Virtual Production: Mixed Reality Filmmaking | NYU Tandon School of Engineering

Virtual Production: Mixed Reality Filmmaking


Mixed Reality

Next Course Run starts: Fall 2023

Format: Technical Intensive


This course will explore the fundamentals of emerging mixed reality cinematic techniques that are quickly becoming integral production tools in both the film and gaming industries. Practical camera and video fundamentals will first be covered and how they translate into Unreal. From there we will quickly build full cinematic scenes in engine using the Marketplace for assets and animated virtual cameras and lighting on the Sequencer.

After establishing an initial cinematic workflow in Unreal, we will then learn how to dynamically control a virtual camera in engine using a mobile device and prep shots for export and integration into Adobe Premiere. Then, in the last week, more advanced forms of camera tracking will be explored, including virtual production green screen techniques that integrate live video into Unreal using a mobile device or DSLR combined with a Vive tracking system.

Key Takeaways

  • Basic camera principles and production terminology
  • Unreal as a cinematic production tool
  • Virtual camera control with a mobile device
  • Realtime chroma key and video compositing
  • Tracking and integrating a DSLR camera into Unreal using Vive components

Who Should Attend

  • Independent Filmmakers: Directors, Cinematographers, Producers, Executive Producers, Technical Directors, VFX Artists/Supervisors, Post-Production Supervisors
  • Experiential Creators: Immersive Storytellers, Game Engine Developers, Creative Technologists, New Media Artists
  • Networks & Media Companies: Content Development Executives, Directors of Video, Interactive Designers, Producers, Directors of Emerging Technology and Innovation
  • Creative Studios & Experience Design Firms: Producers, Technical Directors, Content Directors, Creative Technologists, Technical Producers
  • Corporate/Brand: Producers, Executive Producers, Creative Technologists, Chief Content Officers, Creative Production Managers, Emerging Technology and Innovation Directors
  • Agency: Producers, Executive Producers, Directors of Content Production, Integrated Production Managers, Emerging Technology and Innovation Directors
  • Educators: Teachers, Instructional Designers in Media Production

Technical Requirements

  • An Epic Games Creators account (register here)
  • A computer that runs Unreal Engine 5 (recommended specs)
  • At least 60 GB of free SSD hard drive space
  • A three-button mouse
  • ARKit enabled iPhone or iPad (iOS 11.0 or later and A9 or later processor)
  • Green Screen Backdrop (tabletop or larger setup)
  • Adobe Creative Suite (30-Day free trials are available)
  • Optional:
    • DLSR with HDMI output
    • HDMI Adapter (ex: Elgato Cam Link 4K)
    • Vive Base Stations + Vive Tracker

Course Outline

Module 1

Module 2

Module 3