The Mill “The Human Race”
“The Human Race” was a proof-of-concept commercial production showcasing the future of filmmaking and potential for real-time processes in broadcast productions. The co-production between The Mill, Chevrolet and Epic Games required a real-time tracking solution for transforming The Mill Blackbird, an adjustable vehicle rig, on the fly with live reactions to environment and lighting changes. The resulting production demonstrated the potential for generating real-time visual effects that can be viewed and customized live, with an array of practical applications for broadcast, film and advertising. Arraiy liaised closely with The Mill during production to develop computer vision tools that automated tracking and integrated with traditional photogrammetry techniques and Unreal Engine for generating real-time graphics. Arraiy augmented the tracking solution with a deep learning approach, enabling all rigid objects to be tracked under conditions where traditional computer vision techniques fail, including different lighting conditions and blurred camera shots. Image processing took place in the cloud, enabling the creative team to feed Arraiy the production data, images and assets ahead of time so that the system learned how to track the set and objects and drive 3D graphics for the live shoot.
By providing Arraiy’s software with ground truths from the get go, the system learned how to automate tracking the positions of objects and sets for augmenting with real-time graphics and effects in Unreal Engine. During the live production, the real-time camera feed was signaled to a computer running Arraiy’s tracking software, which processed the feed between 16-60 milliseconds (1-2 frames of footage) and produced 3D data sent to Unreal Engine forgenerating graphics and effects in near real-time. This early prototype of Arraiy’s proprietary deep learning in computer vision technology demonstrated practical applications and the potential for future opportunities for driving value in content creation.