What is virtual production?

Have you ever heard of the term virtual production? This is a general term for a new production workflow that utilizes virtual space to create real-time video in video production. There are various types of virtual production, one of which is LED display-based virtual production. LED display-based virtual production is a technology that uses walls, ceilings, and floors made of LED displays to synthesize the virtual world and the real world in real time (in-camera). With this technology, the 3DCG background projected on the LED display changes in real time according to the camera position and angle, making it possible to blend naturally with reality.

The evolution of virtual production is the application of the high-quality graphics and real-time performance of traditional game engines used for 3D game development, such as Unity and Unreal Engine. A variety of assets and effects can be used in the game engine, and it can be freely customized and expanded. This allows for increased realism and expressiveness in virtual production.

Utilizing this technology in the broadcasting field also increases the freedom of location selection and improves the efficiency of editing work, reducing costs and time and expanding the range of video expression. In this blog, we will introduce know-how on Unity and Unreal Engine that can support video transmission standards such as SMPTE ST 2110 by linking the game engine and our SDK.

SMPTE ST 2110 compliant M2S SDK

In order to achieve video transmission compliant with SMPTE ST 2110, it is necessary to have the ability to convert media data to the SMPTE ST 2110 format, store it in IP packets, and send and receive it. One way to implement this functionality is a software development kit (SDK) called Macnica Media Streaming (M2S). Based on the SMPTE ST 2110 standard, the M2S SDK implements functions such as media data conversion, packetization, synchronization, and traffic control using software. The benefits of using M2S SDK include the following:

・ Video transmission on a general-purpose server: M2S SDK works with general-purpose server equipment and network cards. Video transmission on a general-purpose server also facilitates collaboration with new technologies such as cloud and edge computing.

-Collaboration with open source: M2S SDK can easily be linked with open source video processing software such as GStreamer and OBS Studio. You can freely combine functions such as video input/output, display, compression, and editing.

​-Customization and expansion: M2S SDK is provided as a library written in C language, making it easy to customize and expand. It is possible to support functions not included in the SMPTE ST 2110 standard and newly added standards.

About cooperation between game engine and M2S SDK

M2S SDK can also be linked with game engines such as Unity and Unreal Engine. Below, we will introduce a demo that displays the received SMPTE ST 2110 stream as a game engine asset on a Windows machine with the M2S SDK and game engine installed.

① Structure of game engine cooperation demo

The diagram above shows the configuration of the game engine cooperation demo. SDI IP Gateway or Video Player is sending ST 2110 stream to Windows machine. A Windows machine receives the stream and processes it with its internal game engine (e.g. Unity, Unreal Engine). The processed game engine project is output and received by the ST 2110 Analyzer.

② ST 2110 receiver internal processing on Win machine

The diagram above shows the GStreamer pipeline inside a Windows machine. GStreamer is a general-purpose multimedia framework. m2svideosink is a GStreamer plugin provided in the M2S SDK, and win32ipcvideosink / win32ipcvideosrc are standard GStreamer plugins. The Windows machine extracts the frame data from the received ST 2110 stream and writes it to the upper video memory (raw video memory) via the API. The game engine, on the other hand, reads its video memory via win32ipcvideosrc and eventually displays it in an object within the asset.

③ Project playback in the game engine editor (ST 2110 stream reception)

This is a screenshot when playing the project in the game engine editor. You can see the incoming stream displayed on the object (virtual screen) installed in the asset.

④ ST 2110 transmitter internal processing on Win machine

This shows the GStreamer pipeline for sending an ST 2110 stream inside a Windows machine. The game engine writes frame data to video memory (raw video memory) via the GStreamer standard plugin win32ipcvideosink. m2svideosink is a GStreamer plugin provided in the M2S SDK, while win32ipcvideosink / win32ipcvideosrc are standard GStreamer plugins. The Windows machine extracts frame data from the received ST 2110 stream and writes it to upper-level video memory (raw video memory) via the API. Meanwhile, the game engine reads its video memory via win32ipcvideosrc and ultimately displays it in an object within the asset.

⑤ ST 2110 stream transmission of game engine project

The image above is a monitor screen capture of the ST 2110 Analyzer, which receives the project sent by the game engine as an ST 2110 stream. You can see the project output from the game engine displayed on the ST 2110 analyzer.

At the end