Traditionally, game developers use Unity for real-time 2D and 3D games and shun any use of pre-rendered video, partly out of design dogma but also because the MovieTexture system was a nightmare. However, the recently overhauled VideoPlayer functionality means that *video* games are now much more doable. You can now feasibly make that Her Story clone you always dreamed of!
I'm currently making a video game that makes heavy use of video, chopped into many different video clips. It's been fun trying to figure out how to build basic video functionality like playlists and clean transitions between clips, except in Unity.
The thing they don't tell you about re-inventing wheels is that it's fun and exciting to re-invent the wheel, and it gives much more appreciation for the craft that goes into wheels. It was fun to think about how a live telecast cues up video footage on multiple monitors, and how a real-world broadcast works, and I learned a lot about why they do it like that.
Let's talk video in Unity.
|screenshot of DaVinci Resolve interface... it looks complicated, but it's not so bad|
I won't talk about how to edit and render video here, other than to say, you'll want to export to one of Unity's accepted video container file formats (I usually use .MP4) and use a sensible modern codec like H.264. Most platforms support good H.264 decoding, and Resolve can output video into H.264 by default, so don't worry.
Keep in mind that Unity won't be able to import certain video files. Don't use an unusual codec that your operating system doesn't support natively (Windows list here, MacOS list here) and don't use less common file formats like .MKV which aren't supported by Unity. You might have to use a tool like Handbrake to manually transcode the file to a different format, or use a video editor like Resolve to re-render the whole video entirely.
You should sort out this file format stuff and run tests as soon as possible, because different platforms support different video formats. You don't want to be stuck with unreadable video files.
Once it works in the Unity editor, you also need to make sure it works for your players / users too. The easiest way to ensure compatibility for different platforms is to enable Unity's transcoding, which will re-encode the video for compatibility when you make a build, but that also has its own drawbacks with longer build times and lower video quality.
After I got my video clips, I wanted to build a playlist that seamlessly stitched the video clips together and played them in order, without any weird hiccups or black frames or framerate freeze. This ended up being more difficult than I anticipated: if you just tell the VideoPlayer to play another clip when a clip ends, there will usually be a brief stutter as the system decodes and prepares the next video.
My solution here was to actually use TWO VideoPlayers here. Let's call them Monitor A and Monitor B, and let's say Monitor A is currently playing a video of an aquarium, and Monitor B is turned off.
When Monitor A is about 0.1 seconds from finishing its video, I tell Monitor B to start playing a bear video off-screen. Then, when Monitor A finally finishes its clip, I swap the monitors -- I turn off Monitor A, and show Monitor B. Because I "cue up" Monitor B with a 0.1 second head-start, the result is a very smooth instantaneous cut from an aquarium video to a bear video.
(NOTE: On Twitter, Adam Liss recommends using VideoPlayer.Prepare() and the VideoPlayer.prepareCompleted callback for seamless transitions. I don't really know what Prepare() does, and I couldn't figure it out, but it does seem like Unity's intended solution. In the meantime, I like my approach because it also lets me do cross-fade transitions too.)
My SmoothVideoPlaylist.cs example code is embedded below:
Note the "fast forward" cheat code near the bottom. If you're making a game with heavy use of video, DEFINITELY PUT A FAST-FORWARD CHEAT IN THE GAME. Watching the same videos, over and over, will probably make you very upset -- or worse, disinclined to playtest your own game. So make things easier for yourself, and add a simple fast-forward button to speed through clips or sections that already work.
To implement a fast forward cheat, notice how I increase Time.timeScale to 16x speed, but I also set the VideoPlayer playbackSpeed variable to 16x speed as well. That's because the VideoPlayer runs at a different framerate than the game time. You need to manually speed up (or slow down) both settings.
Anyway, I hope that was all useful. Now go forth and make some video games!