Hollywood Has Gone VR

Virtual Reality has certainly captured the attention of Hollywood. It has been the plot of many major motion pictures, from The Lawnmower Man, Johnny Mnemonic, The Matrix Trilogy, and Ready Player One.

However, producing a major motion picture entirely in the format has not yet been achieved, although many a major studio has been getting their feet wet, mostly by using 360 and VR as a marketing tool (or gimmick, depending on your point of view).
 

We should be clear on some terms, as they impact what gets made and how, because as always, it boils down to cost and the ROI for a studio. We need to also cover the hurdles in making such content, before discussing what works for the format and its potential future. The following explainer video from Thought Café does a lovely job explaining the many “flavors” of 360 video and VR: https://youtu.be/5EMsP8enoIk

For 360 video, whether 2D or 3D (the technical terms are monoscopic – meaning “one view” - and stereoscopic – meaning “two views”). What makes something appear three dimensional, is a term called parallax. The simplest way to explain this is to put your index finger in front of your nose, and keep it still for a moment. Close one eye while keeping the other open. Then switch closing the other eye, while opening the one that was previously closed. Your finger appears to have shifted position slightly, even though you did not move it. That separation from one eye to the other, is parallax, and having proper parallax is what provides the three-dimensional illusion in cinema, video games, 360 and VR or AR.

When it comes to creating a 360 video in 2D format, it’s not considered to be “true” VR, because the point of view is fixed in the center of the spherical projection. That is, wherever the camera was during recording, that remains your point of view. You can shift your field of view, by looking up, down, left or right, but the perspective does not shift. However, it is immersive, and for many genres, specifically the horror format, it lends itself exceptionally well.

Content made in 2D 360 is easier to produce because of the fewer number of cameras and consequent post production involved. When we jump to 3D 360, the immersion, or feeling of “presence” is considerably increased. Why? The perception of three-dimensional depth, even if we still cannot move from the center of the spherical projection, is truly sufficient to trick the mind into believing it is “there.”

Again, this format is still not considered “true VR,” but it is a technical and creative step up from 2D 360, which most people can easily make with cameras purchased from online now, for less than $400. Arguably, the sharpness and quality of those cameras are considerably limited than a professional rig, but for most social media posts and live-stream marketing from influencers, they’re “good enough” for most people watching on a mobile device.

Producing content in 3D 360 is exponentially more expensive because it more than quadruples the cost, and it increases the complexity considerably. It can be difficult to get the cameras to synchronize properly, everything must remain at a certain distance from the cameras (a limitation of parallax and the laws of physics for lenses and light), and issues such as matching exposure, color, etc., must all be considered. If any of these matters are not taken into account, the 3D effect may be seriously compromised, or it might have to be reconstructed manually, which is a tedious task involving the isolation of every item in the scene, frame by frame.

Since most proper 360 video and VR is shot at 90 frames per second (FPS), although some do go as low as 60 FPS (some do less, but it isn’t recommended), one can see how reconstructing 3D, “fixing it in post,” and handling VFX across what is a multiple series of cameras, gets very expensive rather quickly. One of the most common mistakes is getting too close to the cameras, whether in 2D or 3D, and/or not getting proper coverage between the cameras. This results in a visible split, or seam, which can be distractingly noticeable if action cross from one area to another. It can make an object “ghost,” and/or it can make it seem as if it has been oddly “cut off.” Since most producers are not technically savvy enough, or capable of the proper rigs to review on set, this sometimes is not noticed until afterwards, at great expense, or loss of fidelity.

The next step up from 3D 360 video is still in its early stages, but it is considered to be “true VR.” The closest explanation can be made by comparing video in this format, known as 6DoF (6-degrees-of-freedom) to a video game. In many video games, the worlds are modeled in true three-dimensional space. One can move the characters’ point of view, and not just look up, down, left and right, they can look “around” something as if it had true volume.

A simple example would be walking around a 3D car, squatting down and being able to see the undercarriage, jumping up and looking at the roof, and being able to see it from every angle as one moved around it. This is not possible with a standard 3D 360 rig, unless the camera itself performs these actions, which is not something generally acceptable to most viewers, because it disrupts their sense of presence by not being in control of their own movement.

As one can well imagine, this requires an extraordinary amount of high-fidelity camera production, lenses, and post production. It also requires serious amounts of storage, super-computer capabilities for stitching (connecting the different cameras so it appears as one surrounding image), and nothing short of a super-powered workstation to play it back.

 

 

An appropriate, high-end head-mounted-display (HMD) is also needed, one that has trackers for “room scale.” The latter is a term for being able to track the body of a user in a three-dimensional space, so that in a VR game engine, the 6DoF illusion can be reproduced. It can now be done for 6DoF video, but the amount of data required for this type of VR video versus a game engine, has yet to be cracked for delivery via a wireless device (being tethered to a computer to experience VR is not compelling, because it is severely limiting the movement of the user). There are some companies tackling this problem, with highly promising results, and one should expect this kind of content to be more widely available in less than five years’ time.

Presently, most 360 video should be shown at 4K. Ideally, the resolution should be even higher during capture and playback, but not even the most current high-end displays from HTC Vive and Oculus Rift can support showing content at this resolution. As a consequence, for many people experiencing 360 video on a lower end smartphone, 360 video looks blurry or pixelated. Not only do their phones not have the proper resolution nor pixel density, they do not possess the WiFi speeds, storage, nor playback capabilities to show the content at its fullest potential.

The result is a visual quality most people would not find acceptable. When viewed on a desktop, the video is mildly distorted to suit, and one must drag their mouse on the video itself to shift the field of view. Again, if the connection speed to show 4K isn’t available, even if the monitor itself can only display HD at a maximum, it will still look fuzzy, blurry, and/or pixelated. The latter isn’t immersive since one can still see the environment wherever they are watching the footage, and so the feeling of immersion, of “presence” is completely non-existent.

Watching immersive content that’s meant to evoke a response such as fear, tension, horror, etc., is like watching the same content on a TV, but with all the lights on, in the middle of the day, with a dozen friends talking over each other. The whole point of it is ruined.

Unfortunately, Hollywood has itself, in part, to blame for this, as it has done very little to educate the public about how to properly watch said content. This is particularly true when spending a considerable amount of resources to produce and post something in 360, whether 2D or 3D, and/or in VR.

Indeed, Hollywood may very well be shooting itself in the foot with VR in the same way it did with 3D content. James Cameron got the world to shift to digital projection to accommodate displaying Avatar at the highest quality possible, and ideally in 3D. When it came out in 2009, it took the world by storm for its (still) astounding visuals, and it remains the highest-grossing motion picture to date. Yet, rather than use the higher-end “active shutter” 3D glasses, many cinemas opted for the cheap “polarized” glasses. These dim the brightness and reduce the vibrancy of color by some 50%, in the best of circumstances, and it warps the 3D effect, by making only the seats closest to the center capable of showing the most depth.

When 3D HDTVs came out, the manufacturers cut corners by doing the same, and 3D sadly died a quick death in the marketplace, as people became quickly disillusioned by the quality. The same is happening with 3D 360 and VR right now. Unless manufacturers are quick to iterate their devices to something wireless, independent of a computer, and a lot less expensive, than the future of the format may be pretty grim, if a future for it is to exist at all.

Fortunately, the major manufacturers seem to have gotten the hint, and wireless versions of the higher-end displays are available with adapters, the next iterations have been announced as wireless, as well as “independent” versions of the devices that will not require a PC. Granted, the capability of the wireless devices, in this first version, will not be as powerful as one linked to a workstation with a powerful GPU, but, for most people wanting to watch YouTube 180, 360 video in 2D and 3D, streaming Hollywood movies made for 3D, and relatively simplified game graphics for room-scale VR, it might be a reasonable compromise if the cost can be keep under $400 or less. Studies show the magic number for mass adoption is under $200, so that is likely to be version 2.0 of the wireless “independent low-end” VR devices. Though, by their release in 2020, they should be as powerful as most workstations running what is currently considered “high-end” VR, so the circle goes.

So now that we have covered the various flavors of VR, we come to the critical part about making content in VR. For the sake of simplicity, let us presume the content is either 3D 360 or 6DoF video. The first problem is ROI. Most people in Hollywood still consider VR to be a fad, like 3D (it isn’t), and that the install base is not large enough to merit a production at this time.

This is a common misunderstanding, which really needs to be cleared up if VR production and post is to become a commonality. Firstly, VR is a different type of a storytelling medium all together. It will not replace the love that many have for silent films, black and white films, nor even those in 3D black and white, and 3D color.

VR requires a completely different approach to storytelling, one that directors are loathe to give up, and it is editorial control. A director, their cinematographer and editor, work closely together to frame every shot the way they want you to see the story, and to control what you see, hear and understand at any given point in the narrative. With 360/VR, the audience can look anywhere they wish. For those not used to constructing stories in this manner, it’s a creative nightmare.

Many who are masters of the 16x9 (widescreen) format, approach VR the same way, and it doesn’t work. You cannot depict a couple having a conversation at a bar, which is perhaps where one would like the audience to pay attention, and then have nothing happen in the surrounding space, simply because the narrative requires that the dialogue be heard. Unless there is a specific plot reason the bar is empty, it should be full of extras who provide the scene its ambience.

Those used to 16x9 storytelling would frame the dialogue with closeups, two-shots, and maybe an initial establishing shot. The point is, they’re used to cutting back and forth. Most people are told not to do this in 360, and the results are long stretches of dialogue that seem unnatural to modern audiences. This is especially true of action films that have what rapid-fire editing, something akin to 30 cuts with a Steadicam, just to show a character getting punched and thrown to the ground. VR Filmmakers don’t shoot this way in 360. Also, some people cannot handle the insane camera motions tentpole VFX films, such as those depicted by Marvel, because when replicated in VR, some folks tend to get motion sickness (even though they’re never moving).

The natural expression of VR is exploration. This simply means that more thought needs to be given in the pre-production phase, to make Cinematic VR viable. Not only does special consideration for set-building need to be made to hide crew, lighting rigs, cables, shadows, and cameras that would normally be on the other side of a 2D camera, the 360 camera sees all the way around it. Either this must be corrected as a visual effect, which will cost more time and money, and/or the set needs to be shot in sections, with special consideration for blocking and other movement. For those used to 16x( production, which is already considerably tedious and slow, this only makes it more so, with fussy technical limitations that need to be creatively solved or executed as a visual effect, even if it is something not meant to draw attention to itself.

Another consideration of the exploratory nature of VR, is what is commonly referred to as “branching narratives.” The simplest corollary would be to a video game, where a player has the option to choose in which direction they wish to go, and the narrative adapts around those choices. While the player has a fair amount of what is termed “agency,” a kind of “free will” in these environments, there are actually only so many ways that a game story may branch before eventually returning to some part of an overall linear narrative, that forces the game to a conclusion (of which there may be more than one).

This example of a branching narrative is also another sticking point for the ROI of Cinematic VR. Many people, myself included, have likely watched a film more than once. We have our favorite movies that we like to return to, because we enjoy the story, the characters, the costumes, the VFX, whatever it is about it you may enjoy. For some, these films are evergreen and never really become stale in their minds. However, what if your favorite movies had branching narrative capabilities?

Take, for example, The Lord of the Rings trilogy. What if you could “control” the choices of the path, perhaps as Frodo or Gandalf, that the Fellowship of the Ring took to destroy the Ring of Power? Multiply the number of choices that could branch, the number of characters given agency, over three films, and that’s an extraordinary number of possibilities. Yes, it would cost a little more to make these branches, but relative to the production as a whole, relatively negligible since one could charge more for such a feature. Indeed, since Hollywood Guild Writers, Directors, Actors, etc. make money every time such content is sold, streamed, broadcast, etc., it only means the motion picture becomes “stickier,” like a website, encouraging audiences to return to that content. It is also a Marketer’s dream, because the longer an audience becomes attached to certain kinds of content, the more loyal they are to further iterations (read: sequels, prequels, reboots, etc.).

If you doubt this logic, look at everything sites like Facebook do to keep you engaged as much as possible in their ecosystem with video, news, games, 360 content, VR, and then-some. The longer you remain, the more opportunity they have to learn about you, and the more chances they have to market to you. Such VR-enabled films could therefore have similar technologies built into them, that would permit sponsorships and advertising to change as the tastes of the user/player change over time, and/or with who they might engage in these experiences when in groups. It’s an unprecedented opportunity that refreshes content every time it is replayed and that could last as long as the copyright.

Another consideration is that even if there were a large install base to merit producing a VR feature, there are still many who won’t want to watch a feature in that format, or may not have access to the proper equipment. This is where ignorance of the technical aspects of the medium need to be understood by producers, investors, and those who license content for home video, streaming, etc. It is technically feasible to produce content in the 360/VR format, and then to “extract” a traditional 16x9 version, that would be suitable for playback on the majority of devices around the world. This would also enable a traditional 16x9 filmmaker to have their “Director’s Cut.” They could control the audience’s attention, because they will have the entire 360 canvas. The resolution for proper VR should be greater than 8K, and that is much larger than can fit onto most HDTVs and 4K TVs (even though 8K is around the corner too).

While the “extract” of the VR into 16x9 does help the ROI, shooting an entire VR motion picture would be exponentially more expensive to make. On the other hand, it does present unique ways to incorporate sponsorship and advertising in ways not possible for 16x9 that might help to offset the cost. The issue here is cameras and their related data. There is no “one-size-fits-all” camera for even a typical widescreen motion picture. Also, specialty cameras might be brought in for certain action scenes, in addition to the “main” camera selected for the majority of the production.

The same is true for VR. Different cameras, lenses, and rigs would be used for different purposes, and they can vary considerably in availability, cost, and those who can operate them. Furthermore, with most motion pictures, there’s only one camera rolling, maybe a second, or third in certain sequences, but otherwise just one. Depending on the VR rig, one might have anywhere from 8 to 36 cameras running all at once.
 

If you think the storage required for hundreds of hours of footage for one motion picture is crazy, multiply that for all those cameras, plus their backups. After that, one would need to make the master version that has been stitched, and a duplicate of that for backup, at its highest resolution. From an IT standpoint, this is both a technical and security disaster just waiting to happen. Most production facilities are not used to filming like this, nor are most post production studios set up to handle the unbelievable amount of data throughput required to create visual effects at that resolution, format, in 3D, etc.
 

There are companies focused on VR filmmaking working on new kinds of motion pictures CoDecs (which means Compression/Decompression), that claim to reduce the size 300:1. The claim is that 8K stereo 360 can be compressed, without making it blurry or pixelated, and streamed over current 4G LTE without latency (which is to say, no lag or buffering problems, provided the connection is consistent). They claim the result is even better with WiFi, and with 5G around the corner, they say true 6DoF VR Video is possible, also without any latency. While this has been demonstrated in controlled environments at special conferences, the acid test will be when current 360 and VR productions can switch over to this codec for short form content. If it works, then the major problems of capture, backup, sharing for post production, and of course, distribution are rendered moot. Until then, the process remains prohibitively expensive.

 

 

One of the other remaining hurdles for VR is that, for the most part, it is still a singular experience. Due to the complexity required for computing, most games are not multiplayer. While there is a company with nearly $1B USD in backing to make VR infinitely scalable at any resolution, for any device, while connecting users all over the planet via the cloud, it’s not in wide deployment at the moment. Most human beings are social creatures, and VR by its very nature, is an isolating experience.

This is the genius behind Facebook’s VR Spaces, where anyone with an Oculus Rift can join their friends from anywhere in the world, and share in a communal 3D space. They can hear one another and even use video to see one another in real-time, in addition to their avatars. If we take this a step further, and enable legal streaming video on demand via Facebook (something it is actively negotiating), then friends from anywhere in the world could sit down to watch their favorite 2D, 3D, and VR movies together. The movies are still enjoyable specifically because they are a communal experience, often shared with loved ones and friends. In VR Spaces, friends and family could make choices as a group about the narrative VR selections, or talk and crack jokes in their private VR Spaces, without annoying other people the way that would occur in a traditional cinema.

Furthermore, taking a cue from another chapter and earlier in this one as well, since Facebook knows so much about each individual watching content, even though the group may all be watching the same motion picture, the sponsorships, the ads, logo clothing, etc., might all be altered to cater to the preferences of each viewer on the fly. Again, this is one of the promises of VR storytelling and filmmaking for Advertisers and Marketers that has yet to be tapped in any meaningful way, and which has the possibility of offsetting the cost of the VR project to the point where it becomes on par with traditional 16x9.

This same kind of thinking could be applied to a live or pre-recorded concert, one where millions of people who could not travel to see it or afford a ticket, can watch it on their VR device. Now, add laser-focused programmatic advertising and re-targeting thanks to Google, Facebook, Apple, etc. Multiply the number of times to display brands, logos, ads, etc., during set breaks, pre-, mid, and post show, and the number of times people might opt to re-watch the show, and the chance to show the same or new kinds of ads all over again. A partnership with key musical artists with international appeal, under the banner of something such as LiveNation, or the Colosseum in Las Vegas, etc., would rake in exponential amounts of money for the tour/presentation that have hitherto been unable to be leveraged, but now can thanks to VR.

For those into sports, Fox has already launched social VR to watch sports games, and plenty of folks who have been provided the opportunity to try watching their favorite sport(s) in VR, with a camera placement that is courtside at all key strategic points, is enough to make most sign up for the premium of being able to watch the entire season in VR. Indeed, for companies like ESPN, which have been struggling and are about to break free from their cable contracts, the addition of VR to the “skinny bundle” options might be the very thing that places them back into profitability.

As expressed in another chapter, and hinted at earlier, VR as a format for storytelling, is not going to replace 16x9 widescreen motion pictures. It will simply become another way to tell stories. Hopefully, these stories will be designed for the medium, and not shot in the format simply for the sake of being a gimmick. It’s too easy to make it a cheap trick, not unlike using a ping-pong paddle aimed at 3D audiences. There needs to be a reason to use the format. It needs to be executed well, especially due to the extra requirements and costs for production, post, and on behalf of the consumer. Few things will cause VR to die faster than poorly produced and executed experiences, whether that is cinema or game-engine technology.

As the video game and VFX technologies, companies, and artists merge into this new medium and define it: a new form of storyteller, native to the format, and with a comprehensive understanding of its leverage and limits, will appear in the next few years. These new storytellers will take the format to its breaking point, while flexing and expanding its boundaries, and when that magic happens, Hollywood will have another bright set of stars to shine in the sky.