Project Blue Book

Finishing our very first water project, Bharat, which contained a cargo ship fighting its way through a stormy ocean at night, helped us get our biggest and most challenging FX-sequence yet! The introduction to the final episode of the second season of Project Blue Book.
This production was lead by the show's VFX Supervisor John Gajdecki, Producer Gillian Pearson and Stormborn Studios' VFX Supervisors Manuel Tausch and Goran Pavles.

In short, the protagonists, navy officers on the stern of the USS Wisconsin, are looking out to the ocean, when they see something unusual happening. A trolley that was there a second ago is suddenly gone. Instead, they see a dark object they can't really identify rising out of the ocean and charging at them with unnatural velocity. In their panic, they give order to shoot, causing the unidentified flying object to dive back into the ocean just behind their vessel. The UFO's impact with the water surface then generates a massive splash turning into a tidal wave, which rolls under the navy ship, almost causing it to capsize. Later in the episode, we are to have a second encounter with this mysterious object, just to see it dive again and vanish in the depth of the dark Barents sea.

The episode contained multiple sequences of navy ships and UFO's during day and night time on the open ocean. In order to be able to handle the workload, we hired some of the industry's finest Houdini water-FX TD's, Igor Zanic, Nikola Ilic and Mirco Paolini. For shading and lighting of our FX elements, Nathaniel Holroyd joined the team together with Patrick Conaty, who took on the compositing of the CG elements over the plates and ship renders provided by the client. We outsourced the smaller UFO encounter sequence to Tokyo's VFX studio Megalis, while acting as the main vendor building the bridge between the client and Megalis.

By completing this sequence, our team delivered some of the most awe-inspiring shots of our careers and showed what Stormborn is capable of! We are very excited to share the hurdles we had to overcome with you and invite you to a closer look behind the scenes.

Since everything is happening on and in the ocean, let's start with the obvious.
The Ocean!

Houdini comes with an Ocean toolkit which is second to none! Additionally it has built in procedural volume field generation and modifications, height fields for displacements, compositing and other tools that are perfect for the task. Compositing tools for oceans you might ask? Yes! That's right! The ocean consists of salty water with tons of other impurities and the surface is under a constant influence of currents, convection and wind forces, causing it to generate foam that floats on the surface.

This can of course be simulated, which makes sense for close up water. But since the entire 3D ocean surface is a displacement of a flat surface, which depending on how you look at it is a texture, then why not do the foam patterns as textures as well.It's much lighter, more efficient and looks more realistic, since you can patch a few photos of ocean textures together and randomly place them on the ocean. And since the ocean displacement is a 3D deformation that moves the UV's as well, the texture gets stretchted and contracted which gives a very convincing look. Since we already had developed a working system for our first water show Bharat, we only had to do a few more tweaks to adapt it to the current show. Since we are talking hundreds of patches, you somehow have to stamp them together into one texture, which is a perfect task for a for loop in the compositing context of Houdini.

Foam textures scattered in Hoduini on SOP-level, and flattened as one texture in COPs

Now that could be done in Photoshop or Gimp, but we are Houdini users and like to walk the extra mile. With Houdini, we were able to generate the correct UV's and have a LOD texture system, where we get a very high resolution texture based on distance to camera, blending properly into a mid and low resolution texture placement. On top, since everything is generated in 3D, we can create texture trails, masks etc.

Stormborn being a GPU-rendering studio with a pipeline based on the Redshift render engine, we weren't able to use the Mantra ocean procedurals as they are not supported by Redshift.

Instead, up to this point, we were baking all the Ocean-Spectrum detail into the geometry, with a camera distance based mesh density on SOP level. This was working and rendering fine, but was quite slow to work with. Having Igor Zanic on the team who has actually made a course on using the ocean tools with Redshift, he convinced us to go the render time displacement route. This has a few limitations and needed some RnD to be able to set up with multiple Ocean-Spectrums, but gave us a much faster workflow and shorter render times with more detail.

Spectrum detail and LOD mesh density in SOPs

In the end, we had an HDA which we could point to a geometry, and it would run up the node-tree and collect the displacement data from all the spectrums that are connected to it, and composed a shader that we could plug into the displacement for Redshift. It was a quasi Render Time Procedural equivalent for Redshift.

The client had multiple shots of protagonists in dialogue on the ship. At first we were only to deliver a few seconds of generic ocean for the background in those shots. But soon we realized that a consistency in ocean direction and look was necessary due to the different camera angles and the ship's movement. We figured, we could create a full 360 degree ocean render which allowed our client to put it into their shot composites and rotate it to fit each scene correctly. This was done by rendering the ocean in 8 slices and stitching those together in Nuke.

Stormborn Studios Ocean LatLong CG Environment

LatLong 360 degree ocean rendered in 8 slices

After the show was delivered, we were really excited about this idea and decided to put more work into it and push it to the next level. We generated a full 360 clouds, rain and storm environment that you can enjoy in VR below.

LatLong ocean render with post show added cloud environment. rotate view with your mouse or watch in VR

Next we had to face the feat of generating a realistically and excitingly looking ocean, our biggest challenges were the impact with the massive splash and the follow up shot with the rolling wave generated by it. For the UFO impact shot, the asset changed it's size from 300m to only 50m in size during the previsualization stage. This affected our planning and simulations drastically. The UFO was travelling now faster relative to it's size, and had to impact much closer to the stern of the ship where the actors were standing. It was flying at a speed of 180m/s which initially forced our FLIP simulation to explode due to the incredibly high velocities. Since the splash of the impact would inherit some of the UFO's velocity, water would start flying towards the ship, which required a big container with a volume of ~500,000 cubic meters. To be able to simulate detailed fluids at this scale, a few tricks had to be applied. In the first simulation stage, a low resolution FLIP simulation would be computed whilst applying a strong drag force to achieve believable movement. Procedural noise fields were used to break up the splash which resulted in spikey explosive shapes.

The low resolution point cache was then used to drive a series of high resolution fluid and whitewater simulations that would replace the initial simulation based on velocity and age conditions. Those simulations were split up into multiple containers in order to maintain speedy turnarounds and a high fidelity of detail. Using the same attributes to drive masks, the low resolution base cache and high resolution FLIP caches would be combined to produce the final mesh that would be handed off to the lighting department. Instead of using Houdini's out-of-the-box whitewater solver, a custom whitewater solution was developed for the flying spray in order to guarantee fluid-like behavior and intricate features.

Last but not least smoke was sourced from all fluid point caches to generate a fine mist to which an additional layer of points was advected.

Up to this point, we had the experience of running a ship through a flat ocean and adding wave displacement post sim in combination with the foam and white water generated by the forward motion of the ship. In short, the ship affects the water and generates foam. A one directional relationship of cause and effect. The Houdini ocean tools in combination with some tweaking can get the job done.
In the case of our hero shot, there was a second, third and fourth element of difficulty added on top. Namely the tidal wave which is so big that it affected the entire ship. A bidirectional relationship of cause and effect. The tidal wave then was barrelling and breaking generating a massive splash that covers the camera with foam and droplets flying right into the camera and transitioning into the underwater scene, which later was handled by the client.

So to summarize:

350 frames long high-res ocean with a camera just above the water surface
Ship in full motion generating foam and wakes -> high res water interaction sim
Massive navy ship affected by the tidal wave
Wave barrelling, breaking and splashing
Camera transition from above to under water scenario

Let this sink in for a moment. In the eyes of FX-enthusiasts, it doesn't get much better than that! This is some goose bumps material!

The previz we received was very rough, and the sheer size of the shot and effect were difficult to envision, causing many different opinions, directions and uncertainty. With the tight deadline culture of our industry, and being at the end of the line, this is usually a death sentence for a high quality and successful delivery. So we took matter into our own hands and animated multiple versions with different wave heights, speeds and ship animations.

Different camera, ship and wave animations by Stormborn Studios

Once our client picked out an animated version, we started with researching the guided ocean presets that Houdini comes with. Soon we realized, that this is only the starting point to a much ... much bigger setup. We decided to keep the tank flat and use the guiding forces only for the tidal wave itself. This would give us a good base where the flip matched very closely the guiding geometry.

Being passionate surfers, we've been looking at videos of breaking waves for years. Especially one place in Portugal caught our eye demonstrating the pure force of nature. Nazaré! It has waves that we always wanted to simulate which might fit into this shot perfectly. We collected references and started breaking the elements apart. Timings, speed, motions, crest, lip, spit, foam-ball, shoulder, spray, colors, foam patterns....there is so much to a wave, it is a beautiful and intimidating effect of fluid dynamics!

The idea was to start with the main body of water that breaks, and then, like with all other water FX, add the displacement and white water elements on top, spray and mist sims being the final touch. So the question was, how do we get the body of water to leap forward and barrel? We looked at the solutions, the obvious one being the animation rig described in the Surf's Up paper. It's beautiful method of deforming a flat polygonal plane into the shape of a wave worked well for it's time but since our wave had to dynamically interact with the USS Wisconsin we had to find another solution.
With a few volumetric masks and velocity fields that are generated by the guiding wave geometry, we were able to control when and how fast the water body would leap forward and barrel. Due to the nature of Houdini's FLIP simulation, this resulted in a realistically natural breaking wave shape.

Once the base FLIP simulation was locked down, we developed a custom masking system based on various particle attributes, similar to the masking in our white water workflow. This system allowed us to isolate sections of the underlying simulation and do customized secondary simulations such as white water, extra splashes, crest spray, mist, etc. At the point of shot completion, after cleaning up obsolete data, the shot still contained more than 5 terrabytes of simulation caches.

With that many elements, coming from multiple artists, in multiple wedges and versions, things can get messy very quickly for the lighting department.
Our in-house asset manager sbResourceView ensured our lighters would use the correct caches and versions. Notifications similar to contemporary social media made it easy for the lighter to spot new hand-offs from the FX department, while each FX Artist could tag his caches as published and which Lighting Artist the cache would go to. The lighter then could directly load the caches through the sbResourceView, and the loading mechanism would automatically set the correct range, wedge amount and version.

Stormborn's internal asset manager sbResourceView

Rendering this shot wasn't as simple as just to load all caches and applying shaders and lights. No renderer can handle this much data at once ... and even if it could, it would take ages. Therefore the FX elements had to be split up into separate renders, which would have to be arranged in a logical manner for compositing to be able to stack it all on top of each other.

Given the big shot length, the ship continuously traveling and many different events in terms of FX simulations, this provided to be quite the logical challenge. In the image on the right, you can see how many objects had to be rendered and taken into account. Each of them loads and prepares one or multiple caches for rendering. The darker nodes depict matte objects, red nodes points renders, purple nodes volumes, and pink nodes meshes. Each blue box on the left represents a physical FX element like ocean surface, wave crest, foam ball etc.
The cluster on the right-side of the image are mist elements scattered around the ship and on the ocean. For the mist element alone, we had to separate over 70 wedges into physical elements and spacial area.

35 render passes require a neat node layout to stay organized

35 FX Beauty Renders were handed off in total for this shot to the Compositing Department.

There were two more challenign shots that lead up to the above events. The UFO is emerging out of the ocean, and charging towards camera. Since the shots are viewed through binoculars, the lens was very narrow and covered over a kilometer of water. This required a change in our original spectrum settings. With the procedural nature of Houdini and our foam texture workflow, we were able to overcome this difficulty with very little effort. Even repetitions in wave patterns weren't a problem, since we could just mask multiple spectrums with 2D noise volumes, and our inhouse shader HDA would compose the wave displacement properly. It was one of those moments where all the puzzle pieces fall into place and things go smoother than expected.

During the production time of less than 4 months, Stormborn Studios delivered 6 shots with water simulations, 4 of them comped in-house, 2 comped by the client. Additionally we delivered the LatLong ocean render with mist from two camera angles for the client to use in his own shots. It was used in over 35 shots, of which you can see some in the image below.

Client shots with LatLong ocean render in the background

We have enjoyed this project very much. The complexity and scale of the shots forced us many times to think outside of the box, find workarounds and be extremely creative, but with our talented artists and the strong guidance of VFX Sup John Gajdecki we were able to push through to the goal and exceed expectations.

We hope that you enjoyed this article, last but not least we welcome you to check out our Project Blue Book Breakdown Reel below.