Ahead of GDC 2019, we revealed Neon Noir, a research and development project showcasing real-time mesh ray traced reflections and refractions created with an advanced new version of CRYENGINE’s Total Illumination real-time lighting solution. Needless to say, we did receive a lot of questions about the demo and ray tracing in CRYENGINE. So we wanted to make sure to give you a good overview and took some time to read through your questions in order to answer them our in-depth blog interview

We interviewed Vladimir Kajalin, Principal Rendering Engineer for CRYENGINE, and Ron Frölich Principal 3D Artist for Hunt: Showdown, two of the members of the team that created Neon Noir with the new technology for ray tracing reflections that will be added to CRYENGINE development roadmap in 2019.

First, we spoke to Vladimir Kajalin about implementing the new ray tracing technology into CRYENGINE.

Let’s start at the beginning. What is ray tracing?

Vladimir: Ray tracing is a rendering technique which allows the physically correct simulation of complex lighting behaviors. By tracing rays, we simulate the propagation of rays of light through the scene, and also the interaction of light with the scene’s surfaces.

For decades this method of lighting a scene could only be achieved with offline ray tracers which required hours or even days to render a single frame. Now, with the advances in GPU technology and software engineering, ray tracing is increasingly used in real-time applications like video games, usually in combination with more established rendering techniques. The key points of performance are scene pre-processing, allowing very fast tracing at run-time, and then the de-mosaic process (known as de-noise) which produces high-quality lighting using only a limited amount of ray tracing work.

From 2015 CRYENGINE has used a form of ray tracing for our in-house GI solution, Total Illumination, used in shipped titles like Kingdom Come: Deliverance and our very own Hunt: Showdown. Until now, one of the limitations of CRYENGINE’s GI solution was the level of detail in real-time reflections. While most surfaces were supported well, very smooth and reflective surfaces like mirrors were problematic, forcing game creators to rely on pre-baked cube-maps and local screen space reflections.

Neon Noir demonstrates the next upgrade to our GI system which is real-time mesh ray-traced reflections and refractions. This technique shows high-quality scene meshes and textures perfectly reflected in mirrors and reflective surfaces using GPU ray tracing. Neon Noir is a research and development project, but ray-traced reflections will come to the engine by the end of the year. There are, of course, many applications for ray tracing like ambient occlusion or soft shadows, but we currently focus on using this feature for very reflective surfaces.

What benefits will ray-traced reflections give to CRYENGINE users over traditional methods of rendering?

Vladimir: For gamers, ray tracing means more lifelike visuals in games. For game creators, ray tracing helps to automate and simplify the creation of realistic visuals without having to deal with multiple and often highly approximate rendering techniques.

When it comes to specular reflections, game engines typically use static or dynamic cube-maps, local screen space reflections or dynamic planar reflections. While these established techniques work well in many cases, there are examples where they fail, and game developers have to apply a lot of effort to find workarounds or avoid the inclusion of specific elements.

For example, let’s take self-reflections. Imagine a robot made of reflective chrome metal. Every fragment of the robot can reflect the rest of the body. Or let’s take a scene similar to Neon Noir. Imagine a normal shopping area in a city with a lot of reflective windows, which are arbitrarily oriented and can reflect each other as well as everything else in the scene. For a further example, think about an elevator interior with mirrors opposite each other which reflect infinitely. We see these kinds of scenes every day in real life, but we never see them recreated correctly in games. With ray tracing, both the complex and more typical day-to-day examples work out of the box, without the need for developers to combine multiple approximate algorithms or find a workaround for visual errors.

How did you bring real-time mesh ray tracing to CRYENGINE?

Vladimir: Before we added mesh ray tracing into the engine, we already had a lighting system that used a lot of ray tracing, also known as voxel cone tracing. That meant we didn’t have to build an entirely new lighting system from scratch.

One of the important components of any ray tracing implementation is the ray intersection acceleration structure. Our SVOGI (Total Illumination) system already contained what was necessary, so it was a relatively straightforward step to add the data required for ray tracing. In the current implementation, for every voxel, we store a reference to overlapping triangles, plus the usual information like albedo, opacity, and normal data.

Merging voxel and ray tracing data provides great flexibility. For example, for diffuse rays, true mesh tracing is needed only near the beginning of the ray but for the rest of the ray we can use more efficient cone tracing without any visual artifacts. A similar optimization works with specular rays. Only smooth and clean surfaces like mirrors require true mesh ray tracing. Most low-gloss, less shiny surfaces can be traced much faster simply by tracing voxels, which will achieve the same or better visual output. Another important part of most real-time ray tracing implementations is de-noising, which already exists in CRYENGINE.

Part of the process was delivering proper support for ray hit shading. Based on our existing deferred shading solution we implemented an additional render pass for the shading of rays, with the support of every possible light source we have. By doing this, we make sure that we get proper shading in reflections. We also added support for particle system ray tracing, allowing the reflection of particle effects.

Why is now the right time to be able to bring this technology to the engine?

Vladimir: Now is a good time because the average mainstream GPU has reached the computational power required for our new ray tracing implementation. Additionally, people are becoming more aware of what ray tracing can bring to games, but are lacking a practical solution.

Neon Noir is rendered in real-time, in-editor, on an AMD Vega 56 GPU. Will the latest generation of graphics cards deliver even more impressive results?

Yes. Our current implementation is both API and hardware agnostic. It runs in 1080p with 30 fps on a Vega 56. Reducing the resolution of reflections allows much better performance without too much quality loss. For example in half-resolution mode it runs 1440p / 40+ fps. At the moment we don’t benefit from any additional performance that modern APIs like Vulkan or DX12 or dedicated hardware like the latest generation of graphics cards could give us. But of course, we will optimize the feature to achieve an uplift performance from these APIs and graphics cards.

One of the key factors which helps us to run efficiently on non-RTX hardware is the ability to flexibly and dynamically switch from expensive mesh tracing to low-cost voxel tracing, without any loss in quality. Furthermore, whenever possible we still use all the established techniques like environment probes or SSAO. These two factors help to minimize how much true mesh ray tracing we need and means we can achieve good performance on mainstream GPUs. Another factor that helps us is that our SVOGI system has benefitted from five years of development.

However, RTX will allow the effects to run at a higher resolution. At the moment on GTX 1080, we usually compute reflections and refractions at half-screen resolution. RTX will probably allow full-screen 4k resolution. It will also help us to have more dynamic elements in the scene, whereas currently, we have some limitations. Broadly speaking, RTX will not allow new features in CRYENGINE, but it will enable better performance and more details.

The current implementation is for ray-traced reflections only. Are there plans for ray-traced shadows and ambient occlusion too?

Vladimir: We are always looking into ways to improve the engine and its features, but at the moment it’s a case of managing development priorities to achieve the most significant benefits for users compared to our current approach, and, of course, balancing performance. Currently, typical shadow maps in most game scenes already work well, except area lights where ray tracing will produce noticeable visual improvement. Our current voxel-based ambient occlusion solution is already providing great results with a lower performance overhead compared to true mesh ray tracing. However, what we plan regarding ambient occlusion is a combination of voxel tracing and mesh ray tracing, to provide improved detail at a minimal performance cost. Support for ray-traced reflections was our priority because of the very obvious visual improvement it brings compared to traditional methods like cube maps and SSR.

Ron Frölich gives his impressions of the new feature and discusses how the Neon Noir scene was created.

What are your first impressions of the new ray tracing technology?

Ron: Even while we were working on the demo, the ray tracing implementation improved a lot. Initially, the quality of the reflections wasn’t great and probably wouldn’t have held up considering the scene we created. However, while we built the scene, the team did an amazing job improving the reflections. The lighting in the reflections is now taken from the same cubemaps that light all the non-ray tracing pixels in the scene, lights and particles are working in reflections, and the overall visual quality of the reflections has improved significantly. The quality of the reflections is not yet on par with the renderer’s quality, but we made significant progress towards that during the last couple of weeks. While we worked on the demo, our rendering engineers also managed to add extra features like multiple bounces, and basic ray-traced refractions, which we tried our best to incorporate into the scene.

Performance has been improved throughout the entire process, allowing us to use more triangles for dynamic objects. At this stage, our ray-traced reflections are holding up well enough to present them as an experimental feature.

What options does this new feature give you from a creative point of view, compared to traditional methods of rendering?

Ron: We would normally use a combination of static cubemaps and screen space reflections for a scene like this. Cubemaps are static and generally don’t change during run time, which means that dynamic objects will not be visible if they pass by a reflective surface. Screen space reflections allow us to reflect any pixels that are already rendered on the screen but break down once objects are off-screen. A combination of these two methods still has very obvious drawbacks that are alleviated by ray-traced reflections.

With ray tracing, we are free to use very reflective surfaces without thinking about the drawbacks of the previous methods. We can place large mirror-like surfaces into the level without worrying about whether the scene will change to the extent that will make them look out of place. We can also place animated objects across from mirrors knowing that the animations will be reflected accurately in these mirrors. Animated materials and changes in lighting are also correctly reflected.

Essentially, we can worry less about reflections not matching the current state of the world, which allows us to make the world feel more vivid and dynamic.

What did you want to show with the demo?

Ron: We knew that the focus of the demo would be on ray tracing, but during the very early stages of development, we decided to build a scene that could realistically take place in a game, rather than building a straight up ray tracing showcase. We believe that the end result showcases ray-traced reflections fairly well, without shoving the feature in your face at all times!

While searching for themes that could fit a game, as well as serve as a nice test bed for ray tracing, we stumbled upon an artist called Noe Alonzo who takes stunning photos of Seoul in a neon-noir style. Some of his more muted photos became our main inspiration for this scene. We tried to create a neon-infused film noir aesthetic where colorful neon signs juxtapose with an oppressive, gritty inner-city.

There are a few key elements that highlight the new ray tracing feature very well, like the broken mirror shot, the police barrier reflected in the puddle, and the nightclub. We also used ray tracing more subtly throughout the rest of the scene, for instance, with windows, puddles on the ground, and wet surfaces.

How did you combine other rendering features in the scene, and why did you decide to use ray-tracing only for highly reflective surfaces?

Ron: Even though we tried to optimize the scene as much as possible for ray-tracing, a pixel that doesn’t require ray tracing is still cheaper than one that does. For that reason, we limited ray-traced reflections to very glossy surfaces like mirrors, windows, and puddles. Surfaces that fall below our defined “glossiness” threshold can use traditional cubemaps for reflections and still look convincing. The sidewalks in front of the two nightclubs are good examples of where ray-traced reflections in puddles work well next to typical cubemap reflections on the wet concrete.

Let’s talk about this in more details and mention the bullets/shells on the ground when the drone scans the crime-scene. This scene was actually the core of a lot of debate. The reflections on two of them seem to be missing. What are we seeing here?

Ron: To ensure our ray-traced reflections performed as well as possible we didn’t use the original render geometry for reflections, but rather a less detailed version that is easier to process. This optimization is similar to traditional LODs (level of detail objects) that replace the original render geometry as you move away from it, and the object gets smaller on the screen.

In fact, all the objects in the Neon Noir Demo use low-poly versions of themselves for reflections. As a few people have commented, it is noticeable on the bullets, but it’s a lot harder to spot in most cases. That said, we can easily fix the reflection on the bullets by using more detailed LODs or just not using LODs at all.

Since the bullet casings are such small objects in the world, the SVOGI voxel grid didn’t pick them up and we had to turn them into dynamic ray-traced objects which is usually reserved for moving objects that are updated each frame. When we released the video, the limit for the amount of dynamic objects was a lot stricter than it is now. That’s why only about half the bullets had proper reflections in the video. When we release the scene to the public all bullet casings should be reflected properly.

Some people also pointed out, that a sharp reflection in a puddle on a windowsill in the beginning of the video seems to be faulty. Can you elaborate on that?

Ron: In this case, the LOD used for reflections and the render-geometry didn’t match exactly, so there was a visual difference between the two. It was a mistake and has been corrected by now, but as we can’t update the video on YouTube without deleting it, you have to take our word for now. ;)

What benefits do you think this new feature will give CRYENGINE users when it comes to the engine?

Ron: Like most modern rendering features, ray tracing is another step towards visual realism in games. When used aggressively in special cases like mirrors, and subtly in most other cases like windows and wet surfaces, ray tracing can improve the overall image quality. It can also remove a lot of restrictions for developers in the same way it did for us. A dynamic scene with highly reflective surfaces is fairly hard to achieve with traditional methods. Ray tracing breaks down that barrier with relative ease.

What are your favorite parts in the scene?

Ron: Out of the elements we created specifically to showcase ray tracing, the broken mirror is probably the most effective. The reflection of the drone in the cracked pieces of the mirror works just like you would expect it to and seeing the panels of the club across the street open up in the reflections feels genuinely impressive.

However, I believe the most significant impact of ray tracing is captured in all the subtle ways it adds to the scene. Windows accurately reflect what is on the other side of the street, puddles properly reflect the neon signs above them, and the broken street lights flicker in the reflections of the wet surfaces beneath them. To me, these subtle improvements across the entire image help the scene in an entirely different way than the more obvious showcase elements do.

Thanks, guys!

We hope you enjoyed this insight into how we created Neon Noir. The new Total Illumination technology the team used will be added to the CRYENGINE development roadmap this year. We’ll keep you posted on the development of this feature on the usual channels, so stay tuned.

As ever, we look forward to your feedback in the comments, on the forum, and via Facebook and Twitter. Don’t forget to join the community and CRYENGINE staff members over on Discord.

Are you looking for your next career move? At Crytek, we value diversity and actively encourage people from all kinds of backgrounds and experience levels to apply to our open positions, so join us over at LinkedIn and check out our careers page.

• Your CRYENGINE Team