NVIDIA has announced DLSS 3.5, the next major update to the upscaling technology which now accelerates ray tracing and makes it look way better with AI.
NVIDIA DLSS 3.5 Uses AI To Accelerate Ray Tracing, Increased Fidelity Across All RTX GPUs
Today marks 5 years since DLSS was originally introduced at Gamescom 2018. Ever since the tech has become a fundamental component of NVIDIA's RTX GPUs and has seen incremental updates that have made the technology better thanks to the AI supercomputers at NVIDIA who are always hard at work at training the model. Following DLSS 1, the company introduced four major updates which are listed below:
- DLSS 1 (August 2018) - Introduction of Deep Learning Super Sampling To The World
- DLSS 2 (March 2020) - New neural network with better quality and performance
- DLSS 2.3 (November 2021) - Improved detail in motion, reconstruction & temporal stability
- DLSS 3 (October 2022) - New frame generation feature increased performance up to 2X
- DLSS 3.1 (January 2023) - Improved UI stability & quality during fast movement
This year at Gamescom 2023, NVIDIA is taking the wraps off its second major DLSS 3 update known as DLSS 3.5. With DLSS 3.5, NVIDIA is specifically enhancing ray tracing with the same AI model that helps train DLSS. But to understand what DLSS 3.5 does, we have to look at how ray tracing works traditionally.
How's RT Done Today
A typical ray tracing lighting pipeline first generates the scene with materials and geometry. At this point, no lighting effects are applied. This image is then sampled with various lighting effects such as reflections and global illumination.
In the example image below, you can see that the scene isn't fully covered, leaving black pixels which are gaps in lighting information, and these empty spaces are left there because you can't shoot enough rays to fill in every pixel or when they bounce back, they aren't evenly distributed. So the result is that there's always going to be missing pixels despite how many rays are shot at the scene.
This is where denoisers come in which analyze these sampled rays and fill in the missing pixels using spatial and temporal techniques. A game engine can include multiple denoisers for each lightning effect. After the denoisers have finished their work, the RT image is composited & then up-scaled to the native resolution. Denoisers use the following methods:
- Temporal Accumulation: Sample rays across multiple frames (can lead to ghosting or poor RT)
- Spatial Interpolation: Fill in the black gaps & blend with neighboring pixels (can lead to blurry images or non-uniform)
While these denoisers can be hand-tuned for one specific title, they can lead to inaccurate lighting effects, lower-quality global illumination, and lower-quality reflections. Furthermore, the denoisers can also remove high-frequency color data that is required for upscaling, leading to a final image that lacks detail after RT is applied.
This Is Where DLSS 3.5 Comes In!
So NVIDIA has devised a new solution to address these issues and the latest generation of DLSS is here, DLSS 3.5. DLSS 3.5 adds a brand new feature in the DLSS pipeline known as RR or Ray Reconstruction. What Ray Reconstruction does is essentially replace the multiple denoisers which are used in the traditional RT pipeline we discussed above.
NVIDIA states that RR is trained on 5 times more data than DLSS 3 which is why it makes RR a much smarter choice to implement within games and engines. RR is trained to incorporate additional engine data, recognize different ray-traced effects, recognize good and bad temporal & spatial pixels, and retain high-frequency data for upscaling.
Using a close-up comparison between the traditional denoiser and DLSS 3.5 enabled, we can see that the new Super Sampling technology keeps good pixels and discards bad pixels, leading to a much better ray tracing quality versus the denoiser which simply blends the sampled rays & leads to inaccurate lighting. This huge difference is achieved thanks to a huge data set that is trained offline and is far more computationally intensive than what's available in real time.
In multiple comparative images running Portal RTX and Cyberpunk 2077, we can see that DLSS 3.5 with its neural-trained data set offers vastly superior RT lighting which looks more realistic without making the image look blurry or leaving missing lighting points.
Lighting is again just one aspect of RT and reflections also take advantage of the new AI model, offering less blurry images since the data of the original image is retained and not blended together as a denoiser would. This leads to sharper reflections in the final rendered image.
Better RT Quality With Even Better Performance
Well all the RT goodness means that it would be taxing on the GPU right? Well not at all, in fact, enabling DLSS 3.5's Ray Reconstruction would lead to even better performance than vanilla DLSS 3. In one Cyberpunk 2077 example, we can see the performance difference between DLSS off & using the various AI techniques.
With DLSS off and RT-Overdrive (Path Tracing) enabled, the game yields 21 FPS. With DLSS 2 Super Resolution enabled you get a major FPS increase of more than 3X while retaining good IQ (Image Quality), and DLSS 3 further takes it to the next level with even more performance using FG (Frame Generation) technology. And then we have the newly introduced DLSS 3.5 which adds upon these with even better performance than DLSS 3, all while being a visual spectacle of RT goodness. You can see how DLSS 3.5 fills up the lightning on the walls and various surfaces more realistically than the other methods.
NVIDIA explains that one aspect leading to the increased performance is that they can replace multiple deniers with a single AI model and the final performance will depend from game to game. Furthermore, DLSS 3.5 won't be limited to just games and also has major content creation and Pro advantages. The D5 renderer will be getting a DLSS 3.5 update too allowing images to be rendered with RT in all their glory!
On integration, NVIDIA says that they are going to make Ray Reconstruction as easy to implement with the plugins as it is to implement Super Resolution. The developers only need to switch off internal denoisers since the information has to be passed on to the DLSS AI model. So that sounds good from an integration standpoint and we can expect even more RR-enhanced titles in the year ahead.
Cyberpunk 2077 Phantom Liberty DLSS 3.5 Screenshots With Full Path-Tracing:
Alan Wake II DLSS 3.5 Screenshots With Full Path-Tracing:
It Just Works (Across All RTX GPUs)
Coming to support and the good news is that NVIDIA will be offering DLSS 3.5 Ray Reconstruction support across all RTX GPUs starting with the RTX 20 series and going up to RTX 30 and RTX 40 GPUs. The DLSS 3.5 framework will also include the existing Super Resolution, DLAA, and Frame Generation technologies which are supported across their own specific GPU generation. These technologies will harness the NVIDIA Tensor Core technology for unparalleled performance and jaw-dropping visual quality.
Developers will be able to pick and choose between all of these technologies featured within the DLSS 3.5 framework and it's all about their preferences if they want to focus on performance, the image quality, or both.
“Fully ray traced lighting in real-time was a significant milestone on our journey to photorealistic graphics in games, and we met those goals with Cyberpunk 2077’s RT Overdrive Mode, Minecraft with RTX, and Portal with RTX. With the addition of Ray Reconstruction in NVIDIA DLSS 3.5, developers are aided by AI to enhance the worlds they create by increasing image quality further than previously thought possible.”
said Matt Wuebbling, vp of marketing, NVIDIA
"Thanks to DLSS 3.5’s smart technology, fuelled by AI rendering power, you can experience Cyberpunk 2077: Phantom Liberty’s newest location Dogtown at its very best — with sharper images, more accurate lighting, and the highest frame rates possible."
Jakub Knapik, VP of Art, Global Art Director, CD PROJEKT RED
“The new Ray Reconstruction feature in DLSS 3.5 renders our fully ray-traced world more beautifully than ever before, bringing you deeper into the story of Alan Wake 2”
Tatu Aalto, Lead Graphics Programmer Remedy Entertainment
“The integration of DLSS Ray Reconstruction with D5 Render marks our third technical collaboration with NVIDIA, following DLSS SR and FG, which further enhances the image quality of D5 Render's real-time rendering capabilities. We look forward to exploring more possibilities with NVIDIA in the future.”
Jessie Huang, VP, Head of Marketing, D5 Render
When Is It Out?
NVIDIA DLSS 3.5 will be landing this fall in three major titles including Cyberpunk 2077 (with the Phantom Liberty update), Portal with RTX, and also Alan Wake 2 which is awesome! In addition to games, the Omniverse Platform, Chaos Vantage, and the aforementioned D5 Render will also use DLSS 3.5 model moving forward.
