Neural Super Sampling (NSS)

Neural Super Sampling (NSS) is an innovative, efficient network for temporal super sampling on mobile devices. Content rendered at 540p can be upscaled to 1080p, resulting in up to 50% GPU savings. With our retraining tools content creators and game studios can build derivatives of the model suited to artwork style and performance requirements.

🎥 Neural Super Sampling Demo

Model Details

Neural Super Sampling (NSS) is a parameter prediction model for real-time temporal super sampling developed by Arm, optimized for execution on Neural Accelerators (NX) in mobile GPUs. It enables high-resolution rendering at a lower compute cost by reconstructing high-quality output frames from low-resolution temporal inputs. NSS is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases.

Developed by: Arm Limited
Model type: Temporal image super sampling
License: Other
Repository: Neural Graphics Model Gym
Blog: How Neural Super Sampling Works
Paper: Mobile Neural Super Sampling
Quickstart with ML extensions for Vulkan®: ML extensions for Vulkan® Quickstart Guide
Quickstart with Unreal: Neural Super Sampling Quickstart Guide for NSS integration into Unreal Engine

NSS is under active development with regular updates planned. As we increase the size and diversity of the training dataset we expect to see significant quality improvements. Follow Arm to stay up to date on the latest releases.

The model is released under Arm's AI Model Community License which allows NSS to be retrained on datasets captured from your own content. Future releases of the Neural Graphics Model Gym will provide the tools to capture and convert content for use in (re)retraining..

Uses

NSS can be directly integrated into graphics pipelines using ML extensions for Vulkan®. See included ML SDK for Vulkan scenario for the simplest way to evaluate the model. The scenario includes the necessary pre- and post-processing compute shaders along with a single frame worth of input data.

The recommended way of integrating the model into a graphics pipeline is by using the VGF Library from the ML SDK for Vulkan.

NSS is released under a permissive license designed to foster innovation in the graphics industry and provide differentiation to content creators.

Direct Use

NSS has been integrated into Unreal Engine via two plugins, the NSS Plugin for Unreal Engine and Unreal NNE Plugin for ML extensions for Vulkan. See our quick start guide for step-by-step instructions on how to use NSS in Unreal® Engine.

Out-of-Scope Use

Not suited for non-temporal tasks such as a standalone image upsampling

Bias, Risks, and Limitations

Requires accurate motion vectors and frame history for stable output
May underperform in extremely low framerate scenarios (<10 FPS) with fast camera movement
Padding of the input is needed if input dimensions are not divisible by 8

Recommendations

For ultra-low-FPS use cases, reduce the camera speed, acceleration, or both so that the relative motion between frames mimics the application running at a higher frame rate.

How to Get Started with the Model

This repository contains pre-trained weights and compiled NSS model in VGF format ready for integration with Vulkan applications.

The included Scenario demonstrates full execution of the model on a Vulkan compute-capable system. An Emulation Layer is provided to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver.

Download and Prepare the Scenario

Clone the NSS model repository from Hugging Face

git clone https://huggingface.co/Arm/neural-super-sampling
cd neural-super-sampling

Run the Scenario

The NSS Hugging Face repository includes pre-built Windows® binaries for ML Emulation Layer for Vulkan and Scenario Runner. For other platforms,

build from source following the instructions for Building the Emulation Layer from source and Building the Scenario Runner from source
adapt the instructions below accordingly

Set the required environment variables:

On Windows:

$env:VK_LAYER_PATH="$PWD\bin\windows-x86_64"
$env:VK_INSTANCE_LAYERS="VK_LAYER_ML_Graph_Emulation;VK_LAYER_ML_Tensor_Emulation"

On Linux (assuming the Emulation Layer binaries and JSON files and Scenario Runner executable are copied to bin/linux-x86_64):

export LD_LIBRARY_PATH=$PWD/bin/linux-x86_64:$LD_LIBRARY_PATH
export VK_LAYER_PATH=$PWD/bin/linux-x86_64
export VK_INSTANCE_LAYERS=VK_LAYER_ML_Graph_Emulation:VK_LAYER_ML_Tensor_Emulation

Execute the scenario:

On Windows:

bin\windows-x86_64\scenario-runner.exe --scenario scenario\scenario.json --output out

On Linux:

bin/linux-x86_64/scenario-runner --scenario scenario/scenario.json --output out

Output images are encoded as B10G11R11_UFLOAT. This format is common for framebuffers but not widely supported by image viewers. Use RenderDoc to view these images.

Training and Evaluation

For background on NSS architecture and training read our blog: How Neural Super Sampling Works

Training and evaluation details, including model architecture code, training pipeline, and test configurations, are available at:

Model training code: https://github.com/arm/neural-graphics-model-gym
Examples and tutorials: https://github.com/arm/neural-graphics-model-gym-examples
Sample dataset: https://huggingface.co/datasets/Arm/neural-graphics-dataset

🔎 Model Explorer VGF extension

The VGF extension to Model Explorer provides a simple interface to visualize model and analyse VGF composition.

License

The license for the model source code can be found here.
The license for the content of this repository can be found here

More Information

🧑‍🔬 More technical details about the model can be found in the NSS Guide.

👩🏽‍💻 Our Neural Graphics Development Kit contains engine plugins, model training tools, code examples and extensive developer documentation.

🙋🏻‍♀️ For questions or feedback please start a discussion

Trademark notice

Arm® is a registered trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere.

Windows® is a trademark of the Microsoft group of companies.

Vulkan® is a registered trademark of the Khronos® Group.