NVIDIA NeRF Instant: Turn 2D Images into 3D Scenes in Record Time

It was during a session of NVIDIA GTC, last March, that Instant NeRF, a technology based on a neural network capable of transforming a set of 2D photos into high-resolution 3D scenes in a few seconds, was presented. According to the NVIDIA Research team, it would be one of the first models of its kind to combine ultra-fast neural network training and fast rendering.

In its press release, NVIDIA recalls the technological revolution brought by Edwin Land on February 21, 1947 by producing an instant photo with a Polaroid camera. NVIDIA Research pays tribute to him by recreating an iconic photo of Andy Warhol taking an instant photo, transforming it into a 3D scene using Instant NeRF.

Artificial intelligence researchers at NVIDIA Research took the opposite approach with the goal of transforming a set of still images into a 3D digital scene in seconds.

NeRFS, Neural Radiance Fields or neuronal radiation fields

A NeRF is an AI-based technique that creates a three-dimensional scene from 2D images (inverse rendering). Depending on the desired depth, it takes the algorithms hours or days to get results.
According to NVIDIA:

“Collecting data to feed a NeRF is a bit like being a red carpet photographer trying to capture a celebrity’s outfit from all angles – the neural network requires a few dozen images taken from multiple positions around the scene, as well as the camera position of each of them. »

However, if there is a lot of movement when taking photos, the 3D rendering may be blurry, in this case it is better to speed up the shots.

Then the NeRF fills in the blanks, training a small neural network to reconstruct the scene by predicting the color of light radiating in any direction, from any point in 3D space. It can also correct occlusions, when objects seen in some images are hidden in others.

Instant Nerf: 1,000x faster render time

Creating a 3D scene with traditional methods requires at least hours, depending on the complexity and resolution of the visualization. The use of AI has helped to speed up the process and while early NeRFs systems are able to produce sharp artifact-free scenes in minutes, they too require hours of training.

Instant NeRF reduces rendering time: it would only need a few seconds to train on a few dozen still images taken from several angles, then a few tens of milliseconds more to render a 3D view of the stage.

NVIDIA Research has developed a technique called Multi-Resolution Hash Grid Coding, which is optimized to work efficiently on NVIDIA GPUs. Thanks to this new method of encoding inputs and implementing a very fast tiny neural network, researchers can obtain results that combine high quality and speed.

The model was developed using the NVIDIA CUDA toolkit and the Tiny CUDA Neural Networks library. This lightweight neural network offers the advantage that it can be trained and executed on a single NVIDIA GPU, running faster on boards with NVIDIA Tensor Cores.

David Luebke, Vice President of Graphics Research at NVIDIA, said:

“While traditional 3D representations such as polygon meshes are akin to vector images, NeRFs are like bitmap images: they densely capture the way light radiates from an object or within it. ‘a scene. In this sense, Instant NeRF could be as important for 3D as digital cameras, and JPEG compression has been for 2D photography, dramatically increasing the speed, ease and scope of 3D capture and sharing. »

According to NVIDIA, this technology could be used to train robots and self-driving cars or be used in architecture and entertainment to quickly generate digital representations of real environments that creators can modify and expand.

NVIDIA researchers are exploring how this input encoding technique could be used to accelerate several AI challenges, including reinforcement learning, language translation, and general-purpose deep learning algorithms.

We would love to thank the writer of this write-up for this incredible content

NVIDIA NeRF Instant: Turn 2D Images into 3D Scenes in Record Time

Our social media pages here and other pages related to them here.https://www.ai-magazine.com/related-pages/