Towards generalizable and interpretable three-dimensional tracking with inverse neural rendering by Julian Ost & Tanushree Banerjee & Mario Bijelic & Felix Heide ISBN 10.1038/S42256-025-01083-X instant download

Towards generalizable and interpretable three-dimensional tracking with inverse neural rendering by Julian Ost & Tanushree Banerjee & Mario Bijelic & Felix Heide ISBN 10.1038/S42256-025-01083-X instant download

Today, the most successful methods for image-understanding tasks rely

on feed-forward neural networks. Although this approach offers empirical

accuracy, efficiency and task adaptation through fine-tuning, it also comes

with fundamental disadvantages. Existing networks often struggle to

generalize across different datasets, even on the same task. By design, these

networks ultimately reason about high-dimensional scene features, which

are challenging to analyse. This is true especially when attempting to predict

three-dimensional (3D) information based on two-dimensional images. We

propose to recast vision problems with RGB inputs as an inverse rendering

problem by optimizing through a differentiable rendering pipeline over the

latent space of pretrained 3D object representations and retrieving latents

that best represent object instances in a given input image. Specifically,

we solve the task of 3D multi-object tracking by optimizing an image

loss over generative latent spaces that inherently disentangle shape and

appearance properties. Not only do we investigate an alternative take on

tracking but our method also enables us to examine the generated objects,

reason about failure situations and resolve ambiguous cases. We validate

the generalization and scaling capabilities of our method by learning the

generative prior exclusively from synthetic data and assessing camera-based

3D tracking on two large-scale autonomous robot datasets. Both datasets

are completely unseen to our method and do not require fine-tuning.