Physically Based Rendering
in the Latent Space

Vuk Radovanovic1, Vishesh Gupta1, Adrien Gruson2, Binh-Son Hua1
1Trinity College Dublin, 2École de Technologie Supérieure (ÉTS)
Paper arXiv Video Code

Our method takes a single rendering of a 3D scene, encodes it into the latent space of a pretrained autoencoder, and trains our modified renderer to reproduce those latents. This allows us to render directly into latent space under new camera, lighting, and object configurations.

Abstract

Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation.

We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.

Pipeline: an RGB image is encoded to reference latents; latent scene parameters are optimized with differentiable latent path tracing, refined by a latent refiner, and decoded to RGB.

Method overview. A differentiable renderer with a modified signed rendering equation is optimized to reproduce the encoded latents of a single view. A lightweight refiner then cleans up the rendered latents, mostly along edges, before the VAE decoder produces the final image.

Video

Latent Space Analysis

We observe behaviors that are common across the latents of many renderings, and that hold even when the scene is edited or changed. Many mirror the RGB image, such as distinct materials and reflections. However, some effects cannot be produced by a conventional path tracer: (A) emitted light is signed, (B) geometric edges are strongly highlighted, and (C) occluded regions can carry positive or negative energy, sometimes exceeding their unoccluded surroundings. Our method is designed to address these behaviors.

The 16 latent channels of a Cornell Box rendering, with zoom-ins on signed emitters, geometric edges, and signed shadows.
Green is positive, red is negative, fading to black near zero.

Results

Rendered (pre-refinement)
Ours (refined)
Ground truth

Camera motion in the Living Room, trained on a single view. The rendered latents capture lighting, shading, and color, while the refiner sharpens high-frequency detail.

Pick a scene and an edit from the menus, then drag the slider to step through it. Each edit opens on its training view.

Ours (refined)
Ground truth (path traced)
Our refined rendering (left) next to the path-traced ground truth (right).

Application: Text-Guided Editing

Because our renderer outputs latents, it connects a 3D scene directly to a diffusion model for uses such as score distillation sampling (SDS). We skip the cost of encoding a full-resolution image, and gradients flow straight back into the latent scene parameters. Here, we recolor the teapot by optimizing its reflectance with the prompt "a ⟨color⟩ teapot and a lamp on a table".

OriginalBlueBrownGreenPink
Original Lamp scene with a red teapot. SDS with RGB rendering + encoder
Blue teapot, RGB rendering + encoder. Brown teapot, RGB rendering + encoder. Green teapot, RGB rendering + encoder. Pink teapot, RGB rendering + encoder.
SDS with latent rendering (ours)
Blue teapot, latent rendering. Brown teapot, latent rendering. Green teapot, latent rendering. Pink teapot, latent rendering.

Related Links

  • Latent-NeRF renders a NeRF into a diffusion model's latent space for text-to-3D generation.
  • Splatent reconstructs 3D Gaussian splats in the latent space of a VAE.
  • DreamFusion introduced the score distillation sampling used in our editing application.
  • Mitsuba 3 is the differentiable renderer we extend for signed, 16-channel latent rendering.

Acknowledgements

Scenes are from Benedikt Bitterli's rendering resources, via the Mitsuba 3 gallery; license information is available at the link.

This project is supported by Research Ireland under the Research Ireland Frontiers for the Future Programme - Project, award number 22/FFP-P/11522.

BibTeX

@article{radovanovic2026latent,
  author  = {Radovanovic, Vuk and Gupta, Vishesh and Gruson, Adrien and Hua, Binh-Son},
  title   = {Physically Based Rendering in the Latent Space},
  journal = {Computer Graphics Forum},
  year    = {2026},
  doi     = {10.1111/cgf.70633},
  note    = {To appear},
}