Transient forward inverse rendering (Cornell Box)

Overview

This tutorial deals with one of the two forms of inverse rendering: forward inverse rendering, in the transient domain. Mathematically, consider \(\textbf{x}\) as your set of scene parameters (e.g. material properties, albedos…). Our transient path tracing algorithms are functions \(f\) that transform them into a time-resolved image \(\textbf{y}\). Given that we are in the transient domain:

\[\textbf{y}(t) = f(\textbf{x}, t)\]

With this formulation, forward mode differentiation allows you to compute \(\frac{\partial \textbf{y}(t)}{\partial \textbf{x}_i}\) for one parameter \(\textbf{x}_i \in \textbf{x}\) of the scene parameters. As you will see, the forward mode is very useful to generate visualizations of the effect of individual scene parameters. However if you plan to optimize scene parameters, you’ll probably want to look into

🚀 You will learn:

• What is forward inverse rendering and why it is useful

• How to compute transient forward mode derivatives for a given scene parameter

• Visualize the output video and understand what it represents

Importing mitransient

You need to use a Mitsuba variant that enables Automatic Differentiation (AD). Any *_ad_* variant will do. Common choices should be llvm_ad_rgb for CPU and cuda_ad_rgb for GPU.

# If you have compiled Mitsuba 3 yourself, you will need to specify the path
# to the compilation folder
# import sys
# sys.path.insert(0, '<mitsuba-path>/mitsuba3/build/python')
import mitsuba as mi
# To set a variant, you need to have set it in the mitsuba.conf file
# https://mitsuba.readthedocs.io/en/latest/src/key_topics/variants.html
mi.set_variant('llvm_ad_rgb')

import mitransient as mitr
import drjit as dr

print('Using mitsuba version:', mi.__version__)
print('Using Dr.JIT version:', dr.__version__)
print('Using mitransient version:', mitr.__version__)

Using mitsuba version: 3.7.0
Using Dr.JIT version: 1.1.0
Using mitransient version: 1.2.0

Forward mode differentiable rendering begins analogously to reverse mode, by marking the parameters of interest as differentiable (in this example, we do so manually instead of using an Optimizer).

Our goal here is to visualize how changes of the green wall’s color affect the final rendered image. Note that we are rendering this image using a physically-based path tracer, which means that it accounts for globlal illumination, reflection, refraction, and so on. Gradients computated from this simulation will also expose such effects.

Preparing the scene

We load the same Cornell Box as most other examples. As will be seen later it’ll be useful to start recording the video a bit earlier (the original value of start_opl was 3.5, now it is 3). For more information about this parameter you can check the documentation for transient_hdr_film.

d = mitr.cornell_box()
d['sensor']['film']['start_opl'] = 3
scene = mi.load_dict(d)

Using the mi.traverse function will give a list of all parameters that can be differentiated. You should look for parameters with the ∂ symbol. D means ns that there are discontinuities involved and, in practice, computation of these gradients can be very noisy/incorrect sometimes.

params = mi.traverse(scene)

params

SceneParameters[
  -------------------------------------------------------------------------------------------------------
  Name                                                Flags    Type              Parent
  -------------------------------------------------------------------------------------------------------
  allow_thread_reordering                                      bool              Scene
  sensor.near_clip                                             float             PerspectiveCamera
  sensor.far_clip                                              float             PerspectiveCamera
  sensor.shutter_open                                          float             PerspectiveCamera
  sensor.shutter_open_time                                     float             PerspectiveCamera
  sensor.film.size                                             ScalarVector2u    Film
  sensor.film.crop_size                                        ScalarVector2u    Film
  sensor.film.crop_offset                                      ScalarPoint2u     Film
  sensor.film.temporal_bins                                    int               Film
  sensor.film.bin_width_opl                                    float             Film
  sensor.film.start_opl                                        int               Film
  sensor.x_fov                                                 Float             PerspectiveCamera
  sensor.principal_point_offset_x                              Float             PerspectiveCamera
  sensor.principal_point_offset_y                              Float             PerspectiveCamera
  sensor.to_world                                              AffineTransform4f PerspectiveCamera
  white.reflectance.value                             ∂        Color3f           SRGBReflectanceSpectrum
  green.reflectance.value                             ∂        Color3f           SRGBReflectanceSpectrum
  red.reflectance.value                               ∂        Color3f           SRGBReflectanceSpectrum
  red-wall.silhouette_sampling_weight                          float             Rectangle
  red-wall.to_world                                   ∂, D     AffineTransform4f Rectangle
  object_94170931281344.silhouette_sampling_weight             float             Mesh
  object_94170931281344.faces                                  UInt              Mesh
  object_94170931281344.vertex_positions              ∂, D     Float             Mesh
  object_94170931281344.vertex_normals                ∂, D     Float             Mesh
  object_94170931281344.vertex_texcoords              ∂        Float             Mesh
  green-wall.silhouette_sampling_weight                        float             Rectangle
  green-wall.to_world                                 ∂, D     AffineTransform4f Rectangle
  light.emitter.sampling_weight                                float             AreaLight
  light.emitter.radiance.value                        ∂        Color3f           SRGBReflectanceSpectrum
  light.silhouette_sampling_weight                             float             Rectangle
  light.to_world                                      ∂, D     AffineTransform4f Rectangle
]

We select the red.reflectance.value, which refers to how much light the red wall (on the left of the scene) reflects.

Importantly, we mark this parameter for gradient tracking, which will allow to compute the derivatives of the image pixels with respect to this parameter later.

key = 'red.reflectance.value'

# Mark the green wall color parameter as differentiable
dr.enable_grad(params[key])

# Propagate this change to the scene internal state
params.update();

Rendering

We can then perform the simulation to be differentiated. In this case, we simply render an image using the mi.render() routine, which will in turn call the scene’s path tracer integrator.

As we have marked the wall color as differentiable, its role in the rendering process is recorded in the autodiff graph.

For now we are not computing any derivatives, this is exactly the same as the standard render tutorials. However, take note of the order of events in the transient video (first the light at the top lights up, then other elements of the scene, etc.)

result = mi.render(scene, params, spp=1024)  # we use more SPP so the transient result is less noisy

mi.util.convert_to_bitmap(result[0])

transient_tonemap = mitr.vis.tonemap_transient(result[1])
mitr.vis.show_video(transient_tonemap, axis_video=2)