Transient forward inverse rendering (Staircase scene)¶
This tutorial is exactly the same as forward_inverse_rendering_cbox.ipynb. There are two main changes:
This uses a more complex
staircasescene. Executing this example requires a lot of RAM memory (>>32GB). If your computer cannot handle this requirement you can use the Cornell Box which is cheaper.This scene uses
camera_unwarp=Truefor the integrator. That means that, when computing the time of flight, the distance from the last bounce to the camera is not taken into account. This has the effects that you see events happening in world time instead of camera time (i.e. when they are observed by a camera located at a distance).
Overview¶
This tutorial deals with one of the two forms of inverse rendering: forward inverse rendering, in the transient domain. Mathematically, consider \(\textbf{x}\) as your set of scene parameters (e.g. material properties, albedos…). Our transient path tracing algorithms are functions \(f\) that transform them into a time-resolved image \(\textbf{y}\). Given that we are in the transient domain:
With this formulation, forward mode differentiation allows you to compute \(\frac{\partial \textbf{y}(t)}{\partial \textbf{x}_i}\) for one parameter \(\textbf{x}_i \in \textbf{x}\) of the scene parameters. As you will see, the forward mode is very useful to generate visualizations of the effect of individual scene parameters. However if you plan to optimize scene parameters, you’ll probably want to look into
🚀 You will learn:
What is forward inverse rendering and why it is useful
How to compute transient forward mode derivatives for a given scene parameter
Visualize the output video and understand what it represents
Importing mitransient¶
You need to use a Mitsuba variant that enables Automatic Differentiation (AD). Any *_ad_* variant will do. Common choices should be llvm_ad_rgb for CPU and cuda_ad_rgb for GPU.
[1]:
# If you have compiled Mitsuba 3 yourself, you will need to specify the path
# to the compilation folder
# import sys
# sys.path.insert(0, '<mitsuba-path>/mitsuba3/build/python')
import mitsuba as mi
# To set a variant, you need to have set it in the mitsuba.conf file
# https://mitsuba.readthedocs.io/en/latest/src/key_topics/variants.html
mi.set_variant('llvm_ad_rgb')
import mitransient as mitr
import drjit as dr
print('Using mitsuba version:', mi.__version__)
print('Using Dr.JIT version:', dr.__version__)
print('Using mitransient version:', mitr.__version__)
Using mitsuba version: 3.7.0
Using Dr.JIT version: 1.1.0
Using mitransient version: 1.2.0
Forward mode differentiable rendering begins analogously to reverse mode, by marking the parameters of interest as differentiable (in this example, we do so manually instead of using an Optimizer).
Our goal here is to visualize how changes of the green wall’s color affect the final rendered image. Note that we are rendering this image using a physically-based path tracer, which means that it accounts for globlal illumination, reflection, refraction, and so on. Gradients computated from this simulation will also expose such effects.
Preparing the scene¶
We load the same Cornell Box as most other examples. As will be seen later it’ll be useful to start recording the video a bit earlier (the original value of start_opl was 3.5, now it is 3). For more information about this parameter you can check the documentation for transient_hdr_film.
[2]:
scene = mi.load_file('staircase/scene.xml')
Using the mi.traverse function will give a list of all parameters that can be differentiated. You should look for parameters with the ∂ symbol. D means ns that there are discontinuities involved and, in practice, computation of these gradients can be very noisy/incorrect sometimes.
[3]:
params = mi.traverse(scene)
params
[3]:
SceneParameters[
--------------------------------------------------------------------------------------------------------------------------
Name Flags Type Parent
--------------------------------------------------------------------------------------------------------------------------
allow_thread_reordering bool Scene
object_95105942167008.near_clip float PerspectiveCamera
object_95105942167008.far_clip float PerspectiveCamera
object_95105942167008.shutter_open float PerspectiveCamera
object_95105942167008.shutter_open_time float PerspectiveCamera
object_95105942167008.film.size ScalarVector2u Film
object_95105942167008.film.crop_size ScalarVector2u Film
object_95105942167008.film.crop_offset ScalarPoint2u Film
object_95105942167008.film.temporal_bins int Film
object_95105942167008.film.bin_width_opl float Film
object_95105942167008.film.start_opl float Film
object_95105942167008.x_fov Float PerspectiveCamera
object_95105942167008.principal_point_offset_x Float PerspectiveCamera
object_95105942167008.principal_point_offset_y Float PerspectiveCamera
object_95105942167008.to_world AffineTransform4f PerspectiveCamera
WoodFloorBSDF.brdf_0.diffuse_reflectance.data ∂ TensorXf16 BitmapTextureImpl
WoodFloorBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
WoodFloorBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodFloorBSDF.brdf_0.eta ∂, D Float RoughPlastic
WoodStairsBSDF.brdf_0.diffuse_reflectance.data ∂ TensorXf16 BitmapTextureImpl
WoodStairsBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
WoodStairsBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodStairsBSDF.brdf_0.eta ∂, D Float RoughPlastic
WoodChairBSDF.brdf_0.diffuse_reflectance.data ∂ TensorXf16 BitmapTextureImpl
WoodChairBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
WoodChairBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodChairBSDF.brdf_0.eta ∂, D Float RoughPlastic
WoodLampBSDF.brdf_0.diffuse_reflectance.data ∂ TensorXf16 BitmapTextureImpl
WoodLampBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
WoodLampBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodLampBSDF.brdf_0.eta ∂, D Float RoughPlastic
LampshadeBSDF.brdf_0.reflectance.data ∂ TensorXf16 BitmapTextureImpl
LampshadeBSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
BrassBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
BrassBSDF.brdf_0.alpha.value ∂, D Float UniformSpectrum
BrassBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
BrassBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
object_95105947019344.nested_bsdf.brdf_0.specular_reflectance.value ∂, D Color3f SRGBReflectanceSpectrum
object_95105947019344.nested_bsdf.brdf_0.alpha.value ∂, D Float UniformSpectrum
object_95105947019344.nested_bsdf.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
object_95105947019344.nested_bsdf.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
object_95105947019344.nested_texture.data ∂, D TensorXf16 BitmapTextureImpl
object_95105947019344.nested_texture.to_uv , D ScalarAffineTransform3f BitmapTextureImpl
object_95105947019344.scale float BumpMap
ChairSeatBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
GlassBSDF.eta float SmoothDielectric
MagnoliaPaintBSDF.brdf_0.diffuse_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
MagnoliaPaintBSDF.brdf_0.alpha ∂, D Float RoughPlastic
MagnoliaPaintBSDF.brdf_0.eta ∂, D Float RoughPlastic
WhitePaintBSDF.brdf_0.diffuse_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
WhitePaintBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WhitePaintBSDF.brdf_0.eta ∂, D Float RoughPlastic
Painting1BSDF.brdf_0.reflectance.data ∂ TensorXf16 BitmapTextureImpl
Painting1BSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
Painting2BSDF.brdf_0.reflectance.data ∂ TensorXf16 BitmapTextureImpl
Painting2BSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
Painting3BSDF.brdf_0.reflectance.data ∂ TensorXf16 BitmapTextureImpl
Painting3BSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
BlackBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
StainlessSteelBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
StainlessSteelBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
StainlessSteelBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
EmissionBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
CandlesBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
GoldBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
GoldBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
GoldBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
WhiteBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
CopperBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
CopperBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
CopperBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
WallpaperBSDF.brdf_0.reflectance.data ∂ TensorXf16 BitmapTextureImpl
WallpaperBSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl
light.bsdf.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
light.emitter.sampling_weight float AreaLight
light.emitter.radiance.value ∂ Color3f SRGBReflectanceSpectrum
light.silhouette_sampling_weight float Rectangle
light.to_world ∂, D AffineTransform4f Rectangle
object_95105983444672.silhouette_sampling_weight float Mesh
object_95105983444672.faces UInt Mesh
object_95105983444672.vertex_positions ∂, D Float Mesh
object_95105983444672.vertex_normals ∂, D Float Mesh
object_95105983444672.vertex_texcoords ∂ Float Mesh
Glass_0103.silhouette_sampling_weight float OBJMesh
Glass_0103.faces UInt OBJMesh
Glass_0103.vertex_positions ∂, D Float OBJMesh
Glass_0103.vertex_normals ∂, D Float OBJMesh
Glass_0103.vertex_texcoords ∂ Float OBJMesh
object_95105975169616.silhouette_sampling_weight float Mesh
object_95105975169616.faces UInt Mesh
object_95105975169616.vertex_positions ∂, D Float Mesh
object_95105975169616.vertex_normals ∂, D Float Mesh
object_95105975169616.vertex_texcoords ∂ Float Mesh
White.silhouette_sampling_weight float OBJMesh
White.faces UInt OBJMesh
White.vertex_positions ∂, D Float OBJMesh
White.vertex_normals ∂, D Float OBJMesh
White.vertex_texcoords ∂ Float OBJMesh
Wallpaper_0002.silhouette_sampling_weight float OBJMesh
Wallpaper_0002.faces UInt OBJMesh
Wallpaper_0002.vertex_positions ∂, D Float OBJMesh
Wallpaper_0002.vertex_normals ∂, D Float OBJMesh
Wallpaper_0002.vertex_texcoords ∂ Float OBJMesh
object_95105975365008.silhouette_sampling_weight float Mesh
object_95105975365008.faces UInt Mesh
object_95105975365008.vertex_positions ∂, D Float Mesh
object_95105975365008.vertex_normals ∂, D Float Mesh
object_95105975365008.vertex_texcoords ∂ Float Mesh
object_95105975905072.silhouette_sampling_weight float Mesh
object_95105975905072.faces UInt Mesh
object_95105975905072.vertex_positions ∂, D Float Mesh
object_95105975905072.vertex_normals ∂, D Float Mesh
object_95105975905072.vertex_texcoords ∂ Float Mesh
object_95105960591824.silhouette_sampling_weight float Mesh
object_95105960591824.faces UInt Mesh
object_95105960591824.vertex_positions ∂, D Float Mesh
object_95105960591824.vertex_normals ∂, D Float Mesh
object_95105960591824.vertex_texcoords ∂ Float Mesh
object_95105975153504.silhouette_sampling_weight float Mesh
object_95105975153504.faces UInt Mesh
object_95105975153504.vertex_positions ∂, D Float Mesh
object_95105975153504.vertex_normals ∂, D Float Mesh
object_95105975153504.vertex_texcoords ∂ Float Mesh
object_95105966891296.silhouette_sampling_weight float Mesh
object_95105966891296.faces UInt Mesh
object_95105966891296.vertex_positions ∂, D Float Mesh
object_95105966891296.vertex_normals ∂, D Float Mesh
object_95105966891296.vertex_texcoords ∂ Float Mesh
Copper.silhouette_sampling_weight float OBJMesh
Copper.faces UInt OBJMesh
Copper.vertex_positions ∂, D Float OBJMesh
Copper.vertex_normals ∂, D Float OBJMesh
Copper.vertex_texcoords ∂ Float OBJMesh
object_95105942047008.silhouette_sampling_weight float Mesh
object_95105942047008.faces UInt Mesh
object_95105942047008.vertex_positions ∂, D Float Mesh
object_95105942047008.vertex_normals ∂, D Float Mesh
object_95105942047008.vertex_texcoords ∂ Float Mesh
object_95105962408304.silhouette_sampling_weight float Mesh
object_95105962408304.faces UInt Mesh
object_95105962408304.vertex_positions ∂, D Float Mesh
object_95105962408304.vertex_normals ∂, D Float Mesh
object_95105962408304.vertex_texcoords ∂ Float Mesh
ChairSeat.silhouette_sampling_weight float OBJMesh
ChairSeat.faces UInt OBJMesh
ChairSeat.vertex_positions ∂, D Float OBJMesh
ChairSeat.vertex_normals ∂, D Float OBJMesh
ChairSeat.vertex_texcoords ∂ Float OBJMesh
object_95105971888400.silhouette_sampling_weight float Mesh
object_95105971888400.faces UInt Mesh
object_95105971888400.vertex_positions ∂, D Float Mesh
object_95105971888400.vertex_normals ∂, D Float Mesh
object_95105971888400.vertex_texcoords ∂ Float Mesh
object_95105964590256.silhouette_sampling_weight float Mesh
object_95105964590256.faces UInt Mesh
object_95105964590256.vertex_positions ∂, D Float Mesh
object_95105964590256.vertex_normals ∂, D Float Mesh
object_95105964590256.vertex_texcoords ∂ Float Mesh
WoodFloor_0004.silhouette_sampling_weight float OBJMesh
WoodFloor_0004.faces UInt OBJMesh
WoodFloor_0004.vertex_positions ∂, D Float OBJMesh
WoodFloor_0004.vertex_normals ∂, D Float OBJMesh
WoodFloor_0004.vertex_texcoords ∂ Float OBJMesh
object_95105975368176.silhouette_sampling_weight float Mesh
object_95105975368176.faces UInt Mesh
object_95105975368176.vertex_positions ∂, D Float Mesh
object_95105975368176.vertex_normals ∂, D Float Mesh
object_95105975368176.vertex_texcoords ∂ Float Mesh
object_95105962406800.silhouette_sampling_weight float Mesh
object_95105962406800.faces UInt Mesh
object_95105962406800.vertex_positions ∂, D Float Mesh
object_95105962406800.vertex_normals ∂, D Float Mesh
object_95105962406800.vertex_texcoords ∂ Float Mesh
Lampshade_0001.silhouette_sampling_weight float OBJMesh
Lampshade_0001.faces UInt OBJMesh
Lampshade_0001.vertex_positions ∂, D Float OBJMesh
Lampshade_0001.vertex_normals ∂, D Float OBJMesh
Lampshade_0001.vertex_texcoords ∂ Float OBJMesh
object_95105969456720.silhouette_sampling_weight float Mesh
object_95105969456720.faces UInt Mesh
object_95105969456720.vertex_positions ∂, D Float Mesh
object_95105969456720.vertex_normals ∂, D Float Mesh
object_95105969456720.vertex_texcoords ∂ Float Mesh
Wallpaper_0001.silhouette_sampling_weight float OBJMesh
Wallpaper_0001.faces UInt OBJMesh
Wallpaper_0001.vertex_positions ∂, D Float OBJMesh
Wallpaper_0001.vertex_normals ∂, D Float OBJMesh
Wallpaper_0001.vertex_texcoords ∂ Float OBJMesh
object_95105970302480.silhouette_sampling_weight float Mesh
object_95105970302480.faces UInt Mesh
object_95105970302480.vertex_positions ∂, D Float Mesh
object_95105970302480.vertex_normals ∂, D Float Mesh
object_95105970302480.vertex_texcoords ∂ Float Mesh
object_95105970191968.silhouette_sampling_weight float Mesh
object_95105970191968.faces UInt Mesh
object_95105970191968.vertex_positions ∂, D Float Mesh
object_95105970191968.vertex_normals ∂, D Float Mesh
object_95105970191968.vertex_texcoords ∂ Float Mesh
object_95105958790128.silhouette_sampling_weight float Mesh
object_95105958790128.faces UInt Mesh
object_95105958790128.vertex_positions ∂, D Float Mesh
object_95105958790128.vertex_normals ∂, D Float Mesh
object_95105958790128.vertex_texcoords ∂ Float Mesh
WoodLamp.silhouette_sampling_weight float OBJMesh
WoodLamp.faces UInt OBJMesh
WoodLamp.vertex_positions ∂, D Float OBJMesh
WoodLamp.vertex_normals ∂, D Float OBJMesh
WoodLamp.vertex_texcoords ∂ Float OBJMesh
Black_0059.silhouette_sampling_weight float OBJMesh
Black_0059.faces UInt OBJMesh
Black_0059.vertex_positions ∂, D Float OBJMesh
Black_0059.vertex_normals ∂, D Float OBJMesh
Black_0059.vertex_texcoords ∂ Float OBJMesh
object_95105970247040.silhouette_sampling_weight float Mesh
object_95105970247040.faces UInt Mesh
object_95105970247040.vertex_positions ∂, D Float Mesh
object_95105970247040.vertex_normals ∂, D Float Mesh
object_95105970247040.vertex_texcoords ∂ Float Mesh
object_95105964219840.silhouette_sampling_weight float Mesh
object_95105964219840.faces UInt Mesh
object_95105964219840.vertex_positions ∂, D Float Mesh
object_95105964219840.vertex_normals ∂, D Float Mesh
object_95105964219840.vertex_texcoords ∂ Float Mesh
BrushedAluminium_0001.silhouette_sampling_weight float OBJMesh
BrushedAluminium_0001.faces UInt OBJMesh
BrushedAluminium_0001.vertex_positions ∂, D Float OBJMesh
BrushedAluminium_0001.vertex_normals ∂, D Float OBJMesh
BrushedAluminium_0001.vertex_texcoords ∂ Float OBJMesh
object_95105970251728.silhouette_sampling_weight float Mesh
object_95105970251728.faces UInt Mesh
object_95105970251728.vertex_positions ∂, D Float Mesh
object_95105970251728.vertex_normals ∂, D Float Mesh
object_95105970251728.vertex_texcoords ∂ Float Mesh
Gold_0001.silhouette_sampling_weight float OBJMesh
Gold_0001.faces UInt OBJMesh
Gold_0001.vertex_positions ∂, D Float OBJMesh
Gold_0001.vertex_normals ∂, D Float OBJMesh
Gold_0001.vertex_texcoords ∂ Float OBJMesh
object_95105970240848.silhouette_sampling_weight float Mesh
object_95105970240848.faces UInt Mesh
object_95105970240848.vertex_positions ∂, D Float Mesh
object_95105970240848.vertex_normals ∂, D Float Mesh
object_95105970240848.vertex_texcoords ∂ Float Mesh
BrushedAluminium_0002.silhouette_sampling_weight float OBJMesh
BrushedAluminium_0002.faces UInt OBJMesh
BrushedAluminium_0002.vertex_positions ∂, D Float OBJMesh
BrushedAluminium_0002.vertex_normals ∂, D Float OBJMesh
BrushedAluminium_0002.vertex_texcoords ∂ Float OBJMesh
]
We select the WoodChairBSDF.brdf_0.alpha, which refers to how much light the chair reflects.
Importantly, we mark this parameter for gradient tracking, which will allow to compute the derivatives of the image pixels with respect to this parameter later.
[4]:
key = 'WoodChairBSDF.brdf_0.alpha'
# Mark the green wall color parameter as differentiable
dr.enable_grad(params[key])
# Propagate this change to the scene internal state
params.update();
Rendering¶
We can then perform the simulation to be differentiated. In this case, we simply render an image using the mi.render() routine, which will in turn call the scene’s path tracer integrator.
As we have marked the wall color as differentiable, its role in the rendering process is recorded in the autodiff graph.
For now we are not computing any derivatives, this is exactly the same as the standard render tutorials. However, take note of the order of events in the transient video (first the light at the top lights up, then other elements of the scene, etc.)
[5]:
result = mi.render(scene, params, spp=1024) # we use more SPP so the transient result is less noisy
[6]:
mi.util.convert_to_bitmap(result[0])
[6]:
[7]:
transient_tonemap = mitr.vis.tonemap_transient(result[1])
mitr.vis.show_video(transient_tonemap, axis_video=2)
Computing gradients¶
The dr.forward() function will assign a gradient value of 1.0 to the given variables and forward-propagate those gradients through the previously recorded computation graph. During this process, gradient will be accumulated in the output nodes of this graph (here, the rendered image).
Finally, the gradients can be read using dr.grad().
For more detailed information about differentiation with DrJit, please refer to the documentation.
Important detail: Note how, when using a transient integrator, the mi.render function returns two objects (steady state + transient images). That is just a tuple of two objects. You need to pass that same tuple to dr.grad().
[8]:
# Forward-propagate gradients through the computation graph
dr.forward(params[key])
# Fetch the image gradient values
grad_image, grad_video = dr.grad(result)
Visualizing the result (steady state)¶
You get two results: the steady state gradients (the same as in Mitsuba 3), and the transient gradients. First let’s look at the steady state. Because we are tracking changes in the image from a red wall, the gradient is higher on the red channel. You can see all points of the image affected by the red wall. As you may have thought, points closer to the red wall have higher derivatives as the red color bleeding is higher for them.
Note however that gradient values are not necessarily within the [0, 1] range, and so it makes more sense to use a color map and visualize each color channel of the gradient image individually.
[13]:
from matplotlib import pyplot as plt
import matplotlib.cm as cm
import numpy as np
cmap = cm.coolwarm
vlim = np.quantile(np.array(dr.abs(grad_image).array), 0.999)
print(f'Remapping colors within range: [{-vlim:.2f}, {vlim:.2f}]')
fig, axx = plt.subplots(1, 3, figsize=(8, 3))
for i, ax in enumerate(axx):
ax.imshow(grad_image[..., i], cmap=cm.coolwarm, vmin=-vlim, vmax=vlim)
ax.set_title('RGB'[i] + ' gradients')
ax.axis('off')
fig.tight_layout()
plt.show()
Remapping colors within range: [-0.04, 0.04]
Finally, the transient gradient shows when these changes in brightness would take place. Consider one point on the red wall. The time at which its changes would reach the camera is just the time of flight from that point to the camera sensor. So you might think about what happens in this video as:
At \(t=0\) every point in the red wall emits “derivative light”
This “derivative light” propagates through the scene just as normal light would do.
When it reaches the camera, this derivative light is stored in
grad_videowith its corresponding time of flight.
[10]:
grad_video_tonemap = mitr.vis.tonemap_grad_transient(grad_video)
mitr.vis.show_video(grad_video_tonemap)