Transient forward differentiable rendering (Staircase scene)¶
This tutorial is exactly the same as forward_differentiable_rendering_cbox.ipynb. There are two main changes:
This uses a more complex
staircasescene. Executing this example requires a lot of RAM memory (>>32GB). If your computer cannot handle this requirement you can use the Cornell Box which is cheaper.This scene uses
camera_unwarp=Truefor the integrator. That means that, when computing the time of flight, the distance from the last bounce to the camera is not taken into account. This has the effects that you see events happening in world time instead of camera time (i.e. when they are observed by a camera located at a distance).
Overview¶
This tutorial deals with one of the two forms of differentiable rendering: forward differentiable rendering, in the transient domain. Mathematically, consider \(\textbf{x}\) as your set of scene parameters (e.g. material properties, albedos…). Our transient path tracing algorithms are functions \(f\) that transform them into a time-resolved image \(\textbf{y}\). Given that we are in the transient domain:
With this formulation, forward mode differentiation allows you to compute \(\frac{\partial \textbf{y}(t)}{\partial \textbf{x}_i}\) for one parameter \(\textbf{x}_i \in \textbf{x}\) of the scene parameters. As you will see, the forward mode is very useful to generate visualizations of the effect of individual scene parameters. However if you plan to optimize scene parameters, you’ll probably want to look into
🚀 You will learn:
What is forward differentiable rendering and why it is useful
How to compute transient forward mode derivatives for a given scene parameter
Visualize the output video and understand what it represents
Importing mitransient¶
You need to use a Mitsuba variant that enables Automatic Differentiation (AD). Any *_ad_* variant will do. Common choices should be llvm_ad_rgb for CPU and cuda_ad_rgb for GPU.
[1]:
# If you have compiled Mitsuba 3 yourself, you will need to specify the path
# to the compilation folder
# import sys
# sys.path.insert(0, '<mitsuba-path>/mitsuba3/build/python')
import mitsuba as mi
# To set a variant, you need to have set it in the mitsuba.conf file
# https://mitsuba.readthedocs.io/en/latest/src/key_topics/variants.html
mi.set_variant('llvm_ad_rgb')
import mitransient as mitr
import drjit as dr
print('Using mitsuba version:', mi.__version__)
print('Using Dr.JIT version:', dr.__version__)
print('Using mitransient version:', mitr.__version__)
Using mitsuba version: 3.9.1
Using Dr.JIT version: 1.5.0
Using mitransient version: 1.3.1
Forward mode differentiable rendering begins analogously to reverse mode, by marking the parameters of interest as differentiable (in this example, we do so manually instead of using an Optimizer).
Our goal here is to visualize how changes of the green wall’s color affect the final rendered image. Note that we are rendering this image using a physically-based path tracer, which means that it accounts for globlal illumination, reflection, refraction, and so on. Gradients computated from this simulation will also expose such effects.
Preparing the scene¶
We load the same Cornell Box as most other examples. As will be seen later it’ll be useful to start recording the video a bit earlier (the original value of start_opl was 3.5, now it is 3). For more information about this parameter you can check the documentation for transient_hdr_film.
[2]:
scene = mi.load_file('staircase/scene.xml')
Using the mi.traverse function will give a list of all parameters that can be differentiated. You should look for parameters with the ∂ symbol. D means ns that there are discontinuities involved and, in practice, computation of these gradients can be very noisy/incorrect sometimes.
[3]:
params = mi.traverse(scene)
params
[3]:
SceneParameters[
-----------------------------------------------------------------------------------------------------------------------
Name Flags Type Parent
-----------------------------------------------------------------------------------------------------------------------
allow_thread_reordering bool Scene
object_41988723968.near_clip float PerspectiveCamera
object_41988723968.far_clip float PerspectiveCamera
object_41988723968.shutter_open float PerspectiveCamera
object_41988723968.shutter_open_time float PerspectiveCamera
object_41988723968.film.size ScalarVector2u Film
object_41988723968.film.crop_size ScalarVector2u Film
object_41988723968.film.crop_offset ScalarPoint2u Film
object_41988723968.film.temporal_bins int Film
object_41988723968.film.bin_width_opl float Film
object_41988723968.film.start_opl float Film
object_41988723968.film.exhaustive_scan bool Film
object_41988723968.film.laser_scan_width int Film
object_41988723968.film.laser_scan_height int Film
object_41988723968.x_fov Float PerspectiveCamera
object_41988723968.principal_point_offset_x Float PerspectiveCamera
object_41988723968.principal_point_offset_y Float PerspectiveCamera
object_41988723968.to_world AffineTransform4f PerspectiveCamera
WoodFloorBSDF.brdf_0.diffuse_reflectance.data TensorXu8 BitmapTextureImpl[uint8]
WoodFloorBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
WoodFloorBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodFloorBSDF.brdf_0.eta ∂, D Float RoughPlastic
WoodStairsBSDF.brdf_0.diffuse_reflectance.data TensorXu8 BitmapTextureImpl[uint8]
WoodStairsBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
WoodStairsBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodStairsBSDF.brdf_0.eta ∂, D Float RoughPlastic
WoodChairBSDF.brdf_0.diffuse_reflectance.data TensorXu8 BitmapTextureImpl[uint8]
WoodChairBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
WoodChairBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodChairBSDF.brdf_0.eta ∂, D Float RoughPlastic
WoodLampBSDF.brdf_0.diffuse_reflectance.data TensorXu8 BitmapTextureImpl[uint8]
WoodLampBSDF.brdf_0.diffuse_reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
WoodLampBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WoodLampBSDF.brdf_0.eta ∂, D Float RoughPlastic
LampshadeBSDF.brdf_0.reflectance.data TensorXu8 BitmapTextureImpl[uint8]
LampshadeBSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
BrassBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
BrassBSDF.brdf_0.alpha.value ∂, D Float UniformSpectrum
BrassBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
BrassBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
object_41952856368.nested_bsdf.brdf_0.specular_reflectance.value ∂, D Color3f SRGBReflectanceSpectrum
object_41952856368.nested_bsdf.brdf_0.alpha.value ∂, D Float UniformSpectrum
object_41952856368.nested_bsdf.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
object_41952856368.nested_bsdf.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
object_41952856368.nested_texture.data , D TensorXu8 BitmapTextureImpl[uint8]
object_41952856368.nested_texture.to_uv , D ScalarAffineTransform3f BitmapTextureImpl[uint8]
object_41952856368.scale float BumpMap
ChairSeatBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
GlassBSDF.eta float SmoothDielectric
MagnoliaPaintBSDF.brdf_0.diffuse_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
MagnoliaPaintBSDF.brdf_0.alpha ∂, D Float RoughPlastic
MagnoliaPaintBSDF.brdf_0.eta ∂, D Float RoughPlastic
WhitePaintBSDF.brdf_0.diffuse_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
WhitePaintBSDF.brdf_0.alpha ∂, D Float RoughPlastic
WhitePaintBSDF.brdf_0.eta ∂, D Float RoughPlastic
Painting1BSDF.brdf_0.reflectance.data TensorXu8 BitmapTextureImpl[uint8]
Painting1BSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
Painting2BSDF.brdf_0.reflectance.data TensorXu8 BitmapTextureImpl[uint8]
Painting2BSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
Painting3BSDF.brdf_0.reflectance.data TensorXu8 BitmapTextureImpl[uint8]
Painting3BSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
BlackBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
StainlessSteelBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
StainlessSteelBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
StainlessSteelBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
EmissionBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
CandlesBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
GoldBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
GoldBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
GoldBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
WhiteBSDF.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
CopperBSDF.brdf_0.eta.value ∂, D Color3f SRGBReflectanceSpectrum
CopperBSDF.brdf_0.k.value ∂, D Color3f SRGBReflectanceSpectrum
CopperBSDF.brdf_0.specular_reflectance.value ∂ Color3f SRGBReflectanceSpectrum
WallpaperBSDF.brdf_0.reflectance.data TensorXu8 BitmapTextureImpl[uint8]
WallpaperBSDF.brdf_0.reflectance.to_uv ScalarAffineTransform3f BitmapTextureImpl[uint8]
object_41995881472.silhouette_sampling_weight float Mesh
object_41995881472.faces UInt Mesh
object_41995881472.vertex_positions ∂, D Float Mesh
object_41995881472.vertex_normals ∂, D Float Mesh
object_41995881472.vertex_texcoords ∂ Float Mesh
Copper.silhouette_sampling_weight float OBJMesh
Copper.faces UInt OBJMesh
Copper.vertex_positions ∂, D Float OBJMesh
Copper.vertex_normals ∂, D Float OBJMesh
Copper.vertex_texcoords ∂ Float OBJMesh
White.silhouette_sampling_weight float OBJMesh
White.faces UInt OBJMesh
White.vertex_positions ∂, D Float OBJMesh
White.vertex_normals ∂, D Float OBJMesh
White.vertex_texcoords ∂ Float OBJMesh
Black_0059.silhouette_sampling_weight float OBJMesh
Black_0059.faces UInt OBJMesh
Black_0059.vertex_positions ∂, D Float OBJMesh
Black_0059.vertex_normals ∂, D Float OBJMesh
Black_0059.vertex_texcoords ∂ Float OBJMesh
BrushedAluminium_0001.silhouette_sampling_weight float OBJMesh
BrushedAluminium_0001.faces UInt OBJMesh
BrushedAluminium_0001.vertex_positions ∂, D Float OBJMesh
BrushedAluminium_0001.vertex_normals ∂, D Float OBJMesh
BrushedAluminium_0001.vertex_texcoords ∂ Float OBJMesh
object_41995844608.silhouette_sampling_weight float Mesh
object_41995844608.faces UInt Mesh
object_41995844608.vertex_positions ∂, D Float Mesh
object_41995844608.vertex_normals ∂, D Float Mesh
object_41995844608.vertex_texcoords ∂ Float Mesh
object_41995888640.silhouette_sampling_weight float Mesh
object_41995888640.faces UInt Mesh
object_41995888640.vertex_positions ∂, D Float Mesh
object_41995888640.vertex_normals ∂, D Float Mesh
object_41995888640.vertex_texcoords ∂ Float Mesh
Glass_0103.silhouette_sampling_weight float OBJMesh
Glass_0103.faces UInt OBJMesh
Glass_0103.vertex_positions ∂, D Float OBJMesh
Glass_0103.vertex_normals ∂, D Float OBJMesh
Glass_0103.vertex_texcoords ∂ Float OBJMesh
Gold_0001.silhouette_sampling_weight float OBJMesh
Gold_0001.faces UInt OBJMesh
Gold_0001.vertex_positions ∂, D Float OBJMesh
Gold_0001.vertex_normals ∂, D Float OBJMesh
Gold_0001.vertex_texcoords ∂ Float OBJMesh
object_41995885568.silhouette_sampling_weight float Mesh
object_41995885568.faces UInt Mesh
object_41995885568.vertex_positions ∂, D Float Mesh
object_41995885568.vertex_normals ∂, D Float Mesh
object_41995885568.vertex_texcoords ∂ Float Mesh
object_41995889664.silhouette_sampling_weight float Mesh
object_41995889664.faces UInt Mesh
object_41995889664.vertex_positions ∂, D Float Mesh
object_41995889664.vertex_normals ∂, D Float Mesh
object_41995889664.vertex_texcoords ∂ Float Mesh
ChairSeat.silhouette_sampling_weight float OBJMesh
ChairSeat.faces UInt OBJMesh
ChairSeat.vertex_positions ∂, D Float OBJMesh
ChairSeat.vertex_normals ∂, D Float OBJMesh
ChairSeat.vertex_texcoords ∂ Float OBJMesh
object_41995878400.silhouette_sampling_weight float Mesh
object_41995878400.faces UInt Mesh
object_41995878400.vertex_positions ∂, D Float Mesh
object_41995878400.vertex_normals ∂, D Float Mesh
object_41995878400.vertex_texcoords ∂ Float Mesh
object_41995887616.silhouette_sampling_weight float Mesh
object_41995887616.faces UInt Mesh
object_41995887616.vertex_positions ∂, D Float Mesh
object_41995887616.vertex_normals ∂, D Float Mesh
object_41995887616.vertex_texcoords ∂ Float Mesh
object_41995883520.silhouette_sampling_weight float Mesh
object_41995883520.faces UInt Mesh
object_41995883520.vertex_positions ∂, D Float Mesh
object_41995883520.vertex_normals ∂, D Float Mesh
object_41995883520.vertex_texcoords ∂ Float Mesh
Lampshade_0001.silhouette_sampling_weight float OBJMesh
Lampshade_0001.faces UInt OBJMesh
Lampshade_0001.vertex_positions ∂, D Float OBJMesh
Lampshade_0001.vertex_normals ∂, D Float OBJMesh
Lampshade_0001.vertex_texcoords ∂ Float OBJMesh
object_41995879424.silhouette_sampling_weight float Mesh
object_41995879424.faces UInt Mesh
object_41995879424.vertex_positions ∂, D Float Mesh
object_41995879424.vertex_normals ∂, D Float Mesh
object_41995879424.vertex_texcoords ∂ Float Mesh
Wallpaper_0002.silhouette_sampling_weight float OBJMesh
Wallpaper_0002.faces UInt OBJMesh
Wallpaper_0002.vertex_positions ∂, D Float OBJMesh
Wallpaper_0002.vertex_normals ∂, D Float OBJMesh
Wallpaper_0002.vertex_texcoords ∂ Float OBJMesh
object_41995841536.silhouette_sampling_weight float Mesh
object_41995841536.faces UInt Mesh
object_41995841536.vertex_positions ∂, D Float Mesh
object_41995841536.vertex_normals ∂, D Float Mesh
object_41995841536.vertex_texcoords ∂ Float Mesh
object_41995840512.silhouette_sampling_weight float Mesh
object_41995840512.faces UInt Mesh
object_41995840512.vertex_positions ∂, D Float Mesh
object_41995840512.vertex_normals ∂, D Float Mesh
object_41995840512.vertex_texcoords ∂ Float Mesh
object_41995891712.silhouette_sampling_weight float Mesh
object_41995891712.faces UInt Mesh
object_41995891712.vertex_positions ∂, D Float Mesh
object_41995891712.vertex_normals ∂, D Float Mesh
object_41995891712.vertex_texcoords ∂ Float Mesh
object_41995880448.silhouette_sampling_weight float Mesh
object_41995880448.faces UInt Mesh
object_41995880448.vertex_positions ∂, D Float Mesh
object_41995880448.vertex_normals ∂, D Float Mesh
object_41995880448.vertex_texcoords ∂ Float Mesh
object_41995884544.silhouette_sampling_weight float Mesh
object_41995884544.faces UInt Mesh
object_41995884544.vertex_positions ∂, D Float Mesh
object_41995884544.vertex_normals ∂, D Float Mesh
object_41995884544.vertex_texcoords ∂ Float Mesh
object_41995843584.silhouette_sampling_weight float Mesh
object_41995843584.faces UInt Mesh
object_41995843584.vertex_positions ∂, D Float Mesh
object_41995843584.vertex_normals ∂, D Float Mesh
object_41995843584.vertex_texcoords ∂ Float Mesh
light.bsdf.brdf_0.reflectance.value ∂ Color3f SRGBReflectanceSpectrum
light.emitter.sampling_weight float AreaLight
light.emitter.radiance.value ∂ Color3f SRGBReflectanceSpectrum
light.silhouette_sampling_weight float Rectangle
light.to_world ∂, D AffineTransform4f Rectangle
Wallpaper_0001.silhouette_sampling_weight float OBJMesh
Wallpaper_0001.faces UInt OBJMesh
Wallpaper_0001.vertex_positions ∂, D Float OBJMesh
Wallpaper_0001.vertex_normals ∂, D Float OBJMesh
Wallpaper_0001.vertex_texcoords ∂ Float OBJMesh
WoodFloor_0004.silhouette_sampling_weight float OBJMesh
WoodFloor_0004.faces UInt OBJMesh
WoodFloor_0004.vertex_positions ∂, D Float OBJMesh
WoodFloor_0004.vertex_normals ∂, D Float OBJMesh
WoodFloor_0004.vertex_texcoords ∂ Float OBJMesh
object_41995892736.silhouette_sampling_weight float Mesh
object_41995892736.faces UInt Mesh
object_41995892736.vertex_positions ∂, D Float Mesh
object_41995892736.vertex_normals ∂, D Float Mesh
object_41995892736.vertex_texcoords ∂ Float Mesh
BrushedAluminium_0002.silhouette_sampling_weight float OBJMesh
BrushedAluminium_0002.faces UInt OBJMesh
BrushedAluminium_0002.vertex_positions ∂, D Float OBJMesh
BrushedAluminium_0002.vertex_normals ∂, D Float OBJMesh
BrushedAluminium_0002.vertex_texcoords ∂ Float OBJMesh
object_41995838464.silhouette_sampling_weight float Mesh
object_41995838464.faces UInt Mesh
object_41995838464.vertex_positions ∂, D Float Mesh
object_41995838464.vertex_normals ∂, D Float Mesh
object_41995838464.vertex_texcoords ∂ Float Mesh
object_41995842560.silhouette_sampling_weight float Mesh
object_41995842560.faces UInt Mesh
object_41995842560.vertex_positions ∂, D Float Mesh
object_41995842560.vertex_normals ∂, D Float Mesh
object_41995842560.vertex_texcoords ∂ Float Mesh
WoodLamp.silhouette_sampling_weight float OBJMesh
WoodLamp.faces UInt OBJMesh
WoodLamp.vertex_positions ∂, D Float OBJMesh
WoodLamp.vertex_normals ∂, D Float OBJMesh
WoodLamp.vertex_texcoords ∂ Float OBJMesh
object_41995839488.silhouette_sampling_weight float Mesh
object_41995839488.faces UInt Mesh
object_41995839488.vertex_positions ∂, D Float Mesh
object_41995839488.vertex_normals ∂, D Float Mesh
object_41995839488.vertex_texcoords ∂ Float Mesh
object_41995886592.silhouette_sampling_weight float Mesh
object_41995886592.faces UInt Mesh
object_41995886592.vertex_positions ∂, D Float Mesh
object_41995886592.vertex_normals ∂, D Float Mesh
object_41995886592.vertex_texcoords ∂ Float Mesh
object_41995893760.silhouette_sampling_weight float Mesh
object_41995893760.faces UInt Mesh
object_41995893760.vertex_positions ∂, D Float Mesh
object_41995893760.vertex_normals ∂, D Float Mesh
object_41995893760.vertex_texcoords ∂ Float Mesh
]
We select the WoodChairBSDF.brdf_0.alpha, which refers to how much light the chair reflects.
Importantly, we mark this parameter for gradient tracking, which will allow to compute the derivatives of the image pixels with respect to this parameter later.
[4]:
key = 'WoodChairBSDF.brdf_0.alpha'
# Mark the green wall color parameter as differentiable
dr.enable_grad(params[key])
# Propagate this change to the scene internal state
params.update();
Rendering¶
We can then perform the simulation to be differentiated. In this case, we simply render an image using the mi.render() routine, which will in turn call the scene’s path tracer integrator.
As we have marked the wall color as differentiable, its role in the rendering process is recorded in the autodiff graph.
For now we are not computing any derivatives, this is exactly the same as the standard render tutorials. However, take note of the order of events in the transient video (first the light at the top lights up, then other elements of the scene, etc.)
[5]:
result = mi.render(scene, params, spp=1024) # we use more SPP so the transient result is less noisy
[6]:
mi.util.convert_to_bitmap(result[0])
[6]:
[7]:
transient_tonemap = mitr.vis.tonemap_transient(result[1])
mitr.vis.show_video(transient_tonemap, axis_video=2)
Computing gradients¶
The dr.forward() function will assign a gradient value of 1.0 to the given variables and forward-propagate those gradients through the previously recorded computation graph. During this process, gradient will be accumulated in the output nodes of this graph (here, the rendered image).
Finally, the gradients can be read using dr.grad().
For more detailed information about differentiation with DrJit, please refer to the documentation.
Important detail: Note how, when using a transient integrator, the mi.render function returns two objects (steady state + transient images). That is just a tuple of two objects. You need to pass that same tuple to dr.grad().
[8]:
# Forward-propagate gradients through the computation graph
dr.forward(params[key])
# Fetch the image gradient values
grad_image, grad_video = dr.grad(result)
Visualizing the result (steady state)¶
You get two results: the steady state gradients (the same as in Mitsuba 3), and the transient gradients. First let’s look at the steady state. Because we are tracking changes in the image from a red wall, the gradient is higher on the red channel. You can see all points of the image affected by the red wall. As you may have thought, points closer to the red wall have higher derivatives as the red color bleeding is higher for them.
Note however that gradient values are not necessarily within the [0, 1] range, and so it makes more sense to use a color map and visualize each color channel of the gradient image individually.
[9]:
from matplotlib import pyplot as plt
import matplotlib.cm as cm
import numpy as np
cmap = cm.coolwarm
vlim = np.quantile(np.array(dr.abs(grad_image).array), 0.999)
print(f'Remapping colors within range: [{-vlim:.2f}, {vlim:.2f}]')
fig, axx = plt.subplots(1, 3, figsize=(8, 3))
for i, ax in enumerate(axx):
ax.imshow(grad_image[..., i], cmap=cm.coolwarm, vmin=-vlim, vmax=vlim)
ax.set_title('RGB'[i] + ' gradients')
ax.axis('off')
fig.tight_layout()
plt.show()
Remapping colors within range: [-0.04, 0.04]
Finally, the transient gradient shows when these changes in brightness would take place. Consider one point on the red wall. The time at which its changes would reach the camera is just the time of flight from that point to the camera sensor. So you might think about what happens in this video as:
At \(t=0\) every point in the red wall emits “derivative light”
This “derivative light” propagates through the scene just as normal light would do.
When it reaches the camera, this derivative light is stored in
grad_videowith its corresponding time of flight.
[10]:
grad_video_tonemap = mitr.vis.tonemap_grad_transient(grad_video)
mitr.vis.show_video(grad_video_tonemap)