Transforming 2D Images into Depth-Illuminated Experiences with Three.js

Editor’s note: This tutorial continues our focus on fostering creativity within the Three.js community while anticipating the inaugural Three.js Conference in Paris this September. In this article, Dominik Fojcik introduces techniques for animating 2D images to create depth, light, and shadows using Three.js, TSL, and WebGPU.

Effects that manipulate images are ubiquitous online, yet often they lack the depth and immersion that genuinely captures attention. Imagine being able to manipulate an image as if you could reach through the screen. This transformation becomes possible with effective depth mapping.

Understanding Depth Maps

The foundation of this technique lies in depth maps, which reveal the distances of various elements within an image. To generate a depth map, you can employ models like Depth Anything 3, or utilize the Depth Generation Tool created specifically for this tutorial.

It's important to note that depth maps are often in 8-bit formats, resulting in only 256 possible values, which can lead to visible granularity in shading. To ameliorate this issue, depth values should be converted to float precision and subjected to a blurring process that smooths out abrupt transitions.

const { data } = context.getImageData(0, 0, width, height)
const values = new Float32Array(width * height)
for (let i = 0; i < values.length; i++) {
  values[i] = data[i * 4] / 255
}

// blur to diminish 8-bit steps
smoothBands({ values, width, height }, radius)
const halfFloats = new Uint16Array(values.length)
for (let i = 0; i < halfFloats.length; i++) {
  halfFloats[i] = DataUtils.toHalfFloat(values[i])
}

Creating a Normal Map

So, how do we make light interact with our images in a three-dimensional way? Through the use of normal maps, which define the orientation of the surface at every pixel rather than constructing actual 3D models. This method allows the light simulation to respond appropriately, highlighting or shading pixels based on their orientation.

The normal map is derived from the depth map, where gradients are calculated to find surface slopes. This approach creates the illusion of depth through flat images.

const depthGradient = Fn(([vUv, step]) => {
  const left = smoothDepthNode.sample(vUv.sub(alongX)).r
  const right = smoothDepthNode.sample(vUv.add(alongX)).r
  const bottom = smoothDepthNode.sample(vUv.sub(alongY)).r
  const top = smoothDepthNode.sample(vUv.add(alongY)).r

  return vec2(right.sub(left), top.sub(bottom)).mul(0.5)
})

Normal maps can further be enhanced by combining brightness variations from the original image, allowing gradients to simulate detail without actual geometric features.

Implementing Shadows

While normals help define brightness, they don't inherently produce shadows. By tracing rays from each pixel toward the light source, one can identify whether a pixel is shadowed by checking the depth map for obstacles.

const occlusion = float(0).toVar()
Loop(SHADOW_STEPS, ({ i }) => {
  const travel = float(i).add(1).div(SHADOW_STEPS)
  const rayDepth = surfaceDepth.add(headroom.mul(travel))
  const blockerDepth = smoothDepthNode.sample(vUv.add(sweep.mul(travel))).r
  const softness = uShadowSoftness.mul(travel.mul(SOFTNESS_GROWTH).add(1))
  const blocked = blockerDepth.sub(rayDepth).div(softness).clamp(0, 1)

  occlusion.assign(occlusion.max(blocked))
})

Combining Effects

With depth data, normal mapping, and shadow information established, these can be combined within a MeshPhongNodeMaterial to yield realistic rendering:

const material = new MeshPhongNodeMaterial({ specular: 0x000000 })
material.colorNode = diffuseNode(vUv, depth) // our image
material.normalNode = normalNode(vUv)         // fake normals based on depth
material.aoNode = shadowNode(vUv, depth)      // adding shadows

Looking Ahead

This guide aims to spark innovative ideas for creating interactive visual effects using depth maps. The potential within this field remains vast and ripe for exploration.

Best of luck to everyone attending the upcoming Three.js conference! While I won't be able to attend this year, I look forward to participating next time!