Exploring Physics and Sound Design in WebGPU with TSL

Introduction

A new experiment utilizing WebGPU and the Three Shading Language (TSL) highlights the potential for sophisticated real-time graphics and acoustics. Dubbed "Garden Anomaly," this project brings together physics simulations, visual enhancements, and procedurally generated sound to create a captivating interactive experience.

The endeavor stems from a desire to leverage modern rendering techniques, specifically through the integration of TSL and the WebGPURenderer. By engaging with these technologies, I aimed to push boundaries while experimenting with engaging visuals and responsive sound enabled by intricate interactions.

The Concept Behind Garden Anomaly

When embarking on a project like this, crucial first steps include a review of existing resources and examples. I examined various Three.js examples and scoured the documentation on TSL. Armed with this knowledge, I set out to create a visually stimulating environment—a challenge I enjoy tackling through experimentation rather than following a strict tutorial.

The idea originated from a prior experiment I had conducted which laid the groundwork for elements like physics interactions and sound generation. I envisioned a playful scenario, where interactions would reveal unexpected results, leading to an engaging user experience.

Rendering Techniques and Material Properties

The project employs the WebGPURenderer exclusively, utilizing TSL for designing the materials displayed. Here’s an overview of the setup:

• Transmissive Shell: Implemented using MeshPhysicalNodeMaterial, featuring properties like transmission and iridescence, which enhance the visual quality.
• Position Node: A custom positionNode facilitates impact bulges through vertex displacement, controlled by an array of vectors.
• Normal Node: This employs an analytic approach to adjust the normals dynamically, enhancing realism with a tangent space normal map.
• Roughness Node: Custom setup with a pixel-modulated roughness map simulates a material guise akin to ice.
• Emissive Node: Generated by a scrollable pattern enhancing the vibrant aesthetics of the scene.
• Instanced Bubbles: Serving as opaque elements, these bubbles react with the surrounding physical environment effectively.

Physics Engine Implementation

The demo's core revolves around a physics engine based on Position Based Dynamics (PBD). The CPU-driven simulations maintain a focus on realistic interactions and behaviors among the elements. Key components include:

Interaction Dynamics: Users can engage with the bubbles, imparting randomized upward forces through their interactions, adding a hands-on element to the experience.

Velocity Integration: By employing semi-implicit Euler methods for gravity and air drag, the simulation maintains smooth motion while realistic effects unfold.

Collision Handling: The system uses iterative adjustments to manage bubble interactions, with properties determined by their relative sizes, ensuring larger bubbles influence smaller ones accordingly.

Restoring Forces: Each bubble recalibrates its velocity based on its prior position, leading to a more natural settling effect in the visual simulation.

Damping Effects: Contact with surfaces generates dampening effects on bubble velocities, imitating realistic friction against the glass surface.

Collision Sound Generation: Only impactful collisions trigger audio feedback, emphasizing the synergy between visuals and sound.

Sound Design via Web Audio API

The audio framework is entirely generated through the Web Audio API, emphasizing the procedural nature of sound creation. This allows for a responsive auditory environment:

Sound Triggers: Each bubble collision that creates visible deformation in the glass produces a corresponding tone, ensuring an immersive feedback loop.

Pitch Variation: The oscillation frequency is dependent on the size of the bubbles, establishing a harmonious connection between visual size and auditory pitch.

Oscillator Configuration: A stack of oscillators produces the sound, with each component characterized by its temporary loudness and decay rates, establishing a rich acoustic texture.

Mixing and Effects: The finished audio signals undergo processing to ensure clarity and depth, using dynamics compression and flanging effects for spatial quality.

Environmental Considerations: Surroundings are enriched with generated elements from Skybox AI, lending to a captivating garden aesthetic that complements the overall design.

Conclusion

The "Garden Anomaly" experiment encapsulates an ambitious blend of artistic vision and technical prowess, making the most of audio-visual integration within modern web technologies. Through engaging with both TSL and WebGPU, the project showcases new possibilities for developers interested in creating interactive experiences that resonate on multiple sensory levels.

Whether it be the delicate interplay of physics, the depth of sound generation, or the vibrant material interactions, this project exemplifies how modern development tools can lead to exciting advancements in interactive design.

Explore the Demo: You can experience this project firsthand via the provided demo link to see how all these elements come together creatively. Check out the code for deeper insights and lessons in modern web development.