Introduction to Mesh Subdivision

In the realm of 3D renderign, meshes are fundamentally defined by triangles. When exporting assets from modeling software, the geometry is tessellated into a set of primitives that the GPU processes. To increase geometric detail, we often employ subdivision techniques—the process of partitioning a coarse mesh into finer elements without altering the fundamental topology.

A straightforward approach to this involves edge bisection. By iterating through the mesh and identifying triangles where the longest edge exceeds a specific area or length threshold, we can split that edge, generating new vertices and dividing the original face into two smaller ones. Repeating this process recursively allows for a dense mesh derived from a low-poly base.

Deformation and Displacement Concepts

Once a mesh is sufficiently subdivided, the next step is often deformation. While this term broadly covers topological changes, in this context, it refers to the repositioning of vertices to modify the mesh's silhouette. Its critical to maintain the integrity of the connections between faces; otherwise, the mesh may self-intersect or create rendering artifacts due to back-face culling.

The most stable method for deforming a mesh is moving vertices along their normal vectors. A normal vector represents the perpendicular direction relative to the surface at a specific point. By translating vertices inward or outward based on this vector, we can apply displacement.

Challenges with Standard Displacement Mapping

In standard graphics pipelines, displacement mapping is frequently implemented within the vertex shader. The shader reads a value from a texture (the displacement map) and offsets the vertex position along its normal.

// GLSL Example of Shader-based Displacement
float heightSample = texture2D(uDisplacementMap, vUv).r;
float displacementAmount = (heightSample * uScale) + uBias;
vec3 direction = normalize(vNormal);
vec3 newPosition = position + (direction * displacementAmount);

While efficient, this approach has limitations. For the lighting to remain correct on the displaced geometry, the normals must also be updated. Standard pipelines often rely on pre-baked normal maps to approximate these changes. However, if the displacement is significant, these approximations break down, leading to mismatched specular highlights and shadows.

The Importance of Vertex Normals

Normals are the cornerstone of lighting calculations. They define the angle of incidence for light rays. In 3D modeling, vertices can have "split" normals to achieve flat shading (sharp edges) or "averaged" normals for smooth shading.

When a cube is exported with flat shading, the vertices at the corners are duplicated. Although they share the same spatial coordinate, each copy has a normal pointing perpendicular to its specific face. This is crucial for rendering but causes issues during displacement. If we displace these split vertices using their individual face normals, the single corner of the cube will pull apart in multiple directions, tearing the mesh.

To resolve this, we must calculate a unified "geometric normal" for the displacement logic, ensuring that coincident vertices move in unison, while preserving the original "shading normals" for the final render.

Data Structure: The LinkedMesh

To manage this complex relationship between topology and shading, a simple array of vertices is insufficient. I implemented a custom graph-based data structure in Rust, referred to as LinkedMesh. Similar to a half-edge structure but simplified for specific use cases, it maintains bidirectional links between faces, edges, and vertices.

This structure allows for efficient adjacency queries. For example, finding all faces connected to a specific vertex is a direct lookup operation. To handle the complexities of Rust's ownership model, I utilized the slotmap crate. This provides stable, versioned references to entities (indices into an arena), preventing use-after-free bugs while allowing for the dynamic topology modifications required during subdivision.

Calculating Accurate Normals

With the LinkedMesh in place, we can address the normal calculation. The process involves two distinct stages: merging vertices for geometric calculation and splitting them for shading.

First, a spatial search identifies vertices that occupy the same coordinate. These are logically merged into a single node in the graph. The true geometric normal is then calculated by taking the weighted average of the normals of all connected faces. The weight is determined by the angle of the face at that vertex.

// Rust-like pseudo-code for weighted normal calculation
fn calculate_weighted_normal(mesh: &LinkedMesh, vertex_id: VertexID) -> Vec3 {
    let mut accumulated_normal = Vec3::ZERO;
    let mut total_angle = 0.0;

    for face_id in mesh.get_adjacent_faces(vertex_id) {
        let face = mesh.get_face(face_id);
        let angle = mesh.get_angle_at_vertex(face_id, vertex_id);
        accumulated_normal += face.normal * angle;
        total_angle += angle;
    }

    accumulated_normal / total_angle
}

This ensures that when displacement is applied, the mesh expands or contracts uniformly without tearing.

Angle-Based Smooth Shading

After applying displacement, we must restore the visual distinction between sharp edges and smooth surfaces. This is achieved by analyzing the dihedral angle—the angle between the normals of two adjacent faces.

If the angle exceeds a defined threshold (e.g., 30 degrees), the edge is marked as "sharp." The vertices along this edge are then effectively split back into distinct copies, inheriting their original face-specific normals. This mimics the "Auto Smooth" functionality found in professional 3D software. Crucially, during this split, the displaced position remains shared; only the shading normal changes.

Procedural CPU Displacement

Unlike shader-based displacement, performing these calculations on the CPU offers greater flexibility. We can utilize complex algorithms that require knowledge of the entire mesh topology, not just local vertex data.

Using a noise function (such as Simplex or Perlin noise), we can sample a value at every vertex's position. By modulating the displacement strength based on this noise, we can generate organic terrain, jagged rocks, or crystalline structures.

// Procedural noise-based displacement
let noise_sample = fbm_noise(vertex.position, octaves: 4);
let displacement_magnitude = noise_sample * intensity;

// Apply displacement along the calculated geometric normal
vertex.position += vertex.geometric_normal.normalize() * displacement_magnitude;

Advanced Techniques and Refinements

Terrain and Crystal Formation

By tweaking the noise function—for instance, applying a threshold to create "steps" instead of smooth gradients—we can generate crystalline spikes. The combination of high-precision geometric normals and angle-based shading ensures that these sharp features catch light correctly, creating crisp shadows at the base of the spikes.

Handling UV Mapping and Triplanar Textures

Severe mesh deformation often destroys standard UV mappings, causing texture stretching. To counter this, triplanar mapping is employed. This technique projects textures along the X, Y, and Z axes and blends the result based on the surface normal. It produces consistent, non-stretched texturing on procedurally deformed meshes without requiring explicit UV coordinates.

Preserving Sharp Features

During heavy deformation, edges that were originally sharp might become slightly smoothed out by the noise. To maintain the artistic intent, the pipeline allows for explicitly marking edges as "sharp" before processing. These markers lock the shading normals, preventing the smooth shading algorithm from blurring hard creases.