Lab 5.5 — Point Clouds & Gaussian Splatting

← Course 4 syllabus · Module 5 · Prev: « Lab 5.4 · Next: Lab 6.1 »

Goal

Close the module with a different rendering paradigm entirely: no meshes, no rasterized surfaces — a scene represented as millions of 3-D Gaussians and rendered by splatting them, sorted and alpha-blended, onto the screen. The source is a pretrained 3D Gaussian Splatting scene (Kerbl et al. 2023 — the paper that made radiance-field-quality rendering real-time) loaded from the reference implementation’s .ply format. Every ingredient is something this module already built the muscles for: each splat is an anisotropic Gaussian whose covariance comes from a stored scale + rotation (Course 1 Section 1’s linear algebra, cashed as \(\Sigma = R S S^T R^T\)); rendering is depth-sorted, alpha-blended instanced quads with the Gaussian falloff evaluated per fragment; the per-frame sort is a GPU radix sort — Lab 1.3’s scan/sort thinking, spelled in-API; color comes from spherical harmonics, at least the DC band. This is compute-plus-graphics interop at production shape — several compute passes feeding an instanced draw, every frame — and it is a fitting final exam for the module before Module 6 turns the profilers on everything.

Prerequisites

  • Lab 5.4: compute passes writing buffers the draw consumes, indirect draws, and the synchronization discipline between them — this lab reuses all of it.
  • Lab 2.5’s blending state knowledge and Lab 4.2’s graph.
  • Module 1 complete — the sort is not re-taught, it is assumed and re-spelled.

Project & environment setup

  • Asset: a pretrained splat scene in .ply format from the 3DGS reference-implementation ecosystem (the authors’ released scenes and other public pretrained scenes exist; download one, place it under assets/splats/, and record the exact source URL and license in docs/asset-sources.md). Start with a small-to-mid scene — first light on millions of splats is needless pain.
  • Engine work: engine/render/splat_* — the .ply loader, a one-time covariance_pass, per-frame sort_pass (multi-dispatch), and the splat_draw pass. This renderer can live beside the mesh path as an alternate scene mode; compositing splats with meshes is Going-further material.
  • Runtime controls: point-budget slider (draw the first N splats by sorted order), falloff-cutoff slider, front-to-back/back-to-front toggle for the correctness experiment, SH-band toggle if you implement beyond DC.

Where results go:

Artifact Path
Notes, memory worksheet, sort-pass design sketch, reconciliation labs/lab-5-5/notes.md
Screenshots (first light, order experiment, budget sweep), captures labs/lab-5-5/captures/
Budget-sweep timings, sort-cost timestamp dumps labs/lab-5-5/benchmarks/

Background

The representation. Each splat stores a position \(\boldsymbol{\mu}\), an anisotropic covariance factored as scale + rotation, an opacity, and SH color coefficients. The covariance is built from a diagonal scale matrix \(S\) and a rotation \(R\) (stored as a quaternion):

\[ \Sigma \;=\; R\, S\, S^{T} R^{T}, \]

symmetric positive-semidefinite by construction — a congruence transform of a diagonal matrix, which is §1 saying “an ellipsoid with axes \(R\)’s columns and radii \(S\)’s entries.” To render, the 3-D Gaussian is projected to a 2-D Gaussian on screen: with \(W\) the view rotation and \(J\) the Jacobian of the projective mapping (the paper’s local affine approximation),

\[ \Sigma' \;=\; J\, W\, \Sigma\, W^{T} J^{T}, \]

whose upper-left 2×2 block is the screen-space footprint — the ellipse the instanced quad must cover, and the falloff the fragment shader evaluates.

  • Rendering = ordered blending. Splats are semi-transparent; the image is the classic over-composite along each ray,

\[ C \;=\; \sum_{i} c_i\, \alpha_i \prod_{j<i} \bigl(1 - \alpha_j\bigr), \]

which is only correct if splats arrive depth-ordered. Hence the per-frame sort: build a depth key per visible splat, radix-sort the splat indices by it, draw instanced quads in sorted order. Back-to-front with standard over-blending, or front-to-back with the premultiplied variant and an accumulated-transmittance formulation — the tasks make you run both and photograph why order is not optional.

  • The sort is Module 1 coming home. Radix sort = for each digit of the key: histogram, exclusive scan of the histogram, scatter — three dispatch shapes per digit, all built from Lab 1.3’s primitives, now written as GLSL/MSL compute in the render graph with barriers between dispatches. Design (bit-width of the depth key, digits per pass, dispatch count) goes in notes.md as a sketch before implementation; the implementation is yours.
  • Spherical harmonics color. The scene stores view-dependent color as SH coefficients per channel: \(c(\mathbf{d}) = \sum_{\ell, m} c_{\ell m} Y_{\ell m}(\mathbf{d})\) over view direction \(\mathbf{d}\). Band 0 (DC, \(Y_{00} = \tfrac{1}{2\sqrt{\pi}}\)) is view-independent base color — the required minimum; higher bands add glints and sheen and are optional here. The full files carry degree-3 coefficients — \((\deg+1)^2 = 16\) per channel — which dominates the memory worksheet below.
  • No geometry shaders. Each splat becomes a quad by instanced vertex-shader expansion — 4 vertices (or 6) per instance, corner offsets derived from \(\Sigma'\)’s extent in the vertex shader. Geometry shaders would be the textbook-2010 answer; MoltenVK has none, and no modern engine misses them.

Tasks

Engine (both backends)

  1. .ply loader. Parse the reference-implementation layout (structure only is specified: position, scale (log-stored — check and note), rotation quaternion, opacity (pre-sigmoid — check and note), SH coefficients) into tightly packed GPU buffers. The memory worksheet in notes.md comes first: fields × floats × splat count, against the file size on disk as a checksum of your understanding.
  2. Covariance precompute pass. A one-time compute pass turning scale+quaternion into whatever packed form your draw path consumes (3-D covariance’s six unique elements, or leave it factored — defend the choice in a line). Numerically verify a handful of splats against a CPU reference calculation.
  3. First light, unsorted. Instanced-quad expansion + Gaussian falloff + blending, no sort — deliberately. Screenshot the shimmering wrongness; it is the before of the experiment.
  4. Depth key + radix sort. Per-frame: compute pass writes a depth key per splat (view-space depth, quantized — note your key width), radix-sort passes reorder an index buffer, draw consumes it. Frustum-cull splats into the sorted set while you are at it — 5.4’s compaction, reused.
  5. The order experiment. Render the same view back-to-front (correct) and front-to-back with naive over-blending (wrong), screenshot both, and explain the difference from the compositing equation — which term breaks. Then, optionally, the correct front-to-back transmittance variant.
  6. Point-budget sweep. The slider draws the first N splats (by significance if the file’s ordering supplies it, else by your sort). Sweep N over ~4 doublings: screenshot quality + frame time each step → the quality/perf curve in benchmarks/.
  7. SH color. DC band minimum; if ambition allows, band 1+ with the view-direction evaluation in the vertex shader, and an A/B screenshot of a glossy surface.

Backend notes — Vulkan

  • Sort passes are back-to-back compute dispatches with buffer barriers between digit phases — the render graph must chain them; this is the stress test of its compute support. Blending state per Lab 2.5; depth test against the mesh scene’s buffer optional, depth write off.
  • Reverse-Z interaction from Lab 5.3: the sort key is view-space depth, not buffer depth — keep the conventions from tangling, and write the one sentence in your notes that untangles them.

Backend notes — Metal

  • Same passes in MSL; threadgroup memory for the histogram/scan phases maps one-to-one from the CUDA shared-memory version. On Apple GPUs, heavy overlapping blending is where the TBDR story from Lab 5.2 gets stress-tested — blending happens in tile memory, and the measured cost profile will differ from the Linux desktop’s RTX 4090; note what you observe for the reconciliation.

Deliverable & expected results

  • A pretrained 3DGS scene rendering correctly (sorted, blended, DC-SH color) on both backends, with the order experiment and budget sweep documented.
  • notes.md: memory worksheet, sort design sketch, order-experiment explanation, reconciliation.
Quantity Predicted Measured
Memory per splat (reference layout, SH degree 3) from the attribute list: \(3 + 3 + 3(\deg{+}1)^2 + 1 + 3 + 4\) floats — count the file’s actual fields and do the ×4-bytes arithmetic by hand; check total ≈ file size …
GPU buffer total for your scene’s splat count splats × your packed per-splat size (worksheet arithmetic) …
Sort cost share of frame vs. splat count qualitative: grows to dominate at high counts — multiple full passes over the key/index buffers per frame vs. one draw’s worth of quad work; the budget sweep will show the takeover …
Frame cost, camera pushed close to the scene fill-rate/blend-bound: cost tracks covered-pixels × overlap, nearly independent of splat count — the overdraw story Lab 6.4 will quantify properly …
Unsorted vs. sorted image unsorted: popping/shimmering, order-dependent color; sorted: stable — matches the compositing-equation argument …

Profiling & performance

Timestamp the three phases separately — key/cull, sort (all digit passes bracketed together), draw — via the HUD (Lab 6.1 hardens this HUD; the phase split is what matters now). The budget sweep doubles as the profiling deliverable: plot (in your notes, by hand is fine) how the sort share and the draw share move as N doubles. Mac: one Xcode GPU capture, looking specifically at the compute-to-render dependency chain and where blending cost lands. The Linux box: Nsight Systems for the dispatch chain; RenderDoc to inspect the sorted index buffer (spot-check monotonic keys — the cheapest correctness probe in the lab). Tracy confirms the CPU is now a bystander: this frame is almost entirely GPU-authored, the destination Module 5 has been driving toward.

Analysis & reconciliation

Reconcile the memory worksheet first: predicted bytes/splat × count vs. actual file size and GPU allocation — a mismatch means a misread field, and finding it is the point. Then the sort: from your key width and digit choice, count the passes and the bytes each moves, and check the measured sort share against that traffic argument (Course 3 M2 thinking, GPU-sized). Explain the order experiment from the compositing product — one paragraph, equation-anchored. Compare the close-range fill-bound behavior across the two GPUs and connect it to the 5.2 architecture story (tile-memory blending vs. DRAM blending). Close the module with the retrospective paragraph: five techniques, one engine — which of the five moved the most work off the CPU, and what single measurement across the module most changed how you think about GPUs?

Going further

  • Composite splats with the mesh scene: depth-test splats against the terrain’s depth buffer and note every convention (reverse-Z again) that has to line up.
  • View-dependent SH (bands 1–3) and an A/B on a reflective surface from the captured scene.
  • Tile-binned splatting: the paper’s actual rasterizer bins splats per screen tile and sorts per tile — sketch (no build) how that maps onto the module’s tiled-light-culling shapes from Lab 5.2, and what it buys over the global sort.
  • Level-of-detail for splats: cut low-significance splats by opacity×scale heuristics and measure quality vs. the budget slider’s naive truncation.
  • Run the point-budget sweep on the Pi 5 as a stretch portability test — the V3DV compute path gets an honest workout.