Lab 2.4 — Textures, Samplers & Mipmaps
← Course 4 syllabus · Module 2 · Prev: « Lab 2.3 · Next: Lab 2.5 »
Goal
Images as first-class GPU resources: load a texture from disk with stb_image, move it into GPU memory through Vulkan’s image layouts and transitions vs. Metal’s tracked resources, get sRGB vs. linear right on the first try rather than by superstition (Course 1 §16), configure samplers — filter modes, address modes, anisotropy — and build the mipmap chain, on Vulkan by a hand-written blit chain and on Metal by the blit encoder’s generateMipmaps. The intellectual center of the lab is that mipmapping is not a graphics trick: minification is resampling, and the mip chain is a prefiltered pyramid that keeps texture sampling on the right side of the sampling theorem (Course 1 §8). You will build a torture test that makes aliasing undeniable, then watch each sampler setting buy its correction — the sampling theorem, rendered.
Recommended reading
- Vulkan Tutorial — the “Texture mapping” sequence: images, image views, layout transitions and barriers, samplers, and the “Generating mipmaps” chapter (the blit-chain this lab implements). This is the first serious contact with pipeline barriers — read slowly.
- MbT — the textures and materials chapters: loading,
MTLTexture, samplers, and mipmaps on the Metal side (title-level; 5th-ed. numbering — confirm against the copy in hand). - Halladay — the texturing chapters: UVs, sampling, and what filtering looks like on screen; the best pictures-first telling of address and filter modes.
- Course 1 §8 (sampling, Nyquist, prefiltering) and §16 (2-D signals, sRGB and gamma) — skim your own notes before the Background; this lab assumes both.
- Vulkan spec — the “Image Layouts” introduction, once, so
SHADER_READ_ONLY_OPTIMALis a concept and not an incantation.
Prerequisites
Project & environment setup
- Vulkan:
labs/lab-2-4/→vk_textured;engine/vulkan/gains anImageRAII type (image + VMA allocation + view), a layout-transition helper, and sampler creation; descriptor-set layout grows a combined image sampler binding. - Metal:
metal-swift/MetalTextured/; textures viaMTLTextureDescriptor(orMTKTextureLoader— but do the descriptor path at least once by hand), samplers viaMTLSamplerDescriptor. - Test assets into
assets/textures/: one photographic image (any RGBA PNG/JPG), plus a procedural checkerboard generated in code at a crisp power-of-two size (e.g. 512²; 8×8-texel squares) — the aliasing torture instrument. Keepassets/out of git per the repo convention. - UV-mapped quad and cube from Lab 2.2’s layout (uv was already in the vertex spec — now it’s consumed).
Where results go:
| Artifact | Path |
|---|---|
| Notes, filter-mode observation table, Nyquist reconciliation | labs/lab-2-4/notes.md |
Filter-comparison and mip-visualization screenshots, .gputrace |
labs/lab-2-4/captures/ |
Background
Images are not buffers. A sampled texture wants an opaque, tiled, GPU-optimal memory arrangement, so Vulkan gives images layouts — UNDEFINED → TRANSFER_DST_OPTIMAL (receive the copy) → SHADER_READ_ONLY_OPTIMAL (be sampled) — with transitions expressed as pipeline barriers that are also the synchronization between the copy and the first sample. Metal tracks the same hazards automatically (Lab 0.4’s hazardTrackingMode observation, now doing real work) and keeps layout private to the driver: the entire transition vocabulary vanishes. That asymmetry — barriers you author vs. hazards tracked for you — is the module’s recurring theme at its sharpest, and Module 4’s render graph is where the Vulkan side gets civilized.
sRGB vs. linear (§16): color textures are authored in sRGB; lighting and blending math is only valid on linear values. The GPU fixes this in the format: R8G8B8A8_SRGB (Vulkan) / .rgba8Unorm_srgb (Metal) decodes sRGB→linear on sample and the sRGB swapchain re-encodes on write — free, correct, and per-texture. The rule to internalize: color data sRGB, non-color data (normals later, data textures) linear — and the classic bug (washed-out or too-dark output from a double or missing conversion) is one you will produce deliberately.
Sampling theory, applied. A screen pixel’s footprint in texel space grows as the textured surface shrinks or tilts. When that footprint spans \(t\) texels, the shader is sampling a signal at \(1/t\) of its texel rate — below the Nyquist rate for \(t > 1\), so frequencies above the new limit fold back as aliasing (§8): shimmer, moiré, crawling stair-steps. The fix is the one sampling theory always prescribes — prefilter before resampling. The mip chain is exactly that: level \(i\) is the base image band-limited and downsampled by \(2^i\), and the sampler picks \(\lambda \approx \log_2 t\) so the content it reads is already band-limited to the rate it’s sampling at. Trilinear filtering interpolates between the two straddling levels; anisotropic filtering fixes the remaining lie — at grazing angles the footprint is elongated, not square, so isotropic mip selection must blur to the long axis while aniso takes multiple samples along it.
The memory bill is derivable: each level is ¼ the previous, so the full chain costs
\[ \sum_{i=0}^{\infty} 4^{-i} = \tfrac{4}{3} \]
of the base image — a 33% overhead buying alias-free minification and, incidentally, better cache behavior when minified (fewer texels touched per fragment).
Tasks
Vulkan (C++20)
- Upload with transitions. stb_image → staging buffer →
vkCmdCopyBufferToImage→ sample, with explicit layout transitions before and after the copy. Get one barrier’s stage/access masks wrong on purpose, record what synchronization validation says, fix it, and keep the message innotes.md— this is the lab’s postmortem. - sRGB, proven. Load the photographic texture once as
_SRGBand once as_UNORM(no other change) and screenshot both — one is right and one is washed out/too dark. Explain which and why in two sentences innotes.md, in §16’s terms. - The blit chain. Generate mips by hand: allocate the image with full mip count (\(\lfloor\log_2(\max(w,h))\rfloor + 1\) levels), then loop
vkCmdBlitImagelevel \(i \to i{+}1\) with linear filtering, transitioning each level between blit-source and blit-destination roles as you go, ending with the whole chainSHADER_READ_ONLY_OPTIMAL. This is the fiddliest barrier code in the module — budget time accordingly. - Samplers. Create the sampler set this lab compares: nearest; bilinear (no mips); trilinear; trilinear + anisotropy at the device’s
maxSamplerAnisotropy(query it, record it). Address modes: repeat for the floor test below, clamp-to-edge to show the difference at UV borders. - The torture scene. A large floor quad with the checkerboard, UVs tiled many times, camera low so the floor recedes to a grazing horizon — the standard aliasing instrument. A hotkey cycles the four samplers.
Metal (Swift)
- Same textures, tracked resources.
MTLTextureDescriptorwithmipmapLevelCount, upload viareplace(region:...)(shared-storage pragmatism from Lab 2.2 — note it), then a blit encoder’sgenerateMipmaps— one call where Vulkan needed a loop of barriers. Record the asymmetry; also noteMTKTextureLoaderexists and what it would have decided for you (format, sRGB, mips). - Sampler parity. The same four samplers via
MTLSamplerDescriptor(maxAnisotropymirrors the Vulkan query); same torture scene; same hotkey. - Mip visualization. Both APIs: a debug mode sampling with explicit level (
textureLodin GLSL, thelevel()sampler argument in MSL) plus a mode tinting by a per-level color (a tiny 1×1-per-level debug texture is the classic trick) — make the mip selection visible on the receding floor, screenshot the banding of levels marching toward the horizon.
Deliverable & expected results
vk_texturedandMetalTexturedwith the textured cube + torture floor and the sampler hotkey; the four-way filter comparison screenshots (same camera pose — reuse Lab 2.3’s pose dump) and mip-visualization shots incaptures/; the sRGB pair, barrier postmortem, and observation table innotes.md.
| Quantity | Predicted | Measured |
|---|---|---|
| Full mip-chain memory vs. base level | \(\times\,4/3\) — +33%, from the geometric series | … |
| Mip level count for a 512² texture | \(\log_2 512 + 1 = 10\) | … |
| Nearest, minified | hard aliasing: shimmer and moiré on the receding checker | … |
| Bilinear (no mips), minified | still aliases — footprint \(\gg\) 4 texels; blur is not band-limiting | … |
| Trilinear | aliasing gone, horizon over-blurred at grazing angles | … |
| + Anisotropic | grazing detail restored; sharp and stable | … |
| Aniso cost at max setting | small on modern GPUs — single-digit-% direction on this scene, not a number to invent | … |
Profiling & performance
Xcode GPU capture on the Metal app: inspect the texture resource (mip chain visible per level), find the fragment stage’s texture-sample cost, and diff GPU frame time nearest vs. trilinear vs. max-aniso on the torture scene. On the Vulkan side, RenderDoc on the Linux desktop (RTX 4090) earns its keep here — its texture viewer steps through mip levels, and pixel history on a shimmering pixel shows exactly which texels a sample touched. Also capture the upload: the blit-chain generation cost at load time (Tracy zone around it) — cheap, but it exists, and streaming systems later will care.
Analysis & reconciliation
Write the reconciliation as a sampling-theory argument, not a screenshot caption: for one representative floor distance, estimate the pixel footprint \(t\) in texels (from tile count, resolution, and geometry — a hand calculation), state the implied mip level \(\log_2 t\), and check it against the mip-visualization screenshot’s band at that distance. Explain in one paragraph why bilinear-without-mips cannot fix minification (a 2×2 tap is a fixed-width filter; the footprint isn’t) — this is the “prefilter, then resample” lesson in its purest form. Reconcile the aniso screenshots with the elongated-footprint argument. Finally, the API paragraph: list what Vulkan made you say (layouts, barriers, per-level blits) that Metal said for you, and what you’d check first when a Metal texture upload misbehaves without validation naming the hazard.
Going further
- Compressed formats: load a BC/ASTC texture (note which family each GPU supports — BC on desktop-class, ASTC on Apple/mobile) and compare memory and bandwidth against RGBA8 — the production answer to texture memory, deferred until the engine needs it.
- Compute the mip chain in a compute shader (a preview of Module 5’s GPU-driven work) instead of blits, and compare generation time.
- Add a LOD bias control to the sampler and find the bias where trilinear’s horizon blur visibly trades against shimmer — the knob shipping games actually expose as “texture sharpness”.