Lab 9.3 — STFT Overlap-Add Audio Effects

← Course 2 syllabus · Bonus Module 9 · Prev: « Lab 9.2 · Next: Lab 9.4 »

Goal

Move from time-domain filtering to block-based frequency-domain processing on the streamed WAV. You’ll build a real short-time Fourier transform (STFT) pipeline on the device: window each block, forward-FFT with CMSIS-DSP arm_rfft_fast_f32, apply a spectral effect (a noise gate / spectral subtraction / fast-convolution reverb / robotization), inverse-FFT, and overlap-add (OLA) the windowed frames back into a continuous stream that goes home to effected.wav. The verification step is what makes this rigorous: with the effect set to identity, correct OLA must reconstruct the input to within FFT round-off — so you first prove your OLA framework is transparent against a NumPy reference, and only then trust the effected output. This is the standard structure of real-time spectral audio firmware (denoisers, vocoders, convolution reverbs), and the identity-reconstruction test is how you know your windowing and hop are consistent.

Recommended reading

• Lyons Ch. 3–5 — the DFT and FFT, windowing and spectral leakage, and the FFT of real signals (rfft). The core reading: your window choice, hop, and bin interpretation all come from here.
• Lyons Ch. 13 (§13.10, Fast FIR Filtering Using the FFT) — fast convolution and the overlap-add construction: why a block of output is the IFFT of a product of spectra, and why consecutive windowed frames must be summed with the right hop to reconstruct the signal. This is the mechanism the whole lab is built on, and it is the section to read closest.
• Course 1 Lessons 31–32 (../course1/index.qmd#lesson-31) — the Fourier transform and the DTFT underpinning the STFT. They are also where overlap-add gets its justification: the convolution theorem is what lets you multiply spectra instead of convolving sequences, and the identity-reconstruction test below is that theorem holding (or not) in fixed hardware arithmetic.
• Course 1 Lesson 43 (../course1/index.qmd#lesson-43) — audio signal processing: equalizers and fast convolution (overlap-add/overlap-save), and the STFT-based effects this lab implements.
• CMSIS-DSP documentation for arm_rfft_fast_f32 (real FFT, in-place, packed spectrum layout).

Equipment & parts

• STM32 Nucleo-L476RG (small FFT sizes, \(N=256\)–\(512\)) or the Raspberry Pi 5 (larger \(N\), heavier effects) — the lab runs on either; see the tradeoff note in the Background.
• CMSIS-DSP (arm_rfft_fast_f32, arm_cmplx_mag_f32) on the STM32; NumPy on the Pi 5.
• Host Mac with numpy scipy for the reference OLA and audio checks.
• A test WAV: for a noise gate / spectral subtraction, a recording with audible background noise (fan, hiss) makes the effect obvious; for reverb, a dry vocal or clap.

Wiring & bench setup

STM32 route: the same single USB cable as Lab 9.1 (Mac → ST-LINK Micro-B, USART2 VCP). Pi 5 route: no cable at all — the “link” is your LAN/SSH session. Either way the chain is:

flowchart LR
  WAV["media/in/noisy.wav"]
  HOST["Host harness<br/>Lab 9.1 framing + credits"]
  DEV["STM32 or Pi 5<br/>window → FFT → effect → IFFT → OLA"]
  OUT["media/out/effected.wav"]
  REF["NumPy STFT/OLA reference<br/>+ identity test"]
  WAV --> HOST
  HOST -- "PCM blocks" --> DEV
  DEV -- "effected blocks" --> HOST
  HOST --> OUT -- "compare" --> REF

flowchart LR
  WAV["media/in/noisy.wav"]
  HOST["Host harness<br/>Lab 9.1 framing + credits"]
  DEV["STM32 or Pi 5<br/>window → FFT → effect → IFFT → OLA"]
  OUT["media/out/effected.wav"]
  REF["NumPy STFT/OLA reference<br/>+ identity test"]
  WAV --> HOST
  HOST -- "PCM blocks" --> DEV
  DEV -- "effected blocks" --> HOST
  HOST --> OUT -- "compare" --> REF

Safety & don’t-break-it

Data lab; the failure modes are windowing/indexing bugs and real-time budget overruns:

• COLA violation = amplitude modulation. If your window and hop don’t satisfy the constant-overlap-add condition, the identity effect won’t reconstruct flat — you’ll hear a periodic tremolo at the frame rate. Verify COLA numerically before blaming the effect.
• In-place FFT buffer aliasing. arm_rfft_fast_f32 uses a specific packed real-spectrum layout and an in-place scratch buffer. Reusing the same buffer for input and output without respecting the layout corrupts bins silently. Follow the CMSIS buffer contract exactly.
• FFT-size / SRAM budget. Float FFT of size \(N\) needs \(\sim N\) complex floats of scratch plus your OLA history buffer; on the 128 KB L476RG keep \(N \le 512\) and one hop of overlap history. Don’t blow SRAM.
• Latency is structural here. OLA imposes at least one frame of algorithmic latency (\(N/f_s\)). That’s inherent, not a bug — budget for it and don’t try to “fix” it by shrinking the window past your frequency resolution needs.
• Preserve the Lab 9.1 transport discipline: FFT compute per block must fit inside the block’s real-time budget or the RX buffer overruns.

Project & environment setup

Firmware — reuse firmware/m9-media/ (Lab 9.1). No new peripherals; confirm USART2 921600 + DMA + 80 MHz clock (setup essentials) as before. CMSIS-DSP is required this time (arm_rfft_fast_f32, arm_cmplx_mag_f32) — if you didn’t add it in Lab 9.2, add it now (same steps as firmware/m6-dsp/).

Pi 5 route (optional): on the Pi, python3 -m venv ~/venv && ~/venv/bin/pip install numpy scipy — the device-side STFT/OLA script then lives in labs/lab-9-3/edge/ (the SBC-side tree, as in Module 8) and is synced to the board.

Host — course venv; transport is labs/lab-9-1/host/harness.py. Library roles: numpy (Part A COLA check + identity-OLA and effect references; scipy.signal.stft/istft is a useful cross-check of your hand-rolled OLA), scipy.io.wavfile (WAV I/O), matplotlib (COLA and before/after-FFT plots):

source venv/bin/activate
mkdir -p labs/lab-9-3/host labs/lab-9-3/captures

Inputs: media/in/noisy.wav (gate/spectral subtraction) or media/in/dry.wav (reverb). Reference scripts in labs/lab-9-3/host/ (you write them).

Keep this lab’s reconciliation in labs/lab-9-3/host/analysis.ipynb — the notebook convention — and export final figures next to it.

Where results go:

Artifact	Path
Bench note (identity error, latency, headroom)	labs/lab-9-3/notes.md
Effected output	media/out/effected.wav
COLA verification plot	labs/lab-9-3/host/cola.png
Before/after + reference-vs-device FFT	labs/lab-9-3/host/before-after-fft.png
Identity-test error logs	labs/lab-9-3/captures/identity-err.txt

Background

The STFT. Slice \(x[n]\) into frames of length \(N\) taken every hop \(R\) samples, window each frame with \(w[n]\), and DFT it:

\[X_m[k] = \sum_{n=0}^{N-1} w[n]\,x[mR+n]\,e^{-j 2\pi k n / N}, \qquad k = 0,\dots,N-1.\]

A spectral effect modifies \(X_m[k] \to \tilde{X}_m[k]\) (e.g. a noise gate zeroes bins below a threshold; spectral subtraction removes a noise magnitude estimate; fast-convolution reverb multiplies by an impulse-response spectrum; robotization zeroes the phase).

Overlap-add reconstruction. Inverse-DFT each modified frame and sum the overlapping frames back onto the time axis. The default here is an analysis-only window: the frame is windowed once before the FFT, and the reconstructed frames are overlap-added directly (no second window). A synthesis window can also be applied on the way out, in which case the COLA condition below is on the product of the analysis and synthesis windows (see Going further).

\[y[n] = \sum_m \tilde{x}_m[n - mR], \qquad \tilde{x}_m[n]=\frac{1}{N}\sum_k \tilde{X}_m[k]\,e^{j2\pi k n/N}.\]

The COLA condition. For the identity effect (\(\tilde{X}=X\)), OLA reconstructs \(x\) exactly (up to FFT round-off) iff the analysis window satisfies constant overlap-add at hop \(R\):

\[\sum_{m} w[n - mR] = C \quad \text{(a constant, independent of } n\text{)}.\]

A single Hann analysis window satisfies COLA at 50% overlap (\(R=N/2\), \(C=1\)) and at 75% (\(R=N/4\)). This is the property your identity test checks. (If you also apply a Hann synthesis window, the relevant sum is \(\sum_m w^2[n-mR]\), which for Hann at 50% overlap is \(C=0.75\), not 1 — you would then divide the output by \(0.75\), or use \(\sqrt{\text{Hann}}\) on both sides to keep \(C=1\).) Choosing 50% vs. 75% trades compute (more frames) for smoother modification and less inter-frame artifacting.

Window choice & leakage. Hann trades main-lobe width for low side-lobes (Lyons Ch. 3), which matters when a spectral effect edits magnitudes bin-by-bin — leakage smears energy across bins and can make a gate chatter.

Where it runs. On the STM32 at \(N=256\)–\(512\) the M4F + CMSIS arm_rfft_fast_f32 handles a modest effect per block within budget. Heavier effects (long-tail convolution reverb, large FFTs) or higher sample rates belong on the Pi 5, where NumPy/scipy FFTs are cheap but you lose the “runs on a $2 hardware” story. Note the tradeoff in your write-up.

Procedure

Part A — Prove OLA is transparent (identity test) first.

1. Choose \(N=512\), Hann window, 50% overlap (\(R=256\)). On the host, implement analysis→(identity)→synthesis OLA in NumPy and confirm it reconstructs a test WAV to within FFT round-off. Numerically verify COLA:

import numpy as np
N, R = 512, 256
w = np.hanning(N)
cola = np.zeros(N)
for m in range(-4, 5):            # sum shifted copies over one period
    cola += np.roll(np.pad(w, (0, N))[ :N], (m*R) % N)
# with proper accumulation, cola is ~constant  → COLA satisfied

2. Only once the host identity OLA is flat do you move to the device.

Part B — Firmware STFT/OLA.

3. Maintain an input history so each new block forms a full \(N\)-sample frame at hop \(R\). Illustrative per-frame kernel (structure only):

arm_rfft_fast_instance_f32 S;   // init once for size N
/* frame[]: N samples, already Hann-windowed */
arm_rfft_fast_f32(&S, frame, spec, 0);         // forward real FFT (packed)
apply_effect(spec, N);                          // e.g. gate bins below thresh
arm_rfft_fast_f32(&S, spec, recon, 1);          // inverse FFT
/* analysis-only OLA: a single Hann window is COLA with C=1 at 50% overlap, so
   overlap-ADD the reconstructed frames directly. Do NOT window a second time here:
   Hann*Hann = Hann^2 sums to 0.75, not 1, and would scale the output down. */
for (int n = 0; n < N; ++n)
    ola[(head + n) % OLA_LEN] += recon[n];
/* emit R samples from the tail of the completed region, then advance head by R */

4. Stream the effected output back through the Lab 9.1 harness; host writes effected.wav.

Part C — Verify and listen.

5. Identity check on-device: run the device with the effect bypassed; assert the returned WAV matches the input to within a small tolerance (FFT round-off + Q-format if any): np.max(np.abs(y - x)) / 32768 < 1e-4.
6. Enable the effect. For a noise gate, confirm the noise floor drops in the FFT and by ear; for reverb, confirm the tail; for robotization, confirm the monotone pitch.
7. Compare against a NumPy reference of the same effect (same \(N\), window, hop, threshold) — magnitude spectra should match closely frame-by-frame.

Deliverable & expected results

• The COLA verification plot, the on-device identity-reconstruction error, and effected.wav with a before/after FFT (and a reference-vs-device spectral comparison for the chosen effect).

For \(N=512\), Hann, 50% overlap, \(f_s = 48\) kHz:

Quantity	Predicted	Measured
COLA sum \(\sum_m w[n-mR]\) (Hann, 50%)	constant (\(=1\))	…
Identity OLA reconstruction error (device)	\(\lesssim 10^{-4}\) FS (round-off)	…
Algorithmic latency (\(N/f_s\))	10.67 ms	…
Frame rate (\(f_s/R\))	187.5 frames/s	…
FFT bin spacing (\(f_s/N\))	93.75 Hz	…
Effect vs. NumPy reference (per-frame mag)	match to round-off	…

Analysis & reconciliation

The identity test is the linchpin: if bypassed-effect OLA doesn’t reconstruct flat, your window/hop violate COLA (audible tremolo at 187.5 Hz here) or your OLA indexing drops/doubles a hop — fix that before interpreting any effect. A device identity error much larger than round-off points to the arm_rfft_fast_f32 packed-layout contract being mishandled or an in-place buffer alias. Once identity is clean, reconcile the effected output against the NumPy reference frame-by-frame; residual differences should be round-off (float path) plus any threshold/estimator nondeterminism you introduced. Note the measured per-block compute headroom on your chosen platform against the frame period (\(R/f_s = 5.33\) ms here) — that margin is what decides STM32 vs. Pi 5 for a heavier effect.

NoteJetson Orin Nano as the device

Via the Lab 9.1 Jetson transport, the device-side STFT→modify→ISTFT moves to the Jetson with the frames and reference checks unchanged — and this is the M9 lab where the GPU genuinely earns a run: the framed blocks batch naturally into the cupyx.scipy STFT path you measured in Lab 6.3/8.1, so process the stream once on CPU (NumPy server) and once on GPU (CuPy server) and compare achieved real-time factor at several block sizes. The COLA identity test is the arbiter on both. Pi 5: CPU path only.

Going further

• Switch to overlap-save (linear convolution via FFT with discarded wrap-around) for the fast-convolution reverb and compare artifact/latency against OLA.
• Try 75% overlap (Hann still COLA) and hear the reduction in modification artifacts; measure the compute cost increase.
• Add a synthesis window for smoother magnitude-editing effects: use \(\sqrt{\text{Hann}}\) on both the analysis and synthesis sides (the product is Hann, so COLA still gives \(C=1\)), or keep Hann on both sides and normalize the overlap-added output by \(0.75\). Confirm the identity test still reconstructs flat.
• Implement spectral subtraction with a noise magnitude estimated from the first few (silent) frames and quantify the noise-floor drop in dB — a preview of the learned denoiser in Lab 8.3, but hand-built.
• Port the same STFT effect to the Pi 5 at \(N=2048\) and compare artifacts and CPU load to the STM32’s small-\(N\) version.