Lab 9.5 — Real-Time Video Frame Pipeline (Pi 5 / Jetson)

← Course 2 syllabus · Bonus Module 9 · Prev: « Lab 9.4

Goal

The media-signal-processing capstone: extend the file-round-trip discipline of Module 9 from a single image to a stream of video frames, and run it on the edge accelerators from Module 8. The host reads frames from a real MP4, streams them to the Pi 5 (CPU/OpenCV) or Jetson Orin Nano (CUDA/TensorRT), the device runs a per-frame real-time pipeline — grayscale + Sobel/Canny edges and optical flow on the Pi 5, or a small segmentation/detection CNN on the Jetson GPU — and returns processed frames the host reassembles into an output MP4 and verifies (per-frame PSNR vs. a reference + a visual check). You measure the three numbers that decide whether an embedded vision pipeline is shippable: FPS, per-frame latency, and end-to-end throughput, Pi 5 vs. Jetson. This lab ties Module 9’s “stream a file, process on-device, verify against a reference” workflow to Module 8’s accelerators, and is the final lab of the course.

Recommended reading

• Lyons / O-DSP — the filtering and 2-D convolution principles from Lab 9.4 carry straight into the per-frame edge/flow stages; skim the convolution and windowing sections as needed.
• Course 1 Weeks 1–3 (../course1/index.qmd#week-1) — linear algebra behind the optical-flow least-squares (Lucas–Kanade solves a \(2\times2\) normal-equation system per window) and the CNN’s linear layers.
• Course 1 Weeks 5–6 — probability/expectation behind the CNN’s training objective and the confidence thresholds on detections.
• Your own Module 8 labs — Lab 8.1 (GPU spectrogram) for the CUDA/streaming setup and Lab 8.5 (STM32 vs Pi 5 vs Jetson benchmark) for the benchmarking methodology you reuse here.
• OpenCV VideoCapture / VideoWriter, cv2.Canny, cv2.calcOpticalFlowFarneback / cv2.calcOpticalFlowPyrLK; TensorRT / onnxruntime for the Jetson CNN.

Equipment & parts

• Raspberry Pi 5 (CPU/OpenCV path) and/or Jetson Orin Nano (CUDA/TensorRT path).
• Host Mac with numpy opencv-python for read/reassembly and the reference pipeline.
• A short test MP4 (a few hundred frames, e.g. 640×480 @ 30 fps). Something with real motion (a pan, a moving object) so optical flow and edges have content.
• USB link (or TCP over Ethernet — for full frames the framed-block harness rides on TCP just as easily as on serial; see below).

Safety & don’t-break-it

Data/compute lab; the hazards are backpressure, memory bandwidth, and reference mismatch:

• A video stream will overrun a naive link instantly. A 640×480 gray frame is 307 KB; at 30 fps that’s ~9 MB/s of raw frames — far past a UART VCP. Use USB CDC or TCP and keep the Lab 9.1 flow control (credits) so the sender blocks instead of dropping frames. A dropped frame corrupts the reassembled MP4’s timing.
• Don’t confuse codec loss with pipeline error. Re-encoding output frames to MP4 (lossy H.264) adds its own PSNR hit unrelated to your processing. For the verification comparison, compare the raw processed frames against the reference frames, and treat the MP4 as the human-facing artifact — or write a lossless container for scoring.
• Jetson GPU memory and power. Batching frames helps throughput but a too-large batch OOMs the Orin Nano’s shared memory; watch tegrastats. Keep the board’s power mode fixed across a benchmark run or your FPS numbers aren’t comparable (the Lab 8.5 rule).
• Synchronize frame ordering. Tag every frame with a sequence number (you already have seq from Lab 9.1) so the host reassembles frames in order even if the device pipelines/reorders internally.

Background

Per-frame throughput metrics. For a pipeline that processes frames at rate \(f\) frames/s,

\[\text{FPS} = f, \qquad T_{\text{frame}} = \frac{1}{f}\ \text{(steady-state period)}.\]

Latency vs. throughput are different numbers. Latency is the time from a frame entering the host streamer to its processed result returning — it includes transport each way plus device compute:

\[L = t_{\text{tx}} + t_{\text{compute}} + t_{\text{rx}}.\]

A pipelined implementation can hold FPS \(= 1/t_{\text{compute}}\) (throughput bound by the slowest stage) while \(L\) is larger than \(1/\text{FPS}\) because several frames are in flight. End-to-end throughput is the sustained returned-pixel rate:

\[\text{throughput} = \text{FPS} \times (W\cdot H)\ \text{pixels/s} = \text{FPS}\times(W\cdot H\cdot \text{bytes/pixel})\ \text{B/s}.\]

Real-time means \(\text{FPS} \ge\) the source frame rate (e.g. 30) with \(L\) small enough for the application.

The per-frame operations. Sobel/Canny edges are the 2-D convolutions of Lab 9.4 (Canny adds non-max suppression + hysteresis). Optical flow (Lucas–Kanade) assumes brightness constancy, \(I(x,y,t)=I(x{+}u,y{+}v,t{+}1)\), linearizes to \(I_x u + I_y v + I_t = 0\), and solves a per-window \(2\times2\) least-squares system

\[\begin{bmatrix}\sum I_x^2 & \sum I_x I_y\\ \sum I_x I_y & \sum I_y^2\end{bmatrix}\begin{bmatrix}u\\v\end{bmatrix} = -\begin{bmatrix}\sum I_x I_t\\ \sum I_y I_t\end{bmatrix},\]

the Course 1 Week 1–3 normal equations, one per pixel neighborhood. On the Jetson, the per-frame stage is instead a small CNN whose forward pass runs in fp16/int8 on the GPU via TensorRT (the Module 8 toolchain).

PSNR per frame is exactly the Lab 9.4 metric applied to each returned frame \(\hat{I}_t\) vs. reference \(I_{\text{ref},t}\); you report the per-frame PSNR series (mean and worst frame).

Procedure

Part A — Host: stream frames from the MP4.

1. Read frames and stream each as a block (over USB CDC or TCP), reusing the Lab 9.1 framing + credits:

import cv2
cap = cv2.VideoCapture("in.mp4")
fps = cap.get(cv2.CAP_PROP_FPS)
seq = 0
while True:
    ok, frame = cap.read()
    if not ok: break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)     # send gray to save bandwidth
    send_block(seq, gray.tobytes()); seq += 1
    processed = recv_frame()                            # ordered by seq
    writer_append(processed)

Part B — Device pipeline.

2. Pi 5 (CPU/OpenCV): per frame, grayscale → cv2.Sobel/cv2.Canny edges, and calcOpticalFlowFarneback (or LK on tracked points) between consecutive frames; return the edge map (and/or a flow visualization).
3. Jetson (CUDA/TensorRT): per frame, run a small segmentation/detection CNN engine (built with the Module 8 toolchain, fp16/int8), return the mask/overlay. Use CUDA streams / batching to keep the GPU fed.
4. Timestamp receipt and completion per frame on the device to separate transport from compute.

Part C — Reassemble and verify.

5. Reassemble returned frames in seq order into out.mp4 with cv2.VideoWriter at the source fps.
6. Run the host reference (same OpenCV ops, or the same CNN in onnxruntime) on the original frames; compute per-frame PSNR of device-vs-reference on the raw frames, plus a visual side-by-side.
7. Measure FPS (frames returned / wall-clock), per-frame latency (send→receive timestamp), and throughput (pixels/s), for Pi 5 and Jetson separately.

Deliverable & expected results

• out.mp4, the per-frame PSNR series (mean + worst-frame), and a Pi 5 vs. Jetson table of FPS / latency / throughput.

For a 640×480 gray source at 30 fps (illustrative — Predicted from the metric formulas; fill Measured on the bench):

Quantity	Predicted	Measured
Frame size (640×480×1 B)	307,200 B	…
Real-time frame period (\(1/30\))	33.3 ms	…
Raw returned throughput at 30 fps	≈ 9.2 MB/s	…
Pi 5 edges+flow FPS	source-limited (≥30) or lower — measure	…
Jetson CNN FPS (fp16/int8, TensorRT)	≥ Pi 5; measure	…
Per-frame latency \(L=t_{tx}+t_{cmp}+t_{rx}\)	compute-dominated; measure	…
Mean per-frame PSNR vs. reference	\(\infty\) (edges, matched int) / high finite (CNN fp16)	…

Analysis & reconciliation

First separate the numbers: FPS is throughput, latency is delay — a pipelined Jetson can hold 30+ FPS while each frame’s \(L\) spans several frame periods because several are in flight. Reconcile measured throughput against \(\text{FPS}\times W\cdot H\); a shortfall vs. the link’s raw ceiling means the transport, not compute, is the bottleneck (send gray not RGB, widen the credit window, move UART→USB/TCP). If per-frame PSNR is high but not infinite on the OpenCV path, suspect the MP4 re-encode contaminating the comparison — score raw frames instead; on the CNN path, finite PSNR is expected (fp16/int8 vs. the fp32 reference) and its magnitude is your quantization-accuracy budget from Module 8. A worst-frame PSNR far below the mean usually flags a dropped/reordered frame (check seq continuity) rather than a compute error. Compare Pi 5 (CPU/OpenCV) vs. Jetson (CUDA/TensorRT): the Jetson should win on the CNN stage by the accelerator margin you measured in Lab 8.5, while classical edges/flow may be competitive on the Pi 5 CPU — that crossover is the engineering point of the whole edge-ML module.

Going further

• Add temporal stability: run the edge/flow output through a short IIR across frames and quantify flicker reduction — 1-D filtering (Lab 9.2) applied along the time axis of video.
• Batch frames on the Jetson and plot FPS vs. batch size until GPU memory saturates; find the throughput knee.
• Stream RGB instead of gray and re-measure — quantify how much of your throughput budget color costs.
• Close the module’s loop: feed a live camera instead of an MP4 into the same harness, proving the file-based regression pipeline and the real-time pipeline are the same code — the reproducibility payoff of Module 9’s whole design.

---

This is the final lab of Course 2. From Module 0’s power supply to a verified real-time video pipeline on an edge GPU, every result was predicted by hand, built, measured, and reconciled against a reference — the working habit of a DSP embedded / firmware engineer.