Syllabus
10-Week Autonomy ML Systems + Embedded Robotics + Vision + Jetson
Full week-by-week plan. Each entry lists the theme and reading. Detailed theory exercises, implementation tasks, benchmarks, hardware notes, and definitions of done live in the weekly lecture notes. Book-specific exercises live in Books.
Prerequisites — Courses 5 and 6. This course now assumes the mathematical and signal-processing foundations are already in hand. Course 5 supplies the linear algebra, numerical computing (conditioning, QR/SVD, floating point), probability, convex optimization, and information theory; Course 6 supplies sampling, anti-aliasing, digital filtering, and embedded data acquisition. Those topics are no longer re-taught here — they are referenced and applied. This is exactly what lets the course fit in 10 focused weeks instead of 16.
Note on AI use: The weekly lecture notes and syllabus summaries on this site are drafted with AI assistance, so each week has a consistent structure to study from. The substance is mine: every math proof and exercise solution is worked by hand on paper first and converted to LaTeX/KaTeX/MathJax using AI for typesetting only, and all code is written by me. The goal is to learn the material, which only happens by producing the proofs and the code myself.
Weekend rhythm: Saturday = reading + theory exercises + implementation. Sunday = implementation + benchmark + staff-level note. Optional weeknight = 30–60 min cleanup or reading.
General rule for each week: (1) theory exercises (applying Course 5/6 results), (2) implementation, (3) benchmark with recorded data, (4) staff-level note in docs/.
Book abbreviations:
Math (review from Course 5): T&B (Trefethen & Bau) · BT (Bertsekas & Tsitsiklis) · Boyd · Pressley · Morin · DPV (Dasgupta) · C&T (Cover & Thomas)
GPU/Systems: CUDA Book (Motta) · MBT (Metal by Tutorials) · Numerical Python (Johansson) · HLW (How Linux Works) · CA (Computer Architecture) · Embedded AI
Vision: Szeliski (Computer Vision: Algorithms and Applications)
Robotics/AI: PR (Probabilistic Robotics — Thrun/Burgard/Fox) · GBC (Deep Learning — Goodfellow/Bengio/Courville)
Lectures: CS231n · MIT Machine Vision
Main repository structure:
autonomy-ml-systems-course/
docs/
common/
cpp/ python/ geometry/ trajectory/
01-gpu-compute-lab/ 02-prediction-planning-lab/
cuda/ metal/ benchmarks/ prediction/ planning/ models/
03-ml-efficiency-kernels-lab/
profiling/ mixed_precision/ kernels/
04-robot-runtime-control-lab/
runtime/ can/ sensors/ estimation/ safety/ app/Target roles: ML Frameworks & Efficiency Engineer · Prediction & Planning ML Engineer · Embedded/Robotics Software Engineer (e.g. AV and humanoid-robotics roles).
Phase 1 · Weeks 1–3 — GPU Compute and ML Numerical Foundations
Week 1 — GPU Compute First Principles: CUDA and Metal
Theme: CUDA and Metal execution models: threads, blocks/threadgroups, memory hierarchy, launch overhead, reductions.
Read: CUDA Book: threads, blocks, grids, global/shared memory, synchronization, reductions. · MBT: compute pipeline, buffers, threadgroups, dispatching. · CA: memory hierarchy and parallel architectures. · Optional CS231n skim: CNN workloads as matrix/conv workloads. (Linear-algebra and floating-point foundations: assumed from Course 5.)
Week 2 — AV Scene Representation, Coordinate Frames, and Trajectory Geometry
Theme: Synthetic AV scenes, ego-centric coordinate frames, and motion as time-indexed geometry: velocity, acceleration, jerk, curvature, arc length.
Read: Pressley (review): curves, arc length, tangent vectors, curvature. · Morin (review): position/velocity/acceleration, relative motion. · MIT Machine Vision: image formation / perspective projection skim. (Linear maps and coordinate transforms: assumed from Course 5.)
Week 3 — Mixed Precision, Numerical Stability, and Quantization for ML
Theme: fp32/fp16/bf16/int8 tradeoffs, numerical correctness, and accelerator efficiency on real ML workloads.
Read: T&B (review/apply): condition number, perturbation, stability, floating-point behavior. · CUDA Book: mixed precision / tensor-core sections. · C&T: entropy/KL for quantization intuition. · CS231n: training-instability / optimization review. (Conditioning and SVD theory: assumed from Course 5.)
Phase 2 · Weeks 4–7 — Prediction, Planning, and ML Efficiency
Week 4 — Probabilistic, Multimodal Trajectory Prediction
Theme: Future motion is probabilistic and multimodal; represent and score it.
Read: CS231n: linear classifiers / softmax. · PR: motion models, sensor/belief representations. · Embedded AI: model-drift skim. (Probability, expectation, and information theory: assumed from Course 5.)
Week 5 — Loss Functions, Backprop, and a Tiny Learned Predictor
Theme: Train and debug a learned trajectory predictor end to end.
Read: CS231n: neural networks, backpropagation, training/debugging. · GBC: chapters on deep feedforward networks and optimization for training. (Convex sets/functions, gradient methods, and Taylor approximation: assumed from Course 5.)
Week 6 — Planning: Metrics, Physical Feasibility, and Graph Search
Theme: Build a first planner and a clean prediction→planning contract.
Read: DPV (review/apply): graph search, shortest paths, dynamic programming. · Morin: dynamics, constraints, feasibility. · Pressley: curvature/smoothness constraints. · Embedded AI: real-time/failsafe framing.
Week 7 — ML Efficiency and Custom GPU Kernels
Theme: Profile the predictor as a real ML systems workload, then write one ML primitive below the framework in Triton/CUDA and compare CUDA vs Metal.
Read: CUDA Book: occupancy, coalescing, shared memory, reductions, warp-level behavior, tiled matmul. · Triton docs: program IDs, masks, block loads/stores, reductions. · MBT: threadgroup memory, compute pipelines, performance tuning. · CA: parallel architectures. · CS231n: softmax/LayerNorm as kernels.
Phase 3 · Weeks 8–10 — Embedded Robotics and Jetson Integration
Week 8 — Real-Time Control Loops, Timing Jitter, and a CAN Actuator Protocol
Theme: Build a fixed-frequency C++ control loop, measure jitter, and add a robotics/automotive-style CAN actuator+telemetry protocol with fault injection.
Read: HLW: kernel/user space, processes, devices, networking/device interfaces. · CA: CPU chapter skim. · Embedded AI: embedded systems / CAN / automotive context. (Sampling and discrete-time control intuition: assumed from Course 6.)
Week 9 — Sensor Integration and State Estimation: IMU, Bayes and Kalman Filters
Theme: Ingest IMU/sensor streams, model noise, and estimate state with recursive Bayes/Kalman filters.
Read: PR: recursive state estimation, Bayes filters, Kalman filters, motion/sensor models. · BT (review): expectation, variance, covariance. · Embedded AI: smart sensors, IMU preprocessing, sensor fusion. (ADC, sampling, and digital low-pass filtering of the raw signal: assumed from Course 6; probability foundations from Course 5.)
Week 10 — Capstone: Safety Runtime, Calibration, and Full Jetson Integration
Theme: Tie control loop, CAN protocol, sensor/estimation stack, safety state machine, calibration tools, and Jetson deployment into one runtime.
Read: Boyd (apply): least squares, regularized least squares for calibration. · Numerical Python: NumPy/SciPy/matplotlib for fitting/analysis. · HLW: boot, devices, filesystems, processes, logs/system management. · Embedded AI: deployment constraints, debugging, drift.
Book Integration Map
| Book | Weeks |
|---|---|
| CUDA Book / MBT | 1, 3, 7 — GPU execution model, mixed precision, kernels |
| T&B (review) | 3, 7 — stability, floating point, matrix norms |
| BT / C&T (review) | 4, 9 — probability, entropy, covariance |
| Boyd / GBC (review/apply) | 5, 10 — optimization, training, calibration least squares |
| DPV | 6 — graph search, shortest paths, dynamic programming |
| Pressley / Morin | 2, 6 — trajectory geometry, kinematics, feasibility |
| PR | 4, 9 — motion models, Bayes/Kalman state estimation |
| HLW / CA / Embedded AI | 1, 8, 9, 10 — systems, real-time loops, embedded deployment |
| CS231n / MIT Machine Vision / Szeliski | 1, 2, 4, 5, 7 — vision workloads, image formation |
How Courses 5 and 6 Feed This Course
- From Course 5: linear maps and coordinate transforms (Week 2), conditioning/SVD/floating point (Weeks 3, 7), probability and information theory (Weeks 4, 9), convex optimization and least squares (Weeks 5, 10), graph algorithms (Week 6).
- From Course 6: sampling and anti-aliasing, digital filtering, and embedded ADC/data-acquisition pipelines (Weeks 8–9). The raw sensor signal-conditioning that earlier versions of this course taught inline is now Course 6 material, applied here directly to IMU and actuator telemetry.
Optional Weeks 11+ Direction
If prioritizing ML framework efficiency: deeper JAX/XLA, more Triton kernels, ONNX/TensorRT, quantization, distributed/multi-GPU profiling, realistic transformer/vision workloads.
If prioritizing AV prediction and planning: real open-motion datasets, a transformer trajectory model, closed-loop simulation, scenario mining, calibration and uncertainty, a richer planner.
If prioritizing embedded robotics: PREEMPT_RT Linux, a real CAN hardware loop, motor/servo integration, hardware-in-the-loop tests, ROS 2 integration, deeper sensor fusion.
If prioritizing vision: a dedicated classical-vs-learned vision mini-lab (image formation, filtering, edges, a tiny CNN detector) using Szeliski and MIT Machine Vision.