Module 11 Lessons — Embedded Linux Systems Programming
Back to the Course 2 syllabus · Practice: Module 11 exercises
This page is the module’s teaching text for the third tier: a Linux kernel between the program and the hardware, on the Jetson Orin Nano and the Raspberry Pi 5. The subject is the POSIX toolkit that embedded-Linux C is actually written with — descriptors, errno, buffered vs. raw I/O, monotonic time, timerfd and epoll, real-time scheduling, threads — and the std Rust that maps onto it, with nix and libc filling the gaps std leaves. The discipline is the same on both sides: a real-time loop on Linux is not real-time by default, and every knob that makes it so has a failure mode. As with every lessons page in this course, this is AI-drafted teaching text reviewed by me; see the syllabus’s note on AI use. Seacord 7–8, Rust Book 12, 16, and 21, and the nix/gpiod crate docs remain available as optional deep-dives; nothing below requires them.
1 · The tier: what the kernel gives and what it takes
On the bare-metal tier the program is the machine. On Linux the program is a process: a virtual address space, a set of file descriptors, a scheduling class, and one or more threads that the kernel interleaves on the cores it chooses. Everything the microcontroller did by touching a register — timing, I/O, interrupt handling — is now a system call that may block, may fail, and may be preempted in the middle. What the kernel gives in exchange is isolation, drivers, and a userspace ABI that is the same on the Jetson, the Pi, and the Mac’s Docker container.
| Bare-metal concept | Linux equivalent | Where it hides latency |
|---|---|---|
| A polling loop on a timer | clock_nanosleep / timerfd + read |
Scheduler wake-up, CPU frequency, page faults |
| An ISR | A blocking read on a descriptor (gpiod event, signalfd) |
Interrupt → kernel thread → your thread’s wake-up |
| A GPIO register write | ioctl on /dev/gpiochipN |
Syscall entry/exit, the driver’s lock |
| A DMA buffer | mmap’d memory, or a driver’s ring |
Page faults on first touch unless locked |
| Priority = NVIC level | SCHED_FIFO priority 1–99 |
Throttling, priority inversion on a mutex |
The Jetson Orin Nano runs JetPack 6 — Ubuntu 22.04 on kernel 5.15, AArch64, six A78AE cores — and it is not a PREEMPT_RT kernel. The Pi 5 runs a similarly stock kernel. Neither promises bounded latency; the real-time knobs in §7 tighten the distribution and the measurement exercises show by how much. Board setup — packages, groups, nvpmodel, the CLion remote toolchain — is Course 3’s Jetson setup essentials; this module assumes a board that is already flashed and reachable over SSH.
1.1 Descriptors, and the rule that everything is one
A file descriptor is a small integer naming a kernel object: a file, a pipe, a socket, a serial port, a GPIO line request, a timer, a signal set, an event counter. The design consequence is that one blocking primitive handles all of them — read, write, poll/epoll — so an embedded-Linux program’s main loop is a wait on a set of descriptors, exactly as a firmware main loop is a wait on a set of interrupt flags.
int fd = open("/dev/ttyTHS1", O_RDWR | O_NOCTTY | O_NONBLOCK | O_CLOEXEC);
if (fd == -1) {
// errno is set; log it and fail — never continue with fd == -1
}Three habits attach to every descriptor: open with O_CLOEXEC so a child process never inherits it; decide blocking vs. O_NONBLOCK at open time and never mix the two mental models on one descriptor; and close exactly once, on every exit path — the goto cleanup idiom of Module 4 exists for this.
1.2 errno discipline
POSIX reports failure through a return value (-1, NULL, EOF) and a reason through the thread-local errno. errno is only meaningful immediately after a failing call, and any library call in between — including printf in the logging line — may clobber it. The pattern is: copy, then act.
ssize_t n = read(fd, buf, sizeof buf);
if (n == -1) {
int e = errno; // capture before anything else runs
if (e == EINTR) { continue; } // a signal interrupted the syscall: retry
if (e == EAGAIN) { /* O_NONBLOCK: nothing ready */ }
char msg[64];
strerror_r(e, msg, sizeof msg); // the thread-safe form; never strerror()
log_error("read(%d): %s", fd, msg);
}EINTR deserves its own line: any blocking call can return early because a signal arrived, and code that treats EINTR as an error fails randomly under load. Every blocking call in this module is wrapped in a retry-on-EINTR loop, or the signals are routed through signalfd (§5) so that they never interrupt anything.
Rust’s std::io::Error carries the same errno (raw_os_error()), classifies it (kind() == ErrorKind::Interrupted), and — because it is a value returned in Result — cannot be clobbered by a logging call. The ? operator is the copy-then-act pattern with the copy made mandatory.
use std::io::{ErrorKind, Read};
loop {
match port.read(&mut buf) {
Ok(0) => break, // EOF / hang-up
Ok(n) => handle(&buf[..n]),
Err(e) if e.kind() == ErrorKind::Interrupted => continue,
Err(e) if e.kind() == ErrorKind::WouldBlock => wait_for_readiness()?,
Err(e) => return Err(e),
}
}Check every return value, capture errno first, retry on EINTR, and use strerror_r. On the Rust side the compiler enforces the first two (#[must_use] on Result); the last two are still yours.
1.3 Processes, daemons, and systemd
A firmware image starts at reset and runs forever; an embedded-Linux program is started by something — a shell for development, systemd in deployment — and that something owns its lifetime. The modern rule is that a daemon does not daemonize itself (no double-fork, no setsid, no closing of standard descriptors): it runs in the foreground, logs to stderr, and lets the service manager handle backgrounding, restarts, and log capture through the journal. Everything a hand-rolled daemon used to do is a line in a unit file:
# /etc/systemd/system/adc-pipeline.service
[Service]
ExecStart=/opt/pipeline/adc-pipeline --config /etc/pipeline.toml
Restart=on-failure
WatchdogSec=2
LimitMEMLOCK=infinity
LimitRTPRIO=99
CPUAffinity=3WatchdogSec is the software watchdog of Course 3 Lab 7.1’s Jetson section: the service must call sd_notify(0, "WATCHDOG=1") (from libsystemd, or by writing the same string to the $NOTIFY_SOCKET datagram socket — the Rust sd-notify crate does this without linking libsystemd) more often than the interval, or systemd kills and restarts it. LimitMEMLOCK and LimitRTPRIO are the ulimit values §6 needs, set where they belong. The signal contract that comes with this: SIGTERM means “stop cleanly, you have TimeoutStopSec to do it”, SIGHUP conventionally means “reload configuration”, and both arrive as events in the §5 loop.
fork itself is rarely the right tool on this tier — a forked child of a multithreaded process inherits one thread and a copy of every lock, which is a deadlock waiting to happen — so subprocesses are posix_spawn in C and std::process::Command in Rust, both of which avoid the trap by construction.
2 · I/O: streams vs. descriptors
Seacord’s Chapter 8 draws the line the whole module lives on. A stream (FILE *, stdio.h) is a userspace buffer plus a descriptor; a descriptor is the kernel object itself. Streams are convenient for text and files; they are wrong for anything with timing or a wire protocol, because the buffer decides when bytes move.
| Property | Stream (FILE *) |
Descriptor (int) |
|---|---|---|
| Open / close | fopen / fclose |
open / close (fdopen converts, then fclose owns both) |
| Buffering | Fully buffered for files, line-buffered for terminals, unbuffered stderr; setvbuf changes it |
None — every write is a syscall that may be partial |
| Error state | Sticky indicators: ferror, feof; fclose returns EOF on a failed final flush |
Per-call -1 + errno |
| Formatted I/O | fprintf, fscanf |
Format into a buffer with snprintf, then write |
| Multiplexing | Not possible | poll / epoll |
| Use for | Log files, config, CSV captures | Serial ports, sockets, timers, GPIO, signals |
Two stream facts that cause real bugs. First, output you thought was written may still be in the buffer when the process is killed — a capture file that ends mid-line after a SIGINT is this; fflush before any blocking wait, and check fclose’s return value because that final flush is where a full-disk error surfaces. Second, a stream that was last written cannot be read without an intervening fflush or seek, and vice versa; alternating direction on one FILE * is undefined behavior.
Descriptors have their own trap: short writes. write(fd, buf, n) may transfer fewer than n bytes on a socket or a serial port and is not an error; the loop that finishes the job is mandatory.
static int write_all(int fd, const uint8_t *p, size_t n) {
while (n > 0) {
ssize_t w = write(fd, p, n);
if (w == -1) { if (errno == EINTR) continue; return -1; }
p += w; n -= (size_t)w;
}
return 0;
}Rust’s std::io::Write::write_all is exactly this loop, and Read::read_exact is its dual; BufReader/BufWriter are the stream layer, opt-in per handle, with flush() an explicit call. A File, a TcpStream, and a serial port all implement the same Read/Write traits, which is the descriptor model with the type system naming what each one can do.
2.1 Binary I/O and endianness
A capture file or a wire frame is bytes, not a struct. The portable way to put a uint32_t on the wire is to encode it, byte by byte, in a declared order — never fwrite(&s, sizeof s, 1, fp) of a struct, which bakes in padding, alignment, and the CPU’s endianness (AArch64 and the Cortex-M4 are both little-endian, which hides the bug until a big-endian peer or a different compiler’s padding shows it).
static void put_u32_le(uint8_t *out, uint32_t v) {
out[0] = (uint8_t)v; out[1] = (uint8_t)(v >> 8);
out[2] = (uint8_t)(v >> 16); out[3] = (uint8_t)(v >> 24);
}let bytes = v.to_le_bytes(); // [u8; 4], no UB, no padding question
let v = u32::from_le_bytes(bytes);Network byte order (htonl/ntohl) is big-endian by convention; sensor protocols are whatever the datasheet says. The rule is that a frame’s layout is a document — offsets, widths, order — and both the C encoder and the Rust encoder are written from it, then tested against each other (Exercise 11.4).
3 · Strings done safely
The embedded-Linux tier is the only one in this course where text handling is routine — log lines, config files, command parsing. Seacord’s Chapter 7 is the reference; the working subset is short.
| Never | Because | Instead |
|---|---|---|
gets |
Removed from the language in C11; unbounded | fgets(buf, sizeof buf, stream), then strip the newline |
strcpy, strcat, sprintf |
No bound on the destination | snprintf(buf, sizeof buf, …) and check the return value against the size; memcpy with a checked length |
strncpy as a “safe” copy |
Does not terminate on truncation, and zero-fills the rest | snprintf(dst, n, "%s", src), or strnlen + memcpy + explicit terminator |
strlen on untrusted input |
Reads until a NUL that may not exist | strnlen(s, max) |
strtok |
Global state, not thread-safe | strtok_r, or strsep, or a hand parser over a slice |
strerror |
Static buffer, not thread-safe | strerror_r |
Annex K *_s functions |
Optional annex; glibc does not implement it | The bounded functions above |
snprintf’s return value is the number of characters that would have been written; a value >= size means truncation happened and the buffer holds a shortened, still-terminated string. Treat that as an error on anything that goes to a wire or a file name.
Rust splits the type: &str/String are guaranteed UTF-8 and bounds-checked, &[u8]/Vec<u8> are bytes, and the conversion str::from_utf8 is fallible and explicit. Serial protocols and file formats are [u8]; only log lines are str. A String needs alloc, which on this tier is available — the discipline is that a real-time thread does not allocate in its loop (Module 6), so its log messages are formatted into a fixed heapless::String or a stack [u8; N] with core::fmt::Write, and shipped to a logging thread over a channel.
4 · Time and periodic loops
4.1 Which clock
| Clock | Property | Use |
|---|---|---|
CLOCK_REALTIME |
Wall time; jumps on NTP adjustment | Timestamps in logs, never for intervals |
CLOCK_MONOTONIC |
Never jumps; may be slewed; stops in suspend | Every interval and deadline |
CLOCK_MONOTONIC_RAW |
Not slewed either | Measuring the clock itself |
CLOCK_BOOTTIME |
Monotonic including suspend | Uptime |
CLOCK_THREAD_CPUTIME_ID |
CPU time of this thread | How much work a loop iteration cost vs. how long it took |
clock_gettime is a vDSO call on AArch64 — no kernel entry — so timestamping inside a loop is cheap. Rust’s std::time::Instant is CLOCK_MONOTONIC; SystemTime is CLOCK_REALTIME; the others are nix::time::clock_gettime(ClockId::…).
4.2 Sleeping to a deadline, not for a duration
The classic mistake is nanosleep(period) at the end of each iteration: the loop’s period becomes period + work + wake-up latency, and the error accumulates. The correct loop computes the next deadline from the previous one and sleeps absolutely:
struct timespec next;
clock_gettime(CLOCK_MONOTONIC, &next);
for (;;) {
do_work();
timespec_add_ns(&next, PERIOD_NS); // deadline advances by exactly one period
while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL) == EINTR) {}
}Overruns — an iteration that took longer than the period — must be detected, not silently absorbed: compare now to next after waking, count it, and decide whether to skip or catch up. Rust’s std::thread::sleep is relative-only; absolute sleeps are nix::time::clock_nanosleep(ClockId::CLOCK_MONOTONIC, ClockNanosleepFlags::TIMER_ABSTIME, &deadline).
4.3 timerfd: the timer as a descriptor
timerfd turns the same idea into something epoll can wait on, which is what a loop with more than one input needs:
int tfd = timerfd_create(CLOCK_MONOTONIC, TFD_NONBLOCK | TFD_CLOEXEC);
struct itimerspec its = {
.it_value = first_deadline, // absolute with TFD_TIMER_ABSTIME
.it_interval = { .tv_sec = 0, .tv_nsec = PERIOD_NS },
};
timerfd_settime(tfd, TFD_TIMER_ABSTIME, &its, NULL);
uint64_t expirations;
read(tfd, &expirations, sizeof expirations); // blocks until ≥ 1 expiration
if (expirations > 1) { overruns += expirations - 1; }The read returns the number of expirations since the last read — an overrun counter for free, which is why the exercises’ jitter tables have an “overruns” column. In Rust: nix::sys::timerfd::{TimerFd, ClockId, TimerFlags, Expiration, TimerSetTimeFlags}, with TimerFd::new, set(Expiration::IntervalDelayed(first, period), TimerSetTimeFlags::TFD_TIMER_ABSTIME), and wait() (or a raw read of eight bytes via AsFd).
4.4 Where jitter comes from
A periodic loop’s wake-up time is late by the sum of: the timer interrupt’s own granularity (CONFIG_HZ and hrtimer resolution), the scheduler’s decision to run this thread (other runnable threads at equal or higher priority; the throttling of §7), CPU frequency scaling and idle states (a core that was asleep takes longer to wake), page faults on first touch of stack or heap, and cache/TLB coldness after another thread ran on the core. Each has a knob: SCHED_FIFO, mlockall + pre-faulting, nvpmodel/jetson_clocks, CPU affinity, and — the only one this course cannot turn — a PREEMPT_RT kernel. The measurement exercise records the distribution’s p50/p99/max before and after each knob, so that the notes say which one bought what.
4.5 The Rust periodic loop, assembled
The pieces above, in the shape Exercise 11.1 asks for — everything allocated before the loop, nothing allocated inside it, every error a Result:
use nix::sys::timerfd::{ClockId, Expiration, TimerFd, TimerFlags, TimerSetTimeFlags};
use nix::sys::time::TimeSpec;
use std::time::Instant;
fn run(iterations: usize, period: TimeSpec) -> nix::Result<Stats> {
let timer = TimerFd::new(ClockId::CLOCK_MONOTONIC, TimerFlags::TFD_CLOEXEC)?;
timer.set(Expiration::Interval(period), TimerSetTimeFlags::empty())?;
let mut lateness = vec![0u64; iterations]; // allocated once, before the loop
let mut overruns = 0u64;
let mut expected = Instant::now();
for slot in lateness.iter_mut() {
timer.wait()?; // blocks; returns on ≥ 1 expiration
let now = Instant::now();
*slot = now.saturating_duration_since(expected).as_nanos() as u64;
expected += period_as_duration(period);
// do_work(&mut buffer); — fixed work on a pre-allocated buffer
}
Ok(Stats::from(lateness, overruns))
}TimerFd::wait hides the eight-byte read; to see the expiration count (the overrun counter) read the descriptor directly through AsFd with nix::unistd::read. The Vec before the loop is the one allocation; a SCHED_FIFO build pre-faults it (touch every page once) and mlockalls before the first wait. The C version is the same eleven lines with read, clock_gettime, and a uint64_t array — the difference is that the ? operator makes the error path exist.
5 · Event loops: epoll, signalfd, eventfd
A firmware main loop waits on interrupt flags; a Linux main loop waits on descriptors. epoll is the Linux primitive for waiting on many at once with cost independent of how many are idle.
int ep = epoll_create1(EPOLL_CLOEXEC);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = tfd };
epoll_ctl(ep, EPOLL_CTL_ADD, tfd, &ev);
epoll_ctl(ep, EPOLL_CTL_ADD, sfd, &(struct epoll_event){ .events = EPOLLIN, .data.fd = sfd });
epoll_ctl(ep, EPOLL_CTL_ADD, sock, &(struct epoll_event){ .events = EPOLLIN, .data.fd = sock });
for (;;) {
struct epoll_event out[8];
int n = epoll_wait(ep, out, 8, -1);
if (n == -1) { if (errno == EINTR) continue; break; }
for (int i = 0; i < n; ++i) {
if (out[i].data.fd == tfd) { on_tick(tfd); }
else if (out[i].data.fd == sfd) { on_signal(sfd); }
else { on_socket(out[i].data.fd); }
}
}Level-triggered (the default) reports a descriptor as ready every epoll_wait until it is drained; edge-triggered (EPOLLET) reports once per transition and requires the handler to read until EAGAIN on a non-blocking descriptor. Level-triggered is the safe default for this course; edge-triggered is the optimization for high-rate sockets, and forgetting the drain loop under it is a hang.
Two descriptors complete the loop:
signalfd— block the signals of interest withsigprocmask(in every thread, so block them inmainbefore spawning), thensignalfd(-1, &set, SFD_NONBLOCK | SFD_CLOEXEC)delivers them asstruct signalfd_siginforeads. NowSIGINTandSIGTERMare just another event, no async-signal-safety rules apply, andEINTRlargely disappears from the program.eventfd— a 64-bit counter as a descriptor: a worker threadwrites 1 to wake the loop, the loopreads the accumulated count. It is the cross-thread “flag set from an ISR” of Module 9, kernel-mediated.
Rust: nix::sys::epoll::{Epoll, EpollCreateFlags, EpollEvent, EpollFlags} (Epoll::new, .add(&fd, EpollEvent::new(EpollFlags::EPOLLIN, token)), .wait(&mut events, timeout)), nix::sys::signalfd::{SignalFd, SfdFlags} with nix::sys::signal::SigSet, and nix::sys::eventfd::{EventFd, EfdFlags}. Each wraps an OwnedFd, so closing is a drop and double-close is a compile error rather than a bug. The async runtimes (tokio, smol) are epoll loops with a task scheduler on top; they are the right tool for a network-heavy service and are out of scope for a real-time control loop, where the explicit loop above is easier to reason about.
6 · Real-time knobs and their failure modes
Every knob below moves latency; every one also has a way to hurt you. The table is the module’s core content.
| Knob | Call | What it buys | Failure mode |
|---|---|---|---|
| Real-time scheduling class | sched_setscheduler(0, SCHED_FIFO, &(struct sched_param){ .sched_priority = 80 }) — needs root, CAP_SYS_NICE, or an rtprio limit in /etc/security/limits.conf; chrt -f 80 ./prog from the shell |
The thread runs before every SCHED_OTHER thread and is preempted only by higher FIFO priorities |
A FIFO thread that never blocks starves the core — including the SSH session; the kernel’s throttle (/proc/sys/kernel/sched_rt_runtime_us, default 950 000 of every 1 000 000 µs) is what lets you recover, and disabling it (-1) removes the safety net |
| Priority ordering | Higher sched_priority runs first; equal priorities run FIFO until they block |
Deterministic ordering of the pipeline’s stages | Two stages at the same priority do not time-slice |
| Memory locking | mlockall(MCL_CURRENT \| MCL_FUTURE); ulimit -l must allow it |
No page faults in the loop | Locks everything, including a large heap; touch (pre-fault) the stack and buffers once at startup or the first iteration still faults |
| CPU affinity | sched_setaffinity(0, sizeof set, &set) or taskset -c 3; isolcpus=3 on the kernel command line goes further |
The loop’s core stays warm and its cache stays yours | Pinning to a core the kernel also uses for interrupts (/proc/interrupts) buys nothing |
| Priority-inheritance mutex | pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT) |
A low-priority holder is boosted while a high-priority waiter blocks on it | Only mutexes get PI; a semaphore or a spinlock does not, and a plain pthread_mutex_t under SCHED_FIFO is a priority-inversion waiting to happen |
| Clocks and power | sudo nvpmodel -m 0, sudo jetson_clocks; on the Pi, performance governor |
No frequency ramp on wake-up | Thermal throttling if the enclosure cannot take it; tegrastats shows it |
Priority inversion is the failure the exercises reproduce on purpose: a high-priority stage blocks on a mutex held by a low-priority stage, and a medium-priority thread that needs neither runs instead of the low one — so the high one waits for the medium one, which it out-ranks. The PI mutex fixes it by lending the high priority to the holder. This is the same story as FreeRTOS’s mutex priority inheritance and RTIC’s ceiling analysis in Module 10, with one difference: on Linux the medium-priority thread can be any other process on the box, so the reproduction is easy and the fix is not optional.
A SCHED_FIFO thread must block — on a timer, a descriptor, or a condition — every iteration, and the first thing the program does is install a SIGTERM/SIGINT path that cannot be starved (a signalfd in the same loop, or a higher-priority watchdog thread). Test on the board over SSH with a second session already open.
7 · Threads and sharing
7.1 pthreads and C11 atomics
A pipeline stage is a thread with a queue in front of it. The C toolkit: pthread_create with a pthread_attr_t that sets the stack size, the scheduling policy and priority (pthread_attr_setinheritsched(&a, PTHREAD_EXPLICIT_SCHED) — without it the attribute is ignored), and optionally affinity; pthread_mutex_t with the PI protocol from §6; pthread_cond_t for “queue not empty”, always waited on in a while loop against spurious wake-ups; <stdatomic.h> for lock-free flags and counters — on AArch64 every width through 8 bytes is lock-free, which the Module 9 table said the Cortex-M4 could not promise.
static void *stage(void *arg) {
struct stage_ctx *ctx = arg;
for (;;) {
pthread_mutex_lock(&ctx->in.lock);
while (ctx->in.count == 0 && !atomic_load(&ctx->stop)) {
pthread_cond_wait(&ctx->in.not_empty, &ctx->in.lock);
}
if (atomic_load(&ctx->stop) && ctx->in.count == 0) { pthread_mutex_unlock(&ctx->in.lock); break; }
struct frame f = ring_pop(&ctx->in);
pthread_mutex_unlock(&ctx->in.lock);
process(&f);
ring_push_signal(&ctx->out, &f);
}
return NULL;
}glibc since 2.28 also ships C11 <threads.h> (thrd_create, mtx_t, cnd_t) — portable across the Mac and Linux, but without attributes, so the real-time pipeline uses pthreads directly and <threads.h> only for host-side tests.
7.2 std::thread, channels, Arc<Mutex<T>>
Rust Book 16’s three tools are the same three, with ownership deciding who may touch what:
| Need | C | Rust |
|---|---|---|
| Spawn with a name and stack size | pthread_attr_setstacksize |
std::thread::Builder::new().name("adc".into()).stack_size(256 * 1024).spawn(move \|\| …) |
| Hand a frame to the next stage | Ring + mutex + condvar | std::sync::mpsc::sync_channel(N) — bounded, blocking send applies back-pressure; the frame is moved, so the producer cannot touch it afterwards |
| Share a state object | Mutex around a struct, discipline about who locks | Arc<Mutex<T>> — T is unreachable without lock(), and Mutex<T>: Sync only if T: Send |
| A stop flag | atomic_bool |
Arc<AtomicBool> with Ordering::Relaxed for a flag, Acquire/Release for a hand-off |
| Wait for all | pthread_join |
JoinHandle::join() — dropping a handle without joining is legal and detaches |
What std does not provide: scheduling policy and priority (use libc::pthread_setschedparam on JoinHandle::as_pthread_t(), or libc::sched_setscheduler from inside the thread; the thread-priority crate wraps this), affinity (nix::sched::sched_setaffinity), and a priority-inheritance mutex — std::sync::Mutex is a futex-based lock without PI, so a SCHED_FIFO pipeline that shares a lock across priorities builds one on libc::pthread_mutex_t with the PI attribute, wrapped in a small unsafe type with the Module 7 // SAFETY: contract. The exercises do exactly that, once, and then reuse it.
The Rust Book 21 thread-pool shutdown pattern is the general shape of a clean stop: the sender is dropped, every worker’s recv() returns Err, the worker loop ends, Drop for the pool joins each handle. In C the same sequence is a stop atomic, a pthread_cond_broadcast, and pthread_join in order — with the difference that C does not stop you from joining twice or forgetting one.
7.3 What transfers from Module 9, what the kernel took over
| Module 9 rule (interrupts and shared state) | On the Linux tier |
|---|---|
volatile is not atomic |
Unchanged — and now there is no legitimate volatile at all outside mmap’d device memory; sharing is atomics or locks |
| A critical section disables interrupts | Gone — userspace cannot; the equivalents are a mutex (with PI under FIFO) or a lock-free structure |
| ≤ 4-byte atomics are lock-free on the M4 | Every width through 8 bytes is lock-free on AArch64; atomic_is_lock_free still tells the truth |
| SPSC ring between ISR and main | Same structure, same memory orderings, between two threads — Acquire/Release on the indices, nothing else |
Send/Sync decide what may cross the boundary |
Identical: std::thread::spawn requires Send on the closure’s captures; Arc<Mutex<T>> is how a non-Sync T crosses |
| Priority = interrupt level, fixed by hardware | SCHED_FIFO priority, set by you, throttled by the kernel, inverted by any mutex without PI |
| The handler must be short | The FIFO thread must block — the dual failure: a handler that runs too long starves lower interrupts; a FIFO thread that never blocks starves the machine |
8 · Serial and sockets
The Jetson’s 40-pin header exposes a UART on pins 8/10 as /dev/ttyTHS1 (the Pi’s is /dev/ttyAMA0); the user must be in the dialout group. A serial port is a descriptor with a termios configuration attached, and the configuration is where hours go:
struct termios t;
tcgetattr(fd, &t);
cfmakeraw(&t); // no line editing, no CR/LF translation, 8-bit clean
cfsetispeed(&t, B921600); cfsetospeed(&t, B921600);
t.c_cflag |= CLOCAL | CREAD; // ignore modem lines, enable receiver
t.c_cc[VMIN] = 0; t.c_cc[VTIME] = 1; // read returns after ≥0 bytes or 100 ms
tcsetattr(fd, TCSANOW, &t);
tcflush(fd, TCIOFLUSH);VMIN/VTIME define what a blocking read means; with O_NONBLOCK they are ignored and epoll decides. The Course 3 host streaming harness uses this port at 921600 baud with a framing layer on top; the transport abstraction it introduces — the same frame over serial or TCP — is Exercise 11.4’s subject in both languages. Rust: nix::sys::termios mirrors the calls one for one, and the serialport crate wraps them behind Read/Write with a builder for baud and timeouts.
Sockets are descriptors too. A TCP stream to a sensor board or a host needs TCP_NODELAY (Nagle’s algorithm otherwise batches small frames), a decided-upon framing (length prefix — TCP is a byte stream, not a message stream), and the write_all/read_exact loops of §2. std::net::TcpStream has set_nodelay(true) and set_read_timeout; everything else is the same Read/Write code as the serial port, which is the point of the trait.
9 · The Rust std mapping table
For every POSIX call the C half uses, the Rust half’s home:
| POSIX / libc | std |
nix 0.31 / libc |
|---|---|---|
open, read, write, close |
File, Read, Write, OwnedFd (close on drop) |
nix::fcntl::open, nix::unistd::{read, write} for raw fds |
errno, strerror_r |
io::Error::last_os_error(), raw_os_error(), kind() |
nix::errno::Errno |
clock_gettime(CLOCK_MONOTONIC) |
Instant::now() |
nix::time::clock_gettime(ClockId::CLOCK_MONOTONIC_RAW) for the others |
clock_nanosleep(TIMER_ABSTIME) |
— (thread::sleep is relative) |
nix::time::clock_nanosleep |
timerfd_* |
— | nix::sys::timerfd::TimerFd |
epoll_* |
— | nix::sys::epoll::Epoll |
signalfd, sigprocmask |
— (ctrlc crate for the simple case) |
nix::sys::signalfd::SignalFd, nix::sys::signal::SigSet::thread_block |
eventfd |
— | nix::sys::eventfd::EventFd |
sched_setscheduler |
— | libc::sched_setscheduler + libc::sched_param |
sched_setaffinity |
— | nix::sched::{sched_setaffinity, CpuSet} |
mlockall |
— | nix::sys::mman::{mlockall, MlockAllFlags} |
pthread_create + attrs |
thread::Builder (name, stack size only) |
libc::pthread_setschedparam via JoinHandleExt::as_pthread_t |
pthread_mutex_t (PI) |
Mutex<T> (no PI) |
libc::pthread_mutexattr_setprotocol in a hand-written wrapper |
pthread_cond_t |
Condvar |
— |
<stdatomic.h> |
std::sync::atomic |
— |
termios |
— | nix::sys::termios, or the serialport crate |
socket, setsockopt(TCP_NODELAY) |
TcpStream, set_nodelay |
nix::sys::socket for raw options |
mmap |
— | nix::sys::mman::mmap (an unsafe call — it returns memory whose validity the kernel, not the type system, guarantees) |
| libgpiod | — | gpiod 0.3 (pure Rust over the gpiochip ioctls) |
nix is feature-gated: enable the modules you use (features = ["time", "epoll", "signal", "event", "mman", "sched", "term", "fs"] — confirm the exact names against the crate’s Cargo.toml for the pinned version). libc is the escape hatch: every call is unsafe and every struct is C-layout, which is the FFI story of Module 12 arriving early.
9.1 GPIO from userspace
The sysfs GPIO interface (/sys/class/gpio) is deprecated; the character device /dev/gpiochipN with libgpiod is the supported path on both boards. JetPack 6’s Ubuntu 22.04 ships libgpiod 1.6 (gpiodetect --version confirms); the 2.x API is a redesign (gpiod_line_settings, gpiod_line_config, gpiod_chip_request_lines) and the two are not source-compatible, so a C program targets the version on the board. The Rust gpiod crate talks to the kernel uAPI directly and is independent of the installed library. Line numbers come from gpioinfo against the Course 3 pin table (marker pin 7 first, then 29 and 31); requesting an edge-detecting input yields a descriptor that epoll can wait on, which is how Exercise 11.3 turns a GPIO edge into an event-loop wake-up rather than a polling thread.
gpiodetect # chips
gpioinfo gpiochip0 | head # lines, names, current consumers
gpiomon --num-events=5 gpiochip0 <line> # edge events from the shell, before writing code10 · The Jetson checklist
Before any measurement in this module — in this order, and recorded in notes.md with the outputs:
sudo nvpmodel -m 0 && sudo jetson_clocks— max performance mode, clocks pinned;tegrastatsin a second session to watch thermals and confirm the clocks held.groupsshowsgpio,i2c,dialout;ulimit -landulimit -r(or/etc/security/limits.confentries) allow memory locking and real-time priority withoutsudo.cat /proc/sys/kernel/sched_rt_runtime_us— know whether the throttle is on before running a FIFO thread; leave it on.cyclictest -m -p 80 -i 1000 -l 100000— the standard latency benchmark, run before your own loop so that your loop’s numbers have a reference on the same board in the same state;-a 3pins it,chrt/tasksetdo the same for your program.cat /proc/interrupts— which cores service the timer, the UART, and the GPIO controller; choose the loop’s core accordingly.- A second SSH session open, and the FIFO program started with a
signalfdshutdown path, every time.
Course 3’s implementation tracks place the Linux tier against the STM32: sequential, latency-bound work belongs on the microcontroller; block-structured, throughput work belongs here. This module’s measurements are what let that placement be argued from evidence instead of instinct.
11 · Lesson → exercise map
| Section | Exercise it feeds |
|---|---|
§1 descriptors, errno, EINTR |
11.1, 11.5, 11.7 |
| §2 streams vs. descriptors, short writes, binary I/O | 11.4, 11.6 |
| §3 strings | 11.6 (the safe-strings audit) |
§4 clocks, absolute sleeps, timerfd, jitter sources |
11.1 (the 1 kHz loop) |
§5 epoll, signalfd, eventfd |
11.3, 11.5 |
| §6 real-time knobs, priority inversion | 11.1, 11.2 |
| §7 threads, PI mutex, channels, shutdown | 11.2, 11.7 |
| §8 serial and sockets | 11.4 |
| §9 mapping table, GPIO | 11.3, and every Rust half |
| §10 checklist | every measurement |