You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: books/vol4/chapters/01-boundary/01-boundary.qmd
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -295,7 +295,7 @@ In computer systems engineering, developers often ask: *why cannot we run this p
295
295
Therefore, every robust physical AI machine physically partitions its compute across heterogeneous silicon (@fig-01-dual-brain):
296
296
297
297
1.**The Cognitive Proposer (Linux MPU / Edge NPU):** Operates in unprivileged user space at a deliberative cadence ($10\text{--}50\text{ Hz}$). It executes high-capacity neural policies, multi-modal perception backbones, and trajectory generators. It possesses **zero direct electrical wiring to physical actuator registers**. It can only write candidate trajectory proposals into a shared SRAM mailbox or lock-free circular ring buffer across an inter-processor communication (IPC) channel.
298
-
2.**The Real-Time Permission Referee (Bare-Metal / RTOS MCU):** Operates on isolated, dedicated silicon at a hard real-time cadence ($1000\text{ Hz}$, $1\text{ ms}$ deadline, $$< 50\,\mu\text{s}$ jitter) with static, zero-allocation memory (`malloc` is strictly forbidden at runtime). It holds **exclusive physical authority** over the PWM compare registers, digital-to-analog converters, and gate drivers. The MCU continuously checks physical safety invariants and stopping clearance. If the proposed command satisfies all physical constraints, the MCU latches the values to hardware; if the proposal is unsafe, stalls, or panics, the MCU vetoes the command and executes a deterministic physical fallback.
298
+
2.**The Real-Time Permission Referee (Bare-Metal / RTOS MCU):** Operates on isolated, dedicated silicon at a hard real-time cadence ($1000\text{ Hz}$, $1\text{ ms}$ deadline, $< 50\,\mu\text{s}$ jitter) with static, zero-allocation memory (`malloc` is strictly forbidden at runtime). It holds **exclusive physical authority** over the PWM compare registers, digital-to-analog converters, and gate drivers. The MCU continuously checks physical safety invariants and stopping clearance. If the proposed command satisfies all physical constraints, the MCU latches the values to hardware; if the proposal is unsafe, stalls, or panics, the MCU vetoes the command and executes a deterministic physical fallback.
299
299
300
300
To bridge the asynchronous Linux OS and deterministic bare-metal firmware without priority inversion or lock contention, the MPU and MCU communicate across a shared SRAM mailbox using lock-free atomic sequence locks (*seqlocks*) or double-buffered ping-pong ring buffers.[^fn-hw-soc-mailbox] The MPU publishes candidate action chunks with a monotonically incrementing sequence version; the MCU reads the trajectory atomically without ever blocking on a mutex. If an MPU core stalls or experiences an OS scheduling delay, the sequence version fails to update, immediately signaling the MCU that proposed commands have aged beyond their freshness deadline.
Copy file name to clipboardExpand all lines: books/vol4/chapters/08-perception/08-perception.qmd
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -149,12 +149,12 @@ $$\Delta x(y) = v \cdot y \cdot t_{\text{row}}$$
149
149
150
150
Under rotational motion, rolling shutter distortion becomes non-linear. If the camera undergoes angular rotation at angular velocity vector $\boldsymbol{\omega} = [\omega_x, \omega_y, \omega_z]^T$ (e.g., rapid yaw or pitch during aggressive robot maneuvers), the effective camera attitude changes continuously as scanlines are read. For a scanline at vertical coordinate $y$, the differential rotation relative to the frame start is $\Delta \boldsymbol{\theta}(y) \approx \boldsymbol{\omega} \cdot y \cdot t_{\text{row}}$. This scanline-dependent rotation warps straight physical lines into curves, induces focal-length dilation (scaling distortion during pitch), and shears vertical walls into slanted surfaces.
151
151
152
-
As an analytical example, consider a ground robot traversing an aisle at $v = 3.0\text{ m/s}$ equipped with a rolling shutter camera operating at $30\text{ Hz}$ with vertical resolution $H = 1080\text{ rows}$. If the active readout time across all rows is $t_{\text{readout}} = 33.3\text{ ms}$, the inter-row readout interval is $t_{\text{row}} = 33.3\text{ ms} / 1080 \approx $30.8\,\mu\text{s}$$. A vertical pallet edge spanning the full height of the frame does not appear vertical in the captured array. The bottom scanline ($y = 1079$) is sampled $33.3\text{ ms}$ after the top scanline ($y = 0$), producing a total physical spatial skew of:
152
+
As an analytical example, consider a ground robot traversing an aisle at $v = 3.0\text{ m/s}$ equipped with a rolling shutter camera operating at $30\text{ Hz}$ with vertical resolution $H = 1080\text{ rows}$. If the active readout time across all rows is $t_{\text{readout}} = 33.3\text{ ms}$, the inter-row readout interval is $t_{\text{row}} = 33.3\text{ ms} / 1080 \approx 30.8\,\mu\text{s}$. A vertical pallet edge spanning the full height of the frame does not appear vertical in the captured array. The bottom scanline ($y = 1079$) is sampled $33.3\text{ ms}$ after the top scanline ($y = 0$), producing a total physical spatial skew of:
With an exposure duration of $t_{\text{exp}} = 10.0\text{ ms}$, each individual scanline also suffers a motion blur width of $\Delta x_{\text{blur}} = 3.0\text{ m/s} \times 0.010\text{ s} = 0.03\text{ m} = 30\text{ mm}$. If downstream geometry reconstruction treats this image as an instantaneous rigid projection, the perceived $100\text{ mm}$ tilt causes the motion planner to identify an artificial intrusion into the free corridor. Correcting this distortion requires **rolling-shutter motion compensation**, the continuous-time geometric unwarping algorithm that utilizes high-rate ($1000\text{ Hz}$) inertial angular velocities ($\boldsymbol{\omega}$) and linear velocities ($\mathbf{v}$) to interpolate the instantaneous 6-DoF camera pose $\mathbf{T}_{wb}(t(y))$ for each individual scanline $y$, restoring rigid epipolar geometry prior to 3D triangulation.
**System:** Autonomous mobile warehouse transport robot equipped with a forward-facing rolling-shutter CMOS camera ($1080\text{p}$ at $30\text{ Hz}$, row readout time $t_{\text{row}} = $30.8\,\mu\text{s}$$, total frame readout $t_{\text{readout}} = 33.3\text{ ms}$) and an onboard 6-DoF Visual-Inertial Odometry (VIO) pipeline running a sliding-window factor graph optimizer.
157
+
**System:** Autonomous mobile warehouse transport robot equipped with a forward-facing rolling-shutter CMOS camera ($1080\text{p}$ at $30\text{ Hz}$, row readout time $t_{\text{row}} = 30.8\,\mu\text{s}$, total frame readout $t_{\text{readout}} = 33.3\text{ ms}$) and an onboard 6-DoF Visual-Inertial Odometry (VIO) pipeline running a sliding-window factor graph optimizer.
158
158
159
159
**Incident:** While executing a rapid $90^\circ$ turn at an angular velocity of $\omega_z = 2.4\text{ rad/s}$ ($137.5^\circ/\text{s}$) alongside an industrial storage rack, the robot's state estimator suffered an instantaneous divergence fracture ($>0.52\text{ m}$ lateral position jump). The motion planner commanded a corrective steer directly into the steel racking upright at $2.2\text{ m/s}$, shearing the camera mount and causing structural chassis deformation.
160
160
@@ -560,7 +560,7 @@ An observation that arrives with zero sensor noise, perfect spatial resolution,
560
560
561
561
*The scenario.* An autonomous vehicle transitions from direct sunlight ($100{,}000\text{ lux}$) into a dark loading bay ($20\text{ lux}$) at $v = 6.0\text{ m/s}$. The illumination change triggers four cameras to adjust exposure gains simultaneously, bursting raw pixel telemetry across the shared SoC interconnect at $4.8\text{ GB/s}$ against a memory controller limit of $5.2\text{ GB/s}$ ($92\%$ bus utilization).
562
562
563
-
*The systems dilemma.* Real-time safety monitors reading high-frequency IMU and wheel encoder telemetry ($128\text{ kB/s}$ at $1000\text{ Hz}$) experience head-of-line blocking in the shared memory controller, inflating access latency from $$15.0\,\mu\text{s}$$ to $2.8\text{ ms}$ ($P_{99.9}$) and starving the closed-loop traction controller.
563
+
*The systems dilemma.* Real-time safety monitors reading high-frequency IMU and wheel encoder telemetry ($128\text{ kB/s}$ at $1000\text{ Hz}$) experience head-of-line blocking in the shared memory controller, inflating access latency from $15.0\,\mu\text{s}$ to $2.8\text{ ms}$ ($P_{99.9}$) and starving the closed-loop traction controller.
564
564
565
565
*The systems verdict: Interconnect Quality-of-Service Law.* Perceptual data ingestion must never starve real-time safety reflexes. The SoC architecture must enforce hardware AXI QoS priority registers, isolating real-time safety channels on dedicated SRAM or high-priority lanes while rate-limiting or dropping perception frames during transient bus saturation.
As plotted along the dynamic operating curve in @fig-10-lease-dynamics-tradeoff (Panel A), a lease duration longer than $60\text{ ms}$ permits the component to drift outside the gripper's physical capture envelope while the planner is still driving the arm toward the initial coordinate. If the conveyor slows to $v_{\text{drift}} = 0.02\text{ m/s}$ ($20\text{ mm/s}$), the allowable validity window expands to $600\text{ ms}$. Because physical drift rates vary with workspace dynamics, the safe lease duration is a property of the environment and the tooling, not an arbitrary parameter tuned to software convenience.
212
212
213
-
::: {#fig-10-lease-dynamics-tradeoff fig-env="figure" fig-pos="t" fig-cap="**Physics-Derived Lease Validity Horizon and Ingestion Rejection Gateways**: (Panel A) Maximum safe lease duration $\tau$ as a function of scene drift velocity $v_{\text{drift}}$ across distinct task tolerance radii ($r_{\text{tol}} = 5\text{ mm}, 15\text{ mm}, 30\text{ mm}$) derived from $\tau \le (r_{\text{tol}} - \sigma_{\text{sensor}}) / v_{\text{drift}}$. The green shaded region denotes the admissible safe execution manifold, while the red shaded region marks the stale-goal collision hazard zone. Coupling lease duration to reasoning model inference cadence ($P_{99} = 650\text{ ms}$) violates the physical bound under dynamic drift. (Panel B) The 4-stage early rejection pipeline: (1) monotonic sequence and parent perception hash verification ($<1\ \mu\text{s}$), (2) $O(1)$ analytical kinematic workspace manifold $\mathcal{W}$ and static occupancy check ($<5\ \mu\text{s}$), (3) dynamic reachability filter ($t_{\min} = 2\sqrt{d/a_{\max}} \le \tau_{\text{rem}}$), and (4) semantic covariance ambiguity gate ($\lambda_{\max}(\mathbf{\Sigma}) \le \sigma^2_{\max}$). Inadmissible proposals are rejected in $$42\,\mu\text{s}$$ ($P_{99}$) with structured diagnostic feedback, eliminating $15\text{–}80\text{ ms}$ trajectory optimizer stalls and preventing planning queue starvation." fig-alt="Physics-Derived Lease Validity Horizon and Ingestion Rejection Gateways. (Panel A) Maximum safe lease duration \tau as a function of scene drift velocity v{\text{drift}} across distinct task tolerance radii (r{\text{tol}} = 5\text{ mm}, 15\text{ mm}, 30\text{ mm}) derived from \tau \le (r{\text{tol}} - \sigma{\text{sensor}}) / v{\text{drift}}. The green shaded region denotes the admissible safe execution manifold, while the red shaded region marks the stale-goal collision hazard zone. Coupling lease duration to reasoning model inference cadence (P{99} = 650\text{ ms}) violates the physical bound under dynamic drift. (Panel B) The 4-stage early rejection pipeline: (1) monotonic sequence and parent perception hash verification ($<1\,\mu\text{s}$), (2) O(1) analytical kinematic workspace manifold \mathcal{W} and static occupancy check ($<5\,\mu\text{s}$), (3) dynamic reachability filter (t{\min} = 2\sqrt{d/a{\max}} \le \tau{\text{rem}}), and (4) semantic covariance ambiguity gate (\lambda{\max}(\mathbf{\Sigma}) \le \sigma^2{\max}). Inadmissible proposals are rejected in $42\,\mu\text{s}$ (P{99}) with structured diagnostic feedback, eliminating 15\text{–}80\text{ ms} trajectory optimizer stalls and preventing planning queue starvation."}
213
+
::: {#fig-10-lease-dynamics-tradeoff fig-env="figure" fig-pos="t" fig-cap="**Physics-Derived Lease Validity Horizon and Ingestion Rejection Gateways**: (Panel A) Maximum safe lease duration $\tau$ as a function of scene drift velocity $v_{\text{drift}}$ across distinct task tolerance radii ($r_{\text{tol}} = 5\text{ mm}, 15\text{ mm}, 30\text{ mm}$) derived from $\tau \le (r_{\text{tol}} - \sigma_{\text{sensor}}) / v_{\text{drift}}$. The green shaded region denotes the admissible safe execution manifold, while the red shaded region marks the stale-goal collision hazard zone. Coupling lease duration to reasoning model inference cadence ($P_{99} = 650\text{ ms}$) violates the physical bound under dynamic drift. (Panel B) The 4-stage early rejection pipeline: (1) monotonic sequence and parent perception hash verification ($<1\ \mu\text{s}$), (2) $O(1)$ analytical kinematic workspace manifold $\mathcal{W}$ and static occupancy check ($<5\ \mu\text{s}$), (3) dynamic reachability filter ($t_{\min} = 2\sqrt{d/a_{\max}} \le \tau_{\text{rem}}$), and (4) semantic covariance ambiguity gate ($\lambda_{\max}(\mathbf{\Sigma}) \le \sigma^2_{\max}$). Inadmissible proposals are rejected in $42\,\mu\text{s}$ ($P_{99}$) with structured diagnostic feedback, eliminating $15\text{–}80\text{ ms}$ trajectory optimizer stalls and preventing planning queue starvation." fig-alt="Physics-Derived Lease Validity Horizon and Ingestion Rejection Gateways. (Panel A) Maximum safe lease duration \tau as a function of scene drift velocity v{\text{drift}} across distinct task tolerance radii (r{\text{tol}} = 5\text{ mm}, 15\text{ mm}, 30\text{ mm}) derived from \tau \le (r{\text{tol}} - \sigma{\text{sensor}}) / v{\text{drift}}. The green shaded region denotes the admissible safe execution manifold, while the red shaded region marks the stale-goal collision hazard zone. Coupling lease duration to reasoning model inference cadence (P{99} = 650\text{ ms}) violates the physical bound under dynamic drift. (Panel B) The 4-stage early rejection pipeline: (1) monotonic sequence and parent perception hash verification ($<1\,\mu\text{s}$), (2) O(1) analytical kinematic workspace manifold \mathcal{W} and static occupancy check ($<5\,\mu\text{s}$), (3) dynamic reachability filter (t{\min} = 2\sqrt{d/a{\max}} \le \tau{\text{rem}}), and (4) semantic covariance ambiguity gate (\lambda{\max}(\mathbf{\Sigma}) \le \sigma^2{\max}). Inadmissible proposals are rejected in $42\,\mu\text{s}$ (P{99}) with structured diagnostic feedback, eliminating 15\text{–}80\text{ ms} trajectory optimizer stalls and preventing planning queue starvation."}
Copy file name to clipboardExpand all lines: publishing/quarto/contents/vol4/chapters/01-boundary/01-boundary.qmd
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -295,7 +295,7 @@ In computer systems engineering, developers often ask: *why cannot we run this p
295
295
Therefore, every robust physical AI machine physically partitions its compute across heterogeneous silicon (@fig-01-dual-brain):
296
296
297
297
1.**The Cognitive Proposer (Linux MPU / Edge NPU):** Operates in unprivileged user space at a deliberative cadence ($10\text{--}50\text{ Hz}$). It executes high-capacity neural policies, multi-modal perception backbones, and trajectory generators. It possesses **zero direct electrical wiring to physical actuator registers**. It can only write candidate trajectory proposals into a shared SRAM mailbox or lock-free circular ring buffer across an inter-processor communication (IPC) channel.
298
-
2.**The Real-Time Permission Referee (Bare-Metal / RTOS MCU):** Operates on isolated, dedicated silicon at a hard real-time cadence ($1000\text{ Hz}$, $1\text{ ms}$ deadline, $$< 50\,\mu\text{s}$ jitter) with static, zero-allocation memory (`malloc` is strictly forbidden at runtime). It holds **exclusive physical authority** over the PWM compare registers, digital-to-analog converters, and gate drivers. The MCU continuously checks physical safety invariants and stopping clearance. If the proposed command satisfies all physical constraints, the MCU latches the values to hardware; if the proposal is unsafe, stalls, or panics, the MCU vetoes the command and executes a deterministic physical fallback.
298
+
2.**The Real-Time Permission Referee (Bare-Metal / RTOS MCU):** Operates on isolated, dedicated silicon at a hard real-time cadence ($1000\text{ Hz}$, $1\text{ ms}$ deadline, $< 50\,\mu\text{s}$ jitter) with static, zero-allocation memory (`malloc` is strictly forbidden at runtime). It holds **exclusive physical authority** over the PWM compare registers, digital-to-analog converters, and gate drivers. The MCU continuously checks physical safety invariants and stopping clearance. If the proposed command satisfies all physical constraints, the MCU latches the values to hardware; if the proposal is unsafe, stalls, or panics, the MCU vetoes the command and executes a deterministic physical fallback.
299
299
300
300
To bridge the asynchronous Linux OS and deterministic bare-metal firmware without priority inversion or lock contention, the MPU and MCU communicate across a shared SRAM mailbox using lock-free atomic sequence locks (*seqlocks*) or double-buffered ping-pong ring buffers.[^fn-hw-soc-mailbox] The MPU publishes candidate action chunks with a monotonically incrementing sequence version; the MCU reads the trajectory atomically without ever blocking on a mutex. If an MPU core stalls or experiences an OS scheduling delay, the sequence version fails to update, immediately signaling the MCU that proposed commands have aged beyond their freshness deadline.
0 commit comments