Commit e2067d7
authored
Global localization G3 + Boreas crop-cliff root-cause (#91)
* Resolve issue #54/#55 as non-bugs; document executor invariant
#55 (odom_frame_id_ unused) is stale: the field is wired into the
map->odom TF broadcast in publishMapToOdomTransform.
#54 (corrent_pose_with_cov_stamped_ptr_ unlocked) is not a race: the
node is spun by a SingleThreadedExecutor with the default
mutually-exclusive callback group, so the shared pose pointer is
serialized without a lock. Documented that invariant at the executor
site and the member declaration so a future MultiThreadedExecutor /
Reentrant-group change cannot regress it silently.
Also records a Phase 1 correction in development_plan.md: the
SMALL_GICP and SMALL_VGICP backends are already fully wired, so the
Koide backend comparison is a build-and-benchmark task, not new
implementation.
* Add G3 guarded automatic reinitialization (policy + supervisor node)
Close the global-localization recovery loop. The localization component
already raises /reinitialization_requested from its C++ recovery
supervisor; this adds the consumer that queries the G2 service and
republishes /initialpose, behind explicit safety guards.
- scripts/reinitialization_supervisor_policy.py: ROS-free deterministic
state machine owning every safety decision (candidate-score floor,
minimum reset spacing, post-reset recovery evidence, non-self-resetting
attempt ceiling, edge-triggered re-arming). Built around the Phase 3
lesson that a closed loop must not reset the bounds that protect it.
- scripts/reinitialization_supervisor_node.py: thin opt-in ROS shell
wiring /reinitialization_requested + /alignment_status -> G2 ~/query
service -> /initialpose. Not in the default launch.
- test/test_reinitialization_supervisor_policy.py: the roadmap-required
regression test that fails on unsafe publication or false acceptance;
10 adversarial sequences, all green.
Docs: CHANGELOG, global_localization_roadmap.md G3 entry, and the
development_plan execution log. Runtime smoke on the Koide
kidnapped-start window is still pending an idle machine.
* Phase 2: root-cause the Boreas local_map_crop_too_small cliff (code-level)
Read the crop path end to end and pin down the failure chain without an
idle-machine run:
- the local-map crop is centered on the *predicted* seed, not the last
accepted pose, so a runaway prediction starves its own target cloud
(a positive-feedback divergence);
- with local_map_radius=300 and local_map_min_points=100,
local_map_crop_too_small is a lagging symptom (center must be ~300 m
off-map), not an independent map fault at the cliff instant;
- the crop-failure guard then drops every scan until a new initial pose
arrives, freezing the estimate into the flat ~45 m plateau.
Two candidate triggers remain with opposite fixes: prediction
divergence (estimator-side) vs map coverage / map-split (map-side). Add
scripts/diagnose_local_map_crop_coverage.py to decide between them
offline -- it counts map points within local_map_radius of every
ground-truth pose and flags the first under-covered time -- with the
counting core unit-tested in test/test_local_map_crop_coverage.py.
Running it is blocked only on the Boreas map/reference data mount.
Docs: development_plan.md Phase 2 root-cause analysis, CHANGELOG.
* Add global localization operations guide (G2 service + G3 supervisor)
docs/global_localization.md is the how-to-run companion to the
design-focused roadmap: launch commands for the opt-in G2 service and
G3 supervisor nodes, their parameters, the wiring between them, and a
table of the supervisor's safety guards. Cross-linked from the README
(demo section + docs index) and the roadmap G3 section.
* Fix coverage diagnostic: subsample undercount and brute-force memory
Self-review of the diagnostic found two bugs:
- Subsampling the map by stride S divided the in-radius counts by ~S,
which would trip a *false* local_map_crop_too_small cliff against the
absolute min_points (e.g. a real >1000-point neighbourhood read as
~100). Default to no subsample (a KD-tree handles multi-million-point
maps fine); when subsampling is explicitly requested, scale the counts
back by the stride so min_points stays comparable, and print a note.
- The brute-force fallback built a (chunk x map) array, which could OOM
on a real map; bound the effective chunk to ~20M elements.
Adds test_count_scale_compensates_for_subsampling, which fails on the
unscaled path and passes on the scaled one.
* G3 runtime smoke: validate the supervisor node end-to-end; fix shutdown
Ran the supervisor node under ROS for the first time. Two outcomes:
- Found and fixed a shutdown bug: main() caught only KeyboardInterrupt,
so an external SIGTERM / context shutdown raised an uncaught
ExternalShutdownException and dumped a traceback. Now catches both and
guards rclpy.shutdown() with rclpy.ok().
- Added test/test_reinitialization_supervisor_node_ros.py, an rclpy
integration smoke (skipped without a sourced ROS env) that fakes the
localizer + G2 service and asserts the full glue: the node receives
/reinitialization_requested + /alignment_status, debounces, calls the
G2 ~/query service, parses the candidate JSON, and publishes
/initialpose (x/y/yaw + 0.5 m std covariance) before moving to the
settling state. Validates the ROS glue the pure-policy unit tests
cannot.
* Record G3 runtime-glue validation in the roadmap
* G3: add 3-node recovery bringup launch; install supervisor scripts
Author the launch the post-reset recovery-evidence gate needs:
global_localization_recovery.launch.py brings up the core localizer (via
lidar_localization.launch.py), the G2 global_localization_node, and the G3
reinitialization_supervisor_node on a shared cloud_topic / global_frame_id,
with the supervisor wired to the G2 ~/query service and the localizer's
/alignment_status and /reinitialization_requested. The min_candidate_score /
max_attempts guards are launch args, coerced with ParameterValue so the node's
float/int parameter types are respected.
Install reinitialization_supervisor_{policy,node}.py via CMakeLists so they
resolve as ros2 run / launch executables (the node imports the policy as a
sibling module, mirroring global_localization_node.py + global_localization_query.py).
Validated: colcon install resolves both executables and the launch file;
ros2 launch --show-args resolves the full argument graph (including the
included localizer pass-through args). Live 3-node replay on the Koide
kidnapped-start window remains the pending recovery-evidence gate.
* G3: live closed-loop result on Koide outdoor_hard_01a + supervisor timeout launch args
Run the three-node bringup against the real bag + map with a kidnap injected at a
healthy location. Two results, documented in docs/g3_live_closed_loop.md:
- The G3 supervisor mechanism is validated end-to-end against a real localizer +
G2: detect -> debounce -> query -> score-guard -> guarded /initialpose publish ->
recovery-wait -> retry -> max_attempts ceiling -> safe give-up with operator
alert. The ceiling contained three confidently-wrong-or-stale candidates without
an unbounded reset loop (the Phase 3 lesson, validated live).
- Recovery did not complete because of candidate quality, not the supervisor: G2's
BBS_2D returned high-confidence but wrong candidates (107 m and 190 m off at BBS
score ~0.99) on 2 of 3 attempts; the correct pose (5.5 m, score 0.998) arrived too
late. The BBS occupancy score does not separate right from wrong.
Surfaces the evidence-backed next requirement: validate a candidate by registration
fitness before publishing (G2 per-candidate NDT/GICP fitness, and/or walk the ranked
candidate list) rather than trusting the BBS score.
Expose supervisor query_timeout_sec / settle_timeout_sec as launch arguments
(ParameterValue float coercion) so the loop can be made patient enough for the
~10-25 s BBS query latency; this run used query_timeout_sec:=30 settle_timeout_sec:=20.
* G3: ranked-candidate walk so the localizer fitness adjudicates aliased candidates
The live closed-loop run showed G2's BBS occupancy score does not separate a
190 m-wrong candidate (0.99) from the 5.5 m-correct one (0.998), so a score floor
alone cannot pick the right reset. The localizer's NDT fitness can. So the
supervisor now walks the ranked candidate list from a single query: publish the
best, and if fitness does not recover within settle_timeout_sec, publish the
next-best from the same query, letting the localizer's fitness reject aliased poses.
- G2 (global_localization_node) returns the full ranked 'candidates' list in the
~/query reply; 'top' kept for back-compat.
- Policy carries the ranked scores + a candidate index; on a failed settle it walks
to the next candidate above min_candidate_score instead of cooling down. Walking
does NOT spend a max_attempts slot -- only re-querying does -- so the ceiling stays
a true bound on queries while one query can try every pose it found. A wrong reset
is rejected by the localizer (high fitness, no accept), so walking cannot cause the
Phase 3 acceptance blowup.
- Supervisor node delivers the ranked scores to the policy and publishes the
candidate at the policy's requested index.
Regression tests: policy walk recovers on a lower-ranked candidate without spending
attempts; list-exhaust still respects max_attempts and gives up; the score floor
truncates the walk. ROS integration test: the node walks 0->1 within one query and
recovers (query_calls==1). 21 related tests green. Live validation of the walk on
the kidnap window is pending (does the real BBS top-K contain the true pose).
* G3: live walk run pins the blocker to BBS query latency; expose G2 speed args
Second live kidnap run with the ranked-candidate walk enabled corrects the earlier
candidate-quality diagnosis (docs/g3_live_closed_loop.md):
- The walk is live-validated: the supervisor walked all 16 candidates from one query
(each a next_candidate transition, published by index), kept every guard, then
re-queried -- exactly as designed.
- Candidate generation was fine: BBS put the TRUE pose at rank 1 (-107.1, 18.44,
score 1.0; ground truth -107.2, 17.7 at query-issue time, ~0.8 m off).
- It still did not recover because the BBS query took ~23 s (runtime_sec 23.376). The
candidate is computed on the query-issue scan but published ~23 s later; at the
replay rate the moving vehicle was ~12-17 m away by then, outside the registration
basin, so the localizer rejected the now-stale correct pose. Every candidate shares
that stale scan, so the walk cannot rescue it.
So the primary lever for live recovery is G2 query speed, not candidate quality or
ranking. Expose G2 search-cost parameters as launch args (g2_angular_resolution_deg,
g2_max_scan_points, g2_pyramid_depth, g2_max_candidates) so a faster/coarser config can
be selected for the runtime loop and trade a little accuracy for a fresher seed.
* G3: fast-G2 run shows query speedup is necessary but not sufficient
Exposing G2 search cost as launch args and running coarse (yaw 10 deg, 256 pts) cut
the BBS query from ~23 s to ~4-8 s. Recovery still did not complete: the walk spends
settle_timeout_sec on each earlier candidate, so a near-correct candidate that is not
rank 1 (here rank 3, -107.9/12.0) is published ~20 s after the query and is stale
again. The latency budget that matters is query + rank*settle, not just the query.
Caveat for this run: the one-shot seed publisher lost a transient-local discovery race
at the slower replay rate, so the healthy baseline did not establish; the query-timing
and walk-latency observations stand regardless. Documented in g3_live_closed_loop.md
along with the real levers (cut query hard AND keep truth at rank 1, or motion-compensate
the seed, or scope to low-speed kidnaps) and a harness note to republish the seed.
* G3: clean run isolates the blocker to the localizer post-reset re-lock path
Fourth live run, with the seed harness fixed, had a healthy baseline, a fast query
(~5 s), and a correct fresh rank-1 candidate all at once: query 1's top was
(-106.10, 9.04, 150 deg) vs ground truth (-106.8, 8.6, 156 deg) -- 0.8 m and 6 deg
off in full 3-DOF. It still did not re-lock: the localizer went tracking -> recovering
(reject_measurement), streamed fitness > 6, and sat at the seed with fitness ~27 -- at
a pose where the same area scored 0.23 in healthy tracking.
This eliminates everything upstream of the localizer (candidate generation, ranking,
the walk, query latency, yaw). The decisive blocker is the localizer's own post-reset
re-lock path not re-converging from a correct externally-published /initialpose during
the recovering state -- likely the same init_guess / local-map-crop centering the
Boreas root-cause flagged. Confirming needs code-level C++ investigation, not more
replays; the five runs isolated it by elimination. Documented in g3_live_closed_loop.md.
* G3: carry z/roll/pitch from localizer pose onto 2D candidates (the re-lock blocker)
Code investigation of the failed re-lock found the root cause, and it is upstream of
the localizer after all: G2 candidates are 2D (x, y, yaw; z=seed_z_m=0), and the
supervisor published the reset with z=0. On the Koide outdoor map the true z is ~-11 m,
so every reset seeded the pose ~11 m above ground -- outside the NDT z-basin -- and the
lock never took even when x/y/yaw were correct (the fourth live run's rank-1 candidate
was 0.8 m / 6 deg off in x/y/yaw yet held fitness ~27). It worked on the flat HDL demo
only because z=0 was right there.
Fix: the supervisor subscribes to the localizer pose (/pcl_pose) and carries its
z / roll / pitch onto the candidate, overriding only x / y / yaw -- the vehicle's height
and attitude drift slowly even while xy tracking is lost, so the last pose is a good
carry-over. Falls back to reset_default_z_m if no pose seen yet. Adds _quat_from_rpy /
_roll_pitch_from_quat helpers and pose_topic / reset_default_z_m parameters.
Regression test: a faked /pcl_pose at z=-11.05 makes the published reset use z=-11.05,
not 0. 16 related tests green (13 policy + 3 ROS).
* G3: recovery-evidence gate met live -- z-carry fix closes the closed loop
With the z/roll/pitch carry-over fix, the clean kidnap run recovers end-to-end on the
real Koide window: the reset now publishes z~-9.4/-7.3 (carried from /pcl_pose), a
candidate locks, fitness falls from ~30 to 0.10, and the supervisor logs
recovery_confirmed -> standdown -> idle (re-armed). The last G3 Decision Gate --
post-reset recovery evidence with a real localizer -- is satisfied.
Document the root cause (2D candidate z=0 vs ~-11 m true ground, outside the NDT
z-basin) and the success in g3_live_closed_loop.md, and mark the gate met in the
roadmap. Remaining work is quality (faster/first-try recovery via lower query latency
and per-candidate registration scoring), not correctness.
* G3: cap candidate walk depth so a stale query re-queries on a fresher scan
The live kidnap run spent ~95 s walking all 16 candidates of a stale first
query (none of whose poses were the true location) before re-querying -- and
it was the second query, on a newer more distinctive scan, that returned the
true pose at rank 1 and locked. Past the top few BBS occupancy maxima a fresh
query beats a deeper walk, so add max_walk_candidates (default 4): abandon the
query after the top few fail to lock and re-query instead of grinding ranks
5-16. Pure policy change with two unit tests; wired through the node param and
the supervisor_max_walk_candidates launch arg (set high to restore full walk).1 parent 33eed73 commit e2067d7
17 files changed
Lines changed: 2328 additions & 16 deletions
File tree
- docs
- include/lidar_localization
- launch
- scripts
- src
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
19 | 34 | | |
20 | 35 | | |
21 | 36 | | |
| |||
27 | 42 | | |
28 | 43 | | |
29 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
30 | 48 | | |
31 | 49 | | |
32 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| 143 | + | |
| 144 | + | |
143 | 145 | | |
144 | 146 | | |
145 | 147 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
95 | | - | |
96 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
97 | 102 | | |
98 | 103 | | |
99 | 104 | | |
| |||
283 | 288 | | |
284 | 289 | | |
285 | 290 | | |
| 291 | + | |
286 | 292 | | |
287 | 293 | | |
288 | 294 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
81 | | - | |
| 80 | + | |
| 81 | + | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
88 | | - | |
89 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
90 | 92 | | |
91 | 93 | | |
92 | 94 | | |
| |||
166 | 168 | | |
167 | 169 | | |
168 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
169 | 219 | | |
170 | 220 | | |
171 | 221 | | |
| |||
496 | 546 | | |
497 | 547 | | |
498 | 548 | | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
499 | 612 | | |
500 | 613 | | |
501 | 614 | | |
| |||
0 commit comments