fix(vehicle): keep the stream connected while a drive is interrupted by an API outage - #5647
fix(vehicle): keep the stream connected while a drive is interrupted by an API outage#5647onevcat wants to merge 2 commits into
Conversation
…by an API outage When a fetch reported the vehicle offline or asleep, the stream was always torn down, including mid-drive. There the vehicle is usually only unreachable for the API for a while, and the stream keeps delivering positions; those were discarded until the next successful fetch, leaving a hole in the drive. teslamate-org#5535 already reconnects the stream once the vehicle is back online, but everything streamed in between was lost. Keep the stream while the state machine is in a driving state and record D/N/R frames into the open drive also while it is unavailable or offline. The fetch scheduled by the offline handlers stays the only way back to :available; other frames no longer schedule a fetch in those states, since every one of them would fail the same way.
✅ Deploy Preview for teslamate ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
While validating exact head The I reproduced this with streaming enabled: the initial The minimal correction appears to be either:
A regression test with onevtail - an assistant to @onevcat |
|
The review above is technically valid, but the scenario cannot occur in practice. While driving, the vehicle may be reported as Therefore, I think we should keep the current fix as is. |
JakobLichterfeld
added
note:discussion
3 participants
What happens today
When a fetch reports the vehicle as
offlineorasleep, the fetch-result handler always disconnects the stream — including in the middle of a drive.The Data/Fleet API and the streaming socket fail independently. A vehicle that the API cannot reach is very often still streaming: a cellular ↔ Wi-Fi handover during a short stop, a weak-coverage stretch, or a slow Tesla backend is enough to get
vehicle_data -> 408, which the state machine treats asoffline. The car keeps pushing positions, but the socket has just been torn down, so everything until the next successfulvehicle_datais discarded.A concrete case from a self-hosted Fleet Telemetry setup (the code path is the same on
main):Vehicle went offline while driving→Stream disconnecting ..., then 30-second retries, all 408.The window is as long as it takes for one
vehicle_datacall to succeed. Because every one of those calls is billed on the Fleet API, installations that raisePOLLING_MINIMUM_INTERVALto contain 408 storms make it correspondingly longer; in the case above it was 15 minutes.What this PR changes
offline/asleepfetch-result clause no longer disconnects the stream while the state machine is in a driving state. Its comment says the teardown is there for the asleep/offline "real online" probe, and that is still exactly what it does — every path that ends inStart / :offlineorStart / :asleepdisconnects, andStart / :onlinereuses an existingstream_pid, so no socket is leaked.D/N/Rframes into the open drive in every driving status, not only:available. The existing{_status, %Stream.Data{}}branch already anticipated frames arriving while the vehicle is unavailable — it just never received any, because the socket was always gone by then.schedule_fetch(0). The offline handlers already have a fetch scheduled, and one request per frame would only addvehicle_datacalls that are known to fail right now; with a raisedPOLLING_MINIMUM_INTERVALit would also keep re-arming the pending state timeout and push the real fetch further out.Deliberately unchanged: a fetch remains the only thing that can bring the status back to
:available. A stale or replayed frame can therefore, at worst, insert an out-of-order position — the same exposure that exists in:availabletoday — and can never flip the state machine back and forth. The 15-minute drive timeout in{:driving, {:offline, _}, _}is untouched, and after it fires thedrv == nilguard stops further frames from being written.Relation to #5535
#5535 (fixes #5534, reported in #5532) reconnects the stream after the vehicle is back online, so the remainder of the drive is streamed again and carries elevation. That is still needed for a stream process that actually died, and
maybe_reconnect_stream/1stays in place as that safety net.This PR addresses the other half: the data that was streamed during the outage. With the socket kept open, those positions land in the drive as they arrive instead of being lost until the API recovers.
Tests
logs a drive after a significant offline period while drivingnow assert that the stream is neither disconnected nor reconnected during the offline phase.Dframe is still recorded into the drive, aPframe is ignored → the API sees the vehicle again and the drive continues on the same socket.mix format --check-formatted,mix compile --warnings-as-errorsand the vehicle test suite pass locally.