TeslaMate core roadmap: modular data sources, then Rust #5558
Replies: 11 comments 24 replies
|
@jlestel, @Bre77 — you operate the largest proxy-based setup we know of, so two of the open questions above are ones you have real-world data on:
To be clear on scope: the proxy path keeps working as described above, and nothing here asks you to change anything. We're after your operational experience for the event contract design. |
|
Teslemetry never touched the Owners API, so i actually have limited knowledge of the "old way". However i understand Fleet Telemetry a lot, and Tesla does have some enforcemented minimums, but so does Teslemetry (cost control). The contract for this is full documented at this living JSON https://api.teslemetry.com/fields.json I am currently going through a process to document the undocumented enums based on fleet observations. You may also find https://github.com/Teslemetry/tesla-protocol useful, can easily add rust bindings. Ill try answer the other questions when i can validate my memory. |
|
The following is a partially AI generated answer to your questions based on my real-world data. Q1 (format fidelity): we don't operate an owner-shaped stream at all — our SSE is a native 239-field envelope, so nothing is dropped because an old format can't express it. What we do transform: numeric rounding to catalogued scale to reduce floating point waste (91 fields), invalid values become null for the 10 nullable fields and are omitted otherwise, protobuf type tags and unknown-enum raw integers aren't exposed, and units stay native. Rate policy lives in the vehicle config we send (interval/delta/resend) — 89 of 239 fields default slower than their documented minimum (for cost saving), and users can request lower through our UI. There's no second sampler after ingest. Q2 (delivery guarantees): measured from a fixed 7-day production window — out-of-order arrivals are real but rare (4 inversions in ~473k publishes, ≈8 per million, seconds-scale lateness); zero exact duplicates across all record types (the only repeated IDs are the deliberate connect/disconnect pairing); reconnect gaps are hard to prove but I do believe the vehicle sends a buffer of untransmitted events when it reconnects (connectivity related, NOT sleep). We do cache all events server side so that clients can get the latest state even if they were not listening. |
|
Thank you — I checked your figures against What the catalogue settles, and one thing it makes urgent. Every field defaults to Working assumption, and the question that decides it. Interval defaults spread across 1s (115 fields), 10s (51), 30s (4) and 60s (69), with rate policy living in the vehicle config and no second sampler after ingest. Our working assumption is therefore that partial, field-level updates have to be the primitive in our event contract, with the owner API's atomic snapshot as the special case. But Three follow-ups, unchanged in substance:
Three small things that would help everyone consuming the catalogue, if they're cheap on your side:
On semantics that vary by firmware: On One fairness point on the original post. Calling third-party access a "detour" describes how TeslaMate consumes it today, not what you offer. Your stream isn't owner-shaped at all; we simply don't speak it yet. That gap is the thing this work exists to close. 🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
Just my two cents guys, chose Go instead of Rust for this solution. As much as prefer rust, Go is better for this case IMHO. Again, I am not telling what, just adding some advice. |
|
I think we need to rethink our testing strategy. Currently testing occurs by pushing inputs into a Vehicle and looking at the outputs. And maybe the key problem here is that Vehicle is too complicated. The tests are fragile, changes to Vehicle is likely to break the tests, including tests that are not related to the change. And if the tests break it is difficult to find out what went wrong. Even AI models can struggle and go around in circles (seen this happen with my refactoring PR). I consider these tests integration tests just because of the large scope of code being tested. We probably do need to have some integration tests. But we shouldn't rely entirely on them. We should ideally have more unit tests then integration tests. Possible things to think about:
Not sure how practical this will be. I often start new projects with good intentions and then get stuck along the way, e.g. thin API wrappers are not good candidates for unit testing IMHO. If we do encounter problems, we should probably discuss them. |
|
Agreed on the destination, and I think your diagnosis is right — but I'd separate two things that are getting conflated, because once they're apart I don't think we actually disagree. Fragility comes from coupling, not from scope. Today's tests aren't brittle because a lot of code runs; they're brittle because they assert on mock call sequences — The pure functions don't exist yet, and extracting them is the refactor. That's the ordering argument: unit tests written against a new decomposition can't tell us the decomposition preserved behavior — they test the new shape against itself. Only an outer net catches "drives now end one sample later than they used to". So characterization first isn't an alternative to the pyramid, it's how we get to the pyramid without breaking people's data on the way. Where you're right and Phase 1 has no answer: diagnosability. When a golden-master case fails you get "row 47 differs" and no idea which transition caused it. Partial mitigations are small single-purpose fixtures instead of long drives, and a differ that reports the first diverging event index. But the real answer is yours — the unit layer is the diagnostic layer. The two suites do different jobs: the outer net proves behavior was preserved, unit tests tell us where it broke. Two things I'd change in the plan on the back of this:
On your caveat about thin API wrappers: I think that maps exactly onto the source layer, and it's a reason the split helps. Sources are I/O adapters and deserve contract tests against recorded payloads, not unit tests. The core, once it's pure functions over an event stream, is the part that unit-tests well. Put differently, the source abstraction is what creates the seam your pyramid needs — the two pieces of this work aren't independent. And yes, let's discuss the moment we get stuck rather than pushing through. The "AI goes in circles" symptom you saw is worth taking seriously as a signal about the suite itself: if a change can't be localized from a failure, that's the test design telling us something, not the tooling. 🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
I have had several attempts at writing Tesla owners API clients in rust. old now, but still might have some good ideas: https://github.com/brianmay/fake_luxary_api/blob/main/fla_client/src/lib.rs still current: https://github.com/brianmay/robotica-rust/blob/main/robotica-tokio/src/services/tesla/api.rs I haven't looked at these in a while. If I did I imagine I might want to have words with my past self as to why I did various things :-) The new version has support for logging metrics using opentelemetry (see init code https://github.com/brianmay/robotica-rust/blob/main/robotica-backend/src/logging.rs). Good metrics/remote logging is probably a good idea. Although I do find keeping it up-to-date and working can be painful (all relevant crates have to be updated at the same time; and recently AI incorrectly told me the TLS configuration wasn't required which broke all logging). And I never remember to look at the logged data. |
|
Thanks — I went through
The bigger find is It also settles a question about the test strategy: mismatches like that only surface against real recorded payloads. A simulator produces well-formed data by construction, which is why it can't be the safety net — but it's good for exercising paths that are hard to record, like a car that starts charging exactly on a poll boundary. Two different jobs, both useful. For the two API clients, I'd treat them the same way: not as a starting structure, since the source abstraction makes the architecture different anyway, but as a record of the Tesla-specific behavior you already hit — the auth and refresh flow, how the stream behaves on reconnect, and the places where responses don't match documentation. Worth reading before we write the owner-API source, not after. One practical note: the repo is GPL-3.0 and TeslaMate is AGPL-3.0. Those combine cleanly under §13 of both, and since you hold the copyright it's your call regardless — but worth settling explicitly before anything moves across rather than afterwards. On observability, I'd like to turn it into a decision rather than leave it as a good idea, and your last line is the reason: keeping it working is painful and you never look at the data. That's telemetry without a consumer, and it's the default outcome unless the consumer is named up front. Two things shape what makes sense for us. TeslaMate is self-hosted, so there is no central endpoint and for privacy reasons there shouldn't be one — nothing gets exported anywhere by default. What we can rely on is that every user already runs Grafana, so an opt-in Prometheus endpoint plus a shipped dashboard needs no new infrastructure and has a consumer on day one. More importantly, the consumer that pays for itself is support. Our recurring cost is bug reports nobody can reproduce — "my drive was split", "this charge is missing". What resolves those isn't a dashboard, it's structured diagnostics a user can attach to an issue: which events arrived, which transitions followed, why a drive was closed. That argues for instrumenting the core's decisions rather than HTTP plumbing, and for output a person can read and paste. Your maintenance experience argues for one more thing: a thin internal facade for emitting, so a crate version bump doesn't reach into the call sites. If OTel becomes painful we swap the facade, not the core. So: decision-level diagnostics first because they pay for themselves in triage, opt-in Prometheus second into the Grafana users already have, no export anywhere by default. Small enough to stay maintained, which is the failure mode you're describing. None of this needs touching before the event contract settles — I'm flagging it as intent, not as work starting. 🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
I updated the license of the fake_luxary_api to AGPL 3 o greater. Was purely an oversight. Nobody else has contributed changes or even used this project (to the best of my knowledge), so I think I am safe to make license changes. |
|
Name? Teslamate V5? Perhaps the first and one of the most important decisions we will need to make... Need to be something distinct so we don't get confused when referencing old and new code. |
|
Agreed that it matters, and I'd split it into two questions, because I think only one of them is about version numbers. What users install. Next release is 4.1.0 — that's already what's in the tree. The Rust core landing is a major bump, so 5.0.0. That also lines up with what the discussion says about the schema: since the schema is explicitly not an interface, reshaping it isn't a breaking change by our own definition, so the whole source abstraction can ship across 4.x. The major is reserved for the language and runtime change, not the architecture change. What we call the code. This is the part a version number handles badly, and I think it's what you're actually after. During a strangler port both cores exist side by side for a long time, and 4.x releases will already contain new-architecture code — so "the V5 code" is ambiguous exactly when we need it not to be, and it stops meaning anything the day 5.0 ships. What works is naming the component rather than the release: the Rust core gets a crate and directory name and is referred to by it, and the current one is "the Elixir core". Unambiguous in a PR title, in an issue, and in a file path, and still correct in three years. The discussion post already drifts into this without meaning to — "the core", "the Elixir path". I'd keep that name boring and descriptive rather than a codename. A codename costs every new contributor a lookup, and it eventually stops matching what the thing does. One caution on reserving 5.0.0 publicly now: the post deliberately carries no dates, and pinning a number to an undated milestone means either 4.x runs for a very long time, or we end up needing 5.0 for something else and the mapping breaks. I'm happy either way — but if we say it out loud, we should be comfortable with 4.x being where everything happens for a while. 🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
Unfortunately not keeping up with all the discussion here. But wondering do we create a new git project from scratch, or do we start off with a new git tag in existing project and delete most of the files, and start with a clean slate? Or something else? I tend to lean towards new git project, but not too concerned either way. I think should be able create a very basic project, AGPL license, nix build (no module), basic "Hello World" type rust application, maybe simple CI and unit test infrastructure. That should be simple and non-controversial start. Then we can start making some decisions. |
|
Hi @JakobLichterfeld, @brianmay — greetings from Augsburg, and from near Stuttgart as of tomorrow. One note upfront: this is written with help from Claude 5, since my English is nowhere near as fluent as my Swabian. I run a self-hosted TeslaMate whose data comes from Fleet Telemetry streaming: my own receiver, no proxy in between, in production since mid-June. Since you asked for actual field delivery, here is one day of measurements: 24 drives, one Supercharger session, ~15.5k records. 57 of 59 configured fields are delivered. Never seen: Configured intervals bound the rate, they are not a delivery promise:
Fields that rarely change simply arrive rarely. Conversely almost anything can burst down to ~1 s. A source can promise "not faster than N", not "a sample every N". Fidelity: the gap to old streaming is categorical, not gradual. Charging is absent from the old format entirely — Backfill: reconciliation has to be idempotent and keyed on something stable, not on "what happened since last time". After an offline gap the car replays buffered data, and a naive scan creates duplicate sessions. Marking reconstructed ranges explicitly was the only thing that made repeated scans safe. Policy placement: treating the feed as active based on freshness rather than on the connection being up removed a whole class of bugs — a connected stream delivering nothing is indistinguishable from a live one at socket level. Happy to post the full 57-field table with observed intervals if that helps the contract. |
|
Split out as #5586, and the reason it's worth flagging here: it is not a Fleet Telemetry problem.
So the pattern from @kdjkdjkdj's report generalises past the source that surfaced it. His mixed-row case is what made us look; the short-session case was already there, on the path almost everyone runs. To be precise about what the production data does and doesn't show: it proves the short-session mode deterministically, and it does not independently confirm the mixed-row mode — the For the contract discussion this is a second, distinct class from #5583. That one is about building a record from a single message instead of from known state. This one is about an aggregate over rows that silently assumes they're homogeneous — and it survives any change to how records are written, because it runs afterwards. Worth remembering when we get to the segmentation logic: a projection that fills gaps correctly still leaves aggregates that discard rows they can't interpret. 🤖 Drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
Side note: "A session does not begin with a complete state" is a problem Teslemetry solves, we send a full copy of cached telemetry on new SSE connections. |
It landed. The vehicle does buffer and replay — and Before the numbers, the honest framing: this basis is thin and I built it specifically to answer your question, not from accumulated operation. It was 34 °C here today and a couple of the runs are sloppy for that reason — one I had to repeat because the car was still preconditioning and would not suspend. Read it as one clean observation, not as a survey. Waiting for a natural disconnect didn't workMy first idea was a tunnel: a real radio outage while telemetry is flowing, no intervention needed. I drove through the Kappelbergtunnel twice. No disconnect at all — 1,228 records over that hour, event-time gaps of at most 5 s, no arrival gap over 30 s, arrival−event offset never above 2 s. It has good cellular coverage, so it is not a usable test. The two gaps already sitting in my capture were no better. Both (40 min and 895 min) are the car going to sleep — no telemetry was generated during them, so there was nothing to buffer and So I staged oneWith the car awake, parked at home and streaming at a ~3 s cadence, I took the receiver down for exactly 180 seconds:
Result
Raw records (VIN removed). The car was parked, so only
|
| reconnect after … | behaviour | observations |
|---|---|---|
| vehicle sleep | full 57-field snapshot within 0–22 s, no replay | 3 of 3 |
| receiver outage while awake | complete replay, no snapshot | 1 of 1 |
Both are sensible choices. But "did I just receive the present or the past?" is answerable only by inspecting the payload shape, and only in hindsight. If you are declaring source capabilities anyway, this looks like one worth declaring explicitly: does this source replay after a gap, or resynchronise?
What this cannot show
- One staged outage of 180 s, car parked. I don't know the buffer depth, and I don't know whether a longer gap, or one during driving, behaves the same. If the buffer has a limit, my test was well inside it.
- The snapshot-versus-replay contrast rests on 3 + 1 observations.
- I briefly thought snapshots aligned with state transitions — three fell within seconds of a drive boundary. The afternoon's driving refuted it: two of five snapshots sat mid-drive. Mentioning it so nobody builds on a pattern that isn't there.
Still owed from my side: the per-field table on both clocks. That needs a day with real driving in the capture window; the capture is still running.
If I have made a reasoning error anywhere in how I gathered or read this, I would rather hear it than not. And if there is other data that would help — different fields, a longer staged outage, a gap during driving, or anything else that only someone running their own receiver can produce — tell me what to record and I will run it.
🤖 Drafted with Claude Code (Opus 5)
|
@Bre77 — that closes the initial-state question for one source and turns it into a capability difference rather than a universal problem. A provider that ships cached telemetry on every SSE connect means the consumer never needs a warm-up on that path, and a consumer talking directly to |
|
@kdjkdjkdj — staging the outage was the right call after the tunnel didn't cooperate, and the result settles the question. Three things I'd add, one of which may change what the flag means. TeslaMate already implements the comparison you're describing. Your point 3 is the one with teeth. Three minutes of stale positions arriving as current location isn't degraded data, it's wrong data, and drive segmentation would act on it. Parked it's harmless; mid-drive it would produce a fabricated position history. Worth stating plainly in the capability list: a source that cannot expose event time cannot be trusted with segmentation, full stop. On That's testable with the capture you already have running, at no extra cost: over a multi-day window, does One methodological caveat you didn't list. On your recovery-strategy table: with @Bre77's note there are now three observed behaviors, not two — replay after an outage, snapshot after sleep, and provider-side cached state on connect. So the capability isn't a yes/no. Something closer to: on reconnect, does this source replay the gap, resynchronize current state, or neither? A source that does neither forces the consumer to warm up, which is where you started. What would help next, in the order I'd value it:
And noted on the snapshot-versus-state-transition pattern being refuted by your own afternoon. Reporting a hypothesis you disproved is more useful than most confirmations. 🤖 Comment drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
Thanks for putting this together so clearly — the combination of modular sources + explicit event contract, sequenced properly before the Rust port, feels like the right long-term direction. The emphasis on characterization first and keeping the architecture change decoupled from the language change is especially appreciated; that kind of discipline is rare and valuable on a project that holds people’s historical driving/charging data. I’d like to offer concrete help on the database side and the official Grafana dashboards once the shape of the schema changes becomes clearer. Happy to review migration plans, help keep the shipped dashboards in sync, or assist with any performance-related adjustments that surface. A few design questions that would help me (and probably others who maintain custom panels or care about long-term reporting) understand the intended scope:
No pressure on answers; just trying to understand where the boundary sits so any help I offer stays aligned with the actual goals. Happy to stay quiet on the implementation side and only chime in on design / dashboard / migration topics as they surface. |
|
I can't speak on behalf of @JakobLichterfeld so what I say may not be correct. I imagine in the process of doing a rewrite we will come across aspects, e.g. in the database design - that we consider to be broken. And ask why on earth did we ever do it that way. Allowing schema changes, when required, would allow us to fix the problem, rather then be forced to carry around legacy baggage. I would like to at least try to make any schema changes to the Elixir code too. So that they can be both use the same database. But we won't know if that is practical or not until we get more solid detail. |
|
@swiffer — thank you, and yes to all three areas you offered. Really love your dashboard and DB magic. Let me answer the four questions directly, because the boundary you're asking about sits in two different places. 1. What drives the schema changes. Primarily the input side: a field-level event model, partial updates, multi-source merging, backfill semantics. What I do want is for the resulting shape not to foreclose better views — but enabling them is not what drives the cut. 2. Is "richer views" an explicit goal? Not of this work — it's the step after it. The goals here are maintainability, correctness of the state machine, and removing the single-wire-protocol bottleneck. Designing storage around views nobody has specified yet would couple two decisions and leave neither validatable, which is the same mistake as coupling the architecture change to the language change. The order is: get the model right, then design what can now be shown. That second step is real and intended, not a polite deferral — and it is where your input weighs most. 3. Dashboards: a standing commitment, not a one-off. Expect the schema to change repeatedly — not once during this work and then settle, but as an ongoing part of development, whenever the model turns out to be wrong. The commitment is per-change and permanent: the shipped dashboards keep working, and every schema change ships with its dashboard updates in the same release. That is what keeping them in sync means, and it does not expire. What is not committed is restructuring the dashboards around new capabilities — that belongs to the step in answer 2, and it's the part I'd rather not decide alone. 4. Long-term reporting performance. This is never "too late" — what changes is the price. Every access pattern we get wrong and correct later is another migration, and every migration costs each user a maintenance window. Multi-year position tables on modest hardware are unforgiving, and they are exactly the ones you don't want to rewrite twice. So the ask isn't "hurry before we lock it", it's "the earlier it lands, the fewer migrations it costs the people running this". And it's knowledge I don't have: which queries hurt today, on what hardware, over what history depth, and which access patterns the current shape fights against. You and the people maintaining custom panels are the ones who can answer that. @brianmay — agreed, and I'd go further. Making the schema changes in the Elixir code too isn't only about not carrying legacy baggage forward; it's a precondition for the test strategy. Phase 3 gates the Rust core by driving both implementations from the same fixtures and diffing the outputs. If the two ever diverge on schema, that comparison stops being possible and the port loses its safety net. A shared database isn't a nicety during the transition — it's what makes a strangler port verifiable at all. Which also answers where the schema work happens: in Elixir, now, incrementally, with the same migration discipline we already have. Not as a big-bang reshape at the language boundary. Where I'd start: the charging areaSince you asked where the boundary sits, here is a concrete first candidate. It's first because the debt is on record rather than a matter of taste — #5586 exists because the code assumed something the schema permits it to violate.
The fix now going in #5592 is a small preview of the seam. It touches the Elixir integration, a Grafana panel that had replicated the same logic, and a database constraint — because a stored phase count of 0 was read one way by 🤖 Review drafted with Claude Code (Opus 5 high) — sponsored by Claude for Open Source |
|
When we have both Elixir and Rust, challenge will be do we create migrations in Elixir and/or Rust? We don't want to have Elixir and Rust migrations competing to do the same thing. It is likely that they will have different tracking to monitor which migrations have been applied already (depending also on which database library we end up using - I have ideas on rust libraries we might want to use, but that should probably go into another subthread or maybe new discussion even). For early testing, I would recommend migrations in Elixir only. This will mean you for initial installs you will need to install the Elixir code to keep that database migrations current. When this gets annoying we can re-evaluate. And maybe we can add migrations to Rust and somehow make it possible to disable the Elixir migrations so that they don't compete with each other. |
|
Hi @JakobLichterfeld, coming to this late. Most of the delivery side has since been measured far better by @kdjkdjkdj, so let me stick to the one seat nobody else here occupies: I run the only setup that speaks the owner-API shape end to end. I terminate Fleet Telemetry (Pub/Sub) and re-emit it as the two surfaces TeslaMate consumes, which is exactly the "detour" your Q1 was about. Here's where that format caps, concretely. Q1, the cap is 13 fields wide, and the rest isn't streamed at all. TeslaMate's streaming channel is a fixed 13-slot CSV vector: [ts, speed, odometer, soc, elevation, est_heading, est_lat, est_lng, power, shift_state, range, est_range, heading] That's the whole vocabulary the stream can carry. Everything else Fleet Telemetry gives me, the entire charging group, plus doors/windows, sentry, lock, TPMS, climate, software-update progress, has no slot, and reaches TeslaMate only by round-tripping through polled vehicle_data, which I assemble by folding the fields back into the owner-API JSON. So the poll loop you want to delete isn't optional in my topology: it's the only path for ~90% of the fields, and it's drive-shaped, I only forward a frame carrying lat/lng + gear, so charge-only and parked frames never hit the stream at all. That's the concrete form of "categorical, not gradual": on the owner-API path a streaming-only source can't express a charge, full stop. One field-level note (same as @kdjkdjkdj): no Power field, I take charging power from AC/DCChargingPower and reconstruct drive power from EnergyRemaining deltas. A derivation with my own conventions, which is your argument for it living in the source, normalized, before the core sees it. Initial-state / resync. My resync is the poll: I serve a server-side vehicle_data cache, precisely because the stream can't cold-start a consumer. Subscribing alone never produces correct state, so "how does this source bring a consumer to current state" is worth declaring as an explicit capability rather than assuming it. Mine happens to be a pollable cache atm. Q2, guarantees. I've seen a few out-of-order arrivals and normalize them back before they reach TeslaMate with a per-VIN FIFO plus a monotonic createdAt guard that drops any frame older than the last forwarded for that VIN. So it's mine to remove, exactly your "sources normalize before handing over", it shouldn't land in the contract on my account. One production number reinforcing a source obligation already raised: TeslaMate's close_drive/1 deletes any drive with <2 odometer samples or <10 m distance. At coarse odometer downsampling I measured 100% of sub-60 s drives and ~60% of the 60–120 s band silently deleted, until I moved Location/Odometer to a 25 s interval and dropped the odometer minimum_delta to 0.1 mi. So "a drive is at least two positions far enough apart" is a hard floor a source must guarantee, and Fleet Telemetry defaults don't meet it out of the box. Happy to share my full Fleet-Telemetry-field => vehicle_data mapping if it helps the fidelity table. |
|
The per-field clock table I promised on 3 Aug, from a day with real driving. It answers the question, but the answer is a negative result — and the two things I found while producing it matter more than the table itself. Basis: 10,629 The table: there is no per-field structure
Every one of the 57 fields has a median of 0 s. The only spread sits on The Two things reproduce from earlier in the thread, now on a driving day: The comparison also separates three regimes cleanly, which is worth noting as a method: normal operation sits at 0–1 s; a replay after a receiver outage shows a monotone positive ramp up to the outage length (the staged 3-minute outage from 4 Aug is visible in this data as single values stepping to +187 s); and cold session starts show something else entirely. Finding 1 — the session-start snapshot's event time postdates its own arrivalFive records out of 10,629 carry a
Across all 15 session starts in the window the split is clean: ten starts after gaps of 0.14–2.68 h sit at 0 to +9 s, five starts after gaps of 7.97–42.48 h are negative, and within those the magnitude grows with sleep length. The effect is gone immediately — the next record, 1–3 s later, is back to +1…+5 s. Our side is excluded by measurement rather than by argument. That leaves two candidates on the vehicle side which this data cannot separate: either the vehicle's clock was off at that moment, or For what it's worth the shape of the data does suggest a mechanism, offered as a guess rather than a finding: the vehicle's clock free-runs while the car is in deep sleep and resynchronises within seconds of waking, so the snapshot — emitted before that resync — carries the free-running value. That would fit three things at once: the error is gone on the very next record 1–3 s later, its magnitude grows with the length of the sleep, and the threshold sits between 2.68 h and 7.97 h, which is roughly where this car stops being merely idle and goes properly to sleep. It is also direction-agnostic, which is rather the point — it predicts drift whichever way the oscillator errs, and this measurement only ever gets to see the half that runs ahead. Finding 2 — the snapshot's content is the pre-sleep stateThis one is sharper, and I think it bears directly on the contract. Outside temperature as a probe, because it cannot possibly be unchanged across a night:
Five out of five: exactly the pre-sleep value, and 2.0 to 5.5 K away from reality. Inside temperature behaves the same (four exact, one within 0.1 K), as do
The car was plugged in and drawing current while the snapshot said it wasn't. What this means for the contractTaken together, the useful statement doesn't require knowing which clock is off. On a cold start, I'd flag that as something worth looking at critically before ordering or backfill gets keyed on it, rather than draw the conclusion myself — the measurement establishes the discrepancy, not what should follow from it. The two findings also compound in an unhelpful direction. The stamp isn't merely old, which would at least make the staleness self-evident; it postdates arrival, so a consumer trusting This is also the case where "absent is not null" doesn't help. Nothing is absent. All 57 fields are present, complete and plausible — and wrong. Concretely for the backfill framing from your 4 Aug reply: in the 42-hour case, a backfill keyed on If it's useful for the contract, the shape that would fix this is a source declaring whether its initial-state event is a fresh read or a cached last-known-value, rather than consumers having to infer it from a temperature that didn't change overnight. All of the above is measured at the receiver's own 🤖 Measured and drafted with Claude Code (Opus 5) |
|
Follow-up on the two open points from your 7 Aug comment — one with a result that changes what we told you on 4 Aug, one where we have to report a dead end. A. Staged outages, now without a clean closeYour caveat was right, and it turned out to be the whole story rather than a footnote: Setup. Vehicle kept awake with Sentry Mode (parked, at home) so it streams continuously — measured idle cadence 21 records/min over the five minutes before each run. The unclean outage is staged by silently dropping the vehicle's packets at the bridge on the hypervisor, both directions: No FIN, no RST reaches the vehicle — it just stops hearing back. One methodological note for anyone reproducing this: disabling the WAN port-forward is not enough. The established flow keeps being translated from the conntrack table and the vehicle streamed on undisturbed; the drop has to happen on the packet path. Results — four staged outages, the 4 Aug run included for comparison:
The 89 s boundary reproduced to the second across both 180 s runs. The 60 s run ended before it, which is why nothing at all came back. The 20 min clean run came back complete — earliest replayed event at T0 + 0 s, latest at T1 − 0 s, largest gap inside the replay 10 s (i.e. the normal cadence), delivered 7 s after the receiver returned. What this means. The vehicle hands data to a socket it believes is alive and considers it delivered. Nothing is buffered during that time. Only when its TCP gives up — consistently ~89 s here — does application-level buffering start, and from that point the replay is complete and in order. A clean close skips that phase entirely, which is why the 4 Aug run looked perfect. So the limit is not depth. The buffer survived 20 minutes without losing a single record; its upper bound is still unknown. What costs data is the start: a silent disconnect loses the first ~90 seconds, irrecoverably, no matter how long the outage lasts. This corrects our own statement from 4 Aug ("Fleet Telemetry buffers and replays without loss"). That holds for a clean close only. For the contract the consequence is sharper than the original point: a source's recovery capability is not one property but two — does it replay, and from when. A source that replays "from when it noticed" silently drops the beginning of every gap, and the consumer sees nothing unusual: correct For completeness, receiver-side detection lagged even further: Limits of this. One vehicle, one network path, one receiver. The ~89 s is presumably the vehicle's TCP timeout and will depend on the environment, so treat it as "about a minute and a half", not as a constant. The 60 s run only establishes that the threshold is above 60 s. Expected record counts are extrapolated from the pre-outage cadence, so the time boundaries are the solid part, not the ratios. Anything else worth staging? The setup now reproduces on demand: the vehicle can be held awake with Sentry, the connection can be cut either cleanly or silently, for any duration, and both clocks are recorded on every record. If there is a question in the contract that a single-car setup can actually settle, I would rather measure it than guess — tell me what would be useful and I will run it. B.
|
| connections in the window | 40 |
| median connection | 11 min |
| longest uninterrupted connection | 5.11 h |
| connected in total | 18.5 h of 137.2 h (13.5 %) |
A 24 h timer cannot elapse while the connection stands. The vehicle sleeps far too aggressively; even a full driving day (nine drives, 8.2 h awake) produced no connection longer than 5.11 h. So the 21,974 false are not evidence against your reading — the flag simply never had the opportunity to fire.
No field shows a send interval in the 23–25 h band either. The closest candidate is a 42.7 h gap after which 56 slow fields (Version, VehicleName, TPMS, Odometer, …) were sent again — but 87 s after the vehicle woke up, not at the 24 h mark. That is indistinguishable from the wake-up snapshot documented in the clock table from 8 Aug; both explanations fit and the data cannot separate them.
And we see no way to fix that from here. It would need a vehicle that stays connected for more than 24 h without sleeping — and, if the timer turns out to be per-connection, without a single reconnect in between. Neither is producible on a car in normal use. If this matters for the contract, the people who can answer it are the proxy operators seeing many vehicles and long-lived sessions (@jlestel, @Bre77), not a single-car setup.
If the flag is meant to carry weight in the contract, that in itself is worth stating: its semantics could not be established by a normal installation over a week of continuous capture.
🤖 Drafted with Claude Code (Opus 5) — sponsored by Claude for Open Source
|
I checked this against our fleet-wide data. The short answer is: yes, Across the retained fleet-wide window (roughly 20–24 days across the two observation cohorts), I found:
One important denominator caveat: this is a purpose-built sampling corpus. Resend-flagged records are selected unconditionally, while ordinary deltas are rate-limited to at most one sample per field per five minutes. So 81.4% is not an estimate of the fraction of all telemetry messages that are resends. The reliable conclusion is simply that the flag fires extensively across vehicles and fields. Because one flagged payload produces many near-simultaneous field records, I coalesced consecutive flagged writes for the same vehicle within two seconds into approximate resend bursts. That produced 555,091 bursts. Median spacing was 373 seconds in one cohort and 414 seconds in the other—far shorter than 24 hours, but this is likely due to vehicle configuration set to resend after 300s. There is also a visible near-24-hour component: 3,325 inter-burst intervals fell between 23 and 25 hours, and the two cohort p99s were 86,352 and 87,136 seconds. But those near-daily intervals were only 0.60% of measurable intervals. My read is: the default 86,400-second resend interval plausibly contributes to the distribution, but when configured as such, I cannot honestly answer per-connection timer versus absolute timer from my current data. On connection duration, the fleet does cover the regime your single-car run could not. Over the 31-day receiver-log window I queried, there were 1,246,819 completed sessions. Of those, 395,606 reported zero seconds; among the 851,213 positive-duration sessions the distribution was:
The long tail was:
So long-lived sessions definitely exist at fleet scale, yet For the silent-disconnect result, our receiver-side evidence is weaker than yours. We timestamp when the receiver detects a close, not when the path first became silent, so we cannot independently measure the initial ~89 seconds of loss. In a bounded set of unambiguous detected-close→next-connect pairs, most reconnects happened within five seconds, while abnormal closes had a several-minute tail. That is compatible with delayed failure/recovery behavior, but it is not a measurement of the same interval and I would not present it as independent confirmation of 89 seconds. So my numbers-first conclusion is:
|
|
On the ~89 s figure: that was stated more firmly than I intended it. It came from two runs that happened to agree exactly, and I wrote it as if it were a constant. I ran more of them, and the number is softer than my wording suggested — details below. I also have to correct myself on a second point from my previous comment, and it is the more interesting one. I wrote that after a 42.7 h gap the vehicle sent 56 slow-moving fields 87 s after waking, and that I could not tell whether that was a snapshot or a periodic resend. It is a snapshot, it is not tied to long gaps at all, and my reason for thinking otherwise was a bug in my own evaluation. Section 4 has it. @Bre77 — the volume of Up front, so it is not buried: I did not get a single record with Here is the setup and everything I measured, for completeness. Setup
How the gaps were staged. Four tools, all reproducible on demand:
1. Periodic resendAfter the push, all six fields send on an exact 300 s cadence with byte-identical values:
Two details that matter if you are counting field records rather than payloads: the six resends arrive bundled in a single record, not as six, and the moment the config was applied is not in the table above — that one is a full-field record (§4), not a periodic resend. 2. Timer anchoring
3. Clean disconnect spanning a due timeReceiver stopped 07:00:00Z, restarted 07:03:01Z. A resend was due at 07:01:00Z, inside the gap. The phase continued unchanged; 300 s after reconnect would have been 07:08:01Z. 4. Full-field records, and what happens across sleepThis is where my previous comment was wrong, so I will give the evidence rather than the conclusion. The vehicle sends single records that carry every field it ever sends — 57 of them — in one payload. Not the slow fields, not a subset. There are 59 of these in seven days of capture. What I can state cleanly about when they appear: every gap in the stream of 10 minutes or more is followed by one — nineteen out of nineteen, whatever the gap length. The converse does not hold, and I want to be explicit about it rather than imply a rule I cannot support: 27 of the 59 follow no gap at all, and I could not tie them to socket reconnects either (32 of 59 have a reconnect within ±2 min, 46 of 63 reconnects have a full-field record within ±2 min). So a gap reliably produces one; a full-field record does not imply a gap. What triggers the other 27 is open.
The 42.47 h / +87 s row is the one I reported last time as "56 slow fields 87 s after waking". It is this same record, and the delay does not track gap length either — the largest delay of all nineteen, +196 s, sits on the shortest gap but one. What the record contains is the pre-gap state, not a fresh reading. For the 8.58 h sleep on 08-10, 52 of its 57 values are byte-identical to the last value seen before the vehicle went to sleep. The five that differ are
Those values are 8.6 h old and demonstrably wrong at the moment they arrive — and
One further consequence, and it is the reason the correction matters for this thread: the full-field record counts as a send for every field, so it re-anchors all resend timers simultaneously. Nothing is buffered across sleep, unlike the clean disconnect in §3: on the 13 min sleep three resends fell due inside it and none was ever delivered, and across the 8.58 h sleep not a single record carries a 5. State change lost in a silent gapPacket-level drop 08:14:11Z–08:17:31Z. A real state change was triggered 17 s in, inside the window that gets lost. No 6. Silent-gap loss windowNine runs, same rig, packet drop rather than socket close:
The window does not scale with outage duration. It also is not the sharp constant my earlier wording implied: 73–103 s, median ~88. Total loss below ~90 s is the same effect — the outage ends before the vehicle notices. IncidentalAt the moment the config was applied, the vehicle emitted an
Open to suggestionsIf anyone has ideas for further scenarios worth staging, or sees a flaw in how these were set up, I would be glad to hear it. Everything in the setup section reproduces on demand, so additional cases are cheap to run. 🤖 Drafted with Claude Code (Opus 5) |
Self-hosted Fleet Telemetry reportFollowing the three requests in the roadmap: 1. Source in use todayI run a self-hosted Fleet Telemetry receiver for one personal vehicle: TeslaMate uses 2. Fields delivered and intervalsAfter a receiver restart, the vehicle reconnected and Fleet Telemetry logged a 644-record connection. The bridge received MQTT messages and TeslaMate marked the car real online. I then ran a normal ~15-minute drive. TeslaMate created a complete drive record from that stream. The Fleet Telemetry fields observed on the MQTT metric path were:
No other 3. What breaks in the current detourThe compatibility bridge makes the existing setup work, but it can only emit TeslaMate historical streaming fields. That matches the roadmap description: Fleet Telemetry fields outside the old owner-API-shaped stream cannot surface in TeslaMate, and TeslaMate still uses its polling loop alongside the push stream. For this setup, the requested native Fleet Telemetry source and event contract would remove the translation layer and let TeslaMate reason directly about partial Fleet Telemetry events. I can provide another controlled charge sample if there is a preferred field list and interval-report format. |
Uh oh!
There was an error while loading. Please reload this page.
What this is
This documents the direction @brianmay and I are taking the TeslaMate core, and collects input on the open design questions. It is documentation and brainstorming — not a vote, and not a call for contributions.
Two things came up in a closed PR (#5557) and in the older go-forward thread (#3416), and both belong somewhere visible:
We're implementing both. What is genuinely open — and what this thread is for — clusters in three areas: the event contract, the fidelity gap between Fleet Telemetry and old streaming, and what happens to the web UI. The open questions are marked as such below; the rest is decided and documented here so it can be reasoned about.
Ownership — please read before replying
Review capacity is the bottleneck in this project. A half-agreed contract implemented twice consumes it faster than anything else. That is why this section exists, and why it is strict.
This work touches the heart of TeslaMate: the state machine that decides what a drive is, what a charge is, and what ends up in your database. It cannot be assembled from independent pull requests.
If you want to help concretely:
Data sources: modular sources, one core pipeline
To be precise about where we stand today: TeslaMate already works against the Fleet API and Fleet Telemetry — but only by way of a detour. It speaks exactly one wire protocol, the owner-API shape, and alternative backends are reached by pointing
TESLA_API_HOST,TOKENandTESLA_WSS_HOSTat a third party (MyTeslaMate, Teslemetry) or a self-hosted proxy that translates them back into that shape.That detour works, and it will keep working. But it has a price:
vehicle_dataeven when the source underneath is push-only.The target shape, which @brianmay sketched in #5557: a core process receives a stream of events, splits them into drives, charges and so on, and persists them. Which source runs is a matter of configuration:
To set expectations: sources are an internal structuring device, not a plugin API. Which sources exist, and what the contract between them and the core looks like, is a maintainer decision and will stay one. We're documenting the contract so the design can be reasoned about, not so it can be extended from outside.
Today acquisition and interpretation are entangled in
Vehicle, which is why "streaming only" and "additional API" are hard rather than merely new. The polling loop is not a property of the owner API — it's a property of our architecture.The deliverable of this step is the event contract: what an event is, which guarantees a source must provide, and what the core may assume. The refactor is just how we get there.
Open — input wanted
Language: Rust for the core
The core moves to Rust. This decision is made; this thread is not the place to relitigate the language choice, and replies arguing Elixir vs. Rust in the abstract will not move it. What is genuinely open is listed below — most importantly the web UI question.
The reasons, in the order that matters to us:
Elixir/OTP has served this project well, and its supervision model is a good fit for "many vehicles, many long-lived processes". This is not a complaint about Elixir. It's a decision about where the core is easiest to keep correct over the next years.
What it costs
The Phoenix LiveView web UI is the largest single item. Settings, geofences, import and sign-in have no drop-in Rust equivalent. The realistic options are: keep the UI as a separate Elixir service talking to the same database, rewrite it against an HTTP API exposed by the Rust core, or replace it with something else entirely. This is open, and it's the question we'd most like input on.
Beyond that: migration ownership once Ecto is no longer the writer, and CI build times.
What stays stable for you
For everyone running TeslaMate, these hold across all of it:
What is not a stable interface: the database schema
The PostgreSQL schema is internal, and it is worth being blunt about this. It has never been a documented interface, and this work will not make it one. A restructuring of this size is exactly when the freedom to reshape storage is needed, and we intend to use it: expect substantial schema changes, not cosmetic ones.
Concretely: if you read the tables directly — custom Grafana panels, scripts, third-party or commercial tools built on today's schema — expect your integration to break during this work. Not might: will. There is no compatibility promise, no deprecation window, no advance notice beyond the migrations themselves, and we will not slow down, stage, or route the redesign around anything that depends on the current table shapes. Issues asking us to restore a schema detail or to coordinate changes with external tools will be closed. Your data is safe — migrations carry it forward — but every assumption about how it is stored is up for renegotiation, and nobody outside this repository is part of that negotiation. The supported surfaces are the dashboards we ship and MQTT. Everything else is and always was at your own risk.
If something we haven't listed here would break your setup, say so now, not after the migration.
Sequencing
The safety net comes before the surgery, and the architecture change stays decoupled from the language change — a rewrite that also changes the architecture has no reference implementation to test against.
Phase 1 — a characterization suite at the outer boundary. Before any code moves, the current behaviour gets pinned down: a recorded sequence of Tesla API payloads goes in, and the resulting persisted data and MQTT messages are asserted, with the vehicle state machine taken to ~100% coverage — the point of the number is that the suite must reliably catch behavioural deviations, not that a metric looks good. Crucially, the cases are stored as data rather than as code, and nothing in them refers to an internal function.
That boundary already exists —
TeslaMate.VehicleCase.start_vehicle/3drives exactly such a sequence through the API mock today. What's missing is coverage, the move from Elixir literals to data fixtures, and assertions against real persisted rows and messages instead of mock call assertions.This suite is what makes the following phases safe, and it stays valid across both of them: whatever happens inside, identical API payloads must keep producing identical results.
One caveat we'll handle explicitly: pinning current behaviour also pins current bugs. Cases we believe are wrong get marked as such and fixed in their own commits — never silently, in the middle of a refactor.
Phase 2 — source abstraction in Elixir. Introduce the source/event boundary and make the event contract explicit, incrementally, with the Phase 1 suite green at every step. Delivers native streaming-only support without a rewrite. Event-level fixtures — the ones that can only exist once the contract does — are written here, at the new boundary, alongside the outer suite.
Phase 3 — Rust. Port module by module behind the same contract (strangler pattern). Both fixture sets are the gate: the same files drive both cores and the outputs are diffed. Anything the Rust core cannot reproduce does not replace the Elixir path. How the two runtimes coexist during the port — NIFs, a separate process, or running both cores side by side and diffing offline — is deliberately left open here and will be decided in the Phase 3 issues; the single-binary deployment benefit arrives at the end of this phase, not during it.
Users will not be moved onto the Rust core silently. How the switch is offered — an opt-in image, a beta tag, or both cores running against the same data and diffed before anything ships — will be settled there too, and announced before it happens.
Each phase stands on its own. Phase 1 is worth having whatever happens afterwards, Phase 2 removes the bottleneck regardless of the language, and Phase 3 is gated by both.
Where your input goes, and how this gets tracked
The questions marked Open above — the event contract, the fidelity gap between Fleet Telemetry and old streaming, and what happens to the web UI — are the ones where a good answer changes what we build. Corrections to the "what stays stable" list are equally welcome.
There are no dates attached to any of this, and regular development continues throughout: bug fixes, dashboards and features ship as usual, and none of this work blocks a release. Once the design questions have settled, we'll cut the work into issues that link back to this thread — those issues carry scope and acceptance criteria, not deadlines. Hard commitments made in this document — dashboards shipping together with schema changes, MQTT compatibility asserted by the suite — become acceptance criteria in those issues, so they can't silently slip. The issues are the tracking surface; the reasoning lives here. This thread stays open for the design discussion and will not be used as a task list.
What this thread is not: a place to claim implementation work. See "Ownership" above.
cc for visibility: @adriankumpf, @cwanja, @DrMichael, @Dulanic, @swiffer, @tobiasehlert — no action needed, this is a heads-up so you read it here first.
🤖 Drafted with Claude Code (Opus 5 high and Fable 5 high) — sponsored by Claude for Open Source
All reactions