API: publish forecast as arrays with unix timestamps (BC) - #32391
Conversation
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="assets/js/utils/forecast.ts" line_range="38" />
<code_context>
+// milliseconds so components can keep using named fields and `new Date(...)`.
+// State arrives sharded, one key per forecast field, plus the whole object on
+// initial load.
+export function expandForecast(key: string, value: any): any {
+ if (!key.startsWith("forecast") || !value) return value;
+
</code_context>
<issue_to_address>
**issue (complexity):** Consider replacing the generic string-key-based `expandForecast` function with explicit, typed forecast field converters that the store layer calls directly to avoid key parsing and recursion.
You can keep the sharded update behavior while reducing the key-string + recursion complexity by making the transformation API explicit and typed, then using it from your store layer.
### 1. Replace string-based dispatch with typed helpers
Instead of `expandForecast(key: string, value: any)`, you can introduce explicit converters for the known wire types, and let the store call the appropriate one based on its own key logic:
```ts
// Explicit converters, no key parsing
export function expandFullForecast(forecast: {
co2: WireForecastSlot[];
feedin: WireForecastSlot[];
grid: WireForecastSlot[];
planner: WireForecastSlot[];
temperature: WireForecastSlot[];
solar: WireSolarDetails;
// ...add other fields if needed
}) {
return {
co2: expandSlots(forecast.co2),
feedin: expandSlots(forecast.feedin),
grid: expandSlots(forecast.grid),
planner: expandSlots(forecast.planner),
temperature: expandSlots(forecast.temperature),
solar: expandSolar(forecast.solar),
};
}
export type ForecastField =
| "co2"
| "feedin"
| "grid"
| "planner"
| "temperature"
| "solar";
export function expandForecastField(
field: ForecastField,
value: WireForecastSlot[] | WireSolarDetails
): ForecastSlot[] | SolarDetails {
switch (field) {
case "solar":
return expandSolar(value as WireSolarDetails);
default:
return expandSlots(value as WireForecastSlot[]);
}
}
```
Then the store can keep using its `forecast.*` keys but route them through the typed functions instead of having `expandForecast` depend on string conventions:
```ts
// Example usage in the store layer
if (key === "forecast" && value) {
nextState.forecast = expandFullForecast(value);
} else if (key.startsWith("forecast.")) {
const [, field] = key.split(".") as [string, ForecastField];
(nextState.forecast as any)[field] = expandForecastField(field, value);
}
```
This preserves the sharded updates and initial full object handling, but makes the transformation code independent of key naming and removes the recursion on `expandForecast`.
### 2. Make solar wire/UI types explicit and local
You already have `WireSolarDetails` as a widened type. Moving it near the other wire types and delegating the conversion to a dedicated function improves clarity and removes the need for `any`/`Omit` in the main transformer:
```ts
// types.ts (or wherever the wire types live)
export type WireSolarDetails = {
// same shape as SolarDetails, but with optional timeseries in wire form:
timeseries?: WireTimeseriesEntry[];
} & Omit<SolarDetails, "timeseries">;
// converter module
export function toSolarDetails(wire: WireSolarDetails): SolarDetails {
const { timeseries, ...rest } = wire;
return {
...rest,
timeseries: timeseries?.map(([ts, val]) => ({ ts: ts * 1000, val })),
};
}
```
Then your forecast expansion code becomes simpler and more self-contained:
```ts
import { toSolarDetails } from "./toSolarDetails";
function expandSolar(solar: WireSolarDetails): SolarDetails {
return toSolarDetails(solar);
}
```
These changes keep all current behavior but reduce cognitive load by:
- Removing generic `key: string, value: any` routing from the transformation layer.
- Replacing `SLOT_KEYS` + `key.split(".")` indirection with typed, explicit APIs.
- Encapsulating solar wire-to-UI conversion in a dedicated helper.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
Addressed, though not the way suggested. The recursion was the real complexity and it was unreachable: The suggested split was not adopted. Routing in the store means reaching into Added tests for slot shards, the solar timeseries and pass-through of unrelated keys. 🤖 Generated with Claude Code |
|
The change has now been implemented in the Garmin app's v2.0.6-beta4, which supports both the old and the new data format. Testing of the new format will have to wait until the corresponding evcc release, however, as the transformation is performed server-side using JQ. |
|
I have to admit, I AM LOST in the information generated by Claude (and probably that's already the root of my problem)... TooMuch inconsistent information (at least when I try to parse it)... is it now just an int[] - or is this just the "old" change where the timestamp format has been replaced (which is already supported since a couple of weeks)... |
|
Start/end sind int, value ist float, also wie immer. |
|
so 'original' intermed version #31765 (never released) now/new ?! |
|
implemented change in HA Integration https://github.com/marq24/ha-evcc/releases/tag/2026.8.0 |
|
@andig There is also another integration to consider: the openHAB binding by @marcelGoerentz. |
|
ping @marcelGoerentz regarding openHAB |
|
Thanks, I will need to fix it in openHAB. But that should be a no brainer. |
eada57f to
deebae2
Compare
|
@naltatis is this one good to merge? |
|
I've update the forecast data handling in UI to match the pattern we already use for loadpoints: state mirrors api structure, derived (computed)
Yes, fine for me. |
see 45e781d (evcc-io/evcc@45e781d) API: publish forecast as arrays with unix timestamps (BC) (#32391 - evcc-io/evcc#32391)
* fix: support compact EVCC forecasts see 45e781d (evcc-io/evcc@45e781d) API: publish forecast as arrays with unix timestamps (BC) (#32391 - evcc-io/evcc#32391) * feat: add PV forecast auto-scaling with EVCC compatibility fix - New PvAutoscaler collects hourly real yield vs. live EOS forecast and computes per-timeframe scale factors (kWh-based storage) - New PvYieldStore persists history with automatic Wh->kWh migration - PV interface hooks: startup readiness guard prevents incomplete forecast writes; supports EVCC's compact [unix_ts, value] format (EVCC PR #32391) alongside the legacy {ts, val} format - scripts/insert_data_to_db.py: manual test-data seeding helper - Docs updated (config guide, advanced API, developer guide) * feat: update PV autoscaler configuration keys and enhance logging for collection attempts * feat: enhance get_summarized_pv_forecast with scaling option and update tests for autoscaler behavior * feat: implement raw PV forecast retrieval for accurate scaling factor calculations * feat: update PV forecast handling with raw and scaled arrays for improved accuracy in daily average calculations * feat: enhance PV autoscaler and interface with raw forecast handling and improved DST support * fix: correct time boundary calculation in should_collect method for accurate hour counting * fix: correct missed hour calculation in collect_if_needed test for accurate row distribution * feat: enhance today's PV forecast calculation with original and scaled arrays for improved accuracy * fix: repair and relocate PV yield store Retention compared ISO-8601 timestamps ("...T08:00:00+00:00") against SQLite's datetime() output ("... 08:00:00") as strings. 'T' sorts above the space, so rows on the cutoff date always looked newer than the cutoff and were never purged, silently extending retention by a day. The upsert keyed on (local_date, local_hour). On the autumn DST transition local 02:00 occurs twice, so the second reading overwrote the first and an hour of measured yield was lost. It now keys on the UTC timestamp, which identifies the hour uniquely, backed by a matching unique index since the select-then-update is not atomic. Rows written before the local_* columns existed were never back-filled, so anything grouping by local_date dropped them into a phantom bucket. The store holds measurements, not configuration, so it no longer lives in the config_web package: the interface layer can now reach it without importing the config web application. * fix: correct PV autoscaler collection and scaling Fetched every provider twice per cycle, doubling upstream traffic whether or not autoscaling was enabled and overrunning Solcast's fixed request budget. A failure on the second call also served the scaled array back as raw, feeding the correction its own output. Now fetches once and scales locally. The factor averaged days whose forecast was never recorded - which is what gap reconstruction writes - adding yield to the numerator with nothing in the denominator. Three accurate days plus two reconstructed gave 1.67x where the truth was 1.0, inflating the forecast exactly when collection was failing. Both sides must now be present. Source "default" never collected at all: the guard demanded the 48h horizon providers publish, but the built-in curve is one day. Restores EVCC's forecast.solar.scale, dropped earlier in this branch, which left use_real_data_correction with no reader. It applies per source before the aggregate boundary, so the autoscaler learns only the residual bias and the two compose. Also: seed factors at construction so a restart is not unscaled for an hour; step back through the timezone so DST keeps the right hour; cap gap reconstruction; record failures so the UI can show them; share the HA/openHAB reader with the battery interface. * fix: apply PV autoscaler settings without a restart Toggling pv_autoscaling.enabled set the flag and nothing else. The collection thread only ever started at boot, so the API reported the change as applied, no restart banner appeared, and scaling was applied from factors that nothing was updating. It now starts and stops the collector. sensor_entity_id had neither hot_reload nor restart_required, so it fell through the gap in the API's if/elif: the user got a green save and the running autoscaler kept polling the old entity forever. The connection fields are hot-reloadable too, since the next hourly poll picks them up. data_source.type was copied straight into pv_autoscaling.src, but it also allows "default" - its own schema default - which the autoscaler cannot read a counter from and which pv_autoscaling.src forbids. That killed every poll behind a once-an-hour warning. Unsupported types are now left alone and warned about at startup. Adds pv_autoscaling.ssl_ignore: it was only injected from the central data source, so manual-mode users behind a self-signed certificate had no way to set it. Coercion went through bool(), where the string "false" is true. The schema export no longer loads schema.py by file path; the plain import works and the Flask dependency it avoided predates this branch. * fix: correct PV autoscaling overlay and show failures The overlay inferred slot width from the forecast array's length, so an hourly install read its 48-value two-day horizon as one 48-slot day: Today showed the two-day total and Tomorrow was always 0.00 kWh, with factors attributed to the wrong timeframes. Width now comes from the resolution the backend reports, which is the only thing that separates 48 hourly slots from half a 15-minute day. A mistyped sensor, an expired token or an unreachable host rendered exactly like a fresh install - green tick, no data, and a friendly "initializing" banner that never went away. Collection failures are now reported and shown, naming the entity that failed. The panel was reachable only through the two yield badges, whose placeholder text this branch had removed, leaving them zero-width and unclickable until the first optimization cycle filled them in - the window in which a user most wants to check whether collection started. Restores the placeholder and adds a labelled icon. The handler reimplemented the per-day aggregation inline and grouped it on a different key than the factor computation, so the page could disagree with what the optimizer received. It now calls the autoscaler, and every field has a default so one failing section degrades instead of returning 500 on an unbound local. Also reports used_time_frame_base, and stops treating a zero forecast slot as 1 Wh. * test: cover autoscaler behaviour the suite asserted away The DST tests built localized datetimes and never injected them, so they exercised whatever wall-clock time the run happened to start at. Their one assertion, that the UTC offset was 60 or 120 minutes, is true for Europe/Berlin at every instant. Because the seeded row sat months in the past, they also silently drove thousands of gap-reconstruction inserts without asserting anything about them. They now pin local time and use a store fake that reproduces the real upsert, so the fall-back hour loss fails here rather than only in the database. Three EVCC tests defined a mock whose apply_scaling contained the very clamp being asserted, so they only proved the mock multiplied correctly. The real min/max clamp and the near-zero-forecast guard had no test at all. They now exercise the restored EVCC path and the autoscaler's own clamp directly. The collect tests called datetime.now() several times per test and derived indices from each, so a run crossing an hour boundary failed. They now share one pinned clock. Adds coverage for the gaps behind the recent fixes: factor inflation from missing forecasts, the 24-hour default curve, restart seeding, config changes taking effect, and collection failures reaching status. * docs: correct PV autoscaling documentation Three statements did not match the code. The user guide described a "PV Autoscaling card" on the dashboard that does not exist - it is a panel behind the Statistics tile - and hardcoded a 7-day window that is configurable from 1 to 14 days. The API reference documented current_forecast_array, which the endpoint has never emitted; it returns separate raw and scaled arrays. Documents behaviour that was previously undescribed: how to tell a working autoscaler from a stalled one, that factors are reloaded from history at startup, that EVCC's own correction composes with the autoscaler rather than competing with it, and that a single set of factors applies to the summed forecast of every installation - so with mixed sources one source's bias is spread across the others. * fix: adjust import of PvYieldStore for script execution context * fix: simplify error responses in update status and pv autoscaling status
replaces #31765
The forecast is the largest payload evcc publishes and RFC3339 timestamps are the bulk of it. Publishing slots as
[start, end, value]and solar entries as[ts, val]with unix seconds cuts it by two thirds, measured on a live instance with 2784 entries: 209 609 → 72 745 bytes.Loadpoint plan/plan-preview and
/api/tariffresponses are untouched, they still useapi.Ratewith RFC3339 strings.[start, end, value], solar timeseries entries as[ts, val], timestamps in unix seconds{ start, end, value }and{ ts, val }, now in unix milliseconds instead of RFC3339 stringsencoding/json/v2, so this builds on the current Go baseline and does not need the 1.27 upgrade that core: publish forecast start/end/ts as unix timestamps (BC) #31765 pulled inBreaking Change
🤖 Generated with Claude Code