Skip to content

API: publish forecast as arrays with unix timestamps (BC) - #32391

Merged
andig merged 5 commits into
masterfrom
fix/forecast-timestamps-master
Aug 6, 2026
Merged

API: publish forecast as arrays with unix timestamps (BC)#32391
andig merged 5 commits into
masterfrom
fix/forecast-timestamps-master

Conversation

@andig

@andig andig commented Aug 1, 2026

Copy link
Copy Markdown
Member

replaces #31765

The forecast is the largest payload evcc publishes and RFC3339 timestamps are the bulk of it. Publishing slots as [start, end, value] and solar entries as [ts, val] with unix seconds cuts it by two thirds, measured on a live instance with 2784 entries: 209 609 → 72 745 bytes.

Loadpoint plan/plan-preview and /api/tariff responses are untouched, they still use api.Rate with RFC3339 strings.

  • forecast slots are published as [start, end, value], solar timeseries entries as [ts, val], timestamps in unix seconds
  • the UI expands the arrays once in the store, so components keep reading { start, end, value } and { ts, val }, now in unix milliseconds instead of RFC3339 strings
  • no encoding/json/v2, so this builds on the current Go baseline and does not need the 1.27 upgrade that core: publish forecast start/end/ts as unix timestamps (BC) #31765 pulled in

Breaking Change

🤖 Generated with Claude Code

@andig andig added the infrastructure Basic functionality label Aug 1, 2026
@github-actions github-actions Bot added tariffs Specific tariff support enhancement New feature or request needs documentation Triggers issue creation in evcc-io/docs labels Aug 1, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="assets/js/utils/forecast.ts" line_range="38" />
<code_context>
+// milliseconds so components can keep using named fields and `new Date(...)`.
+// State arrives sharded, one key per forecast field, plus the whole object on
+// initial load.
+export function expandForecast(key: string, value: any): any {
+  if (!key.startsWith("forecast") || !value) return value;
+
</code_context>
<issue_to_address>
**issue (complexity):** Consider replacing the generic string-key-based `expandForecast` function with explicit, typed forecast field converters that the store layer calls directly to avoid key parsing and recursion.

You can keep the sharded update behavior while reducing the key-string + recursion complexity by making the transformation API explicit and typed, then using it from your store layer.

### 1. Replace string-based dispatch with typed helpers

Instead of `expandForecast(key: string, value: any)`, you can introduce explicit converters for the known wire types, and let the store call the appropriate one based on its own key logic:

```ts
// Explicit converters, no key parsing
export function expandFullForecast(forecast: {
  co2: WireForecastSlot[];
  feedin: WireForecastSlot[];
  grid: WireForecastSlot[];
  planner: WireForecastSlot[];
  temperature: WireForecastSlot[];
  solar: WireSolarDetails;
  // ...add other fields if needed
}) {
  return {
    co2: expandSlots(forecast.co2),
    feedin: expandSlots(forecast.feedin),
    grid: expandSlots(forecast.grid),
    planner: expandSlots(forecast.planner),
    temperature: expandSlots(forecast.temperature),
    solar: expandSolar(forecast.solar),
  };
}

export type ForecastField =
  | "co2"
  | "feedin"
  | "grid"
  | "planner"
  | "temperature"
  | "solar";

export function expandForecastField(
  field: ForecastField,
  value: WireForecastSlot[] | WireSolarDetails
): ForecastSlot[] | SolarDetails {
  switch (field) {
    case "solar":
      return expandSolar(value as WireSolarDetails);
    default:
      return expandSlots(value as WireForecastSlot[]);
  }
}
```

Then the store can keep using its `forecast.*` keys but route them through the typed functions instead of having `expandForecast` depend on string conventions:

```ts
// Example usage in the store layer
if (key === "forecast" && value) {
  nextState.forecast = expandFullForecast(value);
} else if (key.startsWith("forecast.")) {
  const [, field] = key.split(".") as [string, ForecastField];
  (nextState.forecast as any)[field] = expandForecastField(field, value);
}
```

This preserves the sharded updates and initial full object handling, but makes the transformation code independent of key naming and removes the recursion on `expandForecast`.

### 2. Make solar wire/UI types explicit and local

You already have `WireSolarDetails` as a widened type. Moving it near the other wire types and delegating the conversion to a dedicated function improves clarity and removes the need for `any`/`Omit` in the main transformer:

```ts
// types.ts (or wherever the wire types live)
export type WireSolarDetails = {
  // same shape as SolarDetails, but with optional timeseries in wire form:
  timeseries?: WireTimeseriesEntry[];
} & Omit<SolarDetails, "timeseries">;

// converter module
export function toSolarDetails(wire: WireSolarDetails): SolarDetails {
  const { timeseries, ...rest } = wire;
  return {
    ...rest,
    timeseries: timeseries?.map(([ts, val]) => ({ ts: ts * 1000, val })),
  };
}
```

Then your forecast expansion code becomes simpler and more self-contained:

```ts
import { toSolarDetails } from "./toSolarDetails";

function expandSolar(solar: WireSolarDetails): SolarDetails {
  return toSolarDetails(solar);
}
```

These changes keep all current behavior but reduce cognitive load by:
- Removing generic `key: string, value: any` routing from the transformation layer.
- Replacing `SLOT_KEYS` + `key.split(".")` indirection with typed, explicit APIs.
- Encapsulating solar wire-to-UI conversion in a dedicated helper.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread assets/js/utils/forecast.ts Outdated
@andig

andig commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Addressed, though not the way suggested. The recursion was the real complexity and it was unreachable: publishTariffs always wraps the forecast in util.NewSharder, and server/socket.go emits sharded values as forecast.<field>, never a bare forecast key. The store is fed only from the websocket, so the whole-object branch could not run. It is gone, leaving one string split and two lookups.

The suggested split was not adopted. Routing in the store means reaching into state.forecast directly, which bypasses setProperty and needs an as any, and expandForecastField still dispatches on the same field name, just one layer further out.

Added tests for slot shards, the solar timeseries and pass-through of unrelated keys.

🤖 Generated with Claude Code

@TheNinth7

Copy link
Copy Markdown

The change has now been implemented in the Garmin app's v2.0.6-beta4, which supports both the old and the new data format. Testing of the new format will have to wait until the corresponding evcc release, however, as the transformation is performed server-side using JQ.

METIQ-Solutions/evcc-garmin#183

@marq24

marq24 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

I have to admit, I AM LOST in the information generated by Claude (and probably that's already the root of my problem)... TooMuch inconsistent information (at least when I try to parse it)...

is it now just an int[] - or is this just the "old" change where the timestamp format has been replaced (which is already supported since a couple of weeks)...

@andig

andig commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Start/end sind int, value ist float, also wie immer.

@marq24

marq24 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

so 'original'

{
  "start": "2026-07-13T21:00:00+02:00"
  "end": "2026-07-13T21:15:00+02:00"
  "value": 222
}

intermed version #31765 (never released)

{
  "start": 1783969200 
  "end": 178396xx00
  "value": 222
}

now/new

[1783969200, 178396xx00, 222]

?!

marq24 added a commit to marq24/ha-evcc that referenced this pull request Aug 1, 2026
@marq24

marq24 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

implemented change in HA Integration https://github.com/marq24/ha-evcc/releases/tag/2026.8.0

@TheNinth7

Copy link
Copy Markdown

@andig There is also another integration to consider: the openHAB binding by @marcelGoerentz.

@naltatis naltatis changed the title core: publish forecast as arrays with unix timestamps (BC) API: publish forecast as arrays with unix timestamps (BC) Aug 3, 2026
@andig

andig commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

ping @marcelGoerentz regarding openHAB

@marcelGoerentz

Copy link
Copy Markdown
Contributor

Thanks, I will need to fix it in openHAB. But that should be a no brainer.

@andig
andig force-pushed the fix/forecast-timestamps-master branch from eada57f to deebae2 Compare August 4, 2026 15:31
@andig

andig commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@naltatis could you check if 9e2da18 is correct?

@andig

andig commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

@naltatis is this one good to merge?

@naltatis

naltatis commented Aug 6, 2026

Copy link
Copy Markdown
Member

I've update the forecast data handling in UI to match the pattern we already use for loadpoints: state mirrors api structure, derived (computed) Ui__ types where applicable.

@naltatis is this one good to merge?

Yes, fine for me.

@andig
andig merged commit 45e781d into master Aug 6, 2026
10 checks passed
@andig
andig deleted the fix/forecast-timestamps-master branch August 6, 2026 09:49
ohAnd added a commit to ohAnd/EOS_connect that referenced this pull request Aug 16, 2026
see 45e781d (evcc-io/evcc@45e781d) API: publish forecast as arrays with unix timestamps (BC) (#32391 - evcc-io/evcc#32391)
ohAnd added a commit to ohAnd/EOS_connect that referenced this pull request Aug 22, 2026
* fix: support compact EVCC forecasts
see 45e781d (evcc-io/evcc@45e781d) API: publish forecast as arrays with unix timestamps (BC) (#32391 - evcc-io/evcc#32391)

* feat: add PV forecast auto-scaling with EVCC compatibility fix

- New PvAutoscaler collects hourly real yield vs. live EOS forecast
  and computes per-timeframe scale factors (kWh-based storage)
- New PvYieldStore persists history with automatic Wh->kWh migration
- PV interface hooks: startup readiness guard prevents incomplete
  forecast writes; supports EVCC's compact [unix_ts, value] format
  (EVCC PR #32391) alongside the legacy {ts, val} format
- scripts/insert_data_to_db.py: manual test-data seeding helper
- Docs updated (config guide, advanced API, developer guide)

* feat: update PV autoscaler configuration keys and enhance logging for collection attempts

* feat: enhance get_summarized_pv_forecast with scaling option and update tests for autoscaler behavior

* feat: implement raw PV forecast retrieval for accurate scaling factor calculations

* feat: update PV forecast handling with raw and scaled arrays for improved accuracy in daily average calculations

* feat: enhance PV autoscaler and interface with raw forecast handling and improved DST support

* fix: correct time boundary calculation in should_collect method for accurate hour counting

* fix: correct missed hour calculation in collect_if_needed test for accurate row distribution

* feat: enhance today's PV forecast calculation with original and scaled arrays for improved accuracy

* fix: repair and relocate PV yield store

Retention compared ISO-8601 timestamps ("...T08:00:00+00:00") against
SQLite's datetime() output ("... 08:00:00") as strings. 'T' sorts above
the space, so rows on the cutoff date always looked newer than the
cutoff and were never purged, silently extending retention by a day.

The upsert keyed on (local_date, local_hour). On the autumn DST
transition local 02:00 occurs twice, so the second reading overwrote the
first and an hour of measured yield was lost. It now keys on the UTC
timestamp, which identifies the hour uniquely, backed by a matching
unique index since the select-then-update is not atomic.

Rows written before the local_* columns existed were never back-filled,
so anything grouping by local_date dropped them into a phantom bucket.

The store holds measurements, not configuration, so it no longer lives
in the config_web package: the interface layer can now reach it without
importing the config web application.

* fix: correct PV autoscaler collection and scaling

Fetched every provider twice per cycle, doubling upstream traffic
whether or not autoscaling was enabled and overrunning Solcast's fixed
request budget. A failure on the second call also served the scaled
array back as raw, feeding the correction its own output. Now fetches
once and scales locally.

The factor averaged days whose forecast was never recorded - which is
what gap reconstruction writes - adding yield to the numerator with
nothing in the denominator. Three accurate days plus two reconstructed
gave 1.67x where the truth was 1.0, inflating the forecast exactly when
collection was failing. Both sides must now be present.

Source "default" never collected at all: the guard demanded the 48h
horizon providers publish, but the built-in curve is one day.

Restores EVCC's forecast.solar.scale, dropped earlier in this branch,
which left use_real_data_correction with no reader. It applies per
source before the aggregate boundary, so the autoscaler learns only the
residual bias and the two compose.

Also: seed factors at construction so a restart is not unscaled for an
hour; step back through the timezone so DST keeps the right hour; cap
gap reconstruction; record failures so the UI can show them; share the
HA/openHAB reader with the battery interface.

* fix: apply PV autoscaler settings without a restart

Toggling pv_autoscaling.enabled set the flag and nothing else. The
collection thread only ever started at boot, so the API reported the
change as applied, no restart banner appeared, and scaling was applied
from factors that nothing was updating. It now starts and stops the
collector.

sensor_entity_id had neither hot_reload nor restart_required, so it fell
through the gap in the API's if/elif: the user got a green save and the
running autoscaler kept polling the old entity forever. The connection
fields are hot-reloadable too, since the next hourly poll picks them up.

data_source.type was copied straight into pv_autoscaling.src, but it
also allows "default" - its own schema default - which the autoscaler
cannot read a counter from and which pv_autoscaling.src forbids. That
killed every poll behind a once-an-hour warning. Unsupported types are
now left alone and warned about at startup.

Adds pv_autoscaling.ssl_ignore: it was only injected from the central
data source, so manual-mode users behind a self-signed certificate had
no way to set it.

Coercion went through bool(), where the string "false" is true.

The schema export no longer loads schema.py by file path; the plain
import works and the Flask dependency it avoided predates this branch.

* fix: correct PV autoscaling overlay and show failures

The overlay inferred slot width from the forecast array's length, so an
hourly install read its 48-value two-day horizon as one 48-slot day:
Today showed the two-day total and Tomorrow was always 0.00 kWh, with
factors attributed to the wrong timeframes. Width now comes from the
resolution the backend reports, which is the only thing that separates
48 hourly slots from half a 15-minute day.

A mistyped sensor, an expired token or an unreachable host rendered
exactly like a fresh install - green tick, no data, and a friendly
"initializing" banner that never went away. Collection failures are now
reported and shown, naming the entity that failed.

The panel was reachable only through the two yield badges, whose
placeholder text this branch had removed, leaving them zero-width and
unclickable until the first optimization cycle filled them in - the
window in which a user most wants to check whether collection started.
Restores the placeholder and adds a labelled icon.

The handler reimplemented the per-day aggregation inline and grouped it
on a different key than the factor computation, so the page could
disagree with what the optimizer received. It now calls the autoscaler,
and every field has a default so one failing section degrades instead of
returning 500 on an unbound local.

Also reports used_time_frame_base, and stops treating a zero forecast
slot as 1 Wh.

* test: cover autoscaler behaviour the suite asserted away

The DST tests built localized datetimes and never injected them, so they
exercised whatever wall-clock time the run happened to start at. Their
one assertion, that the UTC offset was 60 or 120 minutes, is true for
Europe/Berlin at every instant. Because the seeded row sat months in the
past, they also silently drove thousands of gap-reconstruction inserts
without asserting anything about them. They now pin local time and use a
store fake that reproduces the real upsert, so the fall-back hour loss
fails here rather than only in the database.

Three EVCC tests defined a mock whose apply_scaling contained the very
clamp being asserted, so they only proved the mock multiplied correctly.
The real min/max clamp and the near-zero-forecast guard had no test at
all. They now exercise the restored EVCC path and the autoscaler's own
clamp directly.

The collect tests called datetime.now() several times per test and
derived indices from each, so a run crossing an hour boundary failed.
They now share one pinned clock.

Adds coverage for the gaps behind the recent fixes: factor inflation
from missing forecasts, the 24-hour default curve, restart seeding,
config changes taking effect, and collection failures reaching status.

* docs: correct PV autoscaling documentation

Three statements did not match the code. The user guide described a
"PV Autoscaling card" on the dashboard that does not exist - it is a
panel behind the Statistics tile - and hardcoded a 7-day window that is
configurable from 1 to 14 days. The API reference documented
current_forecast_array, which the endpoint has never emitted; it returns
separate raw and scaled arrays.

Documents behaviour that was previously undescribed: how to tell a
working autoscaler from a stalled one, that factors are reloaded from
history at startup, that EVCC's own correction composes with the
autoscaler rather than competing with it, and that a single set of
factors applies to the summed forecast of every installation - so with
mixed sources one source's bias is spread across the others.

* fix: adjust import of PvYieldStore for script execution context

* fix: simplify error responses in update status and pv autoscaling status
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request infrastructure Basic functionality needs documentation Triggers issue creation in evcc-io/docs tariffs Specific tariff support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants