Skip to content

Commit 4ec53b2

Browse files
committed
Fix normalize_to_records function which did not traverse into nested elements within a time_series entry
Until now I did not discover nested elements within a time_series entry but the ProductionAndGenerationUnits endpoint returns those. This change may increase the number of rows and columns for existing queries but its absolutely necessary for not silently dropping data. Endpoints like Load and EnergyPrices are not affected as they do not have nested structures in the `time_series` objects.
1 parent 8ae38bf commit 4ec53b2

1 file changed

Lines changed: 12 additions & 7 deletions

File tree

src/entsoe/utils/records.py

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -69,14 +69,18 @@ def _filter_ignored_fields(record: Dict[str, Any]) -> Dict[str, Any]:
6969
return {k: v for k, v in record.items() if k not in ignore_fields}
7070

7171
if isinstance(data, dict):
72-
items = {}
72+
items: Dict[str, Any] = {}
73+
expansions: List[List[Dict[str, Any]]] = []
7374
for k, v in data.items():
7475
new_key = f"{parent_key}{sep}{k}" if parent_key else k
7576
if isinstance(v, dict):
7677
sub_records = normalize_to_records(
7778
v, new_key, sep=sep, ignore_fields=ignore_fields
7879
)
79-
items.update(sub_records[0]) # merge dict
80+
if len(sub_records) == 1:
81+
items.update(sub_records[0])
82+
elif len(sub_records) > 1:
83+
expansions.append(sub_records)
8084
elif isinstance(v, list):
8185
# Expand list elements into multiple records
8286
list_records = []
@@ -88,14 +92,15 @@ def _filter_ignored_fields(record: Dict[str, Any]) -> Dict[str, Any]:
8892
list_records.extend(sub_records)
8993
else:
9094
list_records.append({new_key: elem})
91-
# Cross join if multiple records, else just keep one
9295
if list_records:
93-
return [
94-
_filter_ignored_fields(dict(items, **lr)) for lr in list_records
95-
]
96+
expansions.append(list_records)
9697
else:
9798
items[new_key] = v
98-
return [_filter_ignored_fields(items)]
99+
# Cross-join base items with all collected expansions
100+
result: List[Dict[str, Any]] = [items]
101+
for expansion in expansions:
102+
result = [dict(r, **e) for r in result for e in expansion]
103+
return [_filter_ignored_fields(r) for r in result]
99104
elif isinstance(data, list):
100105
records = []
101106
for elem in data:

0 commit comments

Comments
 (0)