You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: examples/NREL_Data/README.rst
+173-4Lines changed: 173 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,7 +27,7 @@ Definitions
27
27
- ``hsds`` - The highly scalable data service (HSDS) that we recommend to access small chunks of very large cloud-hosted NREL datasets. See the `hsds <https://github.com/HDFGroup/hsds>`_ library for more details.
28
28
- ``meta`` - The ``dataset`` in an NREL h5 file that contains information about the spatial axis. This is typically a `pandas DataFrame <https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html>`_ with columns such as "latitude", "longitude", "state", etc... The DataFrame is typically converted to a records array for storage in an h5 ``dataset``. The length of the meta data should match the length of axis 1 of a 2D spatiotemporal ``dataset``.
29
29
- ``S3`` - Amazon Simple Storage Service (S3) is a basic cloud file storage system we use to store raw .h5 files in their full volume. Downloading files directly from S3 may not be the easiest way to access the data because each file tends to be multiple terabytes. Instead, you can stream small chunks of the files via HSDS.
30
-
- ``scale_factor`` - We frequently scale data by a multiplicative factor, round the data to integer precision, and store the data in integer arrays. The ``scale_factor`` is an attribute associated with the relevant h5 ``dataset`` that defines the multiplicative factor required to unscale the data from integer storage to the original physical units.
30
+
- ``scale_factor`` - We frequently scale data by a multiplicative factor, round the data to integer precision, and store the data in integer arrays. The ``scale_factor`` is an attribute associated with the relevant h5 ``dataset`` that defines the factor required to unscale the data from integer storage to the original physical units. The data should be divided by the ``scale_factor`` to scale back from integer to physical units.
31
31
- ``time_index`` - The ``dataset`` in an NREL h5 file that contains information about the temporal axis. This is typically a `pandas DatetimeIndex <https://pandas.pydata.org/docs/reference/api/pandas.DatetimeIndex.html>`_ that has been converted to a string array for storage in an h5 ``dataset``. The length of this ``dataset`` should match the length of axis 0 of a 2D spatiotemporal ``dataset``.
32
32
33
33
Data Format
@@ -43,6 +43,174 @@ integer. The scale_factor is provided in the ``scale_factor`` attribute. The
43
43
units for each variable are also commonly provided as an attribute called
44
44
``units``.
45
45
46
+
47
+
Many NREL tools have been developed based on the assumption that the data format
48
+
will follow a pseudo-standard definition. Data creators can adhere to the
49
+
following specifications for seamless integration into
- Each file has a top level attribute named ``version`` that defines the model version. In the example above this attribute would have a value of ``v4.0.0``. This is the version that is included in the files users
86
+
will download.
87
+
- Each file includes 2 or 3 specific datasets that are tied to The API functionality. They are:
88
+
89
+
- ``meta`` which contains a table of location specific metadata for each pixel/point/grid-cell of data. There are a few required
90
+
metadata values, and no limit to additional metadata that can be
91
+
included as deemed useful for the specific model. The required
92
+
values are
93
+
94
+
- latitude - either of the actual point or the centroid
95
+
- longitude - either of the actual point or the centroid
96
+
- timezone
97
+
98
+
- The ``meta`` dataset will be 1 a dimensional array that must
99
+
have the same length as the spatial dimension of the datasets
100
+
- ``time_index`` which contains UTC timestamps in ISO 8601
101
+
(``2022-01-01 00:45:00+00:00``) defining the temporal value of each
102
+
data step.
103
+
104
+
- The ``time_index`` dataset will be a 1 dimensional array that must
105
+
have the same length as the temporal dimension of the datasets.
106
+
107
+
- ``tmy_year`` which is ONLY required for TMY data. ``tmy_year`` includes a
108
+
value per timestep and per point which contains a single integer value
109
+
defining the year that the TMY data was derived from.
110
+
111
+
- Each file includes any number of datasets that include the model
112
+
outputs. Each dataset is a variable e.g. “wind_speed” or “ghi”.
113
+
114
+
- Each of these datasets will be a 2 dimensional array with (time, space)
115
+
dimensions
116
+
- The number of locations must be identical within every yearly file of the
117
+
same domain
118
+
- The number of timesteps must represent a consistent temporal
119
+
resolution (hourly, 5 minute, etc), though there can be variations
120
+
in size for leap years
121
+
- Each of these datasets can optionally include any number of attributes. The
122
+
following attribtues are used by the APIs to determine output formatting.
123
+
124
+
- *scale_factor*: In cases where floats have been converted to
125
+
integers for storage efficiency the *scale_factor* is the value to
126
+
divide the raw data by to restore the original value. E.g.
127
+
``h5_val / scale_factor = actual_val``
128
+
- *fill_value*: The value that is used to represent NULL. *\* NOTE
129
+
that empty values in an HDF5 file are technically allowable but will
130
+
result in errors during data post-processing by NREL tools, hence it is
131
+
important to include a fill value in datasets where NULLs are possible*
132
+
- *units*: The string representation of the units that apply to the
133
+
raw data. Only necessary when you want values in Kelvin to be
134
+
converted to Celcius for output. Otherwise included in output file
135
+
metadata for the benefit of users
136
+
137
+
Notes on Chunking
138
+
~~~~~~~~~~~~~~~~~
139
+
140
+
Chunking is vital to achieving the best performance when reading the
141
+
data out of H5. Chunking divides up a potentially massive file into
142
+
discreet blocks on disc. When stored to disc HDF5 stores these chunks in
143
+
physically adjacent blocks for best I/O performance. In the cloud each
144
+
chunk is a discreet object. A bad chunking strategy on a typical data
145
+
product can create enough I/O latency to render the data absolutely
146
+
unusable by our tools due to excessive thread locking, network timeouts,
147
+
memory failures, etc.
148
+
149
+
The strategy is to identify the use cases for reading the data out of
150
+
H5, and then assign chunks that are aligned with that strategy. In
151
+
addition it has been observed that tiny chunks will cause slow read time
152
+
due to the I/O latency of reading many little chunks, while huge chunks
153
+
will cause slow read time and excessive memory usage because the process
154
+
is going to have to load a huge amount of data into memory in order to
155
+
slice out the small percentage that is relevant to the user. The worst
156
+
possible case would be to chunk the data opposite the manner that it
157
+
will be fetched. For example, if we imaging the data as a table, and
158
+
users typically fetch the data one column at a time, but you chunk the
159
+
data row-wise, then a single columnar fetch will require loading ALL of
160
+
the chunks. Imagine this same table however if the data is chunked
161
+
column-wise and also fetched column-wise the fetch will read only the
162
+
necessary data off disc thus minimizing I/O and maximizing performance.
163
+
164
+
For example, the APIs always read out a single pixel of data for an entire year
165
+
at a time, the obvious chunking strategy would appear to be to chunk [8760: 1]
166
+
(for hourly data with 8760 time steps). However this actually results in too
167
+
many small chunks and is not the ideal solution. Through testing we have found
168
+
that the ideal chunk for a year of hourly data is [8760: 1000]
169
+
170
+
There have been instances where a chunking strategy had to be adopted
171
+
that was a compromise between non-complementary use cases. One example
172
+
is where some users wanted to read data in spatially adjacent blocks for
173
+
time spans of only a week or a month at a time. This contradicts The API
174
+
use case of always reading out a year of data for a single pixel at a
175
+
time. The solution was to identify a chunk size that was a satisfacory
176
+
compromise for both. We came up with [2190: 1000]. Essentially this
177
+
breaks up the year into 4 quarters, keeping the 1000 adjacent locations
178
+
in each chunk.
179
+
180
+
A secondary note about chunking is that it makes logical sense for the
181
+
chunks to be a factor of the whole. E.g. if hourly data has 8760 time
182
+
steps then good chunk options would inclue 8760, (8760/2)=4380, or
183
+
(8760/4)=2190.
184
+
185
+
Common Data Formatting Errors
186
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
187
+
These are some common data formatting errors that we see when integrating new
188
+
data products into our tools. These are not limitations of the HDF5 format, but
189
+
rather are conventions NREL tools have adopted.
190
+
191
+
- NaNs in H5s
192
+
- Array sizes between different years of the same resource and/or different
193
+
datasets within the same file that don't align on the *x* or *y* axis. Most
194
+
often this is caused by something like the first year missing the first month
195
+
due to the details of when the underlying source data began to be collected
196
+
(not every weather station goes online at midnight on January 1st UTC time).
197
+
*NOTE: time_index length will vary in leap years*
198
+
- Files that contain less than a year of data. Yearly data files are large, and
199
+
it is often more convenient to work in monthly batches. However for final
200
+
upload yearly is required.
201
+
- Scale factors that are applied by multiplication instead of division. We can
202
+
handle this in code, just let us know how your scale factor is intended to be
203
+
used.
204
+
- Confusion about units. If the datasets don't include the units attribute,
205
+
please provide documentation.
206
+
- Excessive precision. Data at levels of precision beyond the model's accuracy,
207
+
or margin of error can dramatically bloat the size of the data products and
208
+
are very damaging to performance at every stage of the process.
209
+
- Extra dimensions in data. A good example is wind speed at multiple elevations.
210
+
These values could easily be represented as a 3D array, however the supported
211
+
way would be to create multiple datasets for each variable at each elevation.
212
+
E.g. *wind_speed_40m*, *wind_speed_60M*, *wind_speed_80m*, etc.
213
+
46
214
Data Location - NREL Users
47
215
--------------------------
48
216
@@ -105,8 +273,9 @@ for S3 instructions or for how to setup HSDS and how to find the files that
105
273
you're interested in. Then update the file paths to the files you want either
106
274
on HSDS or S3.
107
275
276
+
108
277
The rex Resource Class
109
-
++++++++++++++++++++++
278
+
~~~~~~~~~~~~~~~~~~~~~~
110
279
111
280
Data access in rex is built on the ``Resource`` class. The class can be used to
112
281
open and explore NREL h5 files, extract and automatically unscale data, and
@@ -151,7 +320,7 @@ Toolkit file, but it can be requested by the ``WindResource`` class, which
151
320
interpolates the windspeeds between the available 80 and 100 meter hub heights.
152
321
153
322
The rex Resource Extraction Class
154
-
+++++++++++++++++++++++++++++++++
323
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
155
324
156
325
There are also classes that implement additional quality-of-life features. For
157
326
example, you can use the ``ResourceX`` class to retrieve a timeseries DataFrame
@@ -189,7 +358,7 @@ various renewable energy resource types, see the docs `here
0 commit comments