It is difficult to do parameter filters in RISE

[C-Loftus]

I was chatting with John late last week and I think we have a blocker if we want to keep proxying and some general challenges with filtering by parameters:

Frontend-driven challenges

• We need to be able to fetch the latest reservoir storage for all reservoirs for the teacups;
  • However, this is 140 different parameters and RISE rejects it as being too much fetched data whether as separate queries or all in the same URL
• This essentially means there is no way to get this with a realtime proxy unless we fetch and add contrived time.sleep to stagger it
  • this would make the UI far too slow

So essentially my only way to get this info is to schedule cache the params ahead of time by scheduling them; then after they are cached we no longer need to ping RISE and thus won't get the issue
https://github.com/dbader/schedule

However this is messy and running a scheduled prefetch cache job is not ideal

General parameter filter issues

In general filtering on parameters is difficult in RISE.

In RISE there are multiple different endpoints such as catalogItems, catalogRecords, parameters, and locations.

You can include catalogItems with the locations by doing the following, and you can filter out locations by certain parameters, but I do not believe there is a way to include the parameters joined with the locations in the actual response.

The issue is that if you filter with f"parameterId%5B%5D={prop}&" it returns any location that has that param, so I then need a way of filtering out the param from the location.

        base_url = "https://data.usbr.gov/rise/api/location?order[id]=asc&include=catalogRecords.catalogItems"
        hasTimeseriesData = "itemStructureId=1"
        if only_include_locations_with_data:
            base_url += f"&{hasTimeseriesData}"

        if properties_to_filter_by:
            base_url += "&"
            for prop in properties_to_filter_by:
                assert isinstance(prop, str)
                base_url += f"parameterId%5B%5D={prop}&"
            base_url = base_url.removesuffix("&")

Because I only get the catalog items and not the actual parameter response, there is no good way to filter without doing another fetch for every parameter.

[C-Loftus]

To put it another way,

essentially the consequence of the fact RISE does not return parameters with locations, is that I currently filter on properties after the data is already fetched, that is since otherwise I would need to

1. Fetch all locations
2. Fetch all catalogItems for each
3. For each catalog item response, fetch all the include params
4. Get the param metadata
5. Filter the locations by that metadata

i.e. hence why I have the select properties applied after the covjson already has all the data needed

    def render(
        self,
        location_response: list[DataNeededForCovjson],
        select_properties: list[str] | None,
    ) -> CoverageCollectionDict:
        templated_covjson: CoverageCollectionDict = COVJSON_TEMPLATE
        paramIdToMetadata: EDRFieldsMapping = await_(
            self._cache.get_or_fetch_parameters()
        )
        if select_properties:
            paramIdToMetadata = {
                k: v for k, v in paramIdToMetadata.items() if k in select_properties
            }

parameter queries don't appear to have any way of getting the data unless you fetch the exact id, which comes only from the catalog item response. Hence you have to do multiple layers of fetching

Since catalog items and records have different ids and are highly varied across responses, it is difficult to cache with the url as the key

[C-Loftus]

So to summarize, we have 3 options to resolve this

1. Prefetch / populate the cache on a schedule
   • This is ok if we just want to add the latest reservoir storage values, but it gets very messy to do this for other params (should just crawl at a certain point)
   • This doesn't solve the issue that there is no way to filter params since locations don't contain params
2. Crawl things and put them in a sql db
   • Much better access patterns and speech, but adds complexity
   • Allows us to filter by params much easier
   • Allows for native geospatial filtering instead of just looping over and filtering a huge json object
   • Still not ideal and then we end up needing to duplicate data; no longer possible to update historical data
   • Have to switch a significant part of our tech stack to use dagster or another crawl tool
3. Use a native EDR provider over the rise db directly
   • ideal from a technical perspective, going to be the fastest and easiest to implement technically
   • provider code is mainly written
   • however, for logistical reasons it may take a bit and we can't really test with this or use it for the UI unless we have approval which could take a while.
   • Working with the db may open up unforseen challenges (dependencies, etc)

[C-Loftus]

I was not aware of this previously but it now does appear possible to return param info with the location, but it was not documented. So with that in mind, it becomes a bit easier to keep the proxy in place. Still need to align on how to we would be able to fetch 140 catalogItems for the reservoir storage all at once