Skip to content

ext_filter() performs redundant HTTP requests before a filtered search #192

Description

@rolfsimoes

Describe the bug

A filtered STAC search performs several discovery requests before sending the main request. With services that apply strict rate limits, these additional requests can cause the main POST /search request to fail with HTTP 429.

This was observed with rstac 1.2.0 and the Copernicus Data Space Ecosystem STAC API, which limits the catalogue to 5 requests/second.

To Reproduce

library(rstac)
library(httr)

url <- "https://stac.dataspace.copernicus.eu/v1"

items <- with_verbose({
    query <- stac_search(
        stac(url),
        collections = "sentinel-2-l2a",
        datetime = "2023-05-01T00:00:00Z/2023-09-01T00:00:00Z",
        limit = 1
    )

    query <- ext_filter(
        query,
        `product:type` == "S2MSI2A"
    )

    post_request(query)
})

A single filtered search generates the following request sequence:

GET  /v1/
GET  /v1/
GET  /v1/api
GET  /v1/
POST /v1/search

The first two requests to /v1/ are made by get_request(stac(url)) inside openapi_schema(): one request detects the STAC version and another retrieves the landing page. link_open() then retrieves the OpenAPI document from /v1/api. Later, post_request() requests /v1/ again because the detected version was not retained in the query.

The server then returns:

HTTP/1.0 429 Too Many Requests
Retry-After: 2

The following workaround confirms the cause:

query <- stac_search(
    stac(
        url,
        force_version = "1.0.0"
    ),
    collections = "sentinel-2-l2a",
    datetime = "2023-05-01T00:00:00Z/2023-09-01T00:00:00Z",
    limit = 1
)

query <- ext_filter(
    query,
    `product:type` == "S2MSI2A"
)

items <- post_request(query)

With force_version = "1.0.0", the sequence is reduced to:

GET  /v1/
GET  /v1/api
POST /v1/search

This request completes without HTTP 429.

Expected behavior

A STAC landing page should not be downloaded repeatedly during a single request lifecycle. The response used to detect the STAC version should be reused when possible, and the detected version should be retained for later steps.

Schema validation performed by ext_filter() should not cause duplicate discovery requests.

Additional context

The issue affects downstream packages that construct filtered queries through rstac, including sits.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions