Describe the bug
A filtered STAC search performs several discovery requests before sending the main request. With services that apply strict rate limits, these additional requests can cause the main POST /search request to fail with HTTP 429.
This was observed with rstac 1.2.0 and the Copernicus Data Space Ecosystem STAC API, which limits the catalogue to 5 requests/second.
To Reproduce
library(rstac)
library(httr)
url <- "https://stac.dataspace.copernicus.eu/v1"
items <- with_verbose({
query <- stac_search(
stac(url),
collections = "sentinel-2-l2a",
datetime = "2023-05-01T00:00:00Z/2023-09-01T00:00:00Z",
limit = 1
)
query <- ext_filter(
query,
`product:type` == "S2MSI2A"
)
post_request(query)
})
A single filtered search generates the following request sequence:
GET /v1/
GET /v1/
GET /v1/api
GET /v1/
POST /v1/search
The first two requests to /v1/ are made by get_request(stac(url)) inside openapi_schema(): one request detects the STAC version and another retrieves the landing page. link_open() then retrieves the OpenAPI document from /v1/api. Later, post_request() requests /v1/ again because the detected version was not retained in the query.
The server then returns:
HTTP/1.0 429 Too Many Requests
Retry-After: 2
The following workaround confirms the cause:
query <- stac_search(
stac(
url,
force_version = "1.0.0"
),
collections = "sentinel-2-l2a",
datetime = "2023-05-01T00:00:00Z/2023-09-01T00:00:00Z",
limit = 1
)
query <- ext_filter(
query,
`product:type` == "S2MSI2A"
)
items <- post_request(query)
With force_version = "1.0.0", the sequence is reduced to:
GET /v1/
GET /v1/api
POST /v1/search
This request completes without HTTP 429.
Expected behavior
A STAC landing page should not be downloaded repeatedly during a single request lifecycle. The response used to detect the STAC version should be reused when possible, and the detected version should be retained for later steps.
Schema validation performed by ext_filter() should not cause duplicate discovery requests.
Additional context
The issue affects downstream packages that construct filtered queries through rstac, including sits.
Describe the bug
A filtered STAC search performs several discovery requests before sending the main request. With services that apply strict rate limits, these additional requests can cause the main
POST /searchrequest to fail with HTTP 429.This was observed with
rstac 1.2.0and the Copernicus Data Space Ecosystem STAC API, which limits the catalogue to 5 requests/second.To Reproduce
A single filtered search generates the following request sequence:
The first two requests to
/v1/are made byget_request(stac(url))insideopenapi_schema(): one request detects the STAC version and another retrieves the landing page.link_open()then retrieves the OpenAPI document from/v1/api. Later,post_request()requests/v1/again because the detected version was not retained in the query.The server then returns:
The following workaround confirms the cause:
With
force_version = "1.0.0", the sequence is reduced to:This request completes without HTTP 429.
Expected behavior
A STAC landing page should not be downloaded repeatedly during a single request lifecycle. The response used to detect the STAC version should be reused when possible, and the detected version should be retained for later steps.
Schema validation performed by
ext_filter()should not cause duplicate discovery requests.Additional context
The issue affects downstream packages that construct filtered queries through
rstac, includingsits.