-
Notifications
You must be signed in to change notification settings - Fork 85
HTTP Response Body leak in REST Datasource retriableDo when rate limited #6288
Description
Describe the issue
Describe the bug
While reviewing the internal/datasources/rest/handler.go implementation, I noticed a resource leak in the REST datasource's retry logic.
Specifically, in retriableDo(), when the server responds with a 429 Too Many Requests status code, the function triggers a retry by returning a rate limited error via the backoff library. However, it never closes resp.Body before returning.
Since the *http.Response is overwritten on the next retry attempt, the previous response body becomes completely unreachable and is left unclosed. This holds the underlying TCP connection open, which can lead to file descriptor exhaustion and connection pool starvation if the rate limiting is sustained.
Steps to reproduce
- Configure a REST datasource to hit an endpoint that strictly rate-limits requests.
- Trigger enough evaluations or API requests to provoke
429responses. - If you repeatedly hit the 429 path (with up to 3 max retries), observe the process accumulating open file descriptors or active connections that aren't being cleanly released.
Expected behavior
The HTTP response body should be explicitly drained and closed prior to triggering the retry increment so the connection can be safely released or reused.
Proposed Fix
In internal/datasources/rest/handler.go around line 348, we should add io.Copy(io.Discard, resp.Body) and resp.Body.Close() inside the 429 check:
if resp.StatusCode == http.StatusTooManyRequests {
// Drain and close the response body to release the connection
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
zerolog.Ctx(req.Context()).Debug().
Int("retry", retryCount).
Msg("rate limited, retrying")
retryCount++
return errors.New("rate limited")
}