Skip to content

HTTP Response Body leak in REST Datasource retriableDo when rate limited #6288

@AftAb-25

Description

@AftAb-25

Describe the issue

Describe the bug

While reviewing the internal/datasources/rest/handler.go implementation, I noticed a resource leak in the REST datasource's retry logic.

Specifically, in retriableDo(), when the server responds with a 429 Too Many Requests status code, the function triggers a retry by returning a rate limited error via the backoff library. However, it never closes resp.Body before returning.

Since the *http.Response is overwritten on the next retry attempt, the previous response body becomes completely unreachable and is left unclosed. This holds the underlying TCP connection open, which can lead to file descriptor exhaustion and connection pool starvation if the rate limiting is sustained.

Steps to reproduce

  1. Configure a REST datasource to hit an endpoint that strictly rate-limits requests.
  2. Trigger enough evaluations or API requests to provoke 429 responses.
  3. If you repeatedly hit the 429 path (with up to 3 max retries), observe the process accumulating open file descriptors or active connections that aren't being cleanly released.

Expected behavior

The HTTP response body should be explicitly drained and closed prior to triggering the retry increment so the connection can be safely released or reused.

Proposed Fix

In internal/datasources/rest/handler.go around line 348, we should add io.Copy(io.Discard, resp.Body) and resp.Body.Close() inside the 429 check:

		if resp.StatusCode == http.StatusTooManyRequests {
			// Drain and close the response body to release the connection
			io.Copy(io.Discard, resp.Body)
			resp.Body.Close()
			
			zerolog.Ctx(req.Context()).Debug().
				Int("retry", retryCount).
				Msg("rate limited, retrying")
			retryCount++
			return errors.New("rate limited")
		}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions