Please describe the enhancement
The REST datasource handler (internal/datasources/rest/handler.go) has several hardcoded values and unimplemented stubs that make it hard to use with real-world APIs:
- Timeout is hardcoded to 5s (line 160-161) : there's a TODO acknowledging this. Slower upstream APIs will just time out with no way to adjust.
- Auth is unimplemented (line 62): if your datasource points at an API that isn't the registered provider, there's no way to pass API keys or bearer tokens.
- Fallback is a no-op (lines 61, 228): the struct references it and
doRequest() has a placeholder, but nothing actually happens when a request fails.
- 1MB response limit is hardcoded (line 40):
MaxBytesLimit = 1 << 20 with no override. Some APIs legitimately return more.
- No custom HTTP client support** (line 158): another existing TODO.
These are all flagged with TODO comments in the code already.
Solution Proposal
I think the cleanest approach would be adding a few optional fields to RestDataSource_Def in the proto definition something like a timeout duration, a max_response_bytes int, and an auth block that supports at least API keys and bearer tokens.
The current defaults (5s timeout, 1MB limit) are reasonable and should stay as-is for existing datasources. The new fields would just let people override them when needed.
For fallback, even something simple like "try an alternate endpoint" or "return a structured error instead of nothing" would go a long way.
Describe alternatives you've considered
You could handle some of this at the provider level instead of per-datasource, but that falls apart when a datasource calls a third-party API that has nothing to do with the registered provider. Putting the config on the datasource definition itself gives the most flexibility without overcomplicating things.
Additional context
For reference, here are the exact lines in internal/datasources/rest/handler.go:
- Line 40:
MaxBytesLimit int64 = 1 << 20
- Line 61:
// TODO implement fallback
- Line 62:
// TODO implement auth
- Line 158:
// TODO: Add option to use custom client
- Line 160:
// TODO: Make timeout configurable
- Line 228:
// TODO: Handle fallback here.
Acceptance Criteria
Please describe the enhancement
The REST datasource handler (
internal/datasources/rest/handler.go) has several hardcoded values and unimplemented stubs that make it hard to use with real-world APIs:doRequest()has a placeholder, but nothing actually happens when a request fails.MaxBytesLimit = 1 << 20with no override. Some APIs legitimately return more.These are all flagged with TODO comments in the code already.
Solution Proposal
I think the cleanest approach would be adding a few optional fields to
RestDataSource_Defin the proto definition something like atimeoutduration, amax_response_bytesint, and anauthblock that supports at least API keys and bearer tokens.The current defaults (5s timeout, 1MB limit) are reasonable and should stay as-is for existing datasources. The new fields would just let people override them when needed.
For fallback, even something simple like "try an alternate endpoint" or "return a structured error instead of nothing" would go a long way.
Describe alternatives you've considered
You could handle some of this at the provider level instead of per-datasource, but that falls apart when a datasource calls a third-party API that has nothing to do with the registered provider. Putting the config on the datasource definition itself gives the most flexibility without overcomplicating things.
Additional context
For reference, here are the exact lines in
internal/datasources/rest/handler.go:MaxBytesLimit int64 = 1 << 20// TODO implement fallback// TODO implement auth// TODO: Add option to use custom client// TODO: Make timeout configurable// TODO: Handle fallback here.Acceptance Criteria
doRequest()actually does something, even a structured error is better than silently returning nothing