feat(inference): expose httpx connection pool limits via network config (#6306)

# What does this PR do?

All remote inference providers use httpx internally but never exposed
httpx.Limits, so every provider was stuck with the hardcoded defaults of
100 max connections and 20 max keepalive connections regardless of the
platform's actual concurrency needs. Connection pooling is a standard
HTTP client concern platform admins should be able to tune.

Adds a LimitsConfig model (mirroring httpx.Limits) as a new field on the
existing NetworkConfig, and wires it into build_network_client_kwargs(),
the single shared function already used by OpenAIMixin (and therefore
every OpenAI-compatible remote provider), Bedrock, and the interactions
provider to build their httpx clients from TLS/proxy/timeout/headers
config. VertexAI builds its httpx client separately via the Google GenAI
SDK, so its build_httpx_kwargs() helper got the same handling.

Regenerated provider docs via provider_codegen.py to reflect the new
network.limits.* fields across all remote inference providers.


Closes #5910

---------

Signed-off-by: elina priyadarshinee <elina.priyadarshinee1@ibm.com>
Signed-off-by: Matthew Farrellee <matt@cs.wisc.edu>
Co-authored-by: Matthew Farrellee <matt@cs.wisc.edu>

Author: elinacse

docs/docs/providers/inference/remote_anthropic.mdx

@@ -34,6 +34,10 @@ Anthropic inference provider for accessing Claude models and Anthropic's AI serv
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 
 ## Sample Configuration
 

docs/docs/providers/inference/remote_azure.mdx

@@ -41,6 +41,10 @@ https://learn.microsoft.com/en-us/azure/ai-foundry/openai/overview
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No |  | Azure API base for Azure (e.g., https://your-resource-name.openai.azure.com/openai/v1) |
 | `api_version` | `str \| None` | No |  | Azure API version for Azure (e.g., 2024-12-01-preview) |
 | `api_type` | `str \| None` | No | azure | Azure API type for Azure (e.g., azure) |

docs/docs/providers/inference/remote_bedrock.mdx

@@ -34,6 +34,10 @@ AWS Bedrock inference provider for the OpenAI-compatible runtime, with AWS crede
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `aws_access_key_id` | `SecretStr \| None` | No |  | The AWS access key to use. Default use environment variable: AWS_ACCESS_KEY_ID |
 | `aws_secret_access_key` | `SecretStr \| None` | No |  | The AWS secret access key to use. Default use environment variable: AWS_SECRET_ACCESS_KEY |
 | `aws_session_token` | `SecretStr \| None` | No |  | The AWS session token to use. Default use environment variable: AWS_SESSION_TOKEN |

docs/docs/providers/inference/remote_cerebras.mdx

@@ -34,6 +34,10 @@ Cerebras inference provider for running models on Cerebras Cloud platform.
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No | https://api.cerebras.ai/v1 | Base URL for the Cerebras API |
 
 ## Sample Configuration

docs/docs/providers/inference/remote_databricks.mdx

@@ -34,6 +34,10 @@ Databricks inference provider for running models on Databricks' unified analytic
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No |  | The URL for the Databricks model serving endpoint (should include /serving-endpoints path) |
 
 ## Sample Configuration

docs/docs/providers/inference/remote_deepseek.mdx

@@ -34,6 +34,10 @@ DeepSeek inference provider for accessing DeepSeek models via the DeepSeek API.
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No | https://api.deepseek.com/v1 | Base URL for the DeepSeek API |
 
 ## Sample Configuration

docs/docs/providers/inference/remote_fireworks.mdx

@@ -34,6 +34,10 @@ Fireworks AI inference provider for Llama models and other AI models on the Fire
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No | https://api.fireworks.ai/inference/v1 | The URL for the Fireworks server |
 
 ## Sample Configuration

docs/docs/providers/inference/remote_gemini.mdx

@@ -34,6 +34,10 @@ Google Gemini inference provider for accessing Gemini models and Google's AI ser
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `access_token` | `SecretStr \| None` | No |  | OAuth2 access token for Gemini. When set, used instead of api_key for Bearer authentication. |
 | `project` | `str \| None` | No |  | Google Cloud project ID for quota attribution when using OAuth/ADC credentials. |
 

docs/docs/providers/inference/remote_groq.mdx

@@ -34,6 +34,10 @@ Groq inference provider for ultra-fast inference using Groq's LPU technology.
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No | https://api.groq.com/openai/v1 | The URL for the Groq AI server |
 
 ## Sample Configuration

docs/docs/providers/inference/remote_llama-cpp-server.mdx

@@ -34,6 +34,10 @@ llama.cpp inference provider for connecting to llama.cpp servers with OpenAI-com
 | `network.timeout.connect` | `float \| None` | No |  | Connection timeout in seconds. |
 | `network.timeout.read` | `float \| None` | No |  | Read timeout in seconds. |
 | `network.headers` | `dict[str, str] \| None` | No |  | Additional HTTP headers to include in all requests. |
+| `network.limits` | `LimitsConfig \| None` | No |  | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
+| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
+| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
+| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
 | `base_url` | `HttpUrl \| None` | No | http://localhost:8080/v1 | The URL for the Llama cpp server |
 
 ## Sample Configuration