Skip to content

Commit b24fb5d

Browse files
committed
fix(inference): move network config classes out of model_registry into dedicated module
Create network_config.py with TLSConfig, ProxyConfig, TimeoutConfig, LimitsConfig, and NetworkConfig as a standalone module. These are semantically unrelated to model registry logic. Update all source and test imports to pull network config classes from the new module while keeping RemoteInferenceProviderConfig, ProviderModelEntry, and ModelRegistryHelper in model_registry.py. Keep backward-compatible re-exports in model_registry.py for existing import paths. Regenerate provider documentation with updated field descriptions. Signed-off-by: Matthew Farrellee <matt@cs.wisc.edu>
1 parent 3894af3 commit b24fb5d

33 files changed

Lines changed: 280 additions & 246 deletions

docs/docs/providers/inference/remote_anthropic.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ Anthropic inference provider for accessing Claude models and Anthropic's AI serv
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141

4242
## Sample Configuration
4343

docs/docs/providers/inference/remote_azure.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,9 @@ https://learn.microsoft.com/en-us/azure/ai-foundry/openai/overview
4242
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
4343
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
4444
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
45-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
46-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
47-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
45+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
46+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
47+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4848
| `base_url` | `HttpUrl \| None` | No | | Azure API base for Azure (e.g., https://your-resource-name.openai.azure.com/openai/v1) |
4949
| `api_version` | `str \| None` | No | | Azure API version for Azure (e.g., 2024-12-01-preview) |
5050
| `api_type` | `str \| None` | No | azure | Azure API type for Azure (e.g., azure) |

docs/docs/providers/inference/remote_bedrock.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ AWS Bedrock inference provider for the OpenAI-compatible runtime, with AWS crede
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `aws_access_key_id` | `SecretStr \| None` | No | | The AWS access key to use. Default use environment variable: AWS_ACCESS_KEY_ID |
4242
| `aws_secret_access_key` | `SecretStr \| None` | No | | The AWS secret access key to use. Default use environment variable: AWS_SECRET_ACCESS_KEY |
4343
| `aws_session_token` | `SecretStr \| None` | No | | The AWS session token to use. Default use environment variable: AWS_SESSION_TOKEN |

docs/docs/providers/inference/remote_cerebras.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ Cerebras inference provider for running models on Cerebras Cloud platform.
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `base_url` | `HttpUrl \| None` | No | https://api.cerebras.ai/v1 | Base URL for the Cerebras API |
4242

4343
## Sample Configuration

docs/docs/providers/inference/remote_databricks.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ Databricks inference provider for running models on Databricks' unified analytic
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `base_url` | `HttpUrl \| None` | No | | The URL for the Databricks model serving endpoint (should include /serving-endpoints path) |
4242

4343
## Sample Configuration

docs/docs/providers/inference/remote_deepseek.mdx

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ DeepSeek inference provider for accessing DeepSeek models via the DeepSeek API.
3434
| `network.timeout.connect` | `float \| None` | No | | Connection timeout in seconds. |
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
37+
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
3741
| `base_url` | `HttpUrl \| None` | No | https://api.deepseek.com/v1 | Base URL for the DeepSeek API |
3842

3943
## Sample Configuration

docs/docs/providers/inference/remote_fireworks.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ Fireworks AI inference provider for Llama models and other AI models on the Fire
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `base_url` | `HttpUrl \| None` | No | https://api.fireworks.ai/inference/v1 | The URL for the Fireworks server |
4242

4343
## Sample Configuration

docs/docs/providers/inference/remote_gemini.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ Google Gemini inference provider for accessing Gemini models and Google's AI ser
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `access_token` | `SecretStr \| None` | No | | OAuth2 access token for Gemini. When set, used instead of api_key for Bearer authentication. |
4242
| `project` | `str \| None` | No | | Google Cloud project ID for quota attribution when using OAuth/ADC credentials. |
4343

docs/docs/providers/inference/remote_groq.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ Groq inference provider for ultra-fast inference using Groq's LPU technology.
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `base_url` | `HttpUrl \| None` | No | https://api.groq.com/openai/v1 | The URL for the Groq AI server |
4242

4343
## Sample Configuration

docs/docs/providers/inference/remote_llama-cpp-server.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ llama.cpp inference provider for connecting to llama.cpp servers with OpenAI-com
3535
| `network.timeout.read` | `float \| None` | No | | Read timeout in seconds. |
3636
| `network.headers` | `dict[str, str] \| None` | No | | Additional HTTP headers to include in all requests. |
3737
| `network.limits` | `LimitsConfig \| None` | No | | HTTP connection pool limits (max connections, keepalive connections, keepalive expiry). None uses httpx's own defaults. |
38-
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. |
39-
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. |
40-
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. |
38+
| `network.limits.max_connections` | `int \| None` | No | 100 | Maximum number of concurrent connections in the pool. None means no limit. Values must be >= 1 if set. |
39+
| `network.limits.max_keepalive_connections` | `int \| None` | No | 20 | Maximum number of idle keep-alive connections to retain. None means no limit. Values must be >= 0 if set. |
40+
| `network.limits.keepalive_expiry` | `float \| None` | No | 5.0 | Time in seconds to keep idle keep-alive connections open before closing them. None means no expiry. Values must be >= 0 if set. |
4141
| `base_url` | `HttpUrl \| None` | No | http://localhost:8080/v1 | The URL for the Llama cpp server |
4242

4343
## Sample Configuration

0 commit comments

Comments
 (0)