fix(aws): configurable boto3 timeouts, 10s connect default (#12774)

Author: pedrooot

docs/getting-started/basic-usage/prowler-cli.mdx

@@ -2,6 +2,8 @@
 title: 'Basic Usage'
 ---
 
+import { VersionBadge } from "/snippets/version-badge.mdx"
+
 ## Running Prowler
 
 Running Prowler requires specifying the provider (e.g. `aws`, `gcp`, `azure`, `kubernetes`, `m365`, `github`, `iac` or `mongodbatlas`):
@@ -91,6 +93,18 @@ By default, `prowler` will scan all AWS regions.
 </Note>
 See more details about AWS Authentication in the [Authentication Section](/user-guide/providers/aws/authentication) section.
 
+- **AWS Retrier and Timeout Configuration**
+
+    <VersionBadge version="5.42.0" />
+
+    Tune the Boto3 standard retrier and the endpoint timeouts when AWS throttles the scan or when some endpoints are unreachable from the network Prowler runs in:
+
+    ```console
+    prowler aws --aws-retries-max-attempts 5 --aws-connect-timeout 5 --aws-read-timeout 30
+    ```
+
+    See the [Boto3 configuration](/user-guide/providers/aws/boto3-configuration) page for defaults and environment variables.
+
 ## Azure
 
 Azure requires specifying the auth method:

docs/user-guide/providers/aws/boto3-configuration.mdx

@@ -1,14 +1,39 @@
 ---
-title: "Boto3 Retrier Configuration in Prowler"
+title: "Boto3 Retrier and Timeout Configuration in Prowler"
 ---
 
+import { VersionBadge } from "/snippets/version-badge.mdx"
+
 Prowler's AWS Provider leverages Boto3's [Standard](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html) retry mode to automatically retry client calls to AWS services when encountering errors or exceptions.
 
+## Timeout Configuration
+
+<VersionBadge version="5.42.0" />
+
+Every AWS API call is bounded by two timeouts:
+
+- Connect timeout: seconds to wait to establish a connection (TCP, proxy tunnel and TLS handshake) to the AWS endpoint. Prowler's default is 10 seconds, configurable via `--aws-connect-timeout 5`.
+- Read timeout: seconds to wait for a response once connected. Prowler's default is 60 seconds, configurable via `--aws-read-timeout 30`.
+
+Both timeouts can also be set through environment variables, which is the way to tune them in Prowler Cloud and other deployments without a CLI:
+
+```console
+export PROWLER_AWS_BOTO3_CONNECT_TIMEOUT=5
+export PROWLER_AWS_BOTO3_READ_TIMEOUT=30
+```
+
+CLI flags take precedence over the environment variables. Prowler sets both timeouts explicitly, so `AWS_DEFAULTS_MODE` and a `connect_timeout` in `~/.aws/config` are ignored; use the flag or the environment variable instead.
+
+<Note>
+Boto3 defaults both timeouts to 60 seconds. In networks with restricted egress (for example VPC endpoints for a subset of services, GovCloud or private deployments), every AWS service without a reachable endpoint used to cost up to 4 attempts × 60 seconds (the first call plus the 3 retries) for each region. Prowler lowers the connect timeout to 10 seconds so unreachable endpoints fail fast; lower it further together with `--aws-retries-max-attempts 0`, which disables retries and leaves a single attempt per call, if a scan still spends most of its time waiting on unreachable services.
+
+</Note>
+
 ## Retry Behavior Overview
 
 Boto3's Standard retry mode includes the following mechanisms:
 
-- Maximum Retry Attempts: Default value set to 3, configurable via the `--aws-retries-max-attempts 5` argument.
+- Maximum Retry Attempts: Default value set to 3, configurable via the `--aws-retries-max-attempts 5` argument. `0` disables retries.
 
 - Expanded Error Handling: Retries occur for a comprehensive set of errors.
 

prowler/changelog.d/aws-boto3-connect-timeout-default.changed.md

@@ -0,0 +1 @@
+AWS provider default Boto3 connect timeout lowered from 60 to 10 seconds, so scans in restricted-egress networks (VPC endpoints for a subset of services, GovCloud, private deployments) no longer spend 4 minutes per region on every service whose endpoint is unreachable

prowler/changelog.d/aws-boto3-timeouts.added.md

@@ -0,0 +1 @@
+`--aws-connect-timeout` and `--aws-read-timeout` CLI flags, plus `PROWLER_AWS_BOTO3_CONNECT_TIMEOUT` and `PROWLER_AWS_BOTO3_READ_TIMEOUT` environment variables, to bound how long each AWS API call waits for an endpoint

prowler/changelog.d/aws-retries-max-attempts-zero.fixed.md

@@ -0,0 +1 @@
+`--aws-retries-max-attempts 0` now disables Boto3 retries instead of being silently ignored in favour of the default of 3

prowler/providers/aws/aws_provider.py

@@ -126,6 +126,8 @@ def __init__(
         aws_access_key_id: str = None,
         aws_secret_access_key: str = None,
         aws_session_token: Optional[str] = None,
+        connect_timeout: Optional[int] = None,
+        read_timeout: Optional[int] = None,
     ):
         """
         Initializes the AWS provider.
@@ -155,6 +157,8 @@ def __init__(
             - aws_access_key_id: The AWS access key ID.
             - aws_secret_access_key: The AWS secret access key.
             - aws_session_token: The AWS session token, optional.
+            - connect_timeout: Seconds to wait to establish a connection to an AWS endpoint.
+            - read_timeout: Seconds to wait for a response from an AWS endpoint.
 
         Raises:
             - ArgumentTypeError: If the input MFA ARN is invalid.
@@ -229,7 +233,9 @@ def __init__(
 
         # TODO: Use AwsSetUpSession ?????
         # Configure the initial AWS Session using the local credentials: profile or environment variables
-        session_config = self.set_session_config(retries_max_attempts)
+        session_config = self.set_session_config(
+            retries_max_attempts, connect_timeout, read_timeout
+        )
         aws_session = self.setup_session(
             mfa=mfa,
             profile=profile,
@@ -1165,26 +1171,35 @@ def input_role_mfa_token_and_code() -> AWSMFAInfo:
         return AWSMFAInfo(arn=mfa_ARN, totp=mfa_TOTP)
 
     @staticmethod
-    def set_session_config(retries_max_attempts: int) -> Config:
+    def set_session_config(
+        retries_max_attempts: int,
+        connect_timeout: Optional[int] = None,
+        read_timeout: Optional[int] = None,
+    ) -> Config:
         """
-        set_session_config returns a botocore Config object with the Prowler user agent and the default retrier configuration if nothing is passed as argument
+        set_session_config returns a botocore Config object with the Prowler user agent and the default retrier and timeout configuration if nothing is passed as argument
 
         Args:
             - retries_max_attempts: The maximum number of retries for the standard retrier config
+            - connect_timeout: Seconds to wait to establish a connection to an AWS endpoint
+            - read_timeout: Seconds to wait for a response from an AWS endpoint
 
         Returns:
             - Config: The botocore Config object
         """
         default_session_config = get_default_session_config()
-        if retries_max_attempts:
-            default_session_config = default_session_config.merge(
-                Config(
-                    retries={
-                        "max_attempts": retries_max_attempts,
-                        "mode": "standard",
-                    },
-                )
-            )
+        overrides = {}
+        if retries_max_attempts is not None:
+            overrides["retries"] = {
+                "max_attempts": retries_max_attempts,
+                "mode": "standard",
+            }
+        if connect_timeout:
+            overrides["connect_timeout"] = connect_timeout
+        if read_timeout:
+            overrides["read_timeout"] = read_timeout
+        if overrides:
+            default_session_config = default_session_config.merge(Config(**overrides))
 
         return default_session_config
 

prowler/providers/aws/config.py

@@ -2,14 +2,39 @@
 
 from botocore.config import Config
 
+from prowler.providers.aws.exceptions.exceptions import AWSInvalidBoto3TimeoutError
+
 AWS_STS_GLOBAL_ENDPOINT_REGION = "us-east-1"
 AWS_REGION_US_EAST_1 = "us-east-1"
 BOTO3_USER_AGENT_EXTRA = os.getenv("PROWLER_AWS_BOTO3_USER_AGENT_EXTRA", "APN_1826889")
+BOTO3_RETRIES_MAX_ATTEMPTS = 3
+# botocore defaults both to 60s
+BOTO3_CONNECT_TIMEOUT = 10
+BOTO3_READ_TIMEOUT = 60
 ROLE_SESSION_NAME = "ProwlerAssessmentSession"
 
 
+def get_boto3_timeout_from_env(name: str, default: int) -> int:
+    """Positive integer seconds read from the environment, or default when unset."""
+    raw = os.getenv(name, "").strip()
+    if not raw:
+        return default
+    if not raw.isdecimal() or int(raw) == 0:
+        raise AWSInvalidBoto3TimeoutError(
+            file=os.path.basename(__file__),
+            message=f"{name} must be a positive integer number of seconds, got {raw!r}",
+        )
+    return int(raw)
+
+
 def get_default_session_config() -> Config:
     return Config(
         user_agent_extra=BOTO3_USER_AGENT_EXTRA,
-        retries={"max_attempts": 3, "mode": "standard"},
+        retries={"max_attempts": BOTO3_RETRIES_MAX_ATTEMPTS, "mode": "standard"},
+        connect_timeout=get_boto3_timeout_from_env(
+            "PROWLER_AWS_BOTO3_CONNECT_TIMEOUT", BOTO3_CONNECT_TIMEOUT
+        ),
+        read_timeout=get_boto3_timeout_from_env(
+            "PROWLER_AWS_BOTO3_READ_TIMEOUT", BOTO3_READ_TIMEOUT
+        ),
     )

prowler/providers/aws/exceptions/exceptions.py

@@ -78,6 +78,10 @@ class AWSBaseException(ProwlerException):
             "message": "The provided AWS partition is invalid",
             "remediation": "Check the provided AWS partition and ensure it is valid.",
         },
+        (1918, "AWSInvalidBoto3TimeoutError"): {
+            "message": "The Boto3 timeout configured through the environment is invalid",
+            "remediation": "Set PROWLER_AWS_BOTO3_CONNECT_TIMEOUT and PROWLER_AWS_BOTO3_READ_TIMEOUT to a positive integer number of seconds.",
+        },
     }
 
     def __init__(self, code, file=None, original_exception=None, message=None):
@@ -231,3 +235,12 @@ def __init__(self, file=None, original_exception=None, message=None):
         super().__init__(
             1917, file=file, original_exception=original_exception, message=message
         )
+
+
+class AWSInvalidBoto3TimeoutError(AWSBaseException):
+    """Boto3 timeout configured through the environment is not a positive integer."""
+
+    def __init__(self, file=None, original_exception=None, message=None):
+        super().__init__(
+            1918, file=file, original_exception=original_exception, message=message
+        )

prowler/providers/aws/lib/arguments/arguments.py

@@ -156,7 +156,21 @@ def init_parser(self):
         nargs="?",
         default=None,
         type=int,
-        help="Set the maximum attemps for the Boto3 standard retrier config (Default: 3)",
+        help="Set the maximum retries for the Boto3 standard retrier config, 0 disables retries (Default: 3)",
+    )
+    boto3_config_subparser.add_argument(
+        "--aws-connect-timeout",
+        nargs="?",
+        default=None,
+        type=validate_timeout,
+        help="Seconds to wait to establish a connection (TCP, proxy tunnel and TLS) to an AWS endpoint before retrying (Default: 10)",
+    )
+    boto3_config_subparser.add_argument(
+        "--aws-read-timeout",
+        nargs="?",
+        default=None,
+        type=validate_timeout,
+        help="Seconds to wait for a response from an AWS endpoint before retrying (Default: 60)",
     )
 
     # Scan Unused Services
@@ -190,6 +204,13 @@ def validate_session_duration(session_duration: int) -> int:
         return duration
 
 
+def validate_timeout(value: str) -> int:
+    """validate_timeout validates that the input is a whole number of seconds greater than zero"""
+    if not value.isdecimal() or int(value) == 0:
+        raise ArgumentTypeError(f"{value} is not a positive integer")
+    return int(value)
+
+
 def validate_role_session_name(session_name) -> str:
     """
     Validates that the role session name is valid.

prowler/providers/aws/lib/session/aws_set_up_session.py

@@ -42,6 +42,8 @@ def __init__(
         aws_session_token: Optional[str] = None,
         retries_max_attempts: int = 3,
         regions: set = set(),
+        connect_timeout: Optional[int] = None,
+        read_timeout: Optional[int] = None,
     ) -> None:
         """
         The constructor for the AwsSetUpSession class.
@@ -58,6 +60,8 @@ def __init__(
         - aws_session_token: The AWS session token, optional.
         - retries_max_attempts: The maximum number of retries for the AWS client.
         - regions: A set of regions to audit.
+        - connect_timeout: Seconds to wait to establish a connection to an AWS endpoint.
+        - read_timeout: Seconds to wait for a response from an AWS endpoint.
 
         Returns:
 
@@ -73,7 +77,9 @@ def __init__(
             aws_access_key_id=aws_access_key_id,
             aws_secret_access_key=aws_secret_access_key,
         )
-        session_config = AwsProvider.set_session_config(retries_max_attempts)
+        session_config = AwsProvider.set_session_config(
+            retries_max_attempts, connect_timeout, read_timeout
+        )
         aws_session = AwsProvider.setup_session(
             mfa=mfa,
             profile=profile,