200-python.mdc

---
title: Python Code Review & Enhancement Guidelines
description: Opinionated, performance- and security-minded Python rules for generation and review.
priority: 200
alwaysApply: false
files:
  include:
    - "**/*.py"
    - "pyproject.toml"
    - "uv.lock"
    - "requirements*.txt"
    - ".pre-commit-config.yaml"
---

## Guiding Principle

Apply features only when they add clarity, correctness, performance, or security. Prefer simple, intentional solutions (DRY, KISS, YAGNI, Fail Fast).

## The Zen of Python (`import this`)

The canonical aphorisms by Tim Peters (PEP 20). When in doubt about which Python idiom to choose, re-read these. Quote verbatim; do not paraphrase in code reviews.

```text
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
```

Operative readings (how to apply each in review):

- **Beautiful is better than ugly.** Refactor for clarity even when the ugly version "works".
- **Explicit is better than implicit.** Type hints, named arguments for booleans, no magic numbers, no implicit string-to-int conversion.
- **Simple is better than complex.** Reach for stdlib before a framework. Reach for a function before a class.
- **Complex is better than complicated.** When complexity is intrinsic, structure it (clear names, small functions); don't tangle it.
- **Flat is better than nested.** Early returns, guard clauses, comprehensions over nested loops where readable.
- **Sparse is better than dense.** Whitespace and blank lines are tools. One-liners that need a comment to explain are not Pythonic.
- **Readability counts.** Code is read more often than written. Optimize for the reader.
- **Special cases aren't special enough to break the rules.** No "just this once" pattern violations. Refactor the rule or live with it.
- **Although practicality beats purity.** Ship the working answer; idealize in the next PR.
- **Errors should never pass silently.** No bare `except:`; no `try: ... except: pass`. Raise specific exceptions.
- **Unless explicitly silenced.** When you do want to swallow, document why and catch the *specific* exception.
- **In the face of ambiguity, refuse the temptation to guess.** Ask. Test. Read the spec. Don't ship a heuristic dressed as logic.
- **There should be one-- and preferably only one --obvious way to do it.** Prefer `dict.get(k)` over `dict[k] if k in dict else None`. Use the canonical idiom, not the cleverest one.
- **Although that way may not be obvious at first unless you're Dutch.** A wink at Guido. Translation: when the obvious way isn't obvious to you yet, it's because you haven't read enough Python.
- **Now is better than never.** Ship the boring correct version; don't wait for perfection.
- **Although never is often better than *right* now.** Don't merge under pressure. Bad code now costs more than no code now.
- **If the implementation is hard to explain, it's a bad idea.** If reviewers can't follow it, neither will the on-call.
- **If the implementation is easy to explain, it may be a good idea.** Easy to explain is necessary, not sufficient. Confirm it's also correct.
- **Namespaces are one honking great idea -- let's do more of those!** Modules over flat top-level namespaces. Classes when state warrants. Avoid `from foo import *`.

### Examples

```python
# BAD: Unpythonic - dense, nested, unclear
def process(data):
    return [x*2 for x in [y for y in data if y>0] if x*2<100]

# GOOD: Pythonic - flat, explicit, readable
def process(data: list[int]) -> list[int]:
    """Process positive numbers, doubling values under 100."""
    positive = [x for x in data if x > 0]
    doubled = [x * 2 for x in positive]
    return [x for x in doubled if x < 100]

# BAD: Unpythonic - implicit, two ways to do one thing
def get_value(d, k):
    if k in d:
        return d[k]
    return None

# GOOD: Pythonic - one obvious way
def get_value(d: dict[str, int], k: str) -> int | None:
    """Get value from dictionary, returning None if missing."""
    return d.get(k)

# BAD: Unpythonic - errors pass silently
def divide(a, b):
    try:
        return a / b
    except Exception:
        pass

# GOOD: Pythonic - errors handled explicitly; specific exception
def divide(a: float, b: float) -> float:
    """Divide a by b. Raises ValueError if b is zero."""
    if b == 0:
        raise ValueError("Division by zero")
    return a / b
```

> Run `python -c "import this"` in any Python REPL to see the canonical text.

## Non-negotiables

> [!IMPORTANT]
> These are the hard floors. Every other section in this rule is guidance; the items below are **reject-in-review**.

### NN-1: Python ≥ 3.14 for new applications, services, and Lambda functions

Any new Python application, service, scheduled job, CLI tool, or AWS Lambda function ships with `requires-python = ">=3.14"`. This is non-negotiable for the same reason Module Workers syntax is non-negotiable for new Cloudflare Workers (`401-cloudflare-workers.mdc` NN-1): consistent runtime semantics, access to current type-system features (built-in generics, `|` union syntax, template strings, deferred annotations, free-threading), and the cost of supporting older runtimes that the rest of the codebase doesn't.

**Exception - libraries with broad audiences:**

Libraries published to PyPI or shipped to external customers MAY target a lower floor when there is a stated compatibility commitment. The acceptable lower floor is **Python 3.11** (the oldest CPython version with active security support per [PEP 387 deprecation policy](https://peps.python.org/pep-0387/) as of this rule's last review). The PR description MUST include:

- The audience justification (e.g., "library targets all CPython versions with active security support per PEP 387")
- The features being foregone by the lower floor (no template strings, no deferred annotations, no free-threading, no built-in `type` statement)
- A planned floor-bump date (e.g., "raise to 3.14 when 3.11 reaches EOL")

**Reject in review:**

- New application / service / Lambda with `requires-python = ">=3.11"` (or lower) and no library-audience justification
- New code targeting Python 3.10 or below for any reason (3.10 reaches EOL in October 2026; 3.11 is the absolute floor)
- `pyproject.toml` without `requires-python` at all (must be explicit)
- New code using `from typing import List, Dict, Optional, Tuple, Union` instead of `list[X]`, `dict[K, V]`, `X | None`, `tuple[X, Y]`, `X | Y` (the 3.14+ built-in generics + `|` union syntax are the only acceptable forms; see § Type Hints below)
- `Dockerfile` based on `python:3.13`, `python:3.12`, `python:3.11`, or older for a new app / service / Lambda
- CI workflow using `python-version: '3.13'`, `'3.12'`, `'3.11'`, or older for a new app / service / Lambda
- `runtime = "python3.13"` (or older) in Lambda / serverless configuration for a new function

### NN-2: Leading underscores mean non-public API

Do not prefix functions, methods, classes, variables, modules, or packages with `_` unless they are intentionally non-public implementation details. A leading underscore is a contract with readers and tools: "this is internal; don't import or call it from outside this module/package."

Use public names for public behavior:

```python
# GOOD: public API, public name
def validate_order(order: Order) -> None:
    ...


# GOOD: internal helper, module-private name
def _normalize_order_id(raw_order_id: str) -> str:
    ...
```

**Reject in review:**

- `_process_data()`, `_validate_input()`, or `_build_payload()` called directly from other modules
- Public classes or modules named `_Client`, `_Service`, `_helpers`, or similar without being intentionally internal
- Double-underscore methods (`__method`) unless name-mangling is intentionally required to avoid subclass collisions
- Invented dunder names such as `__process__` or `__validate__` in application code
- Leading underscore added "because it looks cleaner" or because the function is merely small

If a helper starts internal but becomes reused across modules, promote it to a public name or move it behind a clearer public API. Do not leak private-looking names into callers.

## 0) AI Assistant Guidelines

When providing code assistance, follow these guardrails:

- **Avoid Over-Engineering**: Do not recommend `boto3` client caching, `async/await`, `threading`, `multiprocessing`, or other advanced concurrency patterns unless the user explicitly requests them or performance bottlenecks are evident.
- **Keep It Simple**: Prefer straightforward solutions using standard library features over complex architectures.
- **Testing Guidance**: Suggest appropriate tests or improvements to existing test coverage when reviewing code.
- **Explain Trade-offs**: When proposing optimizations or refactoring, clearly explain the benefits and costs (complexity, maintenance, performance).
- **Respect Context**: Analyze the provided code and suggest improvements that align with its scope and purpose - don't transform a 20-line script into a 200-line framework.

## 1) Standards

- **Shebang:** `#!/usr/bin/env -S uv run` (required for new scripts). Use `uv` for modern Python tooling.
- **Python Version:** ≥ 3.14.
- **Formatting:** 4-space indents; soft wrap at 120 chars (adhere to PEP 8, PEP 257, PEP 484).
- **Linting:** Run `black`, `ruff`, `isort`, `mypy`, and `pylint`. Aim for a score of ≥9.5 on `pylint`.
- **Security Scanning:** Run `bandit` to detect common security issues in Python code.
- **String Formatting:** Use double quotes `"` unless the string contains double quotes (use `'` then). `black` handles this.
- **Imports:** Grouped: stdlib / third-party / local. Sorted alphabetically within groups. Separate groups with blank lines. Use `isort` to automatically sort imports.
- **Documentation:**
  - Google-style docstrings for all functions, classes, methods.
  - Keep docstrings accurate: when changing behavior or signatures, update docstrings (and examples, if present) in the same change.
  - Module docstring immediately below shebang (leave space between shebang and docstring).
  - Include workflow with numbered steps if applicable.
  - Add usage syntax guide with command-line examples (not needed for Lambda).
  - Inline comments for complex logic.
- **Type Hints:** Strict typing required. Use Python 3.14 built-in types:
  - `Dict` -> `dict`
  - `List` -> `list`
  - `Tuple` -> `tuple`
  - `Set` -> `set`
  - `Union[X, Y]` -> `X | Y`
  - `Optional[X]` -> `X | None`
- **Type Checking:** Use `ty` (recommended), `mypy --strict`, or `pyright`.
  - **ty** (recommended): Astral's Rust-based type checker, 10-60x faster than mypy with excellent diagnostics. Install: `uv tool install ty@latest`, run: `ty check .`
  - **mypy**: Stable, widely adopted. Run with `--strict` for full coverage.
  - **pyright**: Microsoft's type checker, good VS Code integration.
  - Use `NewType`, `TypedDict`, or `TypeAlias` for specificity.
- **Dependencies:** Use `uv` for package management. Define dependencies in `pyproject.toml` (not `requirements.txt`). Introduce third-party libraries only when needed - stick to stdlib otherwise.
- **Circular Imports:** Avoid by restructuring (move shared logic to separate module).
- **Local Libraries:** Use `aws_utils.py`, `cloudflare_utils.py`, `utils.py` for common tasks to avoid duplication.

## 1a) Code Quality Tools & Commands

**Required Tools:** All Python projects must run these tools before committing.

### Formatting & Linting Commands

**Format code (auto-fix):**

```bash
# Format with black
black .

# Sort imports with isort
isort .

# Run both together
black . && isort .
```

**Check formatting (CI/pre-commit):**

```bash
# Check without modifying files
black --check .
isort --check-only .

# Run both checks
black --check . && isort --check-only .
```

**Lint with ruff:**

```bash
# Check for issues
ruff check .

# Auto-fix issues
ruff check --fix .

# Check specific file
ruff check path/to/file.py
```

**Type checking with mypy:**

```bash
# Strict type checking (recommended)
mypy --strict .

# Type check specific file
mypy --strict path/to/file.py

# Alternative: Use ty (faster, recommended)
ty check .
```

**Lint with pylint:**

```bash
# Run pylint on file/directory
pylint path/to/file.py

# Run with specific score threshold (fail if below 9.5)
pylint --fail-under=9.5 path/to/file.py

# Run on entire project
pylint --fail-under=9.5 .

# Generate report
pylint --output-format=text path/to/file.py > pylint_report.txt
```

### Complete Quality Check Workflow

**Before committing, run:**

```bash
# 1. Format code
black .
isort .

# 2. Lint and auto-fix
ruff check --fix .

# 3. Type check
mypy --strict .
# OR (faster)
ty check .

# 4. Pylint (aim for ≥9.5)
pylint --fail-under=9.5 .

# 5. Security scan
bandit -r . -ll
```

**Quick check (CI/pre-commit):**

```bash
# Check-only mode (no modifications)
black --check . && \
isort --check-only . && \
ruff check . && \
mypy --strict . && \
pylint --fail-under=9.5 . && \
bandit -r . -ll
```

### Configuration Files

**pyproject.toml (Recommended):**

```toml
[tool.black]
line-length = 120
target-version = ['py314']

[tool.isort]
profile = "black"
line_length = 120

[tool.ruff]
line-length = 120
target-version = "py314"

[tool.ruff.lint]
select = ["E", "F", "I", "N", "W", "UP"]
ignore = []

[tool.mypy]
python_version = "3.14"
strict = true
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

[tool.pylint.messages_control]
disable = ["C0103", "C0111"]  # Disable specific checks if needed

[tool.pylint.format]
max-line-length = 120
```

**pylintrc (Optional, for project-specific config):**

```ini
[FORMAT]
max-line-length=120

[MESSAGES CONTROL]
disable=
    C0103,  # invalid-name (if using non-standard naming)
    C0111,  # missing-docstring (if docstrings not required)

[REPORTS]
output-format=text
score=yes
```

### Pylint Score Targets

**Score interpretation:**

- **10.0** - Perfect (rare, may require disabling some checks)
- **9.5-9.9** - Excellent (target for production code)
- **9.0-9.4** - Good (acceptable, but aim higher)
- **<9.0** - Needs improvement

**Common issues that lower score:**

- Missing docstrings
- Long lines (>120 chars)
- Too many arguments (>5)
- Too many local variables (>15)
- Cyclomatic complexity (>10)

**Improving pylint score:**

```bash
# Run pylint to see issues
pylint path/to/file.py

# Fix issues incrementally:
# 1. Add missing docstrings
# 2. Break up complex functions
# 3. Reduce function arguments
# 4. Fix naming conventions
# 5. Add type hints
```

### Pre-commit Integration

**Add to `.pre-commit-config.yaml`:**

```yaml
repos:
  - repo: https://github.com/psf/black
    rev: 24.1.1
    hooks:
      - id: black
        args: [--line-length=120]

  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: [--profile=black, --line-length=120]

  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.1.8
    hooks:
      - id: ruff
        args: [--fix, --exit-non-zero-on-fix]

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.8.0
    hooks:
      - id: mypy
        args: [--strict]
        additional_dependencies: [types-all]

  - repo: local
    hooks:
      - id: pylint
        name: pylint
        entry: pylint
        language: system
        args: [--fail-under=9.5]
        types: [python]
```

### CI/CD Integration

**GitHub Actions example:**

```yaml
name: Code Quality

on: [push, pull_request]

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.14'

      - name: Install dependencies
        run: |
          uv sync

      - name: Format check
        run: |
          black --check .
          isort --check-only .

      - name: Lint
        run: |
          ruff check .

      - name: Type check
        run: |
          mypy --strict .

      - name: Pylint
        run: |
          pylint --fail-under=9.5 .
```

## 1a) Package Management (uv)

**Prefer `uv` for new projects** - Fast, modern Python package installer and resolver.

**Installation:**

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

**Project Setup:**

```bash
uv init my-project
uv sync
```

**Benefits:** 10-100x faster, deterministic resolution, drop-in pip replacement.

## 1b) Import Sorting (isort)

**Use `isort` to automatically sort imports** - Ensures consistent import ordering across projects.

**Installation:**

```bash
uv add --dev isort
# or
pip install isort
```

**Configuration (pyproject.toml):**

```toml
[tool.isort]
profile = "black"  # Compatible with black formatting
line_length = 120
multi_line_output = 3  # Vertical hanging indent
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true
skip_glob = ["*/migrations/*", "*/venv/*", "*/.venv/*"]

# Import sections: stdlib, third-party, local
sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"]
known_first_party = ["my_package"]
```

**Usage:**

```bash
# Check import order
isort --check-only .

# Auto-fix import order
isort .

# Check specific file
isort --check-only src/main.py

# Auto-fix specific file
isort src/main.py
```

**Integration with pre-commit:**

```yaml
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/PyCQA/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: ["--profile", "black"]
```

**Import order example:**

```python
# Standard library
import logging
import os
from datetime import datetime, timezone
from typing import Any

# Third-party
import boto3
import requests
from pydantic import BaseModel

# Local
from utils import setup_logger
from models import User
```

## 2) Code Structure & Readability

- **Function Order:** Order functions by call hierarchy. Helper functions first, `main()` last.
- **Entry Point:** Always use `if __name__ == "__main__":` to call the main function.
- **Small Functions:** Break code into small, reusable functions with meaningful verb-based names.
- **Defensive Programming:** Validate inputs early (fail-fast). Use specific checks (`if x is None`) over broad `try/except`.
- **Meaningful Naming:** Use `fetch_user_details` (not `get`) or `order_total` (not `ot`).
- **No Mutable Defaults:** Avoid `def foo(bar=[])`. Use `def foo(bar: list[str] | None = None): bar = bar or []`.

## 3) Modern Python & Advanced Features

### Core Language Features

- **f-strings:** Use `f"User: {user_name}"` for formatting. Python 3.14: `f"{text!r}"` for debugging.
- **Exception Handling:** Use `try-except-else-finally` for complex flows.
- **Assertions:** Use `assert stock >= 0, "Stock cannot be negative"` for development - not production.
- **Comprehensions:** Use `[price * 2 for price in prices]` where clear - avoid if it reduces readability.
- **Walrus Operator (`:=`):** Use `if (count := len(items)) > 5:` for inline assignments - don't force it.
- **match-case:** Simplify complex conditionals:

  ```python
  def handle_response(code: int) -> str:
      match code:
          case 200: return "Success"
          case 400 | 404: return "Client Error"
          case _: return "Unknown"
  ```

- **Literal Types:** Use `status: Literal["active", "inactive"]` for fixed values.
- **Type Safety:** Use `isinstance` and `issubclass` for safe type checks.

### Python 3.14 Features (Released Oct 2025)

**Template String Literals (PEP 750):**

Safe custom string processing with t-strings for parameterized queries, HTML escaping, and custom formatting:

```python
# SQL query with safe parameterization
query = t"SELECT * FROM users WHERE id = {user_id}"

# HTML generation with auto-escaping
html = t"<div>{user_input}</div>"

# Custom formatting
config = t"server={host}:{port}"
```

**Deferred Annotation Evaluation (PEP 649):**

Annotations are no longer evaluated eagerly, improving startup performance for heavily-annotated code:

```python
# Annotations stored in __annotate__ function
# Evaluated only when inspect.get_annotations() is called
def process(data: ComplexType) -> Result:
    """Annotations evaluated lazily, not at import time."""
    ...
```

**Free-Threading Support (PEP 779):**

True parallelism without the GIL (experimental, use with `python3.14t`):

```python
# Enable with: python3.14t (free-threaded build)
# Performance penalty reduced to 5-10% for single-threaded code
import concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(cpu_intensive_task, items)
```

**datetime Improvements:**

Direct parsing of date and time strings:

```python
from datetime import date, time

# Python 3.14+
d = date.fromisoformat("2025-10-07")
t = time.fromisoformat("14:30:00")
```

**UUID7 and UUID8 Support:**

Modern UUID versions with better properties:

```python
import uuid

# UUID7: Time-ordered, sortable (recommended for databases)
id = uuid.uuid7()

# UUID8: Custom format
id = uuid.uuid8(bytes16)
```

**Enhanced REPL:**

- Colorized syntax highlighting as you type
- Improved error messages with keyword suggestions
- Better interactive debugging experience

### Functional Programming

- **`functools.wraps`:** Preserve metadata in decorators.
- **`functools.cache`:** Use `@functools.cache` for costly, repeated calls.
- **`functools.partial`:** Pre-specify function arguments: `partial(multiply, 2)`.
- **`functools.singledispatch`:** Polymorphic functions based on type.
- **`functools.total_ordering`:** Reduce comparison method boilerplate.
- **map/filter/reduce:** Use where readable - prefer comprehensions otherwise.

### Data Structures

- **dataclasses:** Use for data-centric classes:

  ```python
  from dataclasses import dataclass, field
  @dataclass
  class Product:
      product_id: int
      tags: list[str] = field(default_factory=list)
  ```

- **`collections`:** Use `Counter`, `deque`, `defaultdict`, `namedtuple` where appropriate.
- **Tuple Unpacking:** Use `x, y = get_coordinates()` for cleaner code.
- **`enumerate`:** Use `for idx, value in enumerate(values):` for indexed loops.

### Standard Library Modules to Prefer

**Principle:** Prefer standard library modules over third-party alternatives when they meet your needs. This reduces dependencies, improves portability, and leverages well-tested, maintained code.

**Common Standard Library Modules:**

#### File & Path Operations

**`pathlib` (Preferred over `os.path`):**

```python
# BAD: os.path (old-style, string-based)
import os
file_path = os.path.join("data", "users", "file.txt")
if os.path.exists(file_path):
    with open(file_path) as f:
        content = f.read()

# GOOD: pathlib (object-oriented, cross-platform)
from pathlib import Path

file_path = Path("data") / "users" / "file.txt"
if file_path.exists():
    content = file_path.read_text()

# Advanced pathlib features
config_dir = Path.home() / ".config" / "app"
config_dir.mkdir(parents=True, exist_ok=True)
config_file = config_dir / "config.toml"

# Iterate over directory
for py_file in Path("src").rglob("*.py"):
    print(py_file.stem)  # filename without extension
```

**`shutil` (File operations):**

```python
import shutil
from pathlib import Path

# Copy files/directories
shutil.copy("source.txt", "dest.txt")
shutil.copytree("src_dir", "dest_dir", dirs_exist_ok=True)

# Move/rename
shutil.move("old.txt", "new.txt")

# Archive operations
shutil.make_archive("backup", "zip", "data_dir")
shutil.unpack_archive("backup.zip", "extract_dir")

# Disk usage
total, used, free = shutil.disk_usage("/")
```

#### Functional Programming

**`functools` (Function utilities):**

```python
from functools import partial, cache, wraps, singledispatch

# partial: Pre-specify function arguments
def multiply(x: int, y: int) -> int:
    return x * y

double = partial(multiply, 2)  # Equivalent to: lambda y: multiply(2, y)
result = double(5)  # 10

# cache: Memoization for expensive functions
@cache
def expensive_computation(n: int) -> int:
    # Expensive operation
    return sum(i**2 for i in range(n))

# wraps: Preserve function metadata in decorators
def timing_decorator(func):
    @wraps(func)  # Preserves __name__, __doc__, etc.
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.time() - start:.2f}s")
        return result
    return wrapper

# singledispatch: Polymorphic functions based on type
@singledispatch
def process(data):
    raise NotImplementedError(f"Cannot process {type(data)}")

@process.register
def _(data: dict):
    return {k: str(v) for k, v in data.items()}

@process.register
def _(data: list):
    return [str(item) for item in data]
```

#### Data Structures & Algorithms

**`heapq` (Priority queues):**

```python
import heapq

# Min-heap (default)
heap = []
heapq.heappush(heap, 5)
heapq.heappush(heap, 2)
heapq.heappush(heap, 8)
smallest = heapq.heappop(heap)  # 2

# Max-heap (negate values)
max_heap = []
heapq.heappush(max_heap, -10)
heapq.heappush(max_heap, -5)
largest = -heapq.heappop(max_heap)  # 10

# Priority queue pattern
tasks = []
heapq.heappush(tasks, (1, "low priority task"))
heapq.heappush(tasks, (0, "high priority task"))
priority, task = heapq.heappop(tasks)  # Gets high priority first
```

**`graphlib` (Topological sorting, Python 3.9+):**

```python
from graphlib import TopologicalSorter

# Build dependency graph
graph = {
    "task_a": {"task_b", "task_c"},
    "task_b": {"task_d"},
    "task_c": {"task_d"},
    "task_d": set(),
}

# Get execution order
ts = TopologicalSorter(graph)
execution_order = list(ts.static_order())  # ['task_d', 'task_b', 'task_c', 'task_a']
```

**`secrets` (Cryptographically secure random, preferred over `random` for security):**

```python
import secrets
import string

# Generate secure random token
token = secrets.token_urlsafe(32)  # URL-safe token

# Generate secure random string
alphabet = string.ascii_letters + string.digits
password = ''.join(secrets.choice(alphabet) for _ in range(16))

# Compare securely (constant-time)
if secrets.compare_digest(user_input, expected_token):
    # Secure comparison (prevents timing attacks)
    pass

# BAD: Using random for security
import random
token = ''.join(random.choice(alphabet) for _ in range(16))  # Not cryptographically secure!
```

#### Configuration & Data Formats

**`tomllib` (TOML parsing, Python 3.11+):**

```python
import tomllib
from pathlib import Path

# Read TOML file
with open("config.toml", "rb") as f:  # Note: binary mode required
    config = tomllib.load(f)

# Access nested values
database_url = config["database"]["url"]
api_key = config["api"]["key"]

# BAD: Using third-party library
# import tomli  # Don't need this if Python 3.11+
```

**`json` (JSON parsing):**

```python
import json
from pathlib import Path

# Read JSON
with Path("data.json").open() as f:
    data = json.load(f)

# Write JSON (with formatting)
with Path("output.json").open("w") as f:
    json.dump(data, f, indent=2, sort_keys=True)

# Parse JSON string
config = json.loads('{"key": "value"}')
```

**`configparser` (INI-style config files):**

```python
import configparser

config = configparser.ConfigParser()
config.read("config.ini")

database_host = config.get("database", "host")
debug = config.getboolean("app", "debug", fallback=False)
```

#### Utilities

**`itertools` (Iterator tools):**

```python
from itertools import chain, cycle, islice, pairwise, batched, groupby, combinations, permutations, product, zip_longest

# Chain iterables
combined = list(chain([1, 2], [3, 4]))  # [1, 2, 3, 4]

# Pairwise iteration (Python 3.10+)
for prev, curr in pairwise([1, 2, 3, 4]):
    print(f"{prev} -> {curr}")  # 1 -> 2, 2 -> 3, 3 -> 4

# Batched (Python 3.12+)
for batch in batched(range(10), 3):
    print(list(batch))  # [0, 1, 2], [3, 4, 5], [6, 7, 8], [9]

# Cycle through values
colors = cycle(["red", "green", "blue"])
next(colors)  # "red"
next(colors)  # "green"

# Group consecutive elements
data = [1, 1, 2, 2, 2, 3, 3]
for key, group in groupby(data):
    print(f"{key}: {list(group)}")  # 1: [1, 1], 2: [2, 2, 2], 3: [3, 3]

# Combinations and permutations
items = ['a', 'b', 'c']
list(combinations(items, 2))  # [('a', 'b'), ('a', 'c'), ('b', 'c')]
list(permutations(items, 2))  # [('a', 'b'), ('a', 'c'), ('b', 'a'), ...]

# Cartesian product
list(product([1, 2], ['a', 'b']))  # [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]

# Zip longest (fill missing with default)
list(zip_longest([1, 2], ['a'], fillvalue='-'))  # [(1, 'a'), (2, '-')]

# Slice iterator
for item in islice(range(100), 10, 20):  # Items 10-19
    print(item)
```

**`textwrap` (Text formatting):**

```python
import textwrap

# Wrap text to specified width
text = "This is a long line that needs to be wrapped to fit within 50 characters."
wrapped = textwrap.wrap(text, width=50)
# ['This is a long line that needs to be', 'wrapped to fit within 50 characters.']

# Fill text (wrap and join)
filled = textwrap.fill(text, width=50)
# Multi-line string with line breaks

# Dedent (remove leading whitespace) - great for SQL, templates
sql = textwrap.dedent("""
    SELECT *
    FROM users
    WHERE active = true
""").strip()
# Removes common leading whitespace

# Indent text
indented = textwrap.indent("Line 1\nLine 2", prefix="  ")
# "  Line 1\n  Line 2"

# Shorten text with ellipsis
short = textwrap.shorten("This is a very long string", width=20, placeholder="...")
# "This is a very..."
```

**`contextlib` (Context managers):**

```python
from contextlib import contextmanager, suppress, redirect_stdout
import io

# Custom context manager
@contextmanager
def temporary_file():
    path = Path("/tmp/temp.txt")
    path.write_text("data")
    try:
        yield path
    finally:
        path.unlink()

# Suppress exceptions
with suppress(FileNotFoundError):
    Path("missing.txt").unlink()

# Redirect output
f = io.StringIO()
with redirect_stdout(f):
    print("This goes to StringIO")
output = f.getvalue()
```

**`dataclasses` (Data containers):**

```python
from dataclasses import dataclass, field, asdict

@dataclass
class User:
    name: str
    email: str
    age: int = 0
    tags: list[str] = field(default_factory=list)

user = User("John", "john@acme.com", 30)
user_dict = asdict(user)  # Convert to dict
```

**`enum` (Enumerations):**

```python
from enum import Enum, IntEnum, Flag

class Status(Enum):
    PENDING = "pending"
    ACTIVE = "active"
    INACTIVE = "inactive"

status = Status.ACTIVE
if status == Status.ACTIVE:
    print("Active!")

# IntEnum for integer values
class Priority(IntEnum):
    LOW = 1
    MEDIUM = 2
    HIGH = 3

# Flags for bitwise operations
class Permissions(Flag):
    READ = 1
    WRITE = 2
    EXECUTE = 4

perms = Permissions.READ | Permissions.WRITE
```

**`typing` (Type hints, Python 3.14+):**

```python
from typing import Annotated, Literal, Self, TypeAlias

# Type aliases
UserId: TypeAlias = int

# Literal types
Mode = Literal["dev", "staging", "prod"]

def deploy(env: Mode) -> None:
    ...

# Self type (Python 3.11+)
class Builder:
    def with_name(self, name: str) -> Self:
        self.name = name
        return self

# Annotated (for metadata)
from typing import Annotated
Port = Annotated[int, "Port number between 1 and 65535"]
```

#### When to Use Third-Party Libraries

**Use third-party libraries when:**

- Standard library doesn't provide the functionality
- Third-party library offers significant performance improvements
- Standard library solution is too complex for the use case
- You need features not available in stdlib

**Examples:**

- **`requests`** over `urllib` - Better API, connection pooling, sessions
- **`pydantic`** over `dataclasses` - Advanced validation, JSON schema
- **`click`** over `argparse` - Better CLI framework (if building complex CLIs)
- **`httpx`** over `urllib` - Async HTTP, HTTP/2 support

**Prefer stdlib when:**

- `pathlib` over `path.py`
- `tomllib` over `tomli` (Python 3.11+)
- `secrets` over `random` for security
- `json` over `ujson` (unless performance critical)
- `dataclasses` over `attrs` (for simple cases)
- `heapq` over `heapq` wrapper libraries
- `graphlib` over `networkx` (for simple topological sorting)

### Advanced OOP Features

- **Descriptors:** Control attribute access:

```python
class PositiveNumber:
    """Descriptor that ensures positive numbers."""
    def __init__(self, name: str):
        self.name = name

    def __get__(self, obj: object, objtype: type | None = None) -> int:
        if obj is None:
            return self
        return obj.__dict__.get(self.name, 0)

    def __set__(self, obj: object, value: int) -> None:
        if value < 0:
            raise ValueError(f"{self.name} must be positive")
        obj.__dict__[self.name] = value

class Product:
    price = PositiveNumber("price")
    quantity = PositiveNumber("quantity")

    def __init__(self, price: int, quantity: int):
        self.price = price  # Validated by descriptor
        self.quantity = quantity

# Usage
product = Product(price=100, quantity=5)
product.price = -10  # Raises ValueError
```

- **Metaclasses:** Advanced class customization (use sparingly):

```python
class SingletonMeta(type):
    """Metaclass that creates singleton instances."""
    _instances: dict[type, object] = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super().__call__(*args, **kwargs)
        return cls._instances[cls]

class Database(metaclass=SingletonMeta):
    def __init__(self):
        print("Database initialized")

db1 = Database()  # Prints "Database initialized"
db2 = Database()  # No print - returns same instance
assert db1 is db2  # True
```

- **`__slots__`:** Memory optimization for classes with many instances:

```python
class Point:
    __slots__ = ('x', 'y')  # Prevents __dict__ creation

    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y

# Benefits: Less memory, faster attribute access
# Trade-off: No dynamic attributes, no weak references
```

### When to Use Classes vs Functions vs Dataclasses

**Decision Framework:** Choose the simplest tool that solves the problem. Avoid premature abstraction.

**Use Functions When:**

- Operation is stateless (pure input → output transformation)
- No need to maintain state between calls
- Simple utilities, helpers, or scripts
- Single-purpose operations

```python
# GOOD: Stateless operation - use function
def calculate_total(items: list[dict[str, float]]) -> float:
    """Calculate total price from items."""
    return sum(item['price'] * item['quantity'] for item in items)

# BAD: Unnecessary class for stateless operation
class TotalCalculator:
    def calculate(self, items: list[dict[str, float]]) -> float:
        return sum(item['price'] * item['quantity'] for item in items)
```

**Use Dataclasses When:**

- Need structured data with validation
- Want automatic `__init__`, `__repr__`, `__eq__` without boilerplate
- Creating DTOs, configs, or data containers
- Don't need complex behavior or inheritance

```python
# GOOD: Data container - use dataclass
from dataclasses import dataclass

@dataclass
class User:
    name: str
    email: str
    age: int

# BAD: Full class for simple data container
class User:
    def __init__(self, name: str, email: str, age: int):
        self.name = name
        self.email = email
        self.age = age

    def __repr__(self):
        return f"User(name={self.name!r}, email={self.email!r}, age={self.age})"

    def __eq__(self, other):
        if not isinstance(other, User):
            return False
        return (self.name, self.email, self.age) == (other.name, other.email, other.age)
```

**Use Classes When:**

- Need to maintain state between method calls
- Require polymorphism or inheritance
- Framework requires it (Django models, FastAPI dependencies)
- Rich behavior with mutable state
- Need encapsulation of complex logic

```python
# GOOD: Stateful object - use class
class ConnectionPool:
    """Manages database connections with state."""

    def __init__(self, max_connections: int = 10):
        self.max_connections = max_connections
        self._connections: list[Connection] = []
        self._active = 0

    def acquire(self) -> Connection:
        """Acquire connection from pool."""
        if self._active >= self.max_connections:
            raise PoolExhaustedError("No available connections")
        # ... connection logic
        self._active += 1
        return connection

    def release(self, conn: Connection) -> None:
        """Release connection back to pool."""
        self._active -= 1
        # ... cleanup logic

# BAD: Function trying to manage state (requires global or closure)
_connections = []
_active = 0

def acquire_connection():
    global _active
    # ... problematic global state
```

**Common Anti-Patterns:**

```python
# ANTI-PATTERN: Class with single method and no state
class EmailSender:
    def send(self, to: str, subject: str, body: str) -> None:
        # ... send email

# BETTER: Use function
def send_email(to: str, subject: str, body: str) -> None:
    # ... send email

# ANTI-PATTERN: Class for simple data container
class Config:
    def __init__(self, host: str, port: int):
        self.host = host
        self.port = port

# BETTER: Use dataclass
@dataclass
class Config:
    host: str
    port: int

# ANTI-PATTERN: Function trying to manage state
def process_items(items: list[str]) -> list[str]:
    processed = []  # State maintained across calls?
    # ... unclear if this should be stateful

# BETTER: Use class if state needs to persist
class ItemProcessor:
    def __init__(self):
        self.processed: list[str] = []

    def process(self, items: list[str]) -> list[str]:
        # ... process with state
```

**Decision Tree:**

```
Do you need to maintain state between calls?
├─ Yes → Use Class
└─ No → Is it structured data with validation?
    ├─ Yes → Use Dataclass
    └─ No → Use Function
```

**Framework Requirements:**

- **Django:** Models must be classes (inheritance from `models.Model`)
- **FastAPI:** Dependencies can be functions or classes (use classes for stateful dependencies)
- **Pydantic:** Models are classes (but can use `@dataclass` with Pydantic v2)
- **SQLAlchemy:** Models must be classes

### Object-Oriented Features

- **`@property`:** Use for validation or computed properties:

  ```python
  @property
  def total(self) -> float:
      return self._total
  @total.setter
  def total(self, value: float) -> None:
      if value < 0: raise ValueError("Total cannot be negative")
      self._total = value
  ```

- **Abstract Base Classes (ABC):** Define interfaces:

  ```python
  from abc import ABC, abstractmethod
  class DataWriter(ABC):
      @abstractmethod
      def write(self, data: str) -> None: ...
  ```

- **Protocols:** Use for structural typing (duck typing).
- **MetaClasses:** Use for advanced class customization (e.g., Singleton pattern).
- **`@classmethod`/`@staticmethod`:** Use for utility/factory methods.
- **Magic Methods:** Implement `__str__`, `__repr__`, `__eq__`, etc., for Pythonic behavior.

### Context Managers

- **Built-in:** Use `with` for file handling, network connections.
- **Custom:** Create with `contextlib`:

  ```python
  from contextlib import contextmanager
  @contextmanager
  def database_connection(db_name: str):
      logger.info(f"Connecting to {db_name}")
      yield
      logger.info("Disconnected")
  ```

### Text Processing

- **`dedent` from textwrap:** Remove leading whitespace for SQL queries, templates:

  ```python
  from textwrap import dedent
  query = dedent("""
      SELECT * FROM users
      WHERE age > 18
  """)
  ```

- **`re` module:** Validate inputs to prevent injection attacks.

## 4) Logging & Observability

- **Setup:** Use a custom logger setup function at the start of scripts.
- **Pattern:**

  ```python
  import logging
  import time

  def setup_custom_logger(name: str) -> logging.Logger:
      """Create and configure a logger with UTC timestamps."""
      logger = logging.getLogger(name)
      logger.setLevel(logging.DEBUG)

      class UTCFormatter(logging.Formatter):
          converter = time.gmtime

      formatter = UTCFormatter(
          "%(asctime)s - %(levelname)s - %(message)s",
          datefmt="%c %Z"
      )

      if not logger.handlers:
          handler = logging.StreamHandler()
          handler.setFormatter(formatter)
          logger.addHandler(handler)
      return logger

  logger = setup_custom_logger(__name__)
  ```

- **Structured Logging:** Use `loguru` or JSON format for observability pipelines where needed.
- **Error Logging:** Include `exc_info=True` for tracebacks: `logger.error("Error", exc_info=True)`.

## 5) Security & Validation

- **Security Scanning:** Use `bandit` to detect common security issues:

  ```bash
  # Install
  uv add --dev bandit
  # or
  pip install bandit

  # Scan codebase
  bandit -r .                    # Recursive scan
  bandit -r . -f json -o report.json  # JSON output
  bandit -r . -ll                # Low severity and above
  bandit -r . -ii                # Medium confidence and above

  # Exclude paths
  bandit -r . -x tests/,venv/

  # Use config file (.bandit)
  bandit -r . -c .bandit
  ```

- **Input Validation:** Sanitize early with `re.match(r"^\w+$", user_input)`. For complex patterns, see [Making Regex Readable](#making-regex-readable) below.
- **SQL Injection:** Use parameterized queries: `cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))`.
- **Shell Sanitization:** Use `shlex.quote(user_input)` for shell commands.
- **Pydantic:** Use for structured data validation:

  ```python
  from pydantic import BaseModel, field_validator
  class User(BaseModel):
      name: str
      age: int
      @field_validator("age")
      @classmethod
      def valid_age(cls, v: int) -> int:
          if v < 0 or v > 150: raise ValueError("Invalid age")
          return v
  ```

- **Avoid Broad Exceptions:** Catch specific exceptions (`ValueError`, `KeyError`) instead of bare `except Exception`.

### Making Regex Readable

Regex patterns can become cryptic and hard to maintain. Use these techniques to make them self-documenting:

**Use Variables and Comments:**

```python
# BAD: Cryptic regex
if re.match(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', email):
    ...

# GOOD: Self-documenting with variables
local_part = r'[a-zA-Z0-9._%+-]+'
domain_part = r'[a-zA-Z0-9.-]+'
tld = r'[a-zA-Z]{2,}'
email_pattern = rf'{local_part}@{domain_part}\.{tld}'

if re.match(email_pattern, email):
    ...

# EXCELLENT: Use VERBOSE flag for complex patterns
email_pattern = re.compile(r"""
    [a-zA-Z0-9._%+-]+      # Local part (username)
    @                       # @ symbol
    [a-zA-Z0-9.-]+         # Domain name
    \.                     # Literal dot
    [a-zA-Z]{2,}           # Top-level domain (2+ letters)
""", re.VERBOSE)

if email_pattern.match(email):
    ...
```

**Break Complex Patterns into Functions:**

```python
# BAD: One complex regex
if re.match(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', email):
    ...

# GOOD: Validate components separately
def is_valid_email(email: str) -> bool:
    """Validate email format."""
    if not email or '@' not in email:
        return False

    local, domain = email.rsplit('@', 1)

    # Validate local part
    if not re.match(r'^[a-zA-Z0-9._%+-]+$', local):
        return False

    # Validate domain
    if not re.match(r'^[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$', domain):
        return False

    return True
```

**When Regex Gets Complex:**

For very complex validation patterns (>50 characters, multiple nested groups), consider:

1. **Break into smaller functions** - Validate components separately
2. **Use a validation library** - `pydantic`, `cerberus`, or `marshmallow` for structured validation
3. **Pregex** (optional) - For teams that frequently work with complex regex and need maximum readability

**Preference order:**

1. Simple stdlib `re` with variables/comments
2. `re.VERBOSE` flag for complex patterns
3. Validation library for structured data
4. Pregex only if regex is unavoidable and extremely complex

```python
# Example: Complex pattern broken down
# Pattern: Match URLs with optional protocol, domain, path, query, fragment

# Option 1: Use re.VERBOSE (preferred)
url_pattern = re.compile(r"""
    ^
    (?:https?://)?          # Optional protocol (http:// or https://)
    (?:www\.)?              # Optional www.
    [a-zA-Z0-9-]+          # Domain name
    (?:\.[a-zA-Z0-9-]+)*   # Optional subdomains
    \.[a-zA-Z]{2,}         # Top-level domain
    (?:/[^\s?#]*)?         # Optional path
    (?:\?[^\s#]*)?         # Optional query string
    (?:#[^\s]*)?           # Optional fragment
    $
""", re.VERBOSE)

# Option 2: Use validation library (better for structured data)
from pydantic import BaseModel, HttpUrl, field_validator

class Link(BaseModel):
    url: HttpUrl  # Automatic URL validation

    @field_validator('url', mode='before')
    @classmethod
    def validate_url(cls, v: str) -> str:
        if not v.startswith(('http://', 'https://')):
            return f'https://{v}'
        return v
```

## 6) Error Handling & Resilience

- **Domain Boundary Exception Handling:** Always re-raise exceptions with context at system boundaries. Never swallow errors by returning empty/default data.

  ```python
  #  BAD: Masks failure and returns empty data
  def load_database() -> dict:
      try:
          # Load database data...
          return data
      except Exception as e:
          logger.error(f"Error loading YAML database: {str(e)}")
          return {"books": [], "library": []}  # BAD! Silent failure

  #  GOOD: Re-raises with context, fails fast
  def load_database() -> dict:
      try:
          # Load database data...
          return data
      except Exception as e:
          logger.error(f"Error loading YAML database: {str(e)}")
          raise RuntimeError(f"Failed to load database: {str(e)}") from e
  ```

- **Exception Groups:** For batch operations:

  ```python
  from exceptiongroup import ExceptionGroup
  errors = []
  for item in items:
      try:
          process(item)
      except Exception as e:
          errors.append(e)
  if errors:
      raise ExceptionGroup("Batch failed", errors)
  ```

- **Retry Libraries:** Use `tenacity`, `backoff`, or `retrying` for transient failures:

  ```python
  from tenacity import retry, stop_after_attempt, wait_exponential
  @retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
  def fetch_data(): ...
  ```

- **Custom Exceptions:** Define `class FileProcessingError(Exception): pass` for clarity.
- **Warnings Module:** Use for non-critical issues: `warnings.warn("Deprecated", DeprecationWarning)`.

## 7) Performance Optimization

**Golden Rule:** Profile first, optimize second. Measure before optimizing.

- **Profiling:** Use `cProfile`, `line_profiler`, or `memory_profiler` to identify bottlenecks.
- **Generators:** Use `yield` for large datasets to save memory.
- **Session Reuse:** Use `requests.Session()` for multiple API calls.
- **Concurrency:**
  - **`asyncio`:** For I/O-bound tasks.
  - **`threading`:** For I/O-bound with blocking libraries.
  - **`multiprocessing`:** For CPU-bound tasks.
- **Speed Boosts:** Use `numba` (NumPy-focused), `cython` (C extensions), or `codon` (`@codon.jit` for general Python) for performance-critical code. Codon has fewer restrictions than Numba but isn't a full CPython drop-in. Profile first.
- **Efficient Lookups:** Use `set` for O(1) lookups, `heapq` for priority queues, `bisect` for sorted lists.
- **Parallel Computing:** Use `ray` or `dask` for distributed workloads if needed.
- **Complexity Analysis:** Avoid O(n²) loops - refactor with better data structures.

### Performance Optimization Examples

**String Concatenation:**

```python
# BAD: O(n²) - creates new string each iteration
result = ""
for item in items:
    result += item

# GOOD: O(n) - uses list and join
result = "".join(items)

# GOOD: For complex formatting
parts = [f"{key}={value}" for key, value in data.items()]
result = "&".join(parts)
```

**List Comprehensions vs Loops:**

```python
# BAD: Slower, more verbose
result = []
for x in range(1000):
    if x % 2 == 0:
        result.append(x * 2)

# GOOD: Faster, more Pythonic
result = [x * 2 for x in range(1000) if x % 2 == 0]

# For complex logic, generator expressions save memory
result = (x * 2 for x in range(1000000) if x % 2 == 0)  # Lazy evaluation
```

**Dictionary Lookups:**

```python
# BAD: Multiple lookups
if key in data:
    value = data[key]
    process(value)

# GOOD: Single lookup with default
value = data.get(key)
if value is not None:
    process(value)

# GOOD: Using try/except for rare KeyError (faster when key exists)
try:
    value = data[key]
    process(value)
except KeyError:
    pass
```

**Memory-Efficient Iteration:**

```python
# BAD: Loads all into memory
with open("large_file.txt") as f:
    lines = f.readlines()  # Loads entire file
    for line in lines:
        process(line)

# GOOD: Iterates line by line
with open("large_file.txt") as f:
    for line in f:  # Generator, memory efficient
        process(line)

# GOOD: Using generators for large datasets
def process_large_dataset(items):
    """Process items one at a time."""
    for item in items:
        yield transform(item)  # Lazy evaluation

# Process without loading all into memory
for result in process_large_dataset(huge_list):
    save(result)
```

**Caching Expensive Operations:**

```python
from functools import lru_cache

# GOOD: Cache function results
@lru_cache(maxsize=128)
def expensive_computation(n: int) -> int:
    """Expensive computation cached by input."""
    # Complex calculation
    return sum(i * i for i in range(n))

# GOOD: Custom caching with TTL
from functools import wraps
import time

def cached_with_ttl(ttl_seconds: int):
    """Cache decorator with time-to-live."""
    cache: dict[tuple, tuple[float, any]] = {}

    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            key = (args, tuple(sorted(kwargs.items())))
            now = time.time()

            if key in cache:
                cached_time, cached_value = cache[key]
                if now - cached_time < ttl_seconds:
                    return cached_value

            result = func(*args, **kwargs)
            cache[key] = (now, result)
            return result
        return wrapper
    return decorator

@cached_with_ttl(ttl_seconds=300)
def fetch_data(url: str) -> dict:
    return requests.get(url).json()
```

**Pre-allocation:**

```python
# BAD: Grows list dynamically
result = []
for i in range(1000):
    result.append(i * 2)

# GOOD: Pre-allocate when size is known
result = [0] * 1000
for i in range(1000):
    result[i] = i * 2

# GOOD: Pre-allocate with list comprehension (fastest)
result = [i * 2 for i in range(1000)]
```

**Using Built-in Functions:**

```python
# BAD: Manual implementation
total = 0
for item in items:
    total += item

# GOOD: Built-in is faster
total = sum(items)

# BAD: Manual max
max_val = items[0]
for item in items[1:]:
    if item > max_val:
        max_val = item

# GOOD: Built-in
max_val = max(items)
```

## 7a) Concurrency Patterns & Decision Guide

**When to Use Each Concurrency Model:**

### asyncio (I/O-Bound, Async Libraries)

Use `asyncio` when:

- Working with async libraries (`aiohttp`, `asyncpg`, `aioredis`)
- Many concurrent I/O operations (API calls, database queries)
- Network-bound workloads
- Web servers/frameworks (FastAPI, aiohttp)

```python
import asyncio
import aiohttp

async def fetch_urls(urls: list[str]) -> list[dict]:
    """Fetch multiple URLs concurrently using asyncio."""
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_one(session, url) for url in urls]
        return await asyncio.gather(*tasks)

async def fetch_one(session: aiohttp.ClientSession, url: str) -> dict:
    async with session.get(url) as response:
        return await response.json()
```

### threading (I/O-Bound, Blocking Libraries)

Use `threading` when:

- Using blocking libraries (`requests`, `psycopg2`, synchronous boto3)
- I/O-bound operations with blocking calls
- Need to run blocking code concurrently
- GUI applications (keep UI responsive)

```python
import threading
from queue import Queue
from typing import Callable

def process_with_threads(items: list[str], worker: Callable[[str], None], num_threads: int = 4) -> None:
    """Process items using thread pool."""
    queue: Queue[str] = Queue()

    def worker_thread():
        while True:
            item = queue.get()
            if item is None:  # Sentinel
                break
            worker(item)
            queue.task_done()

    # Start worker threads
    threads = []
    for _ in range(num_threads):
        t = threading.Thread(target=worker_thread)
        t.start()
        threads.append(t)

    # Add items to queue
    for item in items:
        queue.put(item)

    # Wait for completion
    queue.join()

    # Stop threads
    for _ in range(num_threads):
        queue.put(None)
    for t in threads:
        t.join()
```

### multiprocessing (CPU-Bound)

Use `multiprocessing` when:

- CPU-intensive computations (image processing, data analysis)
- Need to bypass GIL (Global Interpreter Lock)
- Parallel number crunching
- Independent CPU-bound tasks

```python
from multiprocessing import Pool, cpu_count
from functools import partial

def cpu_intensive_task(data: list[int]) -> int:
    """CPU-bound computation."""
    return sum(x * x for x in data)

def process_parallel(items: list[list[int]], num_processes: int | None = None) -> list[int]:
    """Process CPU-bound tasks in parallel."""
    if num_processes is None:
        num_processes = cpu_count()

    with Pool(processes=num_processes) as pool:
        results = pool.map(cpu_intensive_task, items)
    return results
```

### Race Condition Detection

**Use `threading` with care - Python's GIL doesn't prevent all race conditions:**

```python
import threading
from typing import Counter

# BAD: Race condition with shared state
counter = {"value": 0}

def increment_unsafe():
    counter["value"] += 1  # Not atomic! Race condition possible

# GOOD: Use locks for shared mutable state
counter_safe = {"value": 0}
lock = threading.Lock()

def increment_safe():
    with lock:
        counter_safe["value"] += 1

# GOOD: Use thread-safe data structures when possible
from collections import Counter
counter_threadsafe = Counter()  # Thread-safe for most operations

# GOOD: Use queue.Queue for thread-safe communication
from queue import Queue
work_queue: Queue[str] = Queue()
result_queue: Queue[dict] = Queue()
```

**Testing for Race Conditions:**

```python
import threading
import time

def test_race_condition():
    """Test for race conditions with multiple threads."""
    shared = {"count": 0}
    lock = threading.Lock()

    def increment(use_lock: bool):
        for _ in range(1000):
            if use_lock:
                with lock:
                    shared["count"] += 1
            else:
                shared["count"] += 1  # Race condition!

    # Test without lock (should show race condition)
    threads = [threading.Thread(target=lambda: increment(False)) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    print(f"Without lock: {shared['count']}")  # May be < 10000

    # Reset and test with lock
    shared["count"] = 0
    threads = [threading.Thread(target=lambda: increment(True)) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    print(f"With lock: {shared['count']}")  # Should be exactly 10000
```

**Best Practices:**

- **Prefer `asyncio`** for I/O-bound async workloads
- **Use `threading`** only when blocking libraries are required
- **Use `multiprocessing`** for CPU-bound tasks
- **Always use locks** when sharing mutable state between threads
- **Prefer immutable data** or thread-safe collections (`queue.Queue`, `collections.deque` with locks)
- **Avoid shared mutable state** - use message passing (queues) instead
- **Profile first** - concurrency adds complexity; measure before optimizing

## 8) AWS & Boto3 Best Practices

### Client Configuration

- **Global Initialization:** In Lambda, create `boto3` clients globally (outside handler) to reuse across invocations:

  ```python
  import boto3
  from botocore.config import Config

  boto_config = Config(
      retries={"max_attempts": 5, "mode": "standard"},
      connect_timeout=5,
      read_timeout=30
  )

  s3_client = boto3.client("s3", config=boto_config, region_name="us-east-1")
  ```

- **Region Handling:** Retrieve from `AWS_REGION` env var, default to `us-east-1`, allow override via parameter.
- **Client Factory:** Create reusable factory function:

  ```python
  def create_boto3_client(service_name: str, region_name: str, **kwargs) -> boto3.client:
      try:
          config = Config(retries={"max_attempts": 5, "mode": "standard"})
          client = boto3.client(service_name, region_name=region_name, config=config, **kwargs)
          logger.info(f"Created boto3 client for {service_name} in {region_name}")
          return client
      except (BotoCoreError, ClientError) as e:
          logger.error(f"Failed to create boto3 client: {e}")
          raise
  ```

### Error Handling

- **Specific Exceptions:** Catch `botocore.exceptions.ClientError`, `ResourceNotFoundError`, etc.
- **Error Codes:** Check `e.response['Error']['Code']` for specific AWS errors.

### Pagination & Waiters

- **Paginators:** Use for large result sets:

  ```python
  paginator = s3_client.get_paginator("list_objects_v2")
  for page in paginator.paginate(Bucket="my-bucket"):
      for obj in page.get("Contents", []):
          process(obj)
  ```

- **NextToken:** Handle manually if paginator unavailable.
- **Waiters:** Pause until resource is ready:

  ```python
  waiter = ec2_client.get_waiter("instance_running")
  waiter.wait(InstanceIds=["i-123456"])
  ```

### Lambda Patterns

**Global Scope Pattern (Recommended):**

```python
import boto3
import os
from botocore.config import Config

#  CORRECT: Create clients in global scope (outside handler)
boto_config = Config(
    retries={"max_attempts": 10, "mode": "adaptive"},
    connect_timeout=5,
    read_timeout=30,
)

region = os.environ["AWS_REGION"]
s3_client = boto3.client("s3", config=boto_config, region_name=region)
dynamodb_client = boto3.client("dynamodb", config=boto_config, region_name=region)

def lambda_handler(event: dict[str, Any], context: Any) -> dict[str, Any]:
    """Lambda handler reuses clients from global scope."""
    response = s3_client.list_buckets()
    return {"statusCode": 200, "body": "Success"}
```

**Anti-Patterns:**

```python
#  WRONG: Creates new client on every invocation (cold start penalty)
def lambda_handler(event: dict[str, Any], context: Any) -> dict[str, Any]:
    s3_client = boto3.client("s3")  # Bad: Recreated every time
    return {"statusCode": 200}

#  WRONG: Ties client lifecycle to class instance
class S3Manager:
    def __init__(self):
        self.s3_client = boto3.client("s3")  # Bad
```

**Cold Start Optimization:**

```python
#  Lazy initialization for rarely-used clients
_sns_client: Any = None

def get_sns_client() -> Any:
    """Lazy-load SNS client only when needed."""
    global _sns_client
    if _sns_client is None:
        _sns_client = boto3.client("sns")
    return _sns_client

#  Always-needed clients in global scope
s3_client = boto3.client("s3")  # Frequently used

def lambda_handler(event: dict[str, Any], context: Any) -> dict[str, Any]:
    s3_client.list_buckets()  # Always available

    if event.get("notify"):
        sns = get_sns_client()  # Loaded only if needed
        sns.publish(TopicArn="...", Message="...")

    return {"statusCode": 200}
```

**AWS Lambda Powertools (Observability):**

```python
from aws_lambda_powertools import Logger, Tracer, Metrics
from aws_lambda_powertools.metrics import MetricUnit
from typing import Any

logger = Logger()
tracer = Tracer()
metrics = Metrics()

@tracer.capture_lambda_handler
@logger.inject_lambda_context(log_event=True)
@metrics.log_metrics(capture_cold_start_metric=True)
def lambda_handler(event: dict[str, Any], context: Any) -> dict[str, Any]:
    """Lambda handler with full observability."""
    logger.info("Processing request", extra={"request_id": event.get("requestId")})
    metrics.add_metric(name="ItemsProcessed", unit=MetricUnit.Count, value=10)
    result = process_items(event["items"])
    return {"statusCode": 200, "body": result}

@tracer.capture_method
def process_items(items: list[str]) -> str:
    """Process items with automatic tracing."""
    logger.info(f"Processing {len(items)} items")
    return "success"
```

**Lambda Best Practices:**

- **Handler Placement:** Place `lambda_handler` at end of file.
- **Clients as Parameters:** Pass clients to functions (don't recreate inside functions).
- **Correlation ID:** Use `context.aws_request_id` for tracing.
- **Handler Signature:** Use `def lambda_handler(event: dict[str, Any], _) -> dict[str, Any]:` when context unused.
- **Dry-Run Mode:** Add flag to preview actions: `if event.get("dry_run"): logger.info(f"Would perform: {action}")`.

### Region Validation

```python
def validate_region(region: str) -> str:
    """Validate AWS region against allowed list."""
    allowed_regions = [
        "us-east-1", "us-west-2", "us-east-2", "ca-central-1",
        "eu-west-1", "eu-west-2", "eu-central-1", "eu-north-1",
        "ap-southeast-1", "ap-southeast-2"
    ]
    if region not in allowed_regions:
        raise ValueError(f"Invalid region: {region}. Allowed: {allowed_regions}")
    return region
```

## 8a) Troubleshooting & Debugging

### Debugging Tools

**Python Debugger (pdb):**

```python
import pdb

def complex_function(data: list[dict]) -> dict:
    """Debug with pdb breakpoint."""
    pdb.set_trace()  # Breakpoint - execution pauses here
    result = process_data(data)
    return result

# Or use breakpoint() built-in (Python 3.7+)
def complex_function(data: list[dict]) -> dict:
    breakpoint()  # Modern way - equivalent to pdb.set_trace()
    result = process_data(data)
    return result
```

**pdb Commands:**

- `n` (next): Execute next line
- `s` (step): Step into function calls
- `c` (continue): Continue execution
- `l` (list): Show current code context
- `p <variable>`: Print variable value
- `pp <variable>`: Pretty-print variable
- `u` (up): Move up stack frame
- `d` (down): Move down stack frame
- `q` (quit): Quit debugger

**IPython Debugger (ipdb):**

```python
import ipdb

def debug_function():
    ipdb.set_trace()  # Enhanced pdb with IPython features
    # Better tab completion, syntax highlighting
```

**VS Code / PyCharm Debugging:**

- Set breakpoints in IDE
- Use "Debug" configuration
- Inspect variables, call stack, watch expressions
- Step through code interactively

### Profiling & Performance Analysis

**cProfile:**

```python
import cProfile
import pstats

def profile_function():
    """Profile function execution."""
    profiler = cProfile.Profile()
    profiler.enable()

    # Code to profile
    result = expensive_operation()

    profiler.disable()
    stats = pstats.Stats(profiler)
    stats.sort_stats('cumulative')
    stats.print_stats(20)  # Top 20 functions

    return result

# Command-line profiling
# python -m cProfile -o profile.stats script.py
# python -m pstats profile.stats
```

**line_profiler:**

```python
# Install: pip install line_profiler
# Run: kernprof -l -v script.py

@profile  # Decorator added by line_profiler
def slow_function():
    result = []
    for i in range(1000):
        result.append(i * 2)  # Line-by-line timing
    return result
```

**memory_profiler:**

```python
# Install: pip install memory_profiler
# Run: python -m memory_profiler script.py

from memory_profiler import profile

@profile
def memory_intensive():
    data = [i for i in range(1000000)]  # Memory usage per line
    return sum(data)
```

### Logging for Debugging

**Structured Debug Logging:**

```python
import logging
import sys

# Configure debug logger
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('debug.log'),
        logging.StreamHandler(sys.stdout)
    ]
)

logger = logging.getLogger(__name__)

def debug_function(data: dict):
    logger.debug(f"Function called with data: {data}")
    logger.debug(f"Data keys: {list(data.keys())}")

    try:
        result = process(data)
        logger.debug(f"Result: {result}")
        return result
    except Exception as e:
        logger.exception("Error in debug_function", exc_info=True)
        raise
```

**Conditional Debug Output:**

```python
import os
import logging

DEBUG = os.getenv("DEBUG", "0") == "1"

def debug_print(message: str):
    """Print only when DEBUG is enabled."""
    if DEBUG:
        print(f"[DEBUG] {message}")

# Or use logging levels
logger.setLevel(logging.DEBUG if DEBUG else logging.INFO)
```

### Common Issues & Solutions

**Import Errors:**

```python
# BAD: Circular import
# module_a.py imports module_b.py
# module_b.py imports module_a.py

# GOOD: Restructure to avoid circular imports
# - Move shared code to separate module
# - Use late imports (inside functions)
# - Use dependency injection

# Late import pattern
def process_data():
    from module_b import helper_function  # Import inside function
    return helper_function()
```

**Memory Leaks:**

```python
# BAD: Keeping references to large objects
cache = {}
def process_large_data(data):
    cache["data"] = data  # Keeps data in memory forever

# GOOD: Use weak references or clear cache
from weakref import WeakValueDictionary

cache = WeakValueDictionary()  # Allows garbage collection

# Or use size limits
from collections import OrderedDict

class LRUCache:
    def __init__(self, max_size: int = 100):
        self.cache: OrderedDict = OrderedDict()
        self.max_size = max_size

    def get(self, key: str):
        if key in self.cache:
            self.cache.move_to_end(key)
            return self.cache[key]
        return None

    def set(self, key: str, value: any):
        if key in self.cache:
            self.cache.move_to_end(key)
        elif len(self.cache) >= self.max_size:
            self.cache.popitem(last=False)  # Remove oldest
        self.cache[key] = value
```

**Performance Bottlenecks:**

```python
# BAD: Inefficient string concatenation
result = ""
for item in items:
    result += item  # Creates new string each time

# GOOD: Use join or list comprehension
result = "".join(items)
# Or for complex cases:
result = "".join(str(item) for item in items)

# BAD: Repeated lookups
for key in keys:
    if key in large_dict:  # O(n) lookup each time
        process(large_dict[key])

# GOOD: Single pass
for key in keys:
    value = large_dict.get(key)  # O(1) lookup
    if value is not None:
        process(value)
```

**Exception Debugging:**

```python
import traceback
import sys

def debug_exception():
    try:
        risky_operation()
    except Exception as e:
        # Print full traceback
        traceback.print_exc()

        # Get traceback as string
        tb_str = traceback.format_exc()
        logger.error(f"Exception occurred: {tb_str}")

        # Get exception info tuple
        exc_type, exc_value, exc_tb = sys.exc_info()
        logger.error(f"Type: {exc_type}, Value: {exc_value}")

        raise  # Re-raise to preserve stack trace
```

### Testing for Race Conditions

**Thread Safety Testing:**

```python
import threading
import time
from queue import Queue

def test_thread_safety(func, num_threads: int = 10, iterations: int = 1000):
    """Test function for thread safety."""
    results: Queue[dict] = Queue()
    errors: Queue[Exception] = Queue()

    def worker():
        try:
            for _ in range(iterations):
                result = func()
                results.put(result)
        except Exception as e:
            errors.put(e)

    threads = [threading.Thread(target=worker) for _ in range(num_threads)]

    start = time.time()
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    elapsed = time.time() - start

    # Check for errors
    error_list = []
    while not errors.empty():
        error_list.append(errors.get())

    return {
        "results_count": results.qsize(),
        "errors": error_list,
        "elapsed_time": elapsed,
        "thread_safe": len(error_list) == 0
    }
```

## 9) Testing & Quality

### Test Structure & Organization

- **pytest:** Write tests for reliability:

  ```python
  def test_parse():
      assert parse_input("123") == 123
  ```

- **Mocking:** Use `unittest.mock.patch` or `pytest-mock` for external dependencies:

```python
from unittest.mock import Mock, patch, MagicMock
import pytest

# Patching functions
@patch('module.expensive_function')
def test_with_mock(mock_func):
    mock_func.return_value = 42
    result = process_data()
    assert result == 42
    mock_func.assert_called_once()

# Patching objects
def test_with_mock_object():
    mock_client = Mock()
    mock_client.get.return_value = {"status": "ok"}
    result = use_client(mock_client)
    assert result["status"] == "ok"

# Using pytest-mock (cleaner)
def test_with_pytest_mock(mocker):
    mock_func = mocker.patch('module.expensive_function')
    mock_func.return_value = 42
    result = process_data()
    assert result == 42
```

- **Parameterized Tests:** Use `@pytest.mark.parametrize` for multiple inputs:

```python
@pytest.mark.parametrize("input,expected", [
    ("123", 123),
    ("-456", -456),
    ("0", 0),
])
def test_parse_input(input: str, expected: int):
    assert parse_input(input) == expected

# Multiple parameters
@pytest.mark.parametrize("a,b,expected", [
    (1, 2, 3),
    (0, 0, 0),
    (-1, 1, 0),
])
def test_add(a: int, b: int, expected: int):
    assert add(a, b) == expected
```

- **Fixtures:** Share test setup and teardown:

```python
import pytest
import tempfile
import os

@pytest.fixture
def temp_file():
    """Create temporary file for testing."""
    with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
        f.write("test data")
        temp_path = f.name

    yield temp_path

    # Cleanup
    os.unlink(temp_path)

@pytest.fixture
def sample_data():
    """Provide sample data for tests."""
    return {"users": [{"id": 1, "name": "Alice"}]}

def test_process_file(temp_file, sample_data):
    result = process_file(temp_file)
    assert result == sample_data
```

- **Test Classes:** Organize related tests:

```python
class TestUserService:
    """Test suite for UserService."""

    @pytest.fixture
    def service(self):
        return UserService()

    def test_create_user(self, service):
        user = service.create_user("Alice")
        assert user.name == "Alice"

    def test_get_user_not_found(self, service):
        with pytest.raises(UserNotFoundError):
            service.get_user("nonexistent")
```

- **Async Testing:** Test async functions:

```python
import pytest

@pytest.mark.asyncio
async def test_async_function():
    result = await fetch_data("https://api.example.com")
    assert result["status"] == "ok"

@pytest.mark.asyncio
async def test_async_with_fixture():
    async with aiohttp.ClientSession() as session:
        result = await fetch_with_session(session)
        assert result is not None
```

- **Test-Driven Development (TDD):** Write tests before implementation where appropriate.
- **Type Checking:** Run `ty check .` (fast, recommended) or `mypy --strict` (stable) to catch type errors.
- **tox:** Use for testing automation across Python versions and environments:

  ```bash
  # Install
  uv add --dev tox
  # or
  pip install tox

  # Run all test environments
  tox

  # Run specific environment
  tox -e py312
  tox -e lint
  tox -e format

  # List available environments
  tox -l

  # Run in parallel
  tox -p auto
  ```

  **Configuration (tox.ini or pyproject.toml):**

  ```ini
  # tox.ini
  [tox]
  envlist = py312,py313,lint,format
  isolated_build = true

  [testenv]
  deps = pytest
  commands = pytest

  [testenv:lint]
  deps = ruff,black,isort,bandit
  commands =
      ruff check .
      black --check .
      isort --check-only .
      bandit -r . -ll

  [testenv:format]
  deps = black,isort
  commands =
      black .
      isort .
  ```

  **Benefits:**
  - Test across multiple Python versions automatically
  - Run linting, formatting, and security checks in isolated environments
  - CI/CD integration (runs same commands locally and in CI)
  - Parallel execution for faster feedback

## 10) Data Handling & Integration

- **pandas:** Use for tabular data analysis: `df[df["sales"] > 100]`.
- **polars:** Rust-based DataFrame library, 10-100x faster than pandas for large datasets. Prefer for ETL/data pipelines.
- **duckdb:** In-process SQL analytics - query CSV/Parquet/DataFrames without loading into memory.
- **SQLAlchemy:** Use for database ORM if needed.
- **cerberus:** Alternative to Pydantic for simple validation.
- **jinja/jinjasql:** Use for templating (HTML, SQL, etc.).
- **httpx:** Modern async HTTP client (alternative to `requests`).
- **Message Brokers:** Integrate RabbitMQ/Kafka if event-driven architecture needed.

## 11) Code Structure & Architecture

- **SOLID Principles:**
  - **Single Responsibility:** One task per class/function.
  - **Open-Closed:** Extend, don't modify.
  - **Liskov Substitution:** Subclasses interchangeable.
  - **Interface Segregation:** Small, focused interfaces.
  - **Dependency Inversion:** Depend on abstractions, not concretions.
- **Separation of Concerns:** Use service/repository/handler pattern for layers.
- **Dependency Injection:** Pass dependencies as parameters for testability.
- **Plugin Architecture:** Enable extensions via registration:

  ```python
  plugins: dict[str, Callable] = {}
  def register_plugin(name: str, func: Callable) -> None:
      plugins[name] = func
  ```

- **Configuration Management:** Use `configparser`, `os.environ`, or `.env` files.
- **Feature Flags:** Implement toggles for gradual rollout.

## 11a) Design Patterns

Design patterns provide reusable solutions to common problems. Use them where they add clarity and maintainability - avoid forcing patterns where simple code suffices.

### Decorator Pattern (Advanced)

**Function Decorators:**

```python
from functools import wraps
from typing import Callable, TypeVar, ParamSpec

P = ParamSpec('P')
T = TypeVar('T')

def retry(max_attempts: int = 3):
    """Decorator that retries function on failure."""
    def decorator(func: Callable[P, T]) -> Callable[P, T]:
        @wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
            last_exception = None
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    last_exception = e
                    if attempt < max_attempts:
                        logger.warning(f"Attempt {attempt} failed: {e}, retrying...")
                    else:
                        logger.error(f"All {max_attempts} attempts failed")
            raise last_exception
        return wrapper
    return decorator

@retry(max_attempts=5)
def fetch_data(url: str) -> dict:
    response = requests.get(url)
    response.raise_for_status()
    return response.json()
```

**Class Decorators:**

```python
def add_logging(cls):
    """Class decorator that adds logging to all methods."""
    for name, method in vars(cls).items():
        if callable(method) and not name.startswith('_'):
            setattr(cls, name, log_method(method))
    return cls

def log_method(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        logger.info(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
        result = func(self, *args, **kwargs)
        logger.info(f"{func.__name__} returned {result}")
        return result
    return wrapper

@add_logging
class DataProcessor:
    def process(self, data: list[int]) -> int:
        return sum(data)
```

**Property Decorators:**

```python
class CachedProperty:
    """Descriptor that caches property value."""
    def __init__(self, func: Callable[[object], T]):
        self.func = func
        self.attrname = None
        self.__doc__ = func.__doc__

    def __set_name__(self, owner: type, name: str):
        self.attrname = f"_{name}"

    def __get__(self, obj: object, objtype: type | None = None) -> T:
        if obj is None:
            return self
        if not hasattr(obj, self.attrname):
            setattr(obj, self.attrname, self.func(obj))
        return getattr(obj, self.attrname)

class ExpensiveComputation:
    @CachedProperty
    def result(self) -> int:
        """Expensive computation cached after first access."""
        print("Computing...")
        return sum(i * i for i in range(1000000))
```

### Factory Method Pattern

Creates objects without specifying their exact class. Useful when object creation logic is complex or varies based on input.

```python
from typing import Protocol

class DatabaseConnection(Protocol):
    """Protocol for database connections."""
    def execute(self, query: str) -> list[dict]: ...

class PostgresConnection:
    def execute(self, query: str) -> list[dict]:
        return [{"result": "postgres_data"}]

class MySQLConnection:
    def execute(self, query: str) -> list[dict]:
        return [{"result": "mysql_data"}]

class DatabaseFactory:
    """Factory for creating database connections."""
    @staticmethod
    def create_connection(db_type: str) -> DatabaseConnection:
        match db_type.lower():
            case "postgres": return PostgresConnection()
            case "mysql": return MySQLConnection()
            case _: raise ValueError(f"Unsupported: {db_type}")
```

### Singleton Pattern

Ensures a class has only one instance. Useful for managing shared resources like boto3 clients, database connections, or configuration objects.

```python
from typing import Any

class SingletonMeta(type):
    """Metaclass that creates a Singleton base class."""
    _instances: dict[type, object] = {}

    def __call__(cls, *args: Any, **kwargs: Any) -> object:
        if cls not in cls._instances:
            cls._instances[cls] = super().__call__(*args, **kwargs)
        return cls._instances[cls]

class AWSClientManager(metaclass=SingletonMeta):
    """Singleton manager for AWS boto3 clients."""
    def __init__(self) -> None:
        self._clients: dict[str, Any] = {}

    def get_client(self, service: str, region: str = "us-east-1") -> Any:
        key = f"{service}:{region}"
        if key not in self._clients:
            self._clients[key] = boto3.client(service, region_name=region)
        return self._clients[key]

# Usage in Lambda or application code
aws_clients = AWSClientManager()
s3_client = aws_clients.get_client("s3", "us-east-1")
```

### Strategy Pattern

Defines a family of interchangeable algorithms. Useful for multi-region AWS operations or varying behaviors based on environment.

```python
from typing import Protocol

class DeploymentStrategy(Protocol):
    def deploy(self, application: str) -> None: ...

class DevelopmentDeployment:
    def deploy(self, application: str) -> None:
        print(f"Deploying {application} to dev (single instance)")

class ProductionDeployment:
    def deploy(self, application: str) -> None:
        print(f"Deploying {application} to prod (multi-AZ, auto-scaling)")

class DeploymentContext:
    def __init__(self, strategy: DeploymentStrategy) -> None:
        self._strategy = strategy

    def execute_deployment(self, application: str) -> None:
        self._strategy.deploy(application)
```

## 12) CLI & User Experience

- **argparse:** Add CLI options:

  ```python
  parser = argparse.ArgumentParser()
  parser.add_argument("--dry-run", action="store_true")
  parser.add_argument("-v", "--verbose", action="count", default=0)
  ```

- **typer:** Modern CLI framework (alternative to argparse).
- **rich:** Rich text and progress bars:

  ```python
  from rich.console import Console
  console = Console()
  console.print("[bold green]Success![/bold green]")
  ```

- **tqdm:** Progress bars: `for item in tqdm(items): process(item)`.
- **colorama:** Colored output for diffs or status messages.

## 13) Deployment & Distribution

### Creating Installable Packages

**Basic Package Structure:**

```
my_utils/
 my_utils/
    __init__.py
    aws_helper.py
    validators.py
 tests/
    test_aws_helper.py
 README.md
 pyproject.toml
```

**Modern Approach: pyproject.toml (Recommended):**

```toml
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my-utils"
version = "1.0.0"
description = "Common utility functions for AWS and validation"
readme = "README.md"
authors = [{name = "Your Name", email = "you@acme.com"}]
requires-python = ">=3.14"
dependencies = [
    "boto3>=1.28.0",
    "pydantic>=2.0.0",
]

[project.optional-dependencies]
dev = ["pytest>=7.0", "black", "ruff"]

[tool.setuptools.packages.find]
where = ["."]
```

**Installation Commands (with uv):**

```bash
# Initialize new project
uv init my-project
cd my-project

# Add dependencies
uv add boto3 pydantic

# Add dev dependencies
uv add --dev pytest black ruff

# Sync dependencies (install from lock file)
uv sync

# Run script with uv
uv run python main.py

# Build distribution
uv build

# Publish to PyPI
uv publish
```

**Legacy pip commands (if uv unavailable):**

```bash
# Development mode (editable install)
pip install -e .

# Production install
pip install .

# With optional dependencies
pip install -e ".[dev]"
```

**Best Practices:**

- **Version Management**: Use semantic versioning (MAJOR.MINOR.PATCH)
- **Dependencies**: Pin major versions, allow minor/patch updates (`boto3>=1.28.0,<2.0.0`)
- **Entry Points**: Add CLI commands via `[project.scripts]` in `pyproject.toml`
- **Documentation**: Include comprehensive README with usage examples
- **Testing**: Include tests and configure with `pytest.ini` or `pyproject.toml`

### Docker & CI/CD

- **Docker:** Use multi-stage builds with `uv`:

  ```dockerfile
  FROM ghcr.io/astral-sh/uv:python3.14 AS builder
  WORKDIR /app
  COPY pyproject.toml uv.lock ./
  RUN uv sync --frozen --no-dev
  COPY . .

  FROM python:3.14-slim
  WORKDIR /app
  COPY --from=builder /app/.venv /app/.venv
  COPY --from=builder /app /app
  ENV PATH="/app/.venv/bin:$PATH"
  CMD ["python", "main.py"]
  ```

- **Executables:** Use `PyInstaller` or `cx_Freeze` for standalone apps.
- **CI/CD:** Automate with GitHub Actions, GitLab CI, etc.

## 14) Timestamps

Use UTC-aware timestamps:

```python
from datetime import datetime, timezone
timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
```

## 15) Dry-Run Mode

Simulate operations without executing:

```python
def update_records(records: dict, dry_run: bool = False) -> None:
    if dry_run:
        logger.info(f"Dry run: would update {records}")
        return
    # Actual update logic
```

## 16) Documentation Template

```python
#!/usr/bin/env -S uv run
"""
Module purpose and overview.

This script performs X by doing Y. It interacts with Z.

Workflow:
1. Load configuration
2. Validate inputs
3. Process data
4. Write results

Usage:
    python script.py --input data.json --output results.json
    python script.py --dry-run
"""

# Standard library
import logging

# Third-party
import requests

# Local
from utils import setup_custom_logger

def function_name(param: str) -> int:
    """
    Brief description of what the function does.

    Args:
        param (str): Description of parameter.

    Returns:
        int: Description of return value.

    Raises:
        ValueError: When param is invalid.
    """
    pass
```

## 17) Scaling Guidelines

- **Small Scripts (<50 lines):** Stick to basics - skip advanced features unless critical.
- **Medium Scripts (50-200 lines):** Add type hints, logging, small functions. Consider `argparse` or `Pydantic`.
- **Large Projects (>200 lines):** Use OOP, SOLID principles, plugins, deployment tools. Optimize with profiling/concurrency.

## 18) Example Script

```python
#!/usr/bin/env -S uv run
"""Process user data with logging, validation, and decorators.

This script demonstrates:
- Pydantic models with validation
- Structured logging with UTC timestamps
- Decorators for timing and retries
- CLI with --dry-run
- Error handling and input validation

Usage:
    python app.py --dry-run
"""

from __future__ import annotations

import argparse
import functools
import logging
import re
import sys
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Callable, Iterable, Iterator, Any

from pydantic import BaseModel, ValidationError, field_validator


# -----------------------------
# Logging setup
# -----------------------------
def setup_custom_logger(name: str) -> logging.Logger:
    """Create and configure a logger with UTC timestamps."""
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)

    class UTCFormatter(logging.Formatter):
        converter = time.gmtime

    formatter = UTCFormatter(
        "%(asctime)s - %(levelname)s - %(message)s",
        datefmt="%c %Z"
    )

    if not logger.handlers:
        handler = logging.StreamHandler(sys.stdout)
        handler.setFormatter(formatter)
        logger.addHandler(handler)
    return logger

logger = setup_custom_logger(__name__)


# -----------------------------
# Decorators
# -----------------------------
def timed(func: Callable) -> Callable:
    """Measure and log execution time."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            elapsed_ms = (time.perf_counter() - start) * 1000
            logger.info(f"Metric: timing_ms func={func.__name__} value={elapsed_ms:.2f}")
    return wrapper


def retry(max_attempts: int = 3, base_delay: float = 0.2, max_delay: float = 2.0) -> Callable:
    """Exponential backoff retry decorator."""
    def decorator(func: Callable) -> Callable:
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            attempt = 1
            delay = base_delay
            while True:
                try:
                    return func(*args, **kwargs)
                except Exception as exc:
                    if attempt >= max_attempts:
                        logger.error(f"Retry exhausted: {exc}", exc_info=True)
                        raise
                    logger.warning(f"Retry {attempt}/{max_attempts}: {exc}")
                    time.sleep(delay)
                    delay = min(delay * 2, max_delay)
                    attempt += 1
        return wrapper
    return decorator


# -----------------------------
# Data models
# -----------------------------
class User(BaseModel):
    """Validated user data model."""
    user_name: str
    user_age: int

    @field_validator("user_name")
    @classmethod
    def valid_name(cls, v: str) -> str:
        if not re.fullmatch(r"[A-Za-z0-9]+", v):
            raise ValueError("user_name must be alphanumeric")
        return v

    @field_validator("user_age")
    @classmethod
    def valid_age(cls, v: int) -> int:
        if v < 0 or v > 150:
            raise ValueError("user_age must be 0-150")
        return v


@dataclass(slots=True)
class ProcessResult:
    """Processing result with accepted and rejected users."""
    accepted: list[User]
    rejected: list[dict]


# -----------------------------
# Business logic
# -----------------------------
def _iter_users(raw: Iterable[dict]) -> Iterator[User]:
    """Yield validated users, logging validation errors."""
    for entry in raw:
        try:
            yield User(**entry)
        except ValidationError as e:
            logger.error(f"Validation error for {entry}: {e.errors()}")


@timed
def process_users(raw: Iterable[dict], *, dry_run: bool = False) -> ProcessResult:
    """Validate and process users."""
    accepted: list[User] = []
    rejected: list[dict] = []

    for user in _iter_users(raw):
        try:
            if dry_run:
                logger.info(f"Would process: {user}")
            else:
                logger.info(f"Processed: {user}")
            accepted.append(user)
        except Exception as exc:
            logger.error(f"Process error for {user}: {exc}", exc_info=True)
            rejected.append({"user": user.model_dump(), "error": str(exc)})

    return ProcessResult(accepted=accepted, rejected=rejected)


# -----------------------------
# CLI
# -----------------------------
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Process user data with validation and logging."
    )
    parser.add_argument(
        "--dry-run",
        action="store_true",
        help="Log actions without executing them."
    )
    return parser.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    """Main entry point."""
    args = _parse_args(argv)
    timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
    logger.info(f"Starting run at {timestamp}")

    # Example input
    raw_data: list[dict] = [
        {"user_name": "Alice123", "user_age": 25},
        {"user_name": "Bob!", "user_age": 30},  # Invalid name
        {"user_name": "Chad", "user_age": -5},  # Invalid age
    ]

    result = process_users(raw_data, dry_run=args.dry_run)
    logger.info(f"Summary: accepted={len(result.accepted)} rejected={len(result.rejected)}")

    return 0 if not result.rejected else 1


if __name__ == "__main__":
    sys.exit(main())
```

## 19) Quick Reference Checklist

Before finalizing code, verify:

- [ ] Shebang line present with `uv`
- [ ] Type hints on all functions/variables
- [ ] Google-style docstrings
- [ ] Imports grouped and sorted (`isort` passes)
- [ ] `if __name__ == "__main__":` guard
- [ ] Logging configured
- [ ] Input validation
- [ ] Error handling (specific exceptions)
- [ ] Security checks (SQL/shell sanitization)
- [ ] Tests written (if applicable)
- [ ] `black`, `isort`, and `ruff` pass
- [ ] `pylint` score ≥ 9.0
- [ ] `bandit` security scan passes (no high-severity issues)
- [ ] Dry-run mode (if applicable)
- [ ] No mutable default arguments
- [ ] No bare `except:` clauses
- [ ] Lambda: Clients in global scope (not in handler)
- [ ] Lambda: AWS Lambda Powertools configured (if applicable)

## 20) Comprehensive Technique Reference

Where appropriate, use these techniques:

**Control Flow & Error Handling:**

- `try-except-else-finally`, `match-case`, Custom exceptions, `exceptiongroup`, Assertions

**String & Data Formatting:**

- `f-strings` (with `!r` for debugging), Context managers, Comprehensions, Tuple unpacking

**Type Safety & Validation:**

- Type annotations (Python 3.14 built-ins), `Literal` types, `Pydantic` models, `dataclasses`, `protocols`, `ABC`

**Functions & Decorators:**

- `*args`/`**kwargs`, `@property`/`getter`/`setter`, Custom decorators with `functools.wraps`, `functools.cache`, `functools.partial`, `functools.singledispatch`, `functools.total_ordering`

**Object-Oriented Programming:**

- Classes with encapsulation, `dataclasses`, `MetaClasses`, `@classmethod`/`@staticmethod`, Magic methods, Inheritance/composition

**Data Structures & Collections:**

- `defaultdict`, `Counter`, `deque`, `enum`, Generators (`yield`), Deep/shallow copying

**Performance & Concurrency:**

- `itertools`, `asyncio`, `threading`, `multiprocessing`, `requests.Session()`

**CLI & Configuration:**

- `argparse`, `os.environ`, `configparser`, Dry-run mode

**AWS & Cloud:**

- `boto3` client configuration, Global scope for Lambda, Paginators/waiters, Error handling (`ClientError` codes), `aws_lambda_powertools`

**Design Patterns:**

- Factory method, Singleton, Strategy, Decorator

**Testing & Quality:**

- `pytest`, `unittest.mock`, Profiling (`cProfile`, `memory_profiler`)

**Deployment:**

- Package creation (`pyproject.toml`), Docker multi-stage builds, CI/CD automation

---