Skip to content

CLAUDE.md

Latest commit

 

History

History
243 lines (203 loc) · 12.6 KB

CLAUDE.md

File metadata and controls

243 lines (203 loc) · 12.6 KB
Raw

CLAUDE.md — Pipedream SDK (Python)

This is a Fern-generated Python SDK with substantial team customizations. Most files in src/pipedream/ are auto-regenerated by the Fern CLI from the Pipedream API spec; a small set of files in .fernignore are hand-maintained and must not be replaced by regen output. Generated and custom code coexist in the same trees.

The .fernignore contract

.fernignore is the only thing standing between custom code and a regen wiping it out. Treat it as a load-bearing file.

  • Never comment out entries before running a regen "to see what happens." That has happened, and it deleted the custom Pipedream wrapper, the entire workflows module, the proxy customizations, the license, and the comprehensive .gitignore. Recovery took a full restoration pass — see git log around 0d5faa8 ("Bump to 2.0.0"), with the original disabling commit at aaa8f29 and the destructive regen at 1eb2b4e.
  • The source-of-truth SHA for restoring custom files is the commit before .fernignore was last disabled. Once it has been re-armed, that's HEAD.
  • Before adding new custom files, add their paths to .fernignore first.
  • When the Fern API spec adds a resource that needs custom client behavior (like proxy or workflows), the custom files go on .fernignore immediately.

Public API: the wrapper is Pipedream / AsyncPipedream

Two layers of clients exist:

  • src/pipedream/client.py — generator-produced base classes Client and AsyncClient. These hold the actual sub-client wiring and the auth bootstrap, and are regenerated on every Fern run. Not exported publicly.
  • src/pipedream/pipedream.py — the Pipedream and AsyncPipedream classes (in .fernignore). They subclass Client / AsyncClient and add the public-facing convenience layer. This is what consumers import (from pipedream import Pipedream).

The wrapper adds: env-var resolution (PIPEDREAM_ACCESS_TOKEN, PIPEDREAM_CLIENT_ID, PIPEDREAM_CLIENT_SECRET, PIPEDREAM_PROJECT_ID, PIPEDREAM_PROJECT_ENVIRONMENT, PIPEDREAM_BASE_URL, PIPEDREAM_WORKFLOW_DOMAIN), the access_token shorthand (turns into a token=lambda: … callable when calling super().__init__), project_id validation (ValueError("Project ID is required")), the raw_access_token property, and the self.workflows = WorkflowsClient(...) attachment.

The wrapper also forwards every new Fern-introduced constructor param (headers, max_retries, follow_redirects, httpx_client, logging) through to super().__init__. When a regen adds another such param, mirror it here too — otherwise consumers can't reach it without instantiating Client directly.

__init__.py: keep regenerated, surgically add workflows

src/pipedream/__init__.py is in .fernignore but is otherwise a normal Fern-generated module index (the lazy _dynamic_imports dict, __all__ list, and if typing.TYPE_CHECKING: block). After every regen we accept the new content as the base and re-apply three additions for the workflows module:

  1. from .workflows.client import AsyncWorkflowsClient, WorkflowsClient plus workflows in the from . import (…) block, both inside the TYPE_CHECKING guard.
  2. "AsyncWorkflowsClient": ".workflows.client", "WorkflowsClient": ".workflows.client", and "workflows": ".workflows" in _dynamic_imports.
  3. "AsyncWorkflowsClient", "WorkflowsClient", and "workflows" in __all__.

Do not restore old ConfigurableProp_* or Emitter_* discriminator aliases — those types no longer exist in the regenerated types/ and importing them raises.

Auth model: access_token shorthand bridged to token callable

The wrapper accepts an access_token: Optional[str] for the common "I already have a bearer token" case and converts it to the generator's token: Callable[[], str] shape:

if access_token:
    super().__init__(token=(lambda: access_token), **common_kwargs)
else:
    super().__init__(client_id=client_id, client_secret=client_secret,
                     **common_kwargs)

The OAuth flow (client_id + client_secret) is owned entirely by the generated base — Client constructs an OAuthTokenProvider and threads its get_token into a SyncClientWrapper. The wrapper does not override this; it just decides which branch to take.

raw_access_token reaches into self._client_wrapper._get_token() to expose the current bearer token regardless of which branch was taken. Don't make _get_token private at the wrapper level.

Custom resources

Workflows (src/pipedream/workflows/) — entirely custom. Calls a different domain (m.pipedream.net by default, overridable via workflow_domain), not api.pipedream.com. Surfaces invoke() and invoke_for_external_user(). The Fern API spec doesn't model this resource; if a regen ever adds a generated workflows/ directory, delete it and keep the custom one. Public surface: HTTPAuthType (NONE/STATIC_BEARER/OAUTH), WorkflowsClient / AsyncWorkflowsClient with the two invoke* methods, all returning httpx.Response. There is no with_raw_response accessor here — the response is already raw.

Proxy (src/pipedream/proxy/) — caller-friendly request shape, not the generator's (url_64, …, request) shape. Callers pass (url, *, external_user_id, account_id, headers=None, params=None[, body]); the client base64-encodes the URL internally, prefixes all caller headers with x-pd-proxy- via RequestOptions(additional_headers=…), and switches between parsed ProxyResponse and binary Iterator[bytes] based on the response's Content-Type. Empty bodies return None. The generator periodically tries to revert the public surface to (url_64, …) returning Iterator[bytes] only — check reference.md Proxy section after every regen.

The proxy raw client returns HttpResponse[Optional[Union[ProxyResponse, Iterator[bytes]]]] and is wrapped by the high-level client through a shared _consume_sync / _consume_async helper that handles the dual return shape (eager-read for JSON, generator with deferred context-manager exit for streams).

Generator-managed files (don't hand-edit)

  • src/pipedream/client.py — base Client / AsyncClient. Auth, sub-client wiring, _make_default_async_client (with httpx_aiohttp autodetect). Regenerates fully on each Fern run.
  • src/pipedream/core/*.py — client_wrapper.py, http_client.py, pagination.py, parse_error.py, logging.py, pydantic_utilities.py, etc. These are the runtime substrate; custom code uses them but should not modify them.
  • src/pipedream/version.py — reads __version__ from package metadata via importlib.metadata. There is no hardcoded version in this file.
  • src/pipedream/types/, src/pipedream/errors/, and every resource client except the ones listed in .fernignore.
  • tests/utils/* and tests/test_aiohttp_autodetect.py — generator produces and updates these. They must keep passing after any change to client.py, client_wrapper.py, or http_client.py.

Version bump checklist

__version__ is auto-read from package metadata, so the source of truth is pyproject.toml. Three places hold the version string:

  1. pyproject.toml — [tool.poetry] version = "X.Y.Z".
  2. src/pipedream/core/client_wrapper.py — User-Agent: pipedream/X.Y.Z header (line ~37).
  3. src/pipedream/core/client_wrapper.py — X-Fern-SDK-Version: "X.Y.Z" header (line ~42).

After bumping, run poetry install --all-extras to refresh the installed package metadata; otherwise pipedream.__version__ will lag. Use semver: new generator types/runtime defaults that change observable behavior are breaking and require a major bump (see 0d5faa8 for the v2.0.0 example).

Regen playbook

When a Fern regen runs (fern-bot PR or local fern generate):

  1. Confirm .fernignore has every custom file listed and is not commented out. Compare against 0d5faa8's .fernignore if unsure.
  2. After regen, run poetry install --all-extras then poetry run python -c "from pipedream import Pipedream, AsyncPipedream; from pipedream.workflows.client import HTTPAuthType; print('ok')". If it fails, errors usually fall into three buckets:
    • Removed types referenced by src/pipedream/__init__.py's _dynamic_imports (e.g., a ConfigurableProp* variant the spec dropped) — drop the dead entry.
    • Sub-client option shape changes in client_wrapper.py (e.g., the generator adds a new constructor param). Mirror it on Pipedream / AsyncPipedream so consumers can reach it.
    • Custom code referencing a renamed/removed generator symbol — see what the regen produced and adjust the bridge. The current proxy and workflows custom code uses encode_path_param, parse_obj_as, ValidationError → ParsingError mapping, and the OMIT sentinel; if the regen changes those names, follow.
  3. Run poetry run pytest -rP -n auto .. Expect 69 passed, 1 skipped (the skip is the aiohttp autodetect guard).
  4. Spot-check the public API: Pipedream(project_id="x", access_token="y") constructs and exposes .workflows, .proxy, .proxy.with_raw_response, and .raw_access_token; Pipedream() with no env vars raises ValueError("Project ID is required").
  5. If the generator left the SDK version unchanged but you made customization changes, bump it manually per the checklist above.
  6. Diff .github/workflows/ci.yml's publish job. The Fern generator emits a poetry publish --username/--password step against secrets.PYPI_USERNAME / PYPI_PASSWORD; this project actually publishes via PyPI Trusted Publishing (OIDC) — permissions: id-token: write plus pypa/gh-action-pypi-publish@release/v1. No PYPI_* secrets exist on the repo or org, so if a regen reverts the auth shape (which already happened once on v2.0.0 — see CI run 25577372729) the next tag push silently fails with 403 Invalid or non-existent authentication information. .fernignore guards this file, but a temporary .fernignore disable + regen will re-open the trapdoor.

Known footguns

  • The workflows client passes both data=body and json=body to httpx_client.request(...). This was inherited from the pre-2.0 code and is preserved for backcompat. Fern's wrapper currently accepts both kwargs, but the dispatch behavior is sensitive to body type — if a workflow expects strict JSON and starts receiving form-encoded payloads (or vice versa), this is the place to look.
  • ConfigurableProp is a plain typing.Union, not a Pydantic discriminated annotation. Payloads with a missing or wrong type may silently parse into a best-fit variant instead of raising. Don't rely on validation alone to catch malformed configurable-prop payloads.
  • max_retries=2 is the default. For mutating proxy / workflow calls, pass max_retries=0 (per-client or per-call via request_options) if duplicate side effects on transient 5xx are unacceptable.
  • The version string lives in three places (see version bump checklist). pipedream.__version__ reads from installed package metadata, so a stale poetry install will mask a fresh bump.

Verification commands

  • Install: poetry install --all-extras
  • Tests: poetry run pytest -rP -n auto . (≈2s; expect 69 passed, 1 skipped).
  • Smoke imports: see step 2 of the regen playbook.
  • Diff sanity after a regen: git diff main -- .fernignore should match the un-commented form; git status should not show src/pipedream/workflows/ as deleted.

Tooling stack

  • Build / package manager: Poetry 2.x (pyproject.toml, poetry.lock). CI uses poetry install --no-root then poetry install.
  • Test runner: pytest 9.x with pytest-xdist (-n auto), pytest-asyncio (mode=auto). Config in pyproject.toml's [tool.pytest.ini_options].
  • Pydantic: v2.x; the SDK supports the v1 compat shim via core/pydantic_utilities.py. Don't import directly from pydantic.v1.
  • HTTP: httpx (sync + async), with optional httpx-aiohttp for the aiohttp transport. Detection in client.py's _make_default_async_client.
  • Python floor: 3.10 (per pyproject.toml python = "^3.10").

CI / publishing

.github/workflows/ci.yml runs the test matrix (default httpx + an aiohttp variant), with cancel-in-progress: false concurrency, and the publish job runs on tag pushes using PYPI_USERNAME / PYPI_PASSWORD secrets. The Python version is pinned to 3.10 in CI even though the floor in pyproject.toml allows higher; bump deliberately if you raise the floor.

.github/workflows/auto-close-empty-prs.yml is custom; the regen does not produce it. Don't let a regen drop it.