The SDK lets you register a webhook_url when creating an experiment
(lab.create_experiment(..., webhook_url=...), client.experiments.create(..., webhook_url=...)),
but it has no support for the other half of that flow: verifying the deliveries that arrive. The
only path the SDK offers for getting results is polling with client.experiments.get_results(...).
So today every integrator hand-writes the HMAC check from the snippet in the API introduction docs.
The snippet itself is correct. The code around it is where things go wrong, and the failure modes
are quiet ones:
- Re-serializing the parsed body (
json.dumps(request.json())) before signing produces different
bytes than the ones that were signed, so every signature fails. The usual "fix" is to switch
verification off.
- Comparing hex digests with
== instead of hmac.compare_digest.
- An unset secret.
hmac.new(b"", body, sha256) returns a perfectly valid signature, so a missing
WEBHOOK_SECRET makes verification pass rather than fail. The control disarms itself and still
looks fine from the outside.
- A missing or malformed
X-Adaptyv-Signature raising TypeError or ValueError out of the
handler instead of being refused as an untrusted request.
- Deliveries retry up to 3 times, so a handler that 500s once is guaranteed a duplicate, and
X-Adaptyv-Delivery-Id is the only thing that lets you dedupe.
Proposal: a small framework-agnostic module, adaptyv/webhooks.py, with one entry point.
from adaptyv.webhooks import verify
event = verify(body=raw_bytes, headers=request.headers, secret=SECRET)
event.event # "experiment_update"
event.delivery_id # "019b8da3-4a91-16c6-fa94-619212bee6a6"
event.payload # the parsed body
- Takes the raw request body as
bytes. A str or a dict is refused, so the re-serialization
mistake is not expressible through the API.
- Raises rather than returning a boolean, because a boolean is what gets assigned to a variable
and then never checked. Two new leaves under the existing AdaptyvError, kept as siblings so
that catching one cannot swallow the other: WebhookVerificationError means the delivery never
proved it came from Foundry and should be answered 4xx, WebhookPayloadError means it did and
the envelope was unreadable. That one must not be answered 4xx, since 4xx is permanent in the
retry model and would discard a real event.
- Stdlib only (
hmac, hashlib, json). No new runtime dependencies and no framework imports.
- Tests need no API key and no network. Signatures are constructed in the test with the same
secret, so the suite is deterministic for anyone who clones the repo.
Plus a short README section: a framework-agnostic example, a FastAPI example, and a note that
handlers have to be idempotent because of the retry behavior.
A PR implementing this is open against main. Happy to change the shape of it, or to close it,
if you would rather take a different approach.
The SDK lets you register a
webhook_urlwhen creating an experiment(
lab.create_experiment(..., webhook_url=...),client.experiments.create(..., webhook_url=...)),but it has no support for the other half of that flow: verifying the deliveries that arrive. The
only path the SDK offers for getting results is polling with
client.experiments.get_results(...).So today every integrator hand-writes the HMAC check from the snippet in the API introduction docs.
The snippet itself is correct. The code around it is where things go wrong, and the failure modes
are quiet ones:
json.dumps(request.json())) before signing produces differentbytes than the ones that were signed, so every signature fails. The usual "fix" is to switch
verification off.
==instead ofhmac.compare_digest.hmac.new(b"", body, sha256)returns a perfectly valid signature, so a missingWEBHOOK_SECRETmakes verification pass rather than fail. The control disarms itself and stilllooks fine from the outside.
X-Adaptyv-SignatureraisingTypeErrororValueErrorout of thehandler instead of being refused as an untrusted request.
X-Adaptyv-Delivery-Idis the only thing that lets you dedupe.Proposal: a small framework-agnostic module,
adaptyv/webhooks.py, with one entry point.bytes. Astror adictis refused, so the re-serializationmistake is not expressible through the API.
and then never checked. Two new leaves under the existing
AdaptyvError, kept as siblings sothat catching one cannot swallow the other:
WebhookVerificationErrormeans the delivery neverproved it came from Foundry and should be answered 4xx,
WebhookPayloadErrormeans it did andthe envelope was unreadable. That one must not be answered 4xx, since 4xx is permanent in the
retry model and would discard a real event.
hmac,hashlib,json). No new runtime dependencies and no framework imports.secret, so the suite is deterministic for anyone who clones the repo.
Plus a short README section: a framework-agnostic example, a FastAPI example, and a note that
handlers have to be idempotent because of the retry behavior.
A PR implementing this is open against
main. Happy to change the shape of it, or to close it,if you would rather take a different approach.