Replace Statsig with GrowthBook for experimentation by aapeliv · Pull Request #8567 · Couchers-org/couchers

aapeliv · 2026-05-14T04:47:31Z

Swaps the backend experimentation framework from Statsig to GrowthBook's sync Python SDK. The public API (check_gate) is unchanged.

statsig-python-core → growthbook>=1.2.0 in pyproject.toml
experimentation.py rewritten on the sync GrowthBook client; the shared feature_repo cache is pre-warmed in setup_experimentation()
STATSIG_SERVER_SECRET_KEY / STATSIG_ENVIRONMENT config → GROWTHBOOK_API_HOST / GROWTHBOOK_CLIENT_KEY (also updated in backend.dev.env and conftest.py)
Dropped the unused get_experiment, get_dynamic_config, and log_event helpers (the last one also collided with couchers.event_log.log_event)
Dropped the elaborate fork-safety boilerplate; the sync GrowthBook client has no background threads, so the comments in app.py / worker.py were pared down accordingly
Smoke-test gate renamed test_statsig_integration → test_growthbook_integration

Testing

make format is clean
make mypy reports no new errors (the 17 pre-existing errors are unrelated — proto attachments, abstract Account.DismissReminder, ics stubs)
from couchers.experimentation import check_gate, setup_experimentation imports cleanly with the new growthbook dep installed
Local smoke check still needed: bring the backend up against a real GrowthBook instance and confirm the test_growthbook_integration gate flips as expected with EXPERIMENTATION_ENABLED=1

Backend checklist

Added tests for any new code or added a regression test if fixing a bug
Run the backend locally and it works
Added migrations if there are any database changes, rebased onto develop if necessary for linear migration history

For maintainers

Maintainers can push commits to my branch
Maintainers can merge this PR for me

This PR was created with the Couchers PR skill.

vercel · 2026-05-14T04:47:37Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
couchers	Ready	Preview	May 18, 2026 3:43am

couchersbot · 2026-05-15T02:44:13Z

Claude finished @CouchersBot[bot]'s task in 1m 13s —— View job

I'll analyze this and get back to you.

couchersbot · 2026-05-15T02:45:33Z

Migration Review

One of our most common issues is migrations that don't work with existing
data in production. This is very hard to catch with CI, so we have this bot
to check for potential problems.

Overall Risk: SAFE

Files Reviewed

0155_replace_statsig_with_growthbook_for_.py — creates a new logging.experiment_exposures table (id, created, version, user_id, experiment_key, variation_id, data JSONB) with a unique constraint on (user_id, experiment_key, variation_id) and two (*, created) indexes.

Findings

No issues found.

The migration was renumbered from 0151 to 0155 since earlier reviews: it is now revision = "0155" / down_revision = "0154". 0154_remove_maybe_from_attendeestatus.py is the current head on develop (revision = "0154" / down_revision = "0153"), so the chain 0153 -> 0154 -> 0155 is linear and the 0151 collision flagged previously is resolved.

The change is purely additive — experiment_exposures is a brand-new empty table in the logging schema (not present in schema.sql), so there is no existing data to backfill, cast, or violate constraints. The model in models/logging.py matches the migration's columns, constraints, and indexes exactly, and the unique constraint name uq_experiment_exposures_user_exp_var matches the on_conflict_do_nothing(constraint=...) reference in experimentation.py. Indexes are created on an empty table, so there is no lock-contention concern. downgrade() correctly drops both indexes and the table.

Summary

The migration is safe to apply against production: a new empty table with no impact on existing data, and the revision chain is linear at 0155.

couchersbot · 2026-05-15T03:19:13Z

Claude finished @CouchersBot[bot]'s task in 1m 27s —— View job

I'll analyze this and get back to you.

WouldYouKindly · 2026-05-16T15:23:33Z

+            on_experiment_viewed=on_experiment_viewed,
+        )
+        context._growthbook = gb  # type: ignore[attr-defined]
+    return context._growthbook  # type: ignore[attr-defined, no-any-return]


Can we add _growthbook as an attribute to CouchersContext, with None as default? Then there will be no need for type ignores

yes, done. thanks

couchersbot · 2026-05-17T04:06:59Z

Claude finished @aapeliv's task in 2m 49s —— View job

Migration Safety Review

Fetch PR diff
Identify migration files
Review migration against existing-data checklist
Post findings

Overall Risk: HIGH RISK — revision-number collision blocks this migration.

This PR's 0151_replace_statsig_with_growthbook_for_.py declares revision = "0151" / down_revision = "0150", but develop already has a migration with that exact revision id (0151_add_machine_readable_data_payload_to_.py) and the chain has since advanced to 0153. Two migrations sharing id 0151 create a branched, non-linear history that Alembic cannot apply.

Fix: rebase onto develop and renumber this migration to revision = "0154" / down_revision = "0153" (and update the filename ordinal to 0154_…). This matches the PR's own unchecked checklist item "rebased onto develop if necessary for linear migration history".

The table itself is fine — logging.experiment_exposures is a purely additive new table, the model matches the migration exactly, and downgrade() is correct. The only blocker is the revision collision.

Full review posted here, with an inline suggestion on the migration file.

couchersbot · 2026-05-17T13:34:12Z

Claude finished @aapeliv's task in 1m 18s —— View job

Migration Safety Review

Fetch PR diff
Identify migration files
Review migration against existing-data checklist
Post findings

Overall Risk: SAFE — the revision collision is resolved and the migration is purely additive.

The migration is now 0154_replace_statsig_with_growthbook_for_.py with revision = "0154" / down_revision = "0153". Since 0153 is the current head on develop, the migration chain is linear — the 0151 collision flagged in earlier reviews is fixed.

logging.experiment_exposures is a brand-new empty table, so there is no existing production data to backfill, cast, or violate constraints. The model in models/logging.py matches the migration's columns, constraints, and indexes exactly, and downgrade() correctly drops both indexes and the table.

Full review updated here.

Swap the backend experimentation framework from Statsig to GrowthBook's sync Python SDK. Drop the unused get_experiment / get_dynamic_config / log_event helpers and the fork-safety boilerplate that was needed for Statsig's threaded internals but not for the sync GrowthBook client.

Adds a generic typed get_feature_value(context, name, default) so callers can read strings, numbers, dicts, and experiment variations - not just boolean gates. Also restores the 'Experimentation is disabled' log line that got trimmed in the initial swap.

The SDK defaults to no timeout, so a slow CDN response could block a worker indefinitely on the periodic refresh. Set http_connect_timeout=1 and http_read_timeout=2 on both the warmup and per-request instances.

Swap @statsig/react-bindings for @growthbook/growthbook-react, slim the experimentation hook surface to useGate + useFeatureValue (matching the backend), and migrate the single donation_amounts consumer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Adds a logging.experiment_exposures table populated by GrowthBook's on_experiment_viewed callback. Each row captures one distinct exposure of a user to an experiment, deduped via a sha256 fingerprint over the assignment-relevant fields - so a re-bucketing or weight change produces a new row with its own timestamp, while identical re-evaluations are collapsed via ON CONFLICT DO NOTHING on (user_id, experiment_key, fingerprint). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Drops the sha256 fingerprint column in favour of a plain unique constraint on (user_id, experiment_key, variation_id). We accept that re-bucketings within the same variation aren't separately recorded; `created` and `data` reflect the first exposure for that variation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

The bot-generated 0151 migration referenced the dropped `fingerprint` column. Removing it so CI can regenerate against the current schema. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Owns the feature fetch directly instead of going through the library's feature_repo singleton. A daemon thread refreshes every minute and request threads only ever read the in-memory snapshot - they never block on the CDN and they keep the last known good state on fetch failure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Points the web frontend (dev/staging/prod) and the dev/staging backend at the gbapi.couchershq.org proxy with their respective SDK keys. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

The only frontend experiment - drop the useFeatureValue hook and use the DONATIONS_BOX_VALUES constant directly at all three reference sites. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Declare _growthbook directly on CouchersContext so _get_growthbook no longer needs type: ignore comments. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

couchersbot · 2026-05-18T03:39:08Z

Claude finished @aapeliv's task in 1m 30s —— View job

Migration Review

One of our most common issues is migrations that don't work with existing
data in production. This is very hard to catch with CI, so we have this bot
to check for potential problems.

Overall Risk: SAFE

Files Reviewed

0155_replace_statsig_with_growthbook_for_.py — creates a new logging.experiment_exposures table (id, created, version, user_id, experiment_key, variation_id, data JSONB) with a unique constraint on (user_id, experiment_key, variation_id) and two (*, created) indexes.

Findings

No issues found.

The migration was renumbered from 0151 to 0155 since earlier reviews: it is now revision = "0155" / down_revision = "0154". 0154_remove_maybe_from_attendeestatus.py is the current head on develop (revision = "0154" / down_revision = "0153"), so the chain 0153 → 0154 → 0155 is linear and the 0151 collision flagged previously is resolved.

The change is purely additive — experiment_exposures is a brand-new empty table in the logging schema (not present in schema.sql), so there is no existing data to backfill, cast, or violate constraints. The model in models/logging.py matches the migration's columns, constraints, and indexes exactly, and the unique constraint name uq_experiment_exposures_user_exp_var matches the on_conflict_do_nothing(constraint=...) reference in experimentation.py. Indexes are created on an empty table, so there is no lock-contention concern. downgrade() correctly drops both indexes and the table.

Summary

The migration is safe to apply against production: a new empty table with no impact on existing data, and the revision chain is linear at 0155.

github-actions · 2026-05-19T13:35:10Z

📝 Release Notes

This PR does not need to be included in release notes.

Reason: This PR primarily replaces the internal experimentation/feature flag infrastructure from Statsig to GrowthBook. The public API is mostly unchanged and there is no clear new user-facing feature or noticeable UX change described. While it includes some technical improvements, this looks like backend/internal tooling work rather than a major end-user-visible reliability or performance improvement that should go in release notes.

🤖 Bot Debug Information

Model: couchers.openai.gpt-5.4
Decision: exclude
Reasoning: This PR primarily replaces the internal experimentation/feature flag infrastructure from Statsig to GrowthBook. The public API is mostly unchanged and there is no clear new user-facing feature or noticeable UX change described. While it includes some technical improvements, this looks like backend/internal tooling work rather than a major end-user-visible reliability or performance improvement that should go in release notes.

vercel Bot deployed to Preview May 14, 2026 04:50 View deployment

vercel Bot deployed to Preview May 14, 2026 05:03 View deployment

vercel Bot deployed to Preview May 14, 2026 05:16 View deployment

aapeliv mentioned this pull request May 14, 2026

Replace Statsig with GrowthBook on the web frontend #8568

Closed

6 tasks

aapeliv marked this pull request as ready for review May 15, 2026 00:49

aapeliv force-pushed the backend/refactor/growthbook-experimentation branch from b3c59ae to b2e1d7e Compare May 15, 2026 00:50

vercel Bot deployed to Preview May 15, 2026 00:53 View deployment

aapeliv requested a review from nabramow as a code owner May 15, 2026 00:58

vercel Bot deployed to Preview May 15, 2026 01:02 View deployment

vercel Bot deployed to Preview May 15, 2026 02:46 View deployment

vercel Bot had a problem deploying to Preview May 15, 2026 03:00 Failure

vercel Bot deployed to Preview May 15, 2026 03:22 View deployment

vercel Bot deployed to Preview May 15, 2026 03:26 View deployment

vercel Bot deployed to Preview May 15, 2026 03:47 View deployment

vercel Bot deployed to Preview May 15, 2026 05:23 View deployment

WouldYouKindly reviewed May 16, 2026

View reviewed changes

aapeliv force-pushed the backend/refactor/growthbook-experimentation branch from b5dd633 to f0d4a0d Compare May 17, 2026 04:06

couchersbot Bot reviewed May 17, 2026

View reviewed changes

Comment thread app/backend/src/couchers/migrations/versions/0151_replace_statsig_with_growthbook_for_.py Outdated

vercel Bot deployed to Preview May 17, 2026 04:10 View deployment

vercel Bot deployed to Preview May 17, 2026 13:36 View deployment

aapeliv and others added 4 commits May 17, 2026 23:36

Set tight HTTP timeouts on GrowthBook feature fetches

732c3df

The SDK defaults to no timeout, so a slow CDN response could block a worker indefinitely on the periodic refresh. Set http_connect_timeout=1 and http_read_timeout=2 on both the warmup and per-request instances.

aapeliv and others added 10 commits May 17, 2026 23:36

Generate migrations

3090de5

Drop stale auto-generated migration

f577baf

The bot-generated 0151 migration referenced the dropped `fingerprint` column. Removing it so CI can regenerate against the current schema. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Generate migrations

11d1145

Configure GrowthBook API host and client keys

83b29ed

Points the web frontend (dev/staging/prod) and the dev/staging backend at the gbapi.couchershq.org proxy with their respective SDK keys. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Remove donation_amounts feature flag from DonationsBox

9b9bd44

The only frontend experiment - drop the useFeatureValue hook and use the DONATIONS_BOX_VALUES constant directly at all three reference sites. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Backend: Add _growthbook attribute to CouchersContext

a08e814

Declare _growthbook directly on CouchersContext so _get_growthbook no longer needs type: ignore comments. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Backend: Renumber GrowthBook migration to 0155

9db8053

aapeliv force-pushed the backend/refactor/growthbook-experimentation branch from 34d32aa to 9db8053 Compare May 18, 2026 03:38

vercel Bot deployed to Preview May 18, 2026 03:43 View deployment

aapeliv requested a review from WouldYouKindly May 18, 2026 03:57

aapeliv merged commit c9338da into develop May 19, 2026
7 checks passed

aapeliv deleted the backend/refactor/growthbook-experimentation branch May 19, 2026 13:34

github-actions Bot added the release notes: not needed label May 19, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace Statsig with GrowthBook for experimentation#8567

Replace Statsig with GrowthBook for experimentation#8567
aapeliv merged 14 commits into
developfrom
backend/refactor/growthbook-experimentation

aapeliv commented May 14, 2026

Uh oh!

vercel Bot commented May 14, 2026 •

edited

Loading

Uh oh!

couchersbot Bot commented May 15, 2026 •

edited

Loading

Uh oh!

couchersbot Bot commented May 15, 2026 •

edited

Loading

Uh oh!

couchersbot Bot commented May 15, 2026 •

edited

Loading

Uh oh!

WouldYouKindly May 16, 2026

Uh oh!

aapeliv May 18, 2026

Uh oh!

couchersbot Bot commented May 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

couchersbot Bot commented May 17, 2026 •

edited

Loading

Uh oh!

couchersbot Bot commented May 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aapeliv commented May 14, 2026

Testing

For maintainers

Uh oh!

vercel Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

couchersbot Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

couchersbot Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration Review

Files Reviewed

Findings

Summary

Uh oh!

couchersbot Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WouldYouKindly May 16, 2026

Choose a reason for hiding this comment

Uh oh!

aapeliv May 18, 2026

Choose a reason for hiding this comment

Uh oh!

couchersbot Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration Safety Review

Uh oh!

Uh oh!

couchersbot Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration Safety Review

Uh oh!

couchersbot Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migration Review

Files Reviewed

Findings

Summary

Uh oh!

Uh oh!

github-actions Bot commented May 19, 2026

📝 Release Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented May 14, 2026 •

edited

Loading

couchersbot Bot commented May 15, 2026 •

edited

Loading

couchersbot Bot commented May 15, 2026 •

edited

Loading

couchersbot Bot commented May 15, 2026 •

edited

Loading

couchersbot Bot commented May 17, 2026 •

edited

Loading

couchersbot Bot commented May 17, 2026 •

edited

Loading

couchersbot Bot commented May 18, 2026 •

edited

Loading