Skip to content

Replace e2e tests with mock HTTP server integration tests - #28

Draft
ofalvai wants to merge 1 commit into
mainfrom
replace-e2e-with-integration-tests
Draft

Replace e2e tests with mock HTTP server integration tests#28
ofalvai wants to merge 1 commit into
mainfrom
replace-e2e-with-integration-tests

Conversation

@ofalvai

@ofalvai ofalvai commented May 15, 2026

Copy link
Copy Markdown
Contributor

Checklist

  • I've read and followed the Contribution Guidelines
  • step.yml and README.md is updated with the changes (if needed)

Version

Requires a PATCH version update

Context

The nightly CI was failing because the e2e test suite depended on two things that are no longer available: a decommissioned auth service (auth.services.bitrise.io) used to obtain BITRISEIO_ARTIFACT_PULL_TOKEN, and hardcoded fixture builds from March 2026 that have since been cleaned up. Since BITRISEIO_ARTIFACT_PULL_TOKEN is only injected by the platform in real pipeline builds — not in standalone e2e worker builds — there is no clean way to restore the old approach.

Changes

  • Delete e2e/bitrise.yml and the e2e/ directory entirely
  • Add step/run_integration_test.go with 5 integration tests that call Run() directly against a mock HTTP server
  • Vendor testify/require (was already in go.mod but not imported by any existing test)

Investigation details

Three approaches were considered:

A. Refresh fixture data + use a stored API token — update the hardcoded build/pipeline IDs and swap the auth service call for BITRISE_API_TOKEN. Quick to implement but perpetuates the same maintenance burden: fixtures go stale on every build retention cycle, and the token format may differ from what the platform injects in production.

B. Run e2e tests inside a real Bitrise pipeline — the step only receives BITRISEIO_ARTIFACT_PULL_TOKEN when running as part of an actual pipeline, so restructuring the CI to trigger a child pipeline would be the most faithful test environment. This requires significant changes to the centralized CI setup and is disproportionate for the coverage gained.

C. Replace with mock HTTP server integration tests — exercise Run() end-to-end with a local httptest.Server handling both the Bitrise API routes and file downloads. No external dependencies, no fixture maintenance, archives created programmatically in the test.

Decisions

Option C was chosen. The e2e tests were primarily validating infrastructure (auth token exchange, stale fixture data) rather than step logic. The unit test suite already covered individual components well; the only genuine gap was a full-stack Run() test with real archive extraction. The new tests fill that gap without any ongoing maintenance cost.

The e2e test suite relied on a decommissioned auth service to obtain
BITRISEIO_ARTIFACT_PULL_TOKEN, and on hardcoded fixture builds from
March 2026 that no longer exist. The new tests cover the same scenarios
using a mock HTTP server, removing both external dependencies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ofalvai
ofalvai marked this pull request as ready for review May 15, 2026 14:15
@ofalvai
ofalvai enabled auto-merge (squash) May 15, 2026 14:21
@ofalvai
ofalvai marked this pull request as draft May 19, 2026 11:39
auto-merge was automatically disabled May 19, 2026 11:39

Pull request was converted to draft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant