TT-17995: run nightly-marked API tests on every trigger - #539
Merged
Conversation
API test execution is fast enough now that there is no need to hold the `nightly`-marked tests back to scheduled runs only. Drop `and not nightly` from every `apimarkers` string in both TUI variation files (5 occurrences each), so PR, push and workflow_dispatch runs pick up the ~58 `@pytest.mark.nightly` tests in tyk-analytics/tests/api too. This also fixes an inconsistency in the scheduled runs: because envfiles accumulate down the variation tree rather than override, the schedule leaves inherited the branch- and global-level envfiles, two of which still carried `not nightly`. Those combos were skipping the very tests the nightly run exists to cover. Follow-ups (separate tickets): - remove the now-meaningless `nightly` marker and its decorators from tyk-analytics/tests/api - delete the unreferenced db-variations.yml in github-actions Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
konrad-sol
had a problem deploying
to
github-pages
August 21, 2026 13:35 — with 
GitHub Actions
Failure
GitHub Actions
Failure
konrad-sol
enabled auto-merge (squash)
August 21, 2026 13:36
![probelabs[bot]](https://avatars.githubusercontent.com/in/1543721?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Drops
and not nightlyfrom everyapimarkersstring in the TUI variation config — 5 occurrences inconfig/tui/prod-variations.yml, 5 inconfig/tui/test-variations.yml.API test execution is fast enough now that there's no reason to hold the
nightly-marked tests back to scheduled runs. This makes PR, push andworkflow_dispatchruns pick up the ~58@pytest.mark.nightlytests intyk-analytics/tests/apias well.Side effect: fixes the scheduled runs too
Envfiles accumulate down the variation tree rather than override (
policy/variations.go:201appends the parent'sEnvFilesonto each child). So thescheduleleaves were inheriting the branch-level and global-level envfiles — two of which still carriednot nightly.Concretely,
prod-variation/tyk-analytics/master/schedule/apiemitted 8 envfile combos, not the 6 declared, and 2 of the 8 skipped exactly the tests the nightly run exists to cover. That's gone now.Verification
go build ./...— cleango test ./policy/...— passgromit policy generate-tuiregenerated cleanly; nonightlyremains anywhere in the generated.ghooutputRollout
pages.ymltriggers on pushes tomastertouchingconfig/tui/**, so merging republishes the static TUI site andtest-controllerpicks it up on the next run. No downstreamrelease.ymlregeneration needed. Reverting is a one-commit rollback.Follow-ups (not in this PR)
nightlymarker intyk-analytics/tests/api/pytest_ci.iniand its 58 decorators are now inert. They should be removed, otherwise people keep adding@pytest.mark.nightlyexpecting a PR-skip that no longer happens.github-actions/.github/actions/tests/test-controller/db-variations.ymlstill carriesnot nightlyin 2 places, but nothing references that file — the action fetches from the TUI API. Dead file, worth deleting.PRmarker is the mirror-image problem: registered as "run only in PR builds" and applied widely, but noapimarkersstring saysnot PR, so those tests run in scheduled runs too.-m "$api_markers"on the command line whilepytest_ci.iniaddopts already has-m "not upgrade and not resilience". pytest's-mis single-valued and the command line wins, so that exclusion is likely being dropped in CI.🤖 Generated with Claude Code