feat(ci): two-step vagrant tests reusing a configurator export#9146
Draft
JeGoi wants to merge 30 commits into
Draft
feat(ci): two-step vagrant tests reusing a configurator export#9146JeGoi wants to merge 30 commits into
JeGoi wants to merge 30 commits into
Conversation
A pf-user mysqldump omits triggers (the pf DB user lacks the TRIGGER privilege), so a restored/imported database loses password_delete_trigger and the ip4log/ip6log/locationlog history triggers. Reapply only the triggers from the matching schema after loading the dump, leaving restored rows untouched.
The base pf-user dump already carries routines (--opt --routines) and PacketFence defines no DB events, so re-dumping them only produced duplicate/no-op objects on restore.
…as-is pf-user dumps omit triggers, so reapply them (from triggers.sql or the matching schema) before any upgrade instead of only for --db-restore. Under --restore-as-is, skip mysql_upgrade and the credential prompt (socket auth).
--restore-as-is restores a same-version backup verbatim: no database/MariaDB/config upgrade, keeps the backup's DB host/port, and asks nothing. Also renames --skip-adjust-conf to --skip-adjust-db-conf since it only adjusts the DB host/port.
Verify config restoration with a marker file and that triggers/routines survive; all restores now use --restore-as-is.
import.sh isolates to packetfence-base and leaves the restart to the operator; the suites now do it via pfcmd service pf start before hitting the webadmin API.
cluster.conf is restored verbatim since it joined stored_config_files, leaving the node with the source cluster's interfaces/IPs. Rewrite the local node's sections from the corrected pf.conf (and propagate interface renames to the CLUSTER VIP sections); skipped under --restore-as-is.
log.conf and several non-.conf files are restored but were undocumented or wrongly listed as excluded; also document cluster.conf adjustment and --restore-as-is verbatim behavior.
…r-conf Default import no longer adopts the export's cluster config: the server is left standalone with an empty cluster.conf and the export's copy is saved as cluster.conf.imported. --update-cluster-conf opts into restoring and adjusting it; --restore-as-is keeps the current cluster.conf untouched.
Rebuilt 37-commit branch on devel via squash-and-replay after PR #9095 landed parallel box-build infra. Preserves devel's EL8/node11 build jobs and the noninteractive dpkg upgrade fix; keeps the branch's golden-box machinery, pf*branch inventory, branch-scoped REF_SLUG upload, exportable backup pretest, and 3-level setup-vagrant-box.sh resolution. Pre-rebase tip preserved on backup/two-step-vagrant-tests-pre-rebase.
The pfel8branch/pfdeb12branch boxes carry stale RHSM consumer certs from the bake, which makes the register playbook's cert-stat guard skip re-registration; downstream yum/dnf then fails to fetch repomd.xml. AD VMs stay on pfad11branch.
…e burst The previous filter gsub'd \r out of CR-redraw bursts, producing a single line of "Progress: 0%Progress: 5%...Progress: 100%". Split each input on \r to keep the final frame, buffer consecutive progress lines, and emit only the most recent one when non-progress output (or EOF) arrives.
Re-apply the Fingerbank upstream api_key from psono when the import didn't carry it, then assert the collector is active; device-profiling/Fingerbank tests need it on 100.64.0.1:4723.
import.sh restores the file with cp -a (older mtime); fingerbank::Config caches by mtime and kept masking the restored api_key, leaving fingerbank-collector unmanaged. Touch forces a reload.
Per the upgrade guide, config changed on disk needs a reload; the fingerbank cache is keyed by mtime and import.sh's cp -a back-dates the file. pfcmd cache fingerbank clear is the idiomatic fix (same command the test uses), replacing the touch hack and dead psono re-apply path.
…removed --db-restore
…st on restore The nightly dump omits large history tables via --ignore-table, leaving trigger targets (locationlog_history) missing after a restore. Export their structure and recreate them empty on import. Only tables that exist are dumped, so obsolete names (iplog_archive) never break the export.
Backups made before excluded_tables_schema.sql existed don't carry it, so fall back to the matching pf-schema (with the devel X.Y.sql fallback) to recreate the excluded tables empty on restore.
Contributor
There was a problem hiding this comment.
Pull request overview
Implements a two-step Vagrant/venom CI pipeline where a single configurator “producer” job exports an exportable backup once, and downstream scenarios restore that backup instead of re-running configurator. It also improves the export/import mechanism so dump-excluded tables (e.g., locationlog_history) are recreated before triggers are reapplied, preventing trigger failures during cleanup.
Changes:
- Add CI “pretest” producer jobs to export a configurator-passed backup and gate downstream scenario jobs on it.
- Add restore playbooks/scripts so scenarios import the pretest export (
--db/--conf --restore-as-is) instead of running configurator. - Enhance export/import tooling to preserve triggers and recreate dump-excluded table structures during restore; update docs and acceptance tests accordingly.
Reviewed changes
Copilot reviewed 62 out of 62 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| t/venom/vars/all.yml | Adds a config-marker path used to validate config restoration from backups. |
| t/venom/test-wrapper.sh | Improves Vagrant progress filtering and logs ansible scenario runs to a file. |
| t/venom/test_suites/backup_db_and_restore/TESTSUITE.md | Updates acceptance-test documentation to reflect new restore flows and checks. |
| t/venom/test_suites/backup_db_and_restore/teardown/00_teardown.yml | Adds teardown step to remove the config marker file. |
| t/venom/test_suites/backup_db_and_restore/00_backup_db_and_restore.yml | Seeds a marker file to validate configuration restores. |
| t/venom/test_suites/backup_db_and_restore/05_backup_db_and_restore.yml | Verifies exported archive contains schema/data/grants/triggers/excluded table schema. |
| t/venom/test_suites/backup_db_and_restore/10_backup_db_and_restore.yml | Switches DB restore to --db --restore-as-is and restarts PF for API readiness. |
| t/venom/test_suites/backup_db_and_restore/15_backup_db_and_restore.yml | Adds validation that excluded tables are recreated and triggers/routines survive restore. |
| t/venom/test_suites/backup_db_and_restore/25_backup_db_and_restore.yml | Adds configuration-only restore verification (--conf --restore-as-is). |
| t/venom/test_suites/backup_db_and_restore/30_backup_db_and_restore.yml | Adds full restore verification (--restore-as-is) for both DB and config. |
| t/venom/scenarios/common/restore_pf_export.yml | Adds a shared playbook to restore the pretest export in CI only. |
| t/venom/scenarios/unit_tests/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/unit_tests/playbooks/run_tests.yml | Removes configurator suite from unit_tests scenario suites. |
| t/venom/scenarios/template/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/template/playbooks/run_tests.yml | Removes configurator suite from template scenario suites. |
| t/venom/scenarios/security_events/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/security_events/playbooks/run_tests.yml | Removes configurator suite from security_events scenario suites. |
| t/venom/scenarios/security_event_virtualswitch/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/security_event_virtualswitch/playbooks/run_tests.yml | Removes configurator suite from security_event_virtualswitch scenario suites. |
| t/venom/scenarios/pfacct_bandwidth_virtualswitch/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/pfacct_bandwidth_virtualswitch/playbooks/run_tests.yml | Removes configurator suite from pfacct_bandwidth_virtualswitch scenario suites. |
| t/venom/scenarios/mac_auth/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/mac_auth/playbooks/run_tests.yml | Removes configurator suite from mac_auth scenario suites. |
| t/venom/scenarios/mac_auth_virtualswitch/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/mac_auth_virtualswitch/playbooks/run_tests.yml | Removes configurator suite from mac_auth_virtualswitch scenario suites. |
| t/venom/scenarios/inline/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/inline/playbooks/run_tests.yml | Removes configurator suite from inline scenario suites. |
| t/venom/scenarios/fingerbank_invalid_db/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/fingerbank_invalid_db/playbooks/run_tests.yml | Removes configurator suite from fingerbank_invalid_db scenario suites. |
| t/venom/scenarios/external_integrations/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/external_integrations/playbooks/run_tests.yml | Removes configurator suite from external_integrations scenario suites. |
| t/venom/scenarios/dot1x_wired_computer_auth_virtualswitch/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/dot1x_wired_computer_auth_virtualswitch/playbooks/run_tests.yml | Removes configurator suite from dot1x_wired_computer_auth_virtualswitch suites. |
| t/venom/scenarios/dot1x_eap_tls/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/dot1x_eap_tls/playbooks/run_tests.yml | Removes configurator suite from dot1x_eap_tls scenario suites. |
| t/venom/scenarios/dot1x_eap_peap/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/dot1x_eap_peap/playbooks/run_tests.yml | Removes configurator suite from dot1x_eap_peap scenario suites. |
| t/venom/scenarios/device_profiling_virtualswitch/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/device_profiling_virtualswitch/playbooks/run_tests.yml | Removes configurator suite from device_profiling_virtualswitch scenario suites. |
| t/venom/scenarios/cli_login/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/cli_login/playbooks/run_tests.yml | Removes configurator suite from cli_login scenario suites. |
| t/venom/scenarios/captive_portal/site.yml | Imports common restore playbook to reuse configurator export in CI. |
| t/venom/scenarios/captive_portal/playbooks/run_tests.yml | Removes configurator suite from captive_portal scenario suites. |
| docs/installation/export_import_mechanism.asciidoc | Documents restored config scope, cluster.conf behavior, and new import options. |
| ci/packer/vagrant_img/vagrant-in-docker.sh | Passes REF_SLUG into the dockerized vagrant build environment. |
| ci/packer/vagrant_img/upload-to-linode.sh | Adds branch-scoped uploads via REF_SLUG and switches to rclone S3 env vars. |
| ci/lib/vagrant/cleanup-pf-exports.sh | Adds host-side cleanup script to prune old exported backups in bucket. |
| ci/lib/vagrant/upload-pf-export.sh | Adds host-side workflow to dump/fetch/upload configurator export tarball. |
| ci/lib/vagrant/restore-pf-export.sh | Adds manual wrapper to run restore playbook outside CI job wiring. |
| ci/lib/vagrant/setup-vagrant-box.sh | Adds branch-scoped box resolution and provenance artifact writing. |
| addons/vagrant/site.yml | Normalizes whitespace in main Vagrant Ansible site playbook. |
| addons/vagrant/playbooks/pf_interfaces_up.yml | Adds playbook to bring up PF NICs pre-import to avoid import prompts. |
| addons/vagrant/playbooks/dump_pf_export.yml | Adds playbook to create and fetch exportable backup tarball from VM. |
| addons/vagrant/playbooks/restore_pf_export.yml | Adds playbook to download/push/import the pretest export into VMs. |
| addons/vagrant/pfservers/Vagrantfile | Makes OS upgrade provisioning OS-aware (apt vs dnf). |
| addons/vagrant/inventory/hosts | Switches some non-PF VMs to public boxes and pins box versions. |
| addons/functions/helpers.functions | Makes prompt() non-interactive under --restore-as-is. |
| addons/functions/database.functions | Recreates excluded tables and reapplies triggers when dump lacks triggers; skips MariaDB upgrade under --restore-as-is. |
| addons/functions/configuration.functions | Adds cluster.conf handling and interface-rename propagation; skips network prompts under --restore-as-is. |
| addons/full-import/import.sh | Adds --restore-as-is, --update-cluster-conf, --skip-adjust-db-conf; changes upgrade/finalization behavior. |
| addons/full-import/export.sh | Exports triggers and excluded-table schema alongside grants for mysqldump-based exports. |
| .gitlab-ci.yml | Adds pretest stage, configurator export producer template, and needs-based gating for consumers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two-step vagrant test pipeline: a configurator pretest exports a PacketFence
backup once, and the 16 downstream venom scenarios restore it instead of
re-running the configurator. Also fixes the restore path so a restored DB keeps
the
locationlog_historytable its trigger writes to (it was excluded from thedump), which had blocked
node_cleanupfrom deleting nodes.Impacts
addons/full-import/export.sh+addons/functions/database.functions— exportand recreate the dump-excluded tables before reapplying triggers.
addons/vagrant/playbooks/restore_pf_export.yml+t/venom/scenarios/*/site.yml— download and restore the export (
--db/--conf --restore-as-is).import.shand the CI pipeline (.gitlab-ci.yml).Code / PR Dependencies
Builds on
fix/restore-reapply-schema-triggers(rebased on top of it).Delete branch after merge
YES
Checklist
docs/installation/export_import_mechanism.asciidoc)backup_db_and_restore)NEWS file entries
New Features
re-running the configurator.
Bug Fixes
locationlog_historywhile keeping its trigger, sonode_cleanupcould not delete nodes.UPGRADE file entries
import.sh:--db-restoreremoved; use--restore-as-is(optionally with--db/--conf).