[Testing] Regression test suite for all three plugins

[thealphacubicle]

Overview

Build a regression test suite covering all three plugins (CKAN, Socrata, ArcGIS)
to complement the existing unit tests. As the project moves through beta and
toward broader municipal adoption, a regression suite ensures new changes don't
silently break existing plugin behavior.

Goals

• Define a regression test matrix covering all MCP tools across all
  three plugins
• Cover both happy path and error path scenarios for each tool
• Use real (but stable) public portal endpoints as test fixtures where
  possible — data.boston.gov for CKAN, a known Socrata portal, a known
  ArcGIS portal
• Integrate regression suite into CI/CD pipeline so it runs on every
  PR to develop

Deliverables

• Regression test matrix defined and documented
• Regression tests implemented for all CKAN plugin tools
• Regression tests implemented for all Socrata plugin tools
• Regression tests implemented for all ArcGIS plugin tools
• Tests integrated into CI/CD pipeline
• README or TESTING.md updated to document how to run regression suite

Notes

• Build on top of existing unit test structure in tests/
• Use CKAN plugin tests as the reference pattern for Socrata and ArcGIS
• Flag any flaky tests that depend on live portal uptime — these should
  be clearly marked and excluded from blocking CI runs

[m13v]

for the regression test matrix across CKAN, Socrata, and ArcGIS, the biggest risk is relying on live public portal endpoints. data.boston.gov and similar portals have downtime, rate limits, and schema changes that will make your CI flaky within weeks. what worked for us: record the actual API responses once (using something like Playwright's route interception or a simple JSON dump), commit those as fixtures, and replay them in CI. then have a separate nightly job that hits the real endpoints and updates fixtures if the schema changed. that way PRs get fast deterministic tests and you still catch upstream changes.

for the test pattern across plugins, define a shared test interface that each plugin implements (list datasets, get dataset by ID, search, error cases). that way adding a fourth plugin later is just filling in the fixture data and implementing the interface, not writing a new test suite from scratch. also mark any test that depends on live portal uptime with a skip-in-ci flag so flaky external dependencies never block a merge.

[m13v]

for the shared test interface pattern and auto-discovering test scenarios across multiple plugins, we built an open source tool that does this. the test discovery and scenario generation logic: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts and the test plan generation API that structures test matrices: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/app/api/generate-plan/route.ts