Auto-paginate the no-arg List* helpers#285
Merged
Merged
Conversation
ListIPs, ListAccounts, ListApplications, ListDatabases, ListKubernetesClusters and ListObjectStores previously sent a single GET /v2/... with no pagination parameters and silently returned whatever fit on the server-side first page (default 20 items). For accounts with more resources than that — typical in production — page 2 onward was invisible to civogo callers including civo-cli (`civo ip ls`) and terraform-provider-civo (`civo_reserved_ip`, `civo_kubernetes_cluster`, `civo_object_store`, `civo_database`). The customer ticket that surfaced this: an account with > 20 Reserved IPs in fra1 / lon1 could not see one specific IP via the CLI or Terraform. Tracked in api-go epic civo&598 (server-side off-by-one, fixed in civo/api-go!1354) and civogo epic civo&599 (this change). The fix introduces a shared `paginateAll` helper in `pagination.go` and routes the six no-arg `List*` functions through it. The merged response always reports `Page=1, Pages=1, PerPage=len(Items)`. Iteration is sequential and capped (100 pages / 10,000 items) with an explicit ErrPaginationCapExceeded rather than spinning forever. Two existing magic-number workarounds were retired in the same change: - ListAllInstances no longer uses ListInstances(1, 99999999). - FindObjectStoreCredential no longer uses ListObjectStoreCredentials(1, 10000). Both depended on api-go's paginator early-return path and would silently re-truncate the day server-side adds a per_page cap. They now share the same iterator. Includes regression tests covering: single page, multi-page in order, trailing-item-on-last-page (the customer's case), and hard-cap exhaustion. Updated affected fixtures that asserted the old buggy semantics (Pages>1 on a single-item response, PerPage=20 on the merged result).
dwoolger
force-pushed
the
fix/pagination-iteration
branch
from
94fb634 to
af6d7bc
Compare
dippydocus
approved these changes
May 11, 2026
dwoolger
pushed a commit
to civo/terraform-provider-civo
that referenced
this pull request
May 11, 2026
The civo_instances plural data source hard-coded per_page=200 with no iteration, so accounts with > 200 instances in a region silently saw a truncated list. Switch to ListAllInstances() — with civogo v0.7.1 that function iterates server pagination internally via the new paginateAll helper (civo/civogo#285), so the cap is removed without re-introducing the brittle "ask for 99999999 at once" workaround. Tracks civo&600. Closes the latent half of the customer-reported pagination truncation class along with the civogo v0.7.1 bump in the parent commit (civo&599). Verified locally with go1.25.10: - go build ./... clean - go test ./civo/instances/... ./internal/datalist/ pass - go test ./... all packages pass
dwoolger
pushed a commit
to dwoolger/civogo
that referenced
this pull request
May 11, 2026
This repo's CHANGELOG.md hasn't been updated by other PRs since 2021 — maintaining release notes for these pagination fixes here would be inconsistent with the rest of the codebase. Releases are documented via git tags + GitHub Releases instead. Removes the bullets I added in civo#285 (now part of v0.7.1) plus the new bullets added earlier in this branch.
dwoolger
added a commit
that referenced
this pull request
May 11, 2026
* Auto-paginate ListVPCIPs (follow-up to #285) ListVPCIPs hits a different endpoint (/v2/vpc/ips) than the classic ListIPs (/v2/ips) and was missed in the audit for #285. Same shape of bug — single un-paginated request, server returns only the first page, caller silently sees a truncated list. This is the path the terraform-provider-civo civo_reserved_ip data source migrated to via FindVPCIP. End-to-end verification against staging with civogo v0.7.1 still showed customers unable to resolve Reserved IPs past the first page, even after the api-go off-by-one fix landed — because the SDK never asked for page 2 of /v2/vpc/ips. Routes ListVPCIPs through the same paginateAll helper introduced in #285. FindVPCIP benefits transparently since it wraps ListVPCIPs. Adds two regression tests covering the trailing-item case via both the direct ListVPCIPs call and the FindVPCIP convenience wrapper. Originating customer report tracked in civo&599 (civogo) and civo&598 (api-go side). Closes the gap exposed during staging verification. * Cover ListDatabaseBackup and add ListAllActions Audit follow-up to civogo#285 + the ListVPCIPs commit in this branch. ListDatabaseBackup /v2/databases/{id}/backups paginates server-side unconditionally and the SDK previously sent no per_page param, so any database with > 20 backups silently truncated. Affects civo-cli `civo db backup list` and `civo db backup delete` via FindDatabaseBackup. Routed through paginateAll using the same caller-transparent pattern as ListIPs. ListAllActions ListActions takes *ActionListRequest with PerPage/Page fields that are optional. Callers passing a zero-valued request hit only the first page silently. Rather than change ListActions semantics (which would break any external Go consumer relying on explicit pagination), added a sibling ListAllActions that iterates internally — pairs with ListActions the way ListAllInstances pairs with ListInstances. Filters on the request are preserved across page fetches; Page/PerPage on the filters are overridden by the iterator. Both fixes covered by regression tests in pagination_test.go using the existing newMultiPageServer helper. No active downstream callers of ListAllActions today; ListDatabaseBackup fix unblocks civo-cli for accounts with > 20 backups per database. * Drop CHANGELOG.md additions This repo's CHANGELOG.md hasn't been updated by other PRs since 2021 — maintaining release notes for these pagination fixes here would be inconsistent with the rest of the codebase. Releases are documented via git tags + GitHub Releases instead. Removes the bullets I added in #285 (now part of v0.7.1) plus the new bullets added earlier in this branch. * Add per-file regression tests for ListAllActions and ListVPCIPs/FindVPCIP The mechanism-focused boundary tests sat in pagination_test.go, but the conventional location for API-contract regression tests is the per-file test file (matching the existing TestListActions / TestListVPCIPs / TestFindVPCIP patterns). action_test.go gains: - TestListAllActions basic API contract, single page - TestListAllActions_AcrossPages verifies iteration across 3 pages - TestListAllActions_PreservesFilters verifies filter fields on ActionListRequest are forwarded to every page request and that Page/PerPage on the filters are overridden by the iterator (callers cannot accidentally short-circuit pagination by passing a small per_page) vpc_test.go gains: - TestListVPCIPs_IteratesAllPages verifies the /v2/vpc/ips path correctly aggregates items across all pages - TestFindVPCIP_OnSecondPage covers the customer-visible terraform-provider symptom: data "civo_reserved_ip" with an id whose owning IP sits past the first server page must now resolve cleanly Removed the corresponding TestListVPCIPs_TrailingItem / TestFindVPCIP_FindsItemPastFirstPage / TestListAllActions_MultiPage from pagination_test.go (now redundant with the per-file versions). TestListDatabaseBackup_TrailingItem stays in pagination_test.go since no database_backup_test.go file exists in the package. All tests pass; no source changes.
dwoolger
pushed a commit
to civo/cli
that referenced
this pull request
May 11, 2026
Pulls in civogo v0.7.2 which: - Auto-paginates the no-arg List* helpers (civo/civogo#285) — fixes silent truncation in `civo ip ls`, `civo kubernetes ls`, `civo database ls`, `civo objectstore ls`, `civo account ls`, and any other CLI command resolving by name/id past page 1. - Auto-paginates `ListVPCIPs` / `FindVPCIP` (follow-up to #285, civogo PR #287). - Auto-paginates `ListDatabaseBackup` / `FindDatabaseBackup` — fixes `civo db backup list` and `civo db backup delete <name>` for any database with > 20 backups. - Adds `ListAllActions` sibling to `ListActions` (latent, no current CLI callers). Customer-visible symptom: an account with > 20 Reserved IPs in a region could not see one specific IP via `civo ip ls`. The same shape of bug affected several other commands at smaller boundary points (>20 K8s clusters, >20 databases, >20 object stores, etc.). With this bump, the CLI iterates server-side pagination transparently. Also bumps the go.mod `go` directive 1.25.9 -> 1.25.10 (transitively required by civogo v0.7.2 to clear stdlib advisories from civo/civogo#286). Verified locally with go1.25.10: - go build ./... clean - go test ./... pass
dwoolger
pushed a commit
to civo/terraform-provider-civo
that referenced
this pull request
May 11, 2026
The civo_instances plural data source hard-coded per_page=200 with no iteration, so accounts with > 200 instances in a region silently saw a truncated list. Switch to ListAllInstances() — with civogo v0.7.2 that function iterates server pagination internally via the new paginateAll helper (civo/civogo#285 / #287), so the cap is removed without re-introducing the brittle "ask for 99999999 at once" workaround. Tracks civo&600. Closes the latent half of the customer-reported pagination truncation class along with the civogo v0.7.2 bump in the parent commit (civo&599).
giornetta
added a commit
to civo/cli
that referenced
this pull request
May 11, 2026
* chore: bump civogo to v0.7.2 Pulls in civogo v0.7.2 which: - Auto-paginates the no-arg List* helpers (civo/civogo#285) — fixes silent truncation in `civo ip ls`, `civo kubernetes ls`, `civo database ls`, `civo objectstore ls`, `civo account ls`, and any other CLI command resolving by name/id past page 1. - Auto-paginates `ListVPCIPs` / `FindVPCIP` (follow-up to #285, civogo PR #287). - Auto-paginates `ListDatabaseBackup` / `FindDatabaseBackup` — fixes `civo db backup list` and `civo db backup delete <name>` for any database with > 20 backups. - Adds `ListAllActions` sibling to `ListActions` (latent, no current CLI callers). Customer-visible symptom: an account with > 20 Reserved IPs in a region could not see one specific IP via `civo ip ls`. The same shape of bug affected several other commands at smaller boundary points (>20 K8s clusters, >20 databases, >20 object stores, etc.). With this bump, the CLI iterates server-side pagination transparently. Also bumps the go.mod `go` directive 1.25.9 -> 1.25.10 (transitively required by civogo v0.7.2 to clear stdlib advisories from civo/civogo#286). Verified locally with go1.25.10: - go build ./... clean - go test ./... pass * ci: bump Go to 1.25.10 in workflow defaults Matches the go.mod directive required by civogo v0.7.2 (transitively bumped in the previous commit). Without this, the Test / Lint / Security jobs install Go 1.25.9 and fail: go: go.mod requires go >= 1.25.10 (running go 1.25.9; GOTOOLCHAIN=local) Same change shape as civo/civogo#286. Bumps the fallback in: - .github/workflows/go.yml (3 refs: test, lint, security) - .github/workflows/goreleaser-check.yml (1 ref) release.yml already uses the floating `1.25.x` tag and is unaffected. The repo-level `vars.GO_VERSION` (if set) overrides the fallback; maintainers may want to update that too. * fix(deps): bump golang.org/x/image to v0.39.0 (GO-2026-4961) GO-2026-4961: panic when decoding large WEBP image on 32-bit platforms in golang.org/x/image. Reachable in this binary via cmd/kubernetes/kubernetes_app_show.go:52 → term.Render → webp.Decode. Fixed in golang.org/x/image v0.39.0. Pre-existing on master — the advisory was published between 2026-04-15 (master's last CI run, all green at the time) and 2026-05-11 (this PR's first CI run). This PR is the first to re-trigger CI in that window, so it picks up the new advisory; bumping the dep here unblocks the Security job and any future PR re-running CI. Also pulls in golang.org/x/text v0.36.0 (transitive). Verified: go test ./... pass; govulncheck ./... reports 0 vulnerabilities. --------- Co-authored-by: david.woolger@civo.com <david.woolger@civo.com> Co-authored-by: Michele Giornetta <michelegiornetta@gmail.com>
dwoolger
added a commit
to civo/terraform-provider-civo
that referenced
this pull request
May 11, 2026
* chore: bump civogo to v0.7.2 Pulls in civogo v0.7.1 which auto-paginates every no-arg / wrapped List* helper that the provider relies on. Affected resources/data sources whose lookups iterate the full account inventory: - `civo_reserved_ip` (resource + data) — via `FindIP`/`FindVPCIP` - `civo_kubernetes_cluster` (resource + data) — via `FindKubernetesCluster` - `civo_object_store` (resource + data) — via `FindObjectStore` - `civo_database` (resource + data) — via `FindDatabase` The follow-up (v0.7.2) over v0.7.1 also closes `ListVPCIPs` and `ListDatabaseBackup` — the VPC IP fix specifically unblocks `data "civo_reserved_ip"` since the provider migrated to FindVPCIP recently. End-to-end verified against staging. Customer-visible symptom that triggered this: a Terraform plan for an account with > 20 Reserved IPs in a region could not find one specific IP, blocking `civo_instance` creation. With v0.7.2 the SDK iterates server pagination transparently — no caller-side changes needed for the data lookups. Also bumps the go.mod `go` directive 1.25.8 -> 1.25.10 (transitively required by civogo v0.7.2 to clear stdlib advisories from civo/civogo#286). * fix: civo_instances data source now returns all instances (civo&600) The civo_instances plural data source hard-coded per_page=200 with no iteration, so accounts with > 200 instances in a region silently saw a truncated list. Switch to ListAllInstances() — with civogo v0.7.2 that function iterates server pagination internally via the new paginateAll helper (civo/civogo#285 / #287), so the cap is removed Tracks civo&600. Closes the latent half of the customer-reported pagination truncation class along with the civogo v0.7.2 bump in the parent commit (civo&599). --------- Co-authored-by: david.woolger@civo.com <david.woolger@civo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ListIPs, ListAccounts, ListApplications, ListDatabases, ListKubernetesClusters and ListObjectStores previously sent a single GET /v2/... with no pagination parameters and silently returned whatever fit on the server-side first page (default 20 items). For accounts with more resources than that — typical in production — page 2 onward was invisible to civogo callers including civo-cli (
civo ip ls) and terraform-provider-civo (civo_reserved_ip,civo_kubernetes_cluster,civo_object_store,civo_database).The customer ticket that surfaced this: an account with > 20 Reserved IPs in fra1 / lon1 could not see one specific IP via the CLI or Terraform. Tracked in api-go epic civo&598 (server-side off-by-one, fixed in civo/api-go!1354) and civogo epic civo&599 (this change).
The fix introduces a shared
paginateAllhelper inpagination.goand routes the six no-argList*functions through it. The merged response always reportsPage=1, Pages=1, PerPage=len(Items). Iteration is sequential and capped (100 pages / 10,000 items) with an explicit ErrPaginationCapExceeded rather than spinning forever.Two existing magic-number workarounds were retired in the same change:
Both depended on api-go's paginator early-return path and would silently re-truncate the day server-side adds a per_page cap. They now share the same iterator.
Includes regression tests covering: single page, multi-page in order, trailing-item-on-last-page (the customer's case), and hard-cap exhaustion. Updated affected fixtures that asserted the old buggy semantics (Pages>1 on a single-item response, PerPage=20 on the merged result).