Skip to content

docs: refresh the querying Kosli tutorial against current cyber-dojo data - #413

Merged
dangrondahl merged 7 commits into
mainfrom
docs/refresh-querying-kosli-tutorial
Sep 14, 2026
Merged

dangrondahl merged 7 commits into
mainfrom
docs/refresh-querying-kosli-tutorial

Conversation

@dangrondahl

@dangrondahl dangrondahl commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Running tutorials/querying_kosli.md as written no longer works. cyber-dojo renamed its flows to *-ci, so kosli list artifacts --flow creator and kosli get artifact creator:344430d fail with "Flow named 'creator' does not exist". The search example resolves to an archived flow with 2022 timestamps, and the aws-beta aws-beta~1 diff example prints nothing today because the two latest snapshots run the same artifacts.

This refresh:

  • Follows one current commit (creator 99d7b74) through every step: search, list artifacts, get artifact, the snapshot where it started running, and the diff that shows its deployment.
  • Replaces every sample output with what the CLI prints today, including the newer TAGS, POLICIES, COMPLIANT, and COMPLIANCE columns.
  • Uses a snapshot pair that actually differs (aws-beta#8339 vs #8338) for the diff example, explains that an empty diff means nothing changed, and mentions --show-unchanged.
  • Notes that one commit can produce several artifacts, since search now also lists Terraform state artifacts for the same commit.
  • Explains the VISIBILITY column: a legacy per-flow field that does not control access. Access comes from the organization's visibility. This spawned chore: drop the VISIBILITY column from kosli list flows cli#1169 to drop the column.

Setup sweep across the three cyber-dojo tutorials, from review:

  • All three (querying_kosli, following_a_git_commit_to_runtime_environments, tracing_a_production_incident_back_to_git_commits) now use the same copy-pasteable Setup block with KOSLI_API_TOKEN=ignored, since cyber-dojo is public and the CLI only needs a non-empty value. A placeholder is used deliberately instead of a real-looking token, to avoid secret scanners and to avoid implying a real token is required.
  • The API token prerequisite is optional on all three.
  • The commit-to-production tutorial's kosli list flows table is refreshed; it listed two flows that no longer exist. The rest of that page is still stale and deserves its own refresh.

No page files move, no navigation or redirect changes.

Verification

Every command on the querying page was run against cyber-dojo on 2026-09-14 and the outputs pasted from those runs, trimmed with [...] where long. The placeholder token was verified to read every flow.

Related: #412 also touched following_a_git_commit_to_runtime_environments.md (frontmatter description only). The hunks do not overlap.

…data

cyber-dojo renamed its flows to *-ci, so the tutorial's creator commands
failed with "Flow named 'creator' does not exist". Follow one current
commit (creator 99d7b74) through search, list, get, snapshot, and diff so
every command runs as written. Update all sample outputs and timestamps
to what the CLI prints today, including the new TAGS, POLICIES, and
COMPLIANT columns, and use a snapshot pair that actually differs for the
diff example.
@dangrondahl
dangrondahl requested a review from a team as a code owner September 14, 2026 09:32
@mintlify

mintlify Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
kosli 🟢 Ready View Preview Sep 14, 2026, 12:19 PM

dangrondahl added a commit that referenced this pull request Sep 14, 2026
The stale "Scaled up" line is removed as part of the full tutorial
refresh in #413, so this PR no longer needs to touch the file.
@claude

claude Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Claude encountered an error after 4m 30s —— View job


Doc review

Checked the three tutorials at branch head (1c2edf5). The two findings from the earlier rounds — the VISIBILITY/access contradiction and the elision marker inside History: — are resolved, and I re-verified the pasted sample data against the generated live examples rather than taking it on trust: the 39-flow list, flow descriptions, per-flow visibility values (including the three public monorepo-* rows), and all six environments with their types, tags and policies match client_reference/kosli_list_flows.md and kosli_list_environments.md exactly. --show-unchanged, the flow:sha / flow@fingerprint expression syntax and the 15-artifact default all check out against the reference pages. No new pages, no moves, so no navigation or redirect work is needed; all internal links are root-relative and every target file exists (I verified these by hand — gh pr checks was not available to this run, so treat link-rot as unverified per CLAUDE.md).

Three findings.

Improvement — the sweep stops one page short

Location: tutorials/cli_and_http_proxy.md:42 and :58

Those two commands are the site's fourth and fifth cyber-dojo invocations, and they still read:

kosli list envs --org cyber-dojo --http-proxy http://localhost:8888 --api-token <your-token>

Both defects this PR removed from the other three pages are still here: the reader is told they need a token for an org that does not require one, and <your-token> is parsed by bash as an input redirection, so pasting the line gives bash: your-token: No such file or directory. This tutorial is the one where a reader is most likely to paste verbatim, because the thing under test is the proxy, not Kosli.

Recommendation: drop --api-token <your-token> from both commands and rely on the KOSLI_API_TOKEN convention the other cyber-dojo pages now set, or use the same ignored placeholder.

Fix this →

Improvement — "public organization" is now load-bearing in six places and documented in none

Location: tutorials/querying_kosli.md:20, :24, :76; tutorials/following_a_git_commit_to_runtime_environments.md:27, :59; tutorials/tracing_a_production_incident_back_to_git_commits.md:19, :23

The three tutorials now rest on a product behavior — a public organization is readable without a valid token, and organization visibility overrides the per-flow VISIBILITY field — that nothing else on the site states. getting_started/authenticating_to_kosli.md:7, the page these tutorials link to, says the opposite in outline: "Most interactions with Kosli — from the CLI, the API, or CI/CD pipelines — require an API token." The only visibility the site documents is the deprecated per-flow --visibility flag (client_reference/kosli_create_flow.md:27). Meanwhile the ten generated live examples (client_reference/kosli_list_flows.md:48 and siblings) still ship a real read-only token for the same org, so the docs currently tell two stories about what a reader needs to query cyber-dojo. You've said the live-docs side is tracked separately — the gap worth closing in this PR is the missing home for the claim: with no page owning it, there is nowhere to correct it if the behavior changes or unauthenticated reads get rate-limited, and a reader who wants this for their own org has nowhere to go.

Recommendation: add one sentence to getting_started/authenticating_to_kosli.md covering public organizations, and link the tutorials' Setup sections to it instead of each asserting the behavior independently.

Fix this →

Suggestion — three sibling pages, three different Setup shapes

Location: tutorials/tracing_a_production_incident_back_to_git_commits.md:19

That page puts the explanatory sentence above the code block, so it tells the reader to "replace ignored" two lines before ignored appears. tutorials/querying_kosli.md:24 puts the same sentence after the block, which reads correctly; tutorials/following_a_git_commit_to_runtime_environments.md has no sentence at all, only the in-block comment. Since all three now share one Setup, worth having them share one shape.

Recommendation: move line 19 below the code block, and add the same sentence after the block on the following_a_git_commit page.

Fix this →


2 Improvements, 1 Suggestion, 0 Critical. None of them blocks the merge — the refresh itself is accurate against the current cyber-dojo data.
• branch docs/refresh-querying-kosli-tutorial

Comment thread tutorials/querying_kosli.md
Comment thread tutorials/querying_kosli.md Outdated
- Move the elision marker out of the History list and say that kosli
  search prints further artifacts as separate blocks.
- Narrow the visibility note: the CLI needs a token, so say any valid
  token can read the public cyber-dojo org rather than "without logging
  in", and point readers without a token at the published read-only one.
- Add the same visibility note to the commit-to-production tutorial,
  whose list flows output has the same private column.
@dangrondahl

Copy link
Copy Markdown
Contributor Author

Review findings addressed in ab52faa:

  • VISIBILITY column contradicts "data is public". A note under the table explains that the column is a legacy per-flow field and that access is decided by the organization's visibility. Checked against the API without a token: all 39 flows come back and a "private" flow's artifacts are served. The column is a CLI leftover from deprecating --visibility, so this review spawned chore: drop the VISIBILITY column from kosli list flows cli#1169 to drop it from kosli list flows. Once that ships the note can be removed.
  • "Readable without logging in" overshoots. Narrowed to "any valid Kosli API token can read every flow listed here". The Setup section now also points readers who have no token yet at the read-only cyber-dojo token already published in the CLI reference live examples, so the tutorial is runnable before they have an org of their own.
  • Elision marker inside History:. Moved to the outer indentation as [... further artifacts for this commit ...], and the sentence below now says kosli search prints each artifact as its own block.
  • Same contradiction on the commit-to-production tutorial. Added the identical note under its kosli list flows output. Duplicated rather than factored into a snippet on purpose: a snippet import would go straight after the frontmatter, next to the description line fix: describe what each Kosli Actions trigger fires on and correct the webhook event_type #412 edits, and would conflict on merge. The note is temporary anyway.

That other tutorial is otherwise as stale as this one was (2022 data, archived flow names). Worth its own refresh, separate from this PR.

dangrondahl added a commit that referenced this pull request Sep 14, 2026
…e webhook event_type (#412)

## Summary

The Kosli Actions page described the "Artifact changed" trigger as
firing "when instances of an artifact are scaled up or down". Scaling
events are no longer recorded, and that trigger fires when an artifact
that is already running becomes compliant or non-compliant, or gains new
provenance. Readers who set it up expecting a notification on a new
deployment got nothing.

Changes to `integrations/kosli_actions.md`:

- Replace the six-bullet trigger list with a table mapping UI label to
API value and stating the exact condition for each trigger. The
"Artifact changed" row says explicitly that it does not fire for new
deployments.
- Add a note that `ON_SCALED_ARTIFACT` keeps its historical name for API
compatibility and never fires on replica count changes.
- Link the table to the API reference and the Terraform `kosli_action`
page so readers can connect the three surfaces.
- Correct the webhook example from `ARTIFACT_STARTED` to
`STARTED_ARTIFACT` and list all six possible `event_type` values below
it.

Adjacent stale scaling prose, flagged in review:

- `tutorials/following_a_git_commit_to_runtime_environments.md`: the
description no longer lists "instance scaling" as a runtime event Kosli
shows.
- `tutorials/querying_kosli.md`: the stale "Scaled up from 1 to 3" line
is handled by the full tutorial refresh in #413 instead, so this PR no
longer touches that file.

The upstream fix for the Terraform `kosli_action` example and schema
descriptions is kosli-dev/terraform-provider-kosli#262.

Closes #411

## Verification

- The API link target was checked against the live OpenAPI spec: `PUT
/actions/{org}/environments` has the summary "Create or update
environment action", which slugifies to the linked path.
- `mint broken-links` reports one broken link, in
`tutorials/working_with_controls.mdx`. It pre-exists on `main` and is
unrelated to this change.
Verified against cyber-dojo: a bogus token reads every flow and artifact,
and only an unset token fails, because the CLI refuses to run without
one. Say so instead of "any valid token".
Comment thread tutorials/querying_kosli.md Outdated
The <your-api-token> placeholder is not copy-pasteable and the page
now says you can follow it before you have a token. Use the read-only
token the CLI reference live examples already publish, in the same
shape, and drop the claim that the server does not check the token.
Comment thread tutorials/following_a_git_commit_to_runtime_environments.md Outdated
Comment thread tutorials/following_a_git_commit_to_runtime_environments.md Outdated
A literal that looks like a real token invites secret scanners and
implies one is required. cyber-dojo is public, so any non-empty value
works; use KOSLI_API_TOKEN=ignored and say so.
Comment thread tutorials/following_a_git_commit_to_runtime_environments.md
Comment thread tutorials/following_a_git_commit_to_runtime_environments.md
Comment thread tutorials/querying_kosli.md
Comment thread tutorials/querying_kosli.md Outdated
- Make the API token prerequisite optional on the commit-to-production
  and incident-tracing tutorials, matching the querying tutorial.
- Replace the incident-tracing Setup block with the same placeholder
  token used by the other two, and drop "any authenticated user".
- Refresh the commit-to-production list flows table from today's
  output; it listed two flows that no longer exist.
- Mention --show-unchanged for an empty snapshot diff.
@dangrondahl
dangrondahl enabled auto-merge (squash) September 14, 2026 12:29
@dangrondahl
dangrondahl merged commit 4acd32b into main Sep 14, 2026
7 of 8 checks passed
@dangrondahl
dangrondahl deleted the docs/refresh-querying-kosli-tutorial branch September 14, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants