Observation scenarios passing with zero effort (validation bypass)

[mihir0209]

While exploring the task scenarios, I noticed that two observation/query scenarios (jsonpath-basics and extract-logs-basics) can be marked as completed with zero user effort by just clicking "Validate" runs the answer command itself and checks the output against the cluster.

Steps to reproduce:

1. Start the jsonpath-basics scenario (or extract-logs-basics)
2. Click the "Validate" tab
3. Click "Run Validation" without typing anything in the terminal
4. Scenario is marked as completed — the validation command is the answer itself

(Attached screenshots here showing: Validate tab (OG one), clicking Run Validation, scenario passing without any user input)

---

As you can see in the second image, the validation is ran just over the "expected" result in scenarios (without running anything in the terminal below)... but I think asking user to submit a command would be a better approach here.

Root cause:
The scenarios use the cluster_state validation mode where the backend runs a predefined command against the cluster. For observation/query tasks, the validation command (e.g., kubectl get pod log-generator -o
jsonpath='{.status.containerStatuses[0].image}') is itself the answer the user should discover, so running it automatically defeats the purpose I thhink...

Proposed fix:
I had opened PR #1 with a fix that adds a command_submission validation mode where the user types their kubectl command in a text input, the backend runs it, and checks the output against the expected result. The fix:

• Preserves the existing cluster_state behavior for all other scenarios (no breaking changes)
• Only affects the two scenarios where the answer is the validation command itself (and will be helpful in developing more such scenarios if needed)
• Adds a new mode field to validation in the scenario schema (documented in SCHEMA.md)

I closed that PR prematurely while restructuring, happy to reopen it once there's agreement on the approach. Will also add screenshots of the results after the solution into PR. Please let me know!

[zeborg]

@mihir0209 Thanks for pointing it out, I'd appreciate it if you could mention all the scenarios which have similar unexpected validation behavior and work on their fixes as well.

I prefer not to change the schema unless it's absolutely necessary. If there's a reason why such scenarios can't be fixed without changing the schema, you must mention that as well.