Skip to content

Commit ec229dc

Browse files
committed
docs(troubleshooting): document the #77 harmony_machines keepalive failure mode
Operators on pre-#77 builds see new task dispatch silently die ~10m after boot with 'FOREIGN KEY constraint failed' / 'failed to check node flags'. Add a dedicated troubleshooting entry (with the harmony_machines zero-rows check) and cross-link it from 'Task is queued but never runs'.
1 parent 19ec149 commit ec229dc

1 file changed

Lines changed: 35 additions & 1 deletion

File tree

docs/help/troubleshooting.md

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,11 @@ Common causes:
5252
**Symptom:** Tasks page shows an unowned task that's been sitting for several
5353
minutes while other tasks complete.
5454

55-
**Cause:** Task's declared `Cost.RAM` exceeds the engine budget.
55+
**Cause 1:** Task's declared `Cost.RAM` exceeds the engine budget.
56+
57+
> If *all* new tasks (not just heavy ones) stop being claimed ~10 minutes after
58+
> boot and you see `FOREIGN KEY constraint failed`, that's a different bug — see
59+
> [New work stops dispatching ~10 minutes after boot](#new-work-stops-dispatching-10-minutes-after-boot).
5660
5761
**Check the log** — curio-core emits a `WARN` line:
5862

@@ -70,6 +74,36 @@ sudo systemctl edit curio-core
7074
sudo systemctl restart curio-core
7175
```
7276

77+
## New work stops dispatching ~10 minutes after boot
78+
79+
**Symptom:** The daemon starts fine and keeps proving existing datasets, but any
80+
**new** action silently stalls after the node has been up for ~10 minutes. A
81+
`createDataSet` hangs at `POST /pdp/data-sets`; new uploads sit unprocessed. The
82+
log fills with:
83+
84+
```
85+
ERROR harmonytask task_type_handler.go ... constraint failed: FOREIGN KEY constraint failed
86+
ERROR harmonytask scheduler.go ... failed to check node flags {"error": "sql: no rows in result set"}
87+
```
88+
89+
**Cause:** The scheduler periodically reaps `harmony_machines` rows whose
90+
`last_contact` is older than 10 minutes (`LOOKS_DEAD_TIMEOUT`). On affected builds
91+
the per-node keepalive that refreshes `last_contact` never ran, so the node's own
92+
machine row got reaped out from under the `harmony_task.owner_id` foreign key —
93+
after which no task can be claimed.
94+
95+
**Check:**
96+
97+
```sql
98+
SELECT id, host_and_port, last_contact FROM harmony_machines;
99+
```
100+
101+
If this returns **zero rows** on a running daemon, you have this bug.
102+
103+
**Fix:** Upgrade to a build that includes the engine keepalive (curio-core #77).
104+
A `systemctl restart curio-core` restores service immediately on a fixed binary;
105+
on an unfixed binary it only buys another ~10 minutes.
106+
73107
## settleRail always reverts with `CannotSettleFutureEpochs`
74108

75109
**Symptom:** Dashboard Rails page shows `last attempt err` on every rail; logs show

0 commit comments

Comments
 (0)