internal: Improve daemon implementation. (#2597)

* Add log date.

* Phase 1.

* Run in background.

* Rework connection.

* Add acquire.

* Add version handshake.

* Clean up usage.

* Add monitoring.

* Add background tasks.

* Fix watcher.

* Fix shutdown.

* Fixes.

Author: milesj

CHANGELOG.md

@@ -18,6 +18,25 @@
     cache, which can improve cache hit rates in some scenarios.
   - When a remote cache hit, we'll now warm the local cache with the hydrated manifest and its
     blobs, so the next run resolves locally instead of round-tripping to the remote.
+- **Daemon**
+  - Server log files will now rotate up to 7 times. Older log files will be automatically deleted.
+  - Webhook delivery and task output archiving are now acknowledged immediately and run in the
+    background on the daemon, so a client exiting or hitting a deadline no longer cancels the work
+    mid-flight.
+  - The daemon now takes exclusive ownership of its workspace through an advisory file lock held for
+    its entire lifetime, replacing PID-liveness checks that could be fooled by zombie processes,
+    reused PIDs, or processes owned by another user. A crashed daemon releases the lock
+    automatically, so a stale socket or state file can no longer block or misdirect the next start.
+  - Whether the daemon is running is now determined by connecting to it rather than probing a PID,
+    and its metadata is recorded in a `daemon.json` state file (replacing `moond.pid`).
+  - Connecting to the daemon and starting it are now a single operation, so a command that needs the
+    daemon will start one itself if the background pre-warm hasn't yet, instead of silently running
+    without it. Concurrent starts still coordinate so only one daemon is spawned.
+  - When connecting, the client now checks the running daemon's moon and protocol version against
+    its own and, on a mismatch, restarts it — so a daemon left over from before a `moon upgrade` is
+    replaced instead of serving the old binary indefinitely.
+  - The daemon now retires itself after a long idle period (no requests), and exits immediately if
+    its workspace is deleted, so an abandoned workspace no longer leaves a daemon running forever.
 - **Processes**
   - Improved our "stream and capture output" child process handling to operate on bytes instead of
     lines, which should resolve some edge cases with output not being written to the console, or
@@ -57,22 +76,39 @@
 
 #### 🐞 Fixes
 
-- Fixed an issue where an explicit head revision was ignored when diffing between revisions, and the
-  current working tree was compared against instead.
-- Fixed an issue where diffing against the previous revision would fail in repositories with a
-  single commit.
-- Fixed an issue where file names with spaces or special characters were excluded from file tree
-  results.
-- Fixed an issue where Git submodules added between 2 revisions were not included when diffing.
-- Fixed an issue where Git hooks could not be set up from the primary working tree when other
-  worktrees exist.
-- Fixed an issue where moon would take over a hooks directory managed by another tool (husky,
-  lefthook, etc) when `core.hooksPath` was already configured, overwriting its hook files, and
-  deleting the entire directory when hooks were disabled.
-- Fixed an issue where Windows hook wrappers would not forward arguments containing spaces
-  correctly, and would arbitrarily cap forwarding at 5 arguments.
-- Fixed an issue where PowerShell hooks would mangle user variables that start with `$ARG`, like
-  `$ARGS`.
+- **CLI**
+  - Fixed an issue where moon would silently exit with code 141 (SIGPIPE) when a child process
+    exited before consuming its stdin. Broken pipes are now handled explicitly instead of resetting
+    the SIGPIPE disposition, while piping moon's output to a consumer that closes early still exits
+    quietly with the conventional code.
+- **Daemon**
+  - Fixed an issue where every daemon RPC was capped by a 1 second client-side timeout, causing slow
+    procedures (webhook delivery, cache cleaning) to be cancelled even though the daemon was
+    healthy. Connection establishment is now bounded separately, and each procedure has an
+    appropriate deadline that is also enforced by the server.
+  - Fixed an issue where connecting to the daemon while it was still starting up — or being started
+    by another process — would fail immediately with "connection refused", and the run would
+    continue without the daemon. Connection attempts are now retried with a bounded backoff.
+  - Fixed an issue on Windows where the daemon briefly had no listening pipe instance between client
+    connections, causing sporadic connection failures, and where busy pipe instances were not
+    retried.
+- **VCS**
+  - Fixed an issue where an explicit head revision was ignored when diffing between revisions, and
+    the current working tree was compared against instead.
+  - Fixed an issue where diffing against the previous revision would fail in repositories with a
+    single commit.
+  - Fixed an issue where file names with spaces or special characters were excluded from file tree
+    results.
+  - Fixed an issue where Git submodules added between 2 revisions were not included when diffing.
+  - Fixed an issue where Git hooks could not be set up from the primary working tree when other
+    worktrees exist.
+  - Fixed an issue where moon would take over a hooks directory managed by another tool (husky,
+    lefthook, etc) when `core.hooksPath` was already configured, overwriting its hook files, and
+    deleting the entire directory when hooks were disabled.
+  - Fixed an issue where Windows hook wrappers would not forward arguments containing spaces
+    correctly, and would arbitrarily cap forwarding at 5 arguments.
+  - Fixed an issue where PowerShell hooks would mangle user variables that start with `$ARG`, like
+    `$ARGS`.
 
 #### ⚙️ Internal
 

Cargo.lock

@@ -63,7 +63,7 @@ semver = "1.0.27"
 serde = { version = "1.0.228", features = ["derive"] }
 serde_json = "1.0.150"
 sha2 = "0.11.0"
-starbase = { version = "0.12.2", features = ["clap", "miette"] }
+starbase = { version = "0.12.3", features = ["clap", "miette"] }
 starbase_archive = { version = "0.13.2", default-features = false, features = [
   "miette",
   "tar-all",

Cargo.toml

@@ -39,9 +39,7 @@ impl Subscriber for CleanupSubscriber {
             debug!("Cleaning stale cache");
 
             if let Some(mut daemon) = self.daemon_client.clone() {
-                let lifetime = self.lifetime.clone();
-
-                tokio::spawn(async move { daemon.clean_cache(lifetime, true).await });
+                daemon.clean_cache(self.lifetime.clone(), true).await?;
             } else {
                 self.cache_engine
                     .clean_stale_cache(&self.lifetime, false)

crates/action-pipeline/src/subscribers/cleanup_subscriber.rs

@@ -8,7 +8,7 @@ pub async fn logs(session: MoonSession) -> SessionResult {
     let connector = session.get_daemon_connector()?;
     let log_path = connector.get_log_file();
 
-    if connector.is_running().is_none() || !log_path.exists() {
+    if !connector.is_running().await || !log_path.exists() {
         session.console.render(element! {
             Container {
                 Notice(variant: Variant::Caution) {

crates/app/src/commands/daemon/logs.rs

@@ -16,7 +16,7 @@ pub async fn restart(session: MoonSession) -> SessionResult {
     }
 
     let connector = session.get_daemon_connector()?;
-    let old_pid = connector.is_running();
+    let old_pid = connector.read_state().map(|state| state.pid);
 
     connector.stop_daemon().await?;
 

crates/app/src/commands/daemon/restart.rs

@@ -13,6 +13,7 @@ fn install_daemon_panic_hook() {
         } else {
             "non-string panic payload".to_owned()
         };
+
         let location = panic_info
             .location()
             .map(|location| {

crates/app/src/commands/daemon/server.rs

@@ -7,7 +7,7 @@ use std::time::Duration;
 pub async fn status(session: MoonSession) -> SessionResult {
     let connector = session.get_daemon_connector()?;
 
-    if connector.is_running().is_none() {
+    if !connector.is_running().await {
         session.console.render(element! {
             Container {
                 Notice(variant: Variant::Caution) {
@@ -19,16 +19,21 @@ pub async fn status(session: MoonSession) -> SessionResult {
         return Ok(None);
     }
 
-    let Some(mut client) = connector.connect().await? else {
-        session.console.render(element! {
-            Container {
-                Notice(variant: Variant::Caution) {
-                    StyledText(content: "Unable to connect to the daemon")
+    // The PID may be stale (a crashed daemon, or the PID was reused by
+    // another process), so a connect failure here is a report, not an error.
+    let mut client = match connector.connect_once().await {
+        Ok(Some(client)) => client,
+        _ => {
+            session.console.render(element! {
+                Container {
+                    Notice(variant: Variant::Caution) {
+                        StyledText(content: "Unable to connect to the daemon")
+                    }
                 }
-            }
-        })?;
+            })?;
 
-        return Ok(None);
+            return Ok(None);
+        }
     };
 
     let status = client.status().await?;
@@ -66,10 +71,10 @@ pub async fn status(session: MoonSession) -> SessionResult {
 
             Section(title: "Paths") {
                 Entry(
-                    name: "PID file",
+                    name: "State file",
                     value: element! {
                         StyledText(
-                            content: connector.get_pid_file().to_string_lossy(),
+                            content: connector.get_state_file().to_string_lossy(),
                             style: Style::Path
                         )
                     }.into_any()