DistSSHKitQueue.jl

English | 日本語

DistSSHKitQueue runs jobs one after another on machines that several people share. You can submit a job, check its status, and cancel. DistSSHKit does the run. Supported on macOS, Linux, and WSL2 Ubuntu (not native Windows).

Even small labs and individuals can keep one always-on machine, add SSH hosts, and use them together as a small set of compute nodes. Not on General yet. Julia 1.12+, DistSSHKit 0.4.1+.

Install

From the Julia REPL, type ] to enter the Pkg REPL mode and run:

pkg> add https://github.com/yamanori99/DistSSHKitQueue.jl#v0.1.0-beta.1

Or, equivalently, via the Pkg API:

julia> import Pkg; Pkg.add(url="https://github.com/yamanori99/DistSSHKitQueue.jl", rev="v0.1.0-beta.1")

Not on General yet, so pin the beta tag by URL. DistSSHKit 0.4.1+ comes from General with it. Do not Pkg.develop Kit for ordinary Queue work.

The queue host also needs ssh, rsync, and (only for git deploys) git — pkg> add does not install them. Full requirements: Requirements.

For everything else, see the Documentation.

Usage

Basic terms

• Queue host — the always-on machine that holds ~/.distsshkitqueue and runs the waiter (macOS or Linux; a VM is fine). A sleeping laptop is not this box.
• Client — a dev machine that submits, lists, watches, or cancels. No cap. It must not become the Kit master.
• Waiter — the serve process on the queue host. It starts DistSSHKit (execute!(…; detached=true)) and waits. Stopping it does not cancel a Kit job that is already running.
• Workers — where the script runs. DistSSHKit tokens: parent[:N] on the queue host, child:NAME[:N] on SSH machines.

  clients = dev machines (no cap)          one queue host (always on)
  ───────────────────────────────          ──────────────────────────
  yours / a colleague's / …                waiter   one Kit job at a time
       │                                   table    ~/.distsshkitqueue
       │  julia -m DistSSHKitQueue         add-host / remove-host
       │    qhost:NAME                     serve    now, this terminal
       │    submit | status | list-host    enable   again after reboot
       │    watch | cancel | …
       └────────────────────────────────►  then DistSSHKit go/drive
                                           → workers (Kit tokens)

qhost:NAME is the SSH name of the queue host (same idea as Kit child:NAME, but it names the queue host, not a worker). Already logged in there? Omit it. One lab: export DISTSSHKITQUEUE_HOST=… and omit qhost: (the token still wins). --hosts / --julia stay on Kit go / drive.

Placement tokens, go / drive flags, and remote setup are DistSSHKit's — see the kit docs.

Examples

From a client (job directory; Queue must be loadable from that env):

julia --project=. -m DistSSHKitQueue qhost:mini list-host
julia --project=. -m DistSSHKitQueue qhost:mini submit go child:host1:4 SCRIPT.jl
julia --project=. -m DistSSHKitQueue qhost:mini status
julia --project=. -m DistSSHKitQueue qhost:mini watch
julia --project=. -m DistSSHKitQueue qhost:mini cancel <id>

submit starts a waiter on the queue host if none is running.

On the queue host (once):

julia -m DistSSHKitQueue setup
julia -m DistSSHKitQueue add-host parent child:host1
julia -m DistSSHKitQueue enable --queue-env ~/.distsshkitqueue/env

setup / serve / enable / disable / add-host / remove-host refuse qhost:. Command reference: User Guide.

Documentation

Introduction	Introduction
First Steps	First Steps
User Guide	User Guide
API	API
News	NEWS.md

Contributing

Bugs and feature requests: Issues. See CONTRIBUTING.md.

License

Source code is MIT.