Initial implementation (#1)

* Initial implementation

* CI: run "go test"

* CI: check for missing "go generate"

* Test on macOS Tahoe because we use cgo

* Integration test that uses Xvfb and a Fyne app

* CI: nicer name for main.yml workflow

* Disable TestSimpleVNC because of chicken and egg

* Implement explicit Run() method for desktop.Desktop and use errgroup

* TestSimpleVNC: only skip it in CI, also enable log/slog debug logging

* A bit more logging when receiving framebuffer updates and CV results

* TestSimpleVNC: test all mtell capabilities, not just <click 'text'>

* Executor: wait a bit before requesting an image and trying CV again

* vision: log found text using debug level

Author: edigaryev

.github/workflows/main.yml

@@ -0,0 +1,30 @@
+name: Main
+
+on:
+  push:
+
+jobs:
+  test:
+    name: Test
+    runs-on: ghcr.io/cirruslabs/macos-runner:tahoe
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: stable
+      - name: Run tests
+        run: go test -v ./...
+  missing-go-generate-check:
+    name: Check for missing "go generate"
+    runs-on: ghcr.io/cirruslabs/ubuntu-runner-amd64:24.04-xs
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: stable
+      - name: Run "go generate ./..."
+        run: go generate ./...
+      - name: Make sure no files had been changed
+        run: |
+          git status --porcelain
+          test -z "$(git status --porcelain)"

README.md

@@ -1,7 +1,61 @@
 # mtell
 
-`mtell` CLI allows you to tell a machine to do something over VNC.
-
-It's indispensable when the actions to be performed cannot be done over a traditional SSH connection, which is a frequent necessity when automating macOS machines.
+`mtell` CLI allows you to tell a machine to do something over VNC. It's indispensable when the actions to be performed cannot be done over a traditional SSH connection, which is a frequent necessity when automating macOS machines.
 
 Besides allowing you to script basic keyboard and mouse interactions, `mtell` can also utilize computer vision to wait for and click certain elements on the screen.
+
+This project is heavily inspired by [Packer's `boot_command`](https://developer.hashicorp.com/packer/integrations/cirruslabs/tart/latest/components/builder/tart#boot-configuration), but extends its command set and allows you to run these commands in any environment where you can start a binary.
+
+Special thanks to [Tor Arne Vestbø](https://github.com/torarnv), who contributed the initial [`<wait 'text'>` implementation](https://github.com/cirruslabs/packer-plugin-tart/pull/178) to [Packer builder for Tart VMs](https://github.com/cirruslabs/packer-plugin-tart), which made us realize that we can further extend the `boot_command` and do pretty cool things with it (e.g. computer vision).
+
+## Installation
+
+### Using Golang
+
+```shell
+go install github.com/cirruslabs/mtell@latest
+```
+
+## Usage
+
+```shell
+mtell --vnc "vnc://:password@localhost:5900" "<wait10s><click 'Select Your Country or Region'>"
+```
+
+## Reference
+
+### Waiting
+
+There commands are useful to work around loading screens.
+
+* `<wait10>` — wait 10 seconds
+* `<wait5m15s>` — wait 5 minutes and 15 seconds
+* `<wait 'Choose Your Country'>` — using computer vision, wait for the pattern (can be a regular expression) to appear on screen
+
+### Keyboard
+
+Introduce the following commands into your program to press a corresponding key:
+
+* `<bs>`, `<del>`, `<enter>`, `<return>`, `<esc>`, `<tab>`, `<spacebar>` — editing keys
+* `<insert>`, `<home>`, `<end>`, `<pageUp>`, `<pageDown>` — navigation keys
+* `<up>`, `<down>`, `<left>`, `<right>` — arrow keys
+* `<f1>`–`<f12>` — <kbd>F1</kbd>–<kbd>F12</kbd> keys
+* `<menu>` — context menu key
+* `<leftAlt>`, `<rightAlt>` — <kbd>Alt</kbd> key
+* `<leftCtrl>`, `<rightCtrl>` — <kbd>Control</kbd> key
+* `<leftShift>`, `<rightShift>` — <kbd>Shift</kbd> key
+* `<leftSuper>`, `<rightSuper>` — <kbd>Super</kbd> key
+* `<leftCommand>`, `<rightCommand>` — <kbd>⌘</kbd> key on macOS
+* `<leftOption>`, `<rightOption>` — <kbd>⌥</kbd> key on macOS
+
+Any keyboard command can be modified with `On` or `Off` modifier, for example:
+
+* `<leftShift>` —  presses <kbd>Shift</kbd> key and releases it
+* `<leftShiftOn>` — presses <kbd>Shift</kbd> key without releasing it
+* `<leftShiftOff>` — releases <kbd>Shift</kbd> key
+
+### Mouse
+
+These commands allow one to utilize a mouse, currently only through computer vision:
+
+* `<click 'Accept'>` — using computer vision, wait for the pattern (can be a regular expression) to appear on screen and click in the center of its bounding box

cmd/main.go

@@ -0,0 +1,36 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"os"
+	"os/signal"
+	"syscall"
+
+	"github.com/cirruslabs/mtell/internal/command"
+	"github.com/cirruslabs/mtell/internal/logginglevel"
+	"github.com/lmittmann/tint"
+)
+
+func main() {
+	if err := mainImpl(); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+
+		os.Exit(1)
+	}
+}
+
+func mainImpl() error {
+	ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer cancel()
+
+	// Use a colored logging handler for log/slog
+	handler := tint.NewHandler(os.Stderr, &tint.Options{
+		Level: logginglevel.Level,
+	})
+	logger := slog.New(handler)
+	slog.SetDefault(logger)
+
+	return command.NewRootCommand().ExecuteContext(ctx)
+}

go.mod

@@ -0,0 +1,78 @@
+module github.com/cirruslabs/mtell
+
+go 1.26
+
+require (
+	github.com/lmittmann/tint v1.1.3
+	github.com/mitchellh/go-vnc v0.0.0-20150629162542-723ed9867aed
+	github.com/mna/pigeon v1.3.0
+	github.com/progrium/darwinkit v0.5.0
+	github.com/spf13/cobra v1.10.2
+	github.com/stretchr/testify v1.11.1
+	golang.org/x/sync v0.19.0
+)
+
+require (
+	dario.cat/mergo v1.0.2 // indirect
+	github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
+	github.com/Microsoft/go-winio v0.6.2 // indirect
+	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/containerd/errdefs v1.0.0 // indirect
+	github.com/containerd/errdefs/pkg v0.3.0 // indirect
+	github.com/containerd/log v0.1.0 // indirect
+	github.com/containerd/platforms v0.2.1 // indirect
+	github.com/cpuguy83/dockercfg v0.3.2 // indirect
+	github.com/distribution/reference v0.6.0 // indirect
+	github.com/docker/docker v28.5.1+incompatible // indirect
+	github.com/docker/go-connections v0.6.0 // indirect
+	github.com/docker/go-units v0.5.0 // indirect
+	github.com/ebitengine/purego v0.8.4 // indirect
+	github.com/felixge/httpsnoop v1.0.4 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/go-ole/go-ole v1.2.6 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
+	github.com/magiconair/properties v1.8.10 // indirect
+	github.com/moby/docker-image-spec v1.3.1 // indirect
+	github.com/moby/go-archive v0.1.0 // indirect
+	github.com/moby/patternmatcher v0.6.0 // indirect
+	github.com/moby/sys/sequential v0.6.0 // indirect
+	github.com/moby/sys/user v0.4.0 // indirect
+	github.com/moby/sys/userns v0.1.0 // indirect
+	github.com/moby/term v0.5.0 // indirect
+	github.com/morikuni/aec v1.0.0 // indirect
+	github.com/opencontainers/go-digest v1.0.0 // indirect
+	github.com/opencontainers/image-spec v1.1.1 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
+	github.com/shirou/gopsutil/v4 v4.25.6 // indirect
+	github.com/sirupsen/logrus v1.9.3 // indirect
+	github.com/tklauser/go-sysconf v0.3.12 // indirect
+	github.com/tklauser/numcpus v0.6.1 // indirect
+	github.com/yusufpapurcu/wmi v1.2.4 // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
+	go.opentelemetry.io/otel v1.40.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 // indirect
+	go.opentelemetry.io/otel/metric v1.40.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.40.0 // indirect
+	go.opentelemetry.io/otel/trace v1.40.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.9.0 // indirect
+	golang.org/x/crypto v0.43.0 // indirect
+	golang.org/x/sys v0.40.0 // indirect
+	google.golang.org/protobuf v1.36.11 // indirect
+)
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/spf13/pflag v1.0.9 // indirect
+	github.com/testcontainers/testcontainers-go v0.40.0
+	golang.org/x/mod v0.21.0 // indirect
+	golang.org/x/tools v0.25.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)