manage browser process directly, auto-detect lightpanda or chrome

Replace chromedp.NewExecAllocator with our own os/exec launch so exec
mode works for both chrome and lightpanda. resolveBrowser searches PATH
lightpanda-first, then chrome variants; explicit exec <path> wins and
the kind is inferred from the basename. Lightpanda is started as
'lightpanda serve --host 127.0.0.1 --port <picked>'; chrome gets the
default flag set + a temp --user-data-dir + --remote-debugging-port.
waitForBrowser polls /json/version until ready. Cleanup sends SIGINT
(5 s grace, then SIGKILL) and removes the temp dir.

Author: jakubkulhan

LIGHTPANDA.md

@@ -43,10 +43,16 @@ same code path is used for Chrome: it's also faster there because each render
 gets its own root browser session instead of paying for
 `Target.createBrowserContext` / disposal on a shared WS reader.
 
-For `exec` mode the middleware launches the browser once with
-`--remote-debugging-port=<picked>`, keeps a long-lived bootstrap WS open so
-chromedp's watchdog doesn't kill the process, and serves every render through
-its own fresh WS to the same browser.
+For `exec` mode the middleware itself launches the browser via `os/exec`
+(not chromedp's allocator), so both chrome and lightpanda are launchable.
+PATH is searched lightpanda-first, falling back to chrome variants;
+explicit `exec <path>` wins. For lightpanda the command is
+`lightpanda serve --host 127.0.0.1 --port <picked>` plus user flags; for
+chrome it's the default flag set + `--user-data-dir=<temp>` +
+`--remote-debugging-port=<picked>`. The middleware polls `/json/version`
+until ready, then serves every render through its own fresh WS to the
+same browser. On `Cleanup` the process gets SIGINT (5 s grace, then SIGKILL)
+and the temp user-data-dir is removed.
 
 `Provision` for `url` mode uses a one-shot WS to log the browser version, then
 discards it.
@@ -180,11 +186,10 @@ workflow does this; the README documents it.
   [middleware.go](middleware.go) `detectLightpanda`, setting `m.lightpanda`
   when `/json/version` says `"Browser":"Lightpanda/..."`.
 - **Provision** — [middleware.go](middleware.go): for `exec` mode picks a
-  free port, launches the browser via `chromedp.NewExecAllocator` with
-  `--remote-debugging-port=<picked>`, and keeps a single bootstrap WS open
-  for the lifetime of the middleware so chromedp's watchdog doesn't kill the
-  process. For `url` mode it stores the configured URL and runs a one-shot
-  remote probe to log the browser version.
+  free port, resolves a binary via `resolveBrowser` (lightpanda first, then
+  chrome), launches it with `exec.Command`, polls `/json/version` until
+  ready, then runs a one-shot remote probe to log the browser version. For
+  `url` mode it stores the configured URL and runs the same one-shot probe.
 - **Per-request flow** — [middleware_serve_http.go](middleware_serve_http.go):
   - bypass-header short-circuit at the top of `ServeHTTP`,
   - one fresh `chromedp.NewRemoteAllocator` + `chromedp.NewContext` per

README.md

@@ -55,17 +55,17 @@ chrome {
 - `timeout` - maximum time to wait for Chrome to render the page, default is `10s`.
 - `mime_types` - list of MIME types to render, default is `text/html`.
 - Browser (only one of these):
-  - `exec` - executes the local browser binary by given path, if the first argument starts with a dash (`-`), the binary is automatically found in the path and all the arguments are treated as additional flags on top of the [default flags](https://pkg.go.dev/github.com/chromedp/chromedp#pkg-variables)
-  - `exec_no_default_flags` - the same as `exec` but without the default flags
+  - `exec` - the middleware launches a browser process itself and connects to it. If a path is given, that binary is used; otherwise PATH is searched: **lightpanda first, then chrome variants** (`google-chrome`, `google-chrome-stable`, `chromium`, `chromium-browser`, plus `/Applications/Google Chrome.app/...` and `/Applications/Chromium.app/...` on macOS). For chrome-kind binaries a sensible default flag set is applied; for lightpanda the middleware runs it as `lightpanda serve --host 127.0.0.1 --port <picked>`. Extra flags after `--` are appended to the launch command. The browser kind is inferred from the binary basename (anything containing "lightpanda" → lightpanda mode).
+  - `exec_no_default_flags` - the same as `exec` but without the default chrome flags (no effect for lightpanda)
   - `url` - URL to the debugging protocol endpoint of a remote browser instance
 - `fullfill_hosts` - a list of hosts to issue as internal requests through the webserver, there's automatically the host of the original request
 - `continue_hosts` - a list of hosts to let Chrome do the regular network requests
 
-Every render opens a fresh CDP WebSocket to the browser. For `exec` mode the middleware launches the browser once on a known port and connects to it per request; for `url` mode it opens a new WS per request to the configured remote endpoint. This avoids the per-request `Target.createBrowserContext` round-trips on a shared connection (faster on Chrome) and gives Lightpanda 0.2.4+ a separate browser per connection (true concurrency).
+Every render opens a fresh CDP WebSocket to the browser. For `exec` mode the middleware launches the browser once on a known port (managed via `os/exec`, not chromedp's allocator, so both chrome and lightpanda are supported) and connects to it per request; for `url` mode it opens a new WS per request to the configured remote endpoint. This avoids the per-request `Target.createBrowserContext` round-trips on a shared connection (faster on Chrome) and gives Lightpanda 0.2.4+ a separate browser per connection (true concurrency).
 
 ## Browsers
 
-In addition to headless Chrome, [Lightpanda](https://lightpanda.io/) is supported as a CDP-compatible backend. Start it as `lightpanda serve --host 127.0.0.1 --port 9222` and point caddy-chrome at it via `url http://127.0.0.1:9222/`. Lightpanda is detected automatically (via `/json/version`) and the middleware switches to a single-target rendering mode:
+In addition to headless Chrome, [Lightpanda](https://lightpanda.io/) is supported as a CDP-compatible backend. The simplest setup is `exec` with no path — if `lightpanda` is in PATH it'll be picked over chrome and the middleware launches `lightpanda serve` itself. Alternatively, run it externally as `lightpanda serve --host 127.0.0.1 --port 9222` and point caddy-chrome at it via `url http://127.0.0.1:9222/`. Lightpanda is detected automatically (via `/json/version`) and the middleware switches to a single-target rendering mode:
 
 - as for Chrome, every render opens a fresh CDP WebSocket — Lightpanda 0.2.4+ gives each connection its own full browser, so concurrent requests render in parallel;
 - `Fetch` interception is skipped — Lightpanda's CDP processes commands serially per session, and dispatching `Fetch.fulfillRequest`/`continueRequest` for a sub-resource while the navigation fulfillment is being parsed deadlocks the WS read loop (see [lightpanda-io/browser#2391](https://github.com/lightpanda-io/browser/issues/2391)). Instead, caddy-chrome sets a `X-Caddy-Chrome-Bypass` header via `Network.setExtraHTTPHeaders` and lets Lightpanda fetch the navigation and every sub-resource (XHR, `fetch()`, `<script>`) directly from the same Caddy server. The middleware sees the marker header and passes the request straight to the next handler;

middleware.go

@@ -12,8 +12,12 @@ import (
 	"github.com/chromedp/cdproto/cdp"
 	"github.com/chromedp/chromedp"
 	"go.uber.org/zap"
+	"io"
 	"net"
 	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
 	"strconv"
 	"strings"
 	"time"
@@ -33,18 +37,15 @@ type Middleware struct {
 	FulfillHosts  []string       `json:"fulfill_hosts,omitempty"`
 	ContinueHosts []string       `json:"continue_hosts,omitempty"`
 	Links         bool           `json:"links,omitempty"`
-	log           *zap.Logger
-	timeout       time.Duration
-	// allocCtx owns the underlying browser process (for exec) or the
-	// long-lived chromedp.RemoteAllocator wrapper. Per-request work uses a
-	// fresh chromedp.NewRemoteAllocator(browserURL) so every render gets its
-	// own CDP WebSocket — this is what gives concurrent requests true
-	// isolation.
-	allocCtx     context.Context
-	allocCancel  context.CancelFunc
-	keepAliveCtx context.Context
-	keepCancel   context.CancelFunc
-	browserURL   string
+	log     *zap.Logger
+	timeout time.Duration
+	// In exec mode the middleware itself launches and owns the browser
+	// process (cmd) and its temp user-data-dir (tempDir). Per-request work
+	// always uses a fresh chromedp.NewRemoteAllocator(browserURL) so every
+	// render gets its own CDP WebSocket.
+	cmd        *exec.Cmd
+	tempDir    string
+	browserURL string
 	lightpanda bool
 }
 
@@ -95,40 +96,63 @@ func (m *Middleware) Provision(ctx caddy.Context) (err error) {
 	}()
 
 	if m.ExecBrowser != nil {
-		// Pick a free port for chrome to listen on so we can build a stable
-		// HTTP debugging URL and reuse it for per-request remote allocators.
 		port, perr := pickFreePort()
 		if perr != nil {
 			return fmt.Errorf("pick debug port: %w", perr)
 		}
-		var opts []chromedp.ExecAllocatorOption
-		if m.ExecBrowser.Path != "" {
-			opts = append(opts, chromedp.ExecPath(m.ExecBrowser.Path))
+
+		execPath, kind, perr := resolveBrowser(m.ExecBrowser.Path)
+		if perr != nil {
+			return perr
 		}
-		if m.ExecBrowser.DefaultFlags {
-			opts = append(opts, chromedp.DefaultExecAllocatorOptions[:]...)
+		m.lightpanda = kind == browserLightpanda
+
+		var args []string
+		switch kind {
+		case browserLightpanda:
+			args = append(args, "serve", "--host", "127.0.0.1", "--port", strconv.Itoa(port))
+			args = append(args, m.ExecBrowser.Flags...)
+		case browserChrome:
+			if m.ExecBrowser.DefaultFlags {
+				args = append(args, defaultChromeFlags...)
+			}
+			args = append(args, m.ExecBrowser.Flags...)
+			m.tempDir, perr = os.MkdirTemp("", "caddy-chrome-")
+			if perr != nil {
+				return fmt.Errorf("create user-data-dir: %w", perr)
+			}
+			args = append(args, "--user-data-dir="+m.tempDir)
+			args = append(args, "--remote-debugging-port="+strconv.Itoa(port))
 		}
-		for _, flag := range m.ExecBrowser.Flags {
-			parts := strings.SplitN(flag, "=", 2)
-			opts = append(opts, chromedp.Flag(parts[0], parts[1]))
+
+		m.log.Info("starting browser",
+			zap.String("path", execPath),
+			zap.Bool("lightpanda", m.lightpanda),
+			zap.Strings("args", args))
+
+		m.cmd = exec.Command(execPath, args...)
+		m.cmd.Stdout = io.Discard
+		m.cmd.Stderr = io.Discard
+		if err = m.cmd.Start(); err != nil {
+			return fmt.Errorf("start %s: %w", execPath, err)
 		}
-		// Override the default --remote-debugging-port=0 so we know the URL up
-		// front. chromedp only sets that flag if it's not already provided.
-		opts = append(opts, chromedp.Flag("remote-debugging-port", strconv.Itoa(port)))
-		m.allocCtx, m.allocCancel = chromedp.NewExecAllocator(context.Background(), opts...)
-		// Bootstrap a long-lived CDP connection so chrome stays running. The
-		// chromedp ExecAllocator kills the chrome process when its single
-		// browser connection drops, so we keep this one open for the lifetime
-		// of the middleware.
-		m.keepAliveCtx, m.keepCancel = chromedp.NewContext(m.allocCtx)
-		if err = chromedp.Run(m.keepAliveCtx, chromedp.ActionFunc(func(ctx context.Context) error {
+		m.browserURL = fmt.Sprintf("http://127.0.0.1:%d/", port)
+
+		if err = waitForBrowser(m.browserURL, 10*time.Second); err != nil {
+			return fmt.Errorf("wait for browser: %w", err)
+		}
+
+		probeAlloc, probeAllocCancel := chromedp.NewRemoteAllocator(context.Background(), m.browserURL)
+		defer probeAllocCancel()
+		probeCtx, probeCancel := chromedp.NewContext(probeAlloc)
+		defer probeCancel()
+		if err = chromedp.Run(probeCtx, chromedp.ActionFunc(func(ctx context.Context) error {
 			c := chromedp.FromContext(ctx)
 			bctx := cdp.WithExecutor(ctx, c.Browser)
-			return logBrowserVersion(bctx, m.log, false)
+			return logBrowserVersion(bctx, m.log, m.lightpanda)
 		})); err != nil {
-			return fmt.Errorf("start chrome: %w", err)
+			return fmt.Errorf("connect to browser: %w", err)
 		}
-		m.browserURL = fmt.Sprintf("http://127.0.0.1:%d/", port)
 
 	} else {
 		m.browserURL = m.RemoteBrowser.URL
@@ -166,6 +190,109 @@ func logBrowserVersion(ctx context.Context, log *zap.Logger, lightpanda bool) er
 	return nil
 }
 
+type browserKind int
+
+const (
+	browserChrome browserKind = iota
+	browserLightpanda
+)
+
+var lightpandaBinaries = []string{"lightpanda"}
+
+var chromeBinaries = []string{
+	"google-chrome",
+	"google-chrome-stable",
+	"chromium",
+	"chromium-browser",
+	"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
+	"/Applications/Chromium.app/Contents/MacOS/Chromium",
+}
+
+var defaultChromeFlags = []string{
+	"--headless",
+	"--disable-gpu",
+	"--no-first-run",
+	"--no-default-browser-check",
+	"--disable-background-networking",
+	"--enable-features=NetworkService,NetworkServiceInProcess",
+	"--disable-background-timer-throttling",
+	"--disable-backgrounding-occluded-windows",
+	"--disable-breakpad",
+	"--disable-client-side-phishing-detection",
+	"--disable-default-apps",
+	"--disable-dev-shm-usage",
+	"--disable-extensions",
+	"--disable-hang-monitor",
+	"--disable-ipc-flooding-protection",
+	"--disable-popup-blocking",
+	"--disable-prompt-on-repost",
+	"--disable-renderer-backgrounding",
+	"--disable-sync",
+	"--force-color-profile=srgb",
+	"--metrics-recording-only",
+	"--mute-audio",
+	"--safebrowsing-disable-auto-update",
+	"--enable-automation",
+	"--password-store=basic",
+	"--use-mock-keychain",
+}
+
+// resolveBrowser picks which browser binary to launch. If execPath is set the
+// caller's choice wins and the kind is inferred from the basename. Otherwise
+// PATH is searched: lightpanda first, then chrome variants.
+func resolveBrowser(execPath string) (string, browserKind, error) {
+	if execPath != "" {
+		base := strings.ToLower(filepath.Base(execPath))
+		if strings.Contains(base, "lightpanda") {
+			return execPath, browserLightpanda, nil
+		}
+		return execPath, browserChrome, nil
+	}
+	for _, name := range lightpandaBinaries {
+		if p, err := exec.LookPath(name); err == nil {
+			return p, browserLightpanda, nil
+		}
+	}
+	for _, name := range chromeBinaries {
+		if filepath.IsAbs(name) {
+			if _, err := os.Stat(name); err == nil {
+				return name, browserChrome, nil
+			}
+			continue
+		}
+		if p, err := exec.LookPath(name); err == nil {
+			return p, browserChrome, nil
+		}
+	}
+	return "", 0, fmt.Errorf("no browser found in PATH (tried lightpanda, chrome variants); set exec <path>")
+}
+
+// waitForBrowser polls /json/version on the launched browser until it answers
+// or the deadline expires.
+func waitForBrowser(remoteURL string, timeout time.Duration) error {
+	versionURL := strings.TrimSuffix(remoteURL, "/") + "/json/version"
+	deadline := time.Now().Add(timeout)
+	client := &http.Client{Timeout: 1 * time.Second}
+	var lastErr error
+	for time.Now().Before(deadline) {
+		resp, err := client.Get(versionURL)
+		if err == nil {
+			resp.Body.Close()
+			if resp.StatusCode == http.StatusOK {
+				return nil
+			}
+			lastErr = fmt.Errorf("status %d", resp.StatusCode)
+		} else {
+			lastErr = err
+		}
+		time.Sleep(50 * time.Millisecond)
+	}
+	if lastErr != nil {
+		return fmt.Errorf("timeout waiting for %s: %w", versionURL, lastErr)
+	}
+	return fmt.Errorf("timeout waiting for %s", versionURL)
+}
+
 func pickFreePort() (int, error) {
 	l, err := net.Listen("tcp", "127.0.0.1:0")
 	if err != nil {
@@ -202,23 +329,22 @@ func detectLightpanda(remoteURL string) bool {
 }
 
 func (m *Middleware) cleanup() {
-	if m.keepCancel != nil {
-		// Cancelling the keep-alive context ends the chromedp managed browser
-		// connection, which lets ExecAllocator's watchdog tear down chrome.
-		m.keepCancel()
-		m.keepCancel = nil
-	}
-	if m.allocCtx != nil {
-		timeoutCtx, c := context.WithTimeout(m.allocCtx, 10*time.Second)
-		defer c()
-		_ = chromedp.Cancel(timeoutCtx)
+	if m.cmd != nil && m.cmd.Process != nil {
+		_ = m.cmd.Process.Signal(os.Interrupt)
+		done := make(chan error, 1)
+		go func() { done <- m.cmd.Wait() }()
+		select {
+		case <-done:
+		case <-time.After(5 * time.Second):
+			_ = m.cmd.Process.Kill()
+			<-done
+		}
+		m.cmd = nil
 	}
-	if m.allocCancel != nil {
-		m.allocCancel()
-		m.allocCancel = nil
+	if m.tempDir != "" {
+		_ = os.RemoveAll(m.tempDir)
+		m.tempDir = ""
 	}
-	m.allocCtx = nil
-	m.keepAliveCtx = nil
 }
 
 func (m *Middleware) Cleanup() error {

middleware_serve_http_test.go

@@ -19,8 +19,15 @@ func TestMiddleware_ServeHTTP(t *testing.T) {
 		return http.ErrUseLastResponse
 	}
 	browserConfig := "exec"
+	isLightpanda := false
 	if u := os.Getenv("CADDY_CHROME_TEST_BROWSER_URL"); u != "" {
 		browserConfig = "url " + u
+		isLightpanda = true
+	} else if _, kind, err := resolveBrowser(""); err == nil && kind == browserLightpanda {
+		// exec mode picked lightpanda from PATH; pass the flag the test
+		// server's self-signed cert needs.
+		browserConfig = "exec -- --insecure-disable-tls-host-verification"
+		isLightpanda = true
 	}
 	tester.InitServer(fmt.Sprintf(`
 		{
@@ -55,7 +62,6 @@ func TestMiddleware_ServeHTTP(t *testing.T) {
 			file_server
 		}`, browserConfig), "caddyfile")
 
-	isLightpanda := os.Getenv("CADDY_CHROME_TEST_BROWSER_URL") != ""
 	for _, testCase := range []struct {
 		url              string
 		statusCode       int