Skip to content

Commit b8b3950

Browse files
committed
Add redact CLI
The package was only usable from Go code. Add a `cmd/redact` binary that accepts text via positional args or stdin and prints the redacted output. Flags map to `Options` (`--mask`, `--min-entropy`, `--min-submatch`); stdin is streamed line by line with `bufio.Scanner` so arbitrarily large inputs do not buffer in memory. Tests use `rogpeppe/go-internal/testscript` (the same pattern as xk6-docs): each `testdata/scripts/*.txtar` invokes the built binary and asserts on stdout. Also adds a minimal `.golangci.yml` so future changes have a single, version-pinned lint command.
1 parent 200369f commit b8b3950

18 files changed

Lines changed: 415 additions & 56 deletions

.github/workflows/build.yml

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: build
2+
3+
on:
4+
push:
5+
branches: [main]
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
test:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v4
16+
- uses: actions/setup-go@v5
17+
with:
18+
go-version-file: go.mod
19+
- run: go test ./...
20+
21+
build:
22+
needs: test
23+
runs-on: ubuntu-latest
24+
strategy:
25+
matrix:
26+
include:
27+
- { goos: linux, goarch: amd64 }
28+
- { goos: linux, goarch: arm64 }
29+
- { goos: darwin, goarch: amd64 }
30+
- { goos: darwin, goarch: arm64 }
31+
- { goos: windows, goarch: amd64, ext: .exe }
32+
steps:
33+
- uses: actions/checkout@v4
34+
- uses: actions/setup-go@v5
35+
with:
36+
go-version-file: go.mod
37+
- name: build
38+
env:
39+
GOOS: ${{ matrix.goos }}
40+
GOARCH: ${{ matrix.goarch }}
41+
CGO_ENABLED: "0"
42+
run: |
43+
mkdir -p dist
44+
go build -trimpath -ldflags='-s -w' \
45+
-o "dist/redact-${{ matrix.goos }}-${{ matrix.goarch }}${{ matrix.ext }}" \
46+
./cmd/redact
47+
- uses: actions/upload-artifact@v4
48+
with:
49+
name: redact-${{ matrix.goos }}-${{ matrix.goarch }}
50+
path: dist/*
51+
if-no-files-found: error

.golangci.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
version: "2"
2+
3+
linters:
4+
default: none
5+
enable:
6+
- errcheck
7+
- govet
8+
- ineffassign
9+
- staticcheck
10+
- unused
11+
- misspell
12+
- unconvert
13+
- unparam
14+
- revive
15+
- gocritic

README.md

Lines changed: 78 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,104 @@
11
# redact
22

3-
Replaces secrets in text with `[REDACTED]`.
3+
Hide secrets in text. Keeps the same length, so logs and dumps stay readable.
4+
5+
```
6+
$ redact 'curl -H "Authorization: Bearer ghp_abcdefghijklmnopqrstuvwxyz0123456789" api.github.com'
7+
curl -H "Authorization: Bearer ****************************************" api.github.com
8+
```
49

510
## Install
611

12+
Grab a prebuilt binary from the [Actions tab](../../actions) (latest `build` run, under "Artifacts"), or build from source:
13+
14+
```
15+
go install github.com/inancgumus/redact/cmd/redact@latest
16+
```
17+
18+
## Use the CLI
19+
20+
Pass text as an argument:
21+
22+
```
23+
$ redact 'user=alice token=ghp_abcdefghijklmnopqrstuvwxyz0123456789 status=ok'
24+
user=alice token=**************************************** status=ok
25+
```
26+
27+
Or pipe a file:
28+
729
```
8-
go get github.com/inancgumus/redact
30+
$ redact < ~/.aws/credentials
31+
[default]
32+
aws_access_key_id = ********************
33+
aws_secret_access_key = ****************************************
934
```
1035

11-
## Use
36+
Database URLs keep their structure:
37+
38+
```
39+
$ redact 'postgresql://app:s3cretP4ss@db.internal:5432/prod'
40+
postgresql://app:**********@db.internal:5432/prod
41+
```
42+
43+
Run with no input to see the help.
44+
45+
### Flags
46+
47+
| Flag | Default | What it does |
48+
|------|---------|--------------|
49+
| `--mask` | `*` | Character used to hide each byte of a secret. |
50+
| `--min-entropy` | `3.5` | How random a value must look before it counts as a secret. Lower means more aggressive. |
51+
| `--min-submatch` | `2` | How strong a match must be before unknown secrets get hidden. Higher means more cautious. |
52+
53+
### Examples
54+
55+
Custom mask character:
56+
57+
```
58+
$ redact --mask=# 'token: ghp_abcdefghijklmnopqrstuvwxyz0123456789'
59+
token: ########################################
60+
```
61+
62+
Catch a wider net of unknown-looking values in a config file:
63+
64+
```
65+
$ redact --min-entropy=2.0 < config.yaml
66+
```
67+
68+
## Use as a Go library
1269

1370
```go
71+
import "github.com/inancgumus/redact"
72+
1473
clean := redact.String(text, redact.DefaultOptions)
1574
```
1675

17-
Override defaults via `Options`:
76+
Tune the defaults:
1877

1978
```go
2079
clean := redact.String(text, redact.Options{
21-
// Placeholder written in place of each detected secret.
22-
// Default: "[REDACTED]".
23-
Mask: "***",
24-
25-
// Minimum Shannon entropy (bits/char) a captured value must have
26-
// to be treated as a secret by the generic detector. Lower values
27-
// redact more aggressively; raise it to cut false positives on
28-
// low-entropy strings. Default: 3.5.
29-
MinEntropy: 4.0,
30-
31-
// Minimum number of regex submatches required before a generic
32-
// match is considered for redaction. Default: 2.
33-
MinSubmatch: 2,
80+
Mask: '#', // Character repeated for each byte. Default '*'.
81+
MinEntropy: 4.0, // Raise to cut false positives. Default 3.5.
82+
MinSubmatch: 2, // Raise to be more cautious on unknown patterns. Default 2.
3483
})
3584
```
3685

37-
## Catches
86+
Any zero field falls back to the default.
87+
88+
## What it catches
3889

39-
Shape-based catches:
90+
### Patterns by shape
4091

4192
- PEM private keys
4293
- JWTs
43-
- `Bearer` auth headers
44-
- `Basic` auth headers
94+
- `Bearer` and `Basic` auth headers
4595
- URL credentials (`user:pass@host`)
4696
- `btoa("user:pass")` calls
4797
- `password` / `secret` / `credential` assignments
48-
- Query-string secrets (`?token=...`, `?key=...`, etc.)
49-
- Generic high-entropy values near words like `key`, `secret`, `token`
98+
- Query-string secrets (`?token=...`, `?key=...`)
99+
- High-entropy values near words like `key`, `secret`, `token`
50100

51-
Provider-token catches (prefix-anchored):
101+
### Provider tokens (prefix-anchored)
52102

53103
- AWS access keys (`AKIA`, `ASIA`, `ABIA`, `ACCA`, `A3T...`)
54104
- AWS secret keys (keyword-anchored 40-char base64)
@@ -159,9 +209,9 @@ Provider-token catches (prefix-anchored):
159209
- HashiCorp Terraform (`<id>.atlasv1.<...>`)
160210
- Telegram bot token
161211

162-
## Skips
212+
### What it skips
163213

164-
- Variable expansions (`${...}`, `$(...)`)
214+
- Variable expansions like `${SECRET}` or `$(cat secret)`
165215
- UUIDs
166-
- Dotted property access (`apiClient.token`)
167-
- Values containing stopwords (`example`, `test`, `placeholder`, etc.)
216+
- Dotted property access like `apiClient.token`
217+
- Values containing words like `example`, `test`, `placeholder`

cmd/redact/main.go

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
// Command redact reads text from positional args or stdin and writes a
2+
// copy with detected secrets replaced by a placeholder.
3+
package main
4+
5+
import (
6+
"bufio"
7+
"errors"
8+
"flag"
9+
"fmt"
10+
"io"
11+
"os"
12+
"strings"
13+
14+
"github.com/inancgumus/redact"
15+
)
16+
17+
func main() {
18+
if err := run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr); err != nil {
19+
fmt.Fprintln(os.Stderr, "redact:", err)
20+
os.Exit(1)
21+
}
22+
}
23+
24+
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) error {
25+
fs := flag.NewFlagSet("redact", flag.ContinueOnError)
26+
fs.SetOutput(stderr)
27+
fs.Usage = func() {
28+
_, _ = fmt.Fprintln(stderr, "usage: redact [flags] [text...]")
29+
_, _ = fmt.Fprintln(stderr, "reads text from args; if none, reads from stdin.")
30+
fs.PrintDefaults()
31+
}
32+
33+
opts := redact.DefaultOptions
34+
fs.Var((*runeFlag)(&opts.Mask), "mask",
35+
"character repeated for each byte of a secret")
36+
fs.Float64Var(&opts.MinEntropy, "min-entropy", opts.MinEntropy,
37+
"how random a value must look to be redacted (lower = redacts more)")
38+
fs.IntVar(&opts.MinSubmatch, "min-submatch", opts.MinSubmatch,
39+
"how strong a match must be to redact unknown secrets (higher = redacts less)")
40+
41+
if err := fs.Parse(args); err != nil {
42+
if errors.Is(err, flag.ErrHelp) {
43+
return nil
44+
}
45+
return err
46+
}
47+
48+
if rest := fs.Args(); len(rest) > 0 {
49+
_, err := io.WriteString(stdout, redact.String(strings.Join(rest, " "), opts))
50+
return err
51+
}
52+
53+
// Peek so empty stdin prints usage instead of silently exiting.
54+
// The peeked byte stays buffered for ReadAll.
55+
br := bufio.NewReader(stdin)
56+
if _, err := br.Peek(1); err != nil {
57+
fs.Usage()
58+
return nil
59+
}
60+
61+
buf, err := io.ReadAll(br)
62+
if err != nil {
63+
return fmt.Errorf("read stdin: %w", err)
64+
}
65+
_, err = io.WriteString(stdout, redact.String(string(buf), opts))
66+
return err
67+
}
68+
69+
// runeFlag adapts a *rune to the flag.Value interface so the mask flag
70+
// accepts exactly one character.
71+
type runeFlag rune
72+
73+
func (r *runeFlag) String() string {
74+
if r == nil || *r == 0 {
75+
return ""
76+
}
77+
return string(*r)
78+
}
79+
80+
func (r *runeFlag) Set(s string) error {
81+
runes := []rune(s)
82+
if len(runes) != 1 {
83+
return fmt.Errorf("mask must be one character, got %q", s)
84+
}
85+
*r = runeFlag(runes[0])
86+
return nil
87+
}

cmd/redact/main_test.go

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"os"
6+
"os/exec"
7+
"path/filepath"
8+
"testing"
9+
10+
"github.com/rogpeppe/go-internal/testscript"
11+
)
12+
13+
func buildRedactBinary(ctx context.Context, t *testing.T) string {
14+
t.Helper()
15+
16+
cacheDir, err := os.UserCacheDir()
17+
if err != nil {
18+
t.Fatalf("cache dir: %v", err)
19+
}
20+
dir := filepath.Join(cacheDir, "redact-test")
21+
if err := os.MkdirAll(dir, 0o755); err != nil {
22+
t.Fatalf("mkdir: %v", err)
23+
}
24+
25+
bin := filepath.Join(dir, "redact")
26+
build := exec.CommandContext(ctx, "go", "build", "-o", bin, ".")
27+
out, err := build.CombinedOutput()
28+
if err != nil {
29+
t.Fatalf("build redact: %v\n%s", err, out)
30+
}
31+
return bin
32+
}
33+
34+
func TestScripts(t *testing.T) {
35+
t.Parallel()
36+
37+
ctx := t.Context()
38+
bin := buildRedactBinary(ctx, t)
39+
40+
testscript.Run(t, testscript.Params{
41+
Dir: "testdata/scripts",
42+
Setup: func(env *testscript.Env) error {
43+
return os.Symlink(bin, filepath.Join(env.WorkDir, "redact"))
44+
},
45+
UpdateScripts: os.Getenv("UPDATE_GOLDEN") != "",
46+
})
47+
}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Redacts a secret passed as a positional arg.
2+
exec ./redact AKIAIOSFODNN7EXAMPLE
3+
! stdout 'AKIAIOSFODNN7EXAMPLE'
4+
stdout '\*+'
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# High --min-entropy suppresses the generic detector.
2+
exec ./redact --min-entropy=10 'api_key="abcdefghij1234567890"'
3+
! stdout '\*'
4+
stdout 'abcdefghij1234567890'
5+
6+
# Low --min-entropy redacts aggressively.
7+
exec ./redact --min-entropy=1 'api_key="abcdefghij1234567890"'
8+
stdout '\*+'
9+
! stdout 'abcdefghij1234567890'
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Default --min-submatch matches generic high-entropy values.
2+
exec ./redact 'api_key="aB3kPq9xZ7mNvR2tYuIo"'
3+
stdout '\*+'
4+
5+
# Raising --min-submatch above the captured group count disables generic matches.
6+
exec ./redact --min-submatch=99 'api_key="aB3kPq9xZ7mNvR2tYuIo"'
7+
! stdout '\*'
8+
stdout 'aB3kPq9xZ7mNvR2tYuIo'
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Multiple positional args are joined with a space.
2+
exec ./redact 'before' AKIAIOSFODNN7EXAMPLE 'after'
3+
stdout '^before \*+ after$'
4+
! stdout 'AKIAIOSFODNN7EXAMPLE'
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# No args and empty stdin prints usage to stderr and exits 0.
2+
exec ./redact
3+
stderr 'usage: redact'
4+
stderr '-mask'
5+
stderr '-min-entropy'
6+
stderr '-min-submatch'
7+
! stdout .
8+
9+
# Explicit -h also prints usage and exits 0.
10+
exec ./redact -h
11+
stderr 'usage: redact'
12+
! stdout .

0 commit comments

Comments
 (0)