Skip to content

Commit 16e2286

Browse files
committed
docs: Docs update
1 parent 2136084 commit 16e2286

9 files changed

Lines changed: 62 additions & 83 deletions

File tree

docs/src/data/changelog/v1.0.3/mark-many-as-read.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ version: "v1.0.3"
33
category: "experiments-added"
44
---
55

6-
#### `mark-many-as-read` — Mark many files as read in one step
6+
#### `mark-many-as-read`: mark many files as read in one step
77

88
Enable the new [`mark-many-as-read`](/reference/experiments) experiment to turn on two behaviors that each mark many files as read in a single call: automatic marking of files inside a local `terraform { source = "..." }` block, and the new `mark_glob_as_read` HCL function.
99

docs/src/data/strict-controls/fast-copy.mdx

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,10 @@ name: fast-copy
33
status: active
44
---
55

6-
Use a compile-once, single-walk implementation for `include_in_copy` and `exclude_from_copy` expansion when Terragrunt copies a module's source folder. Faster on large module sources because the walk no longer pre-expands every pattern in a separate pass per nested directory.
6+
Switches `include_in_copy` and `exclude_from_copy` pattern matching from [zglob](https://pkg.go.dev/github.com/mattn/go-zglob) to [gobwas/glob](https://pkg.go.dev/github.com/gobwas/glob) when Terragrunt copies a module's source folder.
77

88
### `fast-copy` - Reason
99

10-
`include_in_copy` and `exclude_from_copy` patterns have historically been expanded with [zglob](https://pkg.go.dev/github.com/mattn/go-zglob), once per pattern, with directory matches recursively re-expanded. On large source trees the pre-expansion can dominate the cost of the copy itself.
10+
The default implementation runs zglob once per pattern and recursively re-expands every directory match, which can dominate copy time on large module sources. gobwas compiles each pattern once and matches inline during the existing copy walk.
1111

12-
Enabling `fast-copy` switches pattern matching to [gobwas/glob](https://pkg.go.dev/github.com/gobwas/glob), compiles each pattern once, and evaluates matches inline during the existing copy walk.
13-
14-
The semantics differ from zglob in one place worth knowing: gobwas does not collapse `**` when one of its neighbors is a wildcard. For example, `a/**/*.tf` matches `a/sub/main.tf` but not `a/main.tf`. Patterns that depended on zero-depth `**` collapsing should use brace alternation, for example `{*.tf,**/*.tf}`, to cover both depths.
12+
gobwas does not collapse `**` when a neighbor is a wildcard, so `a/**/*.tf` matches `a/sub/main.tf` but not `a/main.tf`. Patterns that depended on zero-depth `**` collapsing need brace alternation like `{*.tf,**/*.tf}` to cover both depths.

internal/glob/glob.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@
1414
// literals. With literals on both sides, "a/**/b" matches "a/b" as well as
1515
// "a/x/b". If either neighbor is a wildcard (for example "a/**/*.tf" or
1616
// "*/**/b.tf"), "**" does not collapse and a zero-depth match fails. Use
17-
// brace alternation — for example "{*.tf,**/*.tf}" to cover both depths
18-
// when the trailing segment contains a wildcard.
17+
// brace alternation like "{*.tf,**/*.tf}" to cover both depths when the
18+
// trailing segment contains a wildcard.
1919
//
2020
// # When to use what
2121
//

internal/runner/run/file_copy_getter.go

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,9 @@ type FileCopyGetter struct {
3232
IncludeInCopy []string
3333
ExcludeFromCopy []string
3434

35-
// FastCopy opts the underlying [util.CopyFolderContents] call into the
36-
// compile-once, single-walk implementation. Set via the `fast-copy`
37-
// strict control at construction time.
35+
// FastCopy routes the [util.CopyFolderContents] call through the
36+
// fast-copy path. Set at construction time from the `fast-copy`
37+
// strict control.
3838
FastCopy bool
3939
}
4040

internal/strict/controls/controls.go

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -80,10 +80,8 @@ const (
8080
// DisableDependentModules is the control that prevents the use of the deprecated `--disable-dependent-modules` flag.
8181
DisableDependentModules = "disable-dependent-modules"
8282

83-
// FastCopy opts into a compile-once, single-walk implementation of
84-
// `include_in_copy` / `exclude_from_copy` expansion in
85-
// `util.CopyFolderContents`. Faster on large module sources, but
86-
// switches pattern matching from zglob to gobwas semantics.
83+
// FastCopy is the control that switches `include_in_copy` and
84+
// `exclude_from_copy` pattern matching from zglob to gobwas.
8785
FastCopy = "fast-copy"
8886
)
8987

@@ -298,7 +296,7 @@ func New() strict.Controls {
298296
},
299297
&Control{
300298
Name: FastCopy,
301-
Description: "Switches `include_in_copy` / `exclude_from_copy` expansion in `util.CopyFolderContents` to a compile-once, single-walk implementation. Pattern matching moves from zglob to gobwas semantics, so `**` no longer collapses when adjacent to a wildcard (for example `a/**/*.tf` will not match `a/foo.tf`). Use brace alternation such as `{*.tf,**/*.tf}` to cover both depths.",
299+
Description: "Switches `include_in_copy` and `exclude_from_copy` pattern matching from zglob to gobwas. `**` no longer collapses when adjacent to a wildcard, so `a/**/*.tf` will not match `a/foo.tf`. Use brace alternation like `{*.tf,**/*.tf}` to cover both depths.",
302300
Category: stageCategory,
303301
},
304302
}

internal/util/file.go

Lines changed: 30 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ func CanonicalResolvedPath(path, basePath string) (string, error) {
112112

113113
// GrepFilesWithSuffix returns true if regex matches the contents of any file
114114
// under rootDir whose name ends with suffix. The walk stops as soon as a match
115-
// is found. A missing rootDir is not an error the function returns false.
115+
// is found. A missing rootDir is not an error; the function returns false.
116116
func GrepFilesWithSuffix(fsys vfs.FS, regex *regexp.Regexp, rootDir, suffix string) (bool, error) {
117117
var found bool
118118

@@ -449,10 +449,10 @@ func CopyFolderContents(l log.Logger, source, destination, manifestFile string,
449449
})
450450
}
451451

452-
// copyFolderContentsFast is the compile-once, single-walk path for
453-
// [CopyFolderContents]. See the `fast-copy` strict control. The walk
454-
// itself runs via [fastwalk.Walk] so directory reads and filter checks
455-
// proceed in parallel.
452+
// copyFolderContentsFast is the [CopyFolderContents] path used when the
453+
// `fast-copy` strict control is enabled. Include and exclude patterns
454+
// are compiled once and the source tree is walked once through
455+
// [vfs.WalkDirParallel].
456456
func copyFolderContentsFast(
457457
l log.Logger,
458458
source,
@@ -491,9 +491,8 @@ func copyFolderContentsFast(
491491
}
492492
}()
493493

494-
// fastwalk runs walkFn concurrently across workers; the gob-encoded
495-
// manifest is not safe for concurrent writes, so every AddFile is
496-
// guarded by this mutex.
494+
// The walk is parallel. The gob-encoded manifest is not safe for
495+
// concurrent writes, so AddFile is guarded.
497496
var manifestMu sync.Mutex
498497

499498
walkFn := func(absolutePath string, d fs.DirEntry, walkErr error) error {
@@ -514,9 +513,8 @@ func copyFolderContentsFast(
514513

515514
isDir := d.IsDir()
516515

517-
// Preserve the cache-dir skip that expandGlobPath enforced at
518-
// expansion time. Must run before include matching — otherwise a
519-
// user include like "**" would pull .terragrunt-cache back in.
516+
// Skip .terragrunt-cache before include matching. A user
517+
// include like "**" would otherwise pull it back in.
520518
if strings.Contains(rel, TerragruntCacheDir) {
521519
if isDir {
522520
return fs.SkipDir
@@ -536,9 +534,9 @@ func copyFolderContentsFast(
536534
included := include.matches(rel)
537535

538536
if !included && TerragruntExcludes(filepath.FromSlash(rel)) {
539-
// Mirror the legacy ancestor pass: a directory on the path
540-
// toward a potential include match must still be descended
541-
// into, even when TerragruntExcludes would reject it.
537+
// A directory on the way to a potential include match must
538+
// still be descended into even when TerragruntExcludes
539+
// would reject it.
542540
if isDir && include.isAncestor(rel) {
543541
return nil
544542
}
@@ -558,9 +556,9 @@ func copyFolderContentsFast(
558556
return errors.New(err)
559557
}
560558

561-
// MkdirAll may already have created `dest` with default perms
562-
// from a sibling file-copy worker racing ahead. Chmod
563-
// normalizes the directory to the source's mode.
559+
// A sibling file-copy worker may have created `dest`
560+
// already with default perms. Chmod forces the source's
561+
// mode.
564562
if err := os.MkdirAll(dest, info.Mode().Perm()); err != nil {
565563
return errors.New(err)
566564
}
@@ -594,24 +592,22 @@ func copyFolderContentsFast(
594592
return nil
595593
}
596594

597-
// includePatterns holds the compiled include matcher plus a cheap
598-
// "is rel an ancestor of any potential match" predicate. Ancestor
599-
// acceptance is what makes the walk descend through dot-prefixed parents
600-
// like `_module/.region3` to reach an include hidden underneath.
595+
// includePatterns holds the compiled include matcher and an ancestor
596+
// predicate. The predicate lets the walk descend through dot-prefixed
597+
// parents like `_module/.region3` to reach an include below.
601598
type includePatterns struct {
602-
// match is a single combined matcher built by OR-ing every user
603-
// pattern as `{<p>,<p>/**}` inside one brace alternation. One
604-
// `Match` call per entry replaces N per-pattern checks.
599+
// match is a single matcher that OR-s every user pattern as
600+
// `{<p>,<p>/**}` so one Match call per entry covers all of them.
605601
match glob.Matcher
606602

607-
// ancestor is a combined matcher over the strict path prefixes of
608-
// each pattern that does not contain `**`. A rel matching this is a
609-
// directory on the way to a potential include match.
603+
// ancestor matches the path prefixes of every pattern that does
604+
// not contain `**`. A rel that matches is a directory on the way
605+
// to a potential include match.
610606
ancestor glob.Matcher
611607

612-
// descendAny is true if any pattern contains a `**` segment — in that
613-
// case any directory could be on the path to a match, so accept all
614-
// non-excluded paths as ancestors regardless of `ancestor`.
608+
// descendAny is true when any pattern contains a `**` segment. In
609+
// that case every directory is a possible ancestor, so `ancestor`
610+
// is not consulted.
615611
descendAny bool
616612
}
617613

@@ -631,11 +627,10 @@ func (p includePatterns) isAncestor(rel string) bool {
631627
return p.ancestor != nil && p.ancestor.Match(rel)
632628
}
633629

634-
// compileIncludePatterns compiles user `include_in_copy` patterns into one
635-
// combined matcher that covers each pattern and all its descendants — so a
636-
// match behaves like the recursive expansion in [expandGlobPath] — plus an
637-
// ancestor matcher that flags directories on the path toward a potential
638-
// match.
630+
// compileIncludePatterns compiles user `include_in_copy` patterns into
631+
// one combined matcher that covers each pattern and all its descendants,
632+
// reproducing the recursive expansion in [expandGlobPath], plus an
633+
// ancestor matcher for directories on the path toward a potential match.
639634
func compileIncludePatterns(patterns []string) (includePatterns, error) {
640635
out := includePatterns{}
641636

internal/util/file_test.go

Lines changed: 11 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -761,14 +761,14 @@ func TestRelPathForLog(t *testing.T) {
761761
}
762762
}
763763

764-
// buildCopyBenchTree lays out a synthetic module source with `topDirs`
765-
// top-level directories. Each top-level dir gets a deeply nested chain
766-
// (`chainDepth` levels) with `filesPerLevel` files at every level, so a
767-
// bare-directory include pattern like the top-level name triggers the
768-
// legacy expandGlobPath recursion once per nested directory.
764+
// buildCopyBenchTree lays out a synthetic module source: topDirs
765+
// top-level directories, each a chain chainDepth levels deep with
766+
// filesPerLevel files at every level. A bare-directory include pattern
767+
// like the top-level name triggers the legacy expandGlobPath recursion
768+
// once per nested directory.
769769
//
770-
// Returns the tree root and the list of top-level directory names, which
771-
// become the include-in-copy patterns fed to the benchmark.
770+
// Returns the tree root and the top-level directory names, which the
771+
// benchmark uses as include patterns.
772772
func buildCopyBenchTree(b *testing.B, topDirs, chainDepth, filesPerLevel int) (string, []string) {
773773
b.Helper()
774774

@@ -783,29 +783,20 @@ func buildCopyBenchTree(b *testing.B, topDirs, chainDepth, filesPerLevel int) (s
783783
current := filepath.Join(root, name)
784784

785785
for depth := range chainDepth {
786-
if err := os.MkdirAll(current, 0o755); err != nil {
787-
b.Fatalf("mkdir: %v", err)
788-
}
786+
require.NoError(b, os.MkdirAll(current, 0o755))
789787

790788
for f := range filesPerLevel {
791789
p := filepath.Join(current, fmt.Sprintf("f%02d.tf", f))
792-
if err := os.WriteFile(p, content, 0o644); err != nil {
793-
b.Fatalf("write: %v", err)
794-
}
790+
require.NoError(b, os.WriteFile(p, content, 0o644))
795791
}
796792

797793
current = filepath.Join(current, fmt.Sprintf("level%02d", depth))
798794
}
799795
}
800796

801797
cache := filepath.Join(root, util.TerragruntCacheDir, "should-be-skipped")
802-
if err := os.MkdirAll(cache, 0o755); err != nil {
803-
b.Fatalf("mkdir cache: %v", err)
804-
}
805-
806-
if err := os.WriteFile(filepath.Join(cache, "skip.tf"), content, 0o644); err != nil {
807-
b.Fatalf("write cache file: %v", err)
808-
}
798+
require.NoError(b, os.MkdirAll(cache, 0o755))
799+
require.NoError(b, os.WriteFile(filepath.Join(cache, "skip.tf"), content, 0o644))
809800

810801
return root, names
811802
}

internal/util/grep_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ func TestGrepFilesWithSuffix(t *testing.T) {
4747
{
4848
name: "no match when suffix differs",
4949
files: map[string]string{
50-
// .tf, not .tf.json must be skipped.
50+
// .tf, not .tf.json, must be skipped.
5151
"/mod/backend.tf": `terraform { backend "s3" {} }`,
5252
},
5353
root: "/mod",

internal/vfs/vfs.go

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -147,18 +147,15 @@ func TryLock(fs FS, name string) (Unlocker, bool, error) {
147147
return locker.TryLock(name)
148148
}
149149

150-
// WalkDirParallel walks the file tree rooted at root the same way
151-
// [WalkDir] does, but reads directories in parallel on filesystems where
152-
// that is safe. On a [NewOSFS] filesystem the walk is driven by
153-
// [fastwalk.Walk]; on any other FS (for example [NewMemMapFS]) it
154-
// transparently degrades to the sequential [WalkDir].
150+
// WalkDirParallel walks the file tree rooted at root like [WalkDir]
151+
// does. On a [NewOSFS] filesystem it reads directories in parallel via
152+
// [fastwalk.Walk]. On any other FS, including [NewMemMapFS], it falls
153+
// back to the sequential [WalkDir].
155154
//
156-
// Unlike [WalkDir], the parallel walk does not guarantee any ordering
157-
// across directories — fn may be called concurrently from multiple
158-
// goroutines. Callers that care about deterministic order, or that write
159-
// to shared state from fn, must use [WalkDir] or serialize access
160-
// themselves. [fs.SkipDir] and [fs.SkipAll] continue to work as with the
161-
// sequential walk.
155+
// The parallel walk calls fn concurrently from multiple goroutines and
156+
// gives no ordering guarantee across directories. Callers that depend
157+
// on deterministic order, or that write to shared state from fn, must
158+
// use [WalkDir] or serialize access themselves.
162159
func WalkDirParallel(fsys FS, root string, fn fs.WalkDirFunc) error {
163160
if _, ok := fsys.(*osFS); !ok {
164161
return WalkDir(fsys, root, fn)

0 commit comments

Comments
 (0)