forked from GoogleCloudPlatform/gcsfuse
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathparams.yaml
More file actions
1357 lines (1202 loc) · 48.5 KB
/
Copy pathparams.yaml
File metadata and controls
1357 lines (1202 loc) · 48.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#################################### DOCUMENTATION STARTS ######################
# Params structure
# flag-name: Name of the CLI flag.
# config-path: Location of the param in the config file. A value of "gcs-auth.anonymous-access" indicates that the param will be present under the gcs-auth:anonymous-access.
# type: data type of the param - supports the following values: ["int", "float64", "bool", "string", "duration", "octal", "[]int",
# "[]string", "logSeverity", "protocol", "resolvedPath"]
# usage: The usage doc that will appear in the helpdoc
# default: The default value of the param.
# deprecated: Specifies whether the param is deprecated. This will cause warnings when the user specifies the flag.
# deprecationWarning: The warning that should be issued when the user specifies a deprecated flag.
# hide-flag: Whether to hide flag from the helpdoc.
#
# The params must be kept sorted in ascending order.
# The linear order between any two params, p1 and p2 is defined as following:
# 1. If p1.config-path != "" && p2.config-path != "" then p1 < p2 iff p1.config-path < p2.config-path else p2 < p1
# 2. If p1.config-path == "" && p2.config-path == "" then p1 < p2 iff p1.flag-name < p2.flag-name else p2 < p1
# 3. If p1.config-path != "" && p2.config-path == "" then p1 < p2 and vice versa.
#
###################################### DOCUMENTATION ENDS ######################
# Keep this section at the top of the file.
machine-type-groups:
high-performance:
- "a2-megagpu-16g"
- "a2-ultragpu-8g"
- "a3-edgegpu-8g"
- "a3-highgpu-8g"
- "a3-megagpu-8g"
- "a3-ultragpu-8g"
- "a4-highgpu-8g"
- "a4-highgpu-8g-lowmem"
- "a4-highgpu-8g-nolssd"
- "a4x-highgpu-4g"
- "a4x-highgpu-4g-nolssd"
- "a4x-maxgpu-4g-metal"
- "ct5l-hightpu-8t"
- "ct5lp-hightpu-8t"
- "ct5p-hightpu-4t"
- "ct5p-hightpu-4t-tpu"
- "ct6e-standard-4t"
- "ct6e-standard-4t-tpu"
- "ct6e-standard-8t"
- "ct6e-standard-8t-tpu"
- "tpu7x-standard-4t"
- "tpu7x-standard-4t-tpu"
- "tpu7x-ultranet-4t"
- "tpu7x-ultranet-4t-tpu"
# Add other machine types here as needed.
# Add other groups here as needed
params:
- config-path: "app-name"
flag-name: "app-name"
type: "string"
usage: "The application name of this mount."
default: ""
- config-path: "cache-dir"
flag-name: "cache-dir"
type: "resolvedPath"
usage: "Enables file-caching. Specifies the directory to use for file-cache."
- config-path: "cloud-profiler.allocated-heap"
flag-name: "cloud-profiler-allocated-heap"
type: "bool"
usage: "Enables allocated heap (HeapProfileAllocs) profiling. This only works when --enable-cloud-profiler is set to true."
default: true
hide-flag: true
- config-path: "cloud-profiler.cpu"
flag-name: "cloud-profiler-cpu"
type: "bool"
usage: "Enables cpu profiling. This only works when --enable-cloud-profiler is set to true."
default: true
hide-flag: true
- config-path: "cloud-profiler.enabled"
flag-name: "enable-cloud-profiler"
type: "bool"
usage: "Enables cloud-profiler, by default disabled."
default: false
hide-flag: true
- config-path: "cloud-profiler.goroutines"
flag-name: "cloud-profiler-goroutines"
type: "bool"
usage: "Enables goroutines cloud-profiler. This only works when --enable-cloud-profiler is set to true."
default: false
hide-flag: true
- config-path: "cloud-profiler.heap"
flag-name: "cloud-profiler-heap"
type: "bool"
usage: "Enables heap cloud-profiler. This only works when --enable-cloud-profiler is set to true."
default: true
hide-flag: true
- config-path: "cloud-profiler.label"
flag-name: "cloud-profiler-label"
type: "string"
usage: >-
Allow setting a profile label to uniquely identify and compare cloud-profiler data with other profiles.
This only works when --enable-cloud-profiler is set to true.
default: "gcsfuse-0.0.0"
hide-flag: true
- config-path: "cloud-profiler.mutex"
flag-name: "cloud-profiler-mutex"
type: "bool"
usage: "Enables mutex cloud-profiler. This only works when --enable-cloud-profiler is set to true."
default: false
hide-flag: true
- config-path: "cloud-profiler.service-name"
flag-name: "cloud-profiler-service-name"
type: "string"
usage: "The service name for cloud-profiler. This only works when --enable-cloud-profiler is set to true."
default: "gcsfuse"
hide-flag: true
- config-path: "debug.exit-on-invariant-violation"
flag-name: "debug_invariants"
type: "bool"
usage: "Exit when internal invariants are violated."
default: false
- config-path: "debug.fuse"
flag-name: "debug_fuse"
type: "bool"
usage: "Enables debug logs."
default: false
deprecated: true
deprecation-warning: "Please set log-severity to TRACE instead."
- config-path: "debug.gcs"
flag-name: "debug_gcs"
type: "bool"
usage: "Enables debug logs."
default: false
deprecated: true
deprecation-warning: "Please set log-severity to TRACE instead."
- config-path: "debug.log-mutex"
flag-name: "debug_mutex"
type: "bool"
usage: "Print debug messages when a mutex is held too long."
default: false
- config-path: "disable-autoconfig"
flag-name: "disable-autoconfig"
type: "bool"
usage: "Disable optimizing configuration automatically for a machine"
default: false
hide-flag: true
- config-path: "disable-list-access-check"
flag-name: "disable-list-access-check"
type: "bool"
usage: "Disables the list object based access check during mount operation"
default: true
hide-flag: true
- config-path: "dummy-io.enable"
flag-name: "enable-dummy-io"
type: "bool"
usage: >-
Enable dummy I/O mode for testing purposes. In this mode all reads
and writes are simulated and no actual data is transferred to or from
Cloud Storage. All the metadata operations like object listing and stats
are real.
default: false
hide-flag: true
- config-path: "dummy-io.per-mb-latency"
flag-name: "dummy-io-per-mb-latency"
type: "duration"
usage: >-
Simulates reading from the reader latency in dummy I/O mode. This value
is only used when dummy I/O mode is enabled.
default: "0s"
hide-flag: true
- config-path: "dummy-io.reader-latency"
flag-name: "dummy-io-reader-latency"
type: "duration"
usage: >-
Simulates reader creation latency in dummy I/O mode. This value
is only used when dummy I/O mode is enabled.
default: "0s"
hide-flag: true
- config-path: "enable-atomic-rename-object"
flag-name: "enable-atomic-rename-object"
type: "bool"
usage: "Enables support for atomic rename object operation on HNS bucket."
default: true
hide-flag: true
- flag-name: "enable-google-lib-auth"
config-path: "enable-google-lib-auth"
type: "bool"
usage: "Enable google library authentication method to fetch the credentials"
default: true
hide-flag: true
- config-path: "enable-hns"
flag-name: "enable-hns"
type: "bool"
usage: "Enables support for HNS buckets"
default: true
hide-flag: true
- config-path: "enable-new-reader"
flag-name: "enable-new-reader"
type: "bool"
usage: "Enables support for new reader implementation."
default: true
hide-flag: true
- config-path: "enable-standard-symlinks"
flag-name: "enable-standard-symlinks"
type: "bool"
usage: >-
Enables the creation and reading of symbolic links using the standard GCS representation.
When enabled, new symlinks created via GCSFuse mount ensure compatibility with other
GCS clients like Storage Transfer Service (STS).
default: true
hide-flag: true
- config-path: "enable-type-cache-deprecation"
flag-name: "enable-type-cache-deprecation"
type: "bool"
usage: "Enables support to deprecate type cache."
default: true
hide-flag: true
- config-path: "enable-unsupported-path-support"
flag-name: "enable-unsupported-path-support"
type: "bool"
usage: >-
Enables support for file system paths with unsupported GCS names (e.g., names containing '//' or starting with /).
When set, GCSFuse will ignore these objects during listing and copying operations.
For rename and delete operations, the flag allows the action to proceed for all specified objects, including those with unsupported names.
default: true
hide-flag: true
- config-path: "file-cache.cache-file-for-range-read"
flag-name: "file-cache-cache-file-for-range-read"
type: "bool"
usage: "Whether to cache file for range reads."
default: false
optimizations:
profiles:
- name: "aiml-serving"
value: true
- name: "aiml-checkpointing"
value: true
- config-path: "file-cache.download-chunk-size-mb"
flag-name: "file-cache-download-chunk-size-mb"
type: "int"
usage: "Size of chunks in MiB that each concurrent request downloads."
default: "200"
- config-path: "file-cache.enable-crc"
flag-name: "file-cache-enable-crc"
type: "bool"
usage: "Performs CRC to ensure that file is correctly downloaded into cache. No op for rapid storage."
default: false
hide-flag: true
- config-path: "file-cache.enable-experimental-shared-chunk-cache"
flag-name: "enable-experimental-shared-chunk-cache"
type: "bool"
usage: >-
[EXPERIMENTAL] Enable chunk-based shared cache that allows multiple
gcsfuse mount instances to safely share the same cache directory (e.g., on NFS).
Uses fixed size chunks with atomic operations (write + rename) to download chunk
file without locks. Ideal for distributed environments where multiple nodes need
to share cached GCS data.
default: false
hide-flag: false
- config-path: "file-cache.enable-o-direct"
flag-name: "file-cache-enable-o-direct"
type: "bool"
usage: "Whether to use O_DIRECT while writing to file-cache in case of parallel downloads."
default: "false"
hide-flag: true
- config-path: "file-cache.enable-parallel-downloads"
flag-name: "file-cache-enable-parallel-downloads"
type: "bool"
usage: "Enable parallel downloads."
default: false
- config-path: "file-cache.exclude-regex"
flag-name: "file-cache-exclude-regex"
type: "string"
usage: "Exclude file paths (in the format bucket_name/object_key) specified by this regex from file caching."
default: ""
- config-path: "file-cache.experimental-disable-size-calculation-fix"
flag-name: "file-cache-experimental-disable-size-calculation-fix"
type: "bool"
usage: "Disable the fix in calculation of disk-utilization of file-cache."
default: false
hide-flag: true
- config-path: "file-cache.experimental-enable-chunk-cache"
flag-name: "file-cache-experimental-enable-chunk-cache"
type: "bool"
usage: "Enable chunk cache mode for random I/O optimization that downloads only requested blocks."
default: false
hide-flag: true
- config-path: "file-cache.experimental-parallel-downloads-default-on"
flag-name: "file-cache-experimental-parallel-downloads-default-on"
type: "bool"
usage: "Enable parallel downloads by default on experimental basis."
default: true
hide-flag: true
- config-path: "file-cache.include-regex"
flag-name: "file-cache-include-regex"
type: "string"
usage: "Include file paths (in the format bucket_name/object_key) specified by this regex for file caching."
default: ""
- config-path: "file-cache.max-parallel-downloads"
flag-name: "file-cache-max-parallel-downloads"
type: "int"
usage: "Sets an uber limit of number of concurrent file download requests that are made across all files."
default: "DefaultMaxParallelDownloads()"
- config-path: "file-cache.max-size-mb"
flag-name: "file-cache-max-size-mb"
type: "int"
usage: "Maximum size of the file-cache in MiBs"
default: "-1"
- config-path: "file-cache.parallel-downloads-per-file"
flag-name: "file-cache-parallel-downloads-per-file"
type: "int"
usage: "Number of concurrent download requests per file."
default: "16"
- config-path: "file-cache.shared-cache-chunk-size-mb"
flag-name: "file-cache-shared-cache-chunk-size-mb"
type: "int"
usage: "Chunk size in MiBs for shared chunk cache. Each chunk is downloaded on-demand."
default: "8"
hide-flag: true
- config-path: "file-cache.write-buffer-size"
flag-name: "file-cache-write-buffer-size"
type: "int"
usage: "Size of in-memory buffer that is used per goroutine in parallel downloads while writing to file-cache."
default: "4194304" # 4MiB
hide-flag: true
- config-path: "file-system.congestion-threshold"
flag-name: "congestion-threshold"
type: "int"
usage: >-
Sets the congestion threshold for background requests. When the number of
outstanding requests exceeds this threshold, the kernel may start blocking
new requests. 0 means system default (typically 75% of max-background; 9).
default: "0"
hide-flag: true
optimizations:
bucket-type-optimization:
- bucket-type: "zonal"
value: "DefaultCongestionThreshold()"
- bucket-type: "pirlo"
value: "DefaultCongestionThreshold()"
- config-path: "file-system.dir-mode"
flag-name: "dir-mode"
type: "octal"
usage: "Permissions bits for directories, in octal."
default: "0755"
- config-path: "file-system.disable-parallel-dirops"
flag-name: "disable-parallel-dirops"
type: "bool"
usage: "Specifies whether to allow parallel dir operations (lookups and readers)"
default: false
hide-flag: true
- config-path: "file-system.enable-kernel-reader"
flag-name: "enable-kernel-reader"
type: "bool"
usage: "Enables kernel reader, disables prefetching gcsfuse side and relies on kernel read-ahead and page-cache."
default: false
hide-flag: true
optimizations:
bucket-type-optimization:
- bucket-type: "zonal"
value: true
- bucket-type: "pirlo"
value: true
- config-path: "file-system.experimental-enable-dentry-cache"
flag-name: "experimental-enable-dentry-cache"
type: "bool"
usage: >-
When enabled, it sets the Dentry cache entry timeout same as metadata-cache-ttl.
This enables kernel to use cached entry to map the file paths to inodes,
instead of making LookUpInode calls to GCSFuse.
default: false
hide-flag: true
- config-path: "file-system.experimental-enable-pirlo"
flag-name: "experimental-enable-pirlo"
type: "bool"
usage: "Enables support for pirlo."
default: false
hide-flag: true
- config-path: "file-system.experimental-enable-readdirplus"
flag-name: "experimental-enable-readdirplus"
type: "bool"
usage: "Enables ReadDirPlus capability"
default: false
hide-flag: true
- config-path: "file-system.experimental-o-direct"
flag-name: "experimental-o-direct"
type: "bool"
usage: >-
Experimental: Bypasses the kernel's page cache for file reads and writes. When enabled,
all I/O operations are sent directly to the GCSFuse process.
default: false
hide-flag: true
- config-path: "file-system.file-mode"
flag-name: "file-mode"
type: "octal"
usage: "Permissions bits for files, in octal."
default: "0644"
- config-path: "file-system.fuse-options"
flag-name: "o"
type: "[]string"
usage: "Additional system-specific mount options. Multiple options can be passed as comma separated. For readonly, use --o ro"
- config-path: "file-system.gid"
flag-name: "gid"
type: "int"
default: -1
usage: "GID owner of all inodes."
- config-path: "file-system.ignore-interrupts"
flag-name: "ignore-interrupts"
type: "bool"
usage: >-
Instructs gcsfuse to ignore system interrupt signals (like SIGINT, triggered
by Ctrl+C). This prevents those signals from immediately terminating gcsfuse
inflight operations.
default: true
- config-path: "file-system.inactive-mrd-cache-size"
flag-name: "inactive-mrd-cache-size"
type: "int"
usage: >-
Sets the cache-size of inactive (no open file) MRD instances. When this limit
is exceeded, the least recently inactive MRD instances will be closed. Set to
0 to disable the cache, which will keep all the inactive MRD instances open forever.
default: "1000"
hide-flag: true
- config-path: "file-system.kernel-list-cache-ttl-secs"
flag-name: "kernel-list-cache-ttl-secs"
type: "int"
usage: >-
How long the directory listing (output of ls <dir>) should be cached in the
kernel page cache. If a particular directory cache entry is kept by kernel
for longer than TTL, then it will be sent for invalidation by gcsfuse on
next opendir (comes in the start, as part of next listing) call. 0 means no
caching. Use -1 to cache for lifetime (no ttl). Negative value other than -1
will throw error.
default: "0"
optimizations:
profiles:
- name: "aiml-serving"
value: -1
- config-path: "file-system.kernel-params-file"
flag-name: "kernel-params-file"
type: "resolvedPath"
usage: >-
File path used to communicate various kernel parameters to CSI Driver in GKE environment.
hide-flag: true
- config-path: "file-system.max-background"
flag-name: "max-background"
type: "int"
usage: >-
Sets the maximum number of outstanding background requests (e.g., request corresponding to
kernel readahead, writeback cache etc.) that the kernel will send to the FUSE daemon. 0 means
system default
(typically 12).
default: "0"
hide-flag: true
optimizations:
bucket-type-optimization:
- bucket-type: "zonal"
value: "DefaultMaxBackground()"
- bucket-type: "pirlo"
value: "DefaultMaxBackground()"
- config-path: "file-system.max-read-ahead-kb"
flag-name: "max-read-ahead-kb"
type: "int"
usage: >-
Sets max kernel-read-ahead for the mount in KiB. 0 means system default.
Requires sudo permission to set this value, otherwise the value will be ignored
and system default will be used.
default: "0"
hide-flag: true
optimizations:
bucket-type-optimization:
- bucket-type: "zonal"
value: 16384 # 16 MiB
- bucket-type: "pirlo"
value: 16384 # 16 MiB
- config-path: "file-system.rename-dir-limit"
flag-name: "rename-dir-limit"
type: "int"
usage: "Allow rename a directory containing fewer descendants than this limit."
default: "0"
optimizations:
machine-based-optimization:
- group: "high-performance"
value: 200000
profiles:
- name: "aiml-checkpointing"
value: 200000
- config-path: "file-system.temp-dir"
flag-name: "temp-dir"
type: "resolvedPath"
usage: >-
Path to the temporary directory where writes are staged prior to upload to
Cloud Storage. (default: system default, likely /tmp)
default: ""
- config-path: "file-system.uid"
flag-name: "uid"
type: "int"
default: -1
usage: "UID owner of all inodes."
- flag-name: "foreground"
config-path: "foreground"
type: "bool"
usage: "Stay in the foreground after mounting."
default: false
- config-path: "gcs-auth.anonymous-access"
flag-name: "anonymous-access"
type: "bool"
usage: "This flag disables authentication."
default: false
- config-path: "gcs-auth.key-file"
flag-name: "key-file"
type: "resolvedPath"
usage: "Absolute path to JSON key file for use with GCS. If this flag is left unset, Google application default credentials are used."
- config-path: "gcs-auth.reuse-token-from-url"
flag-name: "reuse-token-from-url"
type: "bool"
usage: "If false, the token acquired from token-url is not reused."
default: "true"
- config-path: "gcs-auth.token-url"
flag-name: "token-url"
type: "string"
usage: "A url for getting an access token when the key-file is absent."
default: ""
- config-path: "gcs-connection.billing-project"
flag-name: "billing-project"
type: "string"
usage: >-
Project to use for billing when accessing a bucket enabled with "Requester
Pays".
default: ""
- config-path: "gcs-connection.client-protocol"
flag-name: "client-protocol"
type: "protocol"
usage: >-
The protocol used for communicating with the GCS backend.
Value can be 'http1' (HTTP/1.1), 'http2' (HTTP/2) or 'grpc'.
default: "http1"
- config-path: "gcs-connection.custom-endpoint"
flag-name: "custom-endpoint"
type: "string"
usage: >-
To specify a custom storage endpoint, ensure it supports the same resources as the default storage.googleapis.com:443 and includes the port number.
default: ""
- config-path: "gcs-connection.enable-http-dns-cache"
flag-name: "enable-http-dns-cache"
type: "bool"
usage: "Enables DNS cache for HTTP/1 connections"
default: true
hide-flag: true
- config-path: "gcs-connection.experimental-enable-json-read"
flag-name: "experimental-enable-json-read"
type: "bool"
usage: >-
By default, GCSFuse uses the GCS XML API to get and read objects. When this
flag is specified, GCSFuse uses the GCS JSON API instead."
default: false
deprecated: true
deprecation-warning: "Experimental flag: could be dropped even in a minor release."
- config-path: "gcs-connection.experimental-local-socket-address"
flag-name: "experimental-local-socket-address"
type: "string"
usage: "The local socket address to bind to. This is useful in multi-NIC scenarios. This is an experimental flag."
default: ""
hide-flag: true
- config-path: "gcs-connection.grpc-conn-pool-size"
flag-name: "experimental-grpc-conn-pool-size"
type: "int"
usage: "The number of gRPC channel in grpc client."
default: "1"
deprecated: true
deprecation-warning: "Experimental flag: can be removed in a minor release."
- config-path: "gcs-connection.grpc-path-strategy"
flag-name: "grpc-path-strategy"
type: "directPathStrategy"
usage: >-
Strategy for DirectPath connectivity when client-protocol=grpc.
Options: 'direct-path-only' (fail if unavailable), 'direct-path-with-fallback' (always fallback to HTTP/1 when direct path is not available).
default: "direct-path-with-fallback"
hide-flag: true
- config-path: "gcs-connection.http-client-timeout"
flag-name: "http-client-timeout"
type: "duration"
usage: >-
The time duration that http client will wait to get response from the
server. A value of 0 indicates no timeout.
default: "0s"
- config-path: "gcs-connection.limit-bytes-per-sec"
flag-name: "limit-bytes-per-sec"
type: "float64"
usage: "Bandwidth limit for reading data, measured over a 30-second window. (use -1 for no limit)"
default: "-1"
- config-path: "gcs-connection.limit-ops-per-sec"
flag-name: "limit-ops-per-sec"
type: "float64"
usage: "Operations per second limit, measured over a 30-second window (use -1 for no limit)"
default: "-1"
- config-path: "gcs-connection.max-conns-per-host"
flag-name: "max-conns-per-host"
type: "int"
usage: >-
The max number of TCP connections allowed per server. This is effective when
client-protocol is set to 'http1'. A value of 0 indicates no limit on
TCP connections (limited by the machine specifications).
default: "0"
- config-path: "gcs-connection.max-idle-conns-per-host"
flag-name: "max-idle-conns-per-host"
type: "int"
usage: "The number of maximum idle connections allowed per server."
default: "100"
- config-path: "gcs-connection.sequential-read-size-mb"
flag-name: "sequential-read-size-mb"
type: "int"
usage: "File chunk size to read from GCS in one call. Need to specify the value in MB. ChunkSize less than 1MB is not supported"
default: "200"
- config-path: "gcs-retries.chunk-retry-deadline-secs"
flag-name: "chunk-retry-deadline-secs"
type: "int"
usage: >-
We send larger file uploads in 16 MiB (Legacy Writes) or 32MiB (Streaming Writes) chunks. This flag controls the overall duration
that GCSFuse would keep retrying for a single chunk upload completion. 0 means infinity duration for chunk retries.
default: "120"
hide-flag: true
- config-path: "gcs-retries.chunk-transfer-timeout-secs"
flag-name: "chunk-transfer-timeout-secs"
type: "int"
usage: >-
We send larger file uploads in 16 MiB (Legacy Writes) or 32MiB (Streaming Writes) chunks. This flag controls the duration
that the HTTP client will wait for a response after making a request to upload a chunk.
As an example, a value of 10 indicates that the client will wait 10 seconds for upload completion;
otherwise, it cancels the request and retries for that chunk till chunk retry deadline duration. 0 means no timeout.
default: "10"
hide-flag: true
- config-path: "gcs-retries.enable-mount-retries"
flag-name: "enable-mount-retries"
type: "bool"
usage: >-
If true, enables retry logic in GCSFuse during the mount sequence
for additional errors (such as metadata server readiness delays, IAM propagation
delays, and temporary bucket non-existence). Intended specifically for the
GKE GCSFuse CSI Driver.
default: false
hide-flag: true
- config-path: "gcs-retries.experimental-nonrapid-folder-api-stall-retry"
flag-name: "experimental-nonrapid-folder-api-stall-retry"
type: "bool"
usage: "Enables stall-retry-fix for folder APIs for non-rapid buckets."
default: false
hide-flag: true
- config-path: "gcs-retries.max-retry-attempts"
flag-name: "max-retry-attempts"
type: "int"
usage: >-
It sets a limit on the total number of attempts (including the initial call) made for an operation if it fails,
preventing endless retry loops. For example, a value of 5 means up to 5 total attempts (1 initial call plus 4 retries).
A value of 0 indicates unlimited attempts.
default: "0"
- config-path: "gcs-retries.max-retry-sleep"
flag-name: "max-retry-sleep"
type: "duration"
usage: >-
The maximum backoff sleep duration allowed between retry attempts.
Once the exponential backoff exceeds this limit, subsequent retries will use this constant sleep value.
default: "30s"
- config-path: "gcs-retries.multiplier"
flag-name: "retry-multiplier"
type: "float64"
usage: >-
The multiplier factor by which the retry backoff duration increases after each failed attempt.
For example, a multiplier of 2.0 doubles the backoff sleep duration for each subsequent retry.
default: 2
- config-path: "gcs-retries.read-stall.enable"
flag-name: "enable-read-stall-retry"
type: "bool"
usage: >-
To turn on/off retries for stalled read requests. This is based on a timeout
that changes depending on how long similar requests took in the past.
default: true
hide-flag: true
- config-path: "gcs-retries.read-stall.initial-req-timeout"
flag-name: "read-stall-initial-req-timeout"
type: "duration"
usage: Initial value of the read-request dynamic timeout.
default: 20s
hide-flag: true
- config-path: "gcs-retries.read-stall.max-req-timeout"
flag-name: "read-stall-max-req-timeout"
type: "duration"
usage: Upper bound of the read-request dynamic timeout.
default: 20m
hide-flag: true
- config-path: "gcs-retries.read-stall.min-req-timeout"
flag-name: "read-stall-min-req-timeout"
type: "duration"
usage: Lower bound of the read request dynamic timeout.
default: 1500ms
hide-flag: true
- config-path: "gcs-retries.read-stall.req-increase-rate"
flag-name: "read-stall-req-increase-rate"
type: "float64"
usage: Determines how many increase calls it takes for dynamic timeout to double.
default: 15
hide-flag: true
- config-path: "gcs-retries.read-stall.req-target-percentile"
flag-name: "read-stall-req-target-percentile"
type: "float64"
usage: Retry the request which take more than p(targetPercentile * 100) of past similar request.
default: 0.99
hide-flag: true
- config-path: "implicit-dirs"
flag-name: "implicit-dirs"
type: "bool"
usage: "Implicitly define directories based on content. See files and directories in docs/semantics for more information"
default: false
optimizations:
machine-based-optimization:
- group: "high-performance"
value: true
profiles:
- name: "aiml-training"
value: true
- name: "aiml-serving"
value: true
- name: "aiml-checkpointing"
value: true
- config-path: "list.enable-empty-managed-folders"
flag-name: "enable-empty-managed-folders"
type: "bool"
usage: >-
This handles the corner case in listing managed folders.
There are two corner cases (a) empty managed folder (b) nested managed folder which doesn't contain any descendent as object.
This flag always works in conjunction with --implicit-dirs flag.
(a) If only ImplicitDirectories is true, all managed folders are listed other than above two mentioned cases.
(b) If both ImplicitDirectories and EnableEmptyManagedFolders are true, then all the managed folders are listed including the above-mentioned corner case.
(c) If ImplicitDirectories is false then no managed folders are listed irrespective of enable-empty-managed-folders flag.
default: false
hide-flag: true
- config-path: "logging.file-path"
flag-name: "log-file"
type: "resolvedPath"
usage: >-
The file for storing logs that can be parsed by fluentd. When not provided,
plain text logs are printed to stdout when Cloud Storage FUSE is run
in the foreground, or to syslog when Cloud Storage FUSE is run in the
background.
- config-path: "logging.format"
flag-name: "log-format"
type: "string"
usage: "The format of the log file: 'text' or 'json'."
default: "json"
- config-path: "logging.log-rotate.backup-file-count"
flag-name: "log-rotate-backup-file-count"
type: "int"
usage: >-
The maximum number of backup log files to retain after they have been
rotated. A value of 0 indicates all backup files are retained.
default: "10"
- config-path: "logging.log-rotate.compress"
flag-name: "log-rotate-compress"
type: "bool"
usage: "Controls whether the rotated log files should be compressed using gzip."
default: "true"
- config-path: "logging.log-rotate.max-file-size-mb"
flag-name: "log-rotate-max-file-size-mb"
type: "int"
usage: "The maximum size in megabytes that a log file can reach before it is rotated."
default: "512"
- config-path: "logging.severity"
flag-name: "log-severity"
type: "logSeverity"
usage: "Specifies the logging severity expressed as one of [trace, debug, info, warning, error, off]"
default: "info"
- config-path: "logging.wire-log"
flag-name: "wire-log"
type: "resolvedPath"
usage: >-
The file name of the wire log. When specified, GCSFuse will serialize
each FUSE operation as a JSON object and append it to this file.
hide-flag: true
- config-path: "machine-type"
flag-name: "machine-type"
type: "string"
usage: "Type of the machine on which gcsfuse is being run e.g. a3-highgpu-4g"
default: ""
hide-flag: true
- config-path: "metadata-cache.deprecated-stat-cache-capacity"
flag-name: "stat-cache-capacity"
type: "int"
usage: >-
How many entries can the stat-cache hold (impacts memory consumption). This
flag has been deprecated (starting v2.0) and in favor of
stat-cache-max-size-mb. For now, the value of stat-cache-capacity will be
translated to the next higher corresponding value of stat-cache-max-size-mb
(assuming stat-cache entry-size ~= 1720 bytes, including 1464 for positive
entry and 256 for corresponding negative entry), if stat-cache-max-size-mb
is not set."
deprecated: true
deprecation-warning: "Please use --stat-cache-max-size-mb instead."
default: "20460"
- config-path: "metadata-cache.deprecated-stat-cache-ttl"
flag-name: "stat-cache-ttl"
type: "duration"
usage: >-
How long to cache StatObject results and inode attributes. This flag
has been deprecated (starting v2.0) in favor of metadata-cache-ttl-secs. For
now, the minimum of stat-cache-ttl and type-cache-ttl values, rounded up to
the next higher multiple of a second is used as ttl for both stat-cache and
type-cache, when metadata-cache-ttl-secs is not set.
default: "60s"
deprecated: true
deprecation-warning: >-
This flag has been deprecated (starting v2.0) in favor of
metadata-cache-ttl-secs.
- config-path: "metadata-cache.deprecated-type-cache-ttl"
flag-name: "type-cache-ttl"
type: "duration"
usage: >-
Usage: How long to cache StatObject results and inode attributes. This flag
has been deprecated (starting v2.0) in favor of metadata-cache-ttl-secs. For
now, the minimum of stat-cache-ttl and type-cache-ttl values, rounded up to
the next higher multiple of a second is used as ttl for both stat-cache and
type-cache, when metadata-cache-ttl-secs is not set.
default: "60s"
deprecated: true
deprecation-warning: >-
This flag has been deprecated (starting v2.0) in favor of
metadata-cache-ttl-secs.
- config-path: "metadata-cache.enable-metadata-prefetch"
flag-name: "enable-metadata-prefetch"
type: "bool"
usage: >-
Enables background prefetching of object metadata when a directory is first opened.
This reduces latency for subsequent file lookups by pre-filling the metadata cache.
default: false
deprecated: false
hide-flag: false
- config-path: "metadata-cache.enable-nonexistent-entry-caching"
flag-name: "enable-nonexistent-entry-caching"
type: "bool"
usage: "Cache paths confirmed to not exist (404s) in the unified stat cache."
default: false
- config-path: "metadata-cache.enable-nonexistent-type-cache"
flag-name: "enable-nonexistent-type-cache"
type: "bool"
usage: >-
Once set, if an inode is not found in GCS, a type cache entry with type
NonexistentType will be created. This also means new file/dir created might
not be seen. For example, if this flag is set, and metadata-cache-ttl-secs
is set, then if we create the same file/node in the meantime using the same
mount, since we are not refreshing the cache, it will still return nil.
This flag has been deprecated in favour of a single unified flag metadata-cache-negative-ttl-secs.
default: false
- config-path: "metadata-cache.experimental-metadata-prefetch-on-mount"
flag-name: "experimental-metadata-prefetch-on-mount"
type: "string"
usage: >-
Experimental: This indicates whether or not to prefetch the metadata
(prefilling of metadata caches and creation of inodes) of the mounted bucket
at the time of mounting the bucket. Supported values: "disabled", "sync" and
"async". Any other values will return error on mounting. This is applicable
only to static mounting, and not to dynamic mounting.
default: "disabled"
deprecated: true
deprecation-warning: "Experimental flag: could be removed even in a minor release."
- config-path: "metadata-cache.metadata-prefetch-entries-limit"
flag-name: "metadata-prefetch-entries-limit"
type: "int"
usage: >
The maximum number of metadata entries (files and directories) to prefetch
into the cache upon a prefetch trigger. Since a single GCS List call is capped
at 5000 results, values higher than 5000 will trigger multiple sequential GCS
List calls per directory.
default: "5000"
deprecated: false
hide-flag: false
- config-path: "metadata-cache.metadata-prefetch-max-workers"
flag-name: "metadata-prefetch-max-workers"
type: "int"
usage: >
The maximum number of concurrent goroutines (workers) allowed to perform
metadata prefetching across all directories.