sync_gd_to_gcs.qmd

---
title: "Sync Google Drive to GCS"
editor_options:
  chunk_output_type: console
---

## Overview

Mirrors the entire Google Drive `data-public/` folder to
`gs://calcofi-files-public/_sync/` using `rclone`. This transfers
directly between cloud providers (GD API → GCS API) without needing
files cached on the local machine.

This is a standalone utility — not a pipeline target. Run manually
or schedule as a cron job for daily backup.

**Two-tier sync**:

- `calcofi-files-public/_sync/` — living mirror of GD (this notebook).
  Deletes stale GCS files so it always matches GD.
- `calcofi-files-public/archive/` — immutable timestamped snapshots
  (created by ingest QMDs via `sync_to_gcs(archive = TRUE)`).
  Nothing is lost permanently.

**Prerequisites**: `rclone` configured with remotes:

- `gdrive-ecoquants` — Google Drive (or `gdrive-server` on server)
- `gcs-calcofi` — Google Cloud Storage

Check with `rclone config` or `rclone listremotes`.

### rclone exclude patterns

All sync commands below exclude these files:

| Pattern | Reason |
|---------|--------|
| `.DS_Store` | macOS metadata |
| `*.tmp` | temporary files |
| `~$*` | Office lock files |
| `*.gdoc`, `*.gsheet`, `*.gslides` | Google Docs shortcuts (not real files) |

## Setup

```{r}
#| label: setup

librarian::shelf(
  DT,
  fs,
  glue,
  tibble,
  dplyr,
  quiet = T
)
options(DT.options = list(scrollX = TRUE))

# A DT is a CLIENT-SIDE widget: every row is embedded in the HTML as JSON, so a
# table of rclone's verbose output is as large as the sync is. One render of the
# dry run put **322,976** lines into a single datatable() and produced a 52 MB
# _output/sync_gd_to_gcs.html — which was then committed, and GitHub warned about
# it on every push. The full listing is not lost: rclone writes it to
# data/logs/sync_gd_to_gcs_*.log, which is what to read when you need all of it.
#
# So show the head and tail only, and say so in the caption rather than silently
# truncating — a table that quietly shows 200 of 322,976 rows is worse than a
# big one, because it reads as the whole answer.
DT_N <- 100L

dt_head_tail <- function(x, what, n = DT_N) {
  if (!length(x)) return(invisible(NULL))
  total <- length(x)
  if (total <= 2L * n) {
    d   <- tibble(row = seq_len(total), message = x)
    cap <- glue("{what} — all {format(total, big.mark = ',')}")
  } else {
    i   <- c(seq_len(n), seq.int(total - n + 1L, total))
    d   <- tibble(row = i, message = x[i])
    cap <- glue(
      "{what} — first {n} and last {n} of ",
      "{format(total, big.mark = ',')}; the full listing is in the rclone log")
  }
  datatable(d, caption = cap, rownames = FALSE,
            options = list(pageLength = 10, scrollX = TRUE, dom = "tip"))
}

# rclone remote paths — override via env vars for server deployment
# e.g. CALCOFI_GD_REMOTE="gdrive-server:projects/calcofi/data-public"
gd_remote <- Sys.getenv(
  "CALCOFI_GD_REMOTE",
  "gdrive-ecoquants:projects/calcofi/data-public"
)
gcs_remote <- Sys.getenv(
  "CALCOFI_GCS_REMOTE",
  "gcs-calcofi:calcofi-files-public/_sync"
)

# shared exclude flags for all rclone commands
rclone_excludes <- c(
  "--exclude", ".DS_Store",
  "--exclude", "*.tmp",
  "--exclude", "~$*",
  "--exclude", "*.gdoc",
  "--exclude", "*.gsheet",
  "--exclude", "*.gslides"
)

# local log directory
log_dir <- here::here("data/logs")
dir_create(log_dir)
```

## Dry Run

Preview what rclone would do without making changes. This is fast
and useful for verifying before a large sync.

```{r}
#| label: dry_run

cat(glue("dry run: {gd_remote} → {gcs_remote}"), "\n\n")

dry_out <- system2(
  "rclone",
  c("sync", gd_remote, gcs_remote,
    "--dry-run", rclone_excludes, "-v"),
  stdout = TRUE, stderr = TRUE
)

# parse actions from rclone verbose output
actions <- dry_out[grepl("NOTICE|INFO|Copied|Deleted", dry_out)]
cat(glue("{length(actions)} actions would be taken"), "\n\n")

dt_head_tail(actions, "rclone dry-run actions")
```

## Sync

Run the actual sync. `rclone sync` makes the destination match the
source — uploads new/changed files and deletes files in GCS that no
longer exist in GD.

**Initial sync**: The first run transfers all files (~8 GB) and can
take several hours. For a large initial sync, run directly from the
terminal instead of this notebook:

```bash
# run in background with progress logging
rclone sync \
  gdrive-ecoquants:projects/calcofi/data-public \
  gcs-calcofi:calcofi-files-public/_sync \
  --exclude ".DS_Store" --exclude "*.tmp" --exclude "~\$*" \
  --exclude "*.gdoc" --exclude "*.gsheet" --exclude "*.gslides" \
  --progress --stats 30s -v \
  --log-file data/logs/sync_gd_to_gcs_$(date +%Y-%m-%d_%H%M%S).log &

# monitor progress
tail -f data/logs/sync_gd_to_gcs_*.log
```

**Incremental sync**: Subsequent runs only transfer changed files
and complete in seconds to minutes. These run fine in this notebook.

```{r}
#| label: sync

cat(glue("syncing {gd_remote} → {gcs_remote}"), "\n\n")

sync_out <- system2(
  "rclone",
  c("sync", gd_remote, gcs_remote,
    rclone_excludes,
    "--log-level", "INFO",
    "--stats-one-line",
    "-v"),
  stdout = TRUE, stderr = TRUE
)

# write log
log_ts   <- format(Sys.time(), "%Y-%m-%d_%H%M%S")
log_file <- path(log_dir, glue("sync_gd_to_gcs_{log_ts}.log"))
writeLines(sync_out, log_file)
cat(glue("log written to {log_file}"), "\n\n")

# show last 30 lines of output
cat(paste(tail(sync_out, 30), collapse = "\n"))
```

## Summary

```{r}
#| label: summary

# parse stats from rclone output
stats_lines <- sync_out[grepl("Transferred:|Deleted:|Checks:", sync_out)]
if (length(stats_lines) > 0) {
  cat("rclone transfer stats:\n")
  cat(paste(stats_lines, collapse = "\n"), "\n\n")
}

# parse copied/deleted file actions for a table
action_lines <- sync_out[grepl("INFO.*Copied|INFO.*Deleted", sync_out)]
if (length(action_lines) > 0) {
  dt_head_tail(action_lines, "files transferred or deleted")
} else {
  cat("no files changed — GCS already in sync with GD\n")
}

# count log files
logs <- dir_ls(log_dir, glob = "*sync_gd_to_gcs*")
cat(glue("\n{length(logs)} total sync logs in {log_dir}"), "\n")
```

## Server Cron

To run this daily on a server, schedule via cron or systemd timer:

```bash
# crontab -e
0 2 * * * cd /path/to/workflows && \
  rclone sync \
    gdrive-server:projects/calcofi/data-public \
    gcs-calcofi:calcofi-files-public/_sync \
    --exclude ".DS_Store" --exclude "*.tmp" --exclude "~\$*" \
    --exclude "*.gdoc" --exclude "*.gsheet" --exclude "*.gslides" \
    --stats-one-line -v \
    --log-file data/logs/sync_gd_to_gcs_$(date +\%Y-\%m-\%d_\%H\%M\%S).log
```

Or render the full notebook for an HTML report:

```bash
CALCOFI_GD_REMOTE="gdrive-server:projects/calcofi/data-public" \
  Rscript -e "quarto::quarto_render('sync_gd_to_gcs.qmd')"
```