Skip to content

backup-s3.md

Latest commit

 

History

History
330 lines (236 loc) · 9.55 KB

backup-s3.md

File metadata and controls

330 lines (236 loc) · 9.55 KB

Optional S3 Backup Uploads

This guide adds off-server storage for the bundled single-server backup flow.

The local backup remains the source step:

october-backup.timer -> scripts/backup.sh -> /var/backups/october -> optional S3 upload

S3 upload is disabled by default. Enable it per server through runtime secrets, not through the project .env and not through git.

What Gets Uploaded

When enabled, scripts/backup.sh uploads the files created during the current backup run:

  • postgres-<tag>.dump
  • storage-app-<tag>.tar.gz
  • metadata-<tag>.txt
  • env-<tag> only when BACKUP_INCLUDE_SECRETS=1
  • auth-<tag>.json only when BACKUP_INCLUDE_SECRETS=1 and auth.json exists

The script does not manually split files into 100M parts. aws s3 cp uses S3 multipart upload for large files while keeping a single object in the bucket, which keeps restore simple.

Install AWS CLI On The Host

For a Debian server:

sudo apt-get update
sudo apt-get install -y awscli
aws --version

The PHP and Nginx images do not need awscli. Backup upload runs on the host through systemd.

Prepare The Bucket

Create a bucket and prefix, for example:

s3://company-october-backups/projects/example-site/production

Use a dedicated access key for backups. Minimum IAM permissions for upload and restore checks:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::company-october-backups",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["projects/example-site/production/*"]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::company-october-backups/projects/example-site/production/*"
    }
  ]
}

Prefer bucket lifecycle rules for remote retention, for example keep daily backups for 30-90 days depending on the client contract.

Runtime Secrets

scripts/backup.sh reads regular environment variables. The source can be a simple systemd EnvironmentFile, a Vault-rendered file, platform secrets or another runtime secret mechanism.

Recommended order:

  1. Simple VPS: /etc/october-backup.env with 0600 root:root.
  2. Stronger VPS: Vault Agent or another secret manager renders /run/october-backup/env on tmpfs, then systemd reads that file.
  3. Orchestrator: Kubernetes, BeCloud-like platforms, Docker Swarm or another scheduler injects the values as runtime secrets into a backup CronJob.
  4. systemd credentials: use LoadCredential= with a small wrapper or a future script extension that reads $CREDENTIALS_DIRECTORY.

The backup script should not need to know where the values came from. It only expects environment variables.

Simple VPS Environment File

Create a root-readable environment file:

sudo install -m 600 -o root -g root /dev/null /etc/october-backup.env
sudoedit /etc/october-backup.env

Example:

BACKUP_NOTIFY_ENABLED=1
TELEGRAM_BOT_TOKEN=123456:example
TELEGRAM_CHAT_ID=-1001234567890
TELEGRAM_THREAD_ID=

BACKUP_S3_ENABLED=1
BACKUP_S3_URI=s3://company-october-backups/projects/example-site/production
BACKUP_S3_REGION=eu-central-1
BACKUP_S3_STORAGE_CLASS=STANDARD_IA
BACKUP_S3_ENDPOINT=
BACKUP_S3_DELETE_LOCAL_AFTER_UPLOAD=0

AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=eu-central-1

Do not put this file in the repository. Do not include it in regular project backups unless it is encrypted and access-controlled.

For a server where S3 is not ready yet, keep S3 explicitly disabled:

BACKUP_NOTIFY_ENABLED=0
BACKUP_S3_ENABLED=0
BACKUP_S3_DELETE_LOCAL_AFTER_UPLOAD=0

When Telegram credentials are available, enable notifications without enabling S3:

BACKUP_NOTIFY_ENABLED=1
TELEGRAM_BOT_TOKEN=123456:example
TELEGRAM_CHAT_ID=-1001234567890
TELEGRAM_THREAD_ID=

BACKUP_S3_ENABLED=0
BACKUP_S3_DELETE_LOCAL_AFTER_UPLOAD=0

Runtime Key Reference

Backup control:

BACKUP_DIR=/var/backups/october
BACKUP_NOTIFY_ENABLED=0
BACKUP_NOTIFY_SCRIPT=/opt/october/app/scripts/telegram-notify.sh
BACKUP_S3_ENABLED=0
BACKUP_S3_DELETE_LOCAL_AFTER_UPLOAD=0
BACKUP_INCLUDE_SECRETS=0
BACKUP_RETENTION_COUNT=14

Telegram notifications:

TELEGRAM_BOT_TOKEN=
TELEGRAM_CHAT_ID=
TELEGRAM_THREAD_ID=

S3 upload:

BACKUP_S3_URI=s3://company-october-backups/projects/example-site/production
BACKUP_S3_REGION=eu-central-1
BACKUP_S3_STORAGE_CLASS=STANDARD_IA
BACKUP_S3_ENDPOINT=
BACKUP_S3_AWS_BIN=aws

AWS credentials used by aws s3 cp:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=eu-central-1
AWS_SESSION_TOKEN=

Use AWS_SESSION_TOKEN only for temporary credentials. For permanent IAM user keys, leave it unset.

Vault Or Secret Manager Pattern

For stronger deployments, render the same variables to a runtime file outside the project directory:

/run/october-backup/env

Example systemd override:

[Service]
EnvironmentFile=
EnvironmentFile=-/run/october-backup/env

The empty EnvironmentFile= line clears the value generated by the installer. The next line points systemd at the Vault-rendered runtime file.

The secret manager should render values such as:

BACKUP_NOTIFY_ENABLED=1
TELEGRAM_BOT_TOKEN=...
TELEGRAM_CHAT_ID=...
BACKUP_S3_ENABLED=1
BACKUP_S3_URI=s3://company-october-backups/projects/example-site/production
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=eu-central-1

After changing the override:

sudo systemctl daemon-reload
sudo systemctl restart october-backup.timer

systemd Credentials Pattern

systemd credentials keep secret values out of normal environment files. This is a good hardening path for dedicated VPS deployments, but it needs a small wrapper or a script extension that maps credential files into environment variables before calling scripts/backup.sh.

The systemd unit would use credentials like:

[Service]
LoadCredential=TELEGRAM_BOT_TOKEN:/etc/secure/october/telegram_bot_token
LoadCredential=TELEGRAM_CHAT_ID:/etc/secure/october/telegram_chat_id
LoadCredential=AWS_ACCESS_KEY_ID:/etc/secure/october/aws_access_key_id
LoadCredential=AWS_SECRET_ACCESS_KEY:/etc/secure/october/aws_secret_access_key

At runtime, systemd exposes them under $CREDENTIALS_DIRECTORY. A wrapper can export:

export TELEGRAM_BOT_TOKEN="$(cat "$CREDENTIALS_DIRECTORY/TELEGRAM_BOT_TOKEN")"
export TELEGRAM_CHAT_ID="$(cat "$CREDENTIALS_DIRECTORY/TELEGRAM_CHAT_ID")"
export AWS_ACCESS_KEY_ID="$(cat "$CREDENTIALS_DIRECTORY/AWS_ACCESS_KEY_ID")"
export AWS_SECRET_ACCESS_KEY="$(cat "$CREDENTIALS_DIRECTORY/AWS_SECRET_ACCESS_KEY")"
exec /opt/october/app/scripts/backup.sh

For now, the bundled installer uses EnvironmentFile because it is portable and simple for a baseline VPS setup.

Enable The Timer With EnvironmentFile

Install or refresh the timer:

cd /opt/october/app
BACKUP_ENV_FILE=/etc/october-backup.env \
BACKUP_DIR=/var/backups/october \
./scripts/install-backup-timer.sh

The generated service contains:

EnvironmentFile=-/etc/october-backup.env

The leading - means the file is optional. If the file is missing, local backups still run and S3 remains disabled unless variables are provided another way. If you switch to Vault or another runtime secret source, set BACKUP_ENV_FILE to that generated file path or add a systemd override.

Notifications

When BACKUP_NOTIFY_ENABLED=1, scripts/backup.sh calls scripts/telegram-notify.sh.

It sends:

  • [backup:start] with project, host, tag, git commit, image tag and S3 state
  • [backup:success] with local directory, S3 URI, file list and total size
  • [backup:failure] with exit code, partial file list and journal command

Notification failures do not fail the backup itself. The backup job writes a warning to the journal instead.

Manual Test

Run one backup through systemd:

sudo systemctl start october-backup.service
sudo journalctl -u october-backup.service -n 120 --no-pager

Check local files:

ls -lh /var/backups/october

Check S3:

aws s3 ls s3://company-october-backups/projects/example-site/production/

Download and test restore on a disposable machine or temporary container before trusting the setup.

S3-Compatible Storage

For MinIO, Selectel, Cloudflare R2, Yandex Object Storage or another S3-compatible provider, set an endpoint:

BACKUP_S3_ENDPOINT=https://s3.example-provider.com
BACKUP_S3_REGION=ru-1
AWS_DEFAULT_REGION=ru-1

If the provider needs path-style access, configure it in the AWS CLI profile or use provider-specific AWS environment variables. For complex S3-compatible setups, rclone can be added later as a second provider, but the bundled script currently targets aws s3 cp.

Restore From S3

Download the files you need:

mkdir -p /var/backups/october
aws s3 cp \
  s3://company-october-backups/projects/example-site/production/postgres-20260601T031500Z.dump \
  /var/backups/october/

aws s3 cp \
  s3://company-october-backups/projects/example-site/production/storage-app-20260601T031500Z.tar.gz \
  /var/backups/october/

Then follow Backup And Restore for PostgreSQL and storage-app restore.

Multi-Node Deployments

For multi-node production, do not treat a local Docker volume as the source of truth for media/uploads. Use S3 or MinIO as the application media storage, or use a shared RWX storage system and back it up with the platform's backup tooling.

In Kubernetes or BeCloud-like platforms, replace the host systemd timer with a platform CronJob and inject the same variables through Secrets.