Skip to content

Migrate DynamoDB data from dev to prod tables #39

Description

@asmithml

Part of #33

Depends on: Deploy production infrastructure

Context

The current dev DynamoDB tables contain production user data that needs to move to the new prod tables.

Source Table Items Target Table
dev-ftw-projects 428 prd-ftw-projects
dev-ftw-inference-results 658 prd-ftw-inference-results
dev-ftw-images 0 prd-ftw-images (skip, empty)

Steps

  1. Enable Point-in-Time Recovery (PITR) on source tables if not already enabled (it is currently enabled)
  2. Take a backup of the source tables before migration:
    aws dynamodb create-backup --table-name dev-ftw-projects --backup-name pre-migration-projects
    aws dynamodb create-backup --table-name dev-ftw-inference-results --backup-name pre-migration-inference-results
  3. Migrate data using one of these methods:
    • Option A (recommended for this size): Script using scan + batch-write-item. The tables are small enough (428 + 658 items) that this completes in seconds.
    • Option B: DynamoDB Export to S3 + Import. More complex, better for large tables.
  4. Verify item counts match after migration:
    aws dynamodb scan --table-name prd-ftw-projects --select COUNT
    aws dynamodb scan --table-name prd-ftw-inference-results --select COUNT
  5. Spot-check a few items to confirm data integrity

Important

  • Do NOT delete data from dev tables yet. That happens in the cleanup task after prod is verified.
  • All commands should use --region us-west-2 and the appropriate AWS profile.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions