Skip to content

Util: Add PostgreSQL data migration script #1520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
drikusroor wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

Util: Add PostgreSQL data migration script #1520

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions scripts/migrate-postgres-data
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
#!/bin/bash
set -e

# ----------------------------------------------------------------------------
# Step 0: Shut down all running containers for the project.
#
echo "Shutting down all project containers..."
docker compose down

# ----------------------------------------------------------------------------
# Step 1: Run 'docker compose up db' to capture version error output.
#
echo "Running 'docker compose up db' to capture version error output..."
LOGS=$(docker compose up db 2>&1)
echo "$LOGS"

# ----------------------------------------------------------------------------
# Remove any lingering database container so the volume isn’t in use.
#
echo "Removing any lingering 'db' container from docker compose..."
docker compose rm -f db || true

# ----------------------------------------------------------------------------
# Step 2: Extract PostgreSQL version numbers from the logs.
#
OLD_VERSION=$(echo "$LOGS" | grep -oP 'initialized by PostgreSQL version \K\d+')
NEW_VERSION=$(echo "$LOGS" | grep -oP 'this version \K\d+')

# Remove any trailing whitespace or carriage returns.
OLD_VERSION=$(echo "$OLD_VERSION" | tr -d '\r' | xargs)
NEW_VERSION=$(echo "$NEW_VERSION" | tr -d '\r' | xargs)

if [ -z "$OLD_VERSION" ] || [ -z "$NEW_VERSION" ]; then
echo "Error: Could not detect PostgreSQL versions from the logs."
exit 1
fi

echo "Detected migration from PostgreSQL $OLD_VERSION to PostgreSQL $NEW_VERSION"

# ----------------------------------------------------------------------------
# Step 3: Start a temporary container running the old PostgreSQL version.
#
echo "Starting a temporary PostgreSQL $OLD_VERSION container for data dump..."
docker run --name pg_old -d \
-v muscle_db_data:/var/lib/postgresql/data \
postgres:"$OLD_VERSION"

# ----------------------------------------------------------------------------
# Step 4: Wait until PostgreSQL inside the container is ready.
#
echo "Waiting for PostgreSQL to be ready..."
Copy link
Copy Markdown
Contributor

@Evert-R Evert-R Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script hangs here, with this error in the container in docker desktop:

2025-02-12 13:59:18 Error: Database is uninitialized and superuser password is not specified.
2025-02-12 13:59:18        You must specify POSTGRES_PASSWORD to a non-empty value for the
2025-02-12 13:59:18        superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
2025-02-12 13:59:18 
2025-02-12 13:59:18        You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
2025-02-12 13:59:18        connections without a password. This is *not* recommended.
2025-02-12 13:59:18 
2025-02-12 13:59:18        See PostgreSQL documentation about "trust":
2025-02-12 13:59:18        https://www.postgresql.org/docs/current/auth-trust.html

Copy link
Copy Markdown
Collaborator

@BeritJanssen BeritJanssen Feb 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran into something similar when doing this manually. I think what happens is that the env variables aren't loaded in this way. I solved it by running docker compose up db and then running the script afterwards.

until docker exec pg_old pg_isready -U muscle > /dev/null 2>&1; do
sleep 1
done

# ----------------------------------------------------------------------------
# Step 5: Dump the database using the running container (user "muscle").
#
echo "Dumping the database from PostgreSQL $OLD_VERSION..."
docker exec pg_old pg_dumpall -U muscle > migration-dump.sql

# Verify that the dump file exists on the host.
if [ ! -f migration-dump.sql ]; then
echo "Error: migration-dump.sql was not created."
exit 1
fi
echo "Dump file created: $(realpath migration-dump.sql)"

# ----------------------------------------------------------------------------
# Step 6: Stop and remove the temporary old PostgreSQL container.
#
echo "Stopping and removing the temporary PostgreSQL $OLD_VERSION container..."
docker stop pg_old > /dev/null
docker rm pg_old > /dev/null

# ----------------------------------------------------------------------------
# Step 7: Remove the old volume so that a new one can be created.
#
echo "Removing old volume 'muscle_db_data'..."
docker volume rm muscle_db_data

# ----------------------------------------------------------------------------
# Step 8: Restore the dump using a temporary container running the new PostgreSQL version.
#
# Use an absolute path for the dump file.
echo "Starting temporary new PostgreSQL $NEW_VERSION container for restore..."
docker run --name pg_new -d \
-e POSTGRES_HOST_AUTH_METHOD=trust \
-v muscle_db_data:/var/lib/postgresql/data \
postgres:"$NEW_VERSION"

echo "Waiting for PostgreSQL to be ready..."
until docker exec pg_new pg_isready -U postgres > /dev/null 2>&1; do
sleep 1
done

# Copy the migration dump file into the container
echo "Copying migration-dump.sql into the container..."
docker cp migration-dump.sql pg_new:/migration-dump.sql

# Run the restore using the default 'postgres' user and its default database.
echo "Running restore..."
docker exec pg_new psql -U postgres -f ./migration-dump.sql

echo "Stopping and removing temporary new PostgreSQL container..."
docker stop pg_new
docker rm pg_new


# ----------------------------------------------------------------------------
# Step 9: Finally, restart your project with the updated configuration.
#
echo "Restarting the project..."
docker compose up -d --build

echo "Migration complete."