Migration Guide

Step-by-step guide for migrating from other templates and projects

Quick Reference: Getting Started | Architecture | Common Workflows

This guide provides instructions for migrating existing projects to the Research Project Template structure, ensuring a smooth transition while preserving your work.

Overview

Migration involves adapting your existing project structure to match the template's architecture while preserving your code, data, and documentation. This guide covers common migration scenarios and provides step-by-step procedures.

Pre-Migration Assessment

Evaluate Your Current Project

Before migrating, assess:

Current structure - Directory layout
Dependencies - Python packages and versions
Code organization - Module structure
Testing - Test coverage and framework
Documentation - Format and organization
Build system - Current build process

Identify Migration Requirements

Determine what needs to migrate:

Source code modules
Test files
Documentation/manuscript files
Configuration files
Dependencies
Build scripts

Migration Strategy

Phase 1: Preparation

Before starting migration:

Backup everything - Full project backup
Create migration branch - Isolated migration work
Document current state - Note current structure
Plan migration steps - Detailed plan

Phase 2: Structure Setup

Set up template structure:

Create directories - src/, tests/, scripts/, etc.
Copy template files - Build scripts, configs
Set up dependencies - Install uv, configure pyproject.toml

Phase 3: Code Migration

Migrate your code:

Move source code - To src/ directory
Move tests - To tests/ directory
Adapt scripts - Convert to thin orchestrators
Update imports - Fix import paths

Phase 4: Documentation Migration

Migrate documentation:

Move manuscript files - To manuscript/ directory
Update references - Fix cross-references
Adapt formatting - Match template format
Update metadata - Author, project info

Phase 5: Validation

Validate migration:

Run tests - Ensure all tests pass
Check coverage - Verify coverage requirements met
Build PDFs - Generate outputs
Validate outputs - Check PDF quality

Common Migration Scenarios

Scenario 1: Simple Python Project

From: Basic Python project with scripts

Steps:

Create template structure:

mkdir -p projects/my_project/src projects/my_project/tests projects/my_project/scripts projects/my_project/manuscript docs

Move code to src/:

# Move modules
mv *.py projects/my_project/src/
# Keep only root orchestrator scripts at workspace level

Create tests:

# Create test files
for module in projects/my_project/src/*.py; do
  test_file="projects/my_project/tests/test_$(basename $module)"
  # Create test template
done

Adapt scripts:

# Convert scripts to thin orchestrators
# Old: Script with business logic
# New: Script that imports from src/

Update dependencies:

# Create pyproject.toml
uv init
uv add existing-dependencies

Scenario 2: Research Paper Project

From: LaTeX or Word document project

Steps:

Convert to Markdown:

# Convert LaTeX to Markdown
pandoc document.tex -o projects/my_project/manuscript/01_introduction.md

# Or convert Word
pandoc document.docx -o projects/my_project/manuscript/01_introduction.md

Organize sections:

# Create numbered sections
mv introduction.md projects/my_project/manuscript/02_introduction.md
mv methods.md projects/my_project/manuscript/03_methodology.md

Add cross-references:

# Update references
# Old: See Section 2
# New: See \ref{sec:methodology}

Set up build system:

# Copy template build scripts
cp template/scripts/* scripts/

Scenario 3: Jupyter Notebook Project

From: Jupyter notebooks with analysis

Steps:

Extract code:

# Convert notebooks to scripts
jupyter nbconvert --to script notebook.ipynb

Separate logic from presentation:

# Business logic → src/
# Visualization → scripts/

Create tests:

# Extract test cases from notebooks
# Create test files

Convert outputs:

# Export figures
# Convert markdown cells to manuscript/

Scenario 4: Existing Template Fork

From: Previous version of template

Steps:

Compare structures:

# Identify differences
diff -r old_template/ new_template/

Update configuration:

# Update pyproject.toml
# Update build scripts
# Update dependencies

Migrate customizations:

# Preserve custom code
# Preserve custom scripts
# Preserve custom documentation

Test compatibility:

# Run full test suite
# Verify build works
# Check outputs

Step-by-Step Migration Process

Step 1: Backup and Preparation

Create backup:

# Full project backup
tar -czf project_backup_$(date +%Y%m%d).tar.gz project/

# Or use git
git tag backup-before-migration
git push origin backup-before-migration

Create migration branch:

git checkout -b migration/template-upgrade

Step 2: Set Up Template Structure

Clone template structure:

# Create directories
mkdir -p src tests scripts manuscript docs output

# Copy template files
cp template/scripts/* scripts/
cp template/pyproject.toml .
cp template/.cursorrules .

Set up dependencies:

# Initialize uv
uv init

# Add existing dependencies
uv add package1 package2 package3

Step 3: Migrate Source Code

Move code to src/:

# Move modules
mv existing_modules/*.py projects/my_project/src/

# Update imports in moved files
# Fix relative imports

Create test structure:

# Create test files
for module in projects/my_project/src/*.py; do
    module_name=$(basename $module .py)
    cat > projects/my_project/tests/test_${module_name}.py << EOF
"""Tests for ${module_name} module."""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
from ${module_name} import *

def test_example():
    """Example test."""
    pass
EOF
done

Step 4: Adapt Scripts

Convert to thin orchestrators:

# Old script (business logic included)
def process_data(data):
    result = sum(data) / len(data)  # Business logic
    return result

# New script (thin orchestrator)
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
from statistics import calculate_average  # Import from src/

def process_data(data):
    result = calculate_average(data)  # Use src/ method
    return result

Step 5: Migrate Documentation

Organize manuscript files:

# Move documents
mv papers/*.md projects/my_project/manuscript/

# Rename to numbered format
mv introduction.md projects/my_project/manuscript/02_introduction.md
mv methods.md projects/my_project/manuscript/03_methodology.md

Update cross-references:

# Update reference format
# Old: See Section 2
# New: See \ref{sec:methodology}

# Add section labels
# Introduction {#sec:introduction}

Step 6: Update Configuration

Set environment variables:

# Create .env file
cat > .env << EOF
AUTHOR_NAME="Your Name"
AUTHOR_EMAIL="your.email@example.com"
AUTHOR_ORCID="0000-0000-0000-0000"
PROJECT_TITLE="Your Project Title"
EOF

Update pyproject.toml:

[project]
name = "your-project-name"
version = "0.1.0"
description = "Your project description"
authors = [{name = "Your Name", email = "your.email@example.com"}]

Step 7: Test Migration

Run test suite:

# Install dependencies
uv sync

# Run tests
uv run pytest projects/my_project/tests/ --cov=projects/my_project/src

# Fix any import errors
# Add missing tests

Build outputs:

# Clean outputs
# Pipeline automatically handles cleanup

# Run build
uv run python scripts/execute_pipeline.py --project {name} --core-only

# Verify outputs (exemplar layout)
ls -la output/template_code_project/pdf/

Common Migration Challenges

Challenge 1: Import Path Issues

Problem: Import errors after moving files

Solution:

# Add src/ to path
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))

# Or use absolute imports
from projects.my_project.src.module import function

Challenge 2: Test Coverage Gaps

Problem: Coverage below 100% after migration

Solution:

# Identify missing coverage
uv run pytest projects/my_project/tests/ --cov=projects/my_project/src --cov-report=term-missing

# Add tests for uncovered code
# Look for lines marked ">>>>>"

Challenge 3: Build Script Compatibility

Problem: Existing build scripts don't work

Solution:

# Adapt scripts to template structure
# Use template build scripts as reference
# Update paths and commands

Challenge 4: Documentation Format

Problem: Documentation doesn't match template format

Solution:

# Update to template format
# Add section labels: {#sec:section_name}
# Update cross-references: \ref{sec:section_name}
# Use equation environments: \begin{equation}...\end{equation}

Data Preservation

Preserving Code

Ensure nothing is lost:

# Before migration
git add -A
git commit -m "Pre-migration backup"

# After migration
git diff HEAD~1  # Verify nothing missing

Preserving Data

Backup data files:

# Backup data
cp -r data/ data_backup/

# Restore after migration
cp -r data_backup/* output/data/

Preserving Outputs

Save existing outputs:

# Backup outputs
tar -czf outputs_backup.tar.gz output/

# Reference for comparison
# Verify new outputs match

Testing After Migration

Validation Checklist

Verify migration success:

Comparison Testing

Compare old vs new:

# Run old build
cd old_project && ./build.sh

# Run new build
cd new_project && uv run python scripts/execute_pipeline.py --project {name} --core-only

# Compare outputs
diff -r old_project/output/ new_project/output/

Rollback Procedures

If Migration Fails

Rollback steps:

Restore from backup:

# Restore files
tar -xzf project_backup.tar.gz

# Or use git
git checkout backup-before-migration

Investigate issues:

# Review error logs
# Identify problems
# Fix issues

Retry migration:
```
# Apply fixes
# Retry migration steps
```

Migration from Specific Templates

From QuadMath

The template was adapted from QuadMath:

# Copy utilities
cp quadmath/scripts/* scripts/

# Adapt structure
# Update paths
# Update dependencies

See: README.md Migration Section

From Other Research Templates

Generic migration steps:

Identify differences - Compare structures
Map components - Match old to new
Migrate code - Move and adapt
Update configs - Adapt settings
Test thoroughly - Verify everything works

Post-Migration

Optimization

After successful migration:

Optimize structure - Clean up unnecessary files
Improve tests - Enhance test coverage
Update documentation - Align with template
Performance tuning - Optimize build times

Documentation

Update project documentation:

Update README - Reflect new structure
Update guides - Align with template
Add examples - Show new patterns
Document changes - Note what changed

Summary

Migration checklist:

Preparation - Backup, plan, branch
Structure - Set up template directories
Code - Move and adapt source code
Tests - Create tests
Scripts - Convert to thin orchestrators
Documentation - Migrate and format
Configuration - Update settings
Validation - Test everything
Optimization - Improve and refine

See Also:

Getting Started - Template basics
Architecture - Understanding structure
Common Workflows - Step-by-step guides

FilesExpand file tree

migration-guide.md

Latest commit

History

migration-guide.md

File metadata and controls

Migration Guide

Overview

Pre-Migration Assessment

Evaluate Your Current Project

Identify Migration Requirements

Migration Strategy

Phase 1: Preparation

Phase 2: Structure Setup

Phase 3: Code Migration

Phase 4: Documentation Migration

Phase 5: Validation

Common Migration Scenarios

Scenario 1: Simple Python Project

Scenario 2: Research Paper Project

Scenario 3: Jupyter Notebook Project

Scenario 4: Existing Template Fork

Step-by-Step Migration Process

Step 1: Backup and Preparation

Step 2: Set Up Template Structure

Step 3: Migrate Source Code

Step 4: Adapt Scripts

Step 5: Migrate Documentation

Step 6: Update Configuration

Step 7: Test Migration

Common Migration Challenges

Challenge 1: Import Path Issues

Challenge 2: Test Coverage Gaps

Challenge 3: Build Script Compatibility

Challenge 4: Documentation Format

Data Preservation

Preserving Code

Preserving Data

Preserving Outputs

Testing After Migration

Validation Checklist

Comparison Testing

Rollback Procedures

If Migration Fails

Migration from Specific Templates

From QuadMath

From Other Research Templates

Post-Migration

Optimization

Documentation

Summary