Skip to content

testing-with-credentials.md

Latest commit

 

History

History
427 lines (291 loc) · 11 KB

testing-with-credentials.md

File metadata and controls

427 lines (291 loc) · 11 KB

Testing with External Service Credentials

This guide explains how to configure and run tests that require external service credentials (Zenodo, GitHub, arXiv) and local tools (LaTeX).

Overview

The default suite keeps credential-free coverage hermetic with local HTTP servers where possible. Tests that truly make real API calls to external services are marked with special pytest markers and should be selected only in environments where those credentials are configured.

Quick Start

  1. Create the credential files at the repository root:

    # Copy the non-secret env template that ships with the repo
cp infrastructure/config/.env.template .env
# Optionally create test_credentials.yaml directly (no committed example;
# see the YAML Configuration section below for the schema)

  2. Add your credentials to .env (see sections below)

  3. Run all tests (external service tests will be skipped if credentials not configured):

    uv run pytest tests/

  4. Run only tests that don't require external services:

    uv run pytest tests/ -m "not requires_credentials"

Credential Configuration

Environment Variables (.env file)

Create a .env file in the repository root with your credentials:

# Zenodo Sandbox (recommended for testing)
ZENODO_SANDBOX_TOKEN=your_sandbox_token_here

# GitHub
GITHUB_TOKEN=your_github_pat_here
GITHUB_REPO=username/test-repository

# arXiv (optional)
ARXIV_USERNAME=your_username
ARXIV_PASSWORD=your_password

YAML Configuration (test_credentials.yaml)

Optionally, create test_credentials.yaml for additional configuration:

zenodo:
  use_sandbox: true  # Always use sandbox for tests!

github:
  test_tag_prefix: "test-release-"

cleanup:
  auto_cleanup: true  # Automatically delete test artifacts

Obtaining Credentials

Zenodo Sandbox Token

  1. Create account at https://sandbox.zenodo.org/
  2. Navigate to: Account Settings → Applications → Personal access tokens
  3. Create new token with scopes: deposit:write, deposit:actions
  4. Copy token to .env as ZENODO_SANDBOX_TOKEN

Important: Use the sandbox environment for testing, not production Zenodo!

GitHub Personal Access Token

  1. Go to https://github.com/settings/tokens/new
  2. Create token with scopes:
    • repo (full repository access)
    • write:packages (for release assets)
    • delete_repo (optional, for test cleanup)
  3. Set expiration and create token
  4. Copy token to .env as GITHUB_TOKEN

Repository Setup:

  • Create a test repository on GitHub (e.g., username/test-automation)
  • Add repository name to .env as GITHUB_REPO=username/test-automation
  • Tests will create and delete releases in this repository

arXiv Credentials (Optional)

arXiv submission tests are optional and require SWORD API credentials:

  1. Register for arXiv account
  2. Contact arXiv to enable SWORD API access (not available by default)
  3. Add credentials to .env

Most users can skip arXiv tests as they're not required for the core test suite.

Ollama Setup

Ollama tests require a running Ollama server with at least one model installed:

  1. Install Ollama:

    # macOS
brew install ollama

# Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
# Download from https://ollama.ai/download

  2. Start Ollama server:

    ollama serve

  3. Install at least one model:

    # Recommended models for testing
ollama pull llama3.1:latest    # 4.7GB, good quality
ollama pull llama3-gradient    # 4.7GB, 256K context
ollama pull gemma2:2b         # 2GB, fast and lightweight

# Verify models are available
ollama list

  4. Optional: Configure Ollama host (if not using default):

    export OLLAMA_HOST=http://localhost:11434

Note: Ollama tests use the local Ollama installation and do not require external credentials. Tests marked requires_ollama trigger the session harness in tests/conftest.py; the harness starts Ollama when possible, pulls smollm2 by default when no small/fast model is present, and fails with setup guidance if it cannot provide a runnable daemon/model.

Test Markers

Tests are marked with the following markers:

Marker Description Default handling
requires_ollama Needs local Ollama Harness starts/pulls or fails with setup guidance
requires_zenodo Needs Zenodo API access Opt-in when real Zenodo credentials are configured
requires_github Needs GitHub API access Opt-in when GITHUB_TOKEN / GITHUB_REPO are configured
requires_arxiv Needs arXiv API access Opt-in when arXiv credentials are configured
requires_latex Needs LaTeX installed Runs only on LaTeX-capable jobs
requires_network Needs internet access Runs only in network-enabled environments
requires_credentials Needs any external credentials Opt-in credentialed integration marker
private_project Needs a private sidecar project checkout Deselected by default; opt in when that project is present
external_fixture Needs downloaded external fixture trees Deselected by default; opt in after fixture setup

Running Tests Selectively

Run the default test selection

uv run pytest tests/

Run only tests that don't need credentials

uv run pytest tests/ -m "not requires_credentials"

Run only Zenodo tests

uv run pytest tests/ -m requires_zenodo

Run only GitHub tests

uv run pytest tests/ -m requires_github

Run only Ollama tests

# Ensure Ollama server is running first
ollama serve

# Then run tests
uv run pytest tests/ -m requires_ollama

Run only local tests (no network)

uv run pytest tests/ -m "not requires_network"

Run tests without LaTeX

uv run pytest tests/ -m "not requires_latex"

Combine markers

# Run tests that need neither credentials nor LaTeX
uv run pytest tests/ -m "not requires_credentials and not requires_latex"

# Run all external service tests
uv run pytest tests/ -m "requires_zenodo or requires_github"

Test Cleanup

Tests automatically clean up after themselves:

  • Zenodo: Test depositions are deleted after test completion
  • GitHub: Test releases and tags are deleted after test completion
  • Files: Temporary files are cleaned up using pytest's tmp_path fixtures

If cleanup fails (e.g., network error), you may need to manually delete test artifacts:

Manual Zenodo Cleanup

# Visit Zenodo sandbox and delete test depositions
https://sandbox.zenodo.org/deposit

Manual GitHub Cleanup

# Delete test releases via GitHub web UI or CLI
gh release delete test-release-12345678 --repo username/test-repo

# Delete test tags
git push --delete origin test-release-12345678

Security Best Practices

  1. Never commit credentials:

    • .env and test_credentials.yaml are in .gitignore
    • Always use infrastructure/config/.env.template for templates

  2. Use sandbox environments:

    • Always use Zenodo sandbox, not production
    • Create dedicated test repositories on GitHub

  3. Rotate tokens regularly:

    • GitHub tokens: Set expiration dates
    • Zenodo tokens: Regenerate periodically
    • Delete unused tokens

  4. Limit token scopes:

    • Only grant minimum required permissions
    • Use separate tokens for different purposes

  5. Protect your .env file:

    chmod 600 .env  # Owner read/write only

Troubleshooting

Tests are skipped

Problem: Tests marked with requires_* are being skipped

Solution:

  • Check .env file exists and has correct credentials
  • Verify environment variables are loaded: echo $ZENODO_SANDBOX_TOKEN
  • Run with -v to see skip reasons: uv run pytest tests/ -v

Zenodo API errors

Problem: Zenodo tests fail with authentication errors

Solutions:

  • Verify token is for sandbox, not production
  • Check token has deposit:write and deposit:actions scopes
  • Ensure token hasn't expired
  • Test token manually: curl -H "Authorization: Bearer YOUR_TOKEN" https://sandbox.zenodo.org/api/deposit/depositions

GitHub API rate limiting

Problem: Tests fail with rate limit errors

Solutions:

  • Authenticated requests have higher rate limits
  • Wait for rate limit reset (check X-RateLimit-Reset header)
  • Use a dedicated test account to avoid conflicts with personal usage

LaTeX compilation failures

Problem: Rendering tests fail

Solutions:

  • Install LaTeX: sudo apt-get install texlive-latex-base texlive-xetex (Ubuntu) or brew install mactex (macOS)
  • Verify installation: which xelatex
  • Skip LaTeX tests: uv run pytest tests/ -m "not requires_latex"

Cleanup failures

Problem: Test artifacts not being deleted

Solutions:

  • Check token permissions include deletion rights
  • Manually delete via web UI
  • Check cleanup logs for specific errors

Environment Setup Examples

Ubuntu/Debian

# Install LaTeX
sudo apt-get update
sudo apt-get install -y texlive-latex-base texlive-xetex pandoc

# Install Python dependencies
uv sync

# Configure credentials
cp .env.example .env
nano .env  # Add your credentials

# Run tests
uv run pytest tests/

macOS

# Install LaTeX
brew install --cask mactex
brew install pandoc

# Install Python dependencies
uv sync

# Configure credentials
cp .env.example .env
nano .env  # Add your credentials

# Run tests
uv run pytest tests/

Windows

# Install MiKTeX or TeX Live
# Install pandoc from https://pandoc.org/

# Install Python dependencies
uv sync

# Configure credentials
copy .env.example .env
notepad .env  # Add your credentials

# Run tests
uv run pytest tests/

CI/CD Integration

For continuous integration, store credentials as secrets:

GitHub Actions

name: Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6.0.2

      - name: Setup credentials
        run: |
          echo "ZENODO_SANDBOX_TOKEN=${{ secrets.ZENODO_SANDBOX_TOKEN }}" >> .env
          echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" >> .env
          echo "GITHUB_REPO=${{ secrets.GITHUB_REPO }}" >> .env

      - name: Install LaTeX
        run: |
          sudo apt-get update
          sudo apt-get install -y texlive-latex-base pandoc

      - name: Run tests
        run: uv run pytest tests/

Support

For issues or questions:

  1. Check this documentation
  2. Review test output with -v flag for detailed skip reasons
  3. Verify credentials are correctly configured
  4. Check external service status pages
  5. File an issue with details of the problem

See Also