Skip to content

README.md

Latest commit

 

History

History
245 lines (167 loc) · 7.38 KB

README.md

File metadata and controls

245 lines (167 loc) · 7.38 KB
Raw

From APIs to Warehouses: AI-Assisted Data Ingestion with dlt

Welcome to the Data Engineering Zoomcamp 2026 workshop!

In this workshop, you'll use an AI-powered IDE to build a complete data pipeline. Using simple prompts, you can go from an API to a local data warehouse with dlt (data load tool). The AI handles the code generation. You focus on the results.

What You'll Build

By the end of this workshop, you will have:

  1. A working dlt pipeline that extracts data from the Open Library API
  2. Normalized relational tables stored in DuckDB
  3. The ability to query, inspect, and visualize your data
  4. Experience using AI-assisted development for data engineering

No API key required! The Open Library API is completely open and doesn't require authentication. You can start building immediately.

Prerequisites

Before the workshop, make sure you have the following set up:

1. Understand What dlt Does (Recommended for Beginners)

If you're unfamiliar with dlt and what the library does, we recommend reading through the included Jupyter notebook before the workshop.

Open the notebook in Google Colab

It walks through dlt step by step:

  • What a dlt source and pipeline are
  • How data moves through Extract, Normalize, and Load
  • How to inspect the loaded data

Understanding these concepts will help you know what the agent-generated code is actually doing.

You do not need to clone the repo to follow the workshop. The dlt init command scaffolds everything you need.

2. An Agentic IDE

You'll need an AI-powered code editor that can understand context and generate code from natural language. We recommend:

IDE Description
Cursor VS Code fork with built-in AI assistance (recommended)
Windsurf Alternative agentic IDE
VS Code + GitHub Copilot Works, but less integrated

3. Python 3.11+

python --version  # Should be 3.11 or higher

4. uv (Recommended) or pip

We use uv for fast dependency management:

# Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

Workshop Instructions

Step 1: Create a New Project Folder

Create a fresh folder for your pipeline and open it in Cursor (or your preferred agentic IDE):

mkdir my-dlt-pipeline
cd my-dlt-pipeline

Step 2: Add the dlt MCP Server Config

Choose the setup for your IDE:

Cursor - go to Settings → Tools & MCP → New MCP Server and add:

{
  "mcpServers": {
    "dlt": {
      "command": "uv",
      "args": [
        "run",
        "--with",
        "dlt[duckdb]",
        "--with",
        "dlt-mcp[search]",
        "python",
        "-m",
        "dlt_mcp"
      ]
    }
  }
}

VS Code (Copilot) - create .vscode/mcp.json in your project folder:

{
  "servers": {
    "dlt": {
      "command": "uv",
      "args": [
        "run",
        "--with",
        "dlt[duckdb]",
        "--with",
        "dlt-mcp[search]",
        "python",
        "-m",
        "dlt_mcp"
      ]
    }
  }
}

Claude Code - run in your terminal:

claude mcp add dlt -- uv run --with "dlt[duckdb]" --with "dlt-mcp[search]" python -m dlt_mcp

This enables the dlt MCP server, which gives the AI access to dlt documentation, code examples, and your pipeline metadata.

Step 3: Install dlt Workspace

pip install "dlt[workspace]"

Step 4: Initialize the dlt Project

dlt init dlthub:open_library duckdb

This scaffolds the pipeline files and configuration for Open Library. You now have everything you need to start prompting.

📖 Reference: Open Library Workspace Instructions

Step 5: Prompt the Agent to Build and Run the Pipeline

This is where the magic happens. The dlt init command scaffolds sample prompts you can use. Here's an example to get started:

Please generate a REST API Source for Open Library API, as specified in @open_library-docs.yaml
Start with endpoint(s) books and skip incremental loading for now.
Place the code in open_library_pipeline.py and name the pipeline open_library_pipeline.
If the file exists, use it as a starting point.
Do not add or modify any other files.
Use @dlt rest api as a tutorial.
After adding the endpoints, allow the user to run the pipeline with python open_library_pipeline.py and await further instructions.

Feel free to tweak the prompt based on your objective. The agent will:

  1. Generate the pipeline code
  2. Run the pipeline
  3. Load data into your local DuckDB database

All from a single prompt.

Step 6: Debug with the Agent

If there are any errors, paste them into the chat and let the AI resolve them. This is the power of AI-assisted development: you iterate quickly without getting stuck.

Step 7: Inspect Pipeline Data with the dlt Dashboard

Once your pipeline runs successfully, launch the dashboard to inspect your data and metadata:

dlt pipeline open_library_pipeline show

This opens a web app where you can:

  • View pipeline state and run history
  • Explore schemas, tables, and columns
  • Query the loaded data
  • Debug any issues

📖 Reference: dlt Dashboard Documentation

Step 8: Inspect the Pipeline via Chat

With the dlt MCP server configured, you can ask the AI about your pipeline directly:

"What tables were created in the pipeline?"
"Show me the schema for the books table."
"How many rows were loaded?"

The agent has access to your pipeline metadata and can answer these questions.

Step 9 (Bonus): Build Visualizations with marimo + ibis

Take your analysis further by creating interactive reports with marimo notebooks and ibis.

Prompt the agent to build a visualization:

"Create a marimo notebook that visualizes the top 10 authors by book count. Use ibis for data access. Reference: https://dlthub.com/docs/general-usage/dataset-access/marimo"

By providing the docs link, the agent will use the correct stack.

Run your notebook:

# Edit mode (for development)
marimo edit your_notebook.py

# Run mode (view the report)
marimo run your_notebook.py

📖 Reference: Explore Data with marimo

Homework

You've seen me do it, now it's your turn!

See dlt_homework.md for instructions.

Resources

Resource Link
dlt Documentation dlthub.com/docs
Open Library Workspace Guide dlthub.com/workspace/source/open-library
dlt Dashboard Docs dlthub.com/docs/general-usage/dashboard
marimo + dlt Guide dlthub.com/docs/general-usage/dataset-access/marimo
Open Library API openlibrary.org/developers/api

Workshop by dltHub for the Data Engineering Zoomcamp 2026