Skip to content

Add Python model support for MySQL (pandas & PySpark)#182

Open
tripleaceme wants to merge 1 commit intodbeatty10:mainfrom
tripleaceme:feature/python-model-support
Open

Add Python model support for MySQL (pandas & PySpark)#182
tripleaceme wants to merge 1 commit intodbeatty10:mainfrom
tripleaceme:feature/python-model-support

Conversation

@tripleaceme
Copy link
Copy Markdown

@tripleaceme tripleaceme commented Mar 11, 2026

Summary

  • Adds Python model support for MySQL 8.0+ with two execution backends: pandas (default, via SQLAlchemy) and PySpark (via JDBC)
  • Python models run locally as subprocesses and write results back to MySQL
  • Supports both table and incremental materializations with Python language

Implementation Details

New Files

  • python_submissions.py - PandasPythonJobHelper and PySparkPythonJobHelper
  • py_write_table macro for both backends
  • Table materialization with supported_languages=['sql', 'python']
  • Unit tests for submission helpers

Modified Files

  • MySQLAdapter - added Python model properties and methods
  • create_table_as macro - accepts language parameter
  • Incremental materialization - Python language support
  • setup.py - optional extras (python-models, pyspark)
  • README.md - Python models documentation

Architecture

  • Pandas: model(dbt, engine) returns pd.DataFrame, writes via SQLAlchemy
  • PySpark: model(dbt, spark) returns DataFrame, writes via JDBC
  • Configurable via submission_method, timeout, and spark_master config

Test plan

  • Verify existing SQL models still work (no regressions)
  • Test pandas Python model with table and incremental
  • Test PySpark Python model with table and incremental
  • Test special characters in MySQL password
  • Test timeout handling

Closes #180

Implements dbt Python models for the MySQL adapter with two execution
backends: pandas (via SQLAlchemy) and PySpark (via JDBC). Models run
locally as subprocesses and write results back to MySQL.

Closes dbeatty10#180

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MySQL Python model support

1 participant