Is there a way to get a stack dump from DBT-DuckDB when a segmentation fault occurs

[williamkaper]

Is there a way to force DBT-DUCKDB (or DBT itself) to generate a detailed stack dump when a segmentation fault occurs? We are seeing occasional segmentation faults occur when running large projects in DBT through DuckDB that are repeatable with our customer's confidential data, but I can't figure out a way to build a test harness to reproduce it without spending the next month build a project full of fake data to try to replication what is happening. We do know that it seems to happen more frequently with threads > 1 (we were running 4) and it is exacerbated when using something like nice -n10 ionice -c2 -n4 dbt build....

When the issue occurs, we either see a hard segmentation fault, or we see the MD5 function start returning back the same results for different input values (which happened a while ago which caused us to write a test that catches it after the model is built).

We do our builds in Linux Ubuntu 24.04 with Python 3.12.x - all the DBT and DuckDB libraries are latest.

20:30:04  127 of 329 START sql table model datahub_transient.transient.int_org_ledger_entry_components  [RUN]
20:30:04  128 of 329 START snapshot snapshots.stg_ss_invoices ............................ [RUN]
./bd: line 3:  2854 Segmentation fault      (core dumped) dbt build $QUIET_ARG $START_MODEL --vars ''$DBT_ARG'' --exclude $EXCLUDE --project-dir "$DBT_STAGE_DIR" --profiles-dir "$DBT_STAGE_DIR"
DBT failed! Exiting!
ubuntu@ip-10-1-2-197:~/code/HubiFi/data-hub$ /usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

And this is the last part of the dbt.log:

20:30:04.384739 [debug] [Thread-1 (]: SQL status: OK in 0.002 seconds
20:30:04.385983 [debug] [Thread-1 (]: On snapshot.datahub.stg_ss_cost_of_revenue_returns: COMMIT
20:30:04.386371 [debug] [Thread-1 (]: Using duckdb connection "snapshot.datahub.stg_ss_cost_of_revenue_returns"
20:30:04.386757 [debug] [Thread-1 (]: On snapshot.datahub.stg_ss_cost_of_revenue_returns: COMMIT
20:30:04.392315 [debug] [Thread-1 (]: SQL status: OK in 0.005 seconds
20:30:04.393303 [debug] [Thread-1 (]: Using duckdb connection "snapshot.datahub.stg_ss_cost_of_revenue_returns"
20:30:04.393684 [debug] [Thread-1 (]: On snapshot.datahub.stg_ss_cost_of_revenue_returns: /* {"app": "dbt", "dbt_version": "1.9.1", "profile_name": "data-hub", "target_name": "local", "node_id": "snapshot.datahub.stg_ss_cost_of_revenue_returns"} */
drop table if exists "stg_ss_cost_of_revenue_returns__dbt_tmp20250516203003960038" cascade
20:30:04.399013 [debug] [Thread-1 (]: SQL status: OK in 0.005 seconds
20:30:04.400870 [debug] [Thread-1 (]: On snapshot.datahub.stg_ss_cost_of_revenue_returns: Close
20:30:04.670481 [debug] [Thread-4 (]: SQL status: OK in 0.639 seconds
20:30:04.672517 [debug] [Thread-4 (]: On snapshot.datahub.stg_ss_invoice_item_components: COMMIT
20:30:04.673066 [debug] [Thread-4 (]: Using duckdb connection "snapshot.datahub.stg_ss_invoice_item_components"
20:30:04.673571 [debug] [Thread-4 (]: On snapshot.datahub.stg_ss_invoice_item_components: COMMIT
20:30:04.699930 [debug] [Thread-3 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'eaaaaf98-3735-4920-8c81-6c48d63465c8', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7bc5e3fd2120>]}
20:30:04.705738 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'eaaaaf98-3735-4920-8c81-6c48d63465c8', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7bc5e01322a0>]}
20:30:04.742816 [debug] [Thread-3 (]: An error was encountered while trying to send an event
20:30:04.743791 [info ] [Thread-1 (]: 126 of 329 OK snapshotted snapshots.stg_ss_cost_of_revenue_returns ............. [OK in 0.87s]
20:30:04.745519 [info ] [Thread-3 (]: 125 of 329 OK snapshotted snapshots.stg_ss_cost_of_revenue ..................... [OK in 5.18s]
20:30:04.746238 [debug] [Thread-1 (]: Finished running node snapshot.datahub.stg_ss_cost_of_revenue_returns
20:30:04.746835 [debug] [Thread-3 (]: Finished running node snapshot.datahub.stg_ss_cost_of_revenue
20:30:04.747332 [debug] [Thread-1 (]: Began running node model.datahub.int_org_ledger_entry_components
20:30:04.747966 [debug] [Thread-3 (]: Began running node snapshot.datahub.stg_ss_invoices
20:30:04.748982 [info ] [Thread-1 (]: 127 of 329 START sql table model datahub_transient.transient.int_org_ledger_entry_components  [RUN]
20:30:04.749513 [info ] [Thread-3 (]: 128 of 329 START snapshot snapshots.stg_ss_invoices ............................ [RUN]
20:30:04.750036 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly snapshot.datahub.stg_ss_cost_of_revenue_returns, now model.datahub.int_org_ledger_entry_components)
20:30:04.750439 [debug] [Thread-3 (]: Re-using an available connection from the pool (formerly snapshot.datahub.stg_ss_cost_of_revenue, now snapshot.datahub.stg_ss_invoices)
20:30:04.750848 [debug] [Thread-1 (]: Began compiling node model.datahub.int_org_ledger_entry_components

[jwills]

Ooh, interesting one-- okay. So some chatting with Claude and o3 leads me to suggest enabling the python faulthandler module in conjunction with a programmatic invocation of dbt build via: https://docs.getdbt.com/reference/programmatic-invocations

The rub is that's not going to catch anything inside of DuckDB, and I don't know if doing e.g. a debug DuckDB build is worth it in order to track this down (last time I checked, you have to do them yourself and they are pretty slow, but there may be new info here.)

I'm also curious if you can share some more about the environment/setup-- the LLMs feel that the most likely culprit given your description is different threads trying to re-use the same underlying DuckDBPyConnection to perform some operation on the DB, and while we try to be careful about how those connections are handled in dbt-duckdb itself, it's maybe possible that something is going sideways in a plugin or in a dbt-duckdb python model?

I'm going to have Claude cook up an extra safe

[jwills]

ah, so pretty vanilla aside from the postgres scanner extension, okay-- did the faulthandler reveal anything?

[williamkaper]

No sorry I didn't realize that was something I could enable myself. Can you provide an example of how I would enable it?

[jwills]

yeah I was thinking something like this-- I had o3 write it up: https://chatgpt.com/share/682cea9a-9034-800e-a912-b4a4c16576ee

[jwills]

not totally sure this will work-- like, it may just report that the seg fault is happening somewhere inside of duckdb, which isn't super-helpful-- but I thought it would help narrow down the possibilities here and could be useful if it isn't actually happening inside of duckdb, which would be super-interesting

[williamkaper]

Hmm. It might take some work to do this since we run our build through a bash script build driver right now that run through time calling DBT once per accounting period we want to build. I'd need to do something to basically proxy the call through a python shim that did this work. I probably won't get to try that for a few days.