Affected module
Ingestion Framework
Describe the bug
The dbt ingestion pipeline creates Metric entities from dbt metrics, but it never propagates the governance metadata declared in the dbt YAML: owners, domains, glossary terms, tier, custom properties and unit of measurement are all dropped. Only name, displayName, description, metricType, metricExpression, granularity, relatedMetrics, dimensions, measures, filters and tags reach OpenMetadata.
In ingestion/src/metadata/ingestion/source/database/dbt/metadata.py, yield_dbt_metrics() builds the request as:
create_metric = CreateMetricRequest(
name=metric_name,
displayName=label or metric_name,
description=description,
metricType=metric_type,
metricExpression=metric_expression,
granularity=granularity,
relatedMetrics=related_metrics,
dimensions=dimensions or None,
measures=measures or None,
filters=filters or None,
tags=tags or None,
)
owners, domains, dataProducts, reviewers, unitOfMeasurement and extension are valid fields of CreateMetricRequest (see openmetadata-spec/.../api/data/createMetric.json) but are never populated.
The root cause is that the metric path never reads the dbt node's meta / config.meta.
To Reproduce
- Define a dbt metric with OpenMetadata metadata in meta (both meta: and config.meta: were tested same outcome):
metrics:
- name: segmented_drivers
label: Segmented Drivers
description: "Distinct drivers with a loyalty segment in the reference month."
type: simple
type_params:
measure: segmented_drivers_count
time_granularity: month
config:
tags:
- loyalty_governance
meta:
openmetadata:
owner: data_analytics
tier: Tier.Tier1
domain: Customer
glossary:
- Glossary.churn
unit: count
customProperties:
data_steward:
type: user
fqn: some.user
- Declare the same meta.openmetadata block on the table the metric's semantic model points to, as a control.
- Run dbt build and run the dbt ingestion workflow with includeMetrics: true, includeOwners: true, includeTags: true, includeGlossary: true against the same manifest.
- Open the resulting Table and Metric in the UI.
Result: the Table shows owner, tier, domain, glossary term and custom properties. The Metric created from the very same run shows none of them (no owner, no domain, no tier, no glossary, no unit, no custom properties). No error or warning is logged and the metric is created successfully with the fields silently missing.
Expected behavior
yield_dbt_metrics() should parse the metric node's meta / config.meta with the same DbtMeta model used by process_dbt_meta() and populate the corresponding fields on CreateMetricRequest:
- meta.openmetadata.owner -> owners (reusing get_dbt_owner())
- meta.openmetadata.domain -> domains
- meta.openmetadata.tier -> tier tag label
- meta.openmetadata.glossary -> glossary term tags
- meta.openmetadata.customProperties -> extension
- meta.openmetadata.unit (or a dedicated key) -> unitOfMeasurement / customUnitOfMeasurement
so that a metric ends up as governed as the table it is derived from. Today the only workaround is a post-ingestion PATCH against the Metric entity for every metric, which has to be re-run after each ingestion.
OS
Ubuntu 22.04 (ingestion container)
Python version
3.11
OpenMetadata version
2.0.1
OpenMetadata Ingestion package version
openmetadata-ingestion[dbt]==2.0.1
Additional context
No response
Pre-submission checklist
Affected module
Ingestion Framework
Describe the bug
The dbt ingestion pipeline creates Metric entities from dbt metrics, but it never propagates the governance metadata declared in the dbt YAML: owners, domains, glossary terms, tier, custom properties and unit of measurement are all dropped. Only name, displayName, description, metricType, metricExpression, granularity, relatedMetrics, dimensions, measures, filters and tags reach OpenMetadata.
In ingestion/src/metadata/ingestion/source/database/dbt/metadata.py, yield_dbt_metrics() builds the request as:
owners, domains, dataProducts, reviewers, unitOfMeasurement and extension are valid fields of CreateMetricRequest (see openmetadata-spec/.../api/data/createMetric.json) but are never populated.
The root cause is that the metric path never reads the dbt node's meta / config.meta.
To Reproduce
Result: the Table shows owner, tier, domain, glossary term and custom properties. The Metric created from the very same run shows none of them (no owner, no domain, no tier, no glossary, no unit, no custom properties). No error or warning is logged and the metric is created successfully with the fields silently missing.
Expected behavior
yield_dbt_metrics() should parse the metric node's meta / config.meta with the same DbtMeta model used by process_dbt_meta() and populate the corresponding fields on CreateMetricRequest:
so that a metric ends up as governed as the table it is derived from. Today the only workaround is a post-ingestion PATCH against the Metric entity for every metric, which has to be re-run after each ingestion.
OS
Ubuntu 22.04 (ingestion container)
Python version
3.11
OpenMetadata version
2.0.1
OpenMetadata Ingestion package version
openmetadata-ingestion[dbt]==2.0.1
Additional context
No response
Pre-submission checklist