Skip to content

dbt metrics are ingested without owner/domain while the source table gets them #33285

Description

@RayVilaca

Affected module

Ingestion Framework

Describe the bug

The dbt ingestion pipeline creates Metric entities from dbt metrics, but it never propagates the governance metadata declared in the dbt YAML: owners, domains, glossary terms, tier, custom properties and unit of measurement are all dropped. Only name, displayName, description, metricType, metricExpression, granularity, relatedMetrics, dimensions, measures, filters and tags reach OpenMetadata.

In ingestion/src/metadata/ingestion/source/database/dbt/metadata.py, yield_dbt_metrics() builds the request as:

create_metric = CreateMetricRequest(
    name=metric_name,
    displayName=label or metric_name,
    description=description,
    metricType=metric_type,
    metricExpression=metric_expression,
    granularity=granularity,
    relatedMetrics=related_metrics,
    dimensions=dimensions or None,
    measures=measures or None,
    filters=filters or None,
    tags=tags or None,
)

owners, domains, dataProducts, reviewers, unitOfMeasurement and extension are valid fields of CreateMetricRequest (see openmetadata-spec/.../api/data/createMetric.json) but are never populated.

The root cause is that the metric path never reads the dbt node's meta / config.meta.

To Reproduce

  1. Define a dbt metric with OpenMetadata metadata in meta (both meta: and config.meta: were tested same outcome):
metrics:
  - name: segmented_drivers
    label: Segmented Drivers
    description: "Distinct drivers with a loyalty segment in the reference month."
    type: simple
    type_params:
      measure: segmented_drivers_count
    time_granularity: month
    config:
      tags:
        - loyalty_governance
      meta:
        openmetadata:
          owner: data_analytics
          tier: Tier.Tier1
          domain: Customer
          glossary:
            - Glossary.churn
          unit: count
          customProperties:
            data_steward:
              type: user
              fqn: some.user
  1. Declare the same meta.openmetadata block on the table the metric's semantic model points to, as a control.
  2. Run dbt build and run the dbt ingestion workflow with includeMetrics: true, includeOwners: true, includeTags: true, includeGlossary: true against the same manifest.
  3. Open the resulting Table and Metric in the UI.

Result: the Table shows owner, tier, domain, glossary term and custom properties. The Metric created from the very same run shows none of them (no owner, no domain, no tier, no glossary, no unit, no custom properties). No error or warning is logged and the metric is created successfully with the fields silently missing.

Expected behavior

yield_dbt_metrics() should parse the metric node's meta / config.meta with the same DbtMeta model used by process_dbt_meta() and populate the corresponding fields on CreateMetricRequest:

  • meta.openmetadata.owner -> owners (reusing get_dbt_owner())
  • meta.openmetadata.domain -> domains
  • meta.openmetadata.tier -> tier tag label
  • meta.openmetadata.glossary -> glossary term tags
  • meta.openmetadata.customProperties -> extension
  • meta.openmetadata.unit (or a dedicated key) -> unitOfMeasurement / customUnitOfMeasurement

so that a metric ends up as governed as the table it is derived from. Today the only workaround is a post-ingestion PATCH against the Metric entity for every metric, which has to be re-run after each ingestion.

OS

Ubuntu 22.04 (ingestion container)

Python version

3.11

OpenMetadata version

2.0.1

OpenMetadata Ingestion package version

openmetadata-ingestion[dbt]==2.0.1

Additional context

No response

Pre-submission checklist

  • I searched for duplicate issues.
  • I removed credentials, hostnames, emails, and other sensitive data from logs and config.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    • Status
      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions