Skip to content

Reject metadata columns uniformly during table loading #3310

Description

@jtuglu1

Context

PR #3284 added a metadata-column rejection to TableConfiguration::try_new while validating
reconstructed SnapshotHint metadata. That changes normal table loading as well as the new FFI path.
The check was removed from the PR so metadata validation remains uniform while its compatibility
impact is reviewed.

Metadata::try_new already rejects top-level metadata columns for table creation, but metadata
loaded from the log reaches TableConfiguration::try_new through a different path.

Proposed validation

Consider enforcing this check for every table-state construction path:

if let Some(field) = logical_schema.fields().find(|field| field.is_metadata_column()) {
    return Err(Error::Schema(format!(
        "Table schema must not contain metadata columns. Found metadata column: '{}'",
        field.name()
    )));
}

The implementation should avoid a special rule for FFI hints. It should cover ordinary snapshot
loading, incremental loading, reconstructed hints, and table creation through a shared validation
boundary, or document why any path intentionally differs.

Acceptance criteria

  • Determine whether existing Delta tables can contain these annotations in persisted schemas.
  • Choose and document one policy for all table-configuration construction paths.
  • Add tests for ordinary log loading and SnapshotHint loading with equivalent schemas.

Related review: #3284.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions