Add conditional runprov metadata extractor for datasets with run records

[Copilot]

The runprov extractor extracts provenance metadata from DataLad run command records but fails on datasets without them. This adds detection and conditional enablement.

Changes

Database schema

• Added has_run_records boolean field to RepoUrl model
• Migration: f1a2b3c4d5e6_add_has_run_records_to_repourl.py

Detection logic (datalad_registry/utils/datalad_tls.py)

• New has_datalad_run_records(ds) function searches git history for "DATALAD RUNCMD" markers
• Searches all branches using git log --all --grep="DATALAD RUNCMD" -n 1
• Returns boolean, stored in database during dataset processing

Conditional enablement (datalad_registry/tasks/__init__.py)

• _update_dataset_url_info() populates has_run_records field
• New _get_extractors_for_url() adds runprov only when has_run_records=True
• extract_ds_meta() returns ABORTED if runprov called without run records
• Extractor name mapping: "runprov" → "metalad_runprov" for metalad compatibility

Example

# Dataset processing automatically detects run records
dataset_url.has_run_records = has_datalad_run_records(ds)

# Extractor selection is conditional
extractors = _get_extractors_for_url(url)  
# Returns [..., "runprov"] only if url.has_run_records is True

Tests

• Unit tests for has_datalad_run_records() function
• Integration tests for conditional extractor behavior

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• esm.ubuntu.com
  • Triggering command: /usr/lib/apt/methods/https /usr/lib/apt/methods/https unner/pytest-1/t-c sql/16/bin/pg_codiff.ignoreSubmodules=none git -c dules=none -c git for-each-ref --format=%(refna-c refs/heads git -c dules=none -c t commit -m Add file1.txt, f--noprofile git (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>add runprov metadata extractor</issue_title>
<issue_description>https://github.com/datalad/datalad-metalad/blob/fbfbf88c4878cdeabb5e20d58fe2ef7db4a77bf3/datalad_metalad/extractors/runprov.py

• we would need to enable detection based on git log history to enable it. eventually we might need one for git diff as well</issue_description>

<agent_instructions>review current data structure on either we anyhow annotate for presence of run records like we do in the datalad usage dashboard. if not -- we might need to clone with history (but not necessarily with all the blobs (--filter=blob:none option for git clone) ) and check if history has any datalad RUNCMD markers in the commit. And if so -- then enable https://github.com/datalad/datalad-metalad/blob/master/datalad_metalad/extractors/runprov.py metadata extractor</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes add runprov metadata extractor #140

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.