Feature Request: Auto-link resources by name for PM-in-UI workflows

[qjflores]

Feature Request: Auto-link resources by name for PM-in-UI workflows

Summary

When Product Managers create events/categories in the RudderStack UI and engineers use the CLI to manage tracking plans, there's significant friction due to the requirement for explicit metadata.import stanzas with remote_id mappings. This creates maintenance burden and error-prone manual steps.

Current Workflow

Our team follows this workflow:

1. PM creates events/categories in the RudderStack staging UI
2. Engineer imports using rudder-cli import workspace
3. Engineer manually adds metadata.import sections with remote_id mappings to link local definitions to remote resources
4. Engineer pushes to git
5. GitHub Action applies to production on merge

The Problem

Resources created in the UI don't have external_id set, so the CLI treats them as "invisible" when loading remote state:

// category.go line 170
categories, err := p.client.GetCategories(ctx, catalog.ListOptions{HasExternalID: lo.ToPtr(true)})

This means engineers must:

1. Run import workspace to get resources into an imported/ directory
2. Manually copy remote_id values from imported files
3. Add metadata.import.workspaces stanzas to their existing files
4. For each workspace (staging, production), repeat this process with different remote_id values

Example of the friction

A simple category import requires this boilerplate:

metadata:
  name: event-categories
  import:
    workspaces:
      - workspace_id: "STAGING_WORKSPACE_ID"
        resources:
          - local_id: "canvas"
            remote_id: "cat_3xxxxxxxxxxxxexxxxxxxxxxxx"
      - workspace_id: "PRODUCTION_WORKSPACE_ID"
        resources:
          - local_id: "canvas"
            remote_id: "cat_DIFFERENT_PROD_ID"

This scales linearly - every PM-created resource needs entries for every workspace.

Error we encounter without this

Error: syncing the state: creating category resource in upstream catalog: 
sending request: http status code: 400, error code: '', 
error: 'Category with name Canvas already exists'

Proposed Solutions

Option 1: --match-by-name flag (Recommended)

Add a flag to apply that enables name-based matching for resources without external_id:

rudder-cli tp apply -l . --match-by-name

Behavior:

• When a local resource doesn't exist in remote state (no matching external_id)
• AND a remote resource with the same name exists (without external_id)
• Prompt user: "Found existing 'Canvas' category. Link and set external_id? [y/n]"
• If yes, perform an import operation instead of create

Option 2: --auto-link flag with interactive confirmation

rudder-cli tp apply -l . --auto-link

# Output:
# The following resources exist remotely but aren't linked:
#   - category:canvas (remote: cat_37tIcdNnXjm90UeONAx737RgfJm)
#   - event:canvas-project-created (remote: ev_37tL0Edtrdzhz8TWoXm1Jj9Y5Hh)
# 
# Link these resources? [y/n]

Option 3: Import-and-merge mode

rudder-cli import workspace -l . --merge

Instead of writing to imported/ directory, automatically:

1. Match imported resources to existing local definitions by id
2. Add/update metadata.import sections in-place
3. Report what was merged

Option 4: Workspace-agnostic external_id

If the CLI set external_id on resources at creation time AND the UI also set external_id, resources would be trackable across both interfaces without explicit mappings.

Environment

• CLI Version: 0.11.2
• Workflow: PM creates in staging UI → Engineer imports → GitHub Action applies to production

Workaround

Currently we:

1. Temporarily remove new resources from local files
2. Run import workspace
3. Copy remote_id values from imported files
4. Restore local files with added metadata.import stanzas
5. Repeat for each workspace

This is error-prone and doesn't scale.

Impact

• Developer time: ~10-15 minutes per resource for the import dance
• Error prone: Manual copy-paste of UUIDs
• Maintenance burden: Import stanzas grow linearly with resources × workspaces
• Onboarding friction: New engineers struggle with this non-obvious workflow

Related

• The HasExternalID: true filter in LoadResourcesFromRemote methods is the root cause
• Name-based matching would need collision handling (prompt user if ambiguous)

[qjflores]

Additional friction: remote_id not accessible from UI

Just encountered this exact scenario and wanted to document another layer of friction.

The situation:

1. An event my_event_that_exists was created in the RudderStack UI
2. I added the same event to my local YAML with a matching id and name
3. Running apply fails with: Error: syncing the state: track event 'my_event_that_exists' already exists

The workaround requires the remote_id

The problem: The remote_id (ev_... for events, cat_... for categories) is not exposed anywhere in the RudderStack UI. There's no way to look up this internal ID from the Data Catalog interface.

This leaves two options, both painful:

1. Temporary delete + import dance: Remove the local definition, apply to sync, run import workspace, find the ID in the imported files, then re-add with the mapping
2. API spelunking: Manually call the RudderStack API to look up resources by name and find their IDs

For teams where PMs create resources in staging UI and engineers manage the IaC layer, this creates a significant barrier. The --match-by-name or --auto-link proposals would eliminate this friction entirely by handling the linkage automatically.

[sivashanmukh]

Thanks for the thorough write-up and the prs, @qjflores and @ric03uec.

We recognize this gap — the manual remote_id mapping workflow doesn't scale, especially for teams where PMs create resources in the UI and engineers manage the IaC layer. This is something we're actively working on.

We're narrowing down on an approach that addresses the core limitation (UI-created resources being invisible to the CLI across workspaces) and plan to share a proposal next week. Your suggested options are a helpful reference as we evaluate the design.

Appreciate the patience.