Summary
Deleting ANY Switchyard VirtualModel can break inference for OTHER, still-live Switchyard VMs: the surviving VM stays catalog-visible but every inference request answers 500 {"detail": "Factory not found for config hash ..."}.
Tracked internally as NVBug 6563245 (filed 2026-08-05, hit daily by the QA skill-eval suite where VMs are created/destroyed per test).
Root cause
The IGW lifecycle dispatcher (middleware_registry._sdk_vm_to_plugin_vm) builds plugin VirtualModels without mapping the entity id — every on_virtual_model_upserted/on_virtual_model_destroyed hook receives id == "".
nemo_switchyard.middleware keyed its per-VM bookkeeping on that id:
on_virtual_model_upserted: VM_CONFIG_MAPPING[virtual_model.id] = hashes -> ALL VMs collapse into the single "" slot; each upsert overwrites the previous VM's entry.on_virtual_model_destroyed: VM_CONFIG_MAPPING.pop(vm_id) pops the shared slot, computes remaining_hashes without the surviving VMs, and unregisters factories they still reference.
Server log capture of the collision (revision 5d4aa25):
07:48:33.483 Registered factory 'nemo-switchyard-random_routing-0c264eb2f00f7d17' for VM 'default/skill-eval-swy-claude-...'
07:48:33.484 Unregistered factory 'nemo-switchyard-random_routing-0c264eb2f00f7d17' <- destroy of the PREVIOUS VM, 1ms later
(surviving VM's next request => 500 Factory not found)
Live reproduction (same revision, unfixed)
create race-a (switchyard random_routing) -> 201
create race-b (identical config) -> 201
invoke race-b -> 200
delete race-a -> 204
invoke race-b -> 500 "Factory not found for config hash 6e24859f4d219a91 (VM default/race-ctl-b)"
Fix
Key VM_CONFIG_MAPPING by vm_key (workspace/name) — the identity both hooks reliably carry. Two-line change in plugins/nemo-switchyard/src/nemo_switchyard/middleware.py plus regression tests reproducing the production (id-less) VM shape.
With the fix (source deployment on :8090): the same sequence ends invoke race-b -> 200, correctly routed; plugin suite 77 passed; mutation check confirms the new tests fail on the unfixed code.
Note: the SSE double-encoding on /v1/responses for Switchyard VMs is a SEPARATE defect (NVBug 6563029) and is not addressed here.
Summary
Deleting ANY Switchyard VirtualModel can break inference for OTHER, still-live Switchyard VMs: the surviving VM stays catalog-visible but every inference request answers
500 {"detail": "Factory not found for config hash ..."}.Tracked internally as NVBug 6563245 (filed 2026-08-05, hit daily by the QA skill-eval suite where VMs are created/destroyed per test).
Root cause
The IGW lifecycle dispatcher (
middleware_registry._sdk_vm_to_plugin_vm) builds pluginVirtualModels without mapping the entity id — everyon_virtual_model_upserted/on_virtual_model_destroyedhook receivesid == "".nemo_switchyard.middlewarekeyed its per-VM bookkeeping on that id:on_virtual_model_upserted:VM_CONFIG_MAPPING[virtual_model.id] = hashes-> ALL VMs collapse into the single""slot; each upsert overwrites the previous VM's entry.on_virtual_model_destroyed:VM_CONFIG_MAPPING.pop(vm_id)pops the shared slot, computesremaining_hasheswithout the surviving VMs, and unregisters factories they still reference.Server log capture of the collision (revision 5d4aa25):
Live reproduction (same revision, unfixed)
Fix
Key
VM_CONFIG_MAPPINGbyvm_key(workspace/name) — the identity both hooks reliably carry. Two-line change inplugins/nemo-switchyard/src/nemo_switchyard/middleware.pyplus regression tests reproducing the production (id-less) VM shape.With the fix (source deployment on :8090): the same sequence ends
invoke race-b -> 200, correctly routed; plugin suite 77 passed; mutation check confirms the new tests fail on the unfixed code.Note: the SSE double-encoding on
/v1/responsesfor Switchyard VMs is a SEPARATE defect (NVBug 6563029) and is not addressed here.