Deduplicate firmware_image_type and migrate renumbered devices over OTA

[eakorolev]

Problem

firmware_image_type is meant to be unique per device, but 27 ids are each shared by two or more devices. OTA matchers (zigpy / Home Assistant ZHA and Zigbee2MQTT) normally tell such devices apart by manufacturerName, so most of these collisions are currently masked — but the guarantee is fragile: where two colliding devices also share a manufacturer name (the two BOARD__MHCOZY_TS0004 variants both report _TZ3210_imaccztn) the match breaks and neither device is offered an update, and any index entry left without a manufacturerName matches the wrong device.

Fix

Renumber the later-registered device in each collision to a free id (47116–47147), restoring one id per device. A CI check (helper_scripts/check_image_types.py) keeps it that way: on pull requests it validates only the entries changed against the base branch, on main it validates the whole db.

Renumbering on its own would strand already-flashed units — they keep reporting the old id, for which no image exists anymore. To avoid a manual re-flash, each renumbered device also builds a migration OTA: its new firmware wrapped in an OTA header carrying the old id. A deployed unit still reporting the old id matches it, updates once over the air, and afterwards reports the new id. This is driven by a new migration_from_image_type field in device_db.yaml; board.mk emits one migration image per old id into the normal index. Confirmed safe on zigpy 2.0.0 and zigbee-herdsman 10.7.0 — both enforce manufacturerName, so the migration image and the keeper's image at the same old id never collide or cross-flash.

Renumbered devices

old id	stays on id	renumbered
43510	SWITCH_MOES_ALL_TS0012	SWITCH_LERLINK_TS0012 → 47116
43527	MODULE_AVATTO_DRY_TS0001	MODULE_AVATTO_A_TS0001 → 47117
43533	SWITCH_MOES_ALL_TS0013	SWITCH_LERLINK_TS0013 → 47118
43547	MODULE_TUYA_OXT_TS0002	MODULE_EKAZA_TS0002 → 47119
43556	OUTLET_BSEED_PM_TS011F	PLUG___AUBESS_PM_TS011F → 47120
43563	REMOTE_IHSENO_TS0044	REMOTE_IHSENO_TS0046 → 47121
43614	MODULE_NOVADIGITAL_TS0001	SWITCH_NOVADIGITAL_TS0003 → 47122
43616	MODULE_GIRIER_TS130F_1GANG	SWITCH_LORATAP_TS130F → 47123
43623	MODULE_TUYA_ZTU_TS0001	BOARD__SCIMAGIC_TS0001 → 47124
43633	SWITCH_BSEED_TS0726_L_2GANG	MODULE_NOUS_B1Z_TS0001 → 47125
45575	SWITCH_BSEED_TOUCH_TS0001	SWITCH_BSEED_TOUCH_TS0001_2 → 47126
45577	SWITCH_BSEED_TS0726_3GANG	MODULE_TUYA_NOVATO_QS_S05_TS0001 → 47127
45578	SWITCH_BSEED_TOUCH_TS0002	MODULE_TUYA_NOVATO_QS_S05_TS0011 → 47128, SWITCH_BSEED_TOUCH_TS0002_2 → 47129
45585	MODULE_GIRIER_TS0001	MODULE_GIRIER_ZTU_TS0001 → 47130
45589	SWITCH_ZEMISMART_2_TS0012	SWITCH_ZEMISMART_2_TS0011 → 47131
45599	BOARD__SCIMAGIC_TS0002_1	DIN_RELAY_TUYA_TS011F_PM → 47132
45631	BOARD__MHCOZY_TS0004_1	BOARD__MHCOZY_TS0004_3 → 47133
45635	REMOTE_MOES_SWITCH_A_TS0042	REMOTE_LIDL_BUTTON_TS004F → 47134
45636	REMOTE_TUYA_SWITCH_TS0041	REMOTE_TUYA_BUTTON_TS0041 → 47135, REMOTE_TUYA_BUTTON_TS0041_2 → 47136
45637	REMOTE_TUYA_SWITCH_TS0042	REMOTE_TUYA_SWITCH_TS0043 → 47137, REMOTE_TUYA_SWITCH_TS0043_1 → 47138
45681	OUTLET_MOES_PM_TS011F	OUTLET_TUYA_PM_TS011F → 47139
47002	REMOTE_TUYA_TS004F_2	REMOTE_HOBEIAN_BUTTON_TS0001 → 47140, REMOTE_IHSENO_BUTTON_TS0041 → 47141
47003	OUTLET_BSEED_PM_TS011F_1	PLUG___LELLKI_TS0001 → 47142
47004	OUTLET_BSEED_PM_TS011F_2	MODULE_TUYA_COMMON_TS0004 → 47143
47005	SWITCH_TUYA_TS0001	MODULE_TUYA_COMMON2_TS0003 → 47144, SWITCH_TUYA_A_TS0001 → 47145
47007	MODULE_TUYA_COMMON_TS0001_2	SWITCH_BSEED_TS0726_L_3GANG_2 → 47146
47009	SWITCH_TUYA_A_TS0001_1	OUTLET_MAKEGOOD_PM_TS011F_2GANG → 47147

[andrei-lazarov]

Thanks for your work!
However I don't think we need a unique image type for each device. It only complicates things imo.

If they have the same chip, they run the same firmware. Only the default config string is different. So the current approach is very inefficient.. 🙂 We need a big refactor for the project in this regard.

I would like to merge the image types instead, maybe into: Telink router, Telink end-device, Silabs router, Silabs end-device.

Also not sure how the migration would take place? The migration index would have the initial problem where the types are duplicated?

I'm dreaming of a big refactor like this: GitHub action publishes the standard firmware (few image types) -> it works as an update for all devices.
If the user needs to migrate, or to flash it on stock device -> user uploads firmware into a web-tool -> user selects image type, pinout, default config -> tool injects the configuration and generates the custom ota file

[eakorolev]

Thanks @andrei-lazarov, and I like the bigger direction — merging image types, plus a web-tool for migration and stock-flashing, sounds good.

To be honest about where this came from: I started while trying to fix OTA updates on my own devices. After digging in, I found the duplicate image types were not my problem at all. My real issue was the indexes: I have a mix of devices — Telink routers, Silabs routers, and Silabs battery end-devices — so I'd loaded both the router and the end_device index at once, without realizing that puts each device into both, and zigpy then sees it twice and ignores both. So this PR doesn't actually fix my original pain; that part is on the index side, and I'll think about it separately.

---

Looking at the duplicates themselves, they all seem accidental: two people add a device in separate PRs at about the same time, both correctly pick a free id, and the collision only appears once both are merged. Nobody did anything wrong.

I opened the PR mostly to bring the repo in line with what it already says — device_db_explained.md lists firmware_image_type as "unique per device." The CI check just keeps that true. The other option is to drop the check and the uniqueness rule together — but that's the bigger refactor you're describing, changing both the rule and the tooling at once. It seemed simpler to enforce the current rule now, and rethink the rule and the tools together later, rather than leave the repo in a state it contradicts.

The part I'd keep no matter which way image types go: the migration mechanism — a new migration_from_image_type field plus a small board.mk step. Whatever the ids become later (including merging them into a few), the hard part is moving devices that are already installed from one image type to another. This does it over the air: build the firmware, put the old image type in the OTA header, and publish. A device still reporting the old id picks it up, updates once, and then reports the new id — no cable. That's exactly what your refactor would need to move existing devices onto the merged ids over OTA, next to the web-tool for the stock/manual cases.

On your question — "the migration index would have duplicated types" — that part is fine, and I checked it. Both zigpy (2.0.0) and Zigbee2MQTT / zigbee-herdsman (10.7.0) use the entry's manufacturerName when it's present, so two images with the same image type but different manufacturerName never collide and never flash the wrong device — each one only matches its own. The migration image just sits at the old id next to the others, and the manufacturer name keeps them apart. (This is also why the duplicates mostly don't break anything today — which fits your point that we don't strictly need unique ids.)

[andrei-lazarov]

That's true. I allowed duplicates because it was too much work to explain to everybody.. and make sure they still receive updates after the correction. So I left them as is.

It doesn't hurt to add the check, but it's not really needed.
So I prefer to postpone the migration until we do the big refactor 🙂