Stupid feature: official Nintendo header/footer support

[vaguerant]

Feel free to close on sight.

I don't know how consistent Nintendo has been with their format for NES ROM headers (e.g. on Wii they used classic iNES headers but later platforms have not), but is it worth supporting any of these, even just to the extent of recognizing that there is one, skipping over it and trying to match the ROM to the database instead?

As an example, I have the eShop game NES Remix on Wii U, which is a collection of minigames challenges based around a NES emulator. The ROMs are relatively easy to access, assuming you have a modified console. Taking a look at them, I can see that they're pretty normal.

Dr. Mario

The first game I tested with was F_DOCTORJ.bin, which header/footer aside is byte-for-byte identical to Dr. Mario (Japan, USA) (Rev 1).nes from No-Intro.

Here is the header for that file:

00 00 00 00 60 00 01 00 30 00 00 00 40 00 01 00
60 00 01 00 00 00 00 00 4B 00 01 00 4B 00 01 00
57 55 50 2D 46 41 41 4A 00 00 00 00 00 00 00 00  // WUP-FAAJ
4E 45 53 00 02 04 12 00 00 00 00 00 00 00 00 00  // NES

WUP-FAAJ is the internal title ID for the Japanese Virtual Console eShop version of Super Mario Bros.--that's not a typo, Dr. Mario's title ID is WUP-FB5J and that is not what's in the header. Dr. Mario was not released on Virtual Console until several months after the release of NES Remix, so it's likely it did not have a final title ID at this point, so they reused an existing one.

At the end of the file is a footer:

3C 00 00 00 00 00 12 00 E8 03 02 09 00 00 00 01
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Super Mario Bros.

I know this is a bit confusing, but the next ROM I looked at was WUP-FAAJ.bin. This is actually Super Mario Bros., unlike Dr. Mario which just uses its title ID in the header. ROM matches Super Mario Bros. (World).nes from No-Intro.

Header:

00 00 00 00 60 A0 00 00 30 00 00 00 40 A0 00 00
60 A0 00 00 00 00 00 00 4B A0 00 00 4B A0 00 00
57 55 50 2D 46 41 41 4A 00 00 00 00 00 00 00 00  // WUP-FAAJ
4E 45 53 00 02 01 01 00 00 00 00 00 00 00 00 00  // NES

Footer:

3C 00 00 00 00 00 01 00 25 00 02 09 00 00 00 01
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The Legend of Zelda

One more, WUP-FBAE.bin. I can go through all of them if you want more data-points to determine the meaning of the header/footer bytes, but for now you're getting three.

Header:

00 00 00 00 60 00 02 00 30 00 00 00 40 00 02 00
60 00 02 00 00 00 00 00 4B 00 02 00 4B 00 02 00
57 55 50 2D 46 41 38 45 00 00 00 00 00 00 00 00  // WUP-FA8E
4E 45 53 00 08 00 12 00 00 00 00 00 00 00 00 00  // NES

Oddly, while the filename is correct (FBAE is the title ID for Zelda on Virtual Console), the header does not match. WUP-FA8E is actually Metroid. ROM matches Legend of Zelda, The (USA).nes in No-Intro.

Footer:

3C 01 00 01 00 00 12 00 9F 00 02 09 00 00 00 01
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

---

Cards on the table, I'm interested in this because the RetroArch port of Nestopia UE on the Wii U supports reading the proprietary Wii U file system, which means it's entirely possible to navigate through the title folders and see a bunch of NES ROMs just sitting there in this annoying format that you can't use. 100% of my enthusiasm for this idea comes from the fact that I think it would just be funny to play your legitimately purchased NES ROMs, unmodified, directly from the media where they are stored, on the console where you bought them. I have no better reason why you should support this.

[carmiker]

This is not out of the question. I like the novelty of the idea.

[vaguerant]

Thanks for noticing this, carmiker. I hadn't realized how much the state of Nestopia development has changed. This is probably out of scope for UE in 2024, clearly more of a JG issue.

[vaguerant]

While I still have no idea what the Wii U-exclusive stuff is for (the first 48 bytes/last 32 bytes), the last part of the header certainly looked superficially similar to an iNES header. In fact, I believe all Nintendo has done is modify one of the magic bytes:

Offset | 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F
-------|------------------------------------------------
 Data  | 4E 45 53 00 02 04 12 00 00 00 00 00 00 00 00 00 
                  ^^ iNES header expects 0x1A here

Manually removing the rest of the header/footer and altering the byte at 0x33 (0x03 after trim) to 0x1A seems to allow any of these ROMs to work as expected in a variety of NES emulators (FCEUX, Mesen and Nestopia UE all tested).

The tool Wii U Virtual Console Extractor, which extracts ROMs from Virtual Console titles, calls this byte BrokenNesHeaderOffset.

[vaguerant]

I missed this previously, but some of the .bin files in NES Remix 1/2 are for Famicom Disk System titles. The format of these is:

Offset	Length	Data
0x0000	0x0030	standard 48-byte Wii U header (like NES ROMs discussed previously)
0x0030	variable	a disk image in Nintendo's proprietary .qd format
EOF-0x2020	0x0010	16-byte footer?
EOF-0x2010	0x2000	modified Famicom Disk System bootROM (disksys.rom)
EOF-0x0010	0x0010	16-byte footer?

The .qd format is something they've been using since at least Animal Crossing on GameCube.

Ice Hockey

Here's the header and footers from F_ICEHOCKEYJ.bin.

Header:

00 00 00 00 50 20 01 00 30 00 00 00 30 00 01 00
50 20 01 00 00 00 00 00 40 20 01 00 40 00 01 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 // no title ID

QD footer:

3C 04 01 01 02 02 F2 F0 E8 03 02 00 00 00 00 00

BootROM footer:

09 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00

Metroid

And likewise for the Japanese version of Metroid, WUP-FA8J.bin.

Header:

00 00 00 00 50 20 02 00 30 00 00 00 30 00 02 00
50 20 02 00 00 00 00 00 40 20 02 00 40 00 02 00
57 55 50 2D 46 41 38 4A 00 00 00 00 00 00 00 00 // WUP-FA8J

QD footer:

3C 04 01 01 02 02 F2 F0 A0 00 02 00 00 00 00 00

BootROM footer:

09 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00

---

You can get a usable .qd image by simply cropping off the header, additional data and footer; unlike NES, no further modifications are necessary. That said, you may wish to convert to .fds format for greater compatibility.

With a lot of help from ChatGPT, here is a Python 3 script which converts from Wii U .bin format to .nes and .qd (and .fds if you also have @einstein95's qd2fds.py).

wup-nes-trim.py

#!/usr/bin/env python3
"""
Converts Wii U NES/FDS Virtual Console .bins to .nes/.qd
Drop qd2fds.py next to wup-nes-trim.py to convert to .fds
https://gist.github.com/einstein95/6545066905680466cdf200c4cc8ca4f0
"""

import os

def convert_rom(rom_in):
    with open(rom_in, 'rb') as fsrc:
        file_content = fsrc.read()

    # Remove Wii U header/footer
    trimmed_content = file_content[0x30:-0x20]

    # Check if this is a NES or FDS ROM
    if trimmed_content[:3] == b'NES':
        trimmed_content = bytearray(trimmed_content)

        # Replace the null byte at offset 0x03 with expected 0x1A
        trimmed_content[3] = 0x1A

        rom_out = os.path.splitext(rom_in)[0] + '.nes'
        with open(rom_out, 'wb') as fdest:
            fdest.write(trimmed_content)

        print(f"Converted NES ROM {rom_in} to {rom_out}")
    else:
        # Remove bootROM
        trimmed_content = trimmed_content[0:-0x2000]

        rom_out = os.path.splitext(rom_in)[0] + '.qd'
        with open(rom_out, 'wb') as fdest:
            fdest.write(trimmed_content)

        print(f"Converted FDS ROM {rom_in} to {rom_out}")

        # If qd2fds.py is present, also convert .qd to .fds
        if os.path.exists("qd2fds.py"):
            os.system(f"python qd2fds.py {rom_out}")
            print(f"Converted FDS ROM {rom_out} to {os.path.splitext(rom_out)[0] + '.fds'}")
        else:
            print(f"qd2fds not found. Skipping .fds conversion.")

for filename in os.listdir():
    if filename.endswith(".bin"):
        convert_rom(filename)

Place this script in the directory with your .bin files and run it (python wup-nes-trim.py) to convert all ROMs in the directory to .nes and .qd. Also include qd2fds.py in the same directory if you wish to convert your .qd files to .fds. New files are created (with their correct extensions); the original .bin files are left unmodified.