AnvilLauncher Ideas

[Ernegien]

After reviewing the new launcher source code in its current state I have some ideas for improvement, some of which I believe are necessary before it becomes fully usable in my opinion, so I figure I'd get some real discussion going on the topic.

Update Servers

The launcher app settings should be able to reference multiple update servers to be round-robin'ed through to prevent any outages. We don't need (although we likely will) to actually host any of the update servers ourselves (as the content will be signed by us, more on that later) as long as we have access, just the DNS entries pointing to them should we need to point elsewhere at any time.

  <setting name="UpdateServers" serializeAs="Xml">
    <value>
      <ArrayOfString xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <string>https://updates.anvilonline.net/update.json</string>
        <string>https://altupdates.anvilonline.net/update.json</string>
      </ArrayOfString>
    </value>
  </setting>

Update Packages

The update package should consist of a single compressed archive containing a manifest listing original files and their respective checksums along with all required patch content and their checksums.

When being stored on an update server, the update package should be split into file chunks smaller than 512 MB (we'll probably want to size them around 50 to 100 MB) named with the ".bmp" extension (there are others to choose from but it seems fitting enough) so they remain fully eligible for static caching using CloudFlare's free plan. They can be reassembled and extracted client-side.

Packages should be signed by multiple team members (the more the better, but we'd probably want at least 3) before being released to promote accountability and prevent maliciousness. This will require all members to generate their own public/private key pairs. We will then host all of the public keys on a separate auth server controlled solely by us to be used by the launcher to validate content signatures.

Launcher User Experience

Launcher users are assumed to have already obtained base Halo Online content, whether that be in the form of the original leaked alpha, ms30, or some custom community-assembled package. If the launcher is not run from within the same directory as the game files, it should prompt the user to select the path manually.

Connectivity to update servers should not be required in order to use the launcher. The launcher should allow for the ability to specify an update package to load as an alternative to downloading updates from a server. This also allows for offline distribution in case of update server issues.

Users should be able to select from a list of updates, and not just be forced into using the latest. This will allow for people to quickly revert back to the last stable build in case our testing misses something and we end up pushing a broken release.

Valid signatures should not be required in order to use the launcher, but we need to at least issue a warning notifying users that the content may be not be trusted and let them decide if it's worth the risk.

Users shouldn't have to wait for full file content validation each time before being able to launch their game. They should be provided with the option to disable it entirely (with some warnings) or we cache validation output in a secure way allowing it to be skipped during subsequent attempts to run the same build.

Any patched file content should be backed up within an "orig" sub-directory. This will allow us to revert to the original and reapply updates as needed.

Example JSON

Update server's update.json

"packages": {
    "0.3.1": {
        "signatures": {
            "shockfire": "6f7df91d8f973a0619d525c319337741130b77b21f9667dc7d1d74853b644cbe...",
            "kiwidog": "6f7df91d8f973a0619d525c319337741130b77b21f9667dc7d1d74853b644cbe..."
        },
        "parts": {
            "/static/release/0.3.1/part1.bmp",
            "/static/release/0.3.1/part2.bmp",
            "/static/release/0.3.1/part3.bmp"
        }
    },
    "0.1.1.1245": {
        "signatures": {
            "shockfire": "6f7df91d8f973a0619d525c319337741130b77b21f9667dc7d1d74853b644cbe..."
        },
        "parts": {
            "/static/release/0.1.1.1245/part1.bmp"
        }
    }
}

Auth server's keychain.json

"users": {
    "shockfire": {
        "keys": {
            "Latest": {
                "timestamp": 7598327592385,
                "key": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAg7V3m3V+xadQEVVHYE83fkNQEXQTJm7atq9HBIePcR7KepumE1bh+MvYs+mREB3Q4P68mx/6xPqGdSeG1NWqnpSv51SFNfgAZhfoQFJkLJi8ziVeGmY1zrHkd6PXsKZpzlxobid1KuGDDGmNHWPKuFrXwUkqEz90LfemHsE9TWAEE3g9Ntoj8TpCCQFBwUrg1sTYManq0lb1QWqUedxB2pyCpFkka9B9qn+TUlmMNpkvhWLXYDE598GFz00/Xj3pIXVFZpUYUG6+35sd679uWKTj3JoHpDk1l5y5FfRfgrH+cdv4nP6pNRF98L83TZMlUrbw54+4BiC6e9vfQMSoBQIDAQAB"
            },
            "Old Compromised on 20151225": {
                "timestamp": 2349587293472,
                "key": "MIIBITANBgkqhkiG9w0BAQEFAAOCAQ4AMIIBCQKCAQBk08+ojtfLGy0deWEB0EccWtcbNwlL31iIxru+pqyudfxV/ThythKwZO/0/9xLI+uaofDQuo0hlKxO+6zILT0RS/8VnJp/Ln62zXgGBpdDDbvbMGu5PKIXrFW+cyCAqJOqJFINS/rwjOj5VTPWsOTKpM1q9G9HYSUW2h6RINuHqvGiiC25GW7TENUtcLdNYSdnk6WIhFvEgzNY+vtlEhXuFKTtLjS1BVA9GCeZIOx85PfsX+5oGXU75I4UGqN324U44B+T1TKo9ky3qmTTJ1/lRKkjhunT1g6RzpSxg/He/qEVqfWxRRDDVFABREONQk+q2kdVzkN5APHxZediXzStAgMBAAE="
            }
        }
    }
}

Random Thoughts

Perhaps we can have different tiers of public updates available through the launcher, where people can participate in a public beta or use latest builds for example.

How are we planning on handling mod distribution? I'm assuming this will be separate from the launcher.

For generating patches, bsdiff is great for small files, but chokes on large ones due to memory requirements usually being 17x the original file size. I would propose we keep using bsdiff for patching files smaller than 30 MB, and switch to something else like xdelta for larger files. It wouldn't hurt to just use an alternate for everything however.

[kiwidoggie]

Sounds good, The only thing I would have to say about the parts for the launcher is, what happens if someone for example, changes one file and doesn't need to re-download everything, just that one file?

You can disable connectivity to servers to update already, it is in the configuration file.

Also for the signatures, it shouldn't really be needed if we are planning on setting up a CI server. The currently branch structure on git does not permit pushes to master (which will get compiled, and deployed via a CI if that's the way we want to do it) The "verification" requirement can probably be added to GitHub itself so everyone has to sign off on something before it gets pushed to master.

Mod distribution should not be handled by the launcher imo, that should have a completely different subsystem all together. (I was thinking ingame content delivery). I don't think we should use the diff method any longer as its inefficient in a lot of cases. I figure mods, and content patches (maps, tags etc) should have a custom patching format that is designed to work with the data we have. Not plain binary data. What I mean by that is, we could make a patching system that is designed to work with tags, and various other content directly instead of trying to work around the issue of diffing final binaries.

aka Diff this tag, at these offsets. And no matter how the map file is built, it will go and find "that" tag. (Its more dynamic I feel this way)

Otherwise looks good, any thoughts?

[Ernegien]

Yeah, I suppose content probably shouldn't be packaged as aggressively as I had originally mentioned. Individual files will still need to be stored on the update servers as ".bmp" and be broken up if larger than 512 MB (not something we'll encounter often, if ever) to stay under the CloudFlare caching radar.

If we can use CI to automate the signing of some things that would be great. I'd imagine the build process would have a project-wide default key to use, then members of the team would also have their own key to add their personal stamp of approval manually before release if they'd like. The launcher will still need to know where to look up the public keys for verification. Whether that's hosted on github or a separate auth server that we control makes little difference. For simplicity's sake, we might as well put what we can on github as it can be tracked more easily, and it's less custom stuff we have to worry about on our end.

A new diffing strategy custom-tailored towards tag cache data will be pretty much required for effective mod distribution so we might as well use it for our base tag content patches as well. Representing edits via an intermediate language that we design would also allow us to check in and keep track of tag changes via source control so people can work concurrently on different things without having to pass the entire tags file back and forth eliminating some margin of error and wasted time. emoose's mod system for the recode should serve as a good reference point for discussing how we'd like the new one to be laid out.

[StrikerTheHedgefox]

I'd highly suggest against using ".bmp" as the extension, as people will think they are bitmaps and try to mess with them.

[Ernegien]

People that mess with things they shouldn't either deserve to run into problems or already know what they're doing :P I see where you're coming from, but these files won't be user-facing in any way. They'll be automatically pulled off of back-end content servers by the launcher on an as-needed basis and then assembled client-side.

[StrikerTheHedgefox]

Alright, sounds good.

[wub]

What are the downsides to using a torrent-based system for updates (like Blizzard does/used to)?