Do a better job of detecting compatible game versions for add-ons without `package.json`

[layday]

Probably from reading the TOC file.

[torkus]

I have an open issue right now installing Questie from github. One of the problems I found was that Questie is using a dummy .toc file: https://github.com/Questie/Questie/blob/master/Questie.toc

So I can test for the presence of one or more .toc files to guess supported game versions but I won't really know until I download and parse the contents. Hopefully it's only Questie being weird.

[layday]

For add-ons above a certain size (say, 500 KB), could we do something like, perform an HTTP range request to read the zip directory from the end of the add-on zip followed by another range request to read the toc file?

[torkus]

that's really interesting, I'd never considered peeking inside a partial zip file. I'm not sure what it's specification is like. Is the index kept at the end of the file?

Looking at the wiki page it seems like if you could read the file in reverse from the end of the file until the end of the index you could avoid downloading the whole thing, but if all we have is a HTTP range to work with I'm not sure how you'd know where to start downloading from.

Could be a fun investigation.

[layday]

Without knowing a whole lot about zip files or range requests, this is how I would approach it:

1. Do a head request to grab the content length
2. Do a range request where the range is from (content length - ~1 KB or however large the directory typically is) to content length
   1. Repeat until you find the end of directory marker
3. Locate the toc file(s) in the directory
   1. If the filenames are game version-specific, stop here
4. Extract the offset and length of the main toc file from the zip directory
5. Do a range request where the range is from offset to (offset + length)
6. Drop the local header and parse the remainder of the payload as text

[layday]

I took a stab at it in layday/instawow@2f35ca6 without reading the TOC contents.

[torkus]

Ah, very clever. I took a quick skim. Are you using connection pooling with that web_client I see there? It would make the worse case scenario (a requests for each offset) not so bad.

I discovered early on in strongbox that zip files can be pretty robust as I tried to corrupt the files by cutting chunks out of them to get tests to fail.

[layday]

Yeah, they are pooled. I only tried it with Questie so far which has a directory between 25 and 50 KB in size. We could use this project to gather some metrics. I did also try inserting the toc file at the correct offset in the byte stream but the CRC would fail and Python refused to extract it. Sent with [ProtonMail](https://protonmail.com/) secure email.

…

------- Original Message -------

On Saturday, March 12th, 2022 at 08:40, torkus ***@***.***> wrote: Ah, very clever. I took a quick skim. Are you using connection pooling with that web_client I see there? It would make the worse case scenario (a requests for each offset) not so bad. I discovered early on in strongbox that zip files can be pretty robust as I tried to corrupt the files by cutting chunks out of them to get tests to fail. — Reply to this email directly, [view it on GitHub](#3 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWCLEVP6FD5ERNKG24TU7Q34HANCNFSM5P7MNYMA). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you authored the thread.Message ID: ***@***.***>

[layday]

I figured out what I was doing wrong with the TOC file and have got a working prototype in layday/instawow@ebcd262...1a8b7d8.

[torkus]

Not sure if this is related to the work described here but it falls under the ticket title.

I'm looking at the changes to the addons and in a few cases we've lost game versions.

• https://github.com/SnaBe/BetterZoneStats, 'retail' => nil, should be vanilla
• https://github.com/2CDeez/Buvvs, 'retail' => nil, should be unparseable as interface version is weird and it has no packages
• https://github.com/zerasul/ellidice, 'retail' => nil, should be retail
• https://github.com/Cilraaz/EmoteLDB, 'retail' => nil, should be retail
• https://github.com/dendanskealice/farmlog, 'retail' => nil, should be retail (cata)
• https://github.com/naschorr/Of-The-Statistician, 'retail' => nil, should be retail (legion)
• https://github.com/solinisvc/Whisper-Filter-Continued, 'retail' => nil, should be vanilla

Some of these are obvious outliers

[layday]

I’m not on my computer to check right now but I believe the reason some of these have become nil is that the addon files are not placed inside a folder in the zip. Sent with [ProtonMail](https://protonmail.com/) secure email.

…

------- Original Message -------

On Saturday, March 19th, 2022 at 02:58, torkus ***@***.***> wrote: Not sure if this is related to the work described here but it falls under the ticket title. I'm looking at the changes to the addons and in a few cases we've lost game versions. - https://github.com/SnaBe/BetterZoneStats, 'retail' => nil, should be vanilla - https://github.com/2CDeez/Buvvs, 'retail' => nil, should be unparseable as interface version is weird and it has no packages - https://github.com/zerasul/ellidice, 'retail' => nil, should be retail - https://github.com/Cilraaz/EmoteLDB, 'retail' => nil, should be retail - https://github.com/dendanskealice/farmlog, 'retail' => nil, should be retail (cata) - https://github.com/naschorr/Of-The-Statistician, 'retail' => nil, should be retail (legion) - https://github.com/solinisvc/Whisper-Filter-Continued, 'retail' => nil, should be vanilla Some of these are obvious outliers — Reply to this email directly, [view it on GitHub](#3 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWBY4O3P3RHCK7RAXLLVAURDDANCNFSM5P7MNYMA). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you authored the thread.Message ID: ***@***.***>

[layday]

Okay, I've taken a closer look at each one of these:

• https://github.com/SnaBe/BetterZoneStats
  • folder name different, contains version number: BetterZoneStats-v1.0/BetterZoneStats.toc
    • add-on won't load
• https://github.com/2CDeez/Buvvs
  • invalid Interface version: 113
    • add-on will still load
• https://github.com/zerasul/ellidice
  • different capitalisation: ellidice/ElliDice.toc
    • add-on might load on case-insensitive systems or if the WoW client normalises add-on names, haven't checked
• https://github.com/Cilraaz/EmoteLDB
  • add-on files in zip root
    • add-on won't load
• https://github.com/dendanskealice/farmlog
  • as above
    • add-on won't load
• https://github.com/naschorr/Of-The-Statistician
  • folder name different, contains version number
    • add-on won't load
• https://github.com/solinisvc/Whisper-Filter-Continued
  • folder name different from TOC name: WhisperFilter-Continued/WhisperFilter-Cont.toc
    • add-on won't load

[torkus]

When you say the addon will or won't load, are you actually booting WoW and seeing if it will load? Or does WoW have a list of rules somewhere? I've always found the game to be pretty flexible about what it will load (not an addon author though, I don't really know for sure).

[layday]

I didn't check, but it used to be the folder and TOC names had to match. I could try them all out.

[torkus]

I'm not fussed, I think you're doing a great job. There will always be outliers.

[layday]

Yeah, it still works that way. I just checked with BetterZoneStats, Of-The-Statistician and Whisper-Filter-Continued - none of them work.