Loading audio/image resource from compressed file

[Catizard]

Currently this pr's implementation only allow packaging audio files into the archive file. Ideally it should allow packaging image files too.

Motivation

This is a draft of allowing user package the audio/image files into a compressed file for each bms package. The original idea was proposed in upstream. As the statement points out, by implementing this user can much disk spaces and probably benefits the bms package distributors.

I have tried implementing this idea before but was stuck at how to cooperate with gdx & handling compressed file errors. At current stage I have made this feature able to use but extremely slow. See below for more details.

How it shapes like?

Take an arbitrary bms package on my disk for example, before it looks like:

This pr allows it package all the ogg files into a single archive:

Implementation details

The main structure of the implementation is very straightforward: previously, beatoraja encapsulated the actual audio load step in AudioCache::get function, where AudioCache itself is a resource pool. Therefore, in this implementation, I extended it to firstly check if the audio file currently it wants to load is present in the resource.7z archive file, if so, load it from archive file, otherwise, loading it from disk.

However, there're two major issues about the actual file load process:

• 7z file is not allow random access
• 7z file doesn't allow loading parallelly. Race conditions are fatal.

The first problem causes the code have to load the archive file on disk each time loading an audio file. The second problem directly kills th mutli-threading strategy that has already been coded in raja: makes me to lock it exclusively without choice. These two problems may can be solved together by forcing user to use zip format which may support random entry access.

Therefore, in current implementation, a 7z file is eagerly loaded into memory rather than accessing entries when needed.

Compatibility

This is the most concerning part of this pr. While this feature is good, it saves disk space, it makes packaging more simpler. However, this is an one-way street if upstream doesn't have this feature. Imaging a user want to play on vanilla raja temporarily? No it cannot be done¯_(°ペ)_/¯

Other messages

This implementation forces the archive file's name to be resource.7z, this is on purpose. Loading arbitrary compress files are just not a good idea.

[Catizard]

Update: I fixed the performance issue by eagerly loading everything from archive file

[seraxis]

I think it would be fine to impose a limitation of only using .zip but since it appears you have already solved the issue it's not important

Is there a reason we've gone with 7z here? I do like this idea however, it will make distribution much easier in the future. So much file space is lost just on filesystem overhead with the amount of files

[Catizard]

I think it would be fine to impose a limitation of only using .zip but since it appears you have already solved the issue it's not important

Personally I think supporting .zip isn't hard, but it only has one benefit than .7z: random access so it doesn't need to be loaded eagerly. However, forcing the future users only use one specific compress format could reduce our work load imo.

Is there a reason we've gone with 7z here? I do like this idea however, it will make distribution much easier in the future. So much file space is lost just on filesystem overhead with the amount of files

There's no reason of choosing .7z actually, I started with it for prototyping :)