Any tips for implementing playlist type functionality?

[jagthedrummer]

I've implemented a rough play queue and it mostly works, but there are a couple of things that I'm not sure how to address and I'm wondering if anybody has done something similar.

I basically have an array representing sounds that should be played next. When an identifier is added to the queue I pre-emptively call stereo.load(newIdentifier) so that the sound will be ready when it comes time to play it. And the sound that is returned is added to the array. (If we're not already playing something when something is added to the queue it's played immediately.)

My queue service registers a listener for the audio-ended event, and when it fires it checks to see if there's anything in the queue waiting to be played. If so it pulls the first item in the array (which is a sound) and then calls sound.play().

This mostly works, but there are two issues that I'm hoping to solve.

1. Even when this is working at it's best there is a slight but perceptible delay between the end of one track and the beginning of the next track.
2. Sometimes, and especially often in mobile Safari, when I call sound.play() on a pre-loaded sound it seems to re-load the sound.

Are there any best practices around this sort of thing?

Thanks in advance! (I realize this may be uncharted territory, so no worries if I'm on my own.)

[jkeen]

hey @jagthedrummer! This timing is great, because making an audio queue/autoplay thing is on my short list of things to do in the next couple of months and I was kinda wondering if the queue functionality should be built into ember-stereo directly. 🤔 It's been a while since I've had my head in this space, but I remember tangling with this problem long ago (2017ish) and tying into the audio-ended event like you did is the way I got it working before.

1. From what I know gapless playback is extremely tricky to achieve with webaudio. Having the sound all loaded up and making sure each sound is using a separate audio element (by making sure alwaysUseSingleAudioElement is not set to true) is the way to get least gap possible. I stumbled on this thread a few weeks back that I thought was an interesting rundown of the options (with slightly more options than one might have if just having a web app)

2. Not sure about that one–I think in mobile safari it might start sharing a single audio element behind the scenes instead of having separate elements for each sound? It might be a bug?

[jkeen]

An extremely clever/hacky way to achieve gapless playback might be to create an M3U8 playlist and feed it into the HLS connection. I built a whole playback system that records an audio stream, breaks it up into 10 second .aac file chunks, and with the HLS connection it plays it back seamlessly, loading those chunks dynamically.

It might be hacky as hell as it stands now, but I wonder if somewhere in that mess with an abstraction layer on top of it is the path to gapless playback on the web?

[jagthedrummer]

Thanks for the tips, @jkeen! I'll give these a shot and report back.

[jagthedrummer]

It seems like my electron based desktop app is defaulting to alwaysUseSingleAudioElement: false where mobile Safari defaults to true. If I set it true in my config then the desktop app shows the same reloading behavior as mobile Safari. Unfortunately explicitly setting it to false in the config doesn't seem to affect mobile Safari at all.

The M3U8 playlist idea sounds interesting. I don't really know anything about them, so I'm not sure exactly what you're suggesting. To "feed it into the HLS connection" is that some process other than stereo.load or stereo.play? (I see the HLS class mentioned in the docs, but there are no methods described there.) Could I build the playlist as a string from inside the app and somehow use that directly, or would I need to generate it server side so that there's an actual URL involved?

Thanks again for the help!

[jkeen]

It seems like my electron based desktop app is defaulting to alwaysUseSingleAudioElement: false where mobile Safari defaults to true. If I set it true in my config then the desktop app shows the same reloading behavior as mobile Safari. Unfortunately explicitly setting it to false in the config doesn't seem to affect mobile Safari at all.

Yeah I think there might be logic in there that needs to be re-evaluated with handling Mobile safari. Some of that logic is ancient 2016 solutions to 2016 browser problems and things may have changed.

To "feed it into the HLS connection" is that some process other than stereo.load or stereo.play?

Nothing fancy, you're just providing the path of the .m3u8 file to stereo and HLS will pick it up.

It might be more of a thought exercise than a real solution, but the fact that HLS can string together hundreds of file segments and play them back without a gap sounded like it might be able to do the same thing if those segments were instead tracks that you didn't want a gap between. I'm not 100% sure of your usecase so this might be a ridiculous solution, but for me, my m3u8 file looks kinda like this

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:10 <-- just letting it know to expect all the segments to be around 10s
#EXT-X-START:TIME-OFFSET=0 
#EXTINF:9.899,   <--- the exact duration of the segment
https://path/to/segment.aac  <--- the path to the 9.899s segment
#EXTINF:10.027,
https://path/to/segment.aac  <--- the path to the 10.027s segment

I think it would be worth a try to create a playlist.m3u8 file that looks like example 6 on the m3u wikipedia page: https://en.wikipedia.org/wiki/M3U

#EXTM3U
#EXTINF:419,Alice in Chains - Rotten Apple
Alice in Chains_Jar of Flies_01_Rotten Apple.mp3
#EXTINF:260,Alice in Chains - Nutshell
Alice in Chains_Jar of Flies_02_Nutshell.mp3
#EXTINF:255,Alice in Chains - I Stay Away
Alice in Chains_Jar of Flies_03_I Stay Away.mp3
#EXTINF:256,Alice in Chains - No Excuses
Alice in Chains_Jar of Flies_04_No Excuses.mp3
#EXTINF:157,Alice in Chains - Whale And Wasp
Alice in Chains_Jar of Flies_05_Whale And Wasp.mp3
#EXTINF:263,Alice in Chains - Don't Follow
Alice in Chains_Jar of Flies_06_Don't Follow.mp3
#EXTINF:245,Alice in Chains - Swing On This
Alice in Chains_Jar of Flies_07_Swing On This.mp3

with the paths/durations/names of your tracks where appropriate, and then give the URL of the playlist.m3u8 to stereo and see what happens?

  var results = yield this.stereo.loadTask.perform(playlistUrl, {
    useConnections: ['HLS', 'NativeAudio'],
    metadata: {},
    xhr: {
      manifestOnly: true, // just send auth headers when requesting the playlist.m3u8, not each segment. not sure how your auth setup is, you might not need this XHR block at all
      headers: this.session.authenticatedHeaders,
    },
  });

I think as it stands now with how stereo + HLS interact stereo won't see the individual tracks, but I think the connection could be modified/improved in order to do so if it does indeed solve the gap problem

[jagthedrummer]

OK, just to document what I've tried and found.

I'm generating a playlist on the server that the app can point at. I discovered that Ember Stereo (or something farther down the stack) doesn't like a playlist URL in the form of /playlist/1/m3u8. It complains about an unknown MIME type, and explicitly returning a content type from the server didn't seem to make any difference. I finally found that giving it the URL of /playlist/1/m3u8.m3u8 would get around that problem and it stopped complaining. I guess the trick is to make sure that you have a full proper playlist filename with an extension of .m3u8.

Once I got that working I ran into a problem where I'd do this:

let m3u8Url = `${ENV.api.apihost}/jsonapi/v1/playlists/${this.args.playlist.id}/m3u8.m3u8`
this.stereo.play(m3u8Url, {
  useConnections: ['HLS', 'NativeAudio'],
  metadata: {},
});

And I'd get an error saying:

Uncaught Error: levelParsingError
    at Stereo._handleLoadError (stereo.js:753:13)
    at Stereo.loadTask (stereo.js:453:21)
    at loadTask.next (<anonymous>)
    at GeneratorState.step (generator-state.js:20:35)
    at TaskInstanceExecutor.generatorStep (executor.js:240:42)
    at TaskInstanceExecutor.handleResolvedContinueValue (executor.js:120:27)
    at TaskInstanceExecutor.proceedSync (executor.js:84:12)
    at executor.js:74:31
    at invoke (backburner.js:282:1)
    at Queue.flush (backburner.js:197:1)
    at DeferredActionQueues.flush (backburner.js:360:1)
    at Backburner._end (backburner.js:801:1)
    at Backburner._boundAutorunEnd (backburner.js:525:1)

In the network tab I could see that the playlist was being requested and returned successfully. Oddly there would be an immediate second request to the same URL which would be in the "cancelled" state immediately.

According to hls-lint the #EXT-X-TARGETDURATION line is required, so I added that to my playlist (even though I won't really have consistent target durations).

After doing that I'm getting this error:

stereo.js:718 Uncaught Error: fragParsingError
    at Stereo._handlePlaybackError (stereo.js:718:13)
    at Stereo.playTask (stereo.js:564:21)
    at playTask.next (<anonymous>)
    at GeneratorState.step (generator-state.js:20:35)
    at TaskInstanceExecutor.generatorStep (executor.js:240:42)
    at TaskInstanceExecutor.handleResolvedContinueValue (executor.js:120:27)
    at TaskInstanceExecutor.proceedSync (executor.js:84:12)
    at executor.js:74:31
    at invoke (backburner.js:282:1)
    at Queue.flush (backburner.js:197:1)
    at DeferredActionQueues.flush (backburner.js:360:1)
    at Backburner._end (backburner.js:801:1)
    at Backburner._boundAutorunEnd (backburner.js:525:1)

In the network tab I'm seeing two requests for the playlist file, both of which are successful, and then successful requests for each item in the playlist, with several in a row for the last item. And then the whole set of requests repeats again. Then a short pause, and another repeat of all the requests.

The URL for the playlist file looks like this:

https://seshy-api-octolabs.ngrok.io/jsonapi/v1/playlists/11/m3u8.m3u8?Authorization=Bearer%20XXX

And the playlist coming off of the server looks like this:

#EXTM3U
#EXT-X-TARGETDURATION:8
#EXTINF:8,Artis - Song
https://seshy-dev.s3.amazonaws.com/project-478/bounces/237/short1.m4a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx...
#EXTINF:8,Artis - Song
https://seshy-dev.s3.amazonaws.com/project-480/bounces/240/short2.m4a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx...
#EXTINF:8,Artis - Song
https://seshy-dev.s3.amazonaws.com/project-476/bounces/239/short3.m4a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx...

Do you spot anything obviously wrong with that I'm trying to do? And/or do you have any suggestions on what I should try next?

[jkeen]

The mime type thing is annoying, you can either supply a mimeType to the play, or load calls in the options hash, but the easiest IMO is to do what you ended up doing and keep the .m3u8 in the URL

You might try playing that in VLC or https://hlsjs.video-dev.org/demo/ to see if it has a more descriptive error?

[jagthedrummer]

OK, after spending way too much time trying to tweak the format of my playlist file I kind of randomly thought "what if something doesn't like the media format of the files I'm pointing to?" So I tried it with .mp3 files instead of .m4a files and that at least got the playlist to load and play. (I'm kind of surprised because the docs I'm finding make it sound like .m4a is the preferred format for .m3u8 audio playlists...)

Anyway, that does get the playlist to play, and it's almost gapless but not quite. (Which also seems surprising.) There is certainly much less of a gap than with the audio-ended method.

Just to document some observations in case it's useful and/or will help anybody else.

Even after switching to .mp3 files in the playlist the EXT-X-TARGETDURATION still seems to be required. Removing it triggers the levelParsingError mentioned above. I can set it to a very large number and playback seems to work as well as if I set it to the value of the longest file in the playlist.

Once the playlist starts playing it seems that stereo.currentSound and {{ current-sound }} always represent the playlist itself and they don't change as files within the playlist are played.

stereo.currentMetadata and {{ sound-metadata }} don't seem to ever be set when playing a playlist, even when I explicitly pass a metadata hash to stereo.play. They seem to always be blank. (Or I'm somehow passing the metadata incorrectly...) I think ideally I'd want the currentMetadata to be pulled from the #EXTINF lines in the playlist so that it could update track name and what not as the playlist progresses, so I'm not sure it's all that important that passed in metadata be used.

I think for the moment I have more pressing problems in my app than gapless playback, so I'll probably put this effort on hold for a bit while I work on those.

Thanks again for the help, @jkeen! And for the excellent add-on!