Frame durations not handled properly when `isLastVideoFrameComplete` option is not used.

[bp2008]

I discovered another fault in the fragment merging code. This is related to issue #167 where it was discovered that "SPS" and "PPS" and other NAL units could be discarded without being processed as a result of the flawed fragment merging code. It is obvious in hindsight, but the duration and compositionTimeOffset options are also being discarded. That is the subject of this issue.

The issue lies primarily in the feed functions in remuxer/h264.js and remuxer/h265.js.

Consider the case where you feed all your frames with a duration and/or compositionTimeOffset, but without the isLastVideoFrameComplete option. The last NAL unit in each data buffer (possibly an incomplete/partial NAL unit) gets stored in the this.remainingData property to be consumed later. However the duration and compositionTimeOffset options that were provided along with that data are not persisted with it.

In the common case where the last NAL unit in each data buffer is the one containing "VCL" (Video Coding Layer) data, this issue effectively causes every frame to be rendered with the wrong duration and compositionTimeOffset. Further, the very first duration and compositionTimeOffset are completely discarded. If you are monitoring the HTML5 video element's progress (e.g. via its currentTime property), it will always appear to be about 1 frame behind and its playback position may not always line up with expected frame timestamps.

An effective workaround for this is to use the new isLastVideoFrameComplete option (which disables the faulty fragment merging code), but anyone who is unaware of this new option will get video playback that kind of mostly works but has persistent timing errors.

[Leone25]

hm, interesting issue indeed...

Quite unsure what we should do to get around this. Especially because annex-B h264 streams do not contain their length in any reliable way. We could try to parse the begging of the uncomplete frame and based on its type presume what to do:

1. for anything that doesn't actually contain video data (SPS, PPS, SEI, ecc.) and just leave it for later, won't effect timing
2. for anything that does contain video data (basically only NDR and IDR) we leave it in the buffer, but we reduce whatever duration was given to us by one frame length

Does that make any sense? I'm unsure how reliable that would even be...

[bp2008]

I don't know. It is a real mess. To be honest I don't think jmuxer should be concerned with reconstructing NAL fragments. That is more of a transport layer issue. It was a bad decision to put that logic into jmuxer in the first place, and even if it took 4 years for anybody to point that out, it is no less true.

[Leone25]

I agree it's a mess, but also it makes sense why it was done, just to make things work...

We could consider somehow allowing feeding individual nalus with timestamps for those who need it.
In my own project I fetch video through rtsp which does provide it and would be quite easy to forward to the client along with the nalu data.
But still I think some sort of fix could help at least temporarily, then perhaps in the future a major release could be made removing the builtin nalus splitting and only allow single pre-split ones, perhaps with a utility splitter for those who need it.

Honestly as far as I'm concerned that could be done in the next release, but I'm concerned with the repercussions. I don't want to change stuff around too much, that's why I'd love to hear from others first.

I may attempt something tomorrow, but I'm unsure how successful that'll be...