Setting subs-with-matching-audio=no does not work

[low-batt]

System and IINA version:

• macOS 13.6.1
• IINA 1.3.3

Expected behavior:
When the mpv option subs-with-matching-audio is set to no in IINA's Advanced settings, playing the attached video English.mp4 will set the subtitle track to <None> and no subtitles will be displayed.

From the mpv manual:

--subs-with-matching-audio=<yes|no>
When autoselecting a subtitle track, select a full/non-forced one even if the selected audio stream matches your preferred subtitle language (default: yes). If this option is set to no, a non-forced subtitle track that matches the audio language will never be autoselected by mpv regardless of the value of --slang or --subs-fallback.

This allows someone who plays videos in various languages to specify that they only want to see subtitles by default if the subtitle language does not match the audio language.

With the three attached files downloaded to a directory, and Subtitles.en.srt.txt renamed to removed the .txt suffix required by GitHub, playing the video English.mp4 with mpv 0.37.0 as shown here:

mpv --sub-auto=all --subs-with-matching-audio=no English.mp4

Will load the subtitles but not select them. No subtitles will be displayed.

Playing the video Japanese.mp4 as shown here:

mpv --sub-auto=all --subs-with-matching-audio=no Japanese.mp4 

Will load, select and display the subtitles automatically.

Actual behavior:
When the mpv option subs-with-matching-audio is set to no in IINA's Advanced settings, playing the attached video English.mp4 IINA loads, selects and displays the English subtitles as if subs-with-matching-audio was set to yes.

Whenever possible IINA should support mpv options. I see no reason why IINA should not support subs-with-matching-audio, so I have entered this as a defect. Once this option is working, a setting should be added to IINA to avoid the need to configure this option in IINA's Advanced settings.

The ability to automatically suppress subtitle section when the language matches the audio language was requested in IINA's Telegram Group, prompting this investigate into the mpv feature.

Steps to reproduce:

• Start IINA
• Click on Settings… under the IINA menu
• The settings panel appears
• On the left side of the panel click on Advanced
• Slide the Enable advanced settings toggle button to be on (blue)
• In the Additional mpv options section click on +
• A new entry appears in the table
• Double click on name and replace it with subs-with-matching-audio
• Double click on value and replace it with no
• Restart IINA to activate the new settings
• Play the attached video English.mp4
• English subtitles are selected and displayed as if the subs-with-matching-audio setting was ignored

Changing IINA's Auto Load setting to Disabled under the Subtitle tab in IINA's settings and adding sub-auto set to all in the Additional mpv options setting under the Advanced tab so that mpv instead of IINA loads the subtitle file also exhibits the same behavior.

English.mp4

Japanese.mp4

Subtitles.en.srt.txt

• MPV does not have this problem.

How often does this happen?
Every time with the provided files.

[low-batt]

I took a very quick look into this expecting IINA was overriding the mpv subtitle track selection. I put breakpoints in the code IINA uses to set the subtitle track and the breakpoints were not hit. This was with mpv configured to load the subtitle file using sub-auto.

This is the relevant section of the mpv.log when playing using IINA:

[   0.189][v][lavf] select track 0
[   0.189][v][lavf] select track 1
[   0.189][v][lavf] select track 0
[   0.189][i][cplayer]  (+) Video --vid=1 (*) (h264 1920x1080 29.970fps)
[   0.189][i][cplayer]  (+) Audio --aid=1 --alang=eng (*) (aac 2ch 48000Hz)
[   0.189][i][cplayer]  (+) Subs  --sid=1 --slang=en 'Subtitles.en.srt' (subrip) (external)

That same section of the log when playing using mpv:

[   0.029][v][lavf] select track 0
[   0.029][v][lavf] select track 1
[   0.029][i][cplayer]  (+) Video --vid=1 (*) (h264 1920x1080 29.970fps)
[   0.029][i][cplayer]  (+) Audio --aid=1 --alang=eng (*) (aac 2ch 48000Hz)
[   0.029][i][cplayer]      Subs  --sid=1 --slang=en 'Subtitles.en.srt' (subrip) (external)

Only two tracks are selected by mpv. If you run mpv like so:

mpv --osd-duration=10000 --pause --sub-auto=all --subs-with-matching-audio=no English.mp4

And then click on Show Tracks under the Playback menu you can see the subtitles have been loaded, but not selected:

At the moment I need to turn my attention back to other issues, so this is as far as I got in my analysis. I'm starting to wonder if support for this option is missing from libmpv. The next step is to confirm IINA is properly enabling this option. If that seems correct then might need to look into the mpv sources to get an idea on what could be going wrong.

[tgt]

I’m the person who asked about this in the Telegram group. Thank you for investigating the issue for me. Something reminded me of this today and I thought I’d see if I could fix the issue. I built the latest source and when the subs are included as a stream in a media file, using --subs-with-matching-audio=no works as expected. There's now a forced option, which does what no did in the version of mpv you used to test this. (See this mpv commit)

Since the functionality works with internal subs, it’d be nice to have a bit of UI in the subtitles preferences. I can have a go at implementing that, if you’d like.

---

The option is still ignored when the subtitles are external. I had a look into why and it seems to be done when autoloading subtitles:

iina/iina/PlayerCore.swift

Lines 1832 to 1834 in 0107e9d

	// set sub to the first one
	try checkTicket(currentTicket)
	setTrack(1, forType: .sub)

Without the above, the last loaded subs will be selected by --sub-add which is called from loadExternalSubFile above. It looks like it should be possible to pass “auto” and the language information determined from the subtitle file, but that doesn’t seem to work. This is presumably not one of the “special situations” mentioned in the docs (or I’m missing something).

sub-add <url> [<flags> [<title> [<lang>]]]
Load the given subtitle file or stream. By default, it is
selected as current subtitle  after loading.
...
<auto>
Don't select the subtitle. (Or in some special
situations, let the default stream selection mechanism
decide.)
...
The lang argument sets the track language, and can also
influence stream selection with flags set to auto.

[low-batt]

Sorry I'm slow to reply. Totally overloaded lately. I usually wait to reply until I have investigated, but at the moment I need to be testing a release candidate for the first beta of the next IINA release. So I've not had a chance to look into the changes mpv has made, but I did want to let you know I noticed your reply. Here is some quick thoughts.

First some background and a side issue…

The code you identified causes another problem. A significant difference between IINA's subtitle loading feature and mpv's is that IINA's loading is done in the background whereas mpv's occurs before the video starts playing. IINA's background loading breaks the mpv Watch Later feature which will remember the selected subtitle track. But when mpv goes to select the track stored in the watch later file the track is not there because IINA has not loaded the subtitles at that time. Because of problems like this I believe IINA really needs to make it easy to switch to the subtitle loading provided by mpv.

At the end of the steps for reproducing there is this showing I also tried switching to mpv's subtitle loading and that did not work:

Changing IINA's Auto Load setting to Disabled under the Subtitle tab in IINA's settings and adding sub-auto set to all in the Additional mpv options setting under the Advanced tab so that mpv instead of IINA loads the subtitle file also exhibits the same behavior.

I remember expecting that would work and that it didn't really confused me. Using mpv to load the subtitles should by pass the code you identified that sets the subtitle track. I guess we need to verify that code is not executed when mpv is used to load the subtitles.

[tgt]

I hadn't realised you were in the midst of a release. Thank you for taking the time to reply.

I've done a quick test of your suggestion (disabling IINA loading and setting sub-auto=all in the Advanced prefs) and it now appears to work as you'd expected by skipping the problematic code and respecting the subs-with-matching-audio option.

I'm not sure how to test subs-with-matching-audio=forced with external subs. I thought that naming the subs with .eng.forced.srt would mark them as forced, but it doesn't and also stops them being detected as English. Maybe that's not the right way to mark external subs as forced. I tend to just embed subs into an mkv rather than dealing with external ones.

I'll use a locally built IINA with the appropriate settings until your upcoming release. Once you're done with the release, please let me know if I can be of use.

[low-batt]

Actually this is the perfect time to bring this up.

I am a junior IINA developer. One of the things I try and do is to help any developer that wants to contribute to IINA. This particular feature initially looked easy to add and a great task for a first contribution to IINA. But not so much to me at the moment. I can't tell how experienced you are from your profile. I'll give you some information on the state of things and you can decide if you want to open this can of worms…

The release being tested is the first beta of the upcoming release. The plan was to put as many of the risky changes into the first beta and follow up with a second beta containing safer changes. Because the first beta contains risky changes we have been trying hard to uncover regressions and fix them before releasing so hopefully the beta is usable enough that users can give it a good testing. The beta was supposed to be released back in November, but I mentioned in the Telegram Group to users who either build IINA themselves or test out the Nightly builds that now would be a good time to test the latest development version of IINA and report problems so we can fix them before releasing the beta. Users responded and reported some important regressions that we judged to be release blockers. Some of those required fixes from the mpv project. The mpv project was super helpful and responded quickly. While this was going on IINA developers were continuing to test, finding more regressions that were missed and fix them. I believe this latest release candidate will pass internal testing and the release will happen as soon as other work (for example release notes) is completed.

While this was going on there has mostly been a hold on merging pull requests except for those fixing regressions or defects in new functionality. Once the beta is out the door and a branch created I'm expecting merging for beta 2 to commence.

I was unaware of this mpv option when you brought it up in the Telegram Group. At the time I thought it was a very useful feature that IINA should support. Because mpv added controls for secondary subtitles for which there are many IINA issues requesting such support, improved support for subtitles is one of the goals of this next release. Thus adding support for this feature very much aligns with the goals of this next release. I initially thought it would be easy to add and ran into trouble. So I documented what you see here and added this issue to the schedule for beta 2.

This release is massively late. I am expecting there will be a strong push to greatly reduce the planed work for beta 2 to complete the beta cycle and get the official release out the door. The beta 2 issues that already have pull requests are more likely to be included and not pushed out to the next release. As subtitles are a primary goal of this release I view this issue as still in play, assuming it can be accomplished without risky code changes.

One of the first steps is to fully understand this mpv feature. I was pleased to see that you have already been exploring what subs-with-matching-audio=forced does. The SubtitleBee blog post What Are Forced Subtitles? A Comprehensive Guide is a really nice high level guide. But I struggled to find articles discussing what identifies a subtitle as forced. This seems to be another case of the software industry having difficulty getting everyone on the same page. I encountered quite a few articles that indicated forced subtitles are always burned into the frame. Definitely not true. From Specification Notes section Matroska specification:

Forced flag

The “forced” flag tells the Matroska Player that it SHOULD display this subtitle track, even if user preferences usually would not call for any subtitles to be displayed alongside the current selected audio track. This can be used to indicate that a track contains translations of onscreen text, or of dialogue spoken in a different language than the track’s primary one.

So for Matroska files the subtitle track is identified as being forced.

On external files… From the Plex support article Adding Local Subtitles to Your Media:

You specify this by adding forced to the subtitle filename:

• Movie_Name (Release Date).[Language_Code].forced.ext

I too would have thought using the extension .eng.forced.srt would have worked. I checked the master version of the mpv manual and did not find any additional information about forced subtitles. It may be necessary to open a discussion with the mpv project. Before doing that I would search both mpv issues and discussions to see if anyone else has already asked about this feature. If you do post an issue or a discussion make it very detailed. Explain exactly what you tried. Include logs. Given them everything needed to reproduce the behavior. Even if you are just opening a discussion it is good to do this. Look at what their issue template asks for and supply all that. The mpv developers prefer information is given up front, rather than they have to prompt you for logs and other info.

How would we add this to IINA? This is about selecting subtitles, so users would expect a setting on the Subtitle tab of IINA's settings. Of the sections currently there it probably should go into the Other section.

The problem is how to make this work with IINA's subtitle loading. Is the mpv implementation of this feature such that it only kicks in once at the point where mpv's subtitle loading would have been completed? Or is it re-evaluated as tracks are added? Experiments are needed to determine this. Might have to dig into the mpv sources and find and understand the code this option activates.

This is the kind of information that needs to be determined before this feature can be designed and coded. Now is a good time to work on this. Depending upon the answers to these questions the implementation might not be too hard. Much more likely this feature would make it into beta 2 if there is a viable pull request waiting to be merged. But that does mean understanding this mpv feature in detail.

I'm tied up with other work. Are you still interested in trying to move this feature forward?

[tgt]

I thought I'd have a look around to see just how much of a “can of worms” it is. It looks like the mpv developers have been battling the complexity of having lots of options.

Forced Subtitles

mpv handles detecting forced flags for mkv, edl, and anything that lavf handles that supports a forced flag on a stream.

mpv doesn't support marking an external subtitle file as forced.

External subtitles are auto-loaded by append_dir_subtitles, which doesn't set a forced flag.

The language detection was broken by my guess at the appropriate filename format (Name.<lang>.forced.ext) because it expects to find the language as the last part of the filename before the extension either after a . or in [] or (). See mp_guees_lang_from_filename

I can't see any issues or discussions about this on the mpv project.

Loading Subs using IINA

It looks like the way for IINA to load external subs, but then respect the subs-with-matching-audio flag is to add the subs using sub-add <file> select <title> <lang> and then call set sid auto. It's necessary to pass the title and language manually as mpv doesn't determine them from the filename when using sub-add as it does when autoloading them. There's no way to specify that the subtitles are forced. For auto-selection to work correctly, it'll need to know these details.

When auto is passed to sub-add in place of select, it'll treat the file as not being external when auto-selection occurs. I think it's probably expected that external subs take priority over internal ones, so select makes more sense to me. Maybe some users would prefer to treat all subs as internal and let selection occur based on other information.

Things That Need Doing

Assuming things behave as expected, which they seem to from some initial testing with a hacked together implementation of the below:

In IINA

Is the idea to replace IINA's track selection with mpv's?

Assuming not:

• Pass information through sub-add as mentioned above
• Add a preference to use mpv auto-select
  • If enabled, IINA will add the subtitles in its prioritisation order, but leave selection up to mpv. This would be requested by calling set sid auto after adding the tracks.
  • If disabled, it'll select the top subtitle chosen by IINA's prioritisation logic. This would be done by calling setTrack(1, forType: .sub) as it currently does.
• Add a preference to set subs-with-matching-audio (in Subtitles->Other?)
  • Should it be disabled or hidden when not using mpv's auto-select?

Otherwise, ignore the disabled path

In mpv

• Add support for external forced tracks
• Add an option to sub-add to specify forced
• Maybe: Get title, language and forced from filename when using sub-add

[low-batt]

Nice research!

Yes, the complexity of supporting lots of configuration options, the trouble with testing many possible combinations of settings and the resulting high potential for defects to be missed by developers is why many proprietary apps restrict configuration to a small number of settings. Of course the concept of open source came about due to the frustration of trying to get companies to support the ability to configure the behavior of their app. A big benefit of mpv is that it is so configurable, but that does put a lot of burden on developers.

I believe IINA needs to put a greater emphasis on being compatible with advanced mpv configuration. There are certain things you should be able to do with mpv configuration, but you can't because IINA steps on your configuration. Some of that is unavoidable, especially when it comes to UI related configuration. However I can point to cases where IINA could notice the user has provided a setting and not overwrite it. There are now a couple of examples where IINA does this, but a lot more work is needed in this area. Other work has had priority.

On mpv not supporting the concept of forced external subtitles…

Based on responses to unrelated issues my impression is that the mpv project is resistant to implementing certain types of behavior that have not been specified in an actual standards document. This has come up when people have asked mpv to support "home grown" "extensions" to standards. Possibly that is why this file naming concept has not already been implemented. Don't know. But that is something to be aware of.

After writing that I decided to look for a reason why mpv will sometimes turn thumbs down when asked to implement "home grown" behavior. This is an example of a forced subtitle from Open Subtitles:

Open Subtitles named the downloaded file Amsterdam (2022).forced.eng.srt. Not the file naming convention seen in the examples I posted earlier. I would not be surprised if the mpv project decided to sit this one out until the software industry gets its act together on this.

For IINA and this issue I'm thinking we should support the forced setting for subs-with-matching-audio and expect users may open issues indicating IINA is defective because it failed to detect an external subtitle as forced. We will treat those as an "upstream" mpv issue.

On this:

Is the idea to replace IINA's track selection with mpv's?

Remember I'm a junior IINA developer, so for trustworthy answers we have to get the attention of the totally overloaded senior developers. We will need to do that at some point, but it is always good to try and work out things and have a lot of information before bouncing stuff off them.

My expectation is that we will face resistance from the senior developers (and users for that matter) to breaking backward compatibility. Without enabling any new settings IINA needs to find and select the same subtitle file as 1.3.5 does.

I'm thinking yes, this means we do need some sort of new IINA setting to inhibit the direct setting of sid that is done now. As that setting relates to IINA's auto loading that needs to be in the Auto Load section. So then the question is whether it should be under the Advanced disclosure triangle or a setting that is always exposed.

One possibily would be we hide it under Advanced and make it be something like:

• Use legacy subtitle selection

With the default being unchecked. In other words break backward compatibility by default. We'd have to run this past the seniors.

On this:

• Add a preference to set subs-with-matching-audio (in Subtitles->Other?)
  • Should it be disabled or hidden when not using mpv's auto-select?

And we have found a worm in the can. In Apples Human Interface Guidelines this would fall under Toggles in the checkboxes section where it discusses subordinate checkboxes. You can see something like this in IINA's settings on the Video/Audio tab where if you uncheck Enable tone mapping the settings under it are disabled.

But we can't do this because subs-with-matching-audio is not directly tied to subtitle loading. If a video has a bunch of embedded subtitles and there aren't any external subtitles then that option will work fine.

We already have Resume last playback position which is broken by IINA's subtitle loading when it comes to restoring the previously selected subtitle. If we go with breaking compatibility and the Use legacy subtitle selection setting we can have a note under that setting something like:

If enabled, the blah blah blah setting may not work.

Need to figure how what we are going to call the subs-with-matching-audio setting…

On the mpv changes…

I expect the main focus of the mpv project is GPU Next. This is the major project to create a next generation renderer. Just one part of it is to switch from OpenGL to Vulkan/MoltenVK/Metal. It is possible to use the new renderer now by setting --vo=gpu-next, but it is still considered experimental. I do see a lot of other work going on, but finishing up GPU Next must be a key goal.

This is still looking doable to me. Need to come up with a simple description of the proposed changes to the Subtitle settings tab with the exact text that would be used. Then I'll see if I can pull in one of the senior developers and see what they think.

[low-batt]

I just noticed the open mpv PR mpv-player/mpv#15854 "tweaks for --subs-with-matching-audio". Possibly the behavior of this option will change in a future version of mpv.

[tgt]

My inclination is that it should go under Advanced since the automatic selection done by mpv should be better in the majority of situations and it feels odd for new users to be exposed to a “legacy” option. Those are the same reasons I like the idea of breaking backwards compatibility by default, but as you said that'd be up to more senior developers.

So under Auto load we can add something like:

    Use legacy subtitle selection: [yes / no]
Or
    Automatic subtitle selection: intelligent / legacy

With some explanation:

    When using only internal subtitles, the audio and subtitle languages will be taken into
    account when selecting subtitles using either system.

    Legacy subtitle selection works best when using only internal or only external
    subtitles. 
Or
    Legacy subtitle selection may not work as expected when using a mixture of internal
    and external subtitles.
Or
    When using a mixture of internal and external subtitles, legacy selection will select
    the first internal subtitle track rather than the prioritised external subtitles.

    Intelligent subtitle selection will take the audio and subtitle languages
    into account even with a mixture of internal and external subtitles.

Quite a lot of that is referencing the Preferred languages list way down in the Other section.

This feels like far too much text to explain what's going on. Maybe only the middle line would be sufficient? Perhaps with a ? button pointing to a detailed explanation on the IINA wiki.

Suggesting it “works best with” might be read as it being better when using only internal or external. (It could be better when using only external if someone has a system set up using prioritisation, etc.)

Do you know how the prioritisation list is used? I wonder how it'll interact with mpv auto-selection. It'll put the user's preferred subtitles at the top of the external subtitles, but something else may be selected depending on language and other mpv settings. Perhaps this needs mentioning in the explanation as well?

---

For the Other section. (Or would a Subtitle Selection section be nicer? It could contain the preferred language list from Other and any future mpv subtitle selection options.)

    Select subs with matching audio language: Always / When forced / Never
Or
    When preferred subs match audio language: Select / Select when forced / Never select
Or
    When preferred subtitle and audio languages match: Select subtitles / Select when forced / Never select

With some explanation:

    Forced subtitles are shown to communicate information not in the primary language.
Or
    Forced subtitles are shown to communicate information not in the primary language 
    such as foreign or alien speech and signs or sign language.

    This setting will be respected when using only internal subtitles or when using
    intelligent track selection.

(Not sure if we need to explain forced subtitles? In the first line I was trying to convey that they aren't just used for things spoken in another language, but potentially for signs and sign language, and made that more explicit in the alternative line.)

---

If automatic selection of subtitles works well and users don't need to change the subtitle track manually, then the correct subtitles should be selected again on resumption, which might reduce how often the “Resume last playback position” misbehaves.

Would it be possible to get the previously used subtitle track from the resume file and switch to it? I haven't looked at the resume stuff. (And don't plan to before getting this implemented.)

---

Regarding mpv, I can make a small pull request that simply adds a forced parameter to sub-add. Then the more contentious work of extracting forced from the filename can be handled by IINA. It might be worth adding sdh as a parameter too in case we want to do something with that in the future. (Plex also supports this as .sdh. or .cc. in the filename.)

[low-batt]

I really like that you provided all these alternatives. Lets see if we can get the attention of other IINA developers and get their input.

@svobs, @uiryuu Thoughts on the user interface for this setting?

A quick overview:

The mvp option subs-with-matching-audio is a very useful feature. However for it to work mpv has to be allowed to select the subtitle to display. IINA's subtitle loading explicitly selects the subtitle with ID 1, breaking the subs-with-matching-audio feature.

The current proposal is to add a new setting under Advanced to be able to enable the "legacy" behavior. By default IINA would allow mpv to select the subtitle to show. If the user enables the legacy behavior it will break the subs-with-matching-audio feature. So we are trying to come up with some sort of note to put under that new setting to inform the user about that. See the choices of text @tgt is proposing.

I like the idea of putting this new setting under Advanced due to the bad side-effect of re-enabling the current behavior of selecting the subtitle with ID 1.

Since we are discussing that, I might as well mention it is buggy and broken. See issue #5399. There are a number of problems reported against IINA's subtitle loading. This is one reason I would really like to see us add a settings UI for enabling mpv's subtitle loading instead of IINA's.

---

On this…

? button pointing to a detailed explanation on the IINA wiki.

I've added a few help buttons to IINA settings. Most of the ones I added were linked to the description of an option in the mpv manual. Neither the mpv manual nor the IINA wiki is ideal as IINA supports lots of languages that this external documentation does not. For the task at hand we can continue this practice, but this is why it is important to have text in IINA so it can be processed by the translators.

On this…

Do you know how the prioritisation list is used?

As issue #5399 shows, sometimes incorrectly. I've not dug deeply into that aspect of the subtitle auto loading code.

On this…

Would it be possible to get the previously used subtitle track from the resume file and switch to it?

Excellent question. When the file is being loaded mpv reads from the file and then deletes it. IINA would have to read the file before it instructs mpv to load the file. The file it is in is named based on the MD5 hash of the full pathname to the file. The file itself is a name=value property list. I believe this is doable. The problem with failing to restore the selected subtitle is issue #4931. You can see an example of finding and displaying a watch later file in that issue.

On this…

Regarding mpv, I can make a small pull request that simply adds a forced parameter to sub-add.

That sounds good. Small clear changes are more likely to be approved and merged. Definitely read their rules for contributing found here, especially regarding their preferences for commit comments.

I usually enter an issue describing the problem and then enter a pull request that addresses the issue. I'm not sure if the mpv project likes that or just a PR would be good enough.

On the wording for the new settings let's see what guidance we get from the other developers.

[tgt]

I've uploaded my tentative changes to https://github.com/tgt/iina/tree/subs-with-matching-audio to make sure they're somewhere safe.

They require the version of mpv available at https://github.com/tgt/mpv/tree/forced-external-subs, which has been added to the mpv v0.41.0 milestone: mpv-player/mpv#16061

v0.41.0 is only required to add support for forced external subs. A small change will make the subs-with-matching-audio code work for internal subs and external subs without forced / sdh tags.