SLPlugin - libcef.dll - Common crash in plugin infra probably during quit/TP/region crossings.

[monty-linden]

Environment

Second Life Release 7.1.2.7215179142 (64bit)
Release Notes

You are at 159.9, 77.6, 47.4 in LindenWorld B located at simhost-019b8bea6b2cc58b3.agni
SLURL: http://maps.secondlife.com/secondlife/LindenWorld%20B/160/78/47
(global coordinates 237,984.0, 244,302.0, 47.4)
Second Life Server 2024-02-21.7995320426
Release Notes

CPU: 12th Gen Intel(R) Core(TM) i9-12900HK (2918.4 MHz)
Memory: 65208 MB
OS Version: Microsoft Windows 10/11 64-bit (Build 22000.2538)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: NVIDIA GeForce RTX 3050 Ti Laptop GPU/PCIe/SSE2

Windows Graphics Driver Version: 31.0.15.4609
OpenGL Version: 4.6.0 NVIDIA 546.09

Window size: 1506x1020
Font Size Adjustment: 96pt
UI Scaling: 1
Draw distance: 128m
Bandwidth: 3000kbit/s
LOD factor: 1.5
Render quality: 4
Advanced Lighting Model: Enabled
Texture memory: 4095MB
Disk cache: Max size 3993.6 MB (23.1% used)

J2C Decoder Version: KDU v7.10.4
Audio Driver Version: FMOD Studio 2.02.13
Dullahan: 1.14.0.202310131404
CEF: 118.4.1+g3dd6078+chromium-118.0.5993.54
Chromium: 118.0.5993.54
LibVLC Version: 3.0.16
Voice Server Version: Vivox 4.10.0000.32327.5fc3fe7c.399bd0e

Packets Lost: 16/12,357 (0.1%)
March 18 2024 14:50:55

Description

While diagnosing an unrelated problem in an SL and Firestorm environment, I noticed a bunch of crash reports on SLPlugin.exe coming out of dxdiag. I wondered if these were common and I was immediately able to reproduce this on my dev system. The crash gets reported via standard Windows Event Viewer and there will be a pair of events (Event IDs 1000/1001):

Faulting application name: SLPlugin.exe, version: 0.0.0.0, time stamp: 0x657b81b8
Faulting module name: libcef.dll_unloaded, version: 118.4.1.0, time stamp: 0x651cb3d9
Exception code: 0xc0000005
Fault offset: 0x00000000046433f0
Faulting process id: 0x1d6c
Faulting application start time: 0x01da797d35a55042
Faulting application path: C:\Program Files\SecondLifeViewer\SLPlugin.exe
Faulting module path: libcef.dll
Report Id: 86eda01a-dcc8-4278-8595-f3eeafee2090
Faulting package full name: 
Faulting package-relative application ID: 

The faulting offset is consistent over runs so at least this isn't a Heisenbug. Both SL and FS show identical offsets for the same libcef build so I suspect this is a DLL unmap bug possibly in our use of CEF, possibly elsewhere. But it will be reproducible. [sp?]

I don't expect this to be critical but one question is: are we deliberately ignoring this in bugsplat?

Reproduction steps

1. Windows SL session
2. Go to slcorps meeting area.
3. TP to Boston platform: http://maps.secondlife.com/secondlife/LindenWorld%20B/160/78/47
4. Walk around
5. TP back to slcorps meeting area.
6. You should have some crashes in Event Viewer

[vldevel]

TP to Boston platform: http://maps.secondlife.com/secondlife/LindenWorld%20B/160/78/47

"Sorry, you do not have access to this teleport destination"... Linden-restricted, I guess ?

I would have tried with the Linux Cool VL Viewer which is using CEF v122, since I guess it is just one of these many CEF bugs that have likely already been fixed in newer versions.

[monty-linden]

Yes, those will be Linden-only regions. But other pairs of regions with MoaP on them should be able to reproduce the problem. The crash seems very common...

[vldevel]

Yes, those will be Linden-only regions. But other pairs of regions with MoaP on them should be able to reproduce the problem. The crash seems very common...

If you can find another "open" place where you crash, let us know... I'll pay it a visit to try and see if it is CEF-version related or something else.

[monty-linden]

Okay, easy round:

• Login to Morris: http://maps.secondlife.com/secondlife/Morris/128/128/33
• TP to Hippotropolis: http://maps.secondlife.com/secondlife/Hippotropolis/241/21/25

On arrival to Hippo, crash should show up in Event Viewer under Event Viewer > Windows Logs > Application.

[vldevel]

OK, I could reproduce the crash with LL's current GLTF maint2 viewer (7.1.4.something), but only after login + first TP. Afterwards, there is no crash happening.

Also, I could not reproduce the crash with the Windows build of the Cool VL Viewer, even though it uses the same Dullahan/CEF pre-built library as LL's. While there are changes in my media_plugin_cef.cpp module when compared to LL's, I do not expect those to make any difference in this particular issue.

However, there is another, more prominent difference between LL's viewer and mine: the former is launching a "spare media browser" instance on viewer start (mSpareBrowserMediaSource in llviewermedia.cpp), on the pretext it would speed up the subsequent instances... Since I did not see any such "improvement" in all the testing I made, and find it just a waste of memory and OS scheduler cycles to keep such an additional instance (plus all the dullahan_host's child processes it launches), I do not use one in my viewer.

Since the crash only happens once after login and not any more afterwards, I won't be surprised if the crash was coming from this "spare media browser", and then the fix would be pretty easy, by just getting rid of it ! :-P

[beqjanus]

I have been informed of a far simpler repro. Noted in Canny on https://feedback.secondlife.com/bug-reports/p/frequent-slpluginexe-has-stopped-working-errors

rez a prim
close any edit window
right click edit.
refresh windows event viewer (F5)
observe the lovely crash report.

https://gyazo.com/dcb5d80822dc2013de1f1963b976d981

ETA: I am going to try Henri's suggestion of nixing the spare.
ETA: (a few minutes later) createSpareBorwserMediaSource() has not been used since 2017 in either LL or FS.

[beqjanus]

I removed all references and dead code related to mSpareBrowserMediaSource "just in case". It makes no difference. The above repro still works consistently.
One thing that appears to be true is that the crash is triggered by closing the edit window, not the act of editing.

[beqjanus]

OK...so...
Edit prim, select single face, texture tab, media, browse and set homepage to https://www.secondlife.com
close edit
No crash

Edit prim, select the media face, text tabe, media, remove.
close edit
Crash.

This suggests that CEF is being asked to release a media resource it does not have and it trips over its own feet in doing so.

[monty-linden]

(Thanks for additions to the issue... I've been trying to link up various old sources of info including BUG-234655: secondlife/jira-archive#11546 But the reporting tools are giving me grief this weekend.)