Port to Modern OpenGL

[complexlogic]

I've been working on porting the game to modern OpenGL, which uses shaders instead of the fixed function pipeline. The port currently targets OpenGL 2.0 and OpenGL ES 2.0. It should be compatible with most PCs made in the last ~20 years.

This branch establishes a new abstract base class TRenderer for doing drawing, with sub classes implemented by each rendering backend (currently only OpenGL and OpenGL ES). This is similar to how other systems work in USDX, such as the audio system.

I have two motivations for doing this. The first is that it significantly improves the code quality of USDX. All of the OpenGL code is now contained within a single unit. No more having 100+ different pieces of code to draw a texture or quad scattered throughout various places in the codebase.

Now, this becomes Renderer.DrawTexture, Renderer.DrawQuad, Renderer.DrawLine, etc. This is a nice abstraction, so you don't have be an OpenGL expert to work on the graphics anymore. The low level graphics code is hidden and contained.

The second reason for doing this is that it improves the portability of the program. The pluggable rendering interface provides the ability to add other rendering backends in the future if we need to. OpenGL is a legacy API, and is officially deprecated on Apple platforms. For example, if Apple eventually drops OpenGL, we could keep macOS support by adding a Metal backend.

This would also allow us to upgrade ProjectM to the latest 4.x release, instead of being stuck on unmaintained 2.x. With 4.x, the ProjectM library now provides a C API, so we wouldn't need the C Wrapper library anymore. That would greatly simplify the build system.

Summary of Code Changes

Here's a high level summary of the code changes:

• TRenderer class introduced for drawing, with subclasses for each rendering backend
  • Provides an API to draw textures, text, and basic shapes needed by the game
  • TTextureUnit class functionality (LoadTexture, GetTexture, etc.) are now part of TRenderer
• Desktop OpenGL and OpenGL ES are both supported:
  • The game will default to OpenGL ES on ARM-based Linux systems, and desktop OpenGL on all other platforms
  • Added command line option -Renderer to request a specific rendering backend (-Renderer gl or -Renderer gles).
• TTexture is now a class instead of a record. This was needed for polymorphism (ability to support multiple rendering backends)
• Reflections are now built in to the TTexture class, instead of being implemented separately in each of the graphics classes (TMenuButton, TStatic, text, etc).
• A few things have been renamed to abstract from OpenGL
  • TextGL unit is now UText
  • glPrint function is now PrintText
  • glTextWidth function is now TextWidth
• ProjectM related changes:
  • Minimum version updated to 4.0
  • C wrapper library was removed; ProjectM provides its own C API since version 4.0.
  • Configuration option --with-local-projectM-presets was removed. The ProjectM library no longer ships presets, so we always use the provided presets in game/visuals/projectM. The config.inp still allows users to point to a custom preset directory if they wish
  • Linux AppImage OS upgraded from Debian 8 Jessie (2015) to Debian 9 Stretch (2017). The reason is because ProjectM 4.x requires C++14 and can't be compiled with GCC 4.
• The Lua party mode plugin modules Usdx.Gl is now Usdx.Renderer, and Usdx.TextGL is now Usdx.Text. All of provided party mode plugins in this repo have been ported to use the new modules and will continue to work as they did before, but any out-of-tree party mode plugins that use Usdx.Gl and Usdx.TextGL will be broken.
• Some things that were in the code but were completely unused were removed:
  • Bitmap fonts (we only use TrueType)
  • Underline font style
  • Entire UMenuStaticList unit
  • TMenu.SetAnimationProgress function and its overrides in subclasses
  • Pixel Buffer Object (PBO) code for video playback

To Do

• OpenGL ES support
• Particle effects (for golden notes, etc.)
• Oscilloscope
• ProjectM visualizer
• Lua plugin interface (a few plugins do custom drawing with OpenGL 1.x)
• General cleanup, add comments, etc.

[s09bQ5]

Please don't sacrifice support for old OpenGL.
So far no platforms/drivers have dropped support for OpenGL 1.x/2.x.
For OpenGL ES-only systems, GL4ES can be used. Mesa can do OpenGL on top of Vulkan and Direct3D. There are even people trying Zink on top of MoltenVK. Stacking GL4ES on top of ANGLE might be another option to make it run on other APIs.

I have no objections if you move the current OpenGL 1.x/2.x rendering code into your abstracted interface and add OpenGL 3.x rendering as a second implementation.

[complexlogic]

@s09bQ5 Thanks a lot for the review. I removed scaling the textures to power-of-two dimensions, under the assumption that all GPUs will have NPOT support (it's required in OpenGL 2.0 and later).

Now that it's required to still support OpenGL 1.x, should I add support back for GPUs that don't support NPOT? Or can we make the extension mandatory for 1.x contexts?

[basisbit]

I personally am very much in favor of getting rid of the old fixed function based rendering. No need to keep that legacy code. All the other platforms will benefit from getting rid of it and switching to more current OGL. For example RaspberryPi will have much better performance even without us specifically adding GLES support.
Basically any device built in the past 10 (or maybe 15 by now) years does better support OpenGL 3.0+ Core than the old single-threaded fixed function based rendering. The current code is not even OpenGL 2.x, its just 1.4.

We got rid of 32-bit builds, that is much more of an issue for older setups than OpenGL 3 requirement.

[barbeque-squared]

First off, sorry for merge-conflicting your branch, I only saw it was there at all after I already merged a bunch of things.

With that out of the way, I did scroll through the branch diff a bit and I liked what I saw a lot. I remember messing around with the info bar (the one with all the rectangles when you're singing) and oscilloscopes some time ago and I also remember how often something was suddenly taking up half the screen, or invisible, etc. Bottom line: I am heavily interested in eventually merging this.

As to what versions you should (or shouldn't) target, I'll defer to s09bQ5. Though personally, OpenGL 2.0 (or 1.x but requiring the NPOT extension) is perfectly valid. I can't comment on 2 vs 3 as a minimum, though I'm assuming NPOT is the biggest nightmare here anyway (but again I don't know much about this graphics internal stuff)

Regarding merge conflicts, if you want to get some of the trivial renames done already (like the text one, but there's probably more) so that you don't have to constantly keep rebasing, I am okay with that.

[complexlogic]

As a compromise, I could add support for OpenGL 2.0.

2.0 is at least somewhat modern in that it supports shaders and NPOT textures. I would have to adapt the code slightly, but way less than adding back the ancient 1.x fixed function stuff.

Setting 2.0 as a minimum would give us at least 20 years of hardware compatibility for PCs.

[basisbit]

I'd like to again put emphasis on that we just recently dropped 32-bit support, requiring 64bit OS installs, and we just in the last release required that change. Adding the switch to OGL 3 would now fit perfectly. I checked again and from what I could find, all the typical hardware that typically comes with 64 bit OS also already supports OpenGL 3. So there would be no benefit in keeping 2.0 support around - it would just harm performance on some platforms.

[s09bQ5]

I'd like to again put emphasis on that we just recently dropped 32-bit support, requiring 64bit OS installs, and we just in the last release required that change.

We didn't drop 32 bit support. We just stopped providing binaries for 32 bit OSes. And if I remember correctly, in the PR for that it was argued that users of 32 bit machines can still compile USDX for their systems.

For example RaspberryPi will have much better performance even without us specifically adding GLES support.

Proof?

Why do people think that the fixed function OpenGL API is bad? Just because Khronos renamed it to "legacy"? Why do we need shaders? Nobody has plans for visual changes that requires them.

Machines that have support for OpenGL 3+ are fast enough to cope with any performance drawback of OpenGL 1.x/2.x.

[complexlogic]

Why do we need shaders? Nobody has plans for visual changes that requires them.

Here's an idea I have for how USDX could benefit from shader support.

You've probably noticed that video decoding performance in USDX is not great. It often struggles with high resolution files. One of the reasons for this is the use of RGB textures for the video frames.

Almost all video files use YUV colorspace, because it provides superior compression. But YUV data can't be displayed by the GPU. So we convert them to RGB format via the FFmpeg swresample library and then upload them to the GPU.

With support for shaders, we could instead upload frames to the GPU in YUV format, and have the GPU do the YUV->RGB conversion. This would cut the memory bandwidth in half (1.5 bytes per pixel vs 3 bytes per pixel), and reduce the load on the CPU.

This is one example of an optimization that can be made with shaders that isn't possible with the fixed function pipeline.

[s09bQ5]

The beauty of the legacy OpenGL API is that you don't have to use shaders. If they are supported, you can use them to speed things up. If not, you can continue to render the old way.

[complexlogic]

This branch has reached feature parity with master except for the ProjectM visualizer.

How to approach ProjectM will depend strongly on the versions of OpenGL that need to supported. The latest ProjectM 3.x/4.x releases require modern OpenGL (3.x). So this will be on hold until we can reach an agreement on the compatibility question.

I remain in favor of dropping support for the OpenGL 1.x fixed function pipeline API in favor of the shader-based APIs.

[basisbit]

I still think that getting rid of glBegin / glEnd / fixed function pipeline would be worth it and switching to requiring OpenGL 3.3 or newer would be the best way to go forward. Any hardware released within the past 13+ years supports it. No need to maintain two rendering code paths (fixed function and shader based) concurrently. People with machines that are too old can still continue playing using previously released USDX versions.

How much legacy code we decide to maintain has an impact on how fast new features and improvements of UltraStar Deluxe can be developed and tested+released.

I'd like to force getting an agreement within the next few weeks, so that development on this very helpful PR can continue, and thus suggest we just vote on it.

🚀 Option 1: Maintain OpenGL 1.4+ and 2 and 3/4 compatibility for now
🎉 Option 2: Drop OpenGL 1.4+ but maintain 2.1+ and 3/4 compatibility using shaders
❤️ Option 3: Drop fixed function pipeline rendering code and only maintain OpenGL 3 or newer

Please vote via emoticon reaction within the next 7 days.

[bohning]

I am on an ancient 2015 Intel Macbook Pro with an Intel Iris Graphics 6100 which fully supports OpenGL 3.3 and 4.1 Core Profile. In general, I am mostly in favor of supporting old hardware (because I am likely to be affected), but considering that even my ancient computer supports OpenGL 3, dropping support for 1.4+ and 2 is not exactly making a switch that requires brand-new hardware.

[s09bQ5]

You call that ancient? May I remind you of #1156, which was reported this year? That chipset is from 2008 and its GPU barely supports OpenGL 3.3.

[s09bQ5]

A bit of GPU history:

• 2001 ATI R200, OpenGL 1.3
• 2001 Intel Gen2 Extreme Graphics (Pentium 3 Mobile), OpenGL 1.3
• 2002 NVidia GeForce 4, OpenGL 1.2/1.3
• 2002 ATI R300, OpenGL 2.0 without NPOT
• 2003 NVidia GeForce FX, OpenGL 1.5 hardware with 2.1 software emulation
• 2004 NVidia GeForce 6, OpenGL 2.0 hardware with 2.1 software emulation
• 2004 Intel Gen3 GMA 900 (Pentium 4), Windows OpenGL 1.4, Linux OpenGL 2.1
• 2005 ATI R500, OpenGL 2.0 without NPOT
• 2006 NVidia GeForce 7, OpenGL 2.0/2.1 hardware
• 2006 NVidia GeForce 8, OpenGL 3.3
• 2007 ATI R600, OpenGL 3.3
• 2007 Intel Gen4 GMA X3500 (Core 2), Windows OpenGL 2.0, Linux OpenGL 2.1
• 2008 Intel Gen4 GMA 4500 (Core 2), OpenGL 2.1
• 2010 Intel Gen5 HD Graphics (Westmere), OpenGL 2.1
• 2011 Intel Gen6 HD Graphics (Sandy Bridge), OpenGL 3.3

[bohning]

GPU History Continuation (2012–2026)

• 2012 Intel Gen7 HD 4000 (Ivy Bridge): Windows OpenGL 4.0, Linux OpenGL 4.2. Intel's first GPU featuring hardware tessellation support.
• 2012 NVidia GeForce 600 (Kepler): OpenGL 4.6. Introduction of the long-lived Kepler architecture.2012 AMD Radeon HD 7000 (GCN 1.0): OpenGL 4.6. Foundation for later console chips (PS4/Xbox One).
• 2013 Intel Gen7.5 HD 4600 (Haswell): Windows OpenGL 4.3, Linux OpenGL 4.5.
• 2014 NVidia GeForce 900 (Maxwell): OpenGL 4.6. Highly optimized power efficiency.
• 2015 Intel Gen8 HD 5300 (Broadwell): Windows OpenGL 4.4, Linux OpenGL 4.6.
• 2015 Intel Gen9 HD 530 (Skylake): OpenGL 4.6. Also featured early Vulkan 1.3 support on Windows.
• 2016 NVidia GeForce 10 (Pascal): OpenGL 4.6. Legendary generation (e.g., GTX 1080 Ti) with full Vulkan support.
• 2016 AMD Radeon RX 400 (Polaris): OpenGL 4.6. Aggressive price-to-performance cards for the mainstream market.
• 2017 Intel Gen9.5 UHD 620 (Kaby Lake): OpenGL 4.6. Evolutionary refresh of the Skylake graphics engine.
• 2018 NVidia GeForce 20 (Turing): OpenGL 4.6. Introduction of dedicated hardware Ray Tracing (RT) cores and DLSS.
• 2019 AMD Radeon RX 5000 (RDNA 1): OpenGL 4.6. Shift away from the old GCN layout to a gaming-focused architecture.
• 2020 Intel Gen12 Iris Xe (Tiger Lake): OpenGL 4.6. Significant performance leap for integrated Intel graphics.
• 2020 NVidia GeForce 30 (Ampere): OpenGL 4.6. Massive raw compute increase and second-generation ray tracing.
• 2020 AMD Radeon RX 6000 (RDNA 2): OpenGL 4.6. First AMD cards with hardware ray tracing (also powers PS5/Xbox Series X).
• 2022 NVidia GeForce 40 (Ada Lovelace): OpenGL 4.6. Introduction of DLSS 3 (Frame Generation) and extreme energy efficiency.
• 2022 AMD Radeon RX 7000 (RDNA 3): OpenGL 4.6. First consumer gaming GPUs using a chiplet design.
• 2024 Intel Arc Graphics (Meteor Lake / Core Ultra 1): OpenGL 4.6. Integrated "Xe-LPG" architecture with modern ray tracing capabilities.
• 2025 NVidia GeForce 50 (Blackwell): OpenGL 4.6. Massive focus on consumer AI acceleration and next-gen upscaling technologies.
• 2025/2026 AMD Radeon RX 8000 (RDNA 4): OpenGL 4.6. Focus on mid-range price-performance sweet spots and overhauled ray tracing hardware.

[complexlogic]

Thanks @s09bQ5 and @bohning for the history. That tracks closely with my estimate of ~15 years of hardware compatibility for this branch as it currently stands.

As I previously mentioned, I could drop the minimum version down to OpenGL 2.0 with a little bit of added complexity to the code, but not too bad. That would get us another 5 years of hardware compatibility, to about 20 years total.

Adding back the 1.X API would require a major restructuring of the code, and increase the maintenance burden in the long term. I would like to avoid that if possible.

[barbeque-squared]

re: #1296 (comment) to add some explanation to why I picked "drop 1, keep 2.1 and above":

Would a compromise be acceptable? That is:

• gl3+ has all the features
• gl2.1 has the absolute core of USDX, which in my opinion would be: static images (covers, backgrounds) and anything 'static' that is needed for the sing screen (like oscilloscopes). But everything else (videos, screen transitions, visualizers, webcams, the equalizer thingy in the song selection screen) only needs to be bothered with if it's trivial to implement; they're just Nice To Have. Chances are on most of those systems you don't want to use any of these features anyway.
• if this can be extended to gl1: great, but only if it would not have to infect the entire codebase. Otherwise it's time to get rid of it.

My tertiary backup system (I hope I won't ever need it) has an X4500 HD. More pressing, until less than a year ago my main day-to-day system was a 3rd gen i3 with iGPU (~14 years old apparently), and only recently did I "upgrade" to ~9-year old hardware. But that upgrade was mostly because there's so much anti-AI javascript these days, USDX was never the issue.

Personally, I'm actually fine with whatever comes out of the poll, because I have enough systems on GL3 and higher (and I really don't care about projectm at all) that it's not an issue for me. Dropping 1 seems like a sane idea to me, dropping 2 as well won't affect me personally but the "15 years of hardware compatability" is eerily close to my previous daily.

[complexlogic]

@barbeque-squared Good points. I just switched my vote to Option 2. I think it's a fair compromise, and we should now have a final answer on this question.

Last question I have is in regard to ProjectM. As I mentioned, we currently use very outdated ProjectM 2.x, which doesn't support OpenGL 3.x/4.x. The latest 4.x versions of ProjectM have dropped support for OpenGL 1.x/2.x. So they are mutually incompatible.

I would really like to upgrade ProjectM to 4.x as there have been a lot of really good improvements to that library since 2.x, and it would greatly simplify our codebase. So users with 20 year old PCs running OpenGL 2.x would still be able to play the core game, but ProjectM would not be available. IMO this is fine since it's just an optional feature anyway.

[basisbit]

@complexlogic feel free to go ahead and switch it to ProjectM 4.x.
How about checking at startup, if a OpenGL 3.1 context can be obtained (in which case the ProjectM dependency could also be used), if not then try obtaining OpenGL ES 2 context, if that fails, fall back to OpenGL 2.1 context?

[complexlogic]

This is now ready for review. Not going to be an easy review because of the huge diff, however, I think this is a major step forward for the game.

I tested this thoroughly on Windows, Linux, and Raspberry Pi 4. Didn't test at all on Mac because I have no hardware access, but perhaps @bohning could help with that.

[bohning]

Tested on my 2015 Intel Macbook Pro with Intel Iris Graphics 6100 and seems to work as before, couldn't notice anything of concern. Should I be looking at something in particular?

As a side note: I now have src/lib/openCV3/libopencvwrapper.dylib as a changed file, I assume this is now created during the build process (presumably unrelated to this, but a recent PR).

[complexlogic]

From the user's perspective, there should be no noticeable changes from master.

A difficult aspect of graphics programming is that some drivers may not be optimized in the same way as others. This branch could perform well on my PC but struggle on another PC.

That's why I wanted a Mac tester, especially because Apple's OpenGL implementation is not actually a hardware driver but a compatibility layer running on top of Metal. But glad to hear it's working well.

[dgruss]

Would it be possible to restructure the series of commits to something that is easier to review and also look at in hindsight? At the minimum having consistent meaningful commit messages, but I more and more moved to restructuring the entire commit chain and drawing the lines between commits by structure and not timeline, which makes it easier to look at one commit and understand what it does. That would be helpful here as well, considering that this is changing more than a hundred files with a 13000 line diff?