Potential out-of-memory crash selfbooting Dreamcast port

[distay0xGit]

Now, I don't really know if this is just an issue on my part with how I set it up, but I have tried compiling and emulating the port several times across 2 different machines trying to figure this out, and even tried testing some of them on my stock Dreamcast (selfbooting with both MIL-CDs and data/data discs just to be sure), and all of them seem to crash the same way just before startup.

From looking at the logs and messing around with the malloc asserts in my build of KOS, I can only assume libkallisti is chewing through too much memory before SM64 has the chance to set itself up, but I'm not all that experienced with KallistiOS, so I can't really speak on it.

I don't really have a serial adapter to see what real hardware is outputting, but I can at least provide the serial log from me running it on Flycast:
i-suffer-from-skill-issue-syndrome.txt

Machine 1: Asus TUF Gaming FX506LH (w/ Windows 11 v22621.2715)
Machine 1 tools: MinGW (32 bit build), KallistiOS master branch (+ kos-ports), cdi4dc v0.3b, mkisofs

Machine 2: Alienware 17 R2 (w/ UwUntu 22.10 LTS and some update of 22H2 Windows 10)
Machine 2 tools: KallistiOS master branch (+ modified asserts), cdi4dc v0.2b, genisoimage

[bkacjios]

I just managed to compile this today too and get that same assertion.

--
KallistiOS Git revision :
  Sun Apr 21 05:07:57 PM EDT 2024
  root@pi.hole:/opt/toolchains/dc/kos
  sh-elf-gcc (GCC) 13.2.0
thd: pre-emption enabled, HZ=100
maple: active drivers:
    Dreameye (Camera): Camera
    Sound Input Peripheral: Microphone
    PuruPuru (Vibration) Pack: JumpPack
    VMU Driver: Clock, LCD, MemoryCard
    Mouse Driver: Mouse
    Keyboard Driver: Keyboard
    Controller Driver: Controller
    Lightgun: LightGun
  DMA Buffer at acebe560
vid_set_mode: 640x480IL NTSC with 1 framebuffers.
maple: attached devices:
  A0: Dreamcast Controller          (01000000: Controller)
  A1: Visual Memory                 (0e000000: Clock, LCD, MemoryCard)
  A2: Visual Memory                 (0e000000: Clock, LCD, MemoryCard)

Loading configuration from 'sm64config.txt'
Config file 'sm64config.txt' not found. Creating it.
Saving configuration to 'sm64config.txt'

Welcome to GLdc! Git revision:

pvr: enabling vertical scaling for non-VGA
stack: start:8c010000 end:8cea4664
Total Ram: 16777216, Free Ram: 929796, Need: 1270204,  Used Ram: 15847420
GL_VERSION = 1.2 (partial) - GLdc 1.1
GL_EXTENSIONS =
GL_ARB_framebuffer_object, GL_ARB_multitexture, GL_ARB_texture_rg, GL_EXT_paletted_texture, GL_EXT_shared_texture_palette, GL_KOS_multiple_shared_palette, GL_ARB_vertex_array_bgra, GL_ARB_vertex_type_2_10_10_10_rev
snd_init(): loading 3244 bytes into SPU RAM
snd_stream: alloc'd channels 0/1

*** ASSERTION FAILURE ***
Assertion "(old_top == initial_top(av) && old_size == 0) || ((unsigned long)(old_size) >= MINSIZE && prev_inuse(old_top))" failed at pvr_mem_core.c:1544 in `sYSMALLOc'

arch: aborting the system

--
KallistiOS Git revision :
  Sun Apr 21 05:07:57 PM EDT 2024
  root@pi.hole:/opt/toolchains/dc/kos
  sh-elf-gcc (GCC) 13.2.0

[bkacjios]

It's worth noting, the game does seem to run for about 4 frames before dying.. Been trying to dig into what specifically is causing the crash.

stack: start:8c010000 end:8cea62a4
Total Ram: 16777216, Free Ram: 921604, Need: 1278396,  Used Ram: 15855612
GL_VERSION = 1.2 (partial) - GLdc 1.1
GL_EXTENSIONS =
GL_ARB_framebuffer_object, GL_ARB_multitexture, GL_ARB_texture_rg, GL_EXT_paletted_texture, GL_EXT_shared_texture_palette, GL_KOS_multiple_shared_palette, GL_ARB_vertex_array_bgra, GL_ARB_vertex_type_2_10_10_10_rev
DONE
snd_init(): loading 3244 bytes into SPU RAM
snd_stream: alloc'd channels 0/1
audio_init.. DONE
sound_init.. DONE
setup_game_memory.. DONE
init_controllers.. DONE
save_file_load_all.. DONE
set_vblank_handler.. DONE
set_vblank_handler.. DONE: levelCommandAddr=0x8c3186a4
play_music.. DONE
set_sound_mode.. DONE
main_loop
gfx_start_frame.. DONE
profiler_log_thread5_time.. DONE
osContStartReadData.. DONE
audio_game_loop_tick.. DONE
config_gfx_pool.. DONE
read_controller_inputs.. DONE
level_script_execute.. DONE: levelCommandAddr=0x8c3186a8
display_and_vsync.. DONE
render_fps.. DONE
audio_api->play.. DONE
gfx_end_frame.. DONE
gfx_start_frame.. DONE
profiler_log_thread5_time.. DONE
osContStartReadData.. DONE
audio_game_loop_tick.. DONE
config_gfx_pool.. DONE
read_controller_inputs.. DONE
level_script_execute.. DONE: levelCommandAddr=0x8c3186a8
display_and_vsync.. DONE
render_fps.. DONE
audio_api->play.. DONE
gfx_end_frame.. DONE
gfx_start_frame.. DONE
profiler_log_thread5_time.. DONE
osContStartReadData.. DONE
audio_game_loop_tick.. DONE
config_gfx_pool.. DONE
read_controller_inputs.. DONE
level_script_execute.. DONE: levelCommandAddr=0x8c778e48

*** ASSERTION FAILURE ***
Assertion "(old_top == initial_top(av) && old_size == 0) || ((unsigned long)(old_size) >= MINSIZE && prev_inuse(old_top))" failed at pvr_mem_core.c:1544 in `sYSMALLOc'

arch: aborting the system

[bkacjios]

Tracked down the crash to gfx_opengl_upload_texture in gfx_gldc.c

It's crashing on the very first texture it tries to create.

[bkacjios]

Figured out the issue. I had to remove the libgldc that is packaged with this app and use the one from kos-ports.

Only issue is now GL_UNSIGNED_SHORT_1_5_5_5_REV texture format seems unsupported.

[bkacjios]

Managed to figure out the textures and created a fork. https://github.com/bkacjios/sm64-port

I made a pull request in GLdc to implement the texture conversion that SM64 needs here: https://gitlab.com/simulant/GLdc/-/merge_requests/117

Once that gets merged, you should be able to build with my fork with no issues, but there are still a few issues to iron out.