Memory use and garbage collection causing me headaches

[JESteward]

Hi All,

I'm quite new to python and embedded micropython, and I'm on a steep learning curve. I'm hoping I've just made a rookie mistake, and that someone can put me on a path to enlightenment.

I've been hacking a port of micropython with LVGL (lv_bindings_micropython) to a custom STM32 based board with Riverdi 5" display (EVE driver). Everything appears to be functional. I wrote a small C module to add the hardware layer and call to create the EVE display, because gen_mpy.py choked on the function.

I have a minimal test script that runs. It configures the hardware (SPI and pins), creates the display, and puts a button on the screen with a callback for when it is clicked.

For some reason, with every button press, free memory reduces and eventually the app breaks.

I've made the button text display the free memory via

label.set_text(f"mem free {mf}")

I've tried calling gc.collect() earlier but for some reason that causes the app to stop working as well.

I guess some object is being freed, but I don't know how to find out what.

Any suggestions?

Code below...

`import asyncio
import machine
import sys
import gc
import lvgl as lv

EVE driver module for EVE display create and hardware callback function.

import eve_gpu

Bring up native peripherals matching standard configuration paradigms

spi_hardware = machine.SPI(1, baudrate=1000000, polarity=0, phase=0)
cs = machine.Pin('H7', machine.Pin.OUT, value=1)
pd = machine.Pin('H12', machine.Pin.OUT, value=1)

async def async_timer(delay):
while True:
await asyncio.sleep_ms(delay)
lv.tick_inc(delay)
lv.timer_handler()

lvobs = set()
scr = None
label = None

Define event callback for the button

def btn_event_cb(e):
global label
code = e.get_code()
if code == lv.EVENT.CLICKED:
mf = gc.mem_free()
label.set_text(f"mem free {mf}")

async def main():
global lvobs, scr, label

if not lv.is_initialized():
    lv.init()

# Must start the tick increments before create display.
# It uses ticks for a delay during display initialisation
timer_task = asyncio.create_task(async_timer(33))
lvobs.add(timer_task)

# Arguments: (SPI_ID, CS_Pin, PD_Pin, Width, Height)
display = eve_gpu.create_display(1, cs, pd, 800, 480)
lvobs.add(display)

# Draw widgets normally using standard LVGL interfaces
scr = lv.obj()
lvobs.add(scr)

btn = lv.button(scr)
lvobs.add(btn)

btn.align(lv.ALIGN.CENTER, 0, 0)
label = lv.label(btn)
lvobs.add(label)

# Add the event to the button
btn.add_event_cb(btn_event_cb, lv.EVENT.ALL, None)
lvobs.add(btn_event_cb)

# Load the screen
lv.screen_load(scr)

pmf = gc.mem_free()
print(pmf)

while True:
    await asyncio.sleep(1)
    mf = gc.mem_free()
    diff = mf-pmf
    pmf = mf
    if (mf < 150000):
        print(f"mem free {mf} {diff}")
        print(globals())
        gc.collect()
        await asyncio.sleep(0)

Start the event loop

try:
asyncio.run(main())
except KeyboardInterrupt:
print("Program stopped")

scr.delete()

sys.exit()
`

[GitHubsSilverBullet]

For a start, you might want to format your code appropriately.
How to do that is a well hidden secret sitting behind the
»Welcome (Please read before posting)«
link. ;-)

[JESteward]

Good idea. Thanks.

J.

import asyncio
import machine
import sys
import gc
import lvgl as lv

# EVE driver module for EVE display create and hardware callback function.
import eve_gpu

lvobs = set()
scr = None
label = None
refresh_event = None

# Bring up native peripherals matching standard configuration paradigms
spi_hardware = machine.SPI(1, baudrate=1000000, polarity=0, phase=0)
cs = machine.Pin('H7', machine.Pin.OUT, value=1)
pd = machine.Pin('H12', machine.Pin.OUT, value=1)

async def async_timer(delay):
    global refresh_event
    while True:
        await asyncio.sleep_ms(delay)
        await refresh_event.wait()
        lv.tick_inc(delay)
        lv.timer_handler()

# Define event callback for the button
def btn_event_cb(e):
    global label
    code = e.get_code()
    if code == lv.EVENT.CLICKED:
        mf = gc.mem_free()
        label.set_text(f"mem free {mf}")
        
async def main():
    global lvobs, scr, label, refresh_event
    
    if not lv.is_initialized():
        lv.init()

    # Must start the tick increments before create display.
    # It uses ticks for a delay during display initialisation
    refresh_event = asyncio.Event()
    lvobs.add(refresh_event)
    refresh_event.set()

    timer_task = asyncio.create_task(async_timer(33))
    lvobs.add(timer_task)
    
    # Arguments: (SPI_ID, CS_Pin, PD_Pin, Width, Height)
    display = eve_gpu.create_display(1, cs, pd, 800, 480)
    lvobs.add(display)

    # Draw widgets normally using standard LVGL interfaces
    scr = lv.obj()
    lvobs.add(scr)
    
    btn = lv.button(scr)
    lvobs.add(btn)

    btn.align(lv.ALIGN.CENTER, 0, 0)
    label = lv.label(btn)
    lvobs.add(label)
    
    # Add the event to the button
    btn.add_event_cb(btn_event_cb, lv.EVENT.ALL, None)
    lvobs.add(btn_event_cb)

    # Load the screen
    lv.screen_load(scr)

    pmf = gc.mem_free()
    print(pmf)
    
    while True:
        await asyncio.sleep(1)
        mf = gc.mem_free()
        diff = mf-pmf
        pmf = mf
        if (mf < 150000):
            refresh_event.clear()
            print(f"mem free {mf} {diff}")
            print(globals())
            gc.collect()
            await asyncio.sleep(0)
            refresh_event.set()

# Start the event loop
try:
    asyncio.run(main())
except KeyboardInterrupt:
    print("Program stopped")

scr.delete()

[peterhinch]

Re

    if code == lv.EVENT.CLICKED:
        mf = gc.mem_free()
        label.set_text(f"mem free {mf}")

you should always call gc.collect() before gc.mem_free(), otherwise your measurement will include memory which has been allocated but which can now be freed. In other words, you tend to see an ever reducing pool of free memory (until the next auto GC occurs).

I've tried calling gc.collect() earlier but for some reason that causes the app to stop working as well.

If gc.collect() causes a failure something is very wrong, possibly with the LVGL integration.

[JESteward]

Hi Peter,

Yes. I have read that gc.collect() should be called before gc.mem_free(), and I agree that there is something wrong. Thanks for your reply.

Regards,
James.

[sosi-deadeye]

This isn't the issue here, but it's still something to avoid. Don't use global inside functions unless it's actually needed. From what I can see, you're only calling methods on the object, so global is unnecessary. You only need global if you assign a new object to that name inside the function, replacing the object in the global namespace.

[JESteward]

Hi Sosi,

Thanks for your reply. I will take your advice onboard. Please bear with me as I'm quite new to Python/Micropython, so advice like yours is gratefully received. Just having garbage collection is a new concept for me, as my experience has been C code and bare metal microprocessor coding.

Regards,
James.

[beetlegigg]

@JESteward, I don't know if this will help much, but some of your code triggered a memory cell relating to a first, and thus far my last, quick dabble with LVGL. I think it was the similarity in the code and possibly we were reading the same googled example. In my quick test of a lvgl button I used a pre-made binary that included lvgl (dont remember from whence it came) and I made the button update a label, the label being placed on the button itself.

The hw I used was a rpi pico2W with an ili9341 screen. Anyhow I just fished it out and done a quick change to make the button report on the free memory, and included a memory gobbling routine so it reported the free memory accordingly. The free mem reported rapidly declines and then rises again at the variable is reset and gc reclaims memory and then drops again etc, it keeps going and does not crash.

Apart from this small test I did sometime last year I've not played with LVGL so I dont really know how it should be set up an run, but for what its worth I show this small snippet of code.

# test_button.py
#
# MicroPython v1.25.0
# Pico2 W 
# LVGL 9.4

import asyncio
import lvgl as lv
from machine import reset
from display_driver import disp, touch
import time
import gc

lv.init()

# screen
scr = lv.obj()
lv.screen_load(scr)

scr.set_style_bg_color(lv.color_hex(0),lv.PART.MAIN)
scr.set_style_border_width(2, lv.PART.MAIN)
scr.set_style_border_color(lv.palette_main(lv.PALETTE.BLUE),lv.PART.MAIN)

# button style
btnstyle = lv.style_t()
btnstyle.init()
btnstyle.set_radius(5)
btnstyle.set_bg_opa(lv.OPA.COVER)
btnstyle.set_bg_color(lv.palette_main(lv.PALETTE.BLUE))
btnstyle.set_outline_width(2)
btnstyle.set_outline_color(lv.palette_main(lv.PALETTE.BLUE))
btnstyle.set_outline_pad(8)
 
# Button
btn = lv.button(scr)
btn.set_size(100,50)
btn.center()
btn.add_style(btnstyle, lv.PART.MAIN)

lbl = lv.label(btn)
lbl.set_text("Memory")
lbl.center()
lbl.set_style_text_color(lv.color_black(), lv.PART.MAIN)
lbl.set_style_text_font(lv.font_montserrat_24, lv.PART.MAIN)

def btn_cb(event):
    lbl.set_text(str(gc.mem_free()))
 
btn.add_event_cb(btn_cb, lv.EVENT.CLICKED, None)

async def do_something():
    while True:
        mystring = ''
        for num in range(10_000):
            mystring += 'yet more text to gobble memory'
            await asyncio.sleep(0.1)
                              
try:
    asyncio.run(do_something())
except KeyboardInterrupt:
    print("Ctrl C")
finally:
    asyncio.new_event_loop()

[JESteward]

Hi Bg,

Thanks, yes your example looks similar. It must have been a while ago, and I'm not sure how it worked, because for LVGL to work it has to have the timer tick incremented and the timer handler function called to update widgets - which doesn't appear in your example. Maybe they were called in C code behind the scenes.

I have noticed a few times the memory decrease to zero and then rebound. Very rarely that works and the app continues, but mostly it crashes.

It's as though something tries to allocate memory when there is none left, fails, and doesn't raise and exception or print a message or anything.

I don't know how to trace it.

Regards,
James.

[JESteward]

Come to think of it, LVGL 9.4 is recent, so definitely the timer tick increment and timer handler must have been hiding in C code.

JS.

[beetlegigg]

Just a quick note about my small snippet running without error. I ran it from Thonny and noticed today I did not power off the pico. Whilst the pico and screen was still running and reporting the free mem on a button push I noted that Thonny reported a memory issue and was back at the REPL prompt.

I think this is a Thonny type of issue as it runs its python programs within some sort of monitoring framework, and often the advice from the pro's on this forum in such cases is not to run from Thonny. I guess its very possible that Thonny will not work well with installations that include extra C libraries that may interfere with the Thonny structure. I have no further guidance to impart on this, but thought it worth mentioning in case its of relevance for you.

[JESteward]

Good to know! Thanks again.

JS.

[JESteward]

Thanks to all. I've changed my example according to suggestions - except for calling gc.collect() before reading the free memory. That reliably kills the app, as you can see.

If I click the button a number of times the memory drops below 150000 and gc.collect() is called. I've included the REPL output for interest.

Regards,
James.

import asyncio
import machine
import sys
import gc
import lvgl as lv

# EVE driver module for EVE display create and hardware callback function.
import eve_gpu

lvobs = set()
scr = None
label = None
refresh_event = None

# Bring up native peripherals matching standard configuration paradigms
spi_hardware = machine.SPI(1, baudrate=1000000, polarity=0, phase=0)
cs = machine.Pin('H7', machine.Pin.OUT, value=1)
pd = machine.Pin('H12', machine.Pin.OUT, value=1)

async def async_timer(delay):
    while True:
        await asyncio.sleep_ms(delay)
        await refresh_event.wait()
        lv.tick_inc(delay)
        lv.timer_handler()

# Define event callback for the button
def btn_event_cb(e):
    code = e.get_code()
    if code == lv.EVENT.CLICKED:
        mf = gc.mem_free()
        label.set_text(f"mem free {mf}")
        
async def main():
    global scr, label, refresh_event
    
    if not lv.is_initialized():
        lv.init()

    # Must start the tick increments before create display.
    # It uses ticks for a delay during display initialisation
    refresh_event = asyncio.Event()
    lvobs.add(refresh_event)
    refresh_event.set()

    timer_task = asyncio.create_task(async_timer(33))
    lvobs.add(timer_task)
    
    # Arguments: (SPI_ID, CS_Pin, PD_Pin, Width, Height)
    display = eve_gpu.create_display(1, cs, pd, 800, 480)
    lvobs.add(display)

    # Draw widgets normally using standard LVGL interfaces
    scr = lv.obj()
    lvobs.add(scr)

    # Load the screen
    lv.screen_load(scr)

    btn = lv.button(scr)
    lvobs.add(btn)

    btn.align(lv.ALIGN.CENTER, 0, 0)
    label = lv.label(btn)
    lvobs.add(label)
    
    # Add the event to the button
    btn.add_event_cb(btn_event_cb, lv.EVENT.ALL, None)
    lvobs.add(btn_event_cb)

    pmf = gc.mem_free()
    print(pmf)
    
    while True:
        await asyncio.sleep(1)
        mf = gc.mem_free()
        diff = mf-pmf
        pmf = mf
        if (mf < 150000):
            refresh_event.clear()
            print(f"mem free {mf} {diff}")
            print(globals())
            gc.collect()
            await asyncio.sleep(0)
            refresh_event.set()

# Start the event loop
try:
    asyncio.run(main())
except KeyboardInterrupt:
    print("Program stopped")

scr.delete()

sys.exit()

%Run -c $EDITOR_CONTENT
181200
mem free 147488 -2880
{'cs': Pin(Pin.cpu.A15, mode=Pin.OUT), 'label': lvgl label, 'file': '', 'name': 'main', 'pyb': <module 'pyb'>, 'btn_event_cb': <function btn_event_cb at 0x2001c770>, 'lv': <module 'lvgl'>, 'main': , 'pd': Pin(Pin.cpu.B11, mode=Pin.OUT), 'machine': <module 'machine'>, 'gc': <module 'gc'>, 'asyncio': <module 'asyncio' from 'asyncio/init.py'>, 'scr': lvgl obj, 'async_timer': , 'sys': <module 'sys'>, 'lvobs': {<Event object at 2001c7e0>, lvgl obj, lvgl button, lvgl label, , 536996176, <function btn_event_cb at 0x2001c770>}, 'eve_gpu': <module 'eve_gpu'>, 'refresh_event': <Event object at 2001c7e0>, 'spi_hardware': SPI(1, baudrate=625000, polarity=0, phase=0, bits=8)}
PROBLEM IN THONNY'S BACK-END: Exception while handling 'Run' (ConnectionError: device reports readiness to read but returned no data (device disconnected or multiple access on port?)).

[JESteward]

Reply to myself with more info. Looks increasingly like an LVGL problem.

I modified the main loop so that once the free memory had dropped below 150000, the async timer task was disabled, and following that gc.collect() could be called repeatedly and the app didn't crash. This points to a problem in LVGL.

I guess is it fair to say that the garbage collector frees some memory that LVGL was not finished with?

What do I need to do in the C side of things to ensure that memory allocated in the LVGL EVE driver is not garbage, assuming that the rest of LVGL is ok? Should it use tracked allocation and free? MICROPY_TRACKED_ALLOC?

Regards,
James.

    while True:
        await asyncio.sleep(1)
        mf = gc.mem_free()
        diff = mf-pmf
        pmf = mf
        if (mf < 150000):
            refresh_event.clear()
            print(f"mem free {mf} {diff}")
            print(globals())
            gc.collect()
            await asyncio.sleep(0)
            #refresh_event.set()
            while True:
                await asyncio.sleep(1)
                mf = gc.mem_free()
                diff = mf-pmf
                pmf = mf
                print(f"mem free {mf} {diff}")
                print(globals())
                gc.collect()
                await asyncio.sleep(0)

Result:

%Run -c $EDITOR_CONTENT
180976
mem free 147456 -2784
{'cs': Pin(Pin.cpu.A15, mode=Pin.OUT), 'label': lvgl label, 'file': '', 'name': 'main', 'pyb': <module 'pyb'>, 'btn_event_cb': <function btn_event_cb at 0x2001c7a0>, 'lv': <module 'lvgl'>, 'main': , 'pd': Pin(Pin.cpu.B11, mode=Pin.OUT), 'machine': <module 'machine'>, 'gc': <module 'gc'>, 'asyncio': <module 'asyncio' from 'asyncio/init.py'>, 'micropython': <module 'micropython'>, 'scr': lvgl obj, 'sys': <module 'sys'>, 'lvobs': {, 536990496, <Event object at 2001c9e0>, lvgl obj, lvgl button, lvgl label, <function btn_event_cb at 0x2001c7a0>}, 'spi_hardware': SPI(1, baudrate=625000, polarity=0, phase=0, bits=8), 'eve_gpu': <module 'eve_gpu'>, 'refresh_event': <Event object at 2001c9e0>, 'async_timer': }
mem free 191152 43696
{'cs': Pin(Pin.cpu.A15, mode=Pin.OUT), 'label': lvgl label, 'file': '', 'name': 'main', 'pyb': <module 'pyb'>, 'btn_event_cb': <function btn_event_cb at 0x2001c7a0>, 'lv': <module 'lvgl'>, 'main': , 'pd': Pin(Pin.cpu.B11, mode=Pin.OUT), 'machine': <module 'machine'>, 'gc': <module 'gc'>, 'asyncio': <module 'asyncio' from 'asyncio/init.py'>, 'micropython': <module 'micropython'>, 'scr': lvgl obj, 'sys': <module 'sys'>, 'lvobs': {, 536990496, <Event object at 2001c9e0>, lvgl obj, lvgl button, lvgl label, <function btn_event_cb at 0x2001c7a0>}, 'spi_hardware': SPI(1, baudrate=625000, polarity=0, phase=0, bits=8), 'eve_gpu': <module 'eve_gpu'>, 'refresh_event': <Event object at 2001c9e0>, 'async_timer': }
mem free 191200 48
{'cs': Pin(Pin.cpu.A15, mode=Pin.OUT), 'label': lvgl label, 'file': '', 'name': 'main', 'pyb': <module 'pyb'>, 'btn_event_cb': <function btn_event_cb at 0x2001c7a0>, 'lv': <module 'lvgl'>, 'main': , 'pd': Pin(Pin.cpu.B11, mode=Pin.OUT), 'machine': <module 'machine'>, 'gc': <module 'gc'>, 'asyncio': <module 'asyncio' from 'asyncio/init.py'>, 'micropython': <module 'micropython'>, 'scr': lvgl obj, 'sys': <module 'sys'>, 'lvobs': {, 536990496, <Event object at 2001c9e0>, lvgl obj, lvgl button, lvgl label, <function btn_event_cb at 0x2001c7a0>}, 'spi_hardware': SPI(1, baudrate=625000, polarity=0, phase=0, bits=8), 'eve_gpu': <module 'eve_gpu'>, 'refresh_event': <Event object at 2001c9e0>, 'async_timer': }
mem free 191200 0

Etc....