robust ctrl+c interrupt handling

[jkozniewski]

From my experience (using Pico Pi with MicroPico VSCode extension and mpremote CLI on Mac OS) interrupting the code that already runs on microcontroller so one could regain control over the board is super finicky.
I have constant situations in which I have to plug/unplug the board several times before I can stop the code and run / upload something on the board.

Is there any bullet proof, or at least super-reliable way to make Ctrl+C interrupt handled more reliably - both on microcontroller (Pico Pi) and host computer side ?

[peterhinch]

I am strictly a command-line man, so my comments apply to using mpremote at the Bash prompt. In that environment ctrl-c works with normal Python code 100% reliably. It only fails if the program experiences a hard crash i.e. where the bytecode interpreter has crashed. This is unlikely to happen unless you're getting into the weeds with fairly technical code (directly accessing hardware etc.).

It's also worth noting that a soft reset is not the same as a hard reset: some resources are not tidied up unless you trap the KeyboardInterrupt and do it yourself. Timers and RP2 state machines (for example) can continue to run.

Reading comments in here, most of these problems are caused by IDE's.

[jkozniewski]

Seems that might be a main culprit - #13504
Any ETA on when it would be fixed, if my experience is common (and i assume it is) amongst mac os users it really hinders working with micropython quite a bit, as being able to upload code reliably is just some basic workflow element...

[jkozniewski]

@peterhinch thanks for your comment - I'm assuming your not using mac os platform and direct usb A to usb micro cable ?
I suspect there might be some issues related to hardware usb stack especially with hubs and usbc - usb micro converters.
I'm saying this since I recently used mpremote in bash scripts for automating code upload to subsequently connected boards and it was rather unreliable contrary to your experience...

here is the upload script (AI generated) I use that also pulls current commit id from github for versioning etc. works like 70% of the time, sometimes at first, sometimes 3rd retry and sometimes faill completely... curiously if I switch usb ports in between uploads it helps with reliability

#!/usr/bin/env bash

# ... < shortened for readability to show only mpremote part > ...

# connect auto: pick first USB serial device.
# fs cp -r -f: recursive + force overwrite existing files.
# + reset: terminate fs argument list, then hard reset the board.
echo "Stabilizing USB serial for ${MPREMOTE_STABILIZE_DELAY}s before upload..."
sleep "$MPREMOTE_STABILIZE_DELAY"

attempt=1
while (( attempt <= MPREMOTE_RETRIES )); do
	echo "mpremote upload attempt ${attempt}/${MPREMOTE_RETRIES}..."
	if "${MPREMOTE[@]}" connect auto fs cp -rf "$TMP_UPLOAD_DIR/." "$REMOTE_DEST" + reset; then
		echo "Upload complete and device reset."
		exit 0
	fi

	if (( attempt < MPREMOTE_RETRIES )); then
		echo "Warning: mpremote failed (often transient while device is rebooting). Retrying in ${MPREMOTE_RETRY_DELAY}s..." >&2
		sleep "$MPREMOTE_RETRY_DELAY"
	fi

	attempt=$((attempt + 1))
done

echo "Error: mpremote failed after ${MPREMOTE_RETRIES} attempts." >&2
echo "Tip: close any serial monitor (Thonny, screen, etc.) and retry." >&2
exit 1

[Josverl]

I have run many thousands of automatomaterd Mpremote scripts, on windows and Unix quite reliable, after I learned to be patient, and how to properly use Mpremote for speed.

The big snafu that I kept running into is that PCs are much faster than the fastest MCU. And also the DTR/CTS lines dance differently on different OSes, and on some MCU or boards can trigger unseen hardware resets that may take a second or two. ( Which to a script is twice an eternity) And there are only very very few board that implement hardware flow control to add to the challenge.
Also, each Mpremote command, by default resets the MCU. which needs to be accounted for, and which the sample script above appears to blissfully ignore.

As a Contracting example: Mpflash has a built-in Mpremote board API built on batching the Mpremote commandline it has been used across OSes by many different users, for a few years already.
Not 100% proof by any means, but in my personal view more relevant than some script by some AI.

The secret sauce ? moremote resume <rest of the command>

On the IDE Integration.
With IDEs multiplexing board detection and control, file listings and transfers, a REPL, running files and handling output over a single bi-di lane is bound to be challenging as there is no built-in multiplexing protocol to support that.

[jkozniewski]

Thanks for your feedback - seems mac os is lacking in terms of stability then as my experience is pretty bad.
Can you elaborate on mpremote resume <...> is just adding resume argument to each command makes it inherently more reliable ?

How would you improve on the script I've posted (yes, bit ashamed that I used AI generated code :P).

[Josverl]

No need for AI shame,and happy that you are open about it ( as I am when I use one)

And yes, it's a counterintuitive design decision that Mpremote by default starts by resetting the MCU.
resume skips that reset.
It probably made sense for the use case at the time, but not for batching commands from a script. ( And Andrew has a PR open to change the default)

So you can use resume all the time everywhere, or on all but the first command in your script.

Specifying the serial port may add some stability, I have no Mac, but have needed to special-case several aspects in ports selection just for Mac users reporting different behavior. Likely mine are clumsy workarounds as I lacked direct interaction with the systems in question.

[andrewleech]

both on microcontroller (Pico Pi) and host computer side

On host computer do you mean the Unix port? If so there are some known limitations there with how/when stdin is read/checked for the interrupt character.
I've tried to improve this in #18810 but a quick scan through the number of changes/rewrites it's been through should give a feeling for how tricky this has been, there lots of subtle edge cases here.

On device, the complications / unreliability I've run into usually depend on what I'm doing in the application being run.
For context on every port / transport used to connect to repl (uart / usb-cdc etc) there's differences in the transport drivers and how much text stream they can buffer and how early in the buffer they can search for an interrupt character and whether that can be done in an interrupt or whether it has to be a poll.
In many cases the ctrl-c interrupt can't be fully serviced until there's mp_event service of some kind behind the scenes (most commonly during a sleep of some kind or many blocking io operations).
If the application has some "tight loops" or blocking operations that don't defer to mp_event_wait in some way at a C level yes the ctrl-c handling can get deferred or blocked.

[jkozniewski]

To be clear - I'm talking about particular setup:
host - mac os (sequoia, apple silicon)
board - raspberry pi pico (both 1 and 2)

As for mp_event_wait - ok, so on practical level what kind of code structure makes catching those event more reliable - adding time.sleep_ms(1) everywhere ? That is not always possible if the timing is critical...

[andrewleech]

On a separate note there's also some issues with mpremote putting some boards into raw repl mode due often to delays in responding to ctrl-c or reboots, fwiw I'm slowly drafting improvements to this here: andrewleech#48

[jkozniewski]

The TransportError: could not enter raw repl is an error that I see pretty frequently - both using raw mpremote and as an error message using MicroPico VS Code extension - so hope your PR will indeed mitigate this issue.

[peterhinch]

@jkozniewski

I'm assuming your not using mac os platform and direct usb A to usb micro cable ?

Linux with that cable.

Judging by the above I must be leading a charmed life, but then I haven't been scripting mpremote. Re TransportError: could not enter raw repl I have seen this, but in the context of code which was liable to crash. Not with normal Python.

[jkozniewski]

So I guess it might be more of a mac os - usb hardware / firmware - cable - pico pi problem than micropython repl implementation, although given the issue (andrewleech#48) I've rfered above it potentially could be made more resilient on host side... so fingers crossed for that!