API Background tasks created with asyncio.create_task could be garbage collected

[JC-wk]

Describe the bug

Background tasks created with asyncio.create_task are not managed or retained

Description

Background tasks started during the FastAPI application lifespan are created using asyncio.create_task() but are not stored or managed. This results in tasks being unreferenced, which can lead to premature garbage collection, lost exceptions, and uncontrolled shutdown behaviour.

Problem

The code previously used:

asyncio.create_task(deploymentStatusUpdater.receive_messages())
asyncio.create_task(airlockStatusUpdater.receive_messages())

Issues with this approach:

• No strong references to tasks
  • Tasks may be garbage collected while still running
• No lifecycle management
  • Tasks are not cancelled or awaited during shutdown
• Unobserved exceptions
  • Failures inside tasks are silently ignored
• Potential shutdown warnings
  • e.g. Task was destroyed but it is pending!

Impact

• Background workers may terminate unexpectedly
• Errors in long-running tasks can go unnoticed
• Application shutdown may be unreliable or produce warnings
• Difficult to debug production issues due to lack of visibility

Expected behaviour

• Background tasks should:
  • Be strongly referenced
  • Be tied to the FastAPI application lifecycle
  • Be cancelled and awaited on shutdown
  • Have exceptions logged with full context

Suggested fix

Track and manage tasks via app.state, for example:

app.state.background_tasks = set()

task = asyncio.create_task(worker())
app.state.background_tasks.add(task)
task.add_done_callback(app.state.background_tasks.discard)

And on shutdown:

tasks = list(app.state.background_tasks)
for task in tasks:
    task.cancel()

await asyncio.gather(*tasks, return_exceptions=True)

Additional notes

• Tasks should also handle asyncio.CancelledError internally for clean shutdown.
• Naming tasks can significantly improve observability and debugging.

---

Steps to reproduce

I have not noticed any times this has happened in practice

Azure TRE release version (e.g. v0.14.0 or main):
main
Deployed Azure TRE components - click the (i) in the UI:
latest