Implement AiiDA control page

[yakutovicha]

fixes #155

[cpignedoli]

User-Friendly AiiDAlab App for Managing Main Workflows (MWs)

This AiiDAlab app aims to enhance the user experience when working with Main Workflows (MWs) in AiiDA. Currently, the default "verdi process list -a" command provides an overview of all workflows, but it lacks user-centric features and can be cumbersome to navigate, especially when dealing with numerous MWs.

MW is a workflow submitted by a user, usually with a description, that will typically have child workflows

Key Features:

1. Intuitive MW List: The app will present a user-friendly MW list, displaying the descriptions of each workflow along with their statuses. Users can quickly identify MWs, such as "PDOS DBBR on Au," "STM goblet," etc., based on the provided descriptions rather than the AiiDA workchain names.

2. Workflow Progress Visualization: Users can easily track the progress of a specific MW by clicking on it. The app will visually highlight completed steps and indicate which step is currently running, queued, or paused. This provides users with a clear understanding of where their MW stands in terms of execution.

3. Process Control: The app will empower users to interact with their MWs efficiently. Users can pause and resume processes as needed, allowing better control over long-running or resource-intensive calculations. Additionally, the app will provide a "kill" option to terminate unwanted or problematic MWs.

4. Efficient Workflow Retrieval: Searching for past MWs will be a breeze with the app's convenient filtering and sorting options. Users can effortlessly find a specific MW, like "DBBR Br removed on Au" from two months ago, and access its individual steps with ease. The app will allow searching based on date ranges and provide a smooth scrolling experience, avoiding disruptive automatic page refreshes.

5. Optimized Performance: Ensuring the app's responsiveness and speed is a priority. By optimizing database queries and utilizing efficient data retrieval techniques, we aim to create a seamless experience for users, even when dealing with a significant number of MWs.

The goal of this app is to make MW management in AiiDA more intuitive, efficient, and enjoyable for users who frequently submit and monitor complex workflows. Your feedback and contributions are highly appreciated as we strive to create a powerful tool that complements AiiDA's capabilities.

[unkcpz]

The control page dependent on the aiidalab-widgets-base, uses widgets from AWB. It is still an open question whether AWB should be the dependency of aiidalab-home.
A problem I can foresee is since app share the environment with all kernel packages, install/update app will potentially break the control page.

We can also make AWB the core package as defined in aiidalab and prevent user to override with incompatible version. But this brings restriction to app developers.

The ultimate solution could be for every app has its own Python environment created when using aiidalab install by create an independent conda environment. I had played with this idea a bit but stuck at the jupyter kernel as well as the python interpereter's should be set into notebook metadata in advance which makes no sense for app developement. (I was thinking that the kernel can be set such as passing kernel name to URL, but didn't find anything solution like this.)

[danielhollas]

@yakutovicha thanks for this. 🗜️ I like the overall approach and design. I would suggest two things so we can move with an initial version of the control page:

1. Vendor in AWB dependencies. This will allow faster iteration on the design, and in the meantime we can figure out whether / how to add AWB to base image.
2. It is clear that the Process page will require the most discussion. Therefore I would omit it in the first version, so that it can be discussed separately and we can have something basic in the meantime.

Couple minor things / ideas...

1. Perhaps it would make sense to merge the Status and Daemon pages? I would imagine most of the time I'd like to see both at the same time to get the overall state of the system and be able to control it.
2. Provide buttons to increase / decrease number of daemon workers
3. Daemon page: Start / Stop buttons should be deactivated when the daemon is (not) running.
4. Profile page: "Make default" button should not be active for default profile.
5. Get rid of the alert prompt when exiting the page using the same trick as in other pages.

[danielhollas]

It would be great to re-enable the tests in CI before this PR is merged, per #161, and add basic tests for the new functionality.

[danielhollas]

I've merged in main with updated ruff that now checks jupyter notebooks. It reports a lint violation here:

control.ipynb:cell 3:19:25: B006 Do not use mutable data structures for argument defaults
   |
19 | def update_output(value={"new": "Group"}):
   |                         ^^^^^^^^^^^^^^^^ B006
20 |     if value["new"] in correspondance:
21 |         with output:
   |
   = help: Replace with `None`; initialize within function

[danielhollas]

@yakutovicha btw: I am going to be on the coding retreat next month and I've heard you'll be there as well? Perhaps this PR is a good candidate to work on, WDYT?

[danielhollas]

The control page dependent on the aiidalab-widgets-base, uses widgets from AWB. It is still an open question whether AWB should be the dependency of aiidalab-home.
A problem I can foresee is since app share the environment with all kernel packages, install/update app will potentially break the control page.

Just to continue this discussion, I think aiidalab-home cannot depend on AWB. All the apps (including home app) currently share a python environment, which means only one AWB version at a time. But we need to allow application to install different versions of AWB according to their needs, which will always run a risk of breaking home app. Thus I don't see any other way.

If there are components that are currently needed in AWB and needed here, I think the simplest thing is to copy them. I think in general we don't want to expand the home app too much, so the number of such components should not be high.

@edan-bainglass @superstar54 FYI about the status of this PR, please have a look at my earlier comment. #156 (comment)

I don't think it would be too much work to finish this PR in some minimal form and keep iterating on it in subsequent PRs.

[edan-bainglass]

@edan-bainglass @superstar54 FYI about the status of this PR, please have a look at my earlier comment. #156 (comment)

I don't think it would be too much work to finish this PR in some minimal form and keep iterating on it in subsequent PRs.

Thanks @danielhollas. @yakutovicha can we set up a meeting soon (maybe before the break) to discuss the current status and define a roadmap to a minimal release? All are welcome to join.

[yakutovicha]

Thanks @danielhollas. @yakutovicha can we set up a meeting soon (maybe before the break) to discuss the current status and define a roadmap to a minimal release? All are welcome to join.

I'm happy to meet, but frankly, the only time that might work for me is this Wed between 16 and 17. All the remaining days are booked.

[edan-bainglass]

@yakutovicha this is great! I've tested a few things here and there. Looks good to me. I made a few minor updates in a PR to this PR (#231). Other than that, my only concern is the obvious duplication of the process list:

• QE app provides rich process table
• I see that now we have a process list link at the top of the app
• And now the one introduced here

Ideally, I would lean toward using the richer process table, though in its current state, there are some QE-specific parts. Perhaps this can be left for a later PR if we come to an agreement. As for the one at the top of the app and the new one in the control panel, I would lean toward only the control panel one, making sure that it covers all the features the other provides.

If we agree to remove the top process list link in favor of the control panel one, I would recommend to elevate the remaining code setup link to a tile button (icon TBD). This of course can be done in a separate PR. Let's get all of this done on our upcoming coding day 🙂

[edan-bainglass]

A few other comments that I see now:

• The new process list behaves differently than the one in the QE app in the sense that clicking on the PK takes you to the process notebook, not the QE app loaded with the workflow. This is more general and should be kept this way. Might be a justification for keeping both lists. TBD
• In the daemon log under the daemon panel, we get the process report. However, we also get deprecation warnings. It would be nice to introduce the following (in a follow up PR):
  • symmetry with verdi process report, which does not expose deprecation warnings (actually, I'm wondering now if the daemon report covers more than just process reporting?)
  • dropdown selector of report level (default to report, but can be set to debug when the app misbehaves - very useful for assisting users)
  • a copy button (again, helpful when assisting users)
  • a reset/clear button?

[yakutovicha]

@edan-bainglass, thanks a lot for the feedback.

To simplify the review process, I will split this PR into smaller ones and will implement each feature one by one. This Particular PR I won't merge. This was really a vibe-coded prototype.

What concerns the process, I agree to keep only the one in the process table. @cpignedoli is eager to fix some functionality of the process list, so we can do that as well.

If you have time, feel free to open the PR to migrate the QE process list here - I checked it, and I like it a lot 👍