SwarmPAL / Timeline Viewer integration

[eelcodoornbos]

The Space Weather Timeline Viewer is a web application, to interactively visualise time series data from HAPI servers. Part of the development has been and is supported by the Swarm DISC.

The Timeline Viewer ecosystem consists of:

• the Svelte/Javascript web application;
• a HAPI server implementation written in Python that can work with Flask or Apache2 web servers and sqlite3 or PostgreSQL databases (the combination of Flask and sqlite3 allow for the quick setup of a personal HAPI server on a user's notebook or desktop computer);
• The swxtools Python library containing classes and functions for retrieving and processing data as well as evaluating models and offering some utility functions;
• A set of HAPI ingest scripts to tie the downloading, processing and uploading to the HAPI server together.

All of the above links lead to the project's open source repositories.

I have identified a couple of avenues for integration, that could be further explored:

• There is overlap in functionality between swxtools and SwarmPAL, but SwarmPAL seems to be at a higher level of adhering to software development and maintenance standards. So the Timeline Viewer ecosystem could certainly benefit from integration of SwarmPAL, replacing (some of) the swxtools code, where functionality overlaps (most notably retrieving and processing of Swarm data). But there is also overlap with tools in the PyHC ecosystem, like pysat. So the best way forward here perhaps needs to be discussed in this wider context?
• There might also be functionality in swxtools that is not yet readily available in SwarmPAL (as far as I am aware), such as the ability to do some orbit coordinate conversions and model evaluations (IRI, MSIS, Swarm-VIP-Dynamic) along satellite trajectories. The current swxtools codebase could perhaps be a starting point to contribute SwarmPAL toolboxes.
• There are also interesting features in SwarmPAL for obtaining processed timeseries data that could benefit from availability on HAPI servers and visualisation in timeline viewer, offering ease of use, interactivity, and visualisation in combination with other (non-Swarm) data sets. For this to work, it would be necessary to implement an interface between the SwarmPAL processed output and a HAPI server.
• Timeline Viewer also offers the possibility to browse through image sequences and spectogram/keogram-type images. This works by omitting all (axis) annotations and margins from plotted output (this will be provided by the timeline viewer web app), and encoding timing and colour-scale information in the PNG output filename/path. In addition, some work has been done to keep the PNG files as small as possible, so they can be served over the web extremely quickly. SwarmPAL could perhaps be extended to more easily create this type of output and make it available in timeline viewer.

[smithara]

I'll respond now before it already gets to 2026 🙃

• Overlap in functionality between swxtools, SwarmPAL, and more (pyspedas, geospacelab, speasy, ... should also be mentioned here)

  • Indeed it was not an easy decision to start SwarmPAL as a Swarm-specific project rather than building on something more generalised. One of the key differences with SwarmPAL is that data access is designed around web services first (VirES & HAPI) whereas pyspedas|geospacelab|pysat are more designed around file access (but noted that pyspedas now includes access via hapi & vires!). Another aspect is that I want to keep the scope of SwarmPAL open to allow for Solid Earth applications and not lock completely into the Heliophysics environment. But yes - the idea of core functionalities that can be reused in more places needs a wider context discussion.

• Possibility of porting parts of swxtools into SwarmPAL

  • It's a great idea and could be a way to move those parts toward something more community-owned (which could be to SwarmPAL or indeed PyHC packages). So far I have just considered SwarmPAL to house code for "on-demand Swarm products" and whatever is needed to support that so I would be cautious about including more broadly applicable utilities - in those cases I would prioritise moving them to a PyHC core package if one is appropriate.

• Outputting SwarmPAL data via HAPI

  • I've thought of a few approaches to this but I'm lacking experience about what is sensible here, and definitely worth checking up if there are similar activities in the HAPI ecosystem.
    • A while ago I had a short conversation with Martin about installing SwarmPAL into VirES, so that "on-demand / virtual" datasets can be retrieved over the VirES/HAPI operational server just like they are normal datasets (where we could add caching etc). It seems to be plausible and a rational way to avoid having an extra service to maintain.
    • Add a HAPI server in SwarmPAL directly
    • Build an interface with another HAPI server like yours
    • Just make sure that a collection of SwarmPAL-generated data files can be (re-)generated and stored in a standard way so that a generic HAPI server can serve them.

• Image sequences

  • Yes - we're going to have a look at this in connection with consolidating a "quicklook" system for SwarmPAL outputs, which you could then hook into with some scripts to generate the set of PNGs named appropriately. I'll let you know when we have something usable there so we can iterate on it together.
  • Many missions and services have stacks of plots that can be rapidly browsed through (and of course Timeline Viewer extends this concept beautifully!). I think we can consider the ability to generate these flexibly as a core feature of SwarmPAL.

[smithara]

Timeline Viewer 2 will use Zarr data rather than PNG, so that the colour maps can be controlled later in the viewer

See e.g.:
https://gitlab.com/KNMI-OSS/spaceweather/timeline_viewer_2/-/blob/main/docs/zarr-ingest-pipeline-guide.md

Maybe SwarmPAL can have post-processing steps that allows outputting data into certain formats, such as that needed by Timeline Viewer

There might be some useful patterns to look at in the data ingest pipelines

[eelcodoornbos]

First, just a quick note to say that I have just renamed the new pipeline repository, so your last link is broken (sorry!) and should now point here.

The (still developing) idea of my new protocol-based pipeline structure for feeding into timeline viewer is that it consists of three parts:

• sources (for swarm-diss, NOAA, NASA, etc)
• one or more processors (pd.DataFrame and/or xr.Dataset to a new pd.DataFrame or xr.Dataset - for adding fields, transforming data)
• writers (DataFrame to parquet or Dataset to Zarr on the filesystem or cloud storage).

The key here is that writers can be configured to write straight to the HAPI/Zarr datasets on our server in the cloud, requiring cloud authentication, so it is not so useful to pull that functionality into SwarmPAL. I think SwarmPAL integration would fit very well on the other hand in the "processing" part of the pipelines (and perhaps also as a ViRES/HAPI "source"). A SwarmPAL processor that produces for example spectrogram image-type data from high-res Swarm data, and returns it in an xArray structure, can be connected to a "writer", making it available for Timeline Viewer.

I think only small wrappers (or perhaps wrapper-processors) might be required to use SwarmPAL's "toolboxes" to do the work of processors in my pipelines. The wrappers would get DataFrame/DataSet data into/out of SwarmPAL's DataTree structure, pass on toolbox options from YAML files, and perhaps add some necessary parameter renaming/metadata at the end to feed into my pipeline's Zarr or Parquet writers and then the toolbox output would be available on our server compatible with Timeline Viewer.

It might also be of interest that my department's colleagues working on the EarthCARE mission (also and Earth Explorer, like Swarm) have deployed dedicated Timeline Viewer and HAPI server instances on ESA EO's MAAP infrastructure.

See:

• https://portal.maap.eo.esa.int
• https://earth.esa.int/eogateway/tools/earthcare-timeline-viewer

[smithara]

Thank you for the link to the EarthCARE Timeline Viewer. I had seen it before and pleased to see it works nicely!

It's still very convoluted to navigate around the MAAP systems (there are a lot of different documentation sites...) but I went poking around again, wondering what pipelines look like there. All I found so far is running a python script in a Docker container to act on some mounted data files as part of the "PAL" system which seems to be mostly about orchestrating such containerised processors. Another main aspect of MAAP is data access via STAC catalogue.

Anyway I have a long term plan to get SwarmPAL into MAAP (whatever that might mean!) so want to align in some way. Your description of pipeline structure seems sensible so I wonder if your EarthCARE colleagues are looking at something similar and could push the concept within MAAP.

We'll focus on SwarmPAL providing the "processing" part, as you say (xarray structure -> processor -> xarray structure), so we should make it easy to write a small wrapper around it. Any pipeline/application could then use SwarmPAL in this way (whereas "sources" and "writers" are more subject to change), while they can also use the SwarmPAL VirES/HAPI source as well where it makes sense.