Add a possibility to run system commands as a periodic task

[FSeidinger-XI]

Problem description

Currently there is no way to define a task that runs a system command as a periodic task.

Use cases

Many administrative tasks on a Linux based system is by running system processes or scripts via bash, that automate these tasks.

In my case I would like to add a script that takes the Borg backup directory, archives it into a tar file and ships the tar file to an S3 storage.

Currently the Borg backup directory is a mapped kubernetes volume, that can be lost at any time. In our infrastructure there is, besides databases and an on premise S3, no other persistent storage, that can be used as a secure backup location.

Using an external storage location is for security reasons not allowed. So that leaves me with S3.

It such a generic system task would be available it would be easy to map the script into the Weblate container (pod) and define a periodic task in Weblate.

Solution brainstorm

Suggested solution

Add a task to Weblate that takes a command and its arguments and spawns a process running that command line.

Describe alternatives you have considered

Alternatives

1. Add the crontab binary to the docker image and let the Weblate user define a job on the command line.
   This could be done by defining a volume holding the crontab and map it to the system for execution.
2. Add the rclone binary to the docker image and let, for my use case do rclone the shipping of backups. The configuration can also be mapped to the container.

Screenshots

No response

Additional context

No response

[nijel]

I think this is out of scope for Weblate.

Still, you can achieve periodic task execution using a few lines of Python code, but I've just realized that this part of the Docker container is not documented at all. #15494 adds the missing documentation.

[FSeidinger-XI]

I followed your example and put a python file unter /app/data/python/customize.

First of all, I had to create the directory /app/data/python/customize in my custom entry point, because it is not present in the container (using Weblate 5.12). I guess that this should not be necessary and that a django app should already exist there.

And my guess is also, that for this reason it gets not picked up.

Did I miss something here?

Hint: My colleague Markus Schmitt tried to get a support contract with Weblate but it is still not active because we need a CV from your employee.

[FSeidinger-XI]

It should not be necessary, as it is created upon container startup; see https://github.com/WeblateOrg/docker/blob/34c57128198d90a17d7525336288085d0c7ec0ef/start#L116-L122. But the entry point is responsible for this (because the file is in the volume, not shipped in the container), so if you override it, the automatic creation is gone.

There seems to be a chicken-and-egg-problem here.

If I copy my python file to /app/data/python/customize either in the Dockerfile or in the entrypoint, then the creation of the app does not get triggered. Because I delegate to the original entrypoint under /app/bin/start after my setup.

The code in the /app/bin/start file looks like that:

# Create fake Python app for customization
export WEBLATE_PY_PATH="/app/data/python/customize"
...

if [ ! -d "$WEBLATE_PY_PATH" ]; then
    echo "Creating $WEBLATE_PY_PATH"
    mkdir -p "$WEBLATE_PY_PATH/static"
    touch "$WEBLATE_PY_PATH/__init__.py"
    touch "$WEBLATE_PY_PATH/models.py"
fi

So if you copy a file into /app/data/python/customize, then the if statement will skip the app creation. I could copy the file afterwards into the volume and restart the container, but this is way from CI/CD pipeline friendly.

I therefore did something like that in my Dockerfile.

...

# Copy our custom task template
COPY build/modules ./build/modules

And the entrypoint.

# Create fake Python app for customization
# Hint, it is located in the /app/data volume
CUSTOMIZATION_DIR=/app/data/python/customize
CUSTOMIZATION_STATIC_DIR=$CUSTOMIZATION_DIR/static
if [ ! -d "$CUSTOMIZATION_STATIC_DIR" ]; then
  mkdir -p $CUSTOMIZATION_STATIC_DIR
  logger_info "Created $CUSTOMIZATION_STATIC_DIR"
fi

if [ ! -f "$CUSTOMIZATION_DIR/__init__.py" ]; then
  touch "$CUSTOMIZATION_DIR/__init__.py"
  logger_info "Created $CUSTOMIZATION_DIR/__init__.py"
fi

if [ ! -f "$CUSTOMIZATION_DIR/models.py" ]; then
  touch "$CUSTOMIZATION_DIR/models.py"
fi

if [ ! -f "$CUSTOMIZATION_DIR/ba_backup_task.py" ]; then
  # Copy our customization
  cp build/modules/ba_backup_task/ba_backup_task.py $CUSTOMIZATION_DIR
  logger_info "Copied our customization to $CUSTOMIZATION_DIR/ba_backup_task.py"
fi
logger_info "Fake python app in $CUSTOMIZATION_DIR is setup"

After that the directory layout looks like that in the container:

weblate@weblate-feature-backup-5fb8f7f987-2csbv:/app$ ls -al /app/data/python/customize/
total 4
drwxr-sr-x 4 weblate weblate  100 Sep 17 12:33 .
drwxr-sr-x 3 weblate weblate   23 Sep 17 12:33 ..
-rw-r--r-- 1 weblate weblate 2693 Sep 17 12:33 ba_backup_task.py
-rw-r--r-- 1 weblate weblate    0 Sep 17 12:33 __init__.py
-rw-r--r-- 1 weblate weblate    0 Sep 17 12:33 models.py
drwxr-sr-x 2 weblate weblate   68 Sep 17 12:34 __pycache__
drwxr-sr-x 2 weblate weblate    6 Sep 17 12:33 static

But the module is still not get picked up. Do I have to set an additional environment variable like WEBLATE_ADD_APPS?

It is placed on the data volume, so it persists; there is no need to populate it in the entrypoint. See https://docs.weblate.org/en/latest/admin/install/docker.html#docker-container-volumes

Sure, but this is not helpful in automated builds. For every build I create an own database and volumes. The cloud native style, so to say. There won't be a suitable way to write the file to the data volume after helm created the deployment in kubernetes.

[nijel]

You should put the files in your data volume. You can access it from outside or from a running container, and the data will persist there.

[FSeidinger-XI]

You should put the files in your data volume. You can access it from outside or from a running container, and the data will persist there.

They are in the data volume as you can see from my output. The output shows the file layout in the running container after startup with the mounted data volume.

And once again, the procedure to put the file after there after it is deployed from the build is not compatible with our kubernetes/helm CI/CD Pipeline. There is no easy way to do this from the deployment. Manual interventions after creating the pods is not an option here. If this is the only way I have to create an extra init container, to do the job.

That's why I prepare it in my entrypoint. At that time the data volume is mounted and installing the app it is done once from my entrypoint script. It does it the same way you do it in the default entrypoint. As you can see, I copied your code more or less one to one from the official startup script.

To be sure I created an extra app and installed it via WEBLAT_ADD_APPS. if doing so, the django systems throws an error, that the app customize is already installed. So the app itself is not the problem.

My guess is, that the problem is about the activation. I know from further investigations that the app is picked up, but the decorators @app.task and @app.on_after_finalize.connect may be the problem why my task is not installed.

Currently my task looks like that:

import os
import logging

import datetime
import tarfile
import tempfile

from celery.schedules import crontab

from weblate.utils.celery import app

logger = logging.getLogger("weblate.backup")
logger.info(f"Module ${__name__} activated")

@app.on_after_finalize.connect
def setup_periodic_tasks(sender, **kwargs) -> None:
    task_name = "ba-backup-task"
    sender.add_periodic_task(
        crontab(hour=3, minute=0), weblate_backup_task.s(), name=task_name
    )

    logger.info(f"Registered task ${task_name}")

@app.task
def weblate_backup_task() -> None:
    current_time = datetime.datetime.now()
    logger.info(f"Weblate backup task started at {current_time.isoformat(timespec="seconds")}")

    working_directory, archive_file_name, error = weblate_create_archive("/var/lib/weblate/backup")

    if (error):
        logger.error("Failed to create backup archive", exc_info=error)
        remove_working_directory(working_directory)
        return

    ship_backup_archive(archive_file_name)

def weblate_create_archive(backup_path: str) -> tuple[str, str, Exception | None]:
    logger.info("Creating backup archive")

    current_time = datetime.datetime.now()
    file_suffix = current_time.strftime("%Y_%m_%d_%H_%M_%S")
    logger.info(f"File suffix will be ${file_suffix}")

    # receives archive file name
    working_directory = None
    archive_file_name = None

    try:
        # Create a working directory
        working_directory = tempfile.TemporaryDirectory()
        logger.info(f"Created temporary working directory {working_directory}")

        archive_file_name = working_directory + "/" + "weblate_backup_" + file_suffix + "tar.gz"

        tar_file= tarfile.open(archive_file_name, "w:gz")
        logger.info(f"Created archive file {tar_file.name}")

        logger.info(f"Adding {backup_path} to the archive...")
        tar_file.add(backup_path)
        logger.info(f"Archiving {backup_path} done")

        tar_file.close()
    except Exception as error:
        return working_directory, archive_file_name, error

    return working_directory, archive_file_name, None

def ship_backup_archive(file_name: str):
    pass

def remove_working_directory(working_directory: str):
    if (working_directory):
        try:
            # first remove all files in the working directory
            for filename in os.listdir(working_directory):
                file_path = os.path.join(working_directory, filename)
                os.remove(file_path)

            # then remove working directory itself
            os.rmdir(working_directory)
        except OSError as error:
            logger.warning(f"Could not remove directory {working_directory}")

Sorry for the inconvenience, but I'm kinda clueless here, because I'm not that deep in the Weblate internals.

[FSeidinger-XI]

I'm one step further. I created a custom module, like described here (https://docs.weblate.org/en/latest/admin/customize.html#custom-module) and its tasks get registered.

I guess the difference is, that django only expects files named tasks.py. Any custom path name does not get activated.

And a hint on the documentation. The setup tools way with setup.py will not work, because the module setuptools is not part of the Weblate virtual environment.

Using a default python module with __init__.py though is sufficient.

[nijel]

Right, the documentation is not really following current best practices; #16254 updates the docs. For placing it in the folder, the packaging stuff is really not needed (as you've figured out).

I guess the difference is, that django only expects files named tasks.py. Any custom path name does not get activated.

Yes, that's where the task registration looks.