Pipeline Runner

Epic: As a data engineer I want to run repeatable analysis pipelines on a suitable runner.

UPDATE: Working with Prefect, any details you can find here: Pipeline-Automation-of-BFS-based-maps

Suggestions for pipeline runner:

• Airflow
• orchest
• Prefect

Sum up

All are open sourced, using Python framework.

pipeline runner	Passing data between tasks	Parametrization	Docs & community	UI	Hosting	Other
Airflow	X	✔️ (Jinja template)	✔️	✔️	✔️	Take a time to learn, not intuitive
Orchest	✔️	✔️ (grid search)	✔️	✔️	X	intuitive, jupyter notebook only, limited supported connections/staorage..
Prefect	✔️	✔️ (can be dynamic)	✔️	✔️	✔️	intuitive

Passing data between tasks is the most important criteria. Orchest offers only using Jupyter Notebbok. Therefore for Prefect seems to be good choice. It has derived from Airflow (or by people using it who saw the flows of Airflow). Docs are great, has a slack community, and UI as well for monitoring.

Proposal for schema/concept/skeleton of pipelines and how to use pipeline runner for it: concept_proposal

Workflow pipeline runner skeleton

Request comes, with information about type of map. Every map has a "definition" in library. Definition contains files such as:

• sql script,
• template.txt,
• default_dic.txt,
• etc.

Then the tasks in workflow pipeline runner should be:

• fetch the definition of request/map
• fetch data (sql query)
• datapackage pipeline (break in smaller subtasks: create spec, run, exporter)
• upload snapshot

Notes from reasearch

Airflow

First article to read: https://towardsdatascience.com/getting-started-with-airflow-using-docker-cd8b44dbff98

• to see in details why indeed Airflow might not be good solution and it leads me to research Prefect, read this article: https://towardsdatascience.com/is-apache-airflow-good-enough-for-current-data-engineering-needs-c7019b96277d
• not suitable for data pipelines (sharing data between steps/tasks)

For experiment part: Docker image: https://github.com/puckel/docker-airflow Airflow official Docker images: https://github.com/apache/airflow/blob/master/IMAGES.rst#airflow-docker-images
How to run docker image with local storage/mounted volume:

docker run -d -e AIRFLOW__CORE__FERNET_KEY='$FERNET_KEY -p 8080:8080 -v airflow-home:/usr/local/airflow/ puckel/docker-airflow webserver  

Use volumes: https://docs.docker.com/storage/volumes/

Init db:

docker exec -ti <coked id> bash
airflow@573ef8908e96:~$ airflow initdb 

Test task:

airflow test Helloworld task_5 2016-04-15

• Operators, Sensors and Hooks can be custom made, airflow documention you can find pre-defined ones. They are all derived from Base{Operator, Sensor, Hook}.
• option for parametrized notebooks

Articles:

• https://michal.karzynski.pl/blog/2017/03/19/developing-workflows-with-apache-airflow/#:~:text=When%20you%20reload%20the%20Airflow,DAG%20listed%20in%20Airflow%20UI.&text=In%20order%20to%20start%20a,the%20progress%20of%20the%20run.
• https://blog.insightdatascience.com/airflow-101-start-automating-your-batch-workflows-with-ease-8e7d35387f94

Orchest

Docs you can find here: https://orchest.readthedocs.io/en/latest/index.html.

• jupyter notebook only
• grid search
• limited support for types of databases, storage (has PostGRES, S3 bucket)
• suitable for data pipelines

Prefect

Article: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4

• good documentation (https://docs.prefect.io/)
• derived from the experience with Airflow
• intuitive how to use it (or at least more than Airflow)
• python based as well
• possible dynamic parametrization

Hello world looks like this:

from prefect import task, Flow, Parameter


@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))


with Flow("My First Flow") as flow:
    name = Parameter('name')
    say_hello(name)


flow.run(name='world') # "Hello, world!"
flow.run(name='Marvin') # "Hello, Marvin!"