You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: oci/airflow/documentation.yaml
+80-87Lines changed: 80 additions & 87 deletions
Original file line number
Diff line number
Diff line change
@@ -1,19 +1,28 @@
1
-
version: 1
1
+
version: 2
2
2
3
3
application: airflow
4
4
description: >
5
-
Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. This rock ("airflow") packages Airflow built from source on Ubuntu with Python and a wide set of providers.
5
+
Apache Airflow is a platform to programmatically author, schedule, and monitor workflows.
6
+
This rock packages Airflow built from source on Ubuntu 24.04 with Python and a wide set of
7
+
providers pre-installed (Amazon, GCP, Kubernetes, MySQL, Postgres, and more), so common
8
+
integrations work out of the box without requiring additional pip installation inside the container.
The rock uses [Pebble](https://documentation.ubuntu.com/pebble/), a lightweight service manager. Pebble is PID 1 in the container, and it supervises the Airflow service defined in the rock.
14
+
docker:
15
+
parameters: >-
16
+
-p 8080:8080
17
+
-e AIRFLOW_HOME=/opt/airflow
18
+
-v /path/to/dags:/opt/airflow/dags
19
+
-v /path/to/airflow.cfg:/opt/airflow/airflow.cfg
20
+
run_conclusion: |
21
+
Pebble (PID 1) launches the `airflow` service, which runs `airflow standalone`. This
22
+
initializes a local SQLite metadata database, creates a default admin user, and starts
23
+
the webserver and scheduler in a single process group (intended for development use).
24
+
The generated admin password is printed in the container logs.
25
+
Access the Airflow web UI at http://localhost:8080.
17
26
18
27
#### Airflow service
19
28
@@ -23,95 +32,79 @@ docker:
23
32
startup: enabled
24
33
```
25
34
26
-
`airflow standalone` initializes a local metadata DB, creates a default admin user, and starts the webserver, scheduler, and other core components in one process group (intended for development).
35
+
`airflow standalone` initializes a local metadata DB, creates a default admin user,
36
+
and starts the webserver, scheduler, and other core components in one process group
37
+
(intended for development).
27
38
28
39
#### Configuration & State
29
40
30
-
- Runtime state (AIRFLOW_HOME) lives at `/opt/airflow`. This includes `airflow.cfg`, `airflow.db` (Database by default for standalone), logs, and `dags/`.
41
+
- Runtime state (AIRFLOW_HOME) lives at `/opt/airflow`. This includes `airflow.cfg`,
42
+
`airflow.db` (Database by default for standalone), logs, and `dags/`.
31
43
- It is required to mount volumes to persist that data or to inject custom dags.
32
44
For example:
33
45
`-v ./dags:/opt/airflow/dags`
34
46
`-v ./airflow.cfg:/opt/airflow/airflow.cfg`
35
47
36
48
#### Providers
37
49
38
-
The rock is built from Airflow with many providers pre-installed (Amazon, GCP, Kubernetes, MySQL, Postgres, etc.), so common integrations work out of the box without requiring `pip install` inside the container.
50
+
The rock is built from Airflow with many providers pre-installed (Amazon, GCP,
51
+
Kubernetes, MySQL, Postgres, etc.), so common integrations work out of the box without requiring
52
+
`pip install` inside the container.
39
53
40
54
### Full Configuration Reference:
41
-
Airflow offers a large number of configuration options that can be set via environment variables or `airflow.cfg`.
42
-
For the complete list of configurable settings, consult the official [configuration reference](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html)
55
+
Airflow offers a large number of configuration options that can be set via environment variables
56
+
or `airflow.cfg`.
57
+
For the complete list of configurable settings, consult the official
Root directory for Airflow state inside the container (DB, logs, dags, config).
49
-
The rock sets this to /opt/airflow.
50
-
- type: -e
51
-
value: 'AIRFLOW__CORE__LOAD_EXAMPLES=false'
52
-
description: >
53
-
Whether to load the DAG examples that ship with Airflow. It’s good to get started, but you probably want to set this to False in a production environment
54
-
- type: -e
55
-
value: 'AIRFLOW__CORE__EXECUTOR=LocalExecutor'
56
-
description: >
57
-
The executor class that airflow should use. Choices include LocalExecutor, CeleryExecutor, KubernetesExecutor or the full import path to the class when using a custom executor.
SQLAlchemy connection string for the Airflow metadata DB. `airflow standalone` defaults to local SQLite, but for production you generally want Postgres or MySQL. Override this to point at an external DB.
For more information on available authentication managers and the configuration options they require, refer to the official [Airflow Auth Manager documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager/index.html)
66
-
- type: -p
67
-
value: '8080:8080'
68
-
description: >
69
-
Airflow Webserver UI (HTTP).
70
-
This is what you open in the browser.
71
-
These are documented in the [upstream Helm chart](https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html).
72
-
- type: -p
73
-
value: '8794:8794'
74
-
description: >
75
-
Triggerer log server port. Used for log streaming and collection by the Airflow triggerer.
76
-
Officially defined in the Airflow Helm Chart as `ports.triggererLogs`.
77
-
These are documented in the [upstream Helm chart](https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html).
78
-
- type: -p
79
-
value: '8793:8793'
80
-
description: >
81
-
Worker log server port. Used for log streaming and collection by Airflow workers in distributed setups.
82
-
Officially defined in the Airflow Helm Chart as `ports.workerLogs`.
83
-
These are documented in the [upstream Helm chart](https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html).
84
-
- type: -v
85
-
value: '/path/to/dags:/opt/airflow/dags'
86
-
description: >
87
-
Mount your DAGs from the host so you can iterate without rebuilding the rock.
88
-
Anything in /opt/airflow/dags will be auto-discovered.
89
-
- type: -v
90
-
value: '/path/to/logs:/opt/airflow/logs'
91
-
description: >
92
-
Persist task logs outside the container (useful for debugging and restarts).
(Optional) If you extend this rock and want extra Python deps at runtime,
102
-
you can mount and install them in an init hook / derived image.
103
-
- type: -e
104
-
value: 'AIRFLOW__API__SECRET_KEY=change-me'
105
-
description: >
106
-
Secret key used to run your api server. It should be as random as possible. However, when running more than 1 instances of the api, make sure all of them use the same secret_key otherwise one of them will error with “CSRF session token is missing”.
The maximum number of task instances allowed to run concurrently in each dag run.
61
+
config:
62
+
'`AIRFLOW_HOME`':
63
+
type: env
64
+
description: Root directory for Airflow state inside the container (DB, logs, DAGs, config).
65
+
default: /opt/airflow
66
+
'`AIRFLOW__CORE__LOAD_EXAMPLES`':
67
+
type: env
68
+
description: Load the example DAGs shipped with Airflow. Disable in production environments.
69
+
'`AIRFLOW__CORE__EXECUTOR`':
70
+
type: env
71
+
description: Executor class used by Airflow. Supported values include `LocalExecutor`, `CeleryExecutor`, `KubernetesExecutor`, or the full import path to a custom executor class.
72
+
'`AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`':
73
+
type: env
74
+
description: SQLAlchemy connection string for the Airflow metadata database. `airflow standalone` defaults to local SQLite; override for a production Postgres or MySQL database.
75
+
'`AIRFLOW__CORE__AUTH_MANAGER`':
76
+
type: env
77
+
description: Auth manager class. Refer to the [Airflow Auth Manager documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager/index.html) for available options and required configuration.
78
+
'`AIRFLOW__API__SECRET_KEY`':
79
+
type: env
80
+
description: Secret key for the API server. Use a random value; all instances in a cluster must share the same key.
81
+
'`AIRFLOW__API__BASE_URL`':
82
+
type: env
83
+
description: Base URL of the API server.
84
+
'`AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG`':
85
+
type: env
86
+
description: Maximum number of task instances allowed to run concurrently per DAG run.
87
+
'`-p 8080:8080`':
88
+
type: port
89
+
description: Expose port 8080 on the host for the Airflow web UI (HTTP).
90
+
'`-p 8794:8794`':
91
+
type: port
92
+
description: Expose port 8794 on the host for the Airflow triggerer log server.
93
+
'`-p 8793:8793`':
94
+
type: port
95
+
description: Expose port 8793 on the host for the Airflow worker log server.
96
+
'`-v <path>:/opt/airflow/dags`':
97
+
type: mount
98
+
description: Mount a local DAGs directory. Files are auto-discovered by the scheduler.
99
+
'`-v <path>:/opt/airflow/logs`':
100
+
type: mount
101
+
description: Persist task logs outside the container.
102
+
'`-v <path>:/opt/airflow/airflow.cfg`':
103
+
type: mount
104
+
description: Inject a custom `airflow.cfg` in place of the auto-generated one.
105
+
'`-v <path>:/requirements.txt`':
106
+
type: mount
107
+
description: (Optional) Mount additional Python dependencies to install at runtime.
0 commit comments