Skip to content

Commit 79bd38e

Browse files
committed
docs: add docs for conditional Dag loading
1 parent 17d4dc9 commit 79bd38e

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

Readme.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,27 @@ The dags directory contains the various workflows (Directed acyclic graphs)
1212

1313
Dags are automatically loaded from git repositories defined in the helm chart values under `airflow.dagProcessor.dagBundleConfigList`. These have been set up to fetch from the `dags/` directory in each repository, from the `develop` branch for `-dev` and `main` (`master` for older repos) for `-prod`.
1414

15+
### Conditional Dags
16+
17+
If you have a Dag that should be loaded or unloaded based on the environment, there is an `ENVIRONMENT` env var that is exposed as `dev`, `test`, or `prod`. This can be used in the Dag file itself to allow or disallow certain Dags from DagBundles being loaded. A further example of this can be found in the `cas-registration` repository, in the `dags/bc_obps_reset_data.py` file.
18+
19+
sample_conditional_dag.py
20+
21+
```python
22+
import os
23+
# ...other imports and definitions
24+
25+
# Define which environments this Dag loads in to:
26+
CURRENT_ENV = os.environ.get("ENVIRONMENT")
27+
ALLOWED_ENVIRONMENTS = ["dev", "prod"]
28+
29+
# ...Dag here
30+
31+
# This Dag will be found by the Git DagBundle process, but *not* loaded by airflow if not in the allowed environments:
32+
if CURRENT_ENV in ALLOWED_ENVIRONMENTS:
33+
dag_only_in_dev_and_prod()
34+
```
35+
1536
### Running tasks using the Kubernetes executor and KubernetesPodOperator
1637

1738
- The Kubernetes Executor allows us to run tasks on Kubernetes as Pods.

0 commit comments

Comments
 (0)