airflow.example_dags.example_dataset_alias¶
Example DAG for demonstrating the behavior of the DatasetAlias feature in Airflow, including conditional and dataset expression-based scheduling.
Notes on usage:
Turn on all the DAGs.
Before running any DAG, the schedule of the “dataset_alias_example_alias_consumer” DAG will show as “Unresolved DatasetAlias”. This is expected because the dataset alias has not been resolved into any dataset yet.
Once the “dataset_s3_bucket_producer” DAG is triggered, the “dataset_s3_bucket_consumer” DAG should be triggered upon completion. This is because the dataset alias “example-alias” is used to add a dataset event to the dataset “s3://bucket/my-task” during the “produce_dataset_events_through_dataset_alias” task. As the DAG “dataset-alias-consumer” relies on dataset alias “example-alias” which was previously unresolved, the DAG “dataset-alias-consumer” (along with all the DAGs in the same file) will be re-parsed and thus update its schedule to the dataset “s3://bucket/my-task” and will also be triggered.
Module Contents¶
Functions¶
|