Apache Airflow and Cloud Composer: Orchestrating dbt Jobs on Google Cloud

Apache Airflow is a popular open-source tool for orchestrating data workflows, including dbt jobs. It can coordinate tasks across different services. For example, a workflow can wait for a file to appear in Cloud Storage and then start a dbt transformation. Within dbt, dependencies between models, seeds, and other resources are defined using functions such as ref(). Selectors (including selectors.yml) let you define which parts of the dbt project to execute. Airflow starts the selected dbt command, and dbt handles the dependencies within that selection.

Jatka lukemista ”Apache Airflow and Cloud Composer: Orchestrating dbt Jobs on Google Cloud” →