Home / Uncategorized / Migrating ELT cron jobs to Apache Airflow DAGs
single

Migrating ELT cron jobs to Apache Airflow DAGs

We have migrated some ELT jobs to Apache Airflow scripts.
These scripts move Oracle DB data to Google Bigquery.
Steps required
Build apache airflow in docker (used puckel/docker-airflow image)
As root user update docker image
As airflow user install Java 1.8 + (required for running Talend ETL tools)
Convert the export bash script to a DAG
Enjoy the scheduling, retries and logging provided by the airflow scaffolding.
More details will be provided in follow up posts

install docker

docker pull puckel/docker-airflow                                                                                                                                    

 docker run -d –name airflow -p X8080:8080 puckel/docker-airflow webserver 

Leave a Reply

Your email address will not be published. Required fields are marked *