How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie

https://is1-ssl.mzstatic.com/image/thumb/Podcasts221/v4/03/b0/38/03b03894-a103-21c1-aa77-fcd8b0d2ff65/mza_4276056948887286043.jpg/600x600bb.jpg

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer

80 episodes

5 days ago

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/

Technology

RSS

All content for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI is the property of Astronomer and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

https://files.cohostpodcasting.com/cohost/dbcdfaae-58e3-4bce-9cf7-fd6dbc27a8f5/shows/50fd8116-be73-4852-be37-94d8bc0082fa/episodes/d87fa102-b114-4740-b847-92c891b3361c/8184098455.jpg

How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

23 minutes 10 seconds

6 days ago

How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie

Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability.

In this episode, Anja MacKenzie, Expert for Cheminformatics at Covestro, explains how her team uses Airflow and Kubernetes to create a shared, self-service platform for computational chemistry.

Key Takeaways:

00:00 Introduction.

06:19 Custom scripts made sharing and reuse difficult.

09:29 Workflows are manually triggered with user traceability.

10:38 Customization supports varied compute requirements.

12:48 Persistent volumes allow tasks to share large amounts of data.

14:25 Custom operators separate logic from infrastructure.

16:43 Modified triggers connect dependent workflows.

18:36 UI plugins enable file uploads and secure access.

Resources Mentioned:

Anja MacKenzie

https://www.linkedin.com/in/anja-mackenzie/

Covestro | LinkedIn

https://www.linkedin.com/company/covestro/

Covestro | Website

https://www.covestro.com

Apache Airflow

https://airflow.apache.org/

Kubernetes

https://kubernetes.io/

Airflow KubernetesPodOperator

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html

Astronomer

https://www.astronomer.io/

Airflow Academy by Marc Lamberti

https://www.udemy.com/user/lockgfg/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_GammaCatchall_NonP_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Gamma&utm_content=deal4584&utm_term=_._ag_169801645584_._ad_700876640602_._kw__._de_c_._dm__._pl__._ti_dsa-1456167871416_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21341313808&gbraid=0AAAAADROdO1_-I2TMcVyU8F3i1jRXJ24K&gclid=Cj0KCQjwvJHIBhCgARIsAEQnWlC1uYHIRm3y9Q8rPNSuVPNivsxogqfczpKHwhmNho2uKZYC-y0taNQaApU2EALw_wcB

Airflow Documentation

https://airflow.apache.org/docs/

Airflow Plugins

https://airflow.apache.org/docs/apache-airflow/1.10.9/plugins.html

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow