What was the problem?

Our Research & Development department carries out various experiments, and in all of them, we need to create workflow orchestrations for solving tasks in game dev. Previously, we didn’t have any suitable tools with a sufficient number of built-in functions, and we had to orchestrate processes manually and entirely from scratch every time. This led to difficulties with dependencies and monitoring when building complex workflows. We needed a tool that would provide a more centralized approach so that we could see all the logs, the number of retries, and the task performance time. The most important thing that we lacked was the ability to backfill historical data and restart failed tasks.

How did Apache Airflow help to solve this problem?

Apache Airflow offers lots of convenient built-in solutions, including integrative ones. The DAG model helps us avoid errors and follow general patterns when building workflows. In addition, this platform has a large community where we can find plenty of sensors and operators that cover 90% of our cases. This allows us to save ourselves loads of time.

What are the results?

Thanks to Apache Airflow, we’ve managed to simplify the process of building complex workflows. Many procedures that are so important for game development, such as working with the churn rate, processing messages to the support team, and sorting bank offers, now run efficiently, and all issues are resolved centrally. In addition, Apache Airflow is widely used in the industry, allowing us to onboard new people to our team more quickly and smoothly.