2 years ago

#51604

test-img

Savir

Airflow doesn't "see" that the underlying Kubernetes pod completed

We are using a hosted Airflow 1.10.2 in Google Composer 1.7.5 to launch jobs via the KubernetesPodOperator (tasks that will be run in a Kubernetes pod inside a worker cluster)

There has been several occasions in which the Kubernetes pod itself successfully completes, but Airflow doesn't "see" that the pod has completed (it doesn't get the memo), so Airflow thinks the pod is still running and doesn't move onto the next task.

We are planning to move to Composer 2 with Airflow 2.1.4, which I'm fairly confident it manages pods and communication with Kubernetes better, but...

... is there a "quick" tweak we can do? Even a link on how to start investigating would be helpful.

Thank you in advance.

airflow

google-cloud-composer

kubernetespodoperator

0 Answers

Your Answer

Accepted video resources