Skip to content

Add temp code to help with Airflow 2.6 upgrade

Mforns requested to merge add-temp-code-for-upgrade-2.6 into main

This is temporary code to help in the Airflow 2.5 -> 2.6 upgrade. Once all instances are running in Airflow 2.6, we can remove it. ISSUE: The AirflowSparkSubmitHook used in versions 2.5 and older stores the spark binary in a private field self._spark_binary. While the AirflowSparkSubmitHook used in 2.6 and newer stores the spark binary in the field self.spark_binary (without trailing underscore). FIX: The following code checks the Airflow version that is running and uses the corresponding spark binary field in AirflowSparkSubmitHook.

Related phab ticket: https://phabricator.wikimedia.org/T336286

Edited by Mforns

Merge request reports