Upgrade image-suggestions to Spark3.
(Marking as draft since it depends on repos/structured-data/image-suggestions!6 (merged).)
In this MR we upgrade the platform_eng
Airflow instance to default to Spark3, and we upgrade the image-suggestions
job (which is currently the only job on this instance.).
The airflow job runs all the Spark jobs successfully. I did not run the refinery jobs which move data fro Hive to Cassandra, as these point to the production cassandra instance. Once we merge this MR, I'll wait for and monitor the workflow for a full regular run.