Skip to content

Bump up memoryOverhead for edit_hourly_dag SparkSqlOperator

Mforns requested to merge bump-up-edit-hourly-dag into main

The SparkSqlOperator task process_edit_hourly from the DAG edit_hourly has failed recently with:

23/06/03 01:30:52 ERROR YarnScheduler: Lost executor 61 on an-worker1141.eqiad.wmnet: Container killed by YARN for exceeding physical memory limits. 8.9 GB of 8.8 GB physical memory used. Consider boosting spark.executor.memoryOverhead.

This MR bumps up the memoryOverhead for that task.

Merge request reports