Skip to content

T275162 enable spark metrics collection

Gmodena requested to merge github/fork/gmodena/T275162-enable-spark-metrics-collection into main

Created by: gmodena

This PR improves spark session creation in notebooks and scripts, enables control for metric collection, and aligns with resource utilisation patterns defined by Analytics.

Changelog:

  • For notebook we init SparkSession using the wmfdata library.

  • For scripts, the recommended way is to specify a spark.properties file.

  • A metrics.properties.template file is also provided, that shows an example of how to plug in custom configuration files.

Merge request reports