Skip to content

Clean up and fix Spark settings for section_topics

Xcollazo requested to merge fix-skein-settings-for-section-topics into main

(Still need to test these changes. Do not merge.)

While developing the section_topics pipeline, we forgot that, for skein to honor CPU and RAM settings, we cannot use Spark conf flags.

This MR fixes this, and also refactors other confs for readability.

Edited by Xcollazo

Merge request reports