Skip to content

WIP: Convert the model ipython notebook to a script.

Gmodena requested to merge convert-ipynb-to-py into main

Created by: gmodena

This WIP adds the capability to extract a python script from the notebook, and save it as build artifact in Github. I open this PR to collect feedback / discuss possible approaches to releasing the notebook code in a format consumable in non-interactive mode.

The changes add a new py target to Makefile (make py) that runs nbconvert on the algo notebook. The resulting script can be executed on spark with the canonical spark-submit [...] algorithm.py.

I wanted to experiment with a flow that, after a successful build, publishes the resulting artifact to a generally available location.

I've been toying with CI and added make py to the github workflow. With this change we'd generate a new script at each PR / git push, that is available under the Artifacts pane of the Action UI.

Unfortunately there is no straightforward way to generate the artifact URL for remote consumption.

Next I want to explore the capabilities of our internal Gitlab (pipelines are enabled), and eventually move the codebase there.

Merge request reports