Skip to content

Draft: Add Jupyter notebook operator

Mforns requested to merge notebooks into main

This MR is the result of the spike to test the execution of Jupyter notebooks in Airflow using Skein and Papermill. It implements a NotebookOperator (inheriting from SimpleSkeinOperator) that executes a given notebook. The operator's parameters are the notebook itself, a packaged conda env to use as an environment for Skein, and the parameters for the notebook. This MR also implements a test DAG that uses the NotebookOperator.

This code is not final, it is intended to be reused when we prioritize implementing this functionality.

Merge request reports