T292741 implement ci checks
This PR implements a set of basic CI checks as described in T292741.
Checks
A successful CI build should pass linting and testing steps.
Linting
Run lint manually inside a docker container with make lint.
The linting step currently treats errors as warnings: the build won't stop when failures are detected.
We lint with flake8 and the following (conservative) settings:
- McCabe complexity threshold: 10
- maximum allowed line length: 127 (Default PEP8: 79)
- check for syntax errors or undefined names
Tests
Run test manually, inside a docker container, with make test.
Test results and coverage are reported to standard output:
---------- coverage: platform linux, python 3.7.11-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------
spark/__init__.py 0 0 100%
spark/instances_to_filter.py 12 0 100%
spark/raw2parquet.py 21 21 0%
spark/schema.py 7 0 100%
spark/search_table.py 22 22 0%
spark/transform.py 42 17 60%
--------------------------------------------------
TOTAL 104 60 42%
======================== 2 passed, 9 warnings in 9.05s =========================Automation and Github mirroing
Gitlab Pipelines are currently unavailable in Wikimedia's instance. To automate CI, for demo purposes, we propose to mirror this repo to Github, and execute tests on a Github runner
./github/workflow/build.yml implements a CI workflow that runs on every push. Output is available at https://github.com/gmodena/wmf-platform-airflow-dags/actions/workflows/build.yml?query=branch%3AT292741-implement-ci-checks.