Skip to content

Add DAG to load MediaWiki snapshot config to Cassandra

Mforns requested to merge cassandra-load-mediawiki-snapshot-config into main

Adds a DAG to the analytics project, mediawiki namespace, to load the latest MediaWiki History snapshot value to a Cassandra table that serves as config for AQS.

This change (together with the refinery patch https://gerrit.wikimedia.org/r/c/analytics/refinery/+/989558) is the first of 2, to solve https://phabricator.wikimedia.org/T352948. The other one will add a caching mechanism to AQS 2.0, so that the MediaWiki snapshot that is set in the Cassandra AQS config table is automatically picked up and used as base data.

NOTE: The Druid sensor that we had was not used by any DAG so far. It worked, but it required a list of intervals. This change also refactors the Druid sensor to require start and end timestamps instead, since this is more flexible and convenient.

Edited by Mforns

Merge request reports