Merge requests · repos / data-engineering / dumps / mediawiki-content-dump

Fix comment for reconcile_emit_dt column on create-wmf_dumps_wikitext_inconsistent_rows.hql

!47 · created Nov 19, 2024 by Xcollazo
updated Nov 19, 2024
Bump workflow_utils to v0.19.0 to use miniforge

!45 · created Nov 06, 2024 by Btullis
- Merged
- 6
- Approved
updated Nov 18, 2024
Job to submit reconcile events to Kafka via wmf-event-stream

!46 · created Nov 07, 2024 by Xcollazo
- Merged
- 17
updated Nov 18, 2024
T347524 add integration tests to the pyspark jobs 2 of 2 checklist items completed

!17 · created Oct 16, 2023 by Jennifer Ebe
- Merged
- 35
- Approved
updated Nov 12, 2024
Push down earliest rev_dt per wiki on the revision level MERGE INTO

!42 · created Oct 28, 2024 by Xcollazo
- Merged
- 9
- Approved
updated Nov 01, 2024
Workaround blank rev_timestamps when doing consistency check.

!44 · created Oct 30, 2024 by Xcollazo
- Merged
- Approved
updated Oct 31, 2024
Go back to copy-on-write for MERGE INTOs and DELETEs.

!43 · created Oct 30, 2024 by Xcollazo
- Merged
- 1
updated Oct 31, 2024
Allow processing a whole day.

!41 · created Oct 23, 2024 by Xcollazo
- Merged
- 1
- Approved
updated Oct 28, 2024
Fix return bug.

!39 · created Oct 21, 2024 by Xcollazo
updated Oct 22, 2024
Avoid commit conflicts on wikitext_inconsistent_rows.

!38 · created Oct 21, 2024 by Xcollazo
- Merged
- 1
updated Oct 21, 2024
Use adaptive mw spark dataframe function

!37 · created Oct 11, 2024 by Milimetric
- Merged
- 2
- Approved
updated Oct 21, 2024
Use new partitioning and bloom filter strategy for ingestion.

!36 · created Oct 02, 2024 by Xcollazo
- Merged
- 8
- Approved
updated Oct 21, 2024
Use new mediawiki-jdbc datasource

!34 · created Sep 14, 2024 by Milimetric
- Merged
- 13
- Approved
updated Oct 02, 2024
Add jobs to ingest deletes and moves.

!35 · created Sep 17, 2024 by Xcollazo
- Merged
- 24
updated Sep 20, 2024
Modify visibility SQL to account for canary events.

!19 · created Oct 26, 2023 by Xcollazo
- Merged
- 2
updated Sep 19, 2024
Improve User-Agent as per WMF guidelines.

!33 · created Aug 23, 2024 by Xcollazo
- Merged
- Approved
updated Aug 23, 2024
Bump commit.retry.num-retries for wikitext_inconsistent_rows.

!31 · created Aug 12, 2024 by Xcollazo
updated Aug 12, 2024
Job to do consistency check

!30 · created Jul 22, 2024 by Xcollazo
- Merged
- 18
- Approved
updated Jul 31, 2024
Clarify column descriptions of *_is_visible columns

!28 · created Jun 20, 2024 by Ottomata
- Merged
- 3
- Approved
updated Jul 24, 2024
Add notebook from Spike: Figure feasibility to emit (wiki_db, revision_id) pairs

!29 · created Jun 21, 2024 by Xcollazo
updated Jun 21, 2024

Admin message

Admin message

Admin message