T349285 DAG structure for REST API user agents
Contributor checklist
-
I have written tests for this DAG that will be merged into data-engineering/airflow-dags/tests/wmde -
I have locally ran the above tests and code quality checks as outlined in the tests section of the Airflow DAGs project readme -
I have tested the jobs for this DAG in my local database using the process defined in wmde/analytics/hql/airflow_jobs/wd_rest_api_user_agents/_test_monthly -
I have tested the included DAGs in my local database using the process outlined in TEST_AIRFLOW_DAGS.md and the test variable files provided for each DAG -
All Hive tables and HDFS directories that are needed by the included DAG jobs have been created and are accessible by the analytics-wmde
Airflow user-
Hive - wmde.wd_rest_api_user_agents
-
HDFS - Not needed
-
Description
Adds a DAG that updates a new table wmde.wd_rest_api_user_agents
on a monthly basis with the user agents and total requests for the Wikidata REST API. This allows WMDE product to easily check new users of the REST API and their behavior.
Test outputs
Tests are WIP