diff --git a/src/concepts/pipeline/glean_data.md b/src/concepts/pipeline/glean_data.md index bba03a62c..d60f4735e 100644 --- a/src/concepts/pipeline/glean_data.md +++ b/src/concepts/pipeline/glean_data.md @@ -76,3 +76,5 @@ In this case the event's `category` would be `engine_tab` and its name would be You can use the record's `timestamp` and `extra` fields to get the event's timestamp and specifics related to the event. For a complete example, see ["event metrics" under Accessing Glean Data](../../cookbooks/accessing_glean_data.md#event-metrics). + +For most querying needs, the generated `events_stream` table (e.g. `{app}.events_stream`) is now recommended over unnesting the `events` column directly: it flattens each event into its own row with a stable `event_id`, a derived `event_timestamp`, and `event_extra` as a JSON object. See the [FxA datasets](https://docs.telemetry.mozilla.org/datasets/fxa.html) page for an example. diff --git a/src/concepts/sql_style.md b/src/concepts/sql_style.md index 6281cb1c9..5b7efe926 100644 --- a/src/concepts/sql_style.md +++ b/src/concepts/sql_style.md @@ -446,5 +446,9 @@ LIMIT 10 This document was heavily influenced by https://www.sqlstyle.guide/ +For additional conventions on query file structure, naming, incremental +queries, and other practices not covered here, see the +[bigquery-etl Recommended Practices](https://mozilla.github.io/bigquery-etl/reference/recommended_practices/) guide. + Changes to the style guide should be reviewed by at least one member of both the Data Engineering team and the Data Science team. diff --git a/src/contributing/index.md b/src/contributing/index.md index 24677dd9c..5a89d1be6 100644 --- a/src/contributing/index.md +++ b/src/contributing/index.md @@ -24,6 +24,8 @@ This documentation is stored as [CommonMark Markdown](https://commonmark.org/hel [`data-docs` repository](https://github.com/mozilla/data-docs) on GitHub. To build a local copy, fork the repository and check out your copy. Then, [see the README](https://github.com/mozilla/data-docs/blob/master/README.md#building-the-documentation) for up-to-date information on how to build the documentation. +Before submitting a pull request, run the spell checker (`scripts/spell_check.sh`, which uses `markdown-spell-check`) and add any necessary exceptions to the `.spelling` file. + ## Adding a new article You should read the [style guide](./style_guide.md) before adding a new article: it will help you write material that is more useful and fits cohesively into the rest of the documentation. diff --git a/src/cookbooks/analysis/glean_dictionary.md b/src/cookbooks/analysis/glean_dictionary.md index 74228de7c..27a68af0c 100644 --- a/src/cookbooks/analysis/glean_dictionary.md +++ b/src/cookbooks/analysis/glean_dictionary.md @@ -14,6 +14,10 @@ After doing so, you can search for metrics by name (e.g.: `addons.enabled_addons After selecting a metric, you can get more information on it including a reference to its definition in the source code as well as information on how to get the data submitted by this probe in some of our data tools like [STMO], [Looker], and [GLAM]. +## Using the Glean Dictionary with AI assistants + +Because the Glean Dictionary's web UI is a single-page application that loads metrics dynamically via JavaScript, AI assistants and other automated tools that fetch the page directly only see an empty HTML shell. For this reason, the Glean Dictionary also exposes a [Model Context Protocol] (MCP) server at `https://dictionary.telemetry.mozilla.org/mcp`, providing structured, direct access to the same metadata. It offers five tools: `list_apps`, `get_app`, `search_metrics`, `get_metric`, and `get_ping`. + ## Common Questions ### How can I go from a metric to querying it in BigQuery? @@ -30,3 +34,4 @@ For several examples of this along with a more complete explanation, see [Access [looker]: ../../introduction/tools.md#looker [glam]: ../../introduction/tools.md#glean-aggregated-metrics-dashboard-glam [accessing glean data]: ../accessing_glean_data.md +[model context protocol]: https://github.com/mozilla/glean-dictionary/blob/main/docs/mcp-server.md diff --git a/src/cookbooks/data_monitoring/bigquery_etl_integration.md b/src/cookbooks/data_monitoring/bigquery_etl_integration.md index c82c042f6..276685ff2 100644 --- a/src/cookbooks/data_monitoring/bigquery_etl_integration.md +++ b/src/cookbooks/data_monitoring/bigquery_etl_integration.md @@ -17,7 +17,7 @@ To indicate whether a failing check should block any downstream Airflow tasks, a ## Bigconfig -Additional and custom monitors can be defined in a [Bigconfig](https://docs.bigeye.com/docs/bigconfig#example-template) `bigconfig.yml` file that is stored in the same directory as the table query. Bigconfig allows users to deploy other pre-defined monitors, such as row counts or null checks on a table or column level. +Additional and custom monitors can be defined in a [Bigconfig](https://docs.bigeye.com/docs/bigconfig#example-template) `bigconfig.yml` file that is stored in the same directory as the table query. Bigconfig should be applied to tables, not views, since default metrics like volume and freshness rely on insertion metadata that views lack. Bigconfig allows users to deploy other pre-defined monitors, such as row counts or null checks on a table or column level. The file must start with `type: BIGCONFIG_FILE` and defines `table_deployments`, each specifying a `collection.name` and one or more `deployments` (identified by `fq_table_name`); monitors are declared via `table_metrics` (e.g. `saved_metric_id: volume`) and, for column-level checks, `columns[].metrics` (e.g. `saved_metric_id: is_not_null`). ## Custom SQL Rules @@ -74,3 +74,5 @@ To generate a `bigconfig.yml` file with the default metrics when monitoring is e The created file can be manually edited. For tables that do not have a `bigconfig.yml` checked into the repository, the file will get generated automatically before deployment to Bigeye. Files only need to be checked in if there are some customizations. The deployment of Bigconfig files runs automatically as part of the [artifact deployment process](../../concepts/pipeline/artifact_deployment.md), after tables and views have been deployed. + +Bigconfig changes should not be applied locally (e.g. via `bigconfig apply`) against an incomplete file set, since this can delete existing Bigeye metrics. Only the `publish_bigeye_monitors` task in the artifact deployment DAG should push Bigconfig changes to Bigeye. diff --git a/src/cookbooks/new_ping.md b/src/cookbooks/new_ping.md index fbd77942a..edd7b598a 100644 --- a/src/cookbooks/new_ping.md +++ b/src/cookbooks/new_ping.md @@ -7,6 +7,8 @@ to find out. ping document type. To add a histogram, scalar, or event collection to Firefox, please see the documentation on [adding a new probe](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html). +Note also that legacy Telemetry collection APIs (including custom pings) are now deprecated for +Firefox Desktop in favor of Glean. ## Write Your Questions diff --git a/src/cookbooks/operational/connecting_external_data_bigquery.md b/src/cookbooks/operational/connecting_external_data_bigquery.md index 01188e114..900a60ddd 100644 --- a/src/cookbooks/operational/connecting_external_data_bigquery.md +++ b/src/cookbooks/operational/connecting_external_data_bigquery.md @@ -56,6 +56,6 @@ fields: 8. Open a pull-request against [bigquery-etl](https://github.com/mozilla/bigquery-etl) and tag someone for review. - Due to the permissions required, it is not possible to build and QA this table before it is deployed after merging. 9. Once the PR has been reviewed and merged, the table will be available the next day in BigQuery. - - If the table should be made available immediately, then go to the [`bqetl_artifact_deployment` Airflow DAG](https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid) and clear the `publish_new_tables` task. This might need to be done by a data engineer or someone who has permissions to trigger and clear tasks in Airflow. The table will be available as soon as the task finishes. + - If the table should be made available immediately, then go to the [`bqetl_artifact_deployment` Airflow DAG](https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid) and clear the `publish_tables_and_views` task (this task replaced the earlier `publish_new_tables`/`publish_views` tasks). This might need to be done by a data engineer or someone who has permissions to trigger and clear tasks in Airflow. The table will be available as soon as the task finishes. For confidential data it is generally recommended to add these configurations to [private-bigquery-etl](https://github.com/mozilla/private-bigquery-etl). The process and configurations are the same, the only difference is the repository which is not publicly accessible. diff --git a/src/datasets/bigquery/clients_last_seen/reference.md b/src/datasets/bigquery/clients_last_seen/reference.md index a0c0d5217..4cf4d2180 100644 --- a/src/datasets/bigquery/clients_last_seen/reference.md +++ b/src/datasets/bigquery/clients_last_seen/reference.md @@ -80,9 +80,14 @@ Please see this [section](../../../concepts/segments.md) for descriptions regard - `days_visited_1_uri_bits`: Each bit field represents if a client browsed at least 1 URI on that day. - `days_since_visited_1_uri`: Number of days since the client browsed at least 1 URI. +- `days_visited_1_uri_normal_mode_bits`: Each bit field represents if a client browsed at least 1 URI outside of Private Browsing Mode on that day. +- `days_since_visited_1_uri_normal_mode`: Number of days since the client browsed at least 1 URI outside of Private Browsing Mode. +- `days_visited_1_uri_private_mode_bits`: Each bit field represents if a client browsed at least 1 URI in Private Browsing Mode on that day. Only available for Firefox 84 and later. +- `days_since_visited_1_uri_private_mode`: Number of days since the client browsed at least 1 URI in Private Browsing Mode. Only available for Firefox 84 and later. - `days_interacted_bits`: Each bit field represents if a client had at least 1 active tick on that day. This is derived from the `active_hours_sum` in `clients_daily`. - `days_since_interacted`: Number of days since the clients had at least 1 active tick. +- `days_since_qualified_use_v1`: Number of days since the client both browsed at least 1 URI and had at least 1 active tick (computed as `days_visited_1_uri_bits & days_interacted_bits`). This is a view-computed field, not a stored bits column. - `days_had_8_active_ticks_bits`: Each bit field represents if a client had at least 8 active ticks on that day. This can be used to approximate the threshold of 1 URI, and is useful for determining activity for clients using Private Browsing Mode where URI counts are not recorded. diff --git a/src/datasets/experiment_monitoring.md b/src/datasets/experiment_monitoring.md index 64ca56209..5f111f6cb 100644 --- a/src/datasets/experiment_monitoring.md +++ b/src/datasets/experiment_monitoring.md @@ -26,6 +26,10 @@ Experiment monitoring datasets are designed to power dashboards, such as [Experi | `mozdata.telemetry.experiment_cumulative_search_count` | Cumulative number of searches by clients enrolled in experiments | | `mozdata.telemetry.experiment_cumulative_search_with_ads_count` | Cumulative number of searches with ads by clients enrolled in experiments | +## Experiment crash rate data + +`moz-fx-data-shared-prod.telemetry.experiment_crash_rates_live` provides crash-rate monitoring for clients enrolled in experiments, combining per-app `experiment_crash_events_live_v1` materialized views with historical data from `telemetry_derived.experiment_crash_aggregates_v1`. Values are aggregated by experiment, branch, crash process type and crash signature. + ## Derived dataset The derived views have the following schema: diff --git a/src/datasets/glam.md b/src/datasets/glam.md index 686c9c917..d0a48c20a 100644 --- a/src/datasets/glam.md +++ b/src/datasets/glam.md @@ -27,6 +27,8 @@ Each link below points to the dataset's page on [Mozilla's Data Catalog](https:/ ### Firefox Desktop (Legacy Telemetry) +> **_NOTE:_** Firefox Desktop Legacy Telemetry support in GLAM is deprecated; new data stopped being aggregated for these tables on January 26, 2026. + - [`moz-fx-data-shared-prod.glam_etl.glam_desktop_nightly_aggregates`]() - [`moz-fx-data-shared-prod.glam_etl.glam_desktop_beta_aggregates`]() - [`moz-fx-data-shared-prod.glam_etl.glam_desktop_release_aggregates`]() @@ -45,8 +47,8 @@ Most of the GLAM ETL is scheduled to run daily via Airflow. There are separate E - [Firefox Desktop on Glean (release)](https://workflow.telemetry.mozilla.org/dags/glam_fog_release/grid) - Runs weekly - Second "half" of release ETL is processed -- [Firefox Desktop legacy](https://workflow.telemetry.mozilla.org/dags/glam/grid) - - Runs daily +- [Firefox Desktop legacy](https://workflow.telemetry.mozilla.org/dags/glam/grid) (deprecated) + - New aggregations stopped on January 26, 2026 - [Firefox for Android](https://workflow.telemetry.mozilla.org/dags/glam_fenix/grid) - Runs daily diff --git a/src/datasets/new_data.md b/src/datasets/new_data.md index 0b3ac9eb3..cd1f8378a 100644 --- a/src/datasets/new_data.md +++ b/src/datasets/new_data.md @@ -5,6 +5,8 @@ For information about what sorts of data may be collected, and for information on getting a data collection request reviewed, please read the [Data Collection Guidelines.][data_collection] +The data review request and review templates are now maintained as the source of truth in the +[mozilla/data-review GitHub repository](https://github.com/mozilla/data-review). ## Mechanics diff --git a/src/datasets/search/search_clients_last_seen/reference.md b/src/datasets/search/search_clients_last_seen/reference.md index 9041bafa0..a3796bf61 100644 --- a/src/datasets/search/search_clients_last_seen/reference.md +++ b/src/datasets/search/search_clients_last_seen/reference.md @@ -30,8 +30,11 @@ This dataset is scheduled on Airflow As of 2020-04-22, the current version of `search_clients_last_seee` is `v1`, and has a schema as follows. The default `search.search_clients_last_seen` view still returns this `v1` schema; a newer `v2` version with a different -schema (described below) also exists and is used as the basis for LTV -calculations. +schema also exists and is used as the basis for LTV +calculations. Compared to `v1`, `v2` replaces `profile_creation_date`/ +`profile_age_in_days` with `first_seen_date` and `first_reported_country`, +adds `profile_group_id` and a new `total_uri_count_sum`/`days_dau_bytes` +pair for DAU tracking, and drops `days_created_profile_bytes`. It's backfilled through 2020-01-01 ``` diff --git a/src/metrics/metrics.md b/src/metrics/metrics.md index 110788a6b..5a7cd3295 100644 --- a/src/metrics/metrics.md +++ b/src/metrics/metrics.md @@ -18,7 +18,7 @@ Supporting metric information can be found on the [Metrics Inventory](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/620494911/Metrics+Inventory) page in Confluence (Mozilla LDAP required). The canonical, official source code definitions for the metrics can be found in -[metric-hub](https://github.com/mozilla/metric-hub/tree/main/definitions) on GitHub. +[metric-hub](https://github.com/mozilla/metric-hub/tree/main/definitions) on GitHub, which includes change-controlled mobile metric definitions such as `mobile_daily_active_users_v1`, `mobile_dau_kpi_v1`, `mobile_engagement_rate_v1`, `mobile_retention_rate_v1`, and `mobile_new_profile_retention_rate_v1`. ### Details diff --git a/src/tools/spark.md b/src/tools/spark.md index 691481995..03b280eb6 100644 --- a/src/tools/spark.md +++ b/src/tools/spark.md @@ -35,7 +35,7 @@ You can spin up a Dataproc cluster with Jupyter using the following command. Ins ```bash gcloud beta dataproc clusters create cluster-name \ --optional-components=ANACONDA,JUPYTER \ - --image-version=1.4 \ + --image-version=2.3 \ --enable-component-gateway \ --properties=^#^spark:spark.jars=gs://spark-lib/bigquery/spark-bigquery-latest.jar \ --num-workers=3 \