Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
997af76
concepts/pipeline/glean_data.md: Mention events_stream as recommended…
Oct 21, 2026
5aa42f0
concepts/sql_style.md: Link to bigquery-etl Recommended Practices guide
Oct 21, 2026
79d2cf6
contributing.md: Document spell-check requirement from README
Oct 21, 2026
19cc195
cookbooks/analysis/glean_dictionary.md: Document the new MCP server f…
Oct 21, 2026
4939492
cookbooks/data_monitoring/bigquery_etl_integration.md: Warn against r…
Oct 21, 2026
e9113e4
cookbooks/data_monitoring/bigquery_etl_integration.md: Document curre…
Oct 21, 2026
5ecb26a
cookbooks/data_monitoring/bigquery_etl_integration.md: Add caution to…
Oct 21, 2026
9324f04
cookbooks/new_ping.md: Note legacy Telemetry/custom pings are depreca…
Oct 21, 2026
8c12ab3
cookbooks/operational/connecting_external_data_bigquery.md: Update ex…
Oct 21, 2026
f3eab19
datasets/bigquery/clients_last_seen/reference.md: Document normal/pri…
Oct 21, 2026
29cbcba
datasets/bigquery/clients_last_seen/reference.md: Document days_since…
Oct 21, 2026
59da372
datasets/experiment_monitoring.md: Add new crash-rate monitoring data…
Oct 21, 2026
e16ac13
datasets/glam.md: Note legacy desktop telemetry deprecation and stop …
Oct 21, 2026
630d697
datasets/glam.md: Mark legacy desktop DAG deprecated, no longer runni…
Oct 21, 2026
3a5a00f
datasets/new_data.md: Note mozilla/data-review repo as canonical revi…
Oct 21, 2026
0726179
datasets/search/search_clients_last_seen/reference.md: Describe v2 sc…
Oct 21, 2026
e808b7e
metrics/metrics.md: List metric-hub's structured mobile metric defini…
Oct 21, 2026
7060de2
tools/spark.md: Update example to current default Dataproc image version
Oct 21, 2026
71d2dc6
Fix spelling: capitalize Bigconfig
scholtzan Oct 22, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/concepts/pipeline/glean_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,5 @@ In this case the event's `category` would be `engine_tab` and its name would be
You can use the record's `timestamp` and `extra` fields to get the event's timestamp and specifics related
to the event.
For a complete example, see ["event metrics" under Accessing Glean Data](../../cookbooks/accessing_glean_data.md#event-metrics).

For most querying needs, the generated `events_stream` table (e.g. `{app}.events_stream`) is now recommended over unnesting the `events` column directly: it flattens each event into its own row with a stable `event_id`, a derived `event_timestamp`, and `event_extra` as a JSON object. See the [FxA datasets](https://docs.telemetry.mozilla.org/datasets/fxa.html) page for an example.
4 changes: 4 additions & 0 deletions src/concepts/sql_style.md
Original file line number Diff line number Diff line change
Expand Up @@ -446,5 +446,9 @@ LIMIT 10

This document was heavily influenced by https://www.sqlstyle.guide/

For additional conventions on query file structure, naming, incremental
queries, and other practices not covered here, see the
[bigquery-etl Recommended Practices](https://mozilla.github.io/bigquery-etl/reference/recommended_practices/) guide.

Changes to the style guide should be reviewed by at least one member of
both the Data Engineering team and the Data Science team.
2 changes: 2 additions & 0 deletions src/contributing/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ This documentation is stored as [CommonMark Markdown](https://commonmark.org/hel
[`data-docs` repository](https://github.com/mozilla/data-docs) on GitHub.
To build a local copy, fork the repository and check out your copy. Then, [see the README](https://github.com/mozilla/data-docs/blob/master/README.md#building-the-documentation) for up-to-date information on how to build the documentation.

Before submitting a pull request, run the spell checker (`scripts/spell_check.sh`, which uses `markdown-spell-check`) and add any necessary exceptions to the `.spelling` file.

## Adding a new article

You should read the [style guide](./style_guide.md) before adding a new article: it will help you write material that is more useful and fits cohesively into the rest of the documentation.
Expand Down
5 changes: 5 additions & 0 deletions src/cookbooks/analysis/glean_dictionary.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ After doing so, you can search for metrics by name (e.g.: `addons.enabled_addons

After selecting a metric, you can get more information on it including a reference to its definition in the source code as well as information on how to get the data submitted by this probe in some of our data tools like [STMO], [Looker], and [GLAM].

## Using the Glean Dictionary with AI assistants

Because the Glean Dictionary's web UI is a single-page application that loads metrics dynamically via JavaScript, AI assistants and other automated tools that fetch the page directly only see an empty HTML shell. For this reason, the Glean Dictionary also exposes a [Model Context Protocol] (MCP) server at `https://dictionary.telemetry.mozilla.org/mcp`, providing structured, direct access to the same metadata. It offers five tools: `list_apps`, `get_app`, `search_metrics`, `get_metric`, and `get_ping`.

## Common Questions

### How can I go from a metric to querying it in BigQuery?
Expand All @@ -30,3 +34,4 @@ For several examples of this along with a more complete explanation, see [Access
[looker]: ../../introduction/tools.md#looker
[glam]: ../../introduction/tools.md#glean-aggregated-metrics-dashboard-glam
[accessing glean data]: ../accessing_glean_data.md
[model context protocol]: https://github.com/mozilla/glean-dictionary/blob/main/docs/mcp-server.md
4 changes: 3 additions & 1 deletion src/cookbooks/data_monitoring/bigquery_etl_integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To indicate whether a failing check should block any downstream Airflow tasks, a

## Bigconfig

Additional and custom monitors can be defined in a [Bigconfig](https://docs.bigeye.com/docs/bigconfig#example-template) `bigconfig.yml` file that is stored in the same directory as the table query. Bigconfig allows users to deploy other pre-defined monitors, such as row counts or null checks on a table or column level.
Additional and custom monitors can be defined in a [Bigconfig](https://docs.bigeye.com/docs/bigconfig#example-template) `bigconfig.yml` file that is stored in the same directory as the table query. Bigconfig should be applied to tables, not views, since default metrics like volume and freshness rely on insertion metadata that views lack. Bigconfig allows users to deploy other pre-defined monitors, such as row counts or null checks on a table or column level. The file must start with `type: BIGCONFIG_FILE` and defines `table_deployments`, each specifying a `collection.name` and one or more `deployments` (identified by `fq_table_name`); monitors are declared via `table_metrics` (e.g. `saved_metric_id: volume`) and, for column-level checks, `columns[].metrics` (e.g. `saved_metric_id: is_not_null`).

## Custom SQL Rules

Expand Down Expand Up @@ -74,3 +74,5 @@ To generate a `bigconfig.yml` file with the default metrics when monitoring is e
The created file can be manually edited. For tables that do not have a `bigconfig.yml` checked into the repository, the file will get generated automatically before deployment to Bigeye. Files only need to be checked in if there are some customizations.

The deployment of Bigconfig files runs automatically as part of the [artifact deployment process](../../concepts/pipeline/artifact_deployment.md), after tables and views have been deployed.

Bigconfig changes should not be applied locally (e.g. via `bigconfig apply`) against an incomplete file set, since this can delete existing Bigeye metrics. Only the `publish_bigeye_monitors` task in the artifact deployment DAG should push Bigconfig changes to Bigeye.
2 changes: 2 additions & 0 deletions src/cookbooks/new_ping.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ to find out.
ping document type. To add a histogram, scalar, or event collection to Firefox, please see the
documentation on [adding a new
probe](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/start/adding-a-new-probe.html).
Note also that legacy Telemetry collection APIs (including custom pings) are now deprecated for
Firefox Desktop in favor of Glean.

## Write Your Questions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,6 @@ fields:
8. Open a pull-request against [bigquery-etl](https://github.com/mozilla/bigquery-etl) and tag someone for review.
- Due to the permissions required, it is not possible to build and QA this table before it is deployed after merging.
9. Once the PR has been reviewed and merged, the table will be available the next day in BigQuery.
- If the table should be made available immediately, then go to the [`bqetl_artifact_deployment` Airflow DAG](https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid) and clear the `publish_new_tables` task. This might need to be done by a data engineer or someone who has permissions to trigger and clear tasks in Airflow. The table will be available as soon as the task finishes.
- If the table should be made available immediately, then go to the [`bqetl_artifact_deployment` Airflow DAG](https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid) and clear the `publish_tables_and_views` task (this task replaced the earlier `publish_new_tables`/`publish_views` tasks). This might need to be done by a data engineer or someone who has permissions to trigger and clear tasks in Airflow. The table will be available as soon as the task finishes.

For confidential data it is generally recommended to add these configurations to [private-bigquery-etl](https://github.com/mozilla/private-bigquery-etl). The process and configurations are the same, the only difference is the repository which is not publicly accessible.
5 changes: 5 additions & 0 deletions src/datasets/bigquery/clients_last_seen/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,14 @@ Please see this [section](../../../concepts/segments.md) for descriptions regard

- `days_visited_1_uri_bits`: Each bit field represents if a client browsed at least 1 URI on that day.
- `days_since_visited_1_uri`: Number of days since the client browsed at least 1 URI.
- `days_visited_1_uri_normal_mode_bits`: Each bit field represents if a client browsed at least 1 URI outside of Private Browsing Mode on that day.
- `days_since_visited_1_uri_normal_mode`: Number of days since the client browsed at least 1 URI outside of Private Browsing Mode.
- `days_visited_1_uri_private_mode_bits`: Each bit field represents if a client browsed at least 1 URI in Private Browsing Mode on that day. Only available for Firefox 84 and later.
- `days_since_visited_1_uri_private_mode`: Number of days since the client browsed at least 1 URI in Private Browsing Mode. Only available for Firefox 84 and later.
- `days_interacted_bits`: Each bit field represents if a client had at least 1 active tick on that day.
This is derived from the `active_hours_sum` in `clients_daily`.
- `days_since_interacted`: Number of days since the clients had at least 1 active tick.
- `days_since_qualified_use_v1`: Number of days since the client both browsed at least 1 URI and had at least 1 active tick (computed as `days_visited_1_uri_bits & days_interacted_bits`). This is a view-computed field, not a stored bits column.
- `days_had_8_active_ticks_bits`: Each bit field represents if a client had at least 8 active ticks on that day.
This can be used to approximate the threshold of 1 URI, and is useful for determining activity for clients using
Private Browsing Mode where URI counts are not recorded.
Expand Down
4 changes: 4 additions & 0 deletions src/datasets/experiment_monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ Experiment monitoring datasets are designed to power dashboards, such as [Experi
| `mozdata.telemetry.experiment_cumulative_search_count` | Cumulative number of searches by clients enrolled in experiments |
| `mozdata.telemetry.experiment_cumulative_search_with_ads_count` | Cumulative number of searches with ads by clients enrolled in experiments |

## Experiment crash rate data

`moz-fx-data-shared-prod.telemetry.experiment_crash_rates_live` provides crash-rate monitoring for clients enrolled in experiments, combining per-app `experiment_crash_events_live_v1` materialized views with historical data from `telemetry_derived.experiment_crash_aggregates_v1`. Values are aggregated by experiment, branch, crash process type and crash signature.

## Derived dataset

The derived views have the following schema:
Expand Down
6 changes: 4 additions & 2 deletions src/datasets/glam.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ Each link below points to the dataset's page on [Mozilla's Data Catalog](https:/

### Firefox Desktop (Legacy Telemetry)

> **_NOTE:_** Firefox Desktop Legacy Telemetry support in GLAM is deprecated; new data stopped being aggregated for these tables on January 26, 2026.

- [`moz-fx-data-shared-prod.glam_etl.glam_desktop_nightly_aggregates`](<https://mozilla.acryl.io/dataset/urn:li:dataset:(urn:li:dataPlatform:bigquery,moz-fx-data-shared-prod.glam_etl.glam_desktop_nightly_aggregates,PROD)/Schema?is_lineage_mode=false&schemaFilter=>)
- [`moz-fx-data-shared-prod.glam_etl.glam_desktop_beta_aggregates`](<https://mozilla.acryl.io/dataset/urn:li:dataset:(urn:li:dataPlatform:bigquery,moz-fx-data-shared-prod.glam_etl.glam_desktop_beta_aggregates,PROD)/Schema?is_lineage_mode=false&schemaFilter=>)
- [`moz-fx-data-shared-prod.glam_etl.glam_desktop_release_aggregates`](<https://mozilla.acryl.io/dataset/urn:li:dataset:(urn:li:dataPlatform:bigquery,moz-fx-data-shared-prod.glam_etl.glam_desktop_release_aggregates,PROD)/Schema?is_lineage_mode=false&schemaFilter=>)
Expand All @@ -45,8 +47,8 @@ Most of the GLAM ETL is scheduled to run daily via Airflow. There are separate E
- [Firefox Desktop on Glean (release)](https://workflow.telemetry.mozilla.org/dags/glam_fog_release/grid)
- Runs weekly
- Second "half" of release ETL is processed
- [Firefox Desktop legacy](https://workflow.telemetry.mozilla.org/dags/glam/grid)
- Runs daily
- [Firefox Desktop legacy](https://workflow.telemetry.mozilla.org/dags/glam/grid) (deprecated)
- New aggregations stopped on January 26, 2026
- [Firefox for Android](https://workflow.telemetry.mozilla.org/dags/glam_fenix/grid)
- Runs daily

Expand Down
2 changes: 2 additions & 0 deletions src/datasets/new_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
For information about what sorts of data may be collected,
and for information on getting a data collection request reviewed,
please read the [Data Collection Guidelines.][data_collection]
The data review request and review templates are now maintained as the source of truth in the
[mozilla/data-review GitHub repository](https://github.com/mozilla/data-review).

## Mechanics

Expand Down
7 changes: 5 additions & 2 deletions src/datasets/search/search_clients_last_seen/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,11 @@ This dataset is scheduled on Airflow
As of 2020-04-22, the current version of `search_clients_last_seee` is `v1`,
and has a schema as follows. The default `search.search_clients_last_seen`
view still returns this `v1` schema; a newer `v2` version with a different
schema (described below) also exists and is used as the basis for LTV
calculations.
schema also exists and is used as the basis for LTV
calculations. Compared to `v1`, `v2` replaces `profile_creation_date`/
`profile_age_in_days` with `first_seen_date` and `first_reported_country`,
adds `profile_group_id` and a new `total_uri_count_sum`/`days_dau_bytes`
pair for DAU tracking, and drops `days_created_profile_bytes`.
It's backfilled through 2020-01-01

```
Expand Down
2 changes: 1 addition & 1 deletion src/metrics/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Supporting metric information can be found on the
[Metrics Inventory](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/620494911/Metrics+Inventory) page in Confluence (Mozilla LDAP required).

The canonical, official source code definitions for the metrics can be found in
[metric-hub](https://github.com/mozilla/metric-hub/tree/main/definitions) on GitHub.
[metric-hub](https://github.com/mozilla/metric-hub/tree/main/definitions) on GitHub, which includes change-controlled mobile metric definitions such as `mobile_daily_active_users_v1`, `mobile_dau_kpi_v1`, `mobile_engagement_rate_v1`, `mobile_retention_rate_v1`, and `mobile_new_profile_retention_rate_v1`.

### Details

Expand Down
2 changes: 1 addition & 1 deletion src/tools/spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ You can spin up a Dataproc cluster with Jupyter using the following command. Ins
```bash
gcloud beta dataproc clusters create cluster-name \
--optional-components=ANACONDA,JUPYTER \
--image-version=1.4 \
--image-version=2.3 \
--enable-component-gateway \
--properties=^#^spark:spark.jars=gs://spark-lib/bigquery/spark-bigquery-latest.jar \
--num-workers=3 \
Expand Down
Loading