diff --git a/src/concepts/pipeline/event_pipeline.md b/src/concepts/pipeline/event_pipeline.md index 4efffbb9b..33ed0730a 100644 --- a/src/concepts/pipeline/event_pipeline.md +++ b/src/concepts/pipeline/event_pipeline.md @@ -125,6 +125,7 @@ On the pipeline side, the event data is made available in different datasets: its event payload for Firefox versions before 62. - [`events`](../../datasets/batch_view/events/reference.md) contains a row for each event received from main pings and event pings. See [`STMO#52582`](https://sql.telemetry.mozilla.org/queries/52582/source). - For applications that collect events through Glean, each application has a separate `events` dataset. +- [`events_daily`](../../datasets/bigquery/events_daily/reference.md) encodes each client-day's events as a compressed string of Unicode characters, for use with the `mozfun.event_analysis` functions for funnel and event-count analysis. # Data tooling diff --git a/src/concepts/segments.md b/src/concepts/segments.md index a1c818e8c..9d0ec4fee 100644 --- a/src/concepts/segments.md +++ b/src/concepts/segments.md @@ -25,6 +25,8 @@ so that your communication is forwards compatible. ## Current user states/segments +Note: a newer `clients_last_seen_v2` view now exists alongside `clients_last_seen_v1`, adding columns not yet described below: `new_profile_7_day_activated_v1`, `new_profile_14_day_activated_v1`, and `new_profile_21_day_activated_v1` (new-profile activation flags), plus a `segment` column that generalizes `activity_segments_v1` using a `days_active_bits` column. See the view source for exact definitions. + ### Regular users v3 `clients_last_seen.is_regular_user_v3` diff --git a/src/concepts/terminology.md b/src/concepts/terminology.md index 2d9a551fa..c11b33724 100644 --- a/src/concepts/terminology.md +++ b/src/concepts/terminology.md @@ -158,7 +158,7 @@ In Firefox, [Sessions](#session) are split into subsessions after every 24-hour ## STMO (sql.telemetry.mozilla.org) -A service for creating queries and dashboards. See [STMO under analysis tools](../introduction/tools.md#sqltelemetrymozillaorg-stmo). +A service for creating queries and dashboards. While not deprecated, Looker is now the preferred solution for producing data visualizations and dashboards at Mozilla, where possible. See [STMO under analysis tools](../introduction/tools.md#sqltelemetrymozillaorg-stmo). ## Telemetry diff --git a/src/contributing/structure.md b/src/contributing/structure.md index 1944f0a8d..68c47f5f8 100644 --- a/src/contributing/structure.md +++ b/src/contributing/structure.md @@ -9,6 +9,11 @@ The directory structure should feel comfortable to anyone who is familiar with t |--tools - contains tool level documentation |--concepts - contains tutorials meant to introduce a new concept to the reader |--cookbooks - focused code examples for reference + |--introduction - contains introductory material for newcomers to the data platform + |--reference - contains in-depth reference material on the data platform + |--contributing - contains documentation about contributing to this documentation + |--historical - contains archived, historical documentation + |--metrics - contains archived, historical metrics documentation ``` This documentation is meant to take the reader from beginner to expert. diff --git a/src/cookbooks/clients_last_seen_bits.md b/src/cookbooks/clients_last_seen_bits.md index b1d6aa23c..d8ce4b370 100644 --- a/src/cookbooks/clients_last_seen_bits.md +++ b/src/cookbooks/clients_last_seen_bits.md @@ -79,6 +79,9 @@ ORDER BY submission_date ``` +Other bit pattern fields available in the desktop table include `days_active_bits` +and `days_visited_1_uri_bits`. + Adding a new usage criterion is possible, but requires some work especially if a historical backfill is necessary, so [file a bug](../concepts/reporting_a_problem.md) to begin discussions on diff --git a/src/cookbooks/data_monitoring/intro.md b/src/cookbooks/data_monitoring/intro.md index 1fc579776..896d79287 100644 --- a/src/cookbooks/data_monitoring/intro.md +++ b/src/cookbooks/data_monitoring/intro.md @@ -16,6 +16,8 @@ Please refer to additional helpful videos available on the Bigeye channel. In addition to configuring monitors through the Bigeye UI, monitoring can be defined as code in [mozilla/bigquery-etl](https://github.com/mozilla/bigquery-etl). Tables can include a `bigconfig.yml` alongside their `metadata.yaml`, with monitoring enabled via a `monitoring.enabled` field. The `bqetl monitoring` CLI (`deploy`, `update`, `validate`, `deploy_custom_rules`) pushes these configs to Bigeye. +**Caution:** Avoid running `bigconfig apply` or `bqetl monitoring deploy` ad hoc on your local machine outside of the normal deployment automation. Doing so with an incomplete set of config files can delete metrics that aren't included in the plan. + ## Stay Updated on What's Happening [Jira Epic Link](https://mozilla-hub.atlassian.net/browse/DENG-4563) diff --git a/src/cookbooks/live_data.md b/src/cookbooks/live_data.md index b50bc76ae..6ed9bf36d 100644 --- a/src/cookbooks/live_data.md +++ b/src/cookbooks/live_data.md @@ -117,7 +117,7 @@ Google Cloud Functions can process incoming messages in Pub/Sub and stream the d ## Tools for Visualizing Live Data -Currently, live data in BigQuery could be considered "near real time" or nearline (as opposed to online), with data typically loaded within 10 minutes to each live table in our ingestion-sink code base, though a delay of up to 30 minutes is normal. +Currently, live data in BigQuery could be considered "near real time" or nearline (as opposed to online), with latency usually below 1 hour to each live table in our ingestion-sink code base. For near real time product monitoring use cases, it is recommended to use Looker for hosting dashboards that refresh on a regular interval such as every 10 minutes. diff --git a/src/cookbooks/looker/event_counts_explore.md b/src/cookbooks/looker/event_counts_explore.md index 6734bd00c..ee1f47360 100644 --- a/src/cookbooks/looker/event_counts_explore.md +++ b/src/cookbooks/looker/event_counts_explore.md @@ -9,4 +9,4 @@ You can see a quick demo of how to use this explore to answer a simple Firefox f This explore also exists for Mozilla VPN and other products (and you can use the same Glean Dictionary-based workflow to fill it out). -For Firefox Desktop, this explore is currently using legacy (non-Glean data) and you will need to use the [probe dictionary](../analysis/probe_dictionary.md) instead. +For Firefox Desktop, the `event_counts` explore is currently using legacy (non-Glean) data and you will need to use the [probe dictionary](../analysis/probe_dictionary.md) instead. A separate `glean_event_counts` explore is also available for Firefox Desktop, which follows the same Glean Dictionary-based workflow as other Glean products. diff --git a/src/cookbooks/looker/performance_caching.md b/src/cookbooks/looker/performance_caching.md index 32938183f..1261a6106 100644 --- a/src/cookbooks/looker/performance_caching.md +++ b/src/cookbooks/looker/performance_caching.md @@ -27,6 +27,7 @@ infrequently updated, shared views such as `countries.view.lkml` can still use t After finding the source table, in your `explore.lkml` file, include the auto-generated datagroup and add a `persist_with` parameter that matches the datagroup. By convention, the datagroup name matches source tables. +Generated datagroups also set a default `max_cache_age` of 24 hours as a fallback alongside their SQL trigger. See the following example: ``` diff --git a/src/datasets/batch_view/addons/reference.md b/src/datasets/batch_view/addons/reference.md index c3e1e3c36..fc7495991 100644 --- a/src/datasets/batch_view/addons/reference.md +++ b/src/datasets/batch_view/addons/reference.md @@ -15,10 +15,16 @@ there will be a row with nulls for all the addon fields This dataset is updated daily via the [telemetry-airflow](https://github.com/mozilla/telemetry-airflow) infrastructure. The job DAG runs every day after the Main Summary data has been generated. -The DAG defined in [`dags/bqetl_addons.py`](https://github.com/mozilla/bigquery-etl/blob/master/dags/bqetl_addons.py) +The DAG, named `bqetl_addons`, is declaratively defined in [`dags.yaml`](https://github.com/mozilla/bigquery-etl/blob/main/dags.yaml) ## Code Reference This dataset is generated by [BigQuery ETL](https://github.com/mozilla/bigquery-etl/). The query that generates the dataset is [`sql/moz-fx-data-shared-prod/telemetry_derived/addons_v2/query.sql`](https://github.com/mozilla/bigquery-etl/blob/master/sql/moz-fx-data-shared-prod/telemetry_derived/addons_v2/query.sql). You may find the [environment reference](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/environment.html#addons) in The Firefox Source Documentation helpful for understanding the source data. + +## Storage + +The underlying table is partitioned by day on `submission_date`, requires a +partition filter, and expires partitions after 775 days. It is clustered on +`sample_id`. diff --git a/src/datasets/batch_view/clients_daily/reference.md b/src/datasets/batch_view/clients_daily/reference.md index ecf73a7a4..f4b747298 100644 --- a/src/datasets/batch_view/clients_daily/reference.md +++ b/src/datasets/batch_view/clients_daily/reference.md @@ -61,6 +61,8 @@ The data is partitioned by `submission_date`. As of 2019-11-28, the current version of the `clients_daily` dataset is `v6`. +As of Q1 2021, the `telemetry.clients_daily` view points to `clients_daily_joined`, which merges in event-ping, crash, and first/second-seen data on top of the `v6` main-ping table. + ## Code Reference This dataset is generated by [BigQuery ETL](https://github.com/mozilla/bigquery-etl/). The query that generates the dataset is [`sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6/query.sql`](https://github.com/mozilla/bigquery-etl/blob/ad84a15d580333b41d36cfe8331e51238f3bafa1/sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6/query.sql). diff --git a/src/datasets/search/client_ltv/reference.md b/src/datasets/search/client_ltv/reference.md index cd3e77876..72f3af9f6 100644 --- a/src/datasets/search/client_ltv/reference.md +++ b/src/datasets/search/client_ltv/reference.md @@ -123,3 +123,5 @@ ORDER BY ``` This produces a histogram for the observed user frequencies and the model's predicted frequencies, allowing a chart similar to the one shown in the ["assessing model fit" example](https://lifetimes.readthedocs.io/en/latest/Quickstart.html#assessing-model-fit) in the `lifetimes` documentation. This table only checks performance for clients the model expects have, for example, clicked an ad in 0 to 28 days in the past year, since most of the distribution is contained in that interval. + +_Note: this `lifetimes`-based survival model and the `analysis.ltv_daily_model_perf` table are part of the retired v1 design. The current implementation instead derives client states via a Markov-chain model (`mozfun.ltv.\*\_states_\*`UDFs), fed by per-product`ltv*states_v1` tables.* diff --git a/src/datasets/search/search_clients_last_seen/reference.md b/src/datasets/search/search_clients_last_seen/reference.md index a3796bf61..36dd31793 100644 --- a/src/datasets/search/search_clients_last_seen/reference.md +++ b/src/datasets/search/search_clients_last_seen/reference.md @@ -87,6 +87,17 @@ root |- days_created_profile_bytes: bytes ``` +## Mobile Search Clients Last Seen + +A separate dataset, `search.mobile_search_clients_last_seen` (backed by +`search_derived.mobile_search_clients_last_seen_v1`), captures search activity +for mobile clients over the past 365 days and is used as the basis for mobile +LTV calculations. It is scheduled under the `bqetl_mobile_search` DAG and +handles app-migration logic (e.g. legacy Fennec/Focus/Klar transitioning to +their Glean-based successors). See the +[`mobile_search_clients_last_seen_v1` metadata](https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/search_derived/mobile_search_clients_last_seen_v1/metadata.yaml) +for details. + # Code Reference The `search_clients_last_seen` job is diff --git a/src/metrics/index.md b/src/metrics/index.md index 3dbeb97e6..f665022dc 100644 --- a/src/metrics/index.md +++ b/src/metrics/index.md @@ -1,6 +1,6 @@ # Standard Metrics -> **⚠** Some of the information in this section is obsolete. For an up to date explanation of accepted metrics used at Mozilla, [contact the Data team](../concepts/getting_help.md). +> **⚠** Some of the information in this section is obsolete. For an up to date explanation of accepted metrics used at Mozilla, see the [Metrics](https://docs.telemetry.mozilla.org/metrics/metrics) page and the [Growth and Usage Dashboard (GUD)](https://mozilla.github.io/gud/), or [contact the Data team](../concepts/getting_help.md). This section provides an overview of standard metrics used at Mozilla. Here you'll find the definitions and descriptions for each. @@ -8,7 +8,7 @@ Here you'll find the definitions and descriptions for each. The [Telemetry Behavior Reference](../concepts/index.md) section also provides information related to the definitions below. -We are now in the process of setting up the metrics top-level section here. This information will be moved into the appropriate subsection and this page will be replaced with an overview. +Canonical metric definitions are now maintained as code in [Metric Hub](https://github.com/mozilla/metric-hub) (`definitions/*.toml`, `nimbus/definitions/*.toml`) and in BigQuery tables such as `telemetry.active_users_aggregates` and `telemetry.mobile_active_users_aggregates`, rather than in this static page. ## Activity @@ -43,6 +43,10 @@ active during the week beginning one week after the day specified. Among profiles that were active at least once in the week starting on the specified day, what proportion (out of 1) are active during the following week. +Firefox Desktop now defines retention primarily via the "Retained (DAU)" metric +(based on DAU-qualifying days); this legacy "submitted any ping" definition has +been deprecated as of June 2026. See [Metric Hub](https://mozilla.github.io/metric-hub/default_configs/firefox_desktop/) +for details. ## Frequently Asked Questions diff --git a/src/metrics/metrics.md b/src/metrics/metrics.md index 5a7cd3295..3fca76295 100644 --- a/src/metrics/metrics.md +++ b/src/metrics/metrics.md @@ -52,7 +52,7 @@ The day 2-7 activation metric is calculated as: \\[ \frac{\text{Activated New Profiles (day 2-7)}}{\text{New Profiles (day 1-7)}} \\] -Activated New Profiles (day 2-7): Unique count of client ids who use the product at any point starting the day after they created a profile up to 6 days after. +Activated New Profiles (day 2-7): Unique count of client ids who are seen on more than one day during the day 2-7 window after profile creation and have performed at least one search in that period. New Profiles: Unique count of client ids with a given profile creation date. As not all initial pings are received exactly on the day of profile creation, we wait for 7 days after the profile creation date before establishing the New Profile cohort to ensure the data is complete. diff --git a/src/tools/spark.md b/src/tools/spark.md index 03b280eb6..9cb5d254b 100644 --- a/src/tools/spark.md +++ b/src/tools/spark.md @@ -45,6 +45,11 @@ gcloud beta dataproc clusters create cluster-name \ --project project-id ``` +> **Note:** Dataproc image versions 2.1 and above (including the `2.3` image used above) ship with a +> built-in Spark BigQuery connector. Setting `spark.jars` via `--properties` will not override it, since +> the built-in connector takes precedence; to use a different connector version, pass +> `--metadata SPARK_BQ_CONNECTOR_VERSION=` instead. + You can retrieve the Jupyter URL with the following command: ```bash