Retract per server metrics when the mode changes

doc/admin-guide/files/records.yaml.en.rst

@@ -2033,13 +2033,15 @@ Origin Server Connect Attempts
    this setting resolve to the same group -- that is, the same key under
    :ts:cv:`proxy.config.http.per_server.connection.match` -- the transaction that creates the group
    determines its metrics, and later transactions do not change them. A group is discarded once its
-   connection count reaches zero, so *raising* the level of publication is picked up the next time
-   that upstream is reopened: enabling metrics, or enabling the aggregates, takes effect as upstreams
-   reconnect. Lowering it does not. Metrics are never retired once published, so disabling this
-   setting, or switching
-   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` to ``2``, leaves the names that
-   are already published in place, frozen at their last sampled value, until |TS| is restarted. This
-   affects only which metrics exist; enforcement of
+   connection count reaches zero, so a change is picked up the next time that upstream is reopened.
+   A group that never goes idle keeps whatever was in effect when it was created.
+
+   Disabling this setting does not retire metrics that are already published: they stay in place,
+   frozen at their last sampled value, until |TS| is restarted. Changing
+   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` does retract what it no longer
+   asks for, as each group is rebuilt.
+
+   This affects only which metrics exist; enforcement of
    :ts:cv:`proxy.config.http.per_server.connection.max` uses the group's own connection count and is
    unaffected.
 
@@ -2051,24 +2053,30 @@ Origin Server Connect Attempts
    :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Has no effect when that setting
    is ``0``.
 
-   A per hostname aggregate sums a counter across every group belonging to that hostname that has
-   aggregation enabled, and exists only for
+   There are two kinds of per hostname aggregate. The *sums* add ``current_connection``,
+   ``total_connection`` and ``blocked_connection`` across every group belonging to that hostname
+   that has aggregation enabled. The *max* is ``current_connection.max``, the largest
+   ``current_connection`` among those groups, which is the one that answers how close the busiest
+   group is to :ts:cv:`proxy.config.http.per_server.connection.max`. Both exist only for
    :ts:cv:`match type <proxy.config.http.per_server.connection.match>` ``both``, since that is the
    only match type whose group key carries the hostname. See :ref:`per-server-connection-metrics`.
 
-   ===== ======================================================================================
-   Value Effect
-   ===== ======================================================================================
-   ``0`` No aggregates. The per group metrics are published under their own names.
-   ``1`` Publish the per hostname aggregates and the per group metrics.
-   ``2`` Publish only the per hostname aggregates. The per group metrics from which they are
-         computed are collected but not published, which keeps the number of published metrics
-         proportional to hostnames rather than to groups.
-   ===== ======================================================================================
+   ===== =========== ====== =====
+   Value Per group   Sums   Max
+   ===== =========== ====== =====
+   ``0`` published   no     no
+   ``1`` published   yes    yes
+   ``2`` hidden      no     yes
+   ``3`` hidden      yes    yes
+   ===== =========== ====== =====
 
-   With value ``2``, a group that has no aggregate to belong to -- any match type other than
-   ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would be
-   reported for it.
+   ``2`` is the smallest useful configuration: one metric per hostname. ``3`` adds that hostname's
+   totals. Both keep the number of published metrics proportional to hostnames rather than to
+   groups.
+
+   With values ``2`` and ``3``, a group that has no aggregate to belong to -- any match type other
+   than ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would
+   be reported for it.
 
    Values ``0`` and ``1`` can produce a very large number of metrics when the match type includes the
    address or port, since there is then one set per address and port rather than one per hostname.
@@ -2079,12 +2087,14 @@ Origin Server Connect Attempts
    upstream had aggregation enabled, so mappings that disagree for one hostname produce an aggregate
    that covers only part of it.
 
-   The reload is one-directional for the same reason given under
-   :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Raising the value takes effect
-   as upstreams reconnect, but moving to ``2`` does not hide per group metrics that are already
-   published, and moving from ``1`` to ``0`` does not stop the hostname aggregates from publishing.
-   Reducing the number of published metrics therefore requires a restart, which matters most for
-   ``2``, the value chosen specifically to bound that number.
+   A change in either direction takes effect as upstreams reconnect: a group publishes what the
+   new value asks for and withdraws what it does not, when that group is next rebuilt. Metrics
+   withdrawn this way stop appearing in :program:`traffic_ctl` output and in the other metric
+   consumers; they are not destroyed, and moving back republishes them with their accumulated
+   values intact.
+
+   Because the sums are named per hostname rather than per group, where the mappings for one
+   hostname disagree about this setting the last group rebuilt decides whether they are published.
 
 .. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_prefix STRING NULL
    :reloadable:

doc/admin-guide/monitoring/statistics/core/http-connection.en.rst

@@ -234,21 +234,23 @@ blocked_connection
    Counter. The total number of connection attempts to the group blocked by
    :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
 
-For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
-hostname which have aggregation enabled, plus:
+For a hostname aggregate there are two kinds. The *sums* are those same three counters, each added
+across the groups of that hostname which have aggregation enabled, published at
+:ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` ``1`` and
+``3``. The *max*, published at ``1``, ``2`` and ``3``, is:
 
-current_connection_max
+current_connection.max
    Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment
    of sampling, so the maximum rather than the sum of the groups' current counts. This is useful
    because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than
    per hostname, so the busiest group is what determines whether connections are about to be
    blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water
-   mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate.
+   mark. There is no per group ``current_connection.max``; it exists only as a hostname aggregate.
 
 Because :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` is overridable, a group
 joins its hostname's aggregate only if the mapping that first opened that upstream had aggregation
 enabled. Mappings that disagree for one hostname therefore produce an aggregate over part of it: the
-sums cover a subset of the groups and ``current_connection_max`` takes its maximum over that same
+sums cover a subset of the groups and ``current_connection.max`` takes its maximum over that same
 subset, with nothing in the metric to indicate it. Keeping the setting uniform across the mappings
 for a hostname avoids this.
 
@@ -261,13 +263,13 @@ Every published per server metric is recomputed periodically, currently every 5
 on every connection event, so a reader sees a value up to that interval old. This is true of the
 hostname aggregates and of the published per group metrics alike: those are
 mirrored from the internal ones by the same periodic mechanism, not written as connections open and
-close. It applies to ``current_connection_max`` too, which reports the maximum across groups as of
+close. It applies to ``current_connection.max`` too, which reports the maximum across groups as of
 the last sample rather than a running peak. To obtain the peak over a longer window, compute a
 maximum over time from this gauge in the monitoring system.
 
-At :ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` value ``2``
-the per group metrics still exist internally, since the aggregates are computed from them, but are not
-published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which
+At :ts:cv:`metric_aggregate <proxy.config.http.per_server.connection.metric_aggregate>` values
+``2`` and ``3`` the per group metrics still exist internally, since the aggregates are computed from
+them, but are not published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which
 reads them directly and so is not subject to the sampling delay above. That visibility is intended
 for debugging and is not a stable interface: the existence, granularity and naming of the per group
 metrics may change independently of the published aggregates.

doc/developer-guide/internal-libraries/Metrics.en.rst

@@ -188,6 +188,46 @@ sampling point*, not the true peak. There are two ways to arrange this, with dif
 
 Which is appropriate depends on whether the consumer needs to aggregate over time downstream.
 
+Unlisting a metric
+==================
+
+A metric can be taken out of the store's listing after the fact. An unlisted metric is skipped by
+iteration, so it disappears from ``traffic_ctl metric match``, the JSONRPC record lookup and
+``stats_over_http``, without either of those consumers needing to know about it:
+
+.. code-block:: cpp
+
+    auto &m = ts::Metrics::instance();
+
+    m.unlist(id);                       // by id
+    m.unlist("proxy.process.example");  // or by name
+
+    m.relist(id);                       // put it back
+
+The slot, the name and the atomic all survive: an unlisted number that still rings. An unlisted
+metric still resolves through ``lookup``, so an exact name query, a logging field reference and
+``TSStatFindName`` all continue to work, and its value may still be read and written. Creating the
+same name again relists it and returns the same id with its accumulated value intact, so a metric
+that comes and goes with a configuration setting costs nothing to bring back.
+
+``find`` is the exception: it returns ``end()`` for an unlisted metric. Iteration never visits an
+unlisted slot, so an iterator pointing at one would be a range bound that a walk steps straight over
+and never reaches. Use ``lookup`` to read an unlisted metric.
+
+This exists because the decision to publish a name is otherwise made once, when the metric is first
+created, and can never be revisited. Any metric whose name or publication policy depends on a
+runtime changeable setting needs a way to retract a name it has already published.
+
+.. important::
+
+   Unlisting hides; it does not free. The slot and the name remain allocated against the storage
+   limit below. Unlisting does not make an unbounded naming scheme safe.
+
+.. note::
+
+   Iteration is a snapshot taken when the iterator is created. A metric created after ``begin()``
+   is not visited by that iterator.
+
 Storage limits
 ==============
 

include/iocore/net/ConnectionTracker.h

@@ -79,29 +79,51 @@ class ConnectionTracker
    *
    * This is independent of @c TxnConfig::metric_enabled, which decides only whether per server
    * metrics exist for a group at all. The per group metrics are always created in the hidden metric
-   * store; what varies here is what gets published from them:
-   *   - @c AGGREGATE_NONE: no aggregate. The per group metrics are published under their own names.
-   *     This is the default and matches the behavior of releases that had no aggregate support.
-   *   - @c AGGREGATE_GROUP: the per hostname aggregates are published, and so are the per group
-   *     metrics they are computed from.
-   *   - @c AGGREGATE_ONLY: the per hostname aggregates are published and the per group metrics stay
-   *     hidden, which keeps the published metric count proportional to hostnames rather than to
-   *     groups. Where a group has no aggregate to belong to -- see @c Group::host_metric_name, which
-   *     only yields a name for match type @c MATCH_BOTH -- the per group metrics are published
-   *     anyway, since otherwise nothing at all would be reported for that group.
+   * store; what varies here is what gets published from them.
+   *
+   * Two kinds of per hostname aggregate exist. The *sums* are @c current_connection,
+   * @c total_connection and @c blocked_connection added across the groups of a hostname. The *max*
+   * is @c current_connection.max, the largest @c current_connection among those groups. Which of
+   * them are published, and whether the per group metrics are published alongside, is what this
+   * selects:
+   *
+   *   | value                | per group | sums | max |
+   *   |----------------------|-----------|------|-----|
+   *   | @c AGGREGATE_NONE    | yes       | no   | no  |
+   *   | @c AGGREGATE_GROUP   | yes       | yes  | yes |
+   *   | @c AGGREGATE_MAX     | no        | no   | yes |
+   *   | @c AGGREGATE_SUM     | no        | yes  | yes |
+   *
+   * @c AGGREGATE_NONE is the default and matches the behavior of releases that had no aggregate
+   * support. @c AGGREGATE_MAX is the smallest useful configuration: one metric per hostname,
+   * answering how close the busiest group is to @c per_server.connection.max. @c AGGREGATE_SUM adds
+   * the totals for that hostname. Both keep the published metric count proportional to hostnames
+   * rather than to groups.
+   *
+   * Where a group has no aggregate to belong to -- see @c Group::host_metric_name, which only
+   * yields a name for match type @c MATCH_BOTH -- the per group metrics are published whatever this
+   * says, since otherwise nothing at all would be reported for that group.
    *
    * Keeping the per group metrics in the hidden store in every case means changing this at runtime
    * is only a change of what is registered for publication, with no metric to migrate between the
    * two stores.
    *
-   * The records layer validates and clamps this to 0..2. A plugin setting the overridable config
+   * A change is applied per group, when that group is next constructed, which happens on the first
+   * connection after its count last fell to zero. A group that never goes idle keeps whatever was
+   * in effect when it was created. Retracting a published name relies on the metric store's
+   * listing, see @c ts::Metrics::unlist. The sums are named per hostname rather than per group, so
+   * where the mappings for one hostname disagree about this setting, the last group constructed
+   * decides whether they are published.
+   *
+   * The records layer validates and clamps this to 0..3. A plugin setting the overridable config
    * directly is not clamped, see @c METRIC_AGGREGATE_CONV; any other value behaves as
-   * @c AGGREGATE_GROUP, publishing both the aggregate and the per group metrics.
+   * @c AGGREGATE_GROUP, publishing everything.
    */
   enum MetricAggregate : int {
     AGGREGATE_NONE  = 0, ///< No hostname aggregate; the per group metrics are published.
-    AGGREGATE_GROUP = 1, ///< Hostname aggregates published, along with the per group metrics.
-    AGGREGATE_ONLY  = 2, ///< Hostname aggregates published, per group metrics kept hidden.
+    AGGREGATE_GROUP = 1, ///< Hostname sums and max, published along with the per group metrics.
+    AGGREGATE_MAX   = 2, ///< Hostname max only; the per group metrics stay hidden.
+    AGGREGATE_SUM   = 3, ///< Hostname sums and max; the per group metrics stay hidden.
   };
 
   /// Per transaction configuration values.