README-implementation

OpenTelemetry Filter Implementation Review
======================================================================

1  Overview
----------------------------------------------------------------------

The OpenTelemetry (OTel) filter for HAProxy creates, propagates and exports the
spans, metric instruments and log records that the OpenTelemetry specification
defines.  The filter hooks into the HAProxy stream processing pipeline through
the filter API and maps channel analyzer events to span lifecycle operations,
metric recordings and log-record emissions.

The implementation lives at the top of this addon checkout: header files, C
source files, a Makefile.mk fragment, and test configurations with their runner
scripts.


2  Directory Structure
----------------------------------------------------------------------

  haproxy-opentelemetry/
  |-- Makefile.mk        Build integration (loaded via EXTRA_MAKE)
  |-- include/
  |   |-- include.h      Master include (pulls all headers)
  |   |-- config.h       Build-time tunables (pool sizes, limits)
  |   |-- define.h       Utility macros (memory, strings, lists)
  |   |-- debug.h        Debug/logging infrastructure
  |   |-- filter.h       Filter return codes, alert macros
  |   |-- parser.h       Configuration keyword definitions
  |   |-- conf.h         Configuration data structures
  |   |-- conf_funcs.h   Generated init/free function macros
  |   |-- event.h        Event enumeration and data table
  |   |-- scope.h        Runtime span/context structures
  |   |-- pool.h         Memory pool helpers
  |   |-- http.h         HTTP header manipulation
  |   |-- otelc.h        Span context inject/extract wrappers
  |   |-- vars.h         HAProxy variable integration
  |   |-- sample.h       Sample fetch names (otel.context, otel.bytes_*)
  |   |-- util.h         String conversion, sample helpers
  |   |-- group.h        Group action (HAProxy rule integration)
  |   `-- cli.h          CLI command interface
  |-- src/
  |   |-- filter.c       Filter lifecycle and channel callbacks
  |   |-- parser.c       Configuration file parser
  |   |-- conf.c         Configuration structure init/free
  |   |-- event.c        Scope/span execution engine
  |   |-- scope.c        Runtime context and span management
  |   |-- http.c         HTTP header get/set/remove
  |   |-- otelc.c        C wrapper inject/extract bridge
  |   |-- vars.c         HAProxy variable read/write
  |   |-- sample.c       Sample fetch for span-context ACL checks
  |   |-- pool.c         Pool alloc/free, trash buffers
  |   |-- util.c         Argument handling, sample conversion
  |   |-- group.c        Group action parsing and execution
  |   `-- cli.c          CLI command handlers
  |-- dummy/             Build-only wrapper stand-in (see dummy/README)
  `-- test/
      |-- copy-yml.sh    YAML configuration transformer
      |-- test-speed.sh  Performance benchmarking runner
      |-- run-test-config.sh  Single-instance runner (real file)
      |-- run-sa.sh      Standalone test runner (symlink)
      |-- run-full.sh    Full event-coverage test runner (symlink)
      |-- run-fe-be.sh   Frontend-backend chain runner
      |-- run-ctx.sh     Context propagation test runner (symlink)
      |-- run-cmp.sh     Comparison test runner (symlink)
      |-- run-tcp.sh     TCP-mode test runner (symlink)
      |-- run-updown.sh  Up-down counter test runner (symlink)
      |-- run-err.sh     Error-logging test runner (symlink)
      |-- run-empty.sh   Empty configuration test runner (symlink)
      |-- run-optctx.sh  Optional upstream context test runner (symlink)
      |-- run-parser.sh  Configuration parser test driver (real file)
      |-- otlp_http-recorder.py  OTLP/HTTP traffic recorder
      |-- otlp_http-replay.py    OTLP/HTTP traffic replayer
      |-- haproxy-common.cfg     Shared HAProxy configuration part
      |-- index.html     Static page served by the origin server
      |-- README.md      Test overview, coverage and speed results
      |-- README-parser  Configuration parser test description
      |-- sa/            Standalone test configs
      |-- full/          Full event-coverage test configs
      |-- fe/            Frontend-only test configs
      |-- be/            Backend-only test configs
      |-- ctx/           Context propagation test configs
      |-- cmp/           Comparison test configs
      |-- tcp/           TCP-mode test configs
      |-- updown/        Up-down counter test configs
      |-- err/           Error-logging test configs
      |-- empty/         Minimal/empty configuration test
      |-- optctx/        Optional upstream context test configs
      `-- parser/        Parser test cases, tables and generators


3  Build System
----------------------------------------------------------------------

The Makefile.mk fragment is pulled in by the HAProxy top-level Makefile via the
EXTRA_MAKE variable, which lists addon directories whose Makefile.mk fragments
are included unconditionally.  The fragment detects the opentelemetry-c-wrapper
library via pkg-config or manual OTEL_INC/OTEL_LIB paths.

Build options:

  EXTRA_MAKE=<path> Addon directory containing Makefile.mk.  Listing this
                    directory is what enables the filter.  Multiple directories
                    may be passed, space-separated.
  OTEL_DEBUG=1      Compile with DEBUG_OTEL; links the _dbg variant of the
                    wrapper library and enables the debug-only flt_ops callbacks
                    deinit_per_thread, http_payload and http_reset in filter.c.
  OTEL_USE_VARS=1   Define USE_OTEL_VARS (and USE_OTEL_VARS_NAME when struct
                    var has a 'name' member), which allows span context
                    propagation via HAProxy transaction variables in addition
                    to HTTP headers; vars.c itself is always compiled.
  OTEL_INC=<path>   Manual include path for the C wrapper.
  OTEL_LIB=<path>   Manual library path for the C wrapper.
  OTEL_RUNPATH=1    Embed RPATH to the wrapper library.
  OTEL_STATIC=1     Pass --static to pkg-config so OTEL_LDFLAGS picks up the
                    full Libs.private chain; needed for static linking.

Compiled objects (vars.o is built unconditionally):

  cli.o  conf.o  event.o  filter.o  group.o  http.o  otelc.o  parser.o
  pool.o  sample.o  scope.o  util.o  vars.o

The dummy/ subtree stands in for the wrapper API so the filter builds and links
with neither the wrapper nor the OpenTelemetry C++ SDK installed.  Its Makefile
produces a static archive named the way Makefile.mk expects, with OTEL_INC and
OTEL_LIB pointed at it instead of pkg-config, and Makefile.mk builds the archive
in the same pass, handing OTEL_DEBUG on because DEBUG_OTEL and OTELC_DBG_MEM
change the shape of the debug macros and of the external allocator types.  The
resulting executable runs the whole of the filter's own logic but produces no
telemetry, and reports the C++ version as "none" so that it cannot be mistaken
for a real build.  See dummy/README.


4  Configuration Parsing
----------------------------------------------------------------------

Configuration parsing is driven by parser.c.  The filter is declared in the
HAProxy configuration with:

  filter opentelemetry [id <id>] config <file> [<name>]

The flt_otel_parse() function (parser.c) handles the "filter" line, creates an
flt_otel_conf structure, and delegates to parse_cfg() which loads the referenced
YAML/CFG file.  That file is parsed using temporary section registrations for
the section types:

  otel-instrumentation   ->  flt_otel_parse_cfg_instr()
  otel-group             ->  flt_otel_parse_cfg_group()
  otel-scope             ->  flt_otel_parse_cfg_scope()

After each section is fully parsed, a post-parse function validates the section
(e.g., flt_otel_post_parse_cfg_scope() checks that context injection is only
used on events that support it).

An optional trailing name on the filter line selects the [<name>] section of
the file: flt_otel_parse_check_scope() compares it, or the filter id when it
is absent, against HAProxy's cfg_scope while the file is parsed.  That same
function also rejects a keyword line that stands outside of any [<name>] scope.
The uniqueness of the scope name, the other top-level rule of section 4.5, is
checked before the file is parsed, by flt_otel_parse_check_scope_names() over
the loaded content: a section parser is called for the lines of a scope but
never for the declaration itself, so two scopes of the same name that follow
each other cannot be told apart from a single one there.

Before flt_otel_parse_cfg() returns, the instrumentation's YAML configuration
is validated by the wrapper's otelc_cfg_validate(), which parses the document
and probes the context-name resolution per signal without creating a library
context.  The library itself is initialized in the init callback that check
mode never runs, so this validation is what makes 'haproxy -c' reject a broken
YAML file.

4.1  Instrumentation Section

  otel-instrumentation <name>
      config <file> [context]
      log <target>
      debug-level <value>
      rate-limit <value>
      option { disabled | dontlog-normal | hard-errors | noflush | require-context }
      groups <name> ...
      scopes <name> ...
      acl <aclname> <criterion> ...

The instrumentation block defines global filter parameters: the YAML exporter
configuration file, logging, rate limiting, and references to groups and scopes.
Exactly one instrumentation block is allowed per filter instance.

4.2  Group Section

  otel-group <name>
      scopes <name> ...

Groups bundle multiple scopes under a single name for use with HAProxy
http-request/http-response rules via the "otel-group" action.  The group action
(group.c) parses the rule, resolves the scope references at check time, and
executes all referenced scopes when the rule fires.

4.3  Scope Section

  otel-scope <name>
      otel-event <event-name> [{ if | unless } <condition>]
      idle-timeout <time>
      extract <name-prefix> [use-vars | use-headers]
      span <name> [parent <ref>] [link <ref>] [root] [kind <kind>]
        link { <ref> ... | <ref> attr <key> <sample> ... } [{ if | unless } <condition>]
        attribute <key> <sample> ... [{ if | unless } <condition>]
        event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]
        baggage <key> <sample> ... [{ if | unless } <condition>]
        status <code> [<sample> ...] [{ if | unless } <condition>]
        exception <type> [message <sample> ...] [attr <key> <sample> ...] [{ if | unless } <condition>]
        inject <name-prefix> [use-vars] [use-headers]
      finish <name> ...
      instrument { <type> <name> ... | update <name> ... } [{ if | unless } <condition>]
      log-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ... [{ if | unless } <condition>]
      set-var <var-name> <sample> ... [{ if | unless } <condition>]
      set-var-ctx <var-name> <ref> <field> [{ if | unless } <condition>]
      unset-var <var-name> ... [{ if | unless } <condition>]
      acl <aclname> <criterion> ...
      otel-stop [{ if | unless } <condition>]

Each scope ties to a single HAProxy analyzer event (or none, if used only
through groups).  Scopes contain context extraction directives, span
definitions, metric instruments, log records, finish and otel-stop directives.

A span may specify:
  - A parent reference (another span or extracted context).
  - One or more links to other spans/contexts.  Inline link syntax allows one
    link on the span line; the standalone "link" keyword allows multiple.
  - The "root" mark, which declares the root span and changes nothing at run
    time.
  - Attributes, events, baggages and status evaluated from HAProxy sample
    expressions at runtime.
  - An inject directive to propagate the span context via HTTP headers and/or
    HAProxy variables.

4.4  Configuration Structure Initialization

All configuration structures are allocated and freed using macro-generated
functions from conf_funcs.h:

  FLT_OTEL_CONF_FUNC_INIT(type, id_field, max_len, extra_init)
  FLT_OTEL_CONF_FUNC_FREE(type, id_field, extra_free)

These macros produce flt_otel_conf_<type>_init() and _free() functions.
The init function:
  - Checks the identifier length against the limit of the type, which is
    FLT_OTEL_ID_MAXLEN (128) for a name and FLT_OTEL_LEN_UNLIMITED for the text
    of a sample expression.
  - Checks for duplicate identifiers in the target list.
  - Allocates the structure with OTELC_CALLOC.
  - Copies the identifier with OTELC_STRDUP.
  - Appends to the head list.
  - Executes any extra initialization (e.g., LIST_INIT for sub-lists in the
    span structure).

The free function:
  - Executes any extra cleanup (e.g., destroying sub-lists).
  - Frees the identifier string.
  - Removes the node from its list.
  - Frees the structure.

The full init/free chain for all structures:

  flt_otel_conf          flt_otel_conf_init() / flt_otel_conf_free()
    flt_otel_conf_instr  generated via macro
      flt_otel_conf_ph   generated (for ph_groups, ph_scopes)
    flt_otel_conf_group  generated
      flt_otel_conf_ph   generated (for ph_scopes)
    flt_otel_conf_scope  generated
      flt_otel_conf_context    generated
      flt_otel_conf_span       generated
        flt_otel_conf_link       generated
        flt_otel_conf_sample     generated + _init_ex()
          flt_otel_conf_sample_expr  generated
        flt_otel_conf_exception  generated
          flt_otel_conf_sample     generated (message, attributes)
      flt_otel_conf_str        generated (for spans_to_finish)
      flt_otel_conf_stop       generated (for the otel-stop lines)
      flt_otel_conf_instrument generated
      flt_otel_conf_log_record generated
        flt_otel_conf_sample     generated + _init_ex()
          flt_otel_conf_sample_expr  generated
      flt_otel_conf_sample     generated (for set_vars)
      flt_otel_conf_set_var_ctx  generated
      flt_otel_conf_unset_var  generated
        flt_otel_conf_str      generated (for the vars sublist)

  The conf_funcs.h macros also generate the leaf flt_otel_conf_str and
  flt_otel_conf_hdr structures (a plain string node and a header name/value
  pair, respectively), reused wherever a scope needs such a list -- for
  instance spans_to_finish and the unset-var directive's variable names.

4.5  Keyword Definition Rules

This section states the rules of the keywords of the OTel configuration file.
Breaking a rule stated here is an error unless the section says otherwise.

The paragraphs here hold for every keyword; each entry below adds what belongs
to that keyword alone.  An entry indented under 'span' works on the active span
and needs a 'span' line before it.  A section keyword opens its section and the
next section header ends it; every other keyword needs its own section open.

A keyword takes a trailing condition only when its entry names a scheme below,
or says so in words as 'otel-event' does.  Such a keyword repeats, one line per
condition, and the scheme says how the lines combine; 'otel-event' takes its
condition without repeating.  A keyword that takes no condition may repeat too,
when its entry says so.  Section 18.7 records the operation behavior behind the
schemes.  The parser cannot enforce this for 'acl', where everything after the
criterion is pattern text, so a condition written there is read as one more
pattern value.

  first-match - the filter applies the first line of a key whose condition holds
                and skips the rest.  The line without a condition is the default
                and comes last.
  apply-all   - the filter applies every line whose condition holds, in the
                order written.  The lines are separate items: a line without a
                condition always applies, and none of them has to come last.

Lines compete for the first match inside one otel-scope only, inside one span
for a keyword written under a span, and only when they name the same key or the
same name; a keyword that names neither lets them all compete.  A span named
again in another otel-scope starts over: the scopes run one after the other, so
that line is a later operation and not a repetition.

Whatever its scheme allows across the lines, a keyword may not name the same
thing twice on one line: a name, a reference or an attribute key written twice
in the same list is an error, and so is a repeated clause of the keyword, 'root'
or 'link' or 'unit' or 'time' among them.  The same name may still stand in two
lists of one line: a span may name one span as its 'parent' and as its inline
'link'.  A clause that opens one item of a list is written per item instead:
the 'attr' of a 'log-record' and of an 'exception' stands before every key, and
the same clause of a 'link' and of an 'instrument' update may be left out after
the first.  Several sample expressions after a key make one value together,
except after the key of an 'attr' clause and after a 'time' clause, each of
which takes one and reads the next word as the next argument of the line; the
'value' of an 'instrument' refuses a second, and a 'bounds' clause a repeated
boundary value.

Nothing may be called 'if' or 'unless', the filter id, the top-level scope name
and the context name of the 'config' line included, whether the line defines or
references the name.  The ban is on the two words as written: another case never
opens a condition and stays a plain name.  An ACL may not be called 'or' either,
whatever its case, since that word joins the terms of a condition.  A standalone
'link' reads 'attr' as its clause anywhere after the first reference and an
'instrument' update anywhere after its name, so a later link target and a key of
either may not be called 'attr'; an 'event' reads 'time' right after its name as
that clause, so its key may not be called 'time'.

A name is at most 127 characters long, every key and variable name included,
a variable counted with its scope prefix and the key inside a 'set-var-ctx'
field left aside; a longer one is refused with 'name too long'.  No length rule
reaches an ACL name, an 'event' name, an exception type, the description of an
'instrument', the text of a sample expression, the filter id, the context name
of the 'config' line or the top-level scope name, on the 'filter' line as in its
'[<name>]' declaration.

Some names carry a character rule too.  An 'inject' or an 'extract' prefix and a
'baggage' key take letters, digits, '_', '.' and '-' alone; an 'instrument' name
and the key inside a 'set-var-ctx' field follow the rules their entries state;
an ACL name, a section name, a top-level scope name and a variable name follow
HAProxy's own.  The other names take any character.

A reference names a span or an extract context that some otel-scope defines.
The 'parent' and the inline 'link' of a 'span', the standalone 'link', 'finish'
and 'set-var-ctx' take either, the 'span' of a 'log-record' a span alone.  The
'finish' wildcards '*', '*req*' and '*res*' stand for whatever the stream holds
and resolve on their own.

A condition looks each ACL name up in the otel-scope that holds the line, then
in the otel-instrumentation section, then in the HAProxy configuration, so a
scope ACL hides an instrumentation ACL of the same name and one condition may
mix the lists.  The ACL has to be defined by the time the line is read, so an
otel-instrumentation section below the otel-scope, or a HAProxy ACL below the
'filter' line, comes too late.

The parser reports most of these rules as it reads the file, naming the file and
the line.  A rule that pairs two lines of one section is reported on the second
of them: an 'idle-timeout' pairs with the scope event it needs, and so does a
'use-headers' written on an 'inject' or on an 'extract'.  A line that never
comes is reported on the section header or on the span once the section has
been read: the 'config' line of an otel-instrumentation, the 'scopes' line of
an otel-group, the 'otel-event' of a scope carrying an 'idle-timeout' or an
'inject use-headers', the 'idle-timeout' of an 'on-idle-timeout' scope, and
the name of an 'inject' written as '-', whose scope event may stand below the
line.  A top-level scope with no otel-instrumentation section is reported on the
'filter' line once the whole file has been read.  The rules that need the whole
configuration are checked once every section is read, with no line number in the
alert: the references to names defined elsewhere in the file, the 'groups' line
that every otel-group needs, the 'root' rules, the resolved 'inject' names, the
'extract' prefix no span may carry and the agreement of the create lines of one
'instrument'.  Rules this section does not state are checked there too: the
filter id that no other OTel filter of any proxy carries, the 'extract' context
'require-context' needs, the event a scope may carry while that option is set,
the HTTP-mode proxy that a header operation needs, and the filter, the group and
the instrumentation an 'otel-group' action names.

Some checks only warn.  Beside the ones that the 'span' entry names, they break
no rule stated here: an otel-scope no 'scopes' line names, an HTTP event on a
proxy that carries no HTTP message, a frontend-phase or the stream-stop event
for a filter of a backend section, a backend-phase request event on a listen
proxy, a condition or a sample expression the event of its scope, or the action
running it, cannot serve, a 'rate-limit' below 100.0, a 'filter' line with no
id, a 'debug-level' value that a build lacking the debug code ignores, and a
'use-vars' context whose name the HAProxy variable it generates cannot keep as
written.

A scope runs when a 'scopes' line of the instrumentation or of an otel-group
names it: at its event when it carries one, and from the 'otel-group' action
when a group holds it; the rules below mean this by a scope that runs.

A rule that depends on which scope runs first can fail only at execution.  A
'parent' whose span no scope has created yet, or whose extract context found
nothing, is a runtime error and the span carrying it is not created; the other
references do nothing when they do not resolve, and a 'log-record' whose 'span'
is not found is emitted without it.

The rules read one top-level scope, the one the 'filter' line selects; the other
scopes stay unread until a filter line selects them, and only the scope names
are checked over the whole file.  The skipping needs an open section: from the
first section header of the file on, an unread scope may hold anything, a broken
section header included, while a line above every header is an error HAProxy
itself reports.

A frontend's and a backend's filter meet on the streams routed between them,
which the configuration cannot foresee, so keeping their 'inject' names apart is
left to the writer.

Top-level OTel scope:

  [<name>] - the scope name is unique.  The line opens the OTel scope and the
             next [<name>] line ends the previous one.  A line outside of any
             scope is an error.

Section "otel-instrumentation":

  otel-instrumentation - the section is required and stands once in a top-level
                         scope, whatever its name.
  acl                  - <aclname> is unique in the instrumentation; several
                         lines are allowed, and the name may be used in every
                         otel-scope.
  log                  - allowed only once, a prefix included.  The filter reads
                         the prefix too and leaves its own logging off for 'no',
                         on for 'default'.
  config               - required, and allowed only once.
  groups               - <name> is unique in the instrumentation and must be a
                         defined otel-group section; several lines are allowed.
  scopes               - <name> is unique in the instrumentation and must be a
                         defined otel-scope section; several lines are allowed.
  rate-limit           - allowed only once.
  option               - each option is allowed only once, and a prefix does not
                         lift the limit.  The 'no' and 'default' prefixes keep
                         the meaning HAProxy gives them.
  debug-level          - allowed only once.

Section "otel-group":

  otel-group - <name> is unique; another top-level scope may reuse the name.
               The section must hold at least one 'scopes' line, and a 'groups'
               line of the otel-instrumentation must name the group.
  scopes     - <name> is unique in the group; several lines are allowed.  A name
               must be a defined otel-scope section.

Section "otel-scope":

  otel-scope   - <name> is unique; another top-level scope may reuse the name.
  span         - the first line of a <name> in a scope creates the span, with or
                 without creation arguments, and every line of the name makes
                 it the active span for the operations that follow.  A later
                 line of the name in that scope must be bare: one with creation
                 arguments, a defining line, is a repeated definition.  Another
                 otel-scope may carry a defining line of the same name too: at
                 execution the first line to reach the span creates it, bare
                 or defining, a later bare line re-activates it and another
                 defining line is a runtime error.  The creation line carries
                 one inline 'link' at most, and the standalone keyword adds
                 the rest.  The 'parent' may not name the span itself, which
                 is looked up before it exists.  The 'root' mark declares the
                 root span and changes nothing at execution: a span without
                 a 'parent' starts a trace of its own, marked or not.  Only
                 one span name may carry the mark, the same name marked root
                 in another otel-scope being the usual alternative creation.
                 Leaving every span unmarked is not an error, only a warning,
                 and only the spans of the scopes that run count towards it.  A
                 scope that a 'scopes' line names but that never runs draws a
                 warning of its own.  The 'parent' of a marked span may name an
                 extract context, the remote parent of the trace, never a span.
                 A span may not be called by one of the 'finish' wildcards.
    link       - apply-all scheme.
    attribute  - first-match scheme per <key>.
    event      - the lines are gathered by name: each name makes one span event
                 whose attributes are the keys of its lines.  The (name, key)
                 pairs are independent items under the apply-all scheme, while
                 within one pair the value follows the first-match scheme.  The
                 event timestamp comes from the first applied line whose 'time'
                 clause evaluates, and the later lines keep it.
    baggage    - first-match scheme per <key>.
    inject     - no other 'inject' of the configuration, or of the other OTel
                 filters of the proxy, may resolve to the name this one does,
                 one differing only in case included: both carriers keep the
                 name in lower case.  Several lines are allowed, but at most
                 one per span of one otel-scope, so a span that another scope
                 re-activates may carry a second context there.  A '-' line
                 follows the name rule with its resolved name, and at most one
                 context whose name opens with '-', written or resolved, may
                 inject headers on the proxy: the nameless headers would collide
                 whatever the names are.  The resolved name is measured like a
                 written one, the mark counted with it, and the span name it
                 takes when the scope has no event is checked like a written
                 prefix.  Injecting headers, which a line with no storage word
                 does, needs an event that can still change them: any other
                 event is an error, and so is a scope with no event.
    status     - first-match scheme for the span's single status.
    exception  - apply-all scheme, one exception event per applied line.
  extract      - <name-prefix> is unique within its scope; several lines are
                 allowed, and another otel-scope may reuse the name.  No span of
                 the configuration may carry the name: a reference resolves to
                 the span first.  'use-headers' needs a channel to read, so it
                 is an error on an event that carries none; a scope with no
                 event is exempt, the 'otel-group' action that runs it carrying
                 one.
  finish       - <name> is unique within its scope; several lines are allowed,
                 and another otel-scope may finish the same span.
  otel-stop    - first-match scheme.
  instrument   - the create form follows the first-match scheme per its <name>
                 within one otel-scope.  The update form is apply-all for every
                 instrument type, and records the value of the create lines of
                 its own otel-scope, of the scope that created the instrument
                 where its own carries none, and until the instrument exists of
                 the first scope of the file that runs and defines the name.
                 An update needs a create line of its name in some scope of
                 the file, and an update in a scope that runs needs one in a
                 scope that runs, its own or another one.  The lines of one
                 name are one instrument, across the scopes too and with the
                 case of the name folded, so they must agree on the type, the
                 aggregation, the description, the unit and the bounds, while
                 the value and the condition are what the repeated lines vary;
                 only one scope ever creates the instrument, and a create line
                 of another scope that finds it created is a runtime error.  The
                 name begins with a letter and carries only letters, digits
                 and the punctuation '_', '.', '-' and '/', while the unit is
                 at most 63 ASCII characters, both rules of the metric SDK,
                 which records nothing for an instrument it refuses.  A 'bounds'
                 clause stands on a hist_int instrument alone.
  log-record   - apply-all scheme, one record per applied line.  The 'id' and
                 'event' clauses are given together or both omitted.
  idle-timeout - allowed only once, and it belongs to the 'on-idle-timeout'
                 event: a scope bound to that event must carry it, and one bound
                 to another event, or to no event at all, may not.
  acl          - <aclname> is unique within its scope and may be used in that
                 scope alone; several lines are allowed, and another otel-scope
                 may reuse the name.
  otel-event   - the keyword may be given only once in a scope, whatever its
                 name, optionally with a condition.
  set-var      - first-match scheme per <var-name>.
  set-var-ctx  - first-match scheme per <var-name>.  The W3C rules reach the
                 key inside a 'baggage' field, an HTTP token of at most 4096
                 characters, and the one inside a 'tracestate' field: at most
                 256 lower-case letters, digits, '_', '-', '*' and '/' opening
                 with a letter, or a tenant id and a system id of at most 241
                 and 14 around one '@', the tenant opening with a letter or a
                 digit and the system id with a letter.  The other fields take
                 no key.
  unset-var    - first-match scheme per <var-name>.

5  Filter Lifecycle
----------------------------------------------------------------------

The filter registers its operations in the flt_otel_ops structure (filter.c)
and the keyword parser via INITCALL1 (parser.c).

5.1  Proxy-Level Initialization

  flt_otel_ops_init():
    - Registers CLI commands via flt_otel_cli_init().
    - Initializes the OpenTelemetry library via flt_otel_lib_init(): verifies
      the C wrapper version, resolves the absolute path of the YAML
      configuration file, calls otelc_init() to set up exporters, creates the
      tracer, meter and logger objects, and registers custom memory allocation
      and thread-id callbacks with the wrapper via otelc_ext_init().
    - Sets the FLT_CFG_FL_HTX capability flag only for an HTTP-mode proxy, so
      HAProxy attaches the filter to HTX streams; a TCP-mode proxy leaves it
      unset and the filter runs on the raw stream.

  flt_otel_ops_check():
    - Resolves the sample fetch arguments set aside while the OTel file was
      parsed, with the frontend and the backend capabilities combined so that a
      backend-only fetch passes on a frontend proxy; an argument that does not
      resolve is rejected.
    - Validates that filter IDs are unique across all proxies.
    - Resolves group->scope and instrumentation->scope/group placeholder
      references to actual configuration structures (setting the ptr field
      and flag_used).
    - Rejects an otel-group that no 'groups' line of the instrumentation names.
    - Warns about unused scopes, a scope that neither an event nor a group runs,
      and a missing root span; rejects a second root span name and a root span
      parented under a span.
    - Validates metric instruments: binds the lines of one name to the create
      line that owns its single creation, taken from a scope that runs, and
      places the instrument in that scope until one creates it; rejects a
      create-form name that repeats with another instrument type or an otherwise
      differing definition, and an update whose create forms all stand in scopes
      that never run.
    - Rejects a resolved inject context name that another span already carries,
      in this instance or in another OTel filter of the proxy, and a second
      nameless-header context anywhere on the proxy.
    - Rejects an 'extract' context name that a span of the configuration already
      carries.
    - Rejects a 'parent', 'link', 'finish', 'set-var-ctx' or 'log-record' name
      that no otel-scope defines as a span, or as an extracted context where
      one may stand.
    - Rejects a used scope whose event runs before the request context can be
      read while 'require-context' is set, and a configuration where no used
      scope extracts one at all.
    - Rejects an 'inject' or an 'extract' that carries the context in HTTP
      headers on a proxy that is not in HTTP mode.
    - Warns about a used scope bound to an event the proxy keeps from firing: an
      HTTP-phase event on a proxy that is not in HTTP mode, on-stream-start or a
      frontend-phase event for a filter of a backend section, the body event
      among them unless that backend asks for the body itself, on-stream-stop
      for such a filter, which HAProxy releases before that callback, and the
      backend-phase request events on a listen proxy, which fire there only for
      a stream switched to another backend or routed in from another frontend.
    - Computes the aggregated analyzer bitmask from all used scopes.

  flt_otel_ops_init_per_thread():
    - Starts the tracer, meter and logger background threads on first call.
    - Uses an atomic claim on instr->flag_started to guarantee that start runs
      on a single thread only; the FLT_CFG_FL_HTX flag is set elsewhere, in
      flt_otel_ops_init(), not here.

  flt_otel_ops_deinit():
    - Force-flushes the tracer, meter and logger within one shared
      budget, or sets a zero flush budget instead under 'option noflush'.
    - Destroys the tracer, meter and logger.
    - Frees the entire configuration tree.
    - Calls otelc_deinit() to shut down the wrapper library.

5.2  Stream-Level Callbacks

  flt_otel_ops_attach():
    - Checks if the filter is globally disabled; returns IGNORE.
    - Applies rate limiting via ha_random32(); returns IGNORE if the random
      value exceeds the configured rate_limit.
    - Creates the runtime context (flt_otel_runtime_context_init) with a
      generated UUID and initialized span/context lists.
    - Sets pre_analyzers and post_analyzers bitmasks from the instrumentation's
      aggregated analyzer flags.  AN_REQ_WAIT_HTTP and AN_RES_WAIT_HTTP are
      placed in post_analyzers because those analyzers can only be used in the
      post_analyze callback.  AN_REQ_HTTP_TARPIT stays in pre_analyzers; it is
      left out only when channel_start_analyze force-injects the pre_analyzers
      into the channel, so the tarpit event fires only when a tarpit rule has
      armed that analyzer itself.
    - Arms the idle timer from the precomputed minimum idle_timeout and sets
      the first wake-up on the stream task.  The stream-start callback would
      miss a filter attached at backend selection, as HAProxy runs it for the
      frontend filters alone.

  flt_otel_ops_detach():
    - Frees the runtime context, which finishes all remaining active spans and
      destroys all remaining contexts.

  flt_otel_ops_check_timeouts():
    - Disarms the idle timer when the filter is disabled for the stream.
    - Checks whether the idle-timeout timer has expired; if so, fires the
      on-idle-timeout event and reschedules the timer for the next interval
      (disarming instead when the event itself disabled the filter).
    - Sets STRM_EVT_MSG on the stream's pending_events to ensure the filter is
      re-evaluated after a timeout.
    - Re-asserts the idle wake-up on the stream task expiry on every pass.

5.3  Error Handling

Helper functions manage errors:

  flt_otel_return_int() / flt_otel_return_void():
    - If the result indicates an error or an error string is set: in hard-error
      mode, the filter is disabled for the current stream (flag_disabled = 1)
      and the disabled counter is incremented atomically.  In soft-error mode,
      the error is merely logged.
    - The error string is always freed.
    - For int returns, FLT_OTEL_RET_OK is returned regardless, so the stream
      continues processing even after an error.


6  Event Processing (Channel Analyzers)
----------------------------------------------------------------------

The filter maps HAProxy channel analyzer callbacks to a table of named events
defined in event.h (FLT_OTEL_EVENT_DEFINES).

6.1  Event Table

Each event entry carries:
  - an_bit:           the HAProxy analyzer bit (AN_REQ_*, AN_RES_*)
  - an_name:          the analyzer bit name (e.g. "AN_REQ_HTTP_PROCESS_FE")
  - smp_opt_dir:      sample fetch direction (REQ or RES); an event that runs
                      on no channel carries neither, and its samples are fetched
                      on the request side
  - smp_val_fe/be:    SMP_VAL_FE_*/SMP_VAL_BE_* fetch-location masks for the
                      event's processing point, paired per event as the SPOE
                      filter pairs its own; flt_otel_ops_check() ORs the two
                      into the location a bound scope's fetches and conditions
                      are validated against.  The values are not everywhere the
                      same as SPOE's: on-client-session-start carries the HTTP
                      request header location, where SPOE carries the connection
                      accept one, because the HTTP fetches do resolve at that
                      point on an HTTP-mode proxy
  - flag_http_inject: whether span context can be injected into HTTP headers
                      at this point
  - flag_http_extract: whether span context can be extracted from HTTP headers
                      at this point, which needs a channel to read
  - flag_http_only:   whether the event fires only on an HTTP-mode proxy
  - flag_context:     whether the incoming trace context can be read at this
                      point, which 'require-context' needs
  - name:             configuration event name (e.g. "on-frontend-http-request")

Events with an_bit == 0 are pseudo-events not tied to any channel
analyzer.  The stream lifecycle callbacks fire:
  - on-stream-start  (flt_otel_ops_stream_start, before channel processing)
  - on-stream-stop   (flt_otel_ops_stream_stop, after channel processing)

The check_timeouts callback fires periodically:
  - on-idle-timeout  (flt_otel_ops_check_timeouts, when stream is idle)

The stream_set_backend callback fires:
  - on-backend-set   (flt_otel_ops_stream_set_backend, when backend is assigned)

The HTTP lifecycle callbacks fire:
  - on-http-headers-request / on-http-headers-response (flt_otel_ops_http_headers)
  - on-http-end-request / on-http-end-response         (flt_otel_ops_http_end)
  - on-http-reply                                      (flt_otel_ops_http_reply)

But on-http-reply never fires on current HAProxy; section 6.2 says why, and the
full explanation is in README and README-configuration.

The remaining pseudo-events fire from channel start/end callbacks:
  - on-client-session-start / on-client-session-end
  - on-server-session-start / on-server-session-end
  - on-server-unavailable

The events on-stream-start and on-stream-stop pass a NULL channel argument, so
neither context injection nor extraction via HTTP headers can be used there,
while on-idle-timeout and on-backend-set are handed the request channel to read,
which allows extraction but not injection.  These events fetch their samples
with a direction that is neither the request nor the response one, so that the
'finish' wildcards leave their spans alone and a sample fetched there reads the
request side.

6.2  Callback Flow

  attach(s, f):
    - Fires no event; see 5.2.  Runs for a filter of a backend section as well,
      at backend selection.

  stream_start(s, f):
    - Fires on-stream-start with chn=NULL.
    - Called when a new stream begins, before any channel processing, for the
      filters of the stream's frontend alone: a filter of a backend section
      never sees it.

  stream_set_backend(s, f, be):
    - Fires on-backend-set with chn=&s->req.
    - Called when a backend is assigned, the frontend itself included.

  stream_stop(s, f):
    - Fires on-stream-stop with chn=NULL.
    - Called when a stream is destroyed, after all channel processing, over the
      filters still attached: a filter of a backend section is released at the
      end of the channel analysis and never sees it.

  check_timeouts(s, f):
    - Fires on-idle-timeout with chn=&s->req when the idle timer expires, and
      reschedules it; see 5.2.

  channel_start_analyze(chn):
    - Records on the response channel that its analysis started, the sign that
      a server was reached.
    - Enables the per-channel analyzers from pre_analyzers (except
      AN_REQ_HTTP_TARPIT, which only an armed tarpit rule may enable), on an HTX
      stream only, and for a filter attached at backend selection only those
      past the backend start, so the frontend rules do not run a second time.
    - Fires on-client-session-start (request) or on-server-session-start
      (response).

  channel_pre_analyze(chn, an_bit):
    - Looks up the event by an_bit in the event table.
    - Calls flt_otel_event_run() for the matching event.

  channel_post_analyze(chn, an_bit):
    - Same as pre_analyze but for post-analyzers (AN_REQ_WAIT_HTTP,
      AN_RES_WAIT_HTTP).

  channel_end_analyze(chn):
    - Fires on-client-session-end (request) or on-server-session-end (response).
    - On the channel whose analysis ends last: if the response channel never
      began its analysis (no server was reached), fires on-server-unavailable.

  http_headers(s, f, msg):
    - Fires on-http-headers-request or on-http-headers-response depending on
      msg->chn direction.

  http_end(s, f, msg):
    - Fires on-http-end-request or on-http-end-response depending on
      msg->chn direction.

  http_reply(s, f, status, msg):
    - Fires on-http-reply with chn=&s->res.
    - Never reached on current HAProxy: HAProxy dropped the flt_http_reply()
      call in 2.2-dev8 (commit 8dfeccf6d, 2020), so this callback has no caller.

6.3  Scope Execution

  flt_otel_event_run() (event.c):
    - Captures timestamps (CLOCK_MONOTONIC + CLOCK_REALTIME).
    - Iterates all scopes matching the event; calls flt_otel_scope_run() for
      each used scope.

  flt_otel_scope_run() (event.c):
    1. Evaluates the scope's ACL condition and, if it does not hold, returns
       without processing.
    2. Extracts contexts: for each configured extract directive, reads
       the span context from HTTP headers or HAProxy variables via
       flt_otel_scope_context_init().
    3. Runs "set-var" directives via flt_otel_scope_run_set_var(), setting
       HAProxy variables from sample expressions (before any span runs).
    4. Processes spans: for each configured span:
       a. Calls flt_otel_scope_span_init() which either returns an existing
          scope_span (by name) or creates a new one with resolved parent
          reference.
       b. Calls flt_otel_scope_span_start() which creates the OTel span via
          tracer->start_span_with_options(), unless an earlier scope created
          it already, and sets flag_norec when the sampler left the new span
          out.
       c. Resolves span links against the runtime context -- first searching
          active spans, then extracted contexts.  Unresolved links are skipped
          (a debug build traces them; a release build stays silent).
       d. Evaluates the attribute, event and status samples through
          flt_otel_scope_span_samples(), and the baggage samples in place, both
          of them via flt_otel_sample_add().
       e. Calls flt_otel_scope_run_span() which:
          - Adds all resolved links via span->add_link().
          - Sets baggage, attributes, events, and status.
          - Records the configured exceptions.
          - Optionally injects the span context into HTTP headers and/or
            HAProxy variables.

       A span whose flag_norec is set skips the links of step c, the samples
       of step d except the baggages, and the exceptions of step e, all of
       which the SDK would discard.  The span itself is created and injected
       all the same, so the trace still reaches the downstream services with
       the sampled flag cleared.
    5. Runs "set-var-ctx" directives via flt_otel_scope_run_set_var_ctx(),
       storing a field of a referenced span or context into a HAProxy
       variable (after the spans have run, so their contexts are available).
    6. Processes metric instruments via flt_otel_scope_run_instrument(), which
       runs two passes: the first lazily creates the create-form instruments
       whose condition passes, using HA_ATOMIC_CAS for thread-safe one-time
       creation; the second records measurements for update-form instruments,
       creating on the spot one not created yet (UNSET), waiting for one that
       another thread is creating (PENDING) and skipping one given up after
       repeated creation failures (FAILED).
    7. Emits log records via flt_otel_scope_run_log_record(), which iterates
       the scope's log-record list, skips entries below the logger's severity
       threshold, evaluates sample expressions into a body string, resolves
       the optional span reference, and emits the record via the logger.
    8. Runs "unset-var" directives, removing the named HAProxy variables via
       flt_otel_var_unset_byname() (each guarded by its optional condition); a
       variable that an earlier passing line already names is skipped per the
       first-match rule, checked via flt_otel_unset_var_taken().
    9. Marks spans listed in "finish" directives; a fired "otel-stop" marks
       every open span and context for completion.
   10. Calls flt_otel_scope_finish_marked() to end marked spans/contexts.
   11. Calls flt_otel_scope_free_unused() to remove finished and destroyed
       scope_span/scope_context entries from the runtime lists.
   12. If "otel-stop" fired, sets rt_ctx->flag_disabled = 1; the guard at the
       top of flt_otel_scope_run() then skips the stream's remaining scopes.


7  Runtime Data Structures
----------------------------------------------------------------------

7.1  Runtime Context (per stream)

  flt_otel_runtime_context:
    stream         Owning stream pointer.
    filter         Owning filter pointer.
    uuid[40]       Generated UUID v4 for the session.
    flag_harderr   Copied from instrumentation config.
    flag_disabled  Set when the filter encounters a hard error or an 'otel-stop'
                   directive fires, and by 'require-context' when no extract
                   of a scope yields a valid context.
    flag_ctx_valid Set once an extract yields a valid upstream span context;
                   read with 'require-context' only.
    flag_res_started Set when the response channel starts its analysis, that is
                     once a server connection is established.
    logging        Logging flags.
    idle_timeout   Idle timeout interval in milliseconds (0 = off).
    idle_exp       Tick at which the next idle timeout fires.
    bytes_in       Raw payload bytes seen on the request channel (TCP mode).
    bytes_out      Raw payload bytes seen on the response channel (TCP mode).
    spans          Linked list of flt_otel_scope_span.
    contexts       Linked list of flt_otel_scope_context.

7.2  Scope Span

  flt_otel_scope_span:
    id / id_len    Span operation name (borrowed from config).
    smp_opt_dir    Direction in which the span was created.
    flag_finish    Set by finish directives, cleared after ending.
    flag_norec     Set when the created span reports that it does not record,
                   which keeps its samples from being evaluated.
    span           The OTel span object (NULL before start, NULL after
                   end_with_options).
    ref_span       Parent span pointer (resolved at init).
    ref_ctx        Parent context pointer (resolved at init).
    list           Chain in runtime_context.spans.

  flt_otel_scope_span_init() performs memoization: if a span with the same name
  already exists in rt_ctx->spans, it returns the existing entry.  This allows
  multiple scopes to contribute attributes/events to the same logical span.

7.3  Scope Context

  flt_otel_scope_context:
    id / id_len    Context name (borrowed from config).
    smp_opt_dir    Direction in which the context was extracted.
    flag_finish    Marks the context for destruction.
    context        The OTel span_context object.
    baggage        The inbound baggage carrier, or NULL (used by set-var-ctx).
    list           Chain in runtime_context.contexts.

  Similarly memoized: duplicate extraction of the same context name returns the
  existing entry.

7.4  Scope Data (per span per scope run, stack-allocated)

  flt_otel_scope_data:
    baggage        Key-value array for baggage items.
    attributes     Key-value array for span attributes.
    events         Linked list of flt_otel_scope_data_event (each with name,
                   an optional per-event timestamp in ts / ts_set, and a
                   key-value array).
    links          Linked list of flt_otel_scope_data_link (each with span
                   and/or context pointer).
    status         Status code and description string.

  Initialized at the start of each span processing block and freed at the end.
  The link entries hold borrowed pointers to the OTel span/context objects
  owned by the runtime context, so those are not freed; each link's own
  attribute key-value array is owned by the link, however, and is destroyed
  via otelc_kv_destroy() before the link node itself is freed.

7.5  Span Finishing

  finish <name> / finish * / finish *req* / finish *res*

  The "finish" directive marks spans and contexts for completion:
    - "*" marks all.
    - "*req*" / "*res*" marks those created in the request/response direction
      respectively.
    - Otherwise, marks by exact name.

  flt_otel_scope_finish_marked() iterates all marked entries:
    - Spans are ended via span->end_with_options() which NULLs the span pointer;
      an entry whose span creation failed carries none and is skipped.
    - Contexts are destroyed via context->destroy() which NULLs the context
      pointer.

  flt_otel_scope_free_unused() then removes entries with NULL span/context
  pointers from the runtime lists.  For contexts, associated HTTP headers
  and variables are also cleaned up.

  On stream detach (flt_otel_runtime_context_free), any remaining active spans
  are force-ended and all entries are freed.


8  Span Links
----------------------------------------------------------------------

Span links associate a span with other spans or contexts without establishing
a parent-child relationship.

8.1  Configuration

Two syntaxes are supported:

  Inline (one link per span declaration):
    span <name> [parent <ref>] link <ref> [root]

  Standalone (multiple links, or one link with attributes):
    link { <ref> [<ref> ...] | <ref> attr <key> <sample> ... } [{ if | unless } <condition>]

The flt_otel_conf_link structure stores each link target name and, for the
single-link form, a list of attribute samples.  Each line is parsed on its own
private list to bypass the duplicate-name check, so the same target may be named
on several lines, while a repeated target within one line is still rejected.
The links list is initialized in flt_otel_conf_span_init() and destroyed in
flt_otel_conf_span_free().  The optional 'if'/'unless' condition is stored on
each conf_link; on a multi-name line the parser builds it into every created
link, so each of them is guarded independently.

8.2  Runtime Resolution

At scope execution time (event.c, flt_otel_scope_run), for each configured link:
  1. The optional 'if'/'unless' condition is evaluated; when it does not pass,
     the link is skipped before any name resolution.
  2. The name is searched in rt_ctx->spans (active scope_span entries).
     If found, the OTel span pointer is captured.
  3. If not found in spans, the name is searched in rt_ctx->contexts (extracted
     scope_context entries).  If found, the OTel span_context pointer is
     captured.
  4. If neither is found, the link is skipped (a debug build traces the
     miss; a release build stays silent).
  5. A flt_otel_scope_data_link node is allocated and appended to the scope
     data's links list.

In flt_otel_scope_run_span(), all resolved links are applied via
span->add_link(span, link_span, link_context, link->attributes.attr,
link->attributes.cnt).  The last two arguments carry the link's resolved
attribute array and count, so a link can carry its own attributes (from the
"link <ref> attr <key> <sample> ..." form); they are empty when the link was
declared without attributes.


9  Context Propagation
----------------------------------------------------------------------

9.1  Extraction

  extract <name-prefix> [use-vars | use-headers]

Extracts an incoming trace context.  The prefix identifies the header name
pattern (for HTTP) or variable name pattern (for vars).

  - use-headers (default): flt_otel_http_headers_get() iterates HTX headers
    matching the prefix and builds an otelc_text_map.
  - use-vars: flt_otel_vars_get() reads HAProxy variables matching the prefix
    pattern.

The text map is passed to flt_otel_extract_http_headers() which uses the
C wrapper to reconstruct an otelc_span_context.

9.2  Injection

  inject <name-prefix> [use-vars] [use-headers]

Injects the current span's context into outgoing data.  Both storage types can
be used simultaneously.

  flt_otel_inject_http_headers() serializes the span context into an
  otelc_http_headers_writer which produces a text_map.  For each key-value pair:
    - use-headers: flt_otel_http_header_set() adds/replaces the header with the
      prefixed name.
    - use-vars: flt_otel_var_register() + flt_otel_var_set() stores the value
      in a HAProxy transaction variable with normalized name (dashes replaced
      with 'D', spaces with 'S', uppercase lowered; dots serve as component
      separators).


10  HTTP Header Manipulation
----------------------------------------------------------------------

  http.c provides these operations:

  flt_otel_http_headers_get(chn, prefix, prefix_len, err):
    Iterates the HTX message headers.  Headers whose name starts with the given
    prefix are collected into an otelc_text_map.  The prefix is stripped from
    the names in the returned map; a header whose name is only the prefix is
    skipped, as the stripped name would be empty and a zero-length key is read
    as a NUL-terminated string.

  flt_otel_http_header_set(chn, prefix, name, value, err):
    Removes any existing header matching "prefix" + "name", then adds a new
    header with the given value.  If name is NULL, all headers with the prefix
    are removed (bulk delete).  A prefixed name that does not fit the build
    buffer is rejected with a runtime error instead of being truncated.

  flt_otel_http_headers_remove(chn, prefix, err):
    Convenience wrapper; removes all headers matching the prefix.


11  HAProxy Variable Integration
----------------------------------------------------------------------

Enabled with OTEL_USE_VARS=1.  Provides an alternative propagation mechanism
using HAProxy transaction-scoped variables, with the names normalized as 9.2
describes.  In a build without USE_OTEL_VARS_NAME, a meta-variable tracks the
list of context variable names so they can be enumerated for extraction; with
it, the variable store is scanned directly by name prefix.

Key functions:
  flt_otel_var_register()   Registers a variable with HAProxy.
  flt_otel_var_set()        Sets a variable value.
  flt_otel_vars_get()       Reads all context variables into a text_map for
                            extraction.
  flt_otel_vars_unset()     Removes all context variables.


12  Group Action Integration
----------------------------------------------------------------------

The "otel-group" HAProxy action allows triggering trace scopes from
tcp-request, tcp-response, http-request, http-response and
http-after-response rules:

  tcp-request         otel-group <filter-id> <group-name>
  tcp-response        otel-group <filter-id> <group-name>
  http-request        otel-group <filter-id> <group-name>
  http-response       otel-group <filter-id> <group-name>
  http-after-response otel-group <filter-id> <group-name>

  group.c implements:
    flt_otel_group_parse():   Parses the action arguments.
    flt_otel_group_check():   Resolves filter and group references.
    flt_otel_group_action():  At runtime, finds the OTel filter in the stream,
                              iterates all scopes in the group, and calls
                              flt_otel_scope_run() for each.


13  Memory Management
----------------------------------------------------------------------

  pool.c provides wrappers around HAProxy memory pools and standard
  allocation:

  flt_otel_pool_alloc()   Allocates from a pool (when non-NULL) or the heap.
  flt_otel_pool_free()    Returns memory to the pool or frees it.
  flt_otel_trash_alloc()  Acquires a trash buffer chunk.
  flt_otel_trash_free()   Releases a trash buffer chunk.

Pool heads are registered in pool.c for the hot-path structures:
  - otel_scope_span       (allocated in scope.c)
  - otel_scope_context    (allocated in scope.c)
  - otel_runtime_context  (allocated in scope.c)
  - otel_span_context     (allocated by the C wrapper through the
                          otelc_ext_init callbacks in filter.c)

The wrapper library's memory allocations are redirected through
flt_otel_mem_malloc() / flt_otel_mem_free(), which use the otel_span_context
pool.


14  CLI Interface
----------------------------------------------------------------------

  cli.c registers commands under "flt-otel" for runtime control:
  - Introspection: status, instruments and scopes reports.
  - Runtime switches: enable/disable, hard-errors/soft-errors, logging,
    noflush, rate, reset-errors, and -- in a debug build -- the debug
    level.
  - Data control: flush, which force-exports the buffered telemetry.

Logging can be independently controlled via the instrumentation's logging
flags (ON, NOLOGNORM).  Log output goes to the log servers configured in the
instrumentation block.


15  Debug Infrastructure
----------------------------------------------------------------------

When compiled with OTEL_DEBUG=1 (DEBUG_OTEL defined), the filter enables:

  - The debug-only flt_ops callbacks deinit_per_thread, http_payload and
    http_reset are selected via OTELC_DBG_IFDEF() and set to NULL in non-debug
    builds.  (Note: stream_set_backend, http_headers, http_end and http_reply
    are registered unconditionally and back the on-backend-set,
    on-http-headers-*, on-http-end-* and on-http-reply events; tcp_payload is
    likewise unconditional and counts the bytes forwarded on a TCP-mode stream.
    None of these is debug-only.)

  - The OTELC_DBG() macro produces debug output at various levels.

  - flt_otel_scope_data_dump() dumps the complete scope data (baggage,
    attributes, events, links, status) for inspection.

  - Event usage counters (per-event htx_is_empty statistics) are maintained and
    printed at deinit.

  - Pool size information is printed at startup.

The debug level is a bitmask that can be adjusted at runtime via the CLI.


16  Test Infrastructure
----------------------------------------------------------------------

16.1  Test Scenarios

  sa    Standalone: comprehensive test exercising most request and response
        events (six are left to the 'full' configuration), span links (both
        inline and standalone syntax), events with data capture, baggage, and
        the full span hierarchy from client session start to server session
        end.

  full  Full event coverage: extends sa with the remaining lifecycle events,
        covering every filter event except on-http-tarpit-request.

  fe    Frontend-only: tests the request-side span chain with context injection
        into HTTP headers.

  be    Backend-only: tests context extraction from HTTP headers and
        response-side processing.  Designed to run as the backend of
        the fe/ test.

  ctx   Context propagation: deep nesting test that verifies context propagation
        via both HTTP headers and HAProxy variables.

  cmp   Comparison: simplified configuration made for comparison with other
        tracing implementations.

  tcp   TCP mode: traces the raw connection on a TCP-mode proxy and counts the
        forwarded payload via the otel.bytes_in / otel.bytes_out fetches.

  updown Up-down counter: two udcnt_int instruments, one counting the client
         sessions and one recording a signed per-session delta of +1 on client
         session start and -1 on session end, so its value tracks the number of
         active client sessions.

  err   Error logging: the response scope parents its span under an 'orphan
        span' created only by on-server-unavailable, so span creation fails
        on every response and drives the rate-limited error/warning logging
        and the CLI-reported counters.

  empty Minimal: validates that an empty configuration (only the
        instrumentation block, no scopes) does not crash.

  parser Configuration parser: a table-driven set of 'haproxy -c' runs that
         checks the keyword definition rules of section 4.5; each case is one
         generated configuration file that has to be accepted or refused with
         a given alert.  No proxy is started and no socket is opened, so the
         runtime half of the rules, the first-match evaluation of the lines
         of one key, is left to the scenarios above.  Further cases cover the
         arguments each keyword accepts and the validation that runs once the
         whole configuration is known.  See test/README-parser.

16.2  Test Runners

All runners are POSIX shell scripts (/bin/sh) meant to be run from the test/
directory.  The run-*.sh runners accept an optional HAProxy binary path, a
pidfile and a log file that replaces their timestamped default name under
test/_logs/ (run-fe-be.sh appends the -fe and -be suffixes to the given
name); test-speed.sh takes [-b backend] [-d duration] [-r rate-limits] and
a configuration name instead, and run-parser.sh takes [-k pattern] [-x] [-X]
[-v] followed by the same optional binary path.

  run-test-config.sh  Single-instance runner: derives the configuration
                directory from the name it is invoked under (run-<dir>.sh).
                run-sa.sh, run-full.sh, run-cmp.sh, run-ctx.sh, run-tcp.sh,
                run-updown.sh, run-err.sh, run-empty.sh and run-optctx.sh are
                symlinks to it, each starting one HAProxy instance with its
                own config.
  run-fe-be.sh  Launches two HAProxy instances (frontend on port 10080, backend
                on port 11080) forming a trace propagation chain.  Handles
                graceful shutdown via SIGUSR1.  A separate script, not a
                symlink.
  run-parser.sh Runs the configuration parser cases of parser/, one 'haproxy
                -c' per case over a generated configuration file.  It starts
                no proxy, and the cases are described in test/README-parser.
  test-speed.sh Runs performance benchmarks for one or all configurations.

  copy-yml.sh   Transforms a template YAML configuration by replacing
                placeholders with test-specific values (service names, file
                suffixes, etc.).
  otlp_http-recorder.py  Records incoming OTLP/HTTP requests to files, with
                optional forwarding to an upstream collector.
  otlp_http-replay.py    Replays previously recorded OTLP/HTTP requests to a
                collector.

16.3  Exporter Configuration

Each test directory contains an otel.yml file configuring exporter types:
  - OTLP file exporter (writes traces to local files).
  - OTLP gRPC exporter (sends to localhost:4317).
  - OTLP HTTP exporter (sends to localhost:4318 in JSON format).


17  Notable Design Decisions
----------------------------------------------------------------------

  - Span memoization: flt_otel_scope_span_init() and
    flt_otel_scope_context_init() return an existing entry of the same name, so
    several scopes contribute data to one logical span across analyzer events.

  - Lazy span creation: the OTel span object is created on first use in
    flt_otel_scope_span_start(), not at scope_span_init time, and before the
    sample evaluation, so that the sampling decision of the span is known while
    its samples are still unevaluated.

  - Soft/hard error modes: in soft mode, errors are logged but the stream
    continues with tracing effectively abandoned for that span.  In hard mode,
    the filter disables itself for the rest of the stream.  Either way, stream
    processing is never interrupted by a tracing failure (FLT_OTEL_RET_OK is
    always returned).

  - Rate limiting uses a uint32 representation of a percentage
    (FLT_OTEL_FLOAT_U32), compared against ha_random32() for uniform
    distribution without floating-point at runtime.

  - Server-unavailable fallback: when the response channel never started its
    analysis (no server was reached), the on-server-unavailable event is fired
    at the end of the channel analysis to ensure all spans are properly closed.

  - Custom memory allocator: the C wrapper's allocations are routed through
    HAProxy memory pools via otelc_ext_init(), keeping OTel objects in the
    same allocation domain as the rest of the filter.

  - Thread integration: flt_otel_thread_id() returns the HAProxy tid, ensuring
    the wrapper's thread-local operations map to HAProxy worker threads.

  - Single dispatch stage per event: each scope event fires from one callback
    -- channel_pre_analyze or channel_post_analyze for analyzer-bound events,
    or a lifecycle/data callback otherwise.  Exposing both stages of every
    analyzer (an on-X-pre and an on-X-post) was considered and rejected: about
    half the events are not analyzer-bound and have no stage to split; the
    tarpit analyzer never finishes so it has no post stage, and the
    wait-for-HTTP analyzers have no parsed message in the pre stage; and the
    rest complete in a single pass, so their two stages fire microseconds
    apart.  Where a later view is genuinely needed -- the request body, for
    example, is buffered only after AN_REQ_HTTP_BODY -- an optional per-scope
    stage qualifier on the existing event name would be preferable to doubling
    the event table.  The otel-event directive would take an optional 'pre' or
    'post' keyword:

        otel-event on-http-body-request       # default stage (pre, as today)
        otel-event on-http-body-request post  # opt into the post stage

    defaulting to the current stage when omitted.  The filter would list such an
    analyzer in both the pre and post masks and run the scope only from the
    stage it asked for.


18  Tracer, Span and Metrics Internals
----------------------------------------------------------------------

18.1  Tracer Provider Initialization

The tracer provider is set up during the proxy-level flt_otel_ops_init()
callback, which delegates to flt_otel_lib_init() (filter.c):

  1. Version check: OTELC_IS_VALID_VERSION() verifies that the
     OpenTelemetry C wrapper library version matches the header files.

  2. Configuration path: the relative path from the "config" keyword in
     the instrumentation section is resolved to an absolute path using
     getcwd() + snprintf().

  3. SDK initialization: otelc_init(path, err) loads the YAML
     configuration file and sets up the SDK exporters, samplers,
     processors and metric readers.

  4. Tracer creation: otelc_tracer_create(err) allocates the tracer
     handle and stores it in instr->tracer.

  5. Meter creation: otelc_meter_create(err) allocates the meter handle
     and stores it in instr->meter.

  6. Logger creation: otelc_logger_create(err) allocates the logger
     handle and stores it in instr->logger.

  7. Extension callbacks: on success, otelc_ext_init() registers custom
     memory allocation (flt_otel_mem_malloc / flt_otel_mem_free) and
     thread-id (flt_otel_thread_id) callbacks so that OTel SDK objects
     use HAProxy memory pools and thread numbering.

  8. Log handler: otelc_log_set_handler() installs a callback that
     counts SDK diagnostic messages via the flt_otel_drop_cnt counter.

The tracer, meter and logger handles are stored in the flt_otel_conf_instr
structure (conf.h):

  struct flt_otel_conf_instr {
      ...
      struct otelc_tracer *tracer;  /* The OpenTelemetry tracer handle. */
      struct otelc_meter  *meter;   /* The OpenTelemetry meter handle. */
      struct otelc_logger *logger;  /* The OpenTelemetry logger handle. */
      ...
  };

18.2  Per-Thread Tracer, Meter and Logger Startup

The flt_otel_ops_init_per_thread() callback (filter.c) starts the
tracer, meter and logger background threads on the first call:

  if (HA_ATOMIC_BTS(&(conf->instr->flag_started), 0) == 0) {
      retval = OTELC_OPS(conf->instr->tracer, start);
      ...
      if (retval != OTELC_RET_ERROR) {
          retval = OTELC_OPS(conf->instr->meter, start);
          ...
      }
      if (retval != OTELC_RET_ERROR) {
          retval = OTELC_OPS(conf->instr->logger, start);
          ...
      }
  } else {
      retval = FLT_OTEL_RET_OK;
  }

The atomic bit-test-and-set on instr->flag_started ensures that start is
called only once, even when this callback runs on every worker thread and
multiple proxies share the same filter configuration.  Threads that lose the
claim return success without repeating the start.  If any start operation
fails, the error string from the failing handle is forwarded via
FLT_OTEL_ALERT.

18.3  Context, Tracer, Meter and Logger Shutdown

At proxy deinit (flt_otel_ops_deinit, filter.c), the context, tracer,
meter and logger handles are first saved into local variables and
force-flushed (or, under 'option noflush', given a zero flush budget so
the destruction drops the buffered telemetry), then the configuration
tree is freed, and only afterwards are the handles destroyed:

  otel_ctx    = (*conf)->instr->ctx;
  otel_tracer = (*conf)->instr->tracer;
  otel_meter  = (*conf)->instr->meter;
  otel_logger = (*conf)->instr->logger;

  /* tracer, meter and logger: force_flush sharing one deadline. */
  if ((otel_tracer != NULL) && (flt_otel_flush_budget(&ts_deadline, &timeout) == 1))
      (void)OTELC_OPS(otel_tracer, force_flush, &timeout);
  ...

  flt_otel_conf_free(conf);
  flt_otel_pool_destroy();
  otelc_deinit(&otel_ctx, &otel_tracer, &otel_meter, &otel_logger);

Saving the handles into locals first avoids dereferencing the already-freed
instr structure.  Destroying the memory pools before otelc_deinit() is safe:
its teardown never invokes the ext free callback, which serves only otelc_span
and otelc_span_context objects released at runtime.  The otelc_deinit() call
then flushes any pending spans, metric data and log records to the configured
exporters and releases the SDK resources.

The early flush bounds the shutdown time: each handle destruction inside
otelc_deinit() runs a blocking flush of its own, so with the buffers already
drained under the shared FLT_OTEL_FLUSH_DEINIT_S budget, an exporter that
cannot be reached delays the stop by that budget instead of by the sum of the
destruction timeouts.

18.4  Span Lifecycle

Spans progress through identity allocation, OTel span creation, data population
and completion.

18.4.1  Span Identity Allocation

When a scope containing a span definition executes for the first time,
flt_otel_scope_span_init() (scope.c) allocates a scope_span
entry from the otel_scope_span pool and inserts it into the runtime
context's spans list:

  retptr = flt_otel_pool_alloc(pool_head_otel_scope_span, ...);
  retptr->id          = id;       /* Borrowed from config. */
  retptr->id_len      = id_len;
  retptr->smp_opt_dir = dir;
  retptr->ref_span    = ref_span; /* Resolved parent span. */
  retptr->ref_ctx     = ref_ctx;  /* Resolved parent context. */
  LIST_INSERT(&(rt_ctx->spans), &(retptr->list));

The parent reference (ref_id) is resolved at this point by searching the
runtime context's spans list first, then the contexts list.  If the
parent name cannot be found in either list, an error is returned and the
span is not created.

Memoization returns a span whose name already stands in rt_ctx->spans, without
allocating a new entry, as chapter 7.2 describes.

18.4.2  OTel Span Creation (Lazy)

The actual OTel span object is created lazily on first use in
flt_otel_scope_span_start() (event.c):

  span->span = OTELC_OPS(conf->instr->tracer,
      start_span_with_options, span->id,
      span->ref_span, span->ref_ctx,
      ts_steady, ts_system, conf_span->kind, NULL, 0);

The arguments are:

  span->id        The operation name (string identifier from config).
  span->ref_span  The parent span pointer (NULL if root or no parent).
  span->ref_ctx   The parent span context (from extracted context).
  ts_steady       Monotonic timestamp (CLOCK_MONOTONIC) for duration.
  ts_system       Wall-clock timestamp (CLOCK_REALTIME) for events.
  conf_span->kind The per-span configured span kind (default server,
                  overridable via "span ... kind <kind>").
  NULL, 0         Initial attribute array and count (none at creation).

A later scope execution that references the same span name finds the existing
entry, by the memoization above, and adds its data to the already-created OTel
span.

A creation that fails -- the traces signal missing from the configuration,
which is reported through the runtime log, or start_span_with_options()
returning NULL -- leaves the listed entry without an OTel span; completion
skips such an entry, and flt_otel_scope_free_unused() then removes it like
an ended one.

The sampler decides while the span is created, so the new span is asked whether
it records, and flag_norec is set when it does not.  The decision belongs to the
span and not to the stream, because a sampler may record the child of a span it
left out: the delegates of 'parent_based' for a parent that is not sampled say
whether it does.

18.4.3  Span Data Population

After creation, flt_otel_scope_run_span() (event.c) populates
the span with data collected during scope execution:

  Links (event.c):
    Each resolved link is added via span->add_link(), carrying the link's own
    attributes when it has any; see chapter 8.

  Baggage (event.c):
    span->set_baggage_kv_n(data->baggage.attr, data->baggage.cnt)
    sets key-value baggage items propagated across service boundaries.

  Attributes (event.c):
    span->set_attribute_kv_n(data->attributes.attr, data->attributes.cnt)
    sets key-value span attributes evaluated from HAProxy sample
    expressions.

  Events (event.c):
    For each event in data->events (iterated in reverse insertion order):
    span->add_event_kv_n(event->name, ts, event->attr, event->cnt) adds a
    named event with key-value attributes; ts is the event's own evaluated
    'time' value when one was set (the node's ts / ts_set fields), the
    wall-clock timestamp otherwise.

  Exceptions (event.c):
    Each configured 'exception' whose condition holds is emitted through
    span->record_exception() as a span event named 'exception', carrying
    the type, the evaluated message and any additional attributes.

  Status (event.c):
    span->set_status(data->status.code, data->status.description)
    sets the span's status code and description string.  The configured
    status lines are tried in order and the first whose condition holds
    provides the one status that is set.

None of that data is evaluated for a span whose flag_norec is set, the baggage
excepted.  The SDK drops the attributes, the events, the links, the status and
the exceptions of a span that does not record, while the baggage travels in the
span context and reaches the downstream services whatever the sampling decision
was.

18.4.4  Span Context Injection

After populating the span, an "inject" directive (conf_span->ctx_id is non-NULL)
serializes the span context for downstream propagation (event.c).  The ctx_flags
select the carriers, FLT_OTEL_CTX_USE_HEADERS and FLT_OTEL_CTX_USE_VARS, either
or both; chapter 9.2 describes them.

18.4.5  Span Completion

Spans are ended through the marking mechanism described in chapter 7.5.
The actual end call in flt_otel_scope_finish_marked() (scope.c) is:

  if (span->span != NULL)
      OTELC_OPSR(span->span, end_with_options,
                 ts_finish, OTELC_SPAN_STATUS_IGNORE, NULL);

The guard skips an entry whose OTel span creation failed, which carries
only the identity.  The arguments are the monotonic timestamp, a status
hint (IGNORE means "do not override the status already set on the
span"), and NULL for error string.  After end_with_options returns, the
OTELC_OPSR macro NULLs the span pointer, making the entry eligible for
removal by flt_otel_scope_free_unused().

On stream detach, flt_otel_runtime_context_free() (scope.c)
force-ends any remaining active spans with the current monotonic
timestamp, under the same NULL-span guard, and frees all pool entries.

18.5  Metric Instruments

The metric instruments follow the OpenTelemetry data model through a two-form
configuration: an instrument is created once with a fixed identity by a "create"
line, and measurements are recorded against it repeatedly by the "update" lines.
What the create form pins is not only the identity -- name, description, unit,
kind, aggregation and any histogram bounds -- but also the sample expression
that produces the value.  The update form has no value expression of its own; it
contributes only the recording attributes, an optional if/unless condition and,
through the scope that it lives in, the event that drives the recording.

The expression is fixed, the recorded number is not.  Each update re-evaluates
the create form's value expression against the current stream, so a live fetch
such as fe_conn or lat_ns_tot yields a fresh reading on every run, whereas a
constant such as int(1) yields a steady increment.  That value is then applied
through the instrument's native operation: an additive delta for counters and
up-down counters, a distribution sample for histograms, and a last-value reading
for gauges.  A fixed expression is therefore a valid source for the model's
recording side.

A create form emits no data point on its own; a measurement appears only when an
update form for that instrument runs.  Within a scope a name may carry several
create lines, one per condition and the bare line last, and any number of update
lines; the update lines of other scopes record on the same instrument, so one
instrument can be recorded from several sites, each with its own attributes,
condition and driving event.  As section 4.5 states, the create lines of one
name must agree on the type, the aggregation, the description, the unit and the
bounds; the value expression and the condition are what the repeated lines exist
to vary.

18.5.1  Instrument Types

Instrument types (parser.h):

    cnt_int         Counter (uint64)
    hist_int        Histogram (uint64)
    udcnt_int       UpDownCounter (int64)
    gauge_int       Gauge (int64)

Observable (asynchronous) instruments are not supported.  The OTel SDK invokes
their callbacks from an external background thread that is not a HAProxy
thread.  HAProxy sample fetches rely on internal per-thread-group state and
return incorrect results when called from a non-HAProxy thread.

Double-precision types are not supported because HAProxy sample fetches do not
return double values.

  Special:
    update          Update-form instrument (records measurements)

Each create-form instrument carries a description, unit, aggregation type,
sample expression list, and optional histogram bucket boundaries.  Each
update-form instrument carries a reference to its create-form counterpart
and an attribute key-value array for per-scope dimensions.

18.5.2  Instrument Configuration Structure

The flt_otel_conf_instrument structure (conf.h) holds:

  idx          Meter instrument index.  Initially set to
               OTELC_METRIC_INSTRUMENT_UNSET (-1).  Transitions to
               OTELC_METRIC_INSTRUMENT_PENDING (-2) during creation,
               then to the positive meter index on success, or to
               OTELC_METRIC_INSTRUMENT_FAILED (-3) once the creation
               failed FLT_OTEL_INSTR_FAIL_MAX times and is given up.
  fail_num     Number of failed creation attempts, counted up to
               FLT_OTEL_INSTR_FAIL_MAX.
  type         The otelc_metric_instrument_t type constant, or
               OTELC_METRIC_INSTRUMENT_UPDATE (0xff) for update-form.
  aggr_type    The otelc_metric_aggregation_type_t constant.
               Initially OTELC_METRIC_AGGREGATION_UNSET (-1).
  description  Instrument description string (create-form only).
  unit         Instrument unit string (create-form only).
  samples      List of sample expressions for the instrument value.
  bounds       Histogram bucket boundaries array (create-form only).
  bounds_num   Number of histogram bucket boundaries.
  attributes   List of flt_otel_conf_sample entries (update-form only).
  ref          Pointer to the create-form instrument that owns the name, set on
               both forms; the owning entry points at itself.
  scope        The otel-scope the instrument is in, held by the owning entry:
               the scope of that entry until one creates the instrument, the
               creating scope afterwards.
  cond         Optional if/unless ACL condition controlling the recording.

18.5.3  Meter Initialization and Startup

The meter handle is created alongside the tracer in flt_otel_lib_init() and
started per-thread in flt_otel_ops_init_per_thread(), both filter.c and both
described in 18.1 and 18.2.  The meter background thread handles the periodic
collection and export of metric data.

18.5.4  Instrument Creation and Recording

Metric instrument processing is performed by
flt_otel_scope_run_instrument() (event.c), which runs in two
passes during scope execution.

  Pass 1 -- Create-form instruments (event.c):

    Iterates all instruments in the scope.  For each create-form instrument
    whose 'if'/'unless' condition passes and whose owner's idx is not
    OTELC_METRIC_INSTRUMENT_PENDING:

    a. Thread-safe one-time creation: HA_ATOMIC_CAS transitions the idx
       from UNSET to PENDING.  If the CAS fails (another thread is
       already creating this instrument), the current thread skips it.

    b. Instrument creation: meter->create_instrument() is called with
       the instrument name, description, unit, type and callback data.
       On success, the returned index is stored atomically; on failure,
       the idx is reset to UNSET so a transient fault is retried, but
       only up to FLT_OTEL_INSTR_FAIL_MAX attempts, after which the idx
       becomes FAILED and the creation is no longer tried (each attempt
       takes a process-wide lock).  If the instrument carries histogram
       bucket boundaries, meter->add_view() is called before instrument
       creation to register a view with the configured aggregation
       strategy and those bounds; with bounds present and no explicit
       aggregation type, histogram aggregation is used automatically.
       An 'aggr' type given without bounds registers no view.

  Pass 2 -- Update-form instruments (event.c):

    Iterates all instruments again.  For each update-form instrument:

    a. Reference validation: the ref pointer must be non-NULL (resolved
       at check time to the create-form instrument).

    b. Index check: an idx that is still UNSET (the create-form's scope
       has not executed) is created now from the referenced definition,
       so the measurement is not lost; if it is PENDING, the thread
       waits with ha_thread_relax() until the creating thread resolves
       it.  An idx that stays negative (UNSET or FAILED) means the
       creation failed or was given up, and the measurement is skipped.

    c. Recording: flt_otel_scope_run_instrument_record() evaluates the
       sample expression, converts it to an otelc_value, and calls
       meter->update_instrument_kv_n(idx, &value, attr, attr_len).

18.5.5  Sample Evaluation for Metrics

The recording function flt_otel_scope_run_instrument_record()
(event.c) supports two evaluation paths:

  Standard path: evaluates sample_process() on the first expression in
  the create-form instrument's samples list, using the stream's backend,
  session and direction context.

  Log-format path: if sample->lf_used is set, takes a trash buffer through
  flt_otel_trash_alloc(), calls build_logline() to evaluate the log-format
  expression into it, and presents the result as an SMP_T_STR sample.

Both paths converge on flt_otel_sample_to_value(), which converts the HAProxy
sample data to an otelc_value.  A metric instrument value is always int64, so
a string result (for example a value read from a variable, which the filter
stores as a string) is coerced through otelc_value_strtonum() with
OTELC_VALUE_INT64.  A numeric string is coerced and recorded; the measurement
is rejected, with a warning that names the value, only when that conversion
fails.


18.5.6  Instrument Lifecycle Summary

  Configuration time (flt_otel_conf_instrument_init):
    idx = OTELC_METRIC_INSTRUMENT_UNSET (-1)
    type = OTELC_METRIC_INSTRUMENT_UNSET (the actual type constant is
           assigned later, during keyword parsing)

  First scope execution (any thread):
    idx transitions:  UNSET -> PENDING -> meter_index  (success)
                      UNSET -> PENDING -> UNSET        (failure, retried)
                      UNSET -> PENDING -> FAILED       (retries exhausted)

  Subsequent scope executions:
    Create-form: skipped (idx is already a valid meter index).
    Update-form: evaluates samples and records via meter API.

  Shutdown:
    otelc_deinit() flushes and destroys tracer, meter and logger,
    including all registered instruments and their callbacks.


18.6  Log Records

The filter supports OpenTelemetry log records via the "log-record"
keyword inside otel-scope sections.  Each log record is emitted through
the OTel logger at a configured severity level, with an evaluated body,
optional span correlation and optional key-value attributes.

18.6.1  Log Record Configuration Structure

The flt_otel_conf_log_record structure (conf.h) holds:

  severity     The otelc_log_severity_t severity level.
  event_id     Optional numeric event identifier (int64).
  event_name   Optional event name string.
  span         Optional span reference name (resolved at runtime).
  time         Optional timestamp expression (single-entry conf_sample list).
  attributes   List of flt_otel_conf_sample entries for attributes.
  samples      List of sample expressions for the body.
  cond         Optional if/unless ACL condition controlling the record.

The attributes list contains flt_otel_conf_sample entries, one per "attr"
keyword.  Each entry's key field holds the attribute name and its sample
expressions are evaluated at runtime, following the same two-path model
(bare sample or log-format) as span attributes.

The samples list contains exactly one flt_otel_conf_sample entry, which holds
either a list of bare sample expressions or a single log-format expression.  The
log-format path is selected when the value begins with the "%[" sequence; an
explicit '%[ ... ]' wrapper is then stripped before parsing so the inner string
may use HAProxy log-format aliases such as %ci, %t or %ft.

The time list, when populated, holds a single flt_otel_conf_sample produced by
the "time [<unit>] <sample>" keyword.  The unit selector (one of s, ms, us or
ns) is stored in the sample's extra field as an OTELC_VALUE_INT32; a default
of FLT_OTEL_TIME_UNIT_S is recorded when no unit is given.  The sample itself
follows the same parsing rules as any other flt_otel_conf_sample.

18.6.2  Log Record Emission

Log record processing is performed by flt_otel_scope_run_log_record()
(event.c), called from flt_otel_scope_run() after metric instrument
processing and before span finishing.

For each configured log record the function performs:

  1. Severity check: OTELC_OPS(logger, enabled, severity) tests whether
     the logger accepts records at this severity.  If not, the entry is
     skipped.  The threshold is controlled by the "min_severity" option
     in the YAML logs signal configuration.

  2. Attribute evaluation: each entry in the attributes list is evaluated via
     flt_otel_sample_add() into a temporary flt_otel_scope_data structure.
     The evaluated key-value array is passed to logger->log_span() and freed
     after emission.

  3. Body evaluation: the single sample entry is evaluated using one of
     two paths:

     Log-format path (sample->lf_used is true):
       A trash buffer is taken through flt_otel_trash_alloc() and
       build_logline() evaluates the log-format expression into it.

     Bare sample expression path:
       Each expression in sample->exprs is evaluated via
       sample_process() and converted to a string via
       flt_otel_sample_to_str().  Results are concatenated into a
       single buffer.

  4. Span resolution: if conf_log->span is non-NULL, the runtime
     context's spans list is searched for a scope_span with a matching
     name.  If found, the OTel span pointer is captured for correlation.
     A missing span is non-fatal: the record is emitted without span
     correlation (a debug build traces the miss).

  5. Event timestamp selection: when conf_log->time is non-empty, its single
     sample is evaluated via flt_otel_sample_eval() in native mode.  A string
     result is coerced to int64 with otelc_value_strtonum(); on conversion
     failure the wall-clock ts is kept.  A successful int64 result is split
     into a struct timespec by the unit packed in the sample's extra field
     (s, ms, us or ns), and that timespec becomes the event timestamp.

  6. Emission: logger->log_span() is called with the severity, event_id,
     event_name, resolved span (or NULL), the event timestamp, the wall-clock ts
     as the observed timestamp, the evaluated attributes and the evaluated body
     string.

18.6.3  Logger Lifecycle Summary

  flt_otel_lib_init             otelc_logger_create() allocates the handle.
  flt_otel_ops_init_per_thread  logger->start() launches the background thread.
  flt_otel_scope_run            flt_otel_scope_run_log_record() emits records.
  flt_otel_ops_deinit           otelc_deinit() flushes and destroys the logger.

18.7  Operation Repeat Semantics

The wrapper operations behind this chapter differ in what a repeated call does.
Section 4.5 states the binding keyword rules; this section records the behavior
of the operations under them, which a rule may tighten but never contradict.  An
operation is additive (every call appends one more item), overwrite per key (the
last call wins for the named key only), last wins (a single slot that every call
replaces), once (the first call decides and the entry says whether a repeat is a
no-op or an error) or per call (no shared state, every call stands on its own);
update_instrument_kv_n alone splits its class by the instrument type.  Some
classes come from the wrapper itself, the others from the OpenTelemetry C++ SDK
underneath it.

  Span and context operations:

    start_span_with_options - per call: every invocation starts one new and
                              independent span.  Driven by the 'span' line that
                              creates the span; a bare line re-activates it and
                              starts nothing.  The creation-time link array is
                              passed empty and every link goes through add_link.
    set_attribute_kv_n      - overwrite per key: the last value wins for the
                              named key, the other keys stay.  Driven by the
                              'attribute' keyword.
    add_event_kv_n          - additive: every call appends one event to the
                              span.  Driven by the 'event' keyword; the filter
                              merges the lines of one name into a single call.
    add_link                - additive: every call appends one more link; the
                              operation needs OTel ABI 2 and reports an error on
                              the older ABI.  Driven by the 'link' keyword.
    record_exception        - additive: every call appends one more span event
                              named 'exception'.  Driven by the 'exception'
                              keyword.
    set_baggage_kv_n        - overwrite per key: a repeated key replaces its
                              value, per the W3C baggage rules.  Driven by the
                              'baggage' keyword.
    set_status              - last wins: the SDK stores the status code and the
                              description with no precedence, so a later call
                              replaces an earlier error.  The first-match scheme
                              of the 'status' keyword makes the one call that is
                              made deterministic.
    inject_http_headers     - overwrite per key at the carrier: every new call
                              empties and refills the writer's map, and the
                              filter drops a header of the same name before it
                              adds the new one.  Driven by the 'inject' keyword,
                              whose uniqueness rules keep two lines from writing
                              the same header names; inject_text_map behaves the
                              same for the variable storage.
    extract_http_headers    - per call: every call builds one new span context
                              object.  Driven by the 'extract' keyword; the
                              extract_text_map operation behaves the same for
                              the variable storage.
    end_with_options        - once per span: the SDK latches the first call and
                              hands the span data to the exporter pipeline; the
                              wrapper then deletes the handle and nulls the
                              caller's pointer, so a repeated end on it returns
                              silently, and destroying a span that was never
                              ended ends it implicitly.  Driven by the span
                              completion: 'finish', 'otel-stop' and the scope
                              teardown.

  Metric operations:

    create_instrument      - once per name and type pair: a repeated create
                             returns the existing instrument with the first
                             creator's description and unit.  The filter calls
                             it once per name, from the create line that owns
                             it, and refuses a create line of another otel-scope
                             at run time, so the tolerance is never exercised.
    update_instrument_kv_n - additive for cnt_int, hist_int and udcnt_int, whose
                             measurements accumulate into the aggregation; last
                             wins for gauge_int, which accepts every call too
                             but keeps the latest value per attribute set within
                             a collection cycle.  Driven by the update form of
                             the 'instrument' keyword; the gauge's last-wins is
                             how the value is stored, so the update form keeps
                             the apply-all scheme for the gauge as well.
    add_view               - once per view name: a repeated name returns the
                             existing view with the first creator's bounds and
                             aggregation.  Driven by the 'bounds' and 'aggr'
                             arguments of the 'instrument' keyword.

  Log operations:

    log_span - additive: every call emits one more log record.  Driven by the
               'log-record' keyword.

The lifecycle operations fall into the same classes: a repeated start replaces
the provider, the tracer and the propagator, which is why the filter runs it
once, under the atomic claim of section 18.2; set_flush_timeout is a last-wins
setter behind 'option noflush' and force_flush is a per-call action.