T-20356 Deliver every record: batching, flush on Close, retries and error reporting

[PetrHeinz]

The handler sent every record in a goroutine of its own, ignored the response status and the error, never retried and had no way to wait for delivery. Records logged just before the process exited were lost, a wrong token was never noticed, and an outage either dropped records silently or piled up goroutines. This PR replaces the per-record send with a transport that batches, retries, reports and can be flushed, behind the unchanged API and record shape.

• NewBetterstackHandler returns a *BetterstackHandler (still a slog.Handler) with Close, Flush and Stats. defer handler.Close() delivers what is still queued. A program that sleeps before exiting instead keeps working, because a partial batch is sent after BatchInterval.
• Records are batched (1000 records or 1 s), gzip-compressed and sent as the same JSON array as before, so a custom Marshaler still receives the []map[string]any of a batch. Up to 5 uploads run concurrently.
• 408, 429, 5xx and network errors are retried 5 times with exponential backoff and jitter, honouring Retry-After. Other 4xx are terminal. A 413 batch is split and resent; a single record too large on its own is dropped.
• Failures are reported through OnError, by default one line on stderr: rejected batches, exhausted retries and dropped records, with queue-full drops summarised every 5 s instead of one line per record. Nothing is silent anymore and nothing is logged through the handler itself.
• The queue is bounded (100,000 records). When the application outruns delivery, records are dropped and counted instead of blocking the application or starting goroutines without limit. Close waits up to ShutdownTimeout (15 s) and returns an error naming what was left.
• Timeout now bounds each attempt (a hard-coded client timeout used to cap it at 10 s), and a data race on the attribute slice that derived handlers share is fixed.
• The User-Agent carries the version like the other clients.
• A missing token no longer panics at construction, which the other official clients never did either. The handler reports it once through OnError, returns ErrMissingToken from Handle and drops every record, so an unset environment variable cannot take the application down. The example project sets Level: slog.LevelInfo with a comment that Debug is the default.

Unchanged: the record shape (extra, logger.name, logger.version), the Debug default, every existing Option field and the package settings. Upgrading from 1.4.4 is a go get; adding defer handler.Close() is what makes the last records before exit safe, and the docs page needs to say so.

Verified: go test -race green three times in a row on Go 1.25, golangci-lint clean, statement coverage 93%. The race test reproduces the data race on the published 1.4.4 under the race detector (handler.go:90). End to end against a test source on the clanker team: the example project delivers its three records in 0.6 s with Close and no sleep, the rows keep the exact extra shape of 1.4.4, and a wrong token prints slog-betterstack: Better Stack rejected 3 records: 401 Unauthorized on stderr where 1.4.4 printed nothing.

End to end against the real ingest endpoint, built with the race detector: 10 records, a burst of 25,000 from 8 goroutines (all delivered, Close in 0.2 s), 1000 records of 16 KiB each (16 MB, all delivered), the old sleep-only workaround (delivered), a wrong token (401 reported, nothing sent), an unresolvable host (6 attempts, "no such host" reported), an unreachable address (Close bounded by the shutdown timeout), 16,000 records logged while Close runs (every record either sent or counted as dropped, no race), an endpoint without a trailing slash. A process that exits with neither Close nor a sleep loses its records, as documented. The counts in ClickHouse match Stats for every scenario. Four gaps found by that run are fixed in the last four commits, tests first: a batch abandoned at shutdown now reports the error of its last attempt, records logged after Close are reported once, a negative Timeout means the default, and a record whose JSON is over the 10 MiB per-record limit is dropped and reported before sending. That last one came from a 12 MiB record the ingest endpoint answered with a 2xx and then replaced with a notice row in the source, its documented behaviour for records over the 10 MiB limit, so the client counted as sent what never landed; checking locally also saves uploading megabytes that cannot land.

Thanks to @prochac, who reported that records logged before exit were lost and whose analysis of the official clients shaped the defaults and the drop accounting here, and to @alistairjevans for the first batching fork and the race fix.

The first commit holds the tests only and is expected to fail CI. The following commits make them pass.

🤖 Generated with Claude Code