Exponential-histogram table lookup - #5230
Draft
jmacd wants to merge 9 commits into
Draft
Conversation
jmacd
changed the title
3 tasks
oertl
reviewed
Aug 3, 2026
pritishnahar95
pushed a commit
to pritishnahar95/otel-arrow
that referenced
this pull request
Aug 5, 2026
# Change Summary Allocation free fixed-size histogram aggregator for OpenTelemetry exponential histograms, declared `HistogramNN<W>` for a histogram of non-negative values using `W` 64-bit words of space. The implementation uses the best achievable bucket width and base-2 exponential scale. The data structure consumes 6 additional words for min/max/sum/count, so `HistogramNN<10>` occupies 128 bytes (up to 640 1-bit buckets, 160 4-bit buckets--[note the OTel SDK spec gives 160 as a recommended default size](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#base2-exponential-bucket-histogram-aggregation)), `Histogram<26>` occupies 256 bytes (up to 1664 buckets). Data type implements optional min-width and max-scale as limits. Implements "SWAR" i.e., SIMD-within-a-register for its major downscale and widen operations. Table lookup implementation has an experimental specification in open-telemetry/opentelemetry-specification#5230. The table lookup generation is not included in this build; a generated table of size 256 (scale 8) is included here, see unfinished work below. Summary-level (`Mmsc`) instruments export as a regular OTLP `HistogramDataPoint` carrying only count, min, max, and sum with no bucket boundaries, rather than as an exponential point with invented bucket membership. `ExponentialHistogramDataPoint` is reserved for the normal and detailed tiers, which have real buckets. ### Recording cost Recording 1,024 values spanning about 64 octaves into a long-lived instrument on an unloaded development machine, taken as the minimum over many trials: | Instrument | State | Per observation | |---|---|---| | `Counter<f64>` | 8 bytes | ~0.2 ns | | `Mmsc` | 32 bytes | ~0.9 ns | | `HistogramNN<10>` normal, from B1 counters at the finest scale | 128 bytes | ~6 ns | | `HistogramNN<10>` normal, preset to the geometry it settles on | 128 bytes | ~5 ns | | `HistogramNN<26>` detailed, from B1 counters at the finest scale | 256 bytes | ~7 ns | | `HistogramNN<26>` detailed, preset to the geometry it settles on | 256 bytes | ~6 ns | A histogram starting at the default geometry pays to widen its counters and reduce its scale as the population reveals itself; one told the shape up front pays neither. That is worth about a nanosecond, and it is the difference the two rows per tier isolate. It also means the tiers are closer in cost than their footprints suggest: preset, the detailed tier costs about a nanosecond more per observation than the normal tier despite holding twice the state, so that choice is about retained resolution and footprint rather than recording speed. Splitting an observation into its parts puts the remaining cost at roughly 1 ns to map a value to its bucket, 1.6 ns for the exact min, max, sum and count, and 3 ns for the packed counter update. The last is a load, an add and a store at an address the value itself decides, which is close to the floor for counters packed several to a word. An `Mmsc` observation is about five counter additions and a histogram observation about eight `Mmsc` records, so reach for `Mmsc` unless bucket resolution is genuinely needed; `docs/telemetry/metrics-guide.md` carries that guidance for component authors. Roughly half of what `Mmsc` costs over a counter is the check that rejects NaN and the infinities, which the histogram tiers get for free because they already decompose the value to find its bucket. Reproduce with: ```console cargo bench -p otap-df-expohisto --bench hot # the table above cargo bench -p otap-df-telemetry --bench distribution_record # the same at the instrument level cargo bench -p otap-df-expohisto --bench merge --features bench # merging, below ``` Merging selects the combined geometry from stable snapshots, then combines packed counter words with SWAR addition. Against inserting bucket by bucket that runs about 50% faster on dense populations and 40% faster over a range needing circular addressing, and about 20% slower when the two sides differ in scale and the range is sparse. A source holding a single bucket cannot amortize the setup and takes the ordinary recording path instead. ## What issue does this PR close? Part of open-telemetry#2428 Fixes open-telemetry#2458 ## Unfinished work I will open related issues created here as this merges. - Prometheus native-histogram exporter support - Generate variable-size logarithm tables in `build.rs` - Avoid using Prost objects in Metrics ITS path (in general, like Logs ITS path) - Aggregators inside the Prometheus and admin exporter can/should widen their resolution relative to the instruments they aggregate - Selection of Mmsc vs Histogram by configuration - Configurable histogram size/resolution/width. ## How are these changes tested? Alongside unit tests for mapping, downscale, merge, and views, the crate carries a geometry verifier (`crates/expohisto/src/histogram/verify.rs`) that property tests use. Rather than recomputing an expected result and comparing, it checks the properties any correct result must have, given the observations that produced it: - **exact**: the buckets reproduce the population at the reported scale - **representable**: the occupied range fits `N` words at the reported width - **tight**: the width is the narrowest that holds the largest bucket, and the next finer scale genuinely fails to fit Tightness is the part an equality check against a from-scratch optimum cannot express, because a histogram reaches its geometry by a path: values arrive in some order and merges combine two states. The caller states the finest scale that was still reachable, since merging cannot undo coarsening a source already did. Between them the tests reach all seven counter widths, scales -6 through 8, and pools from 2 to 64 words, covering recording, merging in both pool-size directions, repeated merging, configured minimum widths and maximum scales, subnormals, non-finite values, the full normal range, and counter overflow. The verifier found the recording path giving up a scale step it did not owe: a counter overflow asked for a range relaxation at the old width, and the downscale then charged a further step for the wider counters the merged counts required. Widening now spreads into unused words when the wider layout fits and folds one level at a time when it does not, so it stops at the first scale that fits. Quantile estimation also changed as a result of review: the representative value within a bucket is now the geometric midpoint, which keeps a skewed bucket inside the scale's relative error bound. ## Are there any user-facing changes? Users get exponential histogram at normal and detailed level of internal metrics through ITS. ### Changelog * [x] Added a `.chloggen/*.yaml` entry * [ ] This PR is a `chore` (indicated in title) * [ ] This is a documentation-only PR. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> Copilot-Session: c95c1ba9-b197-45aa-9f7b-145982b3e4d6
1 task
Thank you @oertl! Co-authored-by: Otmar Ertl <otmar.ertl@dynatrace.com>
reyang
reviewed
Aug 5, 2026
Comment on lines
+932
to
+933
| - The largest bucket at any given scale will contain the value | ||
| `0x1p+1024`, an unrepresentable value. |
Member
There was a problem hiding this comment.
Why? My understanding is that the largest bucket should go up to double.MaxValue which is 0x1p+1024 - 1.
Member
Author
There was a problem hiding this comment.
I'll try to make this clearer. The largest normal value (more like nextAfter(0x1p+1024, -1)) falls into the largest bucket. The mathematical upper-boundary of the largest bucket is not representable because of upper-inclusivity rules and this is just a note to say that's OK.
MrAlias
reviewed
Aug 5, 2026
MrAlias
left a comment
MrAlias
left a comment
Contributor
There was a problem hiding this comment.
The lookup approach looks workable, but the table-generation procedure and input decomposition need tighter numeric contracts before implementations can reproduce the claimed exact mapping.
| for _ in 0..S: | ||
| x = sqrt(x) | ||
|
|
||
| BOUNDARIES[k+1] = significand(ceiling(x)) |
Contributor
There was a problem hiding this comment.
ceiling(x) here has the ordinary integer-ceiling meaning, so this does not generate the table described above. At S = 1, k = 1, it produces ceiling(sqrt(2)) = 2, whose 52-bit significand is zero; k = 0 likewise overwrites the special BOUNDARIES[1] = 1 entry with zero.
Could we document the reference implementation's scale/verify/mask procedure and the separate k = 0 upper-inclusive adjustment? The prose above should also say S square roots, not N.
| The input is a positive IEEE 754 double-precision value, with the | ||
| zero, NaN and Inf cases handled separately. The significand `s` is a | ||
| 52 bit unsigned integer, and the exponent `e` is the corresponding | ||
| unbiased exponent. |
Contributor
There was a problem hiding this comment.
The decomposition is ambiguous for subnormal values. If s and e are the raw IEEE fields, the smallest subnormal (s = 1, e = -1022) maps to -2044 at scale 1 instead of the exact -2149. The Rust public API avoids that by clamping before decomposition.
Could we state whether callers must normalize or clamp subnormals, and scope the exactness claim in the Overview to match?
|
|
||
| # Exponential-Histogram Table Lookup | ||
|
|
||
| **Status**: [Mixed](../document-status.md) |
Contributor
There was a problem hiding this comment.
Is the entire lookup document intended to be Stable? The file is marked Mixed, but only Overview has a section status; Data Structures, Algorithm, Correctness Proof, and Table Size have none. Could we either mark the document Stable or label those sections with their intended maturity?
pull Bot
pushed a commit
to thompson-tomo/opentelemetry-specification
that referenced
this pull request
Aug 7, 2026
Fixes open-telemetry#5250. ## Problem 1: doctoc never matched the markers doctoc matches the TOC markers case-sensitively, in lowercase (`doctoc/lib/transform.js`): ```js return new RegExp(`${commentEscapedStart} START doctoc `).test(line); ``` Every file in this repo used uppercase markers, `<!-- START DOCTOC -->`. Because the Makefile passes `--update-only`, doctoc found no markers and skipped the file: ```js if (!(tocs.positions?.length > 0) && updateOnly) { return { transformed: false }; } ``` So since open-telemetry#5153: - `make markdown-toc` regenerated nothing. - `make markdown-toc-check` always passed, even with a stale TOC. A stale TOC was only caught later by `markdown-link-check`, as a broken anchor. That happened in open-telemetry#5230. This PR renames the markers to `<!-- START doctoc -->` and `<!-- END doctoc -->` in all 55 files and regenerates the tables of contents. ## Problem 2: an invalid comment truncated a TOC Even with correct markers, `specification/metrics/data-model.md` produced only 5 TOC entries. The file contained: ``` <!--- cSpell:ignore emetry ---> ``` The content ends with `-`, so this is not a valid CommonMark comment. The HTML block never closes and absorbs the rest of the document, and the parser sees almost no headings. Six such comments are rewritten as `<!-- ... -->`, in `development/trace/zpages.md`, `oteps/0035-opentelemetry-protocol.md`, `specification/metrics/data-model.md` and `specification/trace/tracestate-probability-sampling.md`. The Hugo front matter comments use the same `<!--- ... --->` form and are left unchanged. Their enclosing block ends at the TOC marker, so they do not hide any heading. ## Result 10 tables of contents were out of date and are now regenerated. Most changes are bullet prefixes and missing entries left over from the previous TOC tool. Two examples: - `specification/common/README.md`: `[map](#mapstring-anyvalue)` becomes `[map<string, AnyValue>](#mapstring-anyvalue)`. - `oteps/profiles/0239-profiles-data-model.md` gains about 60 headings that were never listed. No heading and no prose is changed. ## Verification - `make markdown-toc` is now idempotent; a second run reports no changes. - `make markdown-toc-check` exits 1 when a heading is added without regenerating. Before this PR it could not fail. - Every anchor in all 55 tables of contents resolves to a real heading. - No table of contents omits a heading. Assisted-by: Copilot --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Carlos Alberto Cortez <calberto.cortez@gmail.com> Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com>
Member
Author
|
Thank you @MrAlias. I agree the pseudocode needs work. |
jmacd
marked this pull request as draft
August 11, 2026 15:14
Pull request dashboard statusWaiting on the author · refreshed 2026-08-20 15:49 UTC Move out of draft to request review. Status above doesn't look right?
|
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 252a5b48-d130-4c6c-9708-029af1c6c82a
Open
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3630
Changes
Adds a supplementary guideline for exact power-of-two table lookup support in OpenTelemetry SDKs.
Original algorithm here is found in related work by @oertl @yzhuge
Dynatrace — Otmar Ertl's DynaHist library (ExponentialHistogramLargeInclusiveLayout) uses
Nlinear buckets with 2 boundary corrections.NewRelic — Yuke Zhuge's NrSketch library (SubBucketLookupIndexer, algorithm description) uses
2Nlinear buckets with 1 boundary correction.CHANGELOG.mdfile updated for non-trivial changes