Sampling your data is a great way get data volume to a manageable size.
Many folks are curious about how sampling works with tracing, given that simply sampling 1/N requests at random will not guarantee that you retain all of the spans for a given trace. The story of how sampling and tracing fit together with Honeycomb is still evolving, but here are some thoughts on how to approach it.
Traditionally, the way traces are sampled is head-based sampling.
When the root span is being processed, a random sampling decision is made.
For example, if randint(10) == 0, the span will be sampled.
If that span is decided to be sampled, it gets sent and propagates that decision out to the descendent spans, who follow suit, usually by a method like HTTP header with something like X-B3-Sampled: 1.
That way, all the spans for a particular trace are preserved.
Our integrations do not support head-based sampling today out of the box, but you could implement such a system yourself.
Some of our integrations do support what we call deterministic sampling. In deterministic sampling, a hash is made of a specific field in the event/span such as the request ID, and a decision to sample is made based on that hash and the intended sample rate. Hence, an approximately correct number of traces will be selected and the decision whether or not to sample a given trace does not need to be propagated around: actors can sample full traces whether they can communicate or not.
There is another option: tail-based sampling, where sampling decisions are made when the full trace information has been gathered. This ensures that if an error or slowness happens way down the tree of service calls, the full events of the trace are more likely to get sampled in. To use this method of sampling, all spans must be collected at a buffer ahead of time. Honeycomb’s Refinery is a tail-based sampler.
The Honeycomb Beelines and OpenTelemetry provide out-of-box support for deterministic sampling of traces. By default, when a sampling rate is configured, the trace ID of each trace-enabled event, or span, will be be used to compute whether the event should be kept. All events that share the same trace ID will receive the same sampling decision.
For more information, see our language specific instrumentation documentation.
Did you find what you were looking for?
