The Go Beeline for Honeycomb is quick and easy way to instrument your Go application. It includes several optional wrappers that automatically instrument HTTP requests and database queries. It also supports tracing out of the box, linking database queries to the HTTP request from which they originated. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey you may find you’d like to add new events or traces to add more details specific to your app. The Go Beeline provides simple interfaces for adding both.
To see an example of the Go Beeline in action, try out the Golang-Gatekeeper Example App.
If you’d like to see more options in the Go Beeline, please file an issue or vote up one already filed! You can also contact us at support@honeycomb.io.
If you prefer lower-level control over your application’s instrumentation, check out our Go SDK.
You can find your API key on your Team Settings page. If you don’t have a API key yet, sign up for a Honeycomb trial.
To install the Go Beeline for your application:
$GOPATH
directory, add the Beeline package and its dependencies: $ git clone https://github.com/honeycombio/beeline-go
$ cd beeline-go; go build
In your code, import the beeline-go
package and initialize it with your API key and dataset name:
import beeline "github.com/honeycombio/beeline-go"
func main() {
beeline.Init(beeline.Config{
WriteKey: "YOUR_API_KEY",
Dataset: "MyGoApp",
})
defer beeline.Close()
...
Add a Beeline wrapper in your http.ListenAndServe
call. This adds basic
instrumentation for each request at the outermost layer of the call stack.
import "github.com/honeycombio/beeline-go/wrappers/hnynethttp"
...
http.ListenAndServe(":8080", hnynethttp.WrapHandler(muxer))
Augment the data with interesting information from your app and extra
context such as errors so that you can see rich information about your app
in Honeycomb. The context.Context
(containing the automatically constructed
HTTP event) to use should be present on the request if the Beeline HTTP wrapper
was used.
func handleEndpoint(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
if err := beeline.AddField(ctx, "user_id", userIDFromSession(r)); err != nil {
beeline.AddField(ctx, "get_session_error", err)
}
// ... do the rest of work for the request ...
}
Add additional spans and turn a series of events into a trace:
func slowOp(ctx context.Context) {
ctx, span := beeline.StartSpan(ctx, "slowOp")
defer span.Send()
// ... go on and do the slow opp, add more data along the way
beeline.AddField(ctx, "interesting_thing", "banana")
}
Make sure to call beeline.Close()
to send any pending events to Honeycomb
before your program exits.
Note: In this example we are using
hnynethttp
,
see the Optional configuration section for
more about the available wrappers. You can also find a more detailed walkthrough
in the Go Beeline tutorial.
The middleware wrapper creates a Honeycomb event in the request context as a span in the overall trace. This span exists throughout the request’s lifecycle, allowing you to add as many additional custom fields as you like.
Here is an example of adding a custom field:
func calcBigNum(ctx context.Context) {
// ... do work, find a BigNum
beeline.AddField(ctx, "big_num", result)
// ... do thing with BigNum, maybe something fails
if err != nil {
beeline.AddField(ctx, "error", err)
}
}
Additional fields are added under the app.
namespace. For example, the
field above would appear in your event as app.big_num
.
The namespace groups your fields together to make them easy to find and examine.
If you prefer to avoid the app.
namespace, for example in order to overwrite an automatically-populated field,
you can extract the current span
instance and call AddField
directly on that instance.
import "github.com/honeycombio/beeline-go/trace"
func calcOtherBigNum(ctx context.Context) {
// ... do work, find another BigNum
span := trace.GetSpanFromContext(r.Context())
span.AddField(ctx, "handler.name", "calc_other_big_num")
}
These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per- user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred.
It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user’s ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request.
If you’re interested in adding custom fields to all spans, use AddFieldToTrace
instead.
AddFieldToTrace
adds the field to both the currently active span and all other
spans that have yet to be sent and are involved in this trace that occur within this process.
Additionally, these fields are packaged up and passed along to downstream processes
if they are also using a beeline.
This function is good for adding context that is better scoped to the request than
this specific unit of work, e.g. user IDs, globally relevant feature flags, errors, etc.
Fields added here are also prefixed with app.
We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder.
Adding spans with the Go Beeline is easy! Here is an example, where calls to slowOp get their own span within the trace:
func slowOp(ctx context.Context) {
ctx, span := beeline.StartSpan(ctx, "slowOp")
defer span.Send()
// ... go on and do the slow opp, add more data along the way
beeline.AddField(ctx, "interesting_thing", "banana")
}
Spans always get a few fields:
slowOp
You are always welcome (and encouraged!) to add additional fields to spans using
the beeline.AddField
function.
After the router has parsed the request, more fields specific to that router are available, such as the specific handler matched or any request parameters that might be attached to the URL.
You can use a wrapper to capture additional fields specific to a particular type of request. The available wrappers are listed below. You can find additional information and detailed instructions in the GoDoc links for each subpackage.
hnynethttp
hnygoji
(Goji v2.0+)hnygorilla
hnyhttprouter
If you would like to use another framework that supports middleware, you may
be able to adapt one of the standard wrappers. We recommend starting with
the hnynethttp
wrapper, as it expects a function that takes a http.Handler
and returns a http.Handler
.
In order to have traces connect HTTP wrapped packages all the way down to the
database (using the sql or sqlx wrappers), you must pass the context from the
*http.Request
through to the SQL package using the appropriate Context-enabled
function calls. This context ties the SQL calls back to specific HTTP requests,
so you can include additional details such as how much time was spent in the DB.
It also connects the request IDs from separate events so you can see exactly
which DB calls were triggered by a given event.
For very high throughput services, you can send only a portion of the events
flowing through your service by setting the SampleRate
during initialization.
This sample rate will send 1/n events, so a sample rate of 5 would send 20% of
all events. For high throughput services, a sample rate of 100 is a good start.
If you have some transformations you would like to make on every span before it
leaves the process for Honeycomb, the PresendHook
is your opportunity to make
these changes. Examples are scrubbing sensitive data (eg you may want to ensure
specific fields are dropped or hashed) or augmenting data (eg making out-of-band
translations between an ID and a more human readable version). Similar to the
SamplerHook
discussed below, you pass the PresendHook
a function that will
be called on every span with the fields of that span as an argument. The
function is free to mutate the map passed in and those mutations will be what
finally gets sent to Honeycomb.
As an example, say we have some database calls that insert secret values into a table. Normally, we want to see all the query arguments and DB queries that come from our instrumented sql
package, but we specifically want to protect things that go into the secrets table. This code will examine all spans just before they’re sent to Honeycomb and replace the arguments to any SQL statement that involves the secrets
table with a sha1
hash of those arguments.
// hashSecretsArguments will replace the query arguments to SQL queries against
// the secrets table with a hash of thsoe arguments, so that the contents of the
// secrets inserted into a table are not passed along.
func hashSecretsArguments(fields map[string]interface{}) {
// if the span is a DB span and has a query
if query, ok := fields["db.query"]; ok {
// and the query accesses the secrets table
if strings.Contains(query, "secrets") {
// scrub the arguments passed in to the query
// replace them with a sha1 hash of the original args
args, ok := fields["db.query_args"]; ok {
hashedArgs := sha1.Sum([]byte(args))
fields["db.query_args"] = hashedArgs
}
}
}
}
func main() {
beeline.Init(beeline.Config{
WriteKey: "YOUR_API_KEY",
Dataset: "MyGoApp",
Debug: true,
PresendHook: hashSecretsArguments,
})
}
To sample a portion of events for very
high throughput services, include an integer SampleRate
in the initialization of the Go Beeline. This sends 1/n of all events, so a sample rate of 5
would send 20% of your events. Try starting with
a value of 10
:
func main() {
beeline.Init(beeline.Config{
WriteKey: "YOUR_API_KEY",
Dataset: "MyGoApp",
Debug: true,
SampleRate: 10,
})
Sampling is performed by default on a per-trace level in the Go Beeline, so adding sampling won’t break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.
The Honeycomb engine reweights query results to ensure that sampled computations return correct results.
Our Beeline lets you define a SamplerHook
in order to customize the logic used for deterministic per-trace sampling.
For example, assume you have instrumented an HTTP server. You’d like to keep all requests to login, skip all health checks, and sample the rest at a default rate. You could define a sampler function like so:
import (
"crypto/sha1"
"math"
)
// Deterministic shouldSample taken from https://github.com/honeycombio/beeline-go/blob/7df4c61d91994bd39cc4c458c2e4cc3c0be007e7/sample/deterministic_sampler.go#L55-L57
func shouldSample(traceId string, sampleRate int) bool {
upperBound := math.MaxUint32 / uint32(sampleRate)
sum := sha1.Sum([]byte(traceId))
// convert last 4 digits to uint32
b := sum[:4]
v := uint32(b[3]) | (uint32(b[2]) << 8) | (uint32(b[1]) << 16) | (uint32(b[0]) << 24)
return v < upperBound
}
func sampler(fields map[string]interface{}) (bool, int) {
// default sample rate of 10
sampleRate = 10
switch fields["request.url"] {
case "/x/alive": // drop health checks
return false, 0
case "/login": // keep all login requests
return true, 1
}
// all the rest of the traffic falls through to the default case
if shouldSample(fields["trace.trace_id"].(string), sampleRate) {
return true, sampleRate
}
return false, 0
}
func main() {
beeline.Init(beeline.Config{
WriteKey: "YOUR_API_KEY",
Dataset: "MyGoApp",
Debug: true,
SamplerHook: sampler,
})
}
Note: Defining a sampling hook overrides the deterministic sampling
behavior for trace IDs. Unless you take trace.trace_id
into account (as we did
above by taking the sha1.Sum
of the trace ID), you will get incomplete traces.
If a single trace in your system can traverse multiple processes, you’ll need a way to connect the spans emitted by each service into a single trace. This is handled by propagating trace context via an HTTP header.
The Honeycomb beelines support trace headers in a Honeycomb specific format as well as the W3C Trace Context format.
If all of your services are using a Honeycomb beeline, and you want to include
trace propagation headers in all outbound HTTP calls, you just have to use the
http.RoundTripper
provided by the github.com/honeycombio/beeline-go/wrappers/hnynethttp
package:
import (
"net/http"
"time"
"github.com/honeycombio/beeline-go/wrappers/hnynethttp"
)
func main() {
client := &http.Client{
Transport: hnynethttp.WrapRoundTripper(http.DefaultTransport),
Timeout: time.Second * 5,
}
req, _ := http.NewRequest("GET", "https://int-srv.local/", nil)
res, err := client.Do(req)
}
In the above example, the outgoing call will include an HTTP header containing information about the current trace. The downstream service, if instrumented with a Honeycomb beeline, will automatically parse the information in this header and use it to construct a span with the appropriate trace and parent ids.
In order to support distributed traces that include services instrumented with a Honeycomb Beeline and OpenTelemetry, the Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers, the format used by OpenTelemetry.
In order to specify that a service should parse W3C Trace Context headers from
incoming requests, you must specify an HTTPParserHook
. This is done by using the
WrapHandlerWithConfig
function in the github.com/honeycombio/beeline-go/wrappers/hnynethttp
package.
An HTTPParserHook
is a function that takes an http.Request
as an argument and
returns a propagation.PropagationContext
. The http.Request
is provided to the function
so that the author can make decisions about whether to trust the incoming headers based
on information contained in the request (e.g. perhaps you don’t want to accept headers
from the public internet).
import (
"fmt"
"log"
"net/http"
"github.com/honeycombio/beeline-go/propagation"
"github.com/honeycombio/beeline-go/wrappers/config"
"github.com/honeycombio/beeline-go/wrappers/hnynethttp"
)
func traceHeaderParserHook(r *http.Request) *propagation.PropagationContext {
headers := map[string]string{
"traceparent": r.Header.Get("traceparent"),
}
ctx := r.Context()
ctx, prop, err := propagation.UnmarshalW3CTraceContext(ctx, headers)
if err != nil {
// ...
}
return prop
}
log.Fatal(
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello!")
})
http.ListenAndServe(":8080", hnynethttp.WrapHandlerWithConfig(mux, config.HTTPIncomingConfig{
HTTPParserHook: traceHeaderParserHook,
}))
)
In order to send trace propagation headers in a supported format, you must specify an
HTTPPropagationHook
. This is done by using the WrapRoundTripperWithConfig
function in the
github.com/honeycombio/beeline-go/wrappers/hnynethttp
package when creating an HTTP client.
An HTTPPropagationHook
is a function that takes an http.Request
and a propagation.PropagationContext
as arguments and returns a map of name, value pairs representing serialized headers. Similar to the parsing
hook described above, the HTTP request and Propagation Context objects are passed to this function so
that the author can make decisions about whether to include trace context in the outgoing request (e.g. you
may not wish to send trace context headers when calling a 3rd party API).
import (
"net/http"
"time"
"github.com/honeycombio/beeline-go/propagation"
"github.com/honeycombio/beeline-go/wrappers/config"
"github.com/honeycombio/beeline-go/wrappers/hnynethttp"
)
func propagateTraceHook(r *http.Request, prop *propagation.PropagationContext) map[string]string {
ctx := r.Context()
ctx, headers := propagation.MarshalW3CTraceContext(ctx, prop)
return headers
}
func main() {
client := &http.Client{
Transport: hnynethttp.WrapRoundTripperWithConfig(
http.DefaultTransport, config.HTTPOutgoingConfig{HTTPPropagationHook: propagateTraceHook}
),
Timeout: time.Second * 5,
}
req, _ := http.NewRequest("GET", "https://int-srv.local/", nil)
res, err := client.Do(req)
// ...
}
Because we specified an HTTPPropagationHook
that returns a serialized header in W3C trace context format,
the outgoing request will include the appropriate trace context header.
There are two general approaches to finding out what’s wrong when the Go Beeline isn’t doing what you expect.
Set STDOUT
to true
in the initialization of the Go Beeline. This will print the JSON representation
of events to the terminal instead of sending them to Honeycomb. This lets you quickly see what’s getting sent
and allows you to modify your code accordingly.
Note: When STDOUT
is set to true, events will only be sent to the
terminal, not to Honeycomb itself. Remove it when you are finished inspecting
the events to start sending to the Honeycomb API again.
func main() {
beeline.Init(beeline.Config{
WriteKey: "YOUR_API_KEY",
Dataset: "MyGoApp",
STDOUT: true,
})
Set Debug
to true
in the initialization of the Go Beeline. This will print the responses that come back from
Honeycomb to the terminal when sending events. These responses will have extra detail explaining why events are being
rejected (or that they are being accepted) by Honeycomb.
func main() {
beeline.Init(beeline.Config{
WriteKey: "YOUR_API_KEY",
Dataset: "MyGoApp",
Debug: true,
})
Here is a sample event created by the Go Beeline. All events
contain Timestamp
, duration_ms
, and meta.type
fields, but each wrapper
adds a different set of additional fields appropriate for its type of request.
{
"Timestamp": "2018-03-20T00:47:25.339Z",
"app.interesting_thing": "banana",
"duration_ms": 772.446625,
"handler.name": "main.hello",
"handler.pattern": "/hello/",
"handler.type": "http.HandlerFunc",
"meta.beeline_version": "0.2.0",
"meta.local_hostname": "cobbler.local",
"meta.span_type": "root",
"meta.type": "http_request",
"name": "main.hello",
"request.content_length": 0,
"request.header.user_agent": "curl/7.54.0",
"request.host": "localhost:8080",
"request.http_version": "HTTP/1.1",
"request.method": "GET",
"request.path": "/hello/",
"request.remote_addr": "127.0.0.1:60379",
"response.status_code": 200,
"service_name": "sample app",
"trace.span_id": "9e4fe697-3ea9-48c9-b673-72d7ddf118a6",
"trace.trace_id": "b64c89a9-7671-4732-bef1-9ef75ab831f6"
}
Here are some examples to get you started querying your app’s behavior:
GROUP BY
: request.path
VISUALIZE
: P99(duration_ms)
WHERE
: meta.type = http request
ORDER BY
: P99(duration_ms) DESC
GROUP BY
: request.path
VISUALIZE
: COUNT
WHERE
: meta.type = http request
user.email
) 🔗 GROUP BY
: app.user.email
VISUALIZE
: COUNT
WHERE
: request.url == <endpoint-url>
Features, bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file!
All contributions will be released under the Apache License 2.0.