We use cookies or similar technologies to personalize your online experience and tailor marketing to you. Many of our product features require cookies to function properly. Your use of this site and online product constitutes your consent to these personalization technologies. Read our Privacy Policy to find out more.

X

Honeycomb Beeline for Go

The Go Beeline for Honeycomb is quick and easy way to instrument your Go application. It includes several optional wrappers that automatically instrument HTTP requests and database queries. It also supports tracing out of the box, linking database queries to the HTTP request from which they originated. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey you may find you’d like to add new events or traces to add more details specific to your app. The Go Beeline provides simple interfaces for adding both.

To see an example of the Go Beeline in action, try out the Golang-Gatekeeper Example App.

If you’d like to see more options in the Go Beeline, please file an issue or vote up one already filed! You can also contact us at support@honeycomb.io.

If you prefer more control over your application’s instrumentation, check out our Go SDK.

Requirements

You can find your API key on your Team Settings page. If you don’t have a API key yet, sign up for a Honeycomb trial.

Quick installation

To install the Go Beeline for your application:

  1. From your $GOPATH directory, add the Beeline package and its dependencies using go get:

        $ go get github.com/honeycombio/beeline-go/...
    
  2. In your code, import the beeline-go package and initialize it with your API key and dataset name:

    import "github.com/honeycombio/beeline-go"
    func main() {
        beeline.Init(beeline.Config{
            WriteKey: "YOUR_API_KEY",
            Dataset: "MyGoApp",
        })
    ...
    

    You are currently logged in to the team, so we have populated the write key here to the first write key for that team.

  3. Add a Beeline wrapper in your http.ListenAndServe call. This adds basic instrumentation for each request at the outermost layer of the call stack.

    import "github.com/honeycombio/beeline-go/wrappers/hnynethttp"
    ...
    http.ListenAndServe(":8080", hnynethttp.WrapHandler(muxer))
    
  4. Augment the data with interesting information from your app and extra context such as errors so that you can see rich information about your app in Honeycomb:

    func calcBigNum(ctx context.Context) {
        // ... do work, find a BigNum
        beeline.AddField(ctx, "big_num", result)
        // ... do thing with BigNum, maybe something fails
        if err != nil {
            beeline.AddField(ctx, "error", err)
        }
    }
    
  5. Add additional spans and turn a series of events into a trace:

    func slowOp(ctx context.Context) {
        ctx, span := beeline.StartSpan(ctx, "slowOp")
        defer span.Send()
        // ... go on and do the slow opp, add more data along the way
        beeline.AddField(ctx, "interesting_thing", "banana")
    }
    

Note: In this example we are using hnynethttp, see the Optional configuration section for more about the available wrappers. You can also find a more detailed walkthrough in the Go Beeline tutorial.

Adding additional context to events

The middleware wrapper creates a Honeycomb event in the request context as a span in the overall trace. This span exists throughout the request’s lifecycle, allowing you to add as many additional custom fields as you like.

Here is an example of adding a custom field:

   func calcBigNum(ctx context.Context) {
       // ... do work, find a BigNum
       beeline.AddField(ctx, "big_num", result)
       // ... do thing with BigNum, maybe something fails
       if err != nil {
           beeline.AddField(ctx, "error", err)
       }
   }

Additional fields are added under the app. namespace. For example, the field above would appear in your event as app.big_num. The namespace groups your fields together to make them easy to find and examine.

These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per- user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred.

It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user’s ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request.

Adding additional traces or additional spans in a trace

We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder.

Adding additional spans with the Go Beeline is easy! Here is an example, where calls to slowOp get their own span within the trace:

func slowOp(ctx context.Context) {
    ctx, span := beeline.StartSpan(ctx, "slowOp")
    defer span.Send()
    // ... go on and do the slow opp, add more data along the way
    beeline.AddField(ctx, "interesting_thing", "banana")
}

Spans always get a few fields:

You are always welcome (and encouraged!) to add additional fields to spans using the beeline.AddField function.

Wrappers and other middleware

After the router has parsed the request, more fields specific to that router are available, such as the specific handler matched or any request parameters that might be attached to the URL.

You can use a wrapper to capture additional fields specific to a particular type of request. The available wrappers are listed below. You can find additional information and detailed instructions in the GoDoc links for each subpackage.

HTTP wrappers:

Database wrappers:

Optional configuration

If you would like to use another framework that supports middleware, you may be able to adapt one of the standard wrappers. We recommend starting with the hnynethttp wrapper, as it expects a function that takes a http.Handler and returns a http.Handler.

In order to have traces connect HTTP wrapped packages all the way down to the database (using the sql or sqlx wrappers), you must pass the context from the *http.Request through to the SQL package using the appropriate Context-enabled function calls. This context ties the SQL calls back to specific HTTP requests, so you can include additional details such as how much time was spent in the DB. It also connects the request IDs from separate events so you can see exactly which DB calls were triggered by a given event.

For very high throughput services, you can send only a portion of the events flowing through your service by setting the SampleRate during initialization. This sample rate will send 1/n events, so a sample rate of 5 would send 20% of all events. For high throughput services, a sample rate of 100 is a good start.

Sampling events

To sample a portion of events for very high throughput services, include an integer SampleRate in the initialization of the Go Beeline. This sends 1/n of all events, so a sample rate of 5 would send 20% of your events. Try starting with a value of 10:

   func main() {
       beeline.Init(beeline.Config{
           WriteKey: "YOUR_API_KEY",
           Dataset: "MyGoApp",
           Debug: "true",
           SampleRate: 10,
       })

Sampling is performed by default on a per-trace level in the Go Beeline, so adding sampling won’t break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.

Customizing sampling logic

Our Beeline lets you define a SamplerHook in order to customize the logic used for deterministic per-trace sampling.

For example, assume you have instrumented an HTTP server. You’d like to keep all errored requests and heavily sample healthy traffic (200 response codes). Also, you don’t really care about 302 redirects in your app, so want to drop those. You could define a sampler function like so:

import (
  "crypto/sha1"
  "math"
)

// Deterministic shouldSample taken from https://github.com/honeycombio/beeline-go/blob/7df4c61d91994bd39cc4c458c2e4cc3c0be007e7/sample/deterministic_sampler.go#L55-L57
func shouldSample(traceId string, sampleRate int) bool {
	upperBound := math.MaxUint32 / uint32(sampleRate)
	sum := sha1.Sum([]byte(traceId))
	// convert last 4 digits to uint32
	b := sum[:4]
	v := uint32(b[3]) | (uint32(b[2]) << 8) | (uint32(b[1]) << 16) | (uint32(b[0]) << 24)
	return v < upperBound
}

func sampler(fields map[string]interface{}) (bool, int) {
	sampleRate := 1
	switch fields["response_code"] {
	case 302:
		return false, 0
	case 200: // Only keep 1 out of every 100 traces for successful requests
		sampleRate = 100
	default: // Capture everything else
		if shouldSample(fields["trace.trace_id"].(string), sampleRate) {
			return true, sampleRate
		}
	}
	return false, 0
}

func main() {
  beeline.Init(beeline.Config{
    WriteKey: "YOUR_API_KEY",
    Dataset: "MyGoApp",
    Debug: "true",
    SamplerHook: sampler,
  })
}

Note: Defining a sampling hook overrides the deterministic sampling behavior for trace IDs. Unless you take trace.trace_id into account (as we did above by taking the sha1.Sum of the trace ID), you will get incomplete traces.

Troubleshooting the Beeline

There are two general approaches to finding out what’s wrong when the Go Beeline isn’t doing what you expect.

“The events I’m generating don’t contain the content I expect”

Set STDOUT to true in the initialization of the Go Beeline. This will print the JSON representation of events to the terminal instead of sending them to Honeycomb. This lets you quickly see what’s getting sent and allows you to modify your code accordingly.

   func main() {
       beeline.Init(beeline.Config{
           WriteKey: "YOUR_API_KEY",
           Dataset: "MyGoApp",
           STDOUT: "true",
       })

“The events I’m sending aren’t being accepted by Honeycomb”

Set Debug to true in the initialization of the Go Beeline. This will print the responses that come back from Honeycomb to the terminal when sending events. These responses will have extra detail explaining why events are being rejected (or that they are being accepted) by Honeycomb.

   func main() {
       beeline.Init(beeline.Config{
           WriteKey: "YOUR_API_KEY",
           Dataset: "MyGoApp",
           Debug: "true",
       })

Example event

Here is a sample event created by the Go Beeline. All events contain Timestamp, duration_ms, and meta.type fields, but each wrapper adds a different set of additional fields appropriate for its type of request.

{
  "Timestamp": "2018-03-20T00:47:25.339Z",
  "app.interesting_thing": "banana",
  "duration_ms": 772.446625,
  "handler.name": "main.hello",
  "handler.pattern": "/hello/",
  "handler.type": "http.HandlerFunc",
  "meta.beeline_version": "0.2.0",
  "meta.local_hostname": "cobbler.local",
  "meta.span_type": "root",
  "meta.type": "http_request",
  "name": "main.hello",
  "request.content_length": 0,
  "request.header.user_agent": "curl/7.54.0",
  "request.host": "localhost:8080",
  "request.http_version": "HTTP/1.1",
  "request.method": "GET",
  "request.path": "/hello/",
  "request.remote_addr": "127.0.0.1:60379",
  "response.status_code": 200,
  "service_name": "sample app",
  "trace.span_id": "9e4fe697-3ea9-48c9-b673-72d7ddf118a6",
  "trace.trace_id": "b64c89a9-7671-4732-bef1-9ef75ab831f6"
}

Queries to try

Here are some examples to get you started querying your app’s behavior:

Which endpoints are the slowest?

Which endpoints are the most frequently hit?

Which users are using the endpoint that I’d like to deprecate? (Using a custom field user.email)

Contributions

Features, bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file!

All contributions will be released under the Apache License 2.0.