1. Docs
2. Troubleshoot
3. Recommended Migrations
4. From Beelines to OpenTelemetry
5. Beeline for Go

Beeline for Go Reference

Table of Contents

The Go Beeline for Honeycomb is quick and easy way to instrument your Go application. It includes several optional wrappers that automatically instrument HTTP requests and database queries. It also supports tracing out of the box, linking database queries to the HTTP request from which they originated. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey you may find you would like to add new events or traces to add more details specific to your app. The Go Beeline provides simple interfaces for adding both.

To see an example of the Go Beeline in action, try one of the Example Apps.

If you would like to see more options in the Go Beeline, please file an issue or vote up an already filed one!

If you prefer to use only Structured Logs, check out our Libhoney library.

Requirements 

• A Go app, preferably one that listens for HTTP requests or makes SQL calls
• Go 1.18+ and uuid 0.2+
• A Honeycomb API key

You can find your API key in your Environment Settings. If you do not have an API key yet, sign up for a free Honeycomb account.

Quick Installation 

4. Augment the data with interesting information from your app and extra attributes, such as errors, so that you can see rich information about your app in Honeycomb. The context.Context (containing the automatically constructed HTTP event) to use should be present on the request if the Beeline HTTP wrapper was used.

   func handleEndpoint(w http.ResponseWriter, r *http.Request) {
     ctx := r.Context()
     if err := beeline.AddField(ctx, "user_id", userIDFromSession(r)); err != nil {
       beeline.AddField(ctx, "get_session_error", err)
     }
     // ... do the rest of work for the request ...
   }
   

5. Add additional spans and turn a series of events into a trace:

   func slowOp(ctx context.Context) {
     ctx, span := beeline.StartSpan(ctx, "slowOp")
     defer span.Send()
     // ... go on and do the slow opp, add more data along the way
     beeline.AddField(ctx, "interesting_thing", "banana")
   }
   

6. Make sure to call beeline.Close() to send any pending events to Honeycomb before your program exits.

Adding Fields to Events 

The middleware wrapper creates a Honeycomb event in the request context as a span in the overall trace. This span exists throughout the request’s lifecycle, allowing you to add as many additional custom fields as you like.

Here is an example of adding a custom field:

func calcBigNum(ctx context.Context) {
  // ... do work, find a BigNum
  beeline.AddField(ctx, "big_num", result)
  // ... do thing with BigNum, maybe something fails
  if err != nil {
    beeline.AddField(ctx, "error", err)
  }
}

Additional fields are added under the app. namespace. For example, the field above would appear in your event as app.big_num. The namespace groups your fields together to make them easy to find and examine.

If you prefer to avoid the app. namespace, for example in order to overwrite an automatically-populated field, you can extract the current span instance and call AddField directly on that instance.

import "github.com/honeycombio/beeline-go/trace"

func calcOtherBigNum(ctx context.Context) {
  // ... do work, find another BigNum
  span := trace.GetSpanFromContext(r.Context())
  span.AddField(ctx, "handler.name", "calc_other_big_num")
}

These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per-user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred.

It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user’s ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request.

If you are interested in adding custom fields to all spans, use AddFieldToTrace instead.

AddFieldToTrace adds the field to both the currently active span and all other spans that have yet to be sent and are involved in this trace that occur within this process. Additionally, these fields are packaged up and passed along to downstream processes if they are also using a Beeline. This function is good for adding fields that is better scoped to the request than this specific unit of work. For example: user IDs, globally relevant feature flags, errors, and so on. Fields added here are also prefixed with app.

Adding Spans to a Trace 

We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder.

Adding spans with the Go Beeline is easy! Here is an example, where calls to slowOp get their own span within the trace:

func slowOp(ctx context.Context) {
  ctx, span := beeline.StartSpan(ctx, "slowOp")
  defer span.Send()
  // ... go on and do the slow opp, add more data along the way
  beeline.AddField(ctx, "interesting_thing", "banana")
}

Spans always get a few fields:

• a name - in this case slowOp
• a duration - how much time elapsed between when the span was started and sent
• a service_name - generally configured during Beeline initialization
• several IDs - trace, span, and parent identifiers (UUIDs)

You are always welcome (and encouraged!) to add additional fields to spans using the beeline.AddField function.

Wrappers and Other Middleware 

After the router has parsed the request, more fields specific to that router are available, such as the specific handler matched or any request parameters that might be attached to the URL.

You can use a wrapper to capture additional fields specific to a particular type of request. The available wrappers are listed below. You can find additional information and detailed instructions in the GoDoc links for each subpackage.

HTTP Wrappers 

• net/http: hnynethttp
• goji/mux: hnygoji (Goji v2.0+)
• gorilla/mux: hnygorilla
• httprouter: hnyhttprouter

Database Wrappers 

• database/sql: hnysql
• github.com/jmoiron/sqlx: hnysqlx
• github.com/gobuffalo/pop: pop

Optional Configuration 

If you would like to use another framework that supports middleware, you may be able to adapt one of the standard wrappers. We recommend starting with the hnynethttp wrapper, as it expects a function that takes a http.Handler and returns a http.Handler.

In order to have traces connect HTTP wrapped packages all the way down to the database (using the sql or sqlx wrappers), you must pass the context from the *http.Request through to the SQL package using the appropriate Context-enabled function calls. This context ties the SQL calls back to specific HTTP requests, so you can include additional details such as how much time was spent in the DB. It also connects the request IDs from separate events so you can see exactly which DB calls were triggered by a given event.

For very high throughput services, you can send only a portion of the events flowing through your service by setting the SampleRate during initialization. This sample rate will send 1/n events, so a sample rate of 5 would send 20% of all events. For high throughput services, a sample rate of 100 is a good start.

Augmenting or Scrubbing Spans 

If you have some transformations you would like to make on every span before it leaves the process for Honeycomb, the PresendHook is your opportunity to make these changes. Examples are scrubbing sensitive data – for example, you may want to ensure specific fields are dropped or hashed – or augmenting data – for example, making out-of-band translations between an ID and a more human readable version. Similar to the SamplerHook discussed below, you pass the PresendHook a function that will be called on every span with the fields of that span as an argument. The function is free to mutate the map passed in and those mutations will be what finally gets sent to Honeycomb.

As an example, say we have some database calls that insert secret values into a table. Normally, we want to see all the query arguments and DB queries that come from our instrumented sql package, but we specifically want to protect things that go into the secrets table. This code will examine all spans just before they are sent to Honeycomb and replace the arguments to any SQL statement that involves the secrets table with a sha1 hash of those arguments.

// hashSecretsArguments will replace the query arguments to SQL queries against
// the secrets table with a hash of those arguments, so that the contents of the
// secrets inserted into a table are not passed along.
func hashSecretsArguments(fields map[string]interface{}) {
  // if the span is a DB span and has a query
  if query, ok := fields["db.query"]; ok {
    // and the query accesses the secrets table
    if strings.Contains(query, "secrets") {
      // scrub the arguments passed in to the query
      // replace them with a sha1 hash of the original args
      args, ok := fields["db.query_args"]; ok {
        hashedArgs := sha1.Sum([]byte(args))
        fields["db.query_args"] = hashedArgs
      }
    }
  }
}

func main() {
  beeline.Init(beeline.Config{
    WriteKey:    "YOUR_API_KEY",
    ServiceName: "MyGoApp",
    Debug:       true,
    PresendHook: hashSecretsArguments,
  })
}

Sampling Events 

To sample a portion of events for very high throughput services, include an integer SampleRate in the initialization of the Go Beeline. This sends 1/n of all events, so a sample rate of 5 would send 20% of your events.

func main() {
  beeline.Init(beeline.Config{
    WriteKey:   "YOUR_API_KEY",
    ServiceName:"MyGoApp",
    Debug:      true,
    SampleRate: 5, // This sends 1/5 (20%) of events
  })
}

The value of SampleRate must be a positive integer.

Sampling is performed by default on a per-trace level in the Go Beeline, so adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.

The Honeycomb engine weights query results based on sample rate to ensure that sampled computations return correct results.

Customizing Sampling Logic 

Our Beeline lets you define a SamplerHook in order to customize the logic used for deterministic per-trace sampling.

For example, assume you have instrumented an HTTP server. You would like to keep all requests to login, skip all health checks, and sample the rest at a default rate. You could define a sampler function like so:

import (
  "crypto/sha1"
  "math"
)

// Deterministic shouldSample taken from https://github.com/honeycombio/beeline-go/blob/7df4c61d91994bd39cc4c458c2e4cc3c0be007e7/sample/deterministic_sampler.go#L55-L57
func shouldSample(traceId string, sampleRate int) bool {
  upperBound := math.MaxUint32 / uint32(sampleRate)
  sum := sha1.Sum([]byte(traceId))
  // convert last 4 digits to uint32
  b := sum[:4]
  v := uint32(b[3]) | (uint32(b[2]) << 8) | (uint32(b[1]) << 16) | (uint32(b[0]) << 24)
  return v < upperBound
}

func sampler(fields map[string]interface{}) (bool, int) {
  // default sample rate of 10
  sampleRate = 10

  switch fields["request.url"] {
  case "/x/alive": // drop health checks
    return false, 0
  case "/login": // keep all login requests
    return true, 1
  }

  // all the rest of the traffic falls through to the default case

  if shouldSample(fields["trace.trace_id"].(string), sampleRate) {
    return true, sampleRate
  }
  return false, 0
}

func main() {
  beeline.Init(beeline.Config{
    WriteKey:    "YOUR_API_KEY",
    ServiceName: "MyGoApp",
    Debug:       true,
    SamplerHook: sampler,
  })
}

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

If all of your services are using a Honeycomb beeline, and you want to include trace propagation headers in all outbound HTTP calls, you just have to use the http.RoundTripper provided by the github.com/honeycombio/beeline-go/wrappers/hnynethttp package:

import (
  "net/http"
  "time"

  "github.com/honeycombio/beeline-go/wrappers/hnynethttp"
)

func main() {
  client := &http.Client{
    Transport: hnynethttp.WrapRoundTripper(http.DefaultTransport),
    Timeout:   time.Second * 5,
  }
  req, _ := http.NewRequest("GET", "https://int-srv.local/", nil)
  res, err := client.Do(req)
}

In the above example, the outgoing call will include an HTTP header containing information about the current trace. The downstream service, if instrumented with a Honeycomb beeline, will automatically parse the information in this header and use it to construct a span with the appropriate trace and parent ids.

Interoperability with OpenTelemetry 

Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the W3C Trace Context specification.

To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers.

The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used.

To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services.

Note: Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed.

To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration.

Parser Hook 

An HTTPParserHook is a function that takes an http.Request as an argument and returns a propagation.PropagationContext. The http.Request is provided to the function so that the author can make decisions about whether to trust the incoming headers based on information contained in the request. For example, perhaps you do not want to accept headers from the public internet.

To specify an HTTPParserHook for incoming W3C headers, use the WrapHandlerWithConfig function in the github.com/honeycombio/beeline-go/wrappers/hnynethttp package.

import (
  "fmt"
  "log"
  "net/http"

  "github.com/honeycombio/beeline-go/propagation"
  "github.com/honeycombio/beeline-go/wrappers/config"
  "github.com/honeycombio/beeline-go/wrappers/hnynethttp"
)

func traceHeaderParserHook(r *http.Request) *propagation.PropagationContext {
  headers := map[string]string{
    "traceparent": r.Header.Get("traceparent"),
  }
  ctx := r.Context()
  ctx, prop, err := propagation.UnmarshalW3CTraceContext(ctx, headers)
  if err != nil {
    // ...
  }
  return prop
}

log.Fatal(
  mux := http.NewServeMux()
  mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello!")
  })
  http.ListenAndServe(":8080", hnynethttp.WrapHandlerWithConfig(mux, config.HTTPIncomingConfig{
    HTTPParserHook: traceHeaderParserHook,
  }))
)

Propagation Hook 

An HTTPPropagationHook is a function that takes an http.Request and a propagation.PropagationContext as arguments and returns a map of name, value pairs representing serialized headers. Similar to the parsing hook described above, the HTTP request and Propagation Context objects are passed to this function so that the author can make decisions about whether to include trace context in the outgoing request. For example, you may not wish to send trace context headers when calling a 3rd party API.

To specify an HTTPPropagationHook for outgoing W3C headers, use the WrapRoundTripperWithConfig function in the github.com/honeycombio/beeline-go/wrappers/hnynethttp package when creating an HTTP client.

import (
  "net/http"
  "time"

  "github.com/honeycombio/beeline-go/propagation"
  "github.com/honeycombio/beeline-go/wrappers/config"
  "github.com/honeycombio/beeline-go/wrappers/hnynethttp"
)

func propagateTraceHook(r *http.Request, prop *propagation.PropagationContext) map[string]string {
  ctx := r.Context()
  ctx, headers := propagation.MarshalW3CTraceContext(ctx, prop)
  return headers
}

func main() {
  client := &http.Client{
    Transport: hnynethttp.WrapRoundTripperWithConfig(
      http.DefaultTransport, config.HTTPOutgoingConfig{HTTPPropagationHook: propagateTraceHook}
    ),
    Timeout:   time.Second * 5,
  }
  req, _ := http.NewRequest("GET", "https://int-srv.local/", nil)
  res, err := client.Do(req)
  // ...
}

Because we specified an HTTPPropagationHook that returns a serialized header in W3C trace context format, the outgoing request will include the appropriate trace context header.

Troubleshooting the Beeline 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

The Events I am Generating Do Not Contain the Content I Expect 

Set STDOUT to true in the initialization of the Go Beeline. This setting will print the JSON representation of events to the terminal instead of sending them to Honeycomb. This output lets you quickly see what is getting sent and allows you to modify your code accordingly.

func main() {
 beeline.Init(beeline.Config{
   WriteKey:    "YOUR_API_KEY",
   ServiceName: "MyGoApp",
   STDOUT:      true,
 })
}

The Events I am Sending Are Not Being Accepted by Honeycomb 

Set Debug to true in the initialization of the Go Beeline. This will print the responses that come back from Honeycomb to the terminal when sending events. These responses will have extra detail explaining why events are being rejected (or that they are being accepted) by Honeycomb.

func main() {
  beeline.Init(beeline.Config{
    WriteKey:    "YOUR_API_KEY",
    ServiceName: "MyGoApp",
    Debug:       true,
  })
}

My Traces Are Showing Missing Root Spans 

There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause “missing span” gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service’s beeline configuration that only Honeycomb propagation headers should be parsed.

To override undesired W3C trace header propagation behavior, configure the Beeline to use a traceParserHook:

func traceParserHook(r *http.Request) *propagation.PropagationContext {
  headers := map[string]string{
    "traceparent": r.Header.Get("traceparent"),
  }
  ctx := r.Context()
  ctx, prop, err := propagation.UnmarshalHoneycombTraceContext(ctx, headers)
  if err != nil {
    fmt.Println("Error unmarshaling header")
    fmt.Println(err)
  }
  return prop
}

The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See Distributed Trace Propagation for more details.

Example Event 

Here is a sample event created by the Go Beeline. All events contain Timestamp, duration_ms, and meta.type fields, but each wrapper adds a different set of additional fields appropriate for its type of request.

{
  "Timestamp": "2018-03-20T00:47:25.339Z",
  "app.interesting_thing": "banana",
  "duration_ms": 772.446625,
  "handler.name": "main.hello",
  "handler.pattern": "/hello/",
  "handler.type": "http.HandlerFunc",
  "meta.beeline_version": "0.2.0",
  "meta.local_hostname": "cobbler.local",
  "meta.span_type": "root",
  "meta.type": "http_request",
  "name": "main.hello",
  "request.content_length": 0,
  "request.header.user_agent": "curl/7.54.0",
  "request.host": "localhost:8080",
  "request.http_version": "HTTP/1.1",
  "request.method": "GET",
  "request.path": "/hello/",
  "request.remote_addr": "127.0.0.1:60379",
  "response.status_code": 200,
  "service_name": "sample app",
  "trace.span_id": "9e4fe697-3ea9-48c9-b673-72d7ddf118a6",
  "trace.trace_id": "b64c89a9-7671-4732-bef1-9ef75ab831f6"
}

Queries to Try 

Here are some examples to get you started querying your app’s behavior:

Which Endpoints Are the Slowest? 

• GROUP BY: request.path
• VISUALIZE: P99(duration_ms)
• WHERE: meta.type = http request
• ORDER BY: P99(duration_ms) DESC

Which Endpoints Are the Most Frequently Hit? 

• GROUP BY: request.path
• VISUALIZE: COUNT
• WHERE: meta.type = http request

Which Users Are Using the Endpoint That I Would Like to Deprecate? (Using a Custom Field user.email) 

• GROUP BY: app.user.email
• VISUALIZE: COUNT
• WHERE: request.url == <endpoint-url>

Contributions 

Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub. Remember to add your name to the CONTRIBUTORS file!

All contributions will be released under the Apache License 2.0.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit