1. Docs
2. Troubleshoot
3. Recommended Migrations
4. From Beelines to OpenTelemetry
5. Beeline for Java

Beeline for Java Reference

Table of Contents

The Java Beeline for Honeycomb is a quick and easy way to instrument your Java application. If you are using Spring Boot 2, it automatically instruments HTTP requests and provides tracing information out of the box. It also provides methods for adding additional information and metadata to your generated events, and creating custom spans to fine-tune your instrumentation.

If you do not have a Spring Boot v2 application, you can still take advantage of the Java Beeline to add tracing to your app. Only a small amount of additional setup code is required. This is detailed below.

If you would like to see more options in the Java Beeline, please file an issue or vote for an existing issue!

If you prefer to use only Structured Logs, check out our Libhoney library.

Requirements 

To take advantage of automatic HTTP instrumentation and tracing support, you will need a Java Spring Boot v2 app. Otherwise, any Java application will suffice. You will just have a bit more configuration code to get traces. You wil also need a Honeycomb API key.

You can find your API key in your Environment Settings. If you do not have an API key yet, sign up for a free Honeycomb account.

Quick Installation With Java Spring Boot v2 

3. In your Spring app, add the following configuration to your application.properties file in src/main/resources:

   # (Required) Give your application a name to identify the origin of your Honeycomb Events/Spans,
   # and to define where trace data is sent.
   honeycomb.beeline.service-name=<service_name>
   
   # (Required) Your honeycomb account API key
   honeycomb.beeline.write-key=<Honeycomb API key>
   
   # (Optional) Sets the global sample rate of traces.
   honeycomb.beeline.sample-rate=1
   
   # (Optional) Allows the entire Beeline AutoConfiguration to be disabled completely.
   honeycomb.beeline.enabled=true
   
   # (Optional) Allows to switch off automatic instrumentation of the RestTemplate.
   honeycomb.beeline.rest-template.enabled=true
   
   # (Optional) Allows overriding of the beeline's servlet filter precedence
   # - in case you want your own filters to come before it.
   #honeycomb.beeline.filter-order=
   
   # (Optional) Enables a form of debug logging of responses from Honeycomb's server
   honeycomb.beeline.log-honeycomb-responses=true
   
   # (Optional) List of paths that should be subject to tracing (ant path pattern)
   honeycomb.beeline.include-path-patterns=/**
   
   # (Optional) List of paths that should not be subject to tracing (ant path pattern)
   honeycomb.beeline.exclude-path-patterns=/exclude-this-path
   
   # (Optional) For testing you can override Honeycomb's hostname and redirect Events/Spans.
   #honeycomb.beeline.api-host=http://localhost:8089
   
   # (Optional) For using the W3C trace context format instead of Honeycomb's
   #honeycomb.beeline.propagators=w3c
   

   After you build and run your app with this configuration, the beeline will automatically start sending events to Honeycomb.

4. Augment the data with interesting information from your app with beeline.getActiveSpan().addField so that you can see rich information about your app in Honeycomb.

   import io.honeycomb.beeline.tracing.Beeline;
   
   @Controller
   class OwnerController {
   
       private final Beeline beeline;
       private static final String VIEWS_OWNER_CREATE_OR_UPDATE_FORM = "owners/createOrUpdateOwnerForm";
       private final OwnerRepository owners;
   
       public OwnerController(OwnerRepository clinicService, Beeline beeline) {
           this.owners = clinicService;
           this.beeline = beeline;
       }
   
       @GetMapping("/owners/new")
       public String initCreationForm(Map<String, Object> model) {
           Owner owner = new Owner();
           this.beeline.getActiveSpan().addField("new_owner", owner);
           model.put("owner", owner);
           return VIEWS_OWNER_CREATE_OR_UPDATE_FORM;
       }
   }
   

5. Add child spans with the @ChildSpan annotation and turn a series of events into a trace. Important: @ChildSpan will only work on methods called in classes other than the root controller instrumented by the Beeline. (This is due to Spring AOP using a proxy mechanism).

   import io.honeycomb.beeline.spring.beans.aspects.ChildSpan;
   
   @ChildSpan("find an owner")
   @GetMapping("/owners/find")
   public String initFindForm(Map<String, Object> model) {
       model.put("owner", new Owner());
       return "owners/findOwners";
   }
   

Tracing Across Services With Spring Boot v2 Beeline 

The Beeline includes a RestTemplateInterceptor, which can be used to propagate tracing information across HTTP requests to ensure the requests can be tied together in queries in the Honeycomb UI. If a Spring Boot v2 app, which has been instrumented with the Java Beeline, uses RestTemplate to call another Beeline instrumented service, the X-Honeycomb-Trace header will be set accordingly and the app on the other side will understand this context to set the tracing information correctly.

The Interceptor is on by default when Beeline automatic instrumentation has been added to a Spring Boot v2 app and can be turned off using the honeycomb.beeline.rest-template.enabled setting in application.properties.

Testing With Spring Boot v2 Beeline 

When testing a Spring Boot controller class that uses the beeline, mock the beeline using Mockito’s @MockBean annotation with Mockito’s Answers.RETURNS_DEEP_STUBS option:

@RunWith(SpringRunner.class)
@WebMvcTest(OwnerController.class)
public class OwnerControllerTests {

    private static final int TEST_OWNER_ID = 1;

    @Autowired
    private MockMvc mockMvc;

    @MockBean
    private OwnerRepository owners;

    @MockBean(answer = Answers.RETURNS_DEEP_STUBS)
    private Beeline beeline;

    private Owner george;

    @Before
    public void setup() {
        ...
    }

Installation With Any Java Application (Simple) 

1. The Java Beeline is available from Maven Central Repository. To get started, add this to your Maven build’s pom.xml:

   <dependency>
     <groupId>io.honeycomb.beeline</groupId>
     <artifactId>beeline-core</artifactId>
     <version>2.0.0</version>
   </dependency>
   

   or in Gradle:

   implementation 'io.honeycomb.beeline:beeline-core:2.0.0'
   

package com.example.MyApp;

import io.honeycomb.beeline.DefaultBeeline;
import io.honeycomb.beeline.builder.BeelineBuilder;

/// ...

public static void main( String[] args ) {
    DefaultBeeline beeline = DefaultBeeline.getInstance(new BeelineBuilder().writeKey("WRITE_KEY"), "myServiceName");
    // ...

    // Call close at program termination to ensure all pending
    // spans are sent.
    beeline.close();
}

package com.example.MyApp;

import io.honeycomb.beeline.DefaultBeeline;

/// ...

public static void main( String[] args ) {
    DefaultBeeline beeline = DefaultBeeline.getInstance("test-dataset", "myServiceName", "WRITE_KEY");

    // ...

    // Call close at program termination to ensure all pending
    // spans are sent.
    beeline.close();
}

Note: If your servers are behind an http proxy, you will need to initialize the beeline with additional transport options:

URI honeycombAPIEndpoint;
try {
   honeycombAPIEndpoint = new URI("https://api.honeycomb.io"); // US instance
   //honeycombAPIEndpoint = new URI("https://api.eu1.honeycomb.io"); // EU instance

} catch (URISyntaxException e) {
    e.printStackTrace();
}
DefaultBeeline beeline = DefaultBeeline.getInstance("test-dataset", "my-app", "WRITE_KEY", honeycombAPIEndpoint,
  LibHoney.transportOptions().setProxy(HttpHost.create("https://myproxy.example.com")).build());

3. Start adding trace spans at interesting points in your code. For example, add a span at the start of an incoming request or the beginning of an asynchronous job. Start more spans as needed, such as when doing a third party API call or an expensive computation. You can add useful context to spans with addField.

// startSpan will begin a trace if there is no active trace.
// Otherwise, it will create child span of the currently active span.
Span rootSpan = beeline.startSpan("request start");

// Add context by calling addField on spans.
// This is useful for describing what's happening
// in your system.
rootSpan.addField("user_id", userId);
rootSpan.addField("endpoint", requestEndpoint);

// You can also call addField directly on the DefaultBeeline
// to add fields to the current 'active' span.
beeline.addField("success", true);

// When needed, add more spans:
Span callDataAPISpan = beeline.startSpan("call data API");

// These child spans can be annotated with
// yet more fields
callDataAPISpan.addField("user_id", userId);
callDataAPISpan.addField("action", "store result");

// Once the work you're wrapping with a span is finished,
// send it to the Honeycomb API by using sendActiveSpan()
// or close() on the span itself.
callDataAPI(); // do the actual work
callDataAPISpan.close(); // finish timing the work

// Don't forget to send the root too.
rootSpan.close();

// Once all spans have been closed end the trace.
beeline.getBeeline().getTracer().endTrace();

Installation With Any Java Application (Advanced) 

1. The Java Beeline is available from Maven Central Repository. To get started, add this to your Maven build’s pom.xml:

   <dependency>
     <groupId>io.honeycomb.beeline</groupId>
     <artifactId>beeline-core</artifactId>
     <version>2.0.0</version>
   </dependency>
   

   or in Gradle:

   implementation 'io.honeycomb.beeline:beeline-core:2.0.0'
   

2. In your Java app, first set up the Beeline and its collaborator classes:

   package com.mycompany.services.Honeycomb;
   
   import io.honeycomb.beeline.tracing.Beeline;
   import io.honeycomb.beeline.tracing.Span;
   import io.honeycomb.beeline.tracing.SpanBuilderFactory;
   import io.honeycomb.beeline.tracing.SpanPostProcessor;
   import io.honeycomb.beeline.tracing.Tracer;
   import io.honeycomb.beeline.tracing.Tracing;
   import io.honeycomb.beeline.tracing.sampling.Sampling;
   import io.honeycomb.libhoney.HoneyClient;
   import io.honeycomb.libhoney.LibHoney;
   
   public class TracerSpans {
       private static final String WRITE_KEY = "test-write-key";
       private static final String DATASET = "test-dataset";
   
       private static final HoneyClient client;
       private static final Beeline beeline;
   
       static {
           client                          = LibHoney.create(LibHoney.options().setDataset(DATASET).setWriteKey(WRITE_KEY).build());
           SpanPostProcessor postProcessor = Tracing.createSpanProcessor(client, Sampling.alwaysSampler());
           SpanBuilderFactory factory      = Tracing.createSpanBuilderFactory(postProcessor, Sampling.alwaysSampler());
           Tracer tracer                   = Tracing.createTracer(factory);
           beeline                         = Tracing.createBeeline(tracer, factory);
       }
   }
   

3. Augment the data with interesting information from your app, so that you can see rich information about your app in Honeycomb. Use the spanBuilderFactory to create a root span. Add custom fields to that span with beeline.getActiveSpan().addField().

   import io.honeycomb.beeline.tracing.Beeline;
   import io.honeycomb.beeline.tracing.Span;
   import io.honeycomb.beeline.tracing.SpanBuilderFactory;
   import io.honeycomb.beeline.tracing.SpanPostProcessor;
   import io.honeycomb.beeline.tracing.Tracer;
   import io.honeycomb.beeline.tracing.Tracing;
   import io.honeycomb.beeline.tracing.sampling.Sampling;
   import io.honeycomb.libhoney.HoneyClient;
   import io.honeycomb.libhoney.LibHoney;
   
   public static void main(String... args) {
           TracerSpans example = new TracerSpans();
       try {
           HttpRequest request = new HttpRequest();
           startTrace(request);
           example.acceptRequest(request);
       } finally {
           beeline.getTracer().endTrace();
           client.close(); // close to flush events and release its thread pool
       }
   }
   
   private DatabaseService db = new DatabaseService();
   
   private static void startTrace(HttpRequest request) {
   
       Span rootSpan = beeline.getSpanBuilderFactory().createBuilder()
           .setSpanName("get-customer-data")
           .setServiceName("customer-db-traced")
           .build();
       beeline.getTracer().startTrace(rootSpan);
   }
   
   public void acceptRequest(HttpRequest request) {
       Span span = beeline.getActiveSpan();
       try {
           db.queryDb(request.getParameter("customer-id"));
           span.addField("result", "OK");
       } catch (Exception e) {
           span.addField("result", "Bad Request")
               .addField("exception-message", e.getMessage());
       }
   }
   

In this example, when the parent context is set in SpanBuilderFactory, the traceId in the context becomes the id for the trace initialized with startTrace. The spanId in the context becomes the parentSpanId of the next span in the trace. Effectively startTrace is more like “continue trace” when there is parent context involved. This is how you can achieve cross-service traces using the Java Beeline.

1. Add additional spans with beeline.startChildSpan() and turn a series of spans into a trace:

   public static class DatabaseService {
       public void queryDb(String id) {
           Span childSpan = beeline.startChildSpan("customer-db-query");
           try {
               String data = getCustomerDataById(id);
               childSpan.addField("customer-data", data);
           } catch (Exception e) {
               childSpan.addField("db.error", e.getMessage());
           } finally {
               childSpan.close();
           }
   
       }
   
       public String getCustomerDataById(String id) {
           return "customer-0123";
       }
   }
   

Instrumenting HTTP Clients and Servers 

The Beeline Spring Boot Starter automatically instruments the HTTP server and client inside your Spring Boot application. If you are not using Spring Boot, you can still instrument your HTTP server and client code using Beeline Core’s HTTP instrumentation utilities.

Instrumenting HTTP Clients 

Beeline Core contains high-level utility classes to help you instrument the HTTP client of your choice. Using this pattern will open and close a child span for each HTTP client call and add a standard set of fields to this span based on the data in the HTTP request and response. The class that does the work is HttpClientPropagator. You pass implementations to it of HttpClientRequestAdapter and HttpClientResponseAdapter, which adapt the HTTP request and response abstractions used in your HTTP client library. This allows the HttpClientPropagator to read and write to the request and response so the client call can be traced. The general pattern is as follows:

public class HttpClientCallInterceptor {
    // Initialized in constructor
    private final HttpClientPropagator clientPropagator;

    ///...

    public HttpResponse instrumentedClientCall(final HttpRequest actualHttpRequest) throws Exception {
            // Adapt your actual HTTP request so it can be read by the HttpClientPropagator
            HttpClientRequestAdapter adaptedHttpRequest = new MyHttpClientRequestAdapter(actualHttpRequest);

            // These two variables are so we can reference the adapted response and error (if they exist) when closing the child span using #endPropagation
            HttpClientResponseAdapter adaptedHttpResponse = null;
            Throwable error = null;

            // Call #startPropagation just before you execute the actual HTTP request; this will start the child span and add standard HTTP span fields.
            Span span = clientPropagator.startPropagation(adaptedHttpRequest);

            try {
                // Execute the actual HTTP request
                HttpResponse actualHttpResponse = makeClientCall(actualHttpRequest);

                // Adapt your actual HTTP response so it can be read by the HttpClientPropagator
                adaptedHttpResponse = new MyHttpClientResponseAdapter(actualHttpResponse);

                //Return the actual HTTP response for processing by your code
                return actualHttpResponse;
            } catch (Exception ex) {
                // Assign the error if present
                error = ex;
                throw ex;
            } finally {
                // Call #endPropagation when request has finished. This closes the child span and adds the standard HTTP span fields based on whether the error/response are non-null.
                clientPropagator.endPropagation(adaptedHttpResponse, error, span);
            }
    }

    private HttpResponse makeClientCall(HttpRequest httpRequest) {
        // Make the actual HTTP client call
        /// ...
    }

    class MyHttpClientRequestAdapter implements HttpClientRequestAdapter {

            private final HttpRequest httpRequest;

            public MyHttpClientRequestAdapter(HttpRequest httpRequest) {
                this.httpRequest = httpRequest;
            }

        //implement methods
        ///...
    }

    class MyHttpClientResponseAdapter implements HttpClientResponseAdapter {

            private final HttpResponse httpResponse;

            public MyHttpClientResponseAdapter(HttpResponse httpResponse) {
                this.httpResponse = httpResponse;
            }

        //implement methods
        ///...
    }
}

It is likely that you would implement this in an interceptor that is added to your HTTP client. An example of this is found in the instrumentation Beeline adds to Spring’s RestTemplate interceptor. This also contains an example of how to adapt an HTTP client request and response.

Instrumenting HTTP Servers 

Beeline Core contains high-level utility classes to help you instrument the HTTP server of your choice. Using this pattern will open and close a span for each HTTP request received by the server and add a standard set of fields to this span based on the data in the HTTP request and response. If the HTTP request is being made by a service that is also using Beeline tracing, then the new span will be the child of the HTTP client’s request span. The class that does the work is HttpServerPropagator. You pass implementations to it of HttpServerRequestAdapter and HttpServerResponseAdapter which adapt your framework’s HTTP request and response abstractions. This allows the HttpServerPropagator to read and write to the request/response so the request handling can be traced. The general pattern is as follows:

// Inside your HTTP server handler class, e.g. a Servlet Filter
public class ServerHandler {

    // Initialized in constructor
    private final HttpServerPropagator serverPropagator;

    ///...

    public HttpResponse instrumentedRequest(final HttpRequest actualHttpRequest) throws Exception {
            // Adapt the actual received HTTP request so it can be read by the HttpServerPropagator
            HttpServerRequestAdapter adaptedHttpRequest = new MyHttpServerRequestAdapter(actualHttpRequest);

            // These two variables are so we can reference the adapted response and error (if they exist) when closing the child span using #endPropagation
            HttpServerResponseAdapter adaptedHttpResponse = null;
            Throwable error = null;

            // Call #startPropagation before you start handling the actual HTTP request; this will start the child span and add standard HTTP span fields.
            Span span = serverPropagator.startPropagation(adaptedHttpRequest);

            try {
                // Handle the HTTP request
                HttpResponse actualHttpResponse = handleRequest(actualHttpRequest);

                // Adapt the actual HTTP response that you will return to the client so it can be read by the HttpServerPropagator
                adaptedHttpResponse = new MyHttpServerResponseAdapter(actualHttpResponse);

                //Return the actual HTTP response to the client
                return actualHttpResponse;
            } catch (Exception ex) {
                // Assign the error if an exception occurred when your code was processing the HTTP request
                error = ex;
                throw ex;
            } finally {
                // Call #endPropagation when the server has finished processing the request. This closes the child span and adds the standard HTTP span fields based on whether the error/response are non-null.
                serverPropagator.endPropagation(adaptedHttpResponse, error, span);
            }
        }

    private HttpResponse handleRequest(HttpRequest actualHttpRequest) {
        ///... handle the actual HTTP request
    }

    class MyHttpServerResponseAdapter implements HttpServerResponseAdapter {

         private final HttpResponse httpResponse;

         public MyHttpServerResponseAdapter(HttpResponse httpResponse) {
             this.httpResponse = httpResponse;
         }
        //implement methods
        ///...
    }

    class MyHttpServerRequestAdapter implements HttpServerRequestAdapter {

        private final HttpRequest httpRequest;

        public MyHttpServerRequestAdapter(HttpRequest httpRequest) {
            this.httpRequest = httpRequest;
        }
        //implement methods
        ///...
    }
}

Beeline Core already uses this approach to add server instrumentation via a Java Servlet Filter. This also contains an example of how to adapt an HTTP server request and response.

Distributed Trace Propagation 

When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace.

Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header.

Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment.

Interoperability With OpenTelemetry 

Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the W3C Trace Context specification.

To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers.

The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used.

To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services.

Note: Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed.

To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration.

Custom Trace Propagation Hook 

Outgoing HTTP requests can contain trace context formatted in HTTP headers. By default, the HttpClientRequestAdapter will be configured to propagate trace context using Honeycomb’s X-Honeycomb-Trace header. The beeline also includes a propagation codec for the W3C trace.

The HttpClientRequestAdapter can be configured to use a custom propagation hook that can write trace context data in other propagation formats. The propagation hook receives both the HTTP request and the propagation context as parameters that can be used in the decision making of what (if any) propagation format is used.

The following example writes propagation context using the built-in Propagation.w3c() propagator.

final HttpClientPropagator propagator = new HttpClientPropagator.Builder(tracer, r -> "span-name")
    .setTracePropagationHook((request, propagationContext) -> {
        return Propagation.w3c().encode(propagationContext);
    })
    .build();

Custom Trace Parser Hook 

Incoming HTTP requests may contain trace context formatted in HTTP headers. By default, the HttpServerPropagator will be configured to parse trace context in Honeycomb’s X-Honeycomb-Trace header. The beeline also includes a propagation codec for the W3C trace format.

The HttpServerPropagator can be configured to use a custom parse hook that can be used to read trace context information from other propagation formats. The parse hook receives the HTTP request, which can be used to determine which (if any) propagation format is used.

The following example tries to extract a propagation context using the built-in W3C propagation format and then falls back to trying to extract context from the Honeycomb format.

final HttpServerPropagator propagator = new HttpServerPropagator.Builder(beeline, "my-service", r -> "span-name")
    .setParsePropagationHook(request -> {
        PropagationContext context = Propagation.w3c.decode(request.getHeaders());
        if (context != PropagationContext.emptyContext()) {
            return context;
        }

        return Propagation.honey().decode(request.getHeaders());
    })
    .build();

Tracing Servlets Outside of Spring Boot 

If you are not using Spring Boot 2, but you are using Java Servlets, you can still use the Beeline Core’s io.honeycomb.beeline.tracing.propagation.BeelineServletFilter to instrument your application for Honeycomb. You just need to manually configure it, depending on your framework choice.

The filter is compatible with Servlet 3.1+, so for instance you could use a ServletContextListener to programmatically configure the filter. In the example below, we use the io.honeycomb.beeline.DefaultBeeline to make the setup concise. However, you could just as well use a io.honeycomb.beeline.tracing.Beeline with a more customized configuration.

import io.honeycomb.beeline.DefaultBeeline;
import io.honeycomb.beeline.builder.BeelineBuilder;
import javax.servlet.DispatcherType;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import java.util.EnumSet;
import io.honeycomb.beeline.tracing.propagation.BeelineServletFilter;

public class ExampleBeelineServletContextListener implements ServletContextListener {
    private static final String SERVICE_NAME = "myServiceName";

    private final DefaultBeeline beeline = DefaultBeeline.getInstance(new BeelineBuilder().writeKey("WRITE_KEY"), SERVICE_NAME);

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        // initialize the filter
        // you may wish to customize it using the builder; for instance to change the default span naming functions
        final BeelineServletFilter beelineServletFilter = BeelineServletFilter.builder()
                                                                            .setBeeline(defaultBeeline.getBeeline())
                                                                            .setServiceName(SERVICE_NAME)
                                                                            .build();

        //add the filter to the context
        sce.getServletContext().addFilter("beelineFilter", beelineServletFilter)
            .addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), true, "/*");

    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        //close the beeline and its underlying resources
        defaultBeeline.close();
    }
}

import io.honeycomb.beeline.DefaultBeeline;
import javax.servlet.DispatcherType;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import java.util.EnumSet;
import io.honeycomb.beeline.tracing.propagation.BeelineServletFilter;

public class ExampleBeelineServletContextListener implements ServletContextListener {
    private static final String SERVICE_NAME = "myServiceName";

    private final DefaultBeeline defaultBeeline = DefaultBeeline.getInstance("myDataset", SERVICE_NAME, "myWriteKey");

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        // initialize the filter
        // you may wish to customize it using the builder; for instance to change the default span naming functions
        final BeelineServletFilter beelineServletFilter = BeelineServletFilter.builder()
                                                                            .setBeeline(defaultBeeline.getBeeline())
                                                                            .setServiceName(SERVICE_NAME)
                                                                            .build();

        //add the filter to the context
        sce.getServletContext().addFilter("beelineFilter", beelineServletFilter)
            .addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), true, "/*");

    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        //close the beeline and its underlying resources
        defaultBeeline.close();
    }
}

More on Adding Fields to Events 

In the above examples, we have gone over adding a custom field to a specific span. If you are interested in adding custom fields to all spans, use Span.addTraceField or Span.AddTraceFields instead. These will add the field(s) to the currently active span and also propagate them down the trace.

Additionally, these fields will be packaged up and passed along to downstream processes if those processes are also using a beeline.

These functions are good for adding information that is better scoped to the request than this specific unit of work, including: user IDs, globally relevant feature flags, errors, and other fields.

More on Spans and Traces 

We encourage people to think about instrumentation in terms of “units of work”. As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize as traces within the Honeycomb query builder.

Spans always get a few fields:

• a name
• a duration - how much time elapsed between when the span was started and sent
• a service_name - generally configured during Beeline initialization
• several IDs - trace, span, and parent identifiers (UUIDs)

Sampling Events 

For very high throughput services, you can send only a portion of the events flowing through your service by enabling sampling. The sample rate N will send 1/N events, so a sample rate of 5 would send 20% of all events. A sample rate of 1 (or omitting sample rate entirely) will send every event. For high throughput services, a sample rate of 100 is a good start.

The value of your sample rate must be a positive integer.

Sampling With Spring Boot 

To start sampling traces with a Spring Boot app, simply set honeycomb.beeline.sample-rate in your application.properties.

# (Optional) Sets the global sample rate of traces.
honeycomb.beeline.sample-rate            :1

The Beeline will deterministically sample 1/N events based on the Span’s trace id. (This means that adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.)

Sampling Without Spring Boot 

Sampling of traces and Spans can be done in two ways:

• First, the SpanBuilderFactory is configured with a “global” DeterministicTraceSampler that accepts a sample rate N and will deterministically sample 1/N events based on a target field such as the Span’s traceId. (This means that adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.) If a trace is not sampled, “noop” spans that carry no data are generated to minimize runtime overhead.

• Second, the SpanBuilderFactory can be configured with a SpanPostProcessor that applies sampling just before a Span is submitted to Honeycomb. This allows sampling to be performed based on the contents of the Span. By writing a custom Sampler you can customize your own sampling logic. The SpanPostProcessor will only apply to Spans that have been sampled by the “global” sampler, because the “noop” Spans carry no data.

The following example creates a SpanPostProcessor that ignores keep alive requests, keeps all login requests and then uses a 1/10 sample rate to all other spans:

final SpanPostProcessor processor = new SpanPostProcessor(client, span -> {
    switch (span.getFields().get("request.url").toString()) {
        case "/x/alive": // drop health checks
            return 0;
        case "/login": // keep all login requests
            return 1;
    }

    return 10; // by default, sample 1/10
})

When the two sampling mechanisms are used together, the effective sample rate is a product of both. However, note that you do not have to use either of the sampling mechanisms and can always configure either one or both to “always sample” with a sampleRate of 1 (Sampling.alwaysSampler() does this).

Troubleshooting 

No Traces for a Service 

The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called unknown_service.

Troubleshooting With Spring Boot 

A DebugResponseObserver is registered by default and enables a form of debug logging of responses from Honeycomb’s server, such as if the event was successfully sent, rejected by the server, or rejected client side. You can also provide your own implementation of io.honeycomb.libhoney.ResponseObserver either as a Spring Bean or by adding it with io.honeycomb.libhoney.HoneyClient#addResponseObserver.

The logging.level setting for both the Beeline and Libhoney packages needs to be set to DEBUG for log entries to be recorded. This can be done via either the command line or by using application.properties entries.

Example application.properties entries:

# enable DEBUG logging in beeline & libhoney
honeycomb.beeline.log-honeycomb-responses=true
logging.level.io.honeycomb.beeline=DEBUG
logging.level.io.honeycomb.libhoney=DEBUG

To explicitly disable debug logging, set honeycomb.beeline.log-honeycomb-responses to false in your application.properties file.

Configuring The Beeline to Disable Sending Events to Honeycomb 

For situations in which you would like to disable sending events to Honeycomb, such as a test environment or in unit tests, use HoneyClient’s constructor in which the Transport can be overridden. Create a mock (substitute) Transport that implements the Transport interface. One similar example can be seen here.

If setting a mock Transport is more than what is needed, simply changing the API key to any other non-null string will prevent sending to Honeycomb. The beeline will not function without an API key, so omitting the key will not have the desired effect.

My Traces Are Showing Missing Root Spans 

There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause “missing span” gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service’s beeline configuration that only Honeycomb propagation headers should be parsed.

To override undesired W3C trace header propagation behavior, configure the Beeline to use a parse hook:

final HttpServerPropagator propagator = new HttpServerPropagator.Builder(beeline, "my-service", r -> "span-name")
    .setParsePropagationHook(request -> {
        PropagationContext context = Propagation.honey().decode(request.getHeaders());
        if (context != PropagationContext.emptyContext()) {
            return context;
        }
    })
    .build();

The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See Distributed Trace Propagation for more details.

Example Event With Spring Boot 

Here is a sample event created by the Java Beeline for Spring Boot:

{
  "Timestamp": "2018-03-20T00:47:25.339Z",
  "duration_ms": 107.446625,
  "meta.beeline_version": "1.0.3",
  "meta.instrumentation_count": 3,
  "meta.instrumentations": ["spring_mvc", "spring_rest_template", "spring_aop"],
  "meta.local_hostname": "cobbler.local",
  "meta.package": "Spring Boot",
  "meta.package_version": "2.1.2.RELEASE",
  "name": "ProcessCreationForm",
  "request.content_length": 99,
  "request.error": "",
  "request.error_detail": "",
  "request.header.accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
  "request.header.content_type": "application/x-www-form-urlencoded",
  "request.header.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36",
  "request.host": "localhost",
  "request.http_version": "HTTP/1.1",
  "request.method": "POST",
  "request.path": "/owners/new",
  "request.query": "",
  "request.remote_addr": "0:0:0:0:0:0:0:1",
  "request.scheme": "http",
  "request.secure": false,
  "request.xhr": false,
  "response.header_content_type": "text/html;charset=UTF-8",
  "response.status_code": 200,
  "service_name": "sample app",
  "spring.method.name": "",
  "spring.request.dispatcher_type": "",
  "spring.request.handler_method": "OwnerController#processCreationForm",
  "spring.request.handler_type": "HandlerMethod",
  "spring.request.matched_pattern": "/owners/new",
  "trace.parent_id": "0676a4de-09f9-4287-baf5-a772b548362c",
  "trace.span_id": "9e4fe697-3ea9-48c9-b673-72d7ddf118a6",
  "trace.trace_id": "b64c89a9-7671-4732-bef1-9ef75ab831f6",
  "type": "http_server"
}

Queries to Try 

Here are some examples to get you started querying your app’s behavior:

Which Endpoints Are The Slowest? 

• GROUP BY: request.path
• VISUALIZE: P99(duration_ms)
• ORDER BY: P99(duration_ms) DESC

Which Endpoints Are The Most Frequently Hit? 

• GROUP BY: request.path
• VISUALIZE: COUNT

Contributions 

Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change via GitHub.

All contributions will be released under the Apache License 2.0.

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit