{"Node.js"}

`, will be sent as `{`lcp.element.data.myAttr`: '' }`, which is inline with the [dataset API](https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/dataset). | | cls | optional | VitalOpts | `undefined` | Pass-through configuration options for web-vitals. Refer to Google Chrome's [ReportOpts](https://github.com/GoogleChrome/web-vitals?tab=readme-ov-file#reportopts). | | cls.applyCustomAttributes | optional | function | `undefined` | A function for adding custom attributes to core web vitals spans. | | inp | optional | VitalOptsWithTimings | `undefined` | Pass-through configuration options for web-vitals. Refer to Google Chrome's [ReportOpts](https://github.com/GoogleChrome/web-vitals?tab=readme-ov-file#reportopts). | | inp.applyCustomAttributes | optional | function | `undefined` | A function for adding custom attributes to core web vitals spans. | | inp.includeTimingsAsSpans | optional | boolean | `false` | When set to `true`, this option will emit `PerformanceLongAnimationFrameTiming` and `PerformanceScriptTiming` as spans. | | fcp | optional | VitalOpts | `undefined` | Pass-through configuration options for web-vitals. Refer to Google Chrome's [ReportOpts](https://github.com/GoogleChrome/web-vitals?tab=readme-ov-file#reportopts). | | fcp.applyCustomAttributes | optional | function | `undefined` | A function for adding custom attributes to core web vitals spans. | | ttf | optional | VitalOpts | `undefined` | Pass-through configuration options for web-vitals. Refer to Google Chrome's [ReportOpts](https://github.com/GoogleChrome/web-vitals?tab=readme-ov-file#reportopts). | | ttf.applyCustomAttributes | optional | function | `undefined` | A function for adding custom attributes to core web vitals spans. | ### Visualizing traces locally Honeycomb's Web Instrumentation package can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation. Enable local visualizations by setting the `localVisualizations` configuration option to `true`. ```javascript theme={} const { HoneycombWebSDK } = require("@honeycombio/opentelemetry-web"); const sdk = new HoneycombWebSDK({ apiKey: "YOUR-API-KEY", // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", serviceName: "YOUR-SERVICE-NAME", localVisualizations: true, debug: true, }); ``` The output displays the name of the root span and a link to Honeycomb that shows its trace. Select the link to view the trace in detail within the Honeycomb UI. ```text theme={} Trace for root-span-name Honeycomb link: ``` Disable local visualizations for production environments. Local visualization creates additional overhead and should only be used during development or testing. ### Enabling debug logging Turn on debug logging by setting the `debug` configuration option to `true`. ```javascript theme={} const { HoneycombWebSDK } = require("@honeycombio/opentelemetry-web"); const sdk = new HoneycombWebSDK({ apiKey: "YOUR-API-KEY", // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", serviceName: "YOUR-SERVICE-NAME", debug: true }); ``` When the debug setting is enabled, the Honeycomb Web Instrumentation package configures a [DiagConsoleLogger](https://github.com/open-telemetry/opentelemetry-js/blob/main/api/src/diag/consoleLogger.ts) that logs telemetry to the console with the log level of Debug. The debug setting in the Honeycomb Web Instrumentation package will also output to the console the main options configuration, including but not limited to API Key and endpoint. Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes. ## Adding automatic instrumentation The Honeycomb OpenTelemetry Web SDK includes [auto-instrumentation](https://github.com/honeycombio/honeycomb-opentelemetry-web/tree/main/packages/honeycomb-opentelemetry-web#auto-instrumentation) for: * [Document & resource loading](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/instrumentation-document-load) * [Fetch requests](https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-instrumentation-fetch) * [XML HTTP requests](https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-instrumentation-xml-http-request) * [User interactions](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/instrumentation-user-interaction) * [Core web vitals](https://github.com/honeycombio/honeycomb-opentelemetry-web/blob/main/docs/web-vitals.md) Automatic instrumentation is enabled by default. You can enable or disable individual auto-instrumentation libraries in your configuration using the `instrumentations` option. ```ts theme={} import { HoneycombWebSDK, WebVitalsInstrumentation } from '@honeycombio/opentelemetry-web'; import { getWebAutoInstrumentations } from '@opentelemetry/auto-instrumentations-web'; const configDefaults = { ignoreNetworkEvents: true, propagateTraceHeaderCorsUrls: [ /.+/g, // Regex to match your backend URLs. Update to the domains you wish to include. ] } const sdk = new HoneycombWebSDK({ // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", apiKey: "YOUR-API-KEY", serviceName: "YOUR-SERVICE-NAME", instrumentations: [ getWebAutoInstrumentations({ // optionally apply defaults config to instrumentation '@opentelemetry/instrumentation-xml-http-request': configDefaults, '@opentelemetry/instrumentation-fetch': configDefaults, // optionally disable document load instrumentation '@opentelemetry/instrumentation-document-load': { enabled: false }, }), new WebVitalsInstrumentation() ], }); sdk.start(); ``` ## Adding custom instrumentation Automatic instrumentation is a fast way to instrument your code, but you get more insight into your application by adding custom, otherwise known as manual, instrumentation. Adding custom instrumentation requires the the OpenTelemetry API package. ```shell npm theme={} npm install --save @opentelemetry/api ``` ```shell yarn theme={} yarn add @opentelemetry/api ``` ### Adding attributes to an active span You can retrieve the currently active span in a trace and add attributes to it. This lets you add more context to traces and gives you more ways to group or filter traces in your queries. ```ts TypeScript theme={} import { trace } from '@opentelemetry/api'; function handleUser(user: User) { let currentActiveSpan = trace.getActiveSpan(); currentActiveSpan.setAttribute('user.id', user.getId()); } ``` ```js JavaScript theme={} const { trace } = require("@opentelemetry/api"); function handleUser(user) { let activeSpan = trace.getActiveSpan(); activeSpan.setAttribute("user.id", user.getId()); } ``` ### Acquiring a tracer For manual tracing, you need to get a tracer: ```typescript TypeScript theme={} import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer("tracer.name.here"); ``` ```js JavaScript theme={} const { trace } = require("@opentelemetry/api"); const tracer = trace.getTracer("tracer.name.here"); ``` When you create a `Tracer`, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter. When traces are sent to Honeycomb, the name of the `Tracer` is turned into the `library.name` field, which can be used to show all spans created from a particular tracer. In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different "layers" of your application, then use the name that corresponds to that "layer". The `library.name` field is also used with traces created from instrumentation libraries. ### Creating spans Create custom spans to get a clear view of the critical parts in your application. ```typescript TypeScript theme={} import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('example-tracer', '0.1.0'); function trackWork() { const span = tracer.startActiveSpan('do work'); console.log('performing work'); span.end(); } ``` ```javascript JavaScript theme={} const { trace } = require("@opentelemetry/api"); const tracer = trace.getTracer("my-service-tracer"); function trackWork() { const span = tracer.startActiveSpan('do work'); console.log('performing work'); span.end(); } ``` ### Adding multi-span attributes Sometimes you want to add the same attribute to many spans within the same trace. This attribute may include variables calculated during your program, or other useful values for correlation or debugging purposes. To add this attribute, leverage the OpenTelemetry concept of [baggage](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#baggage-signal). Baggage allows you to add a `key` with a `value` as an attribute to every subsequent child span within the current application context. ```typescript TypeScript theme={} import { Context, context, propagation, } from '@opentelemetry/api'; tracer.startActiveSpan('main', (span) => { span.setAttribute('app.username', name); // add to current span // new context based on current, with key/values added to baggage const ctx: Context = propagation.setBaggage( context.active(), propagation.createBaggage({ 'app.username': { value: name } }) ); // within the new context, do some work and baggage will be // applied as attributes on child spans context.with(ctx, () => { tracer.startActiveSpan('childSpan', (childSpan) => { doTheWork(); childSpan.end(); }); }); span.end(); }); ``` ```javascript JavaScript theme={} tracer.startActiveSpan('main', (span) => { span.setAttribute('app.username', name); // add to current span // new context based on current, with key/values added to baggage const ctx = propagation.setBaggage( context.active(), propagation.createBaggage({ 'app.username': { value: name } }) ); // within the new context, do some work and baggage will be // applied as attributes on child spans context.with(ctx, () => { tracer.startActiveSpan('childSpan', (childSpan) => { doTheWork(); childSpan.end(); }); }); span.end(); }); ``` ## Adding custom span processing The Honeycomb Web SDK uses span processors as synchronous hooks for when a span starts and when a span ends. This lets you mutate spans after they have been created by automatic instrumentation or manually. Some examples of the actions you can take on spans in a span processor include: * Add attributes * Add span events * Add span links * Update the name of a span * Record an exception on a span * Get the span context to create child spans * Stop spans from being sent ### Example: Basic custom span processor Here is a basic example of a custom span processor: ```typescript theme={} class TestSpanProcessorOne implements SpanProcessor { onStart(span: Span): void { span.setAttributes({ 'processor1.name': 'TestSpanProcessorOne', }); } onEnd(): void {} forceFlush() { return Promise.resolve(); } shutdown() { return Promise.resolve(); } } ``` To use it, add the span processor to your configuration when you initialize the Honeycomb Web SDK: ```typescript theme={} const sdk = new HoneycombWebSDK({ debug: true, apiKey: 'api-key-goes-here', serviceName: 'hny-web-distro', // ... other config spanProcessors: [new TestSpanProcessorOne()], }); sdk.start(); ``` ### Example: Adding user information after SDK initialization Here is an example span processor that adds custom information for users: ```typescript theme={} import { SpanProcessor } from '@opentelemetry/sdk-trace-base'; import { Span } from '@opentelemetry/api'; export class UserInfoSpanProcessor implements SpanProcessor { userInfo: { userId: string; customerId: string; role: string } | undefined; constructor() { const getUserInfo = (): Promise<{ userId: string; customerId: string; role: string; }> => { return new Promise((resolve) => { setTimeout(() => { resolve({ userId: '1234', customerId: '5678', role: 'admin', }); }, 2000); }); }; getUserInfo().then( (userInfo: { userId: string; customerId: string; role: string }) => { this.userInfo = userInfo; }, ); } onStart(span: Span) { if (this.userInfo) { span.setAttributes({ 'app.user.id': this.userInfo.userId, 'app.user.customer_id': this.userInfo.customerId, 'app.user.role': this.userInfo.role, }); } } onEnd() {} forceFlush() { return Promise.resolve(); } shutdown() { return Promise.resolve(); } } ``` To use it, add the span processor to your configuration when you initialize the Honeycomb Web SDK: ```typescript theme={} const sdk = new HoneycombWebSDK({ debug: true, apiKey: 'api-key-goes-here', serviceName: 'hny-web-distro', // ... other configuration spanProcessors: [new UserInfoSpanProcessor()], }); sdk.start(); ``` ### Example: Dynamic page routes with React Router Here is an example span processor that adds attributes to spans based on the state of the React Router. It sets the `page.route` attribute to the generic dynamic route, and records the span as an error if there are errors in the router state. ```typescript theme={} import { SpanProcessor } from '@opentelemetry/sdk-trace-base'; import { Span } from '@opentelemetry/api'; /** * SpanProcessor that adds attributes to spans based on the state of the React Router * Sets the page.route attribute to the generic dynamic route * Records the span as an error if there are errors in the router state (e.g. 404) */ export class ReactRouterSpanProcessor implements SpanProcessor { router; route; constructor({ router }: { router }) { this.router = router; this.route = router.state.matches[router.state.matches.length - 1]?.route.path; this.router.subscribe((state: any) => { this.route = state.matches[state.matches.length - 1]?.route.path; }); } onStart(span: Span) { const { errors } = this.router.state; // If there are errors, set the span status to error and record the error message if (errors !== null) { span.setStatus({ code: 2, message: errors[0].data, }); } // Set the page.route as the generic dynamic route, making things easier to query // e.g. /name/:name/pet/:pet instead of name/123/pet/456 // url.path attribute will have the more specific computed route span.setAttributes({ 'page.route': this.route }); } onEnd() {} forceFlush() { return Promise.resolve(); } shutdown() { return Promise.resolve(); } } ``` To use it, add the span processor to your configuration when you initialize the Honeycomb Web SDK: ```typescript theme={} const sdk = new HoneycombWebSDK({ debug: true, apiKey: 'api-key-goes-here', serviceName: 'hny-web-distro', // ... other configuration spanProcessors: [new ReactRouterSpanProcessor({ router: router })], }); sdk.start(); ``` ## Propagating span context A [Context Manager](https://opentelemetry.io/docs/languages/js/context/#context-manager) stores and propagates global span context through your system. OpenTelemetry provides a context manager for browser instrumentation based on the [Zone.js](https://github.com/angular/angular/tree/main/packages/zone.js) library to track global context across asynchronous execution threads. This context manager can be added used in your instrumentation like so: ```js theme={} import { ZoneContextManager } from '@opentelemetry/context-zone'; const sdk = new HoneycombWebSDK({ // other config options omitted... contextManager: new ZoneContextManager() }); sdk.start(); ``` Zone.js has known limitations with async/await code, and [requires](https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-context-zone-peer-dep#installation) your code to be transpiled down to ES2015. It may also negatively impact your application's performance. For these reasons, the Honeycomb Web SDK does not enable the Zone.js context manager by default. ### Automatically propagate the trace context header Use request automatic instrumentation to automatically send spans for every HTTP request. `@opentelemetry/instrumentation-xml-http-request` automatically instruments XHR requests and `@opentelemetry/instrumentation-fetch` automatically instruments fetch requests. If your browser application uses a request library to make requests, such as [`axios`](https://axios-http.com/) or [`superagent`](https://github.com/ladjs/superagent), these requests are also automatically instrumented by enabling the `xml-http-request` or `fetch` instrumentation, depending on what the library uses to make requests. When using the Honeycomb Instrumentation snippet (as documented on this page), uncomment the `propagateTraceHeaderCorsUrls` array and add regex to include all target domains. This method allows you to propagate to your backend services without leaking trace IDs to third-party services. ```javascript theme={} const configDefaults = { ignoreNetworkEvents: true, propagateTraceHeaderCorsUrls: [ /.+/g, // Regex to match your backend URLs. Update to the domains you wish to include. ] } ``` ### Manually propagate the trace context header It is also possible to manually propagate the [trace context header](https://www.w3.org/TR/trace-context/) if automatic instrumentation is not an option: ```javascript theme={} // General request handler, instrumented with OTel // Forwards traceparent header to connect spans created in the browser // with spans created on the backend const request = async (url, method = 'GET', headers, body) => { return trace .getTracer('request-tracer') .startActiveSpan(`Request: ${method} ${url}`, async (span) => { // construct W3C traceparent header const traceparent = `00-${span.spanContext().traceId}-${span.spanContext().spanId}-01`; try { const response = await fetch(url, { method, headers: { ...headers, // set traceparent header traceparent: traceparent, }, body, }); span.setAttributes({ 'http.method': method, 'http.url': url, 'response.status_code': response.status, }); if (response.ok && response.status >= 200 && response.status < 400) { span.setStatus({ code: SpanStatusCode.OK }); return response.text(); } else { throw new Error(`Request Error ${response.status} ${response.statusText}`); } } catch (error) { span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); throw new Error(error); } finally { span.end(); } }); }; ``` ## Troubleshooting Running into issues? Here are some common problems and ways to fix them. To explore common issues when sending data, visit [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#opentelemetry-sdks-and-honeycomb-distributions). ### Dataset not appearing in Honeycomb The `apiKey` variable is used to send your data to Honeycomb. Make sure you have replaced the placeholder value for it with your Honeycomb Ingest API Key and that your API key permissions include "Can create datasets". If Honeycomb is successfully instantiating but your API key is not included, you should see output similar to the following in your browser console: ```bash theme={} @opentelemetry/api: Registered a global for diag v1.7.0 @honeycombio/opentelemetry-web: Honeycomb Web SDK Debug Mode Enabled @honeycombio/opentelemetry-web: API Key configured for traces: '' @honeycombio/opentelemetry-web: Service Name configured for traces: '' @honeycombio/opentelemetry-web: Endpoint configured for traces: 'https://api.honeycomb.io/v1/traces' @honeycombio/opentelemetry-web: Sample Rate configured for traces: '1' ``` ### Dataset has unexpected name We use the `serviceName` variable to name your dataset in Honeycomb. Be sure you have replaced the placeholder value for it with a name that you will find useful. ### Next.js "Navigator Is Undefined" error If a "navigator is undefined" error appears when you attempt to start your local server while following Next.js instructions, the instrumentation is being run in a server-side rendering path. To fix this, try adding the `'use client';` directive at the top of the file where you instantiate Honeycomb’s web instrumentation. Adding the `'use client';` directive tells React to only execute the file in a client environment. If you are still seeing an error after adding the `'use client';` directive, try wrapping the function in a try/catch block. By wrapping the function, you can catch the error and avoid instantiation in server-side environments, ensuring that your application starts up even if the code is executed in a server-side environment. ```javascript theme={} try { const sdk = new HoneycombWebSDK({ debug: true, apiKey: '[YOUR API KEY HERE]' // Replace with your Honeycomb Ingest API Key serviceName: '[YOUR APPLICATION NAME HERE]', // Replace with your application name. Honeycomb will name your dataset using this variable. instrumentations: [getWebAutoInstrumentations()], // Adds automatic instrumentation }); sdk.start(); } catch (e) {} ``` ### Receiving 464 errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. # Symbolicate JavaScript Stack Traces with the OpenTelemetry Collector Source: https://docs.honeycomb.io/send-data/javascript-browser/symbolicate Use the source map processor in the OpenTelemetry Collector to resolve minified JavaScript stack traces into readable symbols for easier debugging. Use the sourcemap processor in your OpenTelemetry Collector to symbolicate JavaScript (JS) stack traces. The [sourcemap processor](https://github.com/honeycombio/opentelemetry-collector-symbolicator?tab=readme-ov-file#javascript-source-maps) replaces minified names and addresses in your JS stack traces with symbols from provided source maps. ## Before You Start The sourcemap processor is compatible with [Honeycomb OpenTelemetry Web SDK](https://github.com/honeycombio/honeycomb-opentelemetry-web) version 0.12.0 and later. To use the sourcemap processor, you need: * OpenTelemetry Collector built with `CGO` enabled. * An environment or container image with `glibc` support. We recommend `gcr.io/distroless/cc`, a secure and lightweight container image with CGO and `glibc` support. If you're not using the Honeycomb OpenTelemetry Web SDK, make sure your exception data is in [the format the processor expects](https://github.com/honeycombio/opentelemetry-collector-symbolicator/blob/main/README.md#exception-information-format). ## Install By default, the [Honeycomb OpenTelemetry Collector distribution](https://github.com/honeycombio/honeycomb-collector-distro) includes the sourcemap processor, so you can skip to the next section if you're using it. If you use another collector distribution or build your own, it must be built with CGO enabled. You can install the sourcemap processor by adding it to your OpenTelemetry Collector build configuration file: ```yaml theme={} processors: - gomod: github.com/honeycombio/opentelemetry-collector-symbolicator/sourcemapprocessor v0.0.14 ``` You can find the latest sourcemap processor version on the [releases page](https://github.com/honeycombio/opentelemetry-collector-symbolicator/releases) in the GitHub repo. ## Source Files and Source Maps The sourcemap processor requires access to your minimized JS source files and associated source maps. These can be stored in your local file system, Amazon S3, or Google Cloud Storage. To support symbolication, your minified source files must have a comment with a `sourceMappingURL` pointing to the relative path of the source map file. ```js theme={} //# sourceMappingURL=/static/dist/main.c383b093b0b66825a9c3.js.map ``` You should also version your source files and source maps with a file hash in the file name, for example: `vendor.1c285a50f5307be9648d.js`. This helps prevent conflicts and ensure the correct file versions are used. ## Configure a File Store Add the `source_map_symbolicator` as a processor in your OpenTelemetry Collector configuration: ```yaml theme={} processors: source_map_symbolicator: ``` You can then configure where your source maps are stored. By default, the sourcemap processor loads source map files from a local directory. You can set the file path in your collector configuration: ```yaml theme={} processors: source_map_symbolicator: # source_map_store is sets which store to use, in this case local disk source_map_store: file_store local_source_maps: # (optional) path sets the base path of the files, defaults to `.` path: /tmp/sourcemaps ``` Make sure your collector can access the `path` directory you set, and that file paths in stack traces match the structure of your configured file store. #### How the Processor Retrieves Files from Local Disk When retrieving files from local disk, the processor: 1. Gets the base file name from the URL included in the stack trace. 2. The `path`, if configured, is joined with the base file name. 3. Reads the file using the joined path from disk. For example: * Original URL: `https://example.com/static/dist/main.c383b093b0b66825a9c3.js` * `path` is set to `/tmp/sourcemaps` * New path: `/tmp/sourcemaps/main.c383b093b0b66825a9c3.js.map` Optionally, you can load source maps from an Amazon S3 bucket. Add to your OpenTelemetry Collector configuration: ```yaml theme={} processors: source_map_symbolicator: # source_map_store sets which store to use, in this case S3 source_map_store: s3_store s3_source_maps: # name of the bucket the files are stored in bucket: source-maps-bucket # (optional) the bucket's region region: us-east-1 # (optional) prefix is used to nest the files in a sub key of the bucket prefix: source-maps ``` Make sure your collector has permission to access the S3 bucket. Also, ensure the file paths in stack traces match the structure used in your file store. #### How the Processor Retrieves Files from S3 When retrieving files from an Amazon S3 bucket, the processor: 1. Gets the base file name from the URL included in the stack trace. 2. The `prefix`, if configured, is joined with the base file name. 3. Uses this joined path as the key to retrieve the file from the bucket. For example: * Original URL: `https://example.com/static/dist/main.c383b093b0b66825a9c3.js` * `prefix` is set to `source-maps` * New path: `sourcemaps/main.c383b093b0b66825a9c3.js.map` Optionally, you can load source maps from a Google Cloud Storage (GCS) bucket. Add to your OpenTelemetry Collector configuration: ```yaml theme={} processors: source_map_symbolicator: # source_map_store sets which store to use, in this case GCS source_map_store: gcs_store gcs_source_maps: # the name of the bucket the files are stored in bucket: source-maps-bucket # (optional) prefix is used to nest the files in a sub key of the bucket prefix: source-maps ``` Make sure your collector has permission to access the GCS bucket. Also, ensure the file paths in stack traces match the structure used in your file store. #### How the Processor Retrieves Files from GCS When retrieving files from a Google Cloud Storage bucket, the processor: 1. Gets the base file name from the URL included in the stack trace. 2. The `prefix`, if configured, is joined with the base file name. 3. Uses this joined path as the key to retrieve the file from the bucket. For example: * Original URL: `https://example.com/static/dist/main.c383b093b0b66825a9c3.js` * `prefix` is set to `source-maps` * New path: `source-maps/main.c383b093b0b66825a9c3.js.map` ## Advanced Configuration In addition to basic setup, you can customize how the sourcemap processor handles stack traces by configuring attribute mappings and additional processing options. After updating the configuration file, restart the OpenTelemetry Collector to apply the changes. ### Mapping Attributes Use these configuration options to specify which attributes the processor should read from and write to when handling stack traces: | Config Key | Description | Example Value | | ------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | `symbolicator_failure_attribute_key` | Signals if the the symbolicator fails to fully symbolicate the stack trace | `exception.symbolicator.failed` | | `symbolicator_error_attribute_key` | Contains the error message if the the symbolicator fails to fully symbolicate the stack trace | `exception.symbolicator.error` | | `symbolicator_parsing_method_attribute_key` | Stores the stack trace parsing method used by the processor | `exception.symbolicator.parsing_method` | | `columns_attribute_key` | Which attribute should the columns of the stack trace be sourced from | `exception.structured_stacktrace.columns` | | `functions_attribute_key` | Which attribute should the functions of the stack trace be sourced from | `exception.structured_stacktrace.functions` | | `lines_attribute_key` | Which attribute should the lines of the stack trace be sourced from | `exception.structured_stacktrace.lines` | | `urls_attribute_key` | Which attribute should the urls of the stack trace be sourced from | `exception.structured_stacktrace.urls` | | `stack_trace_attribute_key` | Which attribute should the symbolicated stack trace be populated into | `exception.stacktrace` | | `exception_type_attribute_key` | Which attribute contains the exception type | `exception.type` | | `exception_message_attribute_key` | Which attribute contains the exception message | `exception.message` | | `preserve_stack_trace` | After the stack trace has been symbolicated should the original values be preserved as attributes | `true` | | `original_stack_trace_attribute_key` | If the stack trace is being preserved which key should it be copied to | `exception.stacktrace.original` | | `original_columns_attribute_key` | If the stack trace is being preserved which key should the functions be copied to | `exception.structured_stacktrace.functions.original` | | `original_functions_attribute_key` | If the stack trace is being preserved which key should the lines be copied to | `exception.structured_stacktrace.lines.original` | | `original_lines_attribute_key` | If the stack trace is being preserved which key should the columns be copied to | `exception.structured_stacktrace.columns.original` | | `original_urls_attribute_key` | If the stack trace is being preserved which key should the urls be copied to | | ### Additional Processing Options Use these configuration options to control how stack traces are processed and managed: | Config Key | Description | Example Value | | ----------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------- | | `timeout` | Maximum time (in seconds) to wait for symbolication before timing out. | `5` | | `source_map_cache_size` | Maximum number of source maps to cache in memory. Reduce this value if the Collector encounters memory constraints. | `128` | ### Language-Based Routing The source map processor supports language-based routing to ensure it only processes signals from JavaScript/TypeScript applications. This prevents the processor from running on signals from other languages (like Java or Swift), improving performance and avoiding unnecessary processing. | Config Key | Description | Default Value | Example Values | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------- | | `language_attribute_key` | The attribute key that contains the programming language or SDK language of the telemetry signal. | `telemetry.sdk.language` | `telemetry.sdk.language` | | `allowed_languages` | A list of language values that this processor will handle. If the signal's language attribute matches any value in this list, the processor will run. If empty (default), the processor will process all signals regardless of language. **Important:** When `allowed_languages` is configured, signals without a language attribute will be skipped. | `[]` (empty, processes all) | `["javascript", "webjs", "hermesjs"]` | If using Honeycomb's React Native SDK, you'll want to enable `hermesjs` as a value. See the [SDK's default attribute values](https://github.com/honeycombio/honeycomb-opentelemetry-react-native/tree/main?tab=readme-ov-file#default-attributes) **Example configuration:** ```yaml theme={} processors: source_map_symbolicator: allowed_languages: ["javascript", "webjs"] ``` `allowed_languages` configuration behavior: * Empty `allowed_languages` (default): Processes all signals, regardless of language attribute. * With `allowed_languages` configured: Only processes signals where the language attribute matches one of the allowed values (case-insensitive). * Missing language attribute: Skips processing when `allowed_languages` is configured. # Send Data to Honeycomb with Node.js Source: https://docs.honeycomb.io/send-data/javascript-nodejs Explore available methods for sending telemetry from your Node.js application to Honeycomb. When you are working with Node.js, we recommend these methods of sending data to Honeycomb. When you want to instrument Node.js applications in a standard, vendor-agnostic, and future-proof way, we recommend using the OpenTelemetry JavaScript SDK to send telemetry data to Honeycomb. The OpenTelemetry JavaScript SDK allows you to send traces and metrics. Logs is in development. When you need to create and send structured logs to Honeycomb, use Libhoney for Node.js, our structured logging library for Node.js applications. # Send Logs with Libhoney for JavaScript Source: https://docs.honeycomb.io/send-data/javascript-nodejs/libhoney Send structured events to Honeycomb from your JavaScript application using Libhoney, Honeycomb's low-level structured logging library for the Events API. Libhoney for JavaScript is Honeycomb's structured logging library for JavaScript applications. It is a low-level library that helps you send structured events to Honeycomb's [Events API](/api/events/). If you are instrumenting a new application for tracing, we recommend that you use [OpenTelemetry](/send-data/opentelemetry/) instead. For direct use in browser-side JavaScript applications, make sure to generate a separate API key that can only send events. Leaking an API key with those permissions would allow malicious users to access other data such as markers. See our [Browser JS guide](/send-data/javascript-browser/) for more information on how you can more safely send data about your web app from the client to Honeycomb. ## Installation ```shell npm theme={} npm install libhoney --save ``` ```shell yarn theme={} yarn add libhoney ``` ## Links * [Source Code](https://github.com/honeycombio/libhoney-js) * [Examples](https://github.com/honeycombio/libhoney-js/tree/main/examples) ## Initialization Initialize the library by passing in your Team API key and the default dataset name to which it should send events. Using ES6 modules: ```javascript theme={} import Libhoney from "libhoney"; let hny = new Libhoney({ writeKey: "YOUR_API_KEY", dataset: "honeycomb-js-example" // disabled: true // uncomment for testing or development. }); ``` Using `require`: ```javascript theme={} var Libhoney = require("libhoney"); var hny = new Libhoney({ writeKey: "YOUR_API_KEY", dataset: "honeycomb-js-example" }); ``` ### Using a Proxy To route event traffic through a proxy, configure it by passing an additional item, like: ```javascript theme={} import Libhoney from "libhoney"; let hny = new Libhoney({ proxy: "https://proxy-address-or-name:port", writeKey: "YOUR_API_KEY", dataset: "honeycomb-js-example" // disabled: true // uncomment for testing or development. }); ``` Find further configuration options in [the source code](https://github.com/honeycombio/libhoney-js/blob/b43b445a75188d2d2037fb32933262355155dd03/src/libhoney.js#L22). To silence `libhoney` in a test or development environment, include `{ disabled: true }` in the `new libhoney(...)` initialization. ## Building and Sending Events Once initialized, `libhoney` is ready to send events. Events go through three phases: * Creation `event = libhoney.newEvent()` * Adding fields `event.addField("key", "val")`, `event.add(dataMap)` * Transmission `event.send()` Upon calling `.send()`, the event is dispatched to be sent to Honeycomb. All libraries set defaults that will allow your application to function as smoothly as possible during error conditions. If you create events faster than they can be sent, overflowed events are dropped instead of backing up and slowing down your application. In its simplest form, you can add a single attribute to an event with the `.addField(k, v)` method. If you add the same key multiple times, only the last value added is kept. Other JavaScript objects can be added to an event with the `.add(data)` method. Events can have metadata associated with them that is not sent to Honeycomb. This metadata is used to identify the event when processing the response. More detail about metadata is below in the Response section. ## Handling Responses Sending an event is an asynchronous action and will avoid blocking by default—calling `.send()` will enqueue the event to be sent as soon as possible. Assign a `responseCallback` to check whether events were successfully received by Honeycomb's servers. Before sending an event, you have the option to attach metadata to that event. This metadata is not sent to Honeycomb; instead, it is used to help you match up individual responses with sent events. When sending an event, `libhoney` will take the metadata from the event and attach it to the response object for you to consume. Add metadata by populating the `.metadata` attribute directly on an event. For instance: ```javascript theme={} let hny = new Libhoney({ writeKey: "YOUR_API_KEY", dataset: "honeycomb-js-example", responseCallback: responses => { responses.forEach(resp => { console.log(resp); }); } }); let ev = hny.newEvent(); ev.addField("latencyMs", 240); ev.metadata = { id: "recognize-me-later" }; ev.send(); ``` This will print out the asynchronous responses from sending the event or batch of events. A status code of `200` or `202` indicates the event was received by our API successfully, whereas other status codes could indicate issues such as write key authentication failures or rate limiting. ```javascript theme={} { status_code: 202, duration: 320, metadata: {id: "recognize-me-later"}, error: undefined } ``` Responses have a number of fields describing the result of an attempted event send: * **metadata**: the metadata you attached to the event to which this response corresponds * **status\_code**: the HTTP status code returned by Honeycomb when trying to send the event. `2xx` indicates success. * **duration**: the number of milliseconds it took to send the event. * **error**: when the event does not even get to create a HTTP attempt, the reason will be in this field. (For example, when sampled or dropped because of a queue overflow.) You do not have to process responses if you are not interested in them—simply ignoring them is perfectly safe. Unread responses will be dropped. ## Examples Honeycomb can calculate all sorts of statistics, so send the data you care about and let us crunch the averages, percentiles, lower/upper bounds, cardinality—whatever you want—for you. ### Simple: Send a Blob Immediately ```javascript theme={} import Libhoney from "libhoney"; let hny = new Libhoney({ writeKey: "YOUR_API_KEY", dataset: "honeycomb-js-example" }); hny.sendNow({ message: "Test Honeycomb event", randomFloat: Math.random(), hostname: os.hostname(), favoriteColor: "chartreuse" }); ``` ### Intermediate: Override Some Attributes ```javascript theme={} // ... Initialization code ... let params = { hostname: "foo.local", built: false, userId: -1 }; hny.add(params); let builder = hny.newBuilder({ built: true }); // Spawn a new event and override the timestamp let event = builder.newEvent(); event.addField("userId", 15); event.addField("latencyMs", Date.now() - start); event.timestamp = new Date(Date.UTC(2016, 1, 29, 1, 1, 1)); event.send(); ``` Further examples can be found [on GitHub](https://github.com/honeycombio/libhoney-js). ## Middleware Examples: Express [Express](https://expressjs.com/) is light, flexible, and built to make it easy to drop in utility and middleware functionality to augment your application logic. Each inbound HTTP request as received by a framework like Express maps nicely to Honeycomb events, representing "a single thing of interest that happened" in a given system. Express middleware functions are simply functions that have access to the request object, response object, and next middleware function in the application's request-response chain. As such, you can define a simple `express-honey.js` file as in the following: ```javascript theme={} import Libhoney from "libhoney"; module.exports = function(options) { let honey = new Libhoney(options); return function(req, res, next) { honey.sendNow({ app: req.app, baseUrl: req.baseUrl, fresh: req.fresh, hostname: req.hostname, ip: req.ip, method: req.method, originalUrl: req.originalUrl, params: req.params, path: req.path, protocol: req.protocol, query: req.query, route: req.route, secure: req.secure, xhr: req.xhr }); next(); }; }; ``` And, in your Express `app.js`, configure your new `express-honey` and include it in the execution path like the following: ```javascript theme={} let express = require("express"); let hny = require("./express-honey"); let app = express(); app.use( hny({ apiHost: process.env["HONEY_API_HOST"], writeKey: process.env["YOUR_API_KEY"], dataset: process.env["HONEY_DATASET"] }) ); // ... additional Express handling code here ``` See the [`examples/` directory on GitHub](https://github.com/honeycombio/libhoney-js/tree/main/examples) for more sample code demonstrating how to use events, builders, fields, and dynamic fields, specifically in the context of Express middleware. ## Advanced Usage: Using Builders Builders are, at their simplest, a convenient way to avoid repeating common attributes that may not apply globally. Creating a builder for a given component allows a variety of different events to be spawned and sent within the component, without having to repeat the component name as an attribute for each. You can clone builders—the cloned builder will have a copy of all the fields and dynamic fields in the original. As your application forks down into more and more specific functionality, you can create more detailed builders. The final event creation in the leaves of your application's tree will have all the data you have added along the way in addition to the specifics of this event. The global scope is essentially a specialized builder, for capturing attributes that are likely useful to all events (for example, hostname or environment). Adding this kind of peripheral and normally unavailable information to every event gives you enormous power to identify patterns that would otherwise be invisible in the context of a single request. ## Advanced Usage: Dynamic Fields The top-level `libhoney` and Builders support `.addDynamicField(func)`. Adding a dynamic field to a Builder or top-level `libhoney` ensures that each time an event is created, the provided function is executed and the returned key/value pair is added to the event. This may be useful for including dynamic process information such as memory used, number of threads, concurrent requests, and so on to each event. Adding this kind of dynamic data to an event makes it easy to understand the application's context when looking at an individual event or error condition. ## Troubleshooting Refer to [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#libhoney) for Libhoney. ## Contributions Features, bug fixes and other changes to `libhoney` are gladly accepted. Please open issues or a pull request with your change. All contributions will be released under the Apache License 2.0. # Send Data with the OpenTelemetry JavaScript SDK Source: https://docs.honeycomb.io/send-data/javascript-nodejs/opentelemetry-sdk Instrument your Node.js application with the OpenTelemetry JavaScript SDK and send traces, logs, and metrics to Honeycomb. Use the OpenTelemetry JavaScript SDK to instrument Node.js applications in a standard, vendor-agnostic, and future-proof way and send telemetry data to Honeycomb. In this guide, we will walk you through instrumenting with OpenTelemetry for JavaScript, which will include adding automatic instrumentation to your application. For more structured learning, check out the [Instrumentation for OpenTelemetry JavaScript](https://academy.honeycomb.io/app/courses/2ee2544e-866c-4251-90b5-bd17f6e59494) course from Honeycomb Academy. ## Before You Begin Before you can set up automatic instrumentation for your Node.js application, you will need to do a few things. ### Prepare Your Development Environment To complete the required steps, you will need: * A working Node.js environment * An application written in Node.js ### Get Your Honeycomb API Key To send data to Honeycomb, you'll need to [sign up for a free Honeycomb account](https://ui.honeycomb.io/signup) and [create a Honeycomb Ingest API Key](/configure/environments/manage-api-keys/#create-api-key). To get started, you can create a key that you expect to swap out when you deploy to production. Name it something helpful, perhaps noting that it's a getting started key. Make note of your API key; for security reasons, you will not be able to see the key again, and you will need it later! For setup, make sure you check the "Can create datasets" checkbox so that your data will show up in Honeycomb. Later, when you replace this key with a permanent one, you can uncheck that box. If you want to use an API key you previously stored in a secure location, you can also [look up details for Honeycomb API Keys](/configure/environments/manage-api-keys/#find-api-keys) any time in your Environment Settings, and use them to retrieve keys from your storage location. ## Add Automatic Instrumentation Automatic instrumentation is enabled by adding [instrumentation packages](https://github.com/open-telemetry/opentelemetry-js#instrumentations). Add custom, or manual, instrumentation using the OpenTelemetry API. ### Acquire Dependencies Open your terminal, navigate to the location of your project on your drive, and install OpenTelemetry's automatic instrumentation meta package and OpenTelemetry's Node.js SDK package: ```shell theme={} npm install --save \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-node ``` | Module | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `auto-instrumentations-node` | OpenTelemetry's meta package that provides a way to add automatic instrumentation to any Node application to capture telemetry data from a number of popular libraries and frameworks, like `express`, `dns`, `http`, and more. | | `sdk-node` | OpenTelemetry's Node.js distribution package that streamlines configuration and allows you to instrument as quickly and easily as possible. | Alternatively, install [individual instrumentation packages](https://github.com/open-telemetry/opentelemetry-js#instrumentations). If using TypeScript, install `ts-node` to run the code: ```shell theme={} npm install --save-dev ts-node ``` Open your terminal, navigate to the location of your project on your drive, and install OpenTelemetry's automatic instrumentation meta package and OpenTelemetry's Node.js SDK package: ```shell theme={} yarn add \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-node ``` | Module | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `auto-instrumentations-node` | OpenTelemetry's meta package that provides a way to add automatic instrumentation to any Node application to capture telemetry data from a number of popular libraries and frameworks, like `express`, `dns`, `http`, and more. | | `sdk-node` | OpenTelemetry's Node.js distribution package that streamlines configuration and allows you to instrument as quickly and easily as possible. | Alternatively, install [individual instrumentation packages](https://github.com/open-telemetry/opentelemetry-js#instrumentations). If using TypeScript, install `ts-node` to run the code: ```shell theme={} yarn add --dev ts-node ``` ### Initialize Create an initialization file, commonly known as the `tracing.ts` file: ```typescript theme={} // Example filename: tracing.ts import { startNodeSDK } from '@opentelemetry/sdk-node'; import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'; startNodeSDK({ instrumentations: [ getNodeAutoInstrumentations(), ], }); ``` Create an initialization file, commonly known as the `tracing.js` file: ```javascript theme={} // Example filename: tracing.js 'use strict'; const { startNodeSDK } = require('@opentelemetry/sdk-node'); const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); startNodeSDK({ instrumentations: [ getNodeAutoInstrumentations(), ], }); ``` ### Configure Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces # Use the endpoint below for EU # endpoint: https://api.eu1.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Set the following environment variables before running your application: | Environment Variable | Value | | :------------------- | :----------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. The OpenTelemetry declarative configuration is stable at the specification level. Individual fields still under active development are marked with a `/development` suffix in the YAML (see [configuration versioning](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md#experimental-features)). Check the [language support status](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/language-support-status.md) for per-SDK maturity. Add `meter_provider` and `logger_provider` sections to the same file to export metrics and logs. This version also enables resource detectors, which add attributes such as `host.*` and `process.*` automatically: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} detection/development: detectors: - host: - os: - process: - service: - env: tracer_provider: # traces processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # metrics readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/metrics headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} # Legacy metrics only; omit with the current metrics experience: # - name: x-honeycomb-dataset # value: ${HONEYCOMB_METRICS_DATASET} logger_provider: # logs processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` For the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` throughout the file. If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must also specify the Dataset for traces using the `x-honeycomb-dataset` header: ```yaml theme={} headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} - name: x-honeycomb-dataset value: your-dataset ``` If you are sending data directly to Honeycomb, you must configure the API key and service name. If you are using an [OpenTelemetry Collector](/send-data/opentelemetry/collector/), configure your API key at the Collector level instead. ### Run Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run the Node.js app with the initialization file: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml ts-node -r ./tracing.ts YOUR_APPLICATION_NAME.ts ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. Alternatively, you can import the initialization file as the first step in your application lifecycle. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run the Node.js app with the initialization file: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml node -r ./tracing.js YOUR_APPLICATION_NAME.js ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. Alternatively, you can import the initialization file as the first step in your application lifecycle. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. ## Add Custom Instrumentation Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add custom, or manual, instrumentation where appropriate. Follow the instructions below to add custom instrumentation to your code. To learn more about custom, or manual, instrumentation, visit the comprehensive set of topics covered by [Manual Instrumentation for JavaScript](https://opentelemetry.io/docs/languages/js/instrumentation/) in OpenTelemetry's documentation. ### Acquire Dependencies To start adding custom instrumentation, ensure that the OpenTelemetry API package exists as a direct dependency in your project. This package provides access to the high-level instrumentation APIs, which gives the ability to retrieve the current span to enrich with additional attributes, to create new spans, and to generate metrics. ```shell npm theme={} npm install --save @opentelemetry/api ``` ```shell yarn theme={} yarn add @opentelemetry/api ``` ### Add Attributes to Spans Adding attributes to a currently executing span in a trace can be useful. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your service in Honeycomb. To do this, get the current span from the context and set an attribute with the user ID: ```typescript TypeScript theme={} import { trace } from '@opentelemetry/api'; function handleUser(user) { let activeSpan = trace.getActiveSpan(); activeSpan.setAttribute("user.id", user.getId()); } ``` ```javascript JavaScript theme={} const { trace } = require("@opentelemetry/api"); function handleUser(user) { let activeSpan = trace.getActiveSpan(); activeSpan.setAttribute("user.id", user.getId()); } ``` This will add a `user.id` field to the current span, so you can use the field in `WHERE`, `GROUP BY` or `ORDER` clauses in the Honeycomb query builder. ### Initialize a Tracer To create spans, you need to initialize a `Tracer`. ```typescript TypeScript theme={} import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer("tracer.name.here"); ``` ```js JavaScript theme={} const { trace } = require("@opentelemetry/api"); const tracer = trace.getTracer("tracer.name.here"); ``` When you create a `Tracer`, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter. When traces are sent to Honeycomb, the name of the `Tracer` is turned into the `library.name` field, which can be used to show all spans created from a particular tracer. In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different "layers" of your application, then use the name that corresponds to that "layer". The `library.name` field is also used with traces created from instrumentation libraries. You can then use this tracer to create custom spans. ### Create New Spans Automatic instrumentation can show the shape of requests to your system, but only you know the truly important parts. To get the full picture of what is happening, you must add custom, or manual, instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API: ```typescript TypeScript theme={} import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer("my-service-tracer"); function runQuery() { tracer.startActiveSpan("expensive-query", (span) => { // ... do cool stuff span.end(); }); } ``` ```javascript JavaScript theme={} const { trace } = require("@opentelemetry/api"); const tracer = trace.getTracer("my-service-tracer"); function runQuery() { tracer.startActiveSpan("expensive-query", (span) => { // ... do cool stuff span.end(); }); } ``` ### Add Multi-Span Attributes Sometimes you want to add the same attribute to many spans within the same trace. This attribute may include variables calculated during your program, or other useful values for correlation or debugging purposes. To add this attribute to multiple spans, leverage the OpenTelemetry concept of [baggage](https://opentelemetry.io/docs/concepts/signals/baggage/). Baggage allows you to add a `key` with a `value` as an attribute to every subsequent child span within the current application context. 1. Install the baggage span processor package using your terminal: ```shell theme={} npm install --save @opentelemetry/baggage-span-processor ``` 2. When configuring the OpenTelemetry SDK tracer provider, add the `BaggageSpanProcessor`: ```javascript theme={} import { BaggageSpanProcessor } from "@opentelemetry/baggage-span-processor"; const sdk = new NodeSDK({ // ... processors: [ new BaggageSpanProcessor() ] }); ``` 3. Add a baggage entry for the current trace and replace `key` and `value` with your desired key-value pair: ```typescript TypeScript theme={} import { Context, context, propagation, } from '@opentelemetry/api'; tracer.startActiveSpan('main', (span) => { span.setAttribute('key', 'value'); // add to current span // new context based on current, with key/values added to baggage const ctx: Context = propagation.setBaggage( context.active(), propagation.createBaggage({ 'key': { value: 'value' } }) ); // within the new context, do some work and baggage will be // applied as attributes on child spans context.with(ctx, () => { tracer.startActiveSpan('childSpan', (childSpan) => { doTheWork(); childSpan.end(); }); }); span.end(); }); ``` ```javascript JavaScript theme={} tracer.startActiveSpan('main', (span) => { span.setAttribute('key', 'value'); // add to current span // new context based on current, with key/values added to baggage const ctx = propagation.setBaggage( context.active(), propagation.createBaggage({ 'key': { value: 'value' } }) ); // within the new context, do some work and baggage will be // applied as attributes on child spans context.with(ctx, () => { tracer.startActiveSpan('childSpan', (childSpan) => { doTheWork(); childSpan.end(); }); }); span.end(); }); ``` Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do **NOT** put sensitive information in the Baggage attributes. ## Sampling You can configure the OpenTelemetry SDK to [sample the data](/manage-data-volume/sample/guidelines/) it generates. Honeycomb [weights sampled data based on sample rate](/manage-data-volume/sample/sampled-data-in-honeycomb/), so you must set a resource attribute containing the sample rate. Use a [`TraceIdRatioBased` sampler](https://opentelemetry.io//docs/specs/otel/trace/sdk/#traceidratiobased), with a ratio expressed as `1/N`. Then, also create a resource attribute called `SampleRate` with the value of `N`. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data. In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5). To get access to a sampler, install the core OpenTelemetry package: ```shell theme={} npm install --save @opentelemetry/core ``` Import the `TraceIdRatioBasedSampler`, and add as a sampler to the `NodeSDK` along with the `SampleRate` in the `Resource`. ```javascript theme={} // tracing.js const { TraceIdRatioBasedSampler } = require("@opentelemetry/sdk-trace-node"); const sdk = new NodeSDK({ resource: new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: "", SampleRate: 2, }), traceExporter, instrumentations: [getNodeAutoInstrumentations()], sampler: new TraceIdRatioBasedSampler(0.5), }); ``` ## Choosing between gRPC and HTTP Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind: * Some SDKs default to using gRPC, and it may be easiest to start with the default option. * Some firewall policies are not set up to handle gRPC and require using HTTP. * gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery. gRPC default export uses port 4317, whereas HTTP default export uses port 4318. ## Endpoint URLs for OTLP/HTTP When using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable with an SDK and an HTTP exporter, the final path of the endpoint is modified by the SDK to represent the specific signal being sent. For example, when exporting trace data, the endpoint is updated to append `v1/traces`. When exporting metrics data, the endpoint is updated to append `v1/metrics`. So, if you were to set the `OTEL_EXPORTER_OTLP_ENDPOINT` to `https://api.honeycomb.io`, traces would be sent to `https://api.honeycomb.io/v1/traces` and metrics would be sent to `https://api.honeycomb.io/v1/metrics`. The same modification is not necessary for gRPC. ```shell theme={} export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.eu1.honeycomb.io # EU instance ``` If the desired outcome is to send data to a different endpoint depending on the signal, use `OTEL_EXPORTER_OTLP__ENDPOINT` instead of the more generic `OTEL_EXPORTER_OTLP_ENDPOINT`. When using a signal-specific environment variable, these paths must be appended manually. Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` for traces, appending the endpoint with `v1/traces`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` for metrics, appending the endpoint with `v1/metrics`. Send both traces and metrics to Honeycomb using this method by setting the following variables: ```shell theme={} export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces # US instance #export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.eu1.honeycomb.io/v1/traces # EU instance export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics # US instance #export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.eu1.honeycomb.io/v1/metrics # EU instance ``` More details about endpoints and signals can be found in the [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md). To configure the endpoint URL and API key in code instead of with environment variables, specify `/v1/traces` like so: ```javascript theme={} // Configure OTLPTraceExporter const traceExporter = new OTLPTraceExporter({ url: "https://api.honeycomb.io/v1/traces", // US instance //url: "https://api.eu1.honeycomb.io/v1/traces", // EU instance headers: { "x-honeycomb-team": 'your-api-key', }, }); ``` ## Troubleshooting To explore common issues when sending data, visit [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#opentelemetry-sdks-and-honeycomb-distributions). # Send Data from Kubernetes Source: https://docs.honeycomb.io/send-data/kubernetes Instrument your Kubernetes cluster with OpenTelemetry and send traces, logs, and metrics to Honeycomb to monitor workload performance and health. Use OpenTelemetry to collect Kubernetes resource and status metrics from nodes, pods, containers, and volumes. This data answers questions like: * Which pods are using the most CPU? * How do resource limits compare to container resource use? * What do system metrics look like at the node level? * Why are pods failing to start? Adding telemetry to Kubernetes and then analyzing with Honeycomb provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes. This data answers questions like: * How did response time change after a canary deployment? * How does application performance vary with container resource limits? * Are application errors happening on specific nodes, or across the fleet? ## Getting Started: Create Your Telemetry Pipeline Do you have 10 minutes? Then you've come to the right place. Use our Quick Start to create a telemetry pipeline, which will prepare you to instrument your application code. Use Helm to deploy OpenTelemetry Collectors that set up a telemetry pipeline to send Kubernetes metrics and events from your cluster to Honeycomb. ## Add Low-Code Automatic Instrumentation to Your Applications Once you have a telemetry pipeline in place, add automatic instrumentation to your applications. If you have already used OpenTelemetry to instrument your applications to send data to Honeycomb, you can skip this step and jump straight to [configuring OpenTelemetry to forward telemetry data to your Collectors](/send-data/kubernetes/opentelemetry/collect-instrumented-code/). Set up the OpenTelemetry Operator for Kubernetes to add automatic instrumentation to your applications--using very little code. ## Getting Help To ask questions and learn more, join our [Pollinators Community Slack](/troubleshoot/community/#join-pollinators-community-slack). # Add Automatic Instrumentation (Low Code) Source: https://docs.honeycomb.io/send-data/kubernetes/opentelemetry/add-automatic-instrumentation Install the OpenTelemetry Operator for Kubernetes to automatically inject instrumentation into your deployed applications with minimal code changes. Once you have a telemetry pipeline in place, you can gather metrics, logs, and traces from your deployed applications by adding automatic instrumentation to your applications. You can do this with very little code by installing the [OpenTelemetry Operator for Kubernetes](https://opentelemetry.io/docs/kubernetes/operator/) in your cluster. Once installed, the Operator will monitor when pods are created and check whether it should sideload automatic instrumentation packages. We consider this approach to be "low-code" because it requires you to access and modify the Kubernetes deployment manifests for your application. This approach also pushes automatic instrumentation into your code base as the application starts, so that trace data can be gathered. Although this is a very minimal amount of change overall and does not require you to directly modify any application code, we differentiate this from "no-code" because it does require that you modify how your applications are deployed on Kubernetes. Supported languages include: * .NET * Java * Node.js * Python * Go ## Before You Begin Before beginning this guide, you should have: * Created a running Kubernetes cluster. * Created a namespace named `honeycomb`. * [Deployed an OpenTelemetry Collector in DaemonSet mode, listening on the Node IP](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/). ## Step 1: Install the Operator Install the OpenTelemetry Operator Pods into your cluster using a Helm chart: ```shell theme={} helm install \ --set admissionWebhooks.certManager.enabled=false \ --set admissionWebhooks.autoGenerateCert.enabled=true \ --set manager.collectorImage.repository="ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s" \ --namespace honeycomb \ --create-namespace \ opentelemetry-operator open-telemetry/opentelemetry-operator ``` In this example, we want the Operator to add automatic instrumentation to a Go application, so we must include the `manager.featureGates` configuration line to the command. If you are working with a different programming language, you can omit this line. In this example, we use an automatically generated self-signed certificate by setting `admissionWebhooks.certManager.enabled` to `false` and `admissionWebhooks.autoGenerateCert.enabled` to `true`. We do not recommend that you use this configuration in a production environment, but setting up a [cert-manager](https://cert-manager.io/docs/) is outside the scope of this document. ## Step 2: Verify the Operator Installation Check that the Operator is installed by using the `kubectl` command to see if the pod is running: ```shell theme={} kubectl get pods --namespace honeycomb ``` This command should return something like: ```pre theme={} NAME READY STATUS RESTARTS AGE opentelemetry-operator-567bc4ff75-p287g 2/2 Running 0 21s ``` The result should contain one pod, with two containers, running under a name containing the prefix `opentelemetry-operator`. After verifying that the pod is running correctly, you can monitor the work the operator does by tailing the logs with the `kubectl` command: ```shell theme={} kubectl logs -n honeycomb -f opentelemetry-operator-567bc4ff75-p287g ``` This command should return something like: ```pre theme={} {"level":"info","ts":"2023-09-22T15:21:27Z","msg":"Starting Controller","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector"} {"level":"info","ts":"2023-09-22T15:21:27Z","logger":"instrumentation-resource","msg":"default","name":"otel-autoinstrumentation"} {"level":"info","ts":"2023-09-22T15:21:27Z","logger":"collector-upgrade","msg":"no instances to upgrade"} {"level":"info","ts":"2023-09-22T15:21:27Z","logger":"instrumentation-resource","msg":"validate update","name":"otel-autoinstrumentation"} {"level":"info","ts":"2023-09-22T15:21:27Z","msg":"Starting workers","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","worker count":1} {"level":"info","ts":"2023-09-22T15:34:24Z","logger":"instrumentation-resource","msg":"default","name":"otel-autoinstrumentation"} {"level":"info","ts":"2023-09-22T15:34:24Z","logger":"instrumentation-resource","msg":"validate update","name":"otel-autoinstrumentation"} ``` ## Step 3: Configure Automatic Instrumentation Deploy an Instrumentation manifest to your Kubernetes cluster, which will enable the OpenTelemetry Operator to automatically instrument your services: ```shell theme={} kubectl apply --namespace honeycomb -f https://docs.honeycomb.io/_assets/code-samples/kubernetes/values-files/otel-autoinstrumentation.yaml ``` [Download](/_assets/code-samples/kubernetes/values-files/otel-autoinstrumentation.yaml) ```yaml theme={} apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: otel-autoinstrumentation spec: exporter: endpoint: http://$(OTEL_NODE_IP):4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "1" python: env: # Required if endpoint is set to 4317. # Python autoinstrumentation uses http/proto by default # so data must be sent to 4318 instead of 4317. - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(OTEL_NODE_IP):4318 dotnet: env: # Required if endpoint is set to 4317. # Dotnet autoinstrumentation uses http/proto by default # See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/888e2cd216c77d12e56b54ee91dafbc4e7452a52/docs/config.md\#otlp - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(OTEL_NODE_IP):4318 java: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.10.0 env: # Required if endpoint is set to 4317. # Java autoinstrumentation agent v2.0+ uses http/proto by default # See https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/9bbfe7fe4e3f65cb698d6d2320ac87372d5d572f/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/OtlpProtocolPropertiesSupplier.java#L19 - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(OTEL_NODE_IP):4318 nodejs: env: # Optional. # We recommend disabling fs automatic instrumentation because # it can be noisy and expensive during startup - name: OTEL_NODE_DISABLED_INSTRUMENTATIONS value: fs ``` This manifest instructs the Operator to perform the following actions whenever you add annotations to your pods: * Inject instrumentation into your applications * Configure the export to send data to the node's IP address ## Step 4: Add Annotations to Your Application's Kubernetes Manifest In the deployment manifest for your application, add the required annotations. These will vary according to programming language. For example, our Go application requires the following annotations. Note where the annotations are located in the manifest: ```yaml theme={} apiVersion: apps/v1 kind: Deployment metadata: name: my-go-app spec: replicas: 1 template: metadata: labels: app: my-go-app annotations: # Added Annotations for auto-instrumentation go here - points at the namespace where the instrumentation # manifest is installed and the name of meta.name field in the instrumentation manifest instrumentation.opentelemetry.io/inject-go: "honeycomb/otel-autoinstrumentation" # Go Requires an additional annotation to the path of the binary on the container instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/my-go-app" spec: serviceAccountName: my-go-app ``` Each supported programming language requires different annotations: ```yaml theme={} annotations: instrumentation.opentelemetry.io/inject-dotnet: "honeycomb/otel-autoinstrumentation" ``` ```yaml theme={} annotations: instrumentation.opentelemetry.io/inject-java: "honeycomb/otel-autoinstrumentation" ``` ```yaml theme={} annotations: instrumentation.opentelemetry.io/inject-nodejs: "honeycomb/otel-autoinstrumentation" ``` ```yaml theme={} annotations: instrumentation.opentelemetry.io/inject-python: "honeycomb/otel-autoinstrumentation" ``` Go requires an extra annotation that tells the Operator where to find the Go binary of your application on the container. ```yaml theme={} annotations: instrumentation.opentelemetry.io/inject-go: "honeycomb/otel-autoinstrumentation" instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/app/frontend-service" ``` Possible values for the annotation include: * `"true"` - inject an `Instrumentation` Kubernetes Custom Resource (CR) instance from the current namespace * `"my-instrumentation"` - inject a specific `Instrumentation` CR instance from the current namespace * `"my-other-namespace/my-instrumentation"` - inject a specific `Instrumentation` CR instance from another namespace * `"false"` - do not inject an `Instrumentation` CR instance ## Explore Your Data In Honeycomb The next time requests are made to your application, data should start flowing to Honeycomb. Once your applications are sending data, you can [explore trace data](/investigate/) and gain insights from the Kubernetes attributes that have been attached by the [Kubernetes Attributes Processor](/send-data/kubernetes/opentelemetry/components/#kubernetes-attributes-processor) in your Collector. If several requests have been made to your application and after several minutes, you still do not see any data, reach out for help in our [Pollinators Community Slack](/troubleshoot/community/#join-pollinators-community-slack). Pro/Enterprise users can [visit Honeycomb Support](https://support.honeycomb.io/) or [email support@honeycomb.io](mailto:support@honeycomb.io). ## Additional Resources * [Operator documentation on opentelemetry.io](https://opentelemetry.io/docs/kubernetes/operator/) * [OpenTelemetry Operator README in GitHub](https://github.com/open-telemetry/opentelemetry-operator/blob/main/README.md) # Send Data from Instrumented Code to Collectors Source: https://docs.honeycomb.io/send-data/kubernetes/opentelemetry/collect-instrumented-code Configure your OpenTelemetry-instrumented applications to export telemetry to the Collectors running in your Kubernetes cluster. If you're already using OpenTelemetry, or you've decided to instrument your code with OpenTelemetry's SDKs, you need to tell the SDKs where to send the telemetry data. ## Before You Begin Before beginning this guide, you should have: * Created a running Kubernetes cluster. * Deployed some applications to your cluster and instrumented them with OpenTelemetry. * [Deployed an OpenTelemetry Collector in DaemonSet mode, listening on the Node IP](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/#step-4-deploy-collectors). ## Forward Data from Your Application Code to the Collectors Now that you have a telemetry pipeline in your cluster, you must configure the SDKs to forward the telemetry data to your Collectors. To do this, you will use the [Kubernetes Downward API](https://kubernetes.io/docs/concepts/workloads/pods/downward-api/), which allows you to pass information about the wider context of your Kubernetes environment into your deployments and therefore your pods. For example, you can use the Downward API to add environment variables that can store pieces of metadata, like the Cluster name. In our Kubernetes Quick Start, you deployed a DaemonSet-mode Collector listening on the IP address of the node, which is where your SDK must send telemetry data. 1. Using the Downward API, create an environment variable and pass in the node's IP address: ```yaml theme={} env: - name: NODE_IP valueFrom: fieldRef: fieldPath: status.hostIP ``` 2. Configure the node's environment variable name and value. In this example, we use .NET's standard environment variable, which is named `OTEL_EXPORTER_OTLP_ENDPOINT`, and set it to the value of the node's IP address. ```yaml theme={} env: - name: NODE_IP valueFrom: fieldRef: fieldPath: status.hostIP - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://$(NODE_IP):4317 ``` For a list of OTLP exporter configuration options, visit [OpenTelemetry's Protocol Exporter](https://opentelemetry.io/docs/specs/otel/protocol/exporter/). After a few minutes, data should start flowing through your observability pipeline and into Honeycomb. # OpenTelemetry's Kubernetes Components Source: https://docs.honeycomb.io/send-data/kubernetes/opentelemetry/components Reference the OpenTelemetry Collector components used to gather data from Kubernetes, including receivers, processors, and exporters for cluster telemetry. With Honeycomb and OpenTelemetry, you can analyze and query data across not only your applications, but also your entire Kubernetes cluster. The following components available in the OpenTelemetry Collector will allow you to gather the best data from your Kubernetes cluster and applications to be analyzed in Honeycomb. ## Kubernetes Attributes Processor OpenTelemetry's [Kubernetes Attributes Processor](https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-attributes-processor) automatically discovers Kubernetes pods, extracts their metadata (for example, pod name or node name), and adds the extracted metadata to spans, metrics, and logs as resource attributes. Because the Kubernetes Attributes Processor adds Kubernetes context to your telemetry, you can correlate your application's traces, metrics, and logs with your Kubernetes telemetry, such as pod metrics and events. We highly recommend that you include this processor in any Collector that receives telemetry from Kubernetes pods. ## Kubeletstats Receiver OpenTelemetry's [Kubeletstats Receiver](https://opentelemetry.io/docs/kubernetes/collector/components/#kubeletstats-receiver) gathers metrics about the Kubernetes node and its workloads. This receiver can gather metrics like container memory usage, pod cpu usage, and node network errors. All of the gathered telemetry includes Kubernetes metadata, such as pod name or node name. When you use the Kubeletstats Receiver with the Kubernetes Attributes Processor to enhance your application telemetry, you can correlate your application traces, metrics, and logs with the metrics produced by the Kubeletstats Receiver. ## Kubernetes Cluster Receiver OpenTelemetry Collectors use the [Kubernetes Cluster Receiver](https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-cluster-receiver) to collect metrics about the state of the cluster as a whole. This receiver can gather metrics about node conditions, pod phases, container restarts, available and desired deployments, and more. Since the receiver gathers telemetry for the cluster as a whole, only one instance of the receiver is needed across the cluster in order to collect all the data. ## Kubernetes Objects Receiver OpenTelemetry's [Kubernetes Objects Receiver](https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-objects-receiver) collects objects from the Kubernetes API server. Because the receiver gathers telemetry for the cluster as a whole, only one instance of the receiver is needed across the cluster in order to collect all the data. Most commonly, this receiver is used to watch Kubernetes Events. To learn more about using the Kubernetes Objects Receiver to send Kubernetes Events to Honeycomb, visit [Kubernetes Events with the OpenTelemetry Collector](/send-data/kubernetes/send-events-objects-receiver/). ## OpenTelemetry Filelog Receiver OpenTelemetry's [Filelog Receiver](https://opentelemetry.io/docs/kubernetes/collector/components/#filelog-receiver) tails and parses logs from files. Although the Filelog Receiver is not Kubernetes specific, we recommend it as the primary solution for collecting any logs from Kubernetes. The Filelog Receiver processes logs through a series of operators that are chained together, each of which performs a simple responsibility, such as parsing JSON. Configuring a Filelog Receiver can be complicated. The easiest way to get started is to use the OpenTelemetry Collector Helm chart with the `logsCollection` preset. ## OpenTelemetry Marker Exporter The OpenTelemetry Collector's [Marker Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/honeycombmarkerexporter) allows you to send a [Honeycomb Marker](/configure/environments/manage-markers/) based on the shape of incoming telemetry. In your OpenTelemetry Collector exporter configuration, you can specify a set of rules that will be evaluated against incoming telemetry. When a condition is met, a marker is sent and appears in Honeycomb. For example, whenever a Kubernetes Event contains a `reason` of `Backoff`, the configuration below sends a marker: ```yaml theme={} exporters: honeycombmarker: api_key: YOUR-API-KEY-HERE markers: # Creates a new marker each time the exporter sees a Kubernetes event with a reason of Backoff - type: k8s-backoff-events rules: - log_conditions: - IsMap(body) and IsMap(body["object"]) and body["object"]["reason"] == "Backoff" ``` # Create a Telemetry Pipeline Source: https://docs.honeycomb.io/send-data/kubernetes/opentelemetry/create-telemetry-pipeline Set up an OpenTelemetry Collector pipeline for your Kubernetes cluster and start sending traces, logs, and metrics to Honeycomb in under 10 minutes. Whether you want to maximize Honeycomb's capabilities or begin with a more limited set of telemetry data for your Kubernetes applications, we recommend that you use [OpenTelemetry](https://opentelemetry.io/), a highly configurable, open-source, and vendor-neutral instrumentation framework. In this guide, you will learn how to get answers about how your applications on Kubernetes are performing in production using OpenTelemetry Collectors and Honeycomb--and you'll do it in under 10 minutes. When you finish, you'll have visibility into in-depth Kubernetes data, including Kubernetes logs, events, and node/cluster metrics. And you'll be given the opportunity to take the next step toward leveraging Honeycomb's full potential by instrumenting your code. ## Overview In the next 10 minutes, you will create a series of OpenTelemetry Collectors that will work together to pull in the correct telemetry and apply Kubernetes-specific data to it, which will help correlate issues. Your implementation will also lay the foundation for your applications to send telemetry data, if you choose to instrument them. Collectors Overview

1. Each node will contain a Collector, which will use the node's Kubelet API to gather metrics data about the node and the node's pod resources. 2. The entire cluster will contain a separate Collector, which will use the Kubernetes API to get details about Kubernetes Events, such as active deployments. 3. Applications will be able to use the node's IP address to send telemetry data (logs, metrics, and traces) to the Collector that is local to the node, if you instrument them. 4. Each Collector will send telemetry data directly to Honeycomb over gRPC. ## Collected Data When you finish, you will have access to: * Default [metrics provided by Kubelet Stats](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/documentation.md) for nodes and pods, plus the optional `uptime`, `*_request_utilization`, and `*_limit_utilization` metrics * Default [Kubernetes Cluster metrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sclusterreceiver/documentation.md) * All Kubernetes events from the cluster * Additional Kubernetes metadata, plus all the pod's labels and annotations as resource attributes whenever possible, including: * `k8s.namespace.name` * `k8s.deployment.name` * `k8s.statefulset.name` * `k8s.daemonset.name` * `k8s.cronjob.name` * `k8s.job.name` * `k8s.node.name` * `k8s.pod.name` * `k8s.pod.uid` * `k8s.pod.start_time` ## Before You Begin Before beginning this guide, you should have: * Created a running Kubernetes cluster. * Installed the `kubectl` command-line utility locally. * Installed [Helm 3.9+](https://helm.sh/) locally. * Deployed some applications to Kubernetes. You'll also need your Honeycomb API Key. You can [find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys) in your Environment Settings. Let's get started! ## Step 1: Create a Namespace To help you manage your objects in the cluster, create a namespace to contain the collector infrastructure. In this example, we call the namespace `honeycomb`. ```shell theme={} kubectl create namespace honeycomb ``` ## Step 2: Configure Kubernetes with Your Honeycomb API Key Within your new namespace, create a Kubernetes Secret that contains your Honeycomb API Key. You can [find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys) in your environment in Honeycomb. ```shell theme={} export HONEYCOMB_API_KEY=mykey kubectl create secret generic honeycomb --from-literal=api-key=$HONEYCOMB_API_KEY --namespace=honeycomb ``` ## Step 3: Add OpenTelemetry's Helm Repository [OpenTelemetry's Helm GitHub repository](https://github.com/open-telemetry/opentelemetry-helm-charts) includes Helm charts with all of the resources you need to deploy Collectors to your Kubernetes cluster. 1. Add the repo: ```shell theme={} helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts ``` 2. Update your repos to ensure Helm is aware of the latest versions: ```shell theme={} helm repo update ``` ## Step 4: Deploy Collectors Deploy your Collectors: * A Deployment-mode Collector to collect your cluster metrics. * A DaemonSet-mode Collector to collect application telemetry data and metrics from your cluster's node(s). You can deploy both Collectors using the same Helm chart, but with different names and values files. Deploy the Deployment-mode Collector: ```shell theme={} helm install otel-collector-cluster open-telemetry/opentelemetry-collector --namespace honeycomb --values https://docs.honeycomb.io/_assets/code-samples/kubernetes/values-files/values-deployment.yaml ``` [Download](/_assets/code-samples/kubernetes/values-files/values-deployment.yaml) ```yaml theme={} mode: deployment image: repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s extraEnvs: - name: HONEYCOMB_API_KEY valueFrom: secretKeyRef: name: honeycomb key: api-key # We only want one of these collectors - any more and we'd produce duplicate data replicaCount: 1 presets: # enables the k8sclusterreceiver and adds it to the metrics pipelines clusterMetrics: enabled: true # enables the k8sobjectsreceiver to collect events only and adds it to the logs pipelines kubernetesEvents: enabled: true config: receivers: k8s_cluster: collection_interval: 30s metrics: # Disable replicaset metrics by default. These are typically high volume, low signal metrics. # If volume is not a concern, then the following blocks can be removed. k8s.replicaset.desired: enabled: false k8s.replicaset.available: enabled: false jaeger: null zipkin: null processors: transform/events: error_mode: ignore log_statements: - context: log statements: # adds a new watch-type attribute from the body if it exists - set(attributes["watch-type"], body["type"]) where IsMap(body) and body["type"] != nil # create new attributes from the body if the body is an object - merge_maps(attributes, body, "upsert") where IsMap(body) and body["object"] == nil - merge_maps(attributes, body["object"], "upsert") where IsMap(body) and body["object"] != nil # Transform the attributes so that the log events use the k8s.* semantic conventions - merge_maps(attributes, attributes[ "metadata"], "upsert") where IsMap(attributes[ "metadata"]) - set(attributes["k8s.pod.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Pod" - set(attributes["k8s.node.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Node" - set(attributes["k8s.job.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Job" - set(attributes["k8s.cronjob.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "CronJob" - set(attributes["k8s.namespace.name"], attributes["regarding"]["namespace"]) where attributes["regarding"]["kind"] == "Pod" or attributes["regarding"]["kind"] == "Job" or attributes["regarding"]["kind"] == "CronJob" # Transform the type attribtes into OpenTelemetry Severity types. - set(severity_text, attributes["type"]) where attributes["type"] == "Normal" or attributes["type"] == "Warning" - set(severity_number, SEVERITY_NUMBER_INFO) where attributes["type"] == "Normal" - set(severity_number, SEVERITY_NUMBER_WARN) where attributes["type"] == "Warning" exporters: otlp/k8s-metrics: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-metrics" otlp/k8s-events: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-events" service: pipelines: traces: null metrics: receivers: [k8s_cluster] exporters: [ otlp/k8s-metrics ] logs: receivers: [k8sobjects] processors: [ memory_limiter, transform/events, batch ] exporters: [ otlp/k8s-events ] ports: jaeger-compact: enabled: false jaeger-thrift: enabled: false jaeger-grpc: enabled: false zipkin: enabled: false ``` Deploy the DaemonSet-mode Collector: ```shell theme={} helm install otel-collector open-telemetry/opentelemetry-collector --namespace honeycomb --values https://docs.honeycomb.io/_assets/code-samples/kubernetes/values-files/values-daemonset.yaml ``` [Download](/_assets/code-samples/kubernetes/values-files/values-daemonset.yaml) ```yaml theme={} mode: daemonset image: repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s # Required to use the kubeletstats cpu/memory utilization metrics clusterRole: create: true rules: - apiGroups: - "" resources: - nodes/proxy verbs: - get extraEnvs: - name: HONEYCOMB_API_KEY valueFrom: secretKeyRef: name: honeycomb key: api-key presets: # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines kubernetesAttributes: enabled: true extractAllPodLabels: true extractAllPodAnnotations: true # enables the kubeletstatsreceiver and adds it to the metrics pipelines kubeletMetrics: enabled: true config: receivers: jaeger: null zipkin: null kubeletstats: insecure_skip_verify: true # required as most clusters use self-signed certificates collection_interval: 30s metric_groups: - node - pod metrics: k8s.node.uptime: enabled: true k8s.pod.uptime: enabled: true k8s.pod.cpu_limit_utilization: enabled: true k8s.pod.cpu_request_utilization: enabled: true k8s.pod.memory_limit_utilization: enabled: true k8s.pod.memory_request_utilization: enabled: true exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" otlp_http/k8s-metrics: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-metrics" otlp_http/k8s-logs: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-logs" service: pipelines: traces: receivers: [otlp] exporters: [otlp_http] metrics: receivers: [kubeletstats] exporters: [otlp/k8s-metrics] logs: exporters: [otlp/k8s-logs] ports: jaeger-compact: enabled: false jaeger-thrift: enabled: false jaeger-grpc: enabled: false zipkin: enabled: false ``` Deploy the Deployment-mode Collector: ```shell theme={} helm install otel-collector-cluster open-telemetry/opentelemetry-collector --namespace honeycomb --values https://docs.honeycomb.io/_assets/code-samples/kubernetes/values-files/eu-values-deployment.yaml ``` [Download](/_assets/code-samples/kubernetes/values-files/eu-values-deployment.yaml) ```yaml theme={} mode: deployment image: repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s extraEnvs: - name: HONEYCOMB_API_KEY valueFrom: secretKeyRef: name: honeycomb key: api-key # We only want one of these collectors - any more and we'd produce duplicate data replicaCount: 1 presets: # enables the k8sclusterreceiver and adds it to the metrics pipelines clusterMetrics: enabled: true # enables the k8sobjectsreceiver to collect events only and adds it to the logs pipelines kubernetesEvents: enabled: true config: receivers: k8s_cluster: collection_interval: 30s metrics: # Disable replicaset metrics by default. These are typically high volume, low signal metrics. # If volume is not a concern, then the following blocks can be removed. k8s.replicaset.desired: enabled: false k8s.replicaset.available: enabled: false jaeger: null zipkin: null processors: transform/events: error_mode: ignore log_statements: - context: log statements: # adds a new watch-type attribute from the body if it exists - set(attributes["watch-type"], body["type"]) where IsMap(body) and body["type"] != nil # create new attributes from the body if the body is an object - merge_maps(attributes, body, "upsert") where IsMap(body) and body["object"] == nil - merge_maps(attributes, body["object"], "upsert") where IsMap(body) and body["object"] != nil # Transform the attributes so that the log events use the k8s.* semantic conventions - merge_maps(attributes, attributes[ "metadata"], "upsert") where IsMap(attributes[ "metadata"]) - set(attributes["k8s.pod.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Pod" - set(attributes["k8s.node.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Node" - set(attributes["k8s.job.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Job" - set(attributes["k8s.cronjob.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "CronJob" - set(attributes["k8s.namespace.name"], attributes["regarding"]["namespace"]) where attributes["regarding"]["kind"] == "Pod" or attributes["regarding"]["kind"] == "Job" or attributes["regarding"]["kind"] == "CronJob" # Transform the type attribtes into OpenTelemetry Severity types. - set(severity_text, attributes["type"]) where attributes["type"] == "Normal" or attributes["type"] == "Warning" - set(severity_number, SEVERITY_NUMBER_INFO) where attributes["type"] == "Normal" - set(severity_number, SEVERITY_NUMBER_WARN) where attributes["type"] == "Warning" exporters: otlp/k8s-metrics: # endpoint: "api.honeycomb.io:443" # US instance endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-metrics" otlp/k8s-events: # endpoint: "api.honeycomb.io:443" # US instance endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-events" service: pipelines: traces: null metrics: receivers: [k8s_cluster] exporters: [ otlp/k8s-metrics ] logs: receivers: [k8sobjects] processors: [ memory_limiter, transform/events, batch ] exporters: [ otlp/k8s-events ] ports: jaeger-compact: enabled: false jaeger-thrift: enabled: false jaeger-grpc: enabled: false zipkin: enabled: false ``` Deploy the DaemonSet-mode Collector: ```shell theme={} helm install otel-collector open-telemetry/opentelemetry-collector --namespace honeycomb --values https://docs.honeycomb.io/_assets/code-samples/kubernetes/values-files/eu-values-daemonset.yaml ``` [Download](/_assets/code-samples/kubernetes/values-files/eu-values-daemonset.yaml) ```yaml theme={} mode: daemonset image: repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s # Required to use the kubeletstats cpu/memory utilization metrics clusterRole: create: true rules: - apiGroups: - "" resources: - nodes/proxy verbs: - get extraEnvs: - name: HONEYCOMB_API_KEY valueFrom: secretKeyRef: name: honeycomb key: api-key presets: # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines kubernetesAttributes: enabled: true extractAllPodLabels: true extractAllPodAnnotations: true # enables the kubeletstatsreceiver and adds it to the metrics pipelines kubeletMetrics: enabled: true config: receivers: jaeger: null zipkin: null kubeletstats: insecure_skip_verify: true # required as most clusters use self-signed certificates collection_interval: 30s metric_groups: - node - pod metrics: k8s.node.uptime: enabled: true k8s.pod.uptime: enabled: true k8s.pod.cpu_limit_utilization: enabled: true k8s.pod.cpu_request_utilization: enabled: true k8s.pod.memory_limit_utilization: enabled: true k8s.pod.memory_request_utilization: enabled: true exporters: otlp_http: # endpoint: "https://api.honeycomb.io:443" # US instance endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" otlp_http/k8s-metrics: # endpoint: "https://api.honeycomb.io:443" # US instance endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-metrics" otlp_http/k8s-logs: # endpoint: "https://api.honeycomb.io:443" # US instance endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-logs" service: pipelines: traces: receivers: [otlp] exporters: [otlp_http] metrics: receivers: [kubeletstats] exporters: [otlp_http/k8s-metrics] logs: exporters: [otlp_http/k8s-logs] ports: jaeger-compact: enabled: false jaeger-thrift: enabled: false jaeger-grpc: enabled: false zipkin: enabled: false ``` If Collector installation fails and returns an error like the following: ```bash theme={} Error: INSTALLATION FAILED: template: opentelemetry-collector/templates/service.yaml:38:28: executing "opentelemetry-collector/templates/service.yaml" at : error calling include: template: opentelemetry-collector/templates/_helpers.tpl:148:44: executing "opentelemetry-collector.serviceInternalTrafficPolicy" at : error calling eq: incompatible types for comparison ``` Make sure you have [Helm 3.9+](https://helm.sh/) installed, as older Helm versions typically cause this error. ## Step 5: Verify the Collector Installation Check that the Collectors are installed by using the `kubectl` command to see if the pods are running: ```shell theme={} kubectl get pods --namespace honeycomb ``` This command should return something like: ```pre theme={} NAME READY STATUS RESTARTS AGE otel-collector-cluster-opentelemetry-collector-7c9cc9f8d-k9ncw 1/1 Running 0 9m21s otel-collector-opentelemetry-collector-agent-fcn5v 1/1 Running 0 17m ``` The result should contain one pod running under a name containing the prefix `otel-collector-cluster`, and one pod running under a name containing the word `agent` for each node in your Kubernetes cluster. You should now have an OpenTelemetry installation in your cluster that can: * Receive tracing data from service applications in your cluster and forward it to Honeycomb. * Gather and send metrics data from all of the pods in your cluster. * Gather and send metrics data about the nodes in your cluster. ## Explore Your Data in Honeycomb After a few minutes, data should start flowing into Honeycomb. If you do not see any data after several minutes, reach out for help in our [Pollinators Community Slack](/troubleshoot/community/#join-pollinators-community-slack). Pro/Enterprise users can [visit Honeycomb Support](https://support.honeycomb.io/) or [email support@honeycomb.io](mailto:support@honeycomb.io). To explore metrics related to your Kubernetes cluster, log in to Honeycomb and query the `k8s-metrics` dataset in your environment. Try asking the Query Assistant questions like: * "Show me the average CPU of my pods" * "What's the P99 memory usage of my nodes?" To explore the events emitted by Kubernetes itself, query the `k8s-events` dataset. These may be a little harder to understand, so try asking the Query Assistant questions like: * "Show me the pods that have a reason of Started" * "Show me pods that are crashing" For each question you ask, the Query Assistant will create a general query. You can further customize each query by adding visualizations, or by filtering or grouping the data. ## What's Next? Now that you have created an observability pipeline and have gotten some metrics, you can use these to get even more visibility into your Kubernetes cluster. * **Configure your applications to send data to the OpenTelemetry Collectors** Have you already instrumented your applications with OpenTelemetry? You'll need to [configure your pods and applications to send data to your new Collectors](/send-data/kubernetes/opentelemetry/collect-instrumented-code/). * **Add low-code, automatic instrumentation to your applications** Do you want more insight into your application data, but can't fully instrument your code yet? You can get even more insight by using the OpenTelemetry Operator to automatically instrument your applications. To learn more, visit [Low-Code Auto-Instrumentation with the OpenTelemetry Operator for Kubernetes](/send-data/kubernetes/opentelemetry/add-automatic-instrumentation/). ## Additional Information The DaemonSet-mode Collector uses the following components: * [Kubernetes Attribute Processor](/send-data/kubernetes/opentelemetry/components/#kubernetes-attributes-processor) * [Kubeletstats Receiver](/send-data/kubernetes/opentelemetry/components/#kubeletstats-receiver) The Deployment-mode Collector uses the following components: * [Kubernetes Cluster Receiver](/send-data/kubernetes/opentelemetry/components/#kubernetes-cluster-receiver) * [Kubernetes Objects Receiver](/send-data/kubernetes/opentelemetry/components/#kubernetes-objects-receiver) # Send Events with the Kubernetes Objects Receiver Source: https://docs.honeycomb.io/send-data/kubernetes/send-events-objects-receiver Send Kubernetes Events to Honeycomb using the OpenTelemetry Kubernetes Objects Receiver, even if your applications are not instrumented with OpenTelemetry. If you are running applications that are not using OpenTelemetry in your Kubernetes cluster, you can still collect the events from your cluster. In this guide, you will learn how to use an OpenTelemetry Collector to get additional insight into your data by sending your Kubernetes Events to Honeycomb using OpenTelemetry's [Kubernetes Objects Receiver](https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-objects-receiver). ## Before You Begin Before beginning this guide, you should have: * Created a running Kubernetes cluster. * Created a namespace named `honeycomb`. * [Deployed an OpenTelemetry Collector in Deployment mode with a Helm chart](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/#step-4-deploy-collectors). ## Collect Events Enable the Kubernetes Objects Receiver to collect events by adding the `kubernetesEvents` preset in the values file for your [OpenTelemetry Deployment-mode Collector](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/#step-4-deploy-collectors). Place it near the top of the values file under the `config` section: ```yaml theme={} presets: kubernetesEvents: enabled: true ``` To review the configuration, [download the values file for the Deployment-mode Collector](/_assets/code-samples/kubernetes/values-files/values-deployment.yaml). Apply this to only a Deployment-mode Collector with a replica count of 1. Running multiple receivers on the same Kubernetes cluster will cause duplicate events to be sent for every Collector running this receiver. ## Format Event Data By default, the `kubernetesEvents` preset in the OpenTelemetry Helm chart will configure the Collector to pull all of the events from the Kubernetes cluster and export them as logs. Honeycomb users can derive a great deal of value from these logs--when they are structured appropriately. To transform the bodies of these logs into structured content that is easily queried in Honeycomb, use the [OpenTelemetry Transform Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md) to parse the event data. To force the event data to conform to the standard `k8s.*` attribute naming that all other telemetry types use: 1. Add the following to the values file for your [OpenTelemetry Deployment-mode Collector](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/#step-4-deploy-collectors). Place it near the top of the values file under the `config` section: ```yaml theme={} processors: transform/events: error_mode: ignore log_statements: - context: log statements: - set(attributes["watch-type"], body["type"]) where IsMap(body) and body["type"] != nil - merge_maps(attributes, body, "upsert") where IsMap(body) and body["object"] == nil - merge_maps(attributes, body["object"], "upsert") where IsMap(body) and body["object"] != nil - merge_maps(attributes, attributes[ "metadata"], "upsert") where IsMap(attributes[ "metadata"]) # Maps the name of the resource to the right k8s.* attribute - set(attributes["k8s.pod.name"], attributes["regarding"] ["name"]) where attributes["regarding"]["kind"] == "Pod" - set(attributes["k8s.node.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Node" - set(attributes["k8s.job.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "Job" - set(attributes["k8s.cronjob.name"], attributes["regarding"]["name"]) where attributes["regarding"]["kind"] == "CronJob" - set(attributes["k8s.namespace.name"], attributes["regarding"]["namespace"]) where attributes["regarding"]["kind"] == "Pod" or attributes["regarding"]["kind"] == "Job" or attributes["regarding"]["kind"] == "CronJob" # Converts event types to Otel log Severities - set(severity_text, attributes["type"]) where attributes["type"] == "Normal" or attributes["type"] == "Warning" - set(severity_number, SEVERITY_NUMBER_INFO) where attributes["type"] == "Normal" - set(severity_number, SEVERITY_NUMBER_WARN) where attributes["type"] == "Warning" ``` 2. Under the `config` section, update your `pipelines` section to match the following: ```yaml theme={} service: pipelines: metrics: exporters: [ otlp/k8s-metrics ] logs: processors: [ memory_limiter, transform/events, batch ] exporters: [ otlp/k8s-logs ] ``` # Send Logs with OpenTelemetry's Filelog Receiver Source: https://docs.honeycomb.io/send-data/kubernetes/send-logs-filelog-receiver Send container logs from your Kubernetes cluster to Honeycomb using the OpenTelemetry Filelog Receiver, even without application-level instrumentation. If you are running applications that are not using OpenTelemetry in your Kubernetes cluster, you can still collect the logs from your containers. In this guide, you will learn how to use an OpenTelemetry Collector to get additional insight into your data by sending your container logs to Honeycomb using OpenTelemetry's [Filelog Receiver](https://opentelemetry.io/docs/kubernetes/collector/components/#filelog-receiver). ## Before You Begin Before beginning this guide, you should have: * Created a running Kubernetes cluster. * [Deployed an OpenTelemetry Collector in DaemonSet mode using a Helm chart](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/). ## Collect Logs Enable the Filelog Receiver to collect logs by adding the `logsCollection` preset in the values file for your [OpenTelemetry DaemonSet-mode Collector](/send-data/kubernetes/opentelemetry/create-telemetry-pipeline/#step-4-deploy-collectors). Place it near the top of the values file under the `config` section: ```yaml theme={} presets: logsCollection: enabled: true ``` [Download](/_assets/code-samples/kubernetes/values-files/values-daemonset.yaml) ```yaml theme={} mode: daemonset image: repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s # Required to use the kubeletstats cpu/memory utilization metrics clusterRole: create: true rules: - apiGroups: - "" resources: - nodes/proxy verbs: - get extraEnvs: - name: HONEYCOMB_API_KEY valueFrom: secretKeyRef: name: honeycomb key: api-key presets: # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines kubernetesAttributes: enabled: true extractAllPodLabels: true extractAllPodAnnotations: true # enables the kubeletstatsreceiver and adds it to the metrics pipelines kubeletMetrics: enabled: true config: receivers: jaeger: null zipkin: null kubeletstats: insecure_skip_verify: true # required as most clusters use self-signed certificates collection_interval: 30s metric_groups: - node - pod metrics: k8s.node.uptime: enabled: true k8s.pod.uptime: enabled: true k8s.pod.cpu_limit_utilization: enabled: true k8s.pod.cpu_request_utilization: enabled: true k8s.pod.memory_limit_utilization: enabled: true k8s.pod.memory_request_utilization: enabled: true exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" otlp_http/k8s-metrics: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-metrics" otlp_http/k8s-logs: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" "x-honeycomb-dataset": "k8s-logs" service: pipelines: traces: receivers: [otlp] exporters: [otlp_http] metrics: receivers: [kubeletstats] exporters: [otlp/k8s-metrics] logs: exporters: [otlp/k8s-logs] ports: jaeger-compact: enabled: false jaeger-thrift: enabled: false jaeger-grpc: enabled: false zipkin: enabled: false ``` ## Tune Logs Collection By default, the `logsCollection` preset in the OpenTelemetry Helm chart will configure the Collector to collect all the pod logs in a cluster. In larger clusters, you may want to configure the preset to target specific pods or applications to avoid being overwhelmed. ### Restrict Logs by Location or Name To collect only logs in a specific directory or with a specific filename, combine the preset with some configuration: ```yaml theme={} presets: logsCollection: enabled: true config: receivers: filelog: include: - /var/log/pods/my-namespace*/*/*.log - /var/log/pods/*mypodname*/*/*.log - /var/log/pods/*/my-containername/*.log ``` ### Restrict Logs by Label Selector To collect logs for only certain label selectors, use the Kubernetes Attributes processor to enable labels, and then the Filter processor to remove unwanted data: ```yaml theme={} presets: kubernetesAttributes: enabled: true extractAllPodLabels: true logsCollection: enabled: true config: processors: filter: error_mode: ignore logs: log_record: - resource.attributes["app.kubernetes.io/component"] != "myapp1" exporters: otlp: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "myapp1-logs" service: pipelines: logs: receivers: - filelog processors: - memory_limiter - k8sattributes - filter - batch exporters: - otlp ``` ### Send Logs to Different Datasets To send specific logs to different datasets, use the [Filter processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor) and multiple OTLP exporters: ```yaml theme={} presets: kubernetesAttributes: enabled: true extractAllPodLabels: true logsCollection: enabled: true config: processors: filter/myapp1: error_mode: ignore logs: log_record: - resource.attributes["app.kubernetes.io/component"] != "myapp1" filter/myapp2: error_mode: ignore logs: log_record: - resource.attributes["app.kubernetes.io/component"] != "myapp2" exporters: otlp/myapp1: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "myapp1-logs" otlp/myapp2: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "myapp2-logs" service: pipelines: logs: receivers: - filelog processors: - memory_limiter - k8sattributes - filter/myapp1 - batch exporters: - otlp/myapp1 logs/myapp2: receivers: - filelog processors: - memory_limiter - k8sattributes - filter/myapp2 - batch exporters: - otlp/myapp2 ``` # Send Logs from OpenTelemetry SDKs Source: https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk Configure an OpenTelemetry SDK to produce structured logs directly from your application and send them to Honeycomb, correlated with your existing traces. This guide details how to send OpenTelemetry Logs from OpenTelemetry SDKs to Honeycomb. ## What are OpenTelemetry Logs? OpenTelemetry Logs are **structured logs** that wrap the bodies of existing logs and optionally correlate them with traces. For example, structured application logs from a logging framework in your application will be automatically correlated with any tracing you also add to that application. OpenTelemetry Logs also enable you to centrally process logs data along with traces and metrics within the OpenTelemetry Collector. ## Send Logs from Your Application to Honeycomb You can send your structured application logs via OpenTelemetry to an endpoint, such as Honeycomb's endpoint or the OpenTelemetry Collector. This option is especially useful if structured logging exists in your application, since it allows for automatic correlation of structured logs with traces later. When you use an OpenTelemetry SDK to create OpenTelemetry Logs, you send to Honeycomb directly or an OpenTelemetry Collector configured to export to Honeycomb. OpenTelemetry Logs are not supported in all languages yet. Refer to the [OpenTelemetry availability status for each language](https://opentelemetry.io/status/). Explore examples related to sending OpenTelemetry logs to Honeycomb using the OpenTelemetry .NET SDK. Explore examples related to sending OpenTelemetry logs to Honeycomb using the OpenTelemetry Python SDK. Explore examples related to sending OpenTelemetry logs to Honeycomb using the OpenTelemetry Java SDK. Explore examples related to sending OpenTelemetry logs to Honeycomb using the OpenTelemetry JavaScript SDK. Explore examples related to sending OpenTelemetry logs to Honeycomb using the OpenTelemetry Go SDK. # Example: Send OpenTelemetry Logs with the OpenTelemetry .NET SDK Source: https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/dotnet Configure the OpenTelemetry .NET SDK logger provider to send structured logs from your .NET application to Honeycomb or an OpenTelemetry Collector. This example shows how to configure the OpenTelemetry .NET SDK's logger provider to send logs from a .NET application to Honeycomb or to an OpenTelemetry Collector. Install the OpenTelemetry SDK and OTLP exporter packages: ```shell theme={} dotnet add package OpenTelemetry dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol ``` During application setup, create and configure the logger provider: ```csharp theme={} using Microsoft.Extensions.Logging; using OpenTelemetry; using OpenTelemetry.Logs; // Configure a logger factory with OpenTelemetry and the OTLP log exporter using var loggerFactory = LoggerFactory.Create(builder => { builder.AddOpenTelemetry(options => { options.AddOtlpExporter(); }); }); // Create an ILogger instance from the logger factory var logger = loggerFactory.CreateLogger(); // Use the logger in your application logger.LogInformation("Something interesting happened"); ``` Set environment variables to configure the exporter and define your service name, then run your application. To send logs directly to Honeycomb: ```shell theme={} OTEL_SERVICE_NAME="my-service" \ OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io" \ OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=" \ dotnet run ``` To send logs to an OpenTelemetry Collector instead: ```shell theme={} OTEL_SERVICE_NAME="my-service" \ OTEL_EXPORTER_OTLP_ENDPOINT="my-collector:4317" \ OTEL_EXPORTER_OTLP_INSECURE=true \ dotnet run ``` # Example: Send OpenTelemetry Logs with the OpenTelemetry Go SDK Source: https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/go Configure the OpenTelemetry Go SDK to send structured logs from your Go application to Honeycomb or a Collector. Logs support in the OpenTelemetry Go SDK is currently experimental and subject to change. To check the status for each language, refer to the [OpenTelemetry status page](https://opentelemetry.io/status/). This example shows how to configure the OpenTelemetry Go SDK with `slog` instrumentation to send logs from a Go application to Honeycomb or an OpenTelemetry Collector. Install the OpenTelemetry `otelconf/x` and `otelslog` bridge packages: ```shell theme={} go get \ go.opentelemetry.io/contrib/otelconf/x \ go.opentelemetry.io/contrib/bridges/otelslog ``` During application setup, initialize the SDK from the config file and set the global logger provider: ```go theme={} package main import ( "context" "go.opentelemetry.io/contrib/bridges/otelslog" otelconf "go.opentelemetry.io/contrib/otelconf/x" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/log/global" ) func main() { // Set up the OpenTelemetry SDK from the config file sdk, err := otelconf.NewSDK() if err != nil { panic("failed to initialize OTel SDK") } defer sdk.Shutdown(context.Background()) otel.SetTracerProvider(sdk.TracerProvider()) otel.SetTextMapPropagator(sdk.Propagator()) // Set the logger provider globally global.SetLoggerProvider(sdk.LoggerProvider()) // Create a new slog logger instance logger := otelslog.NewLogger() // Use the logger directly anywhere in your app logger.Debug("Something interesting happened") } ``` Create an `otelconfig.yaml` file to configure your exporter. To send logs to Honeycomb: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} ``` To send logs to an OpenTelemetry Collector instead: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_grpc: endpoint: http://my-collector:4317 ``` When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. Point the SDK at your configuration file using `OTEL_CONFIG_FILE`, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml go run app.go ``` # Example: Send OpenTelemetry Logs with the OpenTelemetry Java SDK Source: https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/java Configure the OpenTelemetry Java SDK or Java agent to intercept log messages from your logging framework and send them to Honeycomb as structured OTLP logs. OpenTelemetry Java uses log appenders to intercept log messages from popular logging frameworks and convert them into OTLP logs. Log appenders work with both the OpenTelemetry Java agent and the OpenTelemetry Java SDK. The following example shows how to configure a Java application using Gradle to send Log4j logs to Honeycomb or an OpenTelemetry Collector. Install the OpenTelemetry Java agent. The Log4j appender is bundled with the agent and installs automatically. ```groovy theme={} dependencies { // OpenTelemetry Java Agent agent "io.opentelemetry.javaagent:opentelemetry-javaagent:${otelAgentVersion}" } ``` Install Log4j and the OpenTelemetry SDK, OTLP exporter and Log4j appender packages: ```groovy theme={} dependencies { // Log4j implementation("org.apache.logging.log4j:log4j-api:2.17.2") implementation("org.apache.logging.log4j:log4j-core:2.17.2") // OpenTelemetry SDK & OTLP exporter implementation("io.opentelemetry:opentelemetry-sdk") implementation("io.opentelemetry:opentelemetry-exporter-otlp") // OpenTelemetry Log4j appender implementation("io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17") } ``` The Log4j appender is bundled with the agent and requires no additional configuration. During application setup, install the OpenTelemetry Log4j appender when configuring the SDK: ```java theme={} import io.opentelemetry.instrumentation.log4j.appender.v2_17.OpenTelemetryAppender; import io.opentelemetry.sdk.OpenTelemetrySdk; // Setup OpenTelemetry SDK OpenTelemetrySdk sdk = OpenTelemetrySdk.builder() .build(); // Install the OpenTelemetry log4j log appender that intercepts log messages and create OTLP logs from them OpenTelemetryAppender.install(sdk); ``` Configure Log4j using a configuration file. The following example logs all messages to both the console and the OpenTelemetry log appender. The `packages` property on the top-level `Configuration` element tells Log4j to scan for custom appenders in the package. The OpenTelemetry appender is defined in `Appenders` and referenced in the root logger (`Loggers` > `Root`). ```xml theme={} ``` Then, in your application code you can use Log4j's `LogManager` to create loggers and emit log messages: ```java theme={} import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.LogManager; // Create a logger and use it in your application Logger logger = LogManager.getLogger("my-logger") Map mapMessage = new HashMap<>(); mapMessage.put("app.message", "Something interesting happened"); logger.info(new ObjectMessage(mapMessage)); ``` Create an `otelconfig.yaml` file and point the agent at it using `OTEL_CONFIG_FILE`. When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. To send logs directly to Honeycomb: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml ./gradlew run ``` To send logs to an OpenTelemetry Collector instead: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_grpc: endpoint: my-collector:4317 insecure: true propagator: composite: - tracecontext: - baggage: ``` ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml ./gradlew run ``` ## Available Log Appenders OpenTelemetry Java includes log appenders for popular logging frameworks. Each appender intercepts log messages and routes them through the OpenTelemetry export pipeline. Available log appenders include: * [Log4j Appender](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/log4j/log4j-appender-2.17/library) * [Logback](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-appender-1.0/library) * [JBoss Logmanager](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jboss-logmanager) (agent only) # Example: Send OpenTelemetry Logs with the OpenTelemetry JavaScript SDK Source: https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/javascript Configure the OpenTelemetry JavaScript SDK to send structured logs from your JavaScript application to Honeycomb or a Collector. Logs support in the OpenTelemetry JavaScript SDK is currently experimental and subject to change. To check the status for each language, refer to the [OpenTelemetry status page](https://opentelemetry.io/status/). This example shows how to configure the OpenTelemetry JavaScript SDK to capture Bunyan logging calls and send logs from a JavaScript application to Honeycomb or an OpenTelemetry Collector. Install the OpenTelemetry Node.js SDK, Bunyan instrumentation, and Bunyan logging framework packages: ```shell theme={} npm install --save @opentelemetry/sdk-node \ @opentelemetry/instrumentation-bunyan \ bunyan ``` Create an initialization file (for example, `telemetry.js`) that sets up the SDK with Bunyan instrumentation. The exporter configuration is provided by the `otelconfig.yaml` file at runtime. ```js theme={} const { startNodeSDK } = require('@opentelemetry/sdk-node'); const { BunyanInstrumentation } = require('@opentelemetry/instrumentation-bunyan'); const sdk = startNodeSDK({ instrumentations: [ new BunyanInstrumentation(), ] }); // Shut down the SDK gracefully before exiting to export all pending logs process.on('SIGTERM', () => { sdk .shutdown() .finally(() => process.exit(0)); }); ``` Then, in your application, create and use Bunyan loggers to capture interesting things: ```javascript theme={} const bunyan = require('bunyan'); // Create a logger and use it in your app const logger = bunyan.createLogger({name: 'myapp', level: 'info'}); logger.info({'app.message':'Something interesting happened'}); ``` Create an `otelconfig.yaml` file to configure your exporter. To send logs to Honeycomb: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} ``` To send logs to an OpenTelemetry Collector instead: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_grpc: endpoint: http://my-collector:4317 ``` When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. Point the SDK at your configuration file using `OTEL_CONFIG_FILE`, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml node -r ./telemetry.js app.js ``` ## Supported Logging Frameworks The OpenTelemetry JavaScript SDK can automatically create logs from these supported logging frameworks: * Bunyan * Winston # Example: Send OpenTelemetry Logs with the OpenTelemetry Python SDK Source: https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/python Configure the OpenTelemetry Python SDK logger provider to send structured logs from your Python application to Honeycomb or a Collector. Logs support in the OpenTelemetry Python SDK is currently experimental and subject to change. To check the status for each language, refer to the [OpenTelemetry status page](https://opentelemetry.io/status/). This example shows how to configure the OpenTelemetry Python SDK with declarative configuration to send logs from a Python application to Honeycomb or an OpenTelemetry Collector. Install the OpenTelemetry distro, instrumentation, and OTLP exporter packages: ```shell theme={} pip install opentelemetry-distro \ opentelemetry-instrumentation \ opentelemetry-exporter-otlp ``` Create an `otelconfig.yaml` file to configure your exporter. To send logs to Honeycomb: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} ``` To send logs to an OpenTelemetry Collector instead: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} logger_provider: processors: - batch: exporter: otlp_grpc: endpoint: http://my-collector:4317 ``` When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. Attach the OpenTelemetry `LoggingHandler` to the root logger during application startup so that standard library `logging` records reach the configured logger provider: ```python theme={} import logging from opentelemetry._logs import get_logger_provider from opentelemetry.sdk._logs import LoggingHandler logging.getLogger().addHandler( LoggingHandler(logger_provider=get_logger_provider()) ) # Use logging directly anywhere in your app logging.warning("Something interesting happened") ``` This step is a temporary workaround. Under `OTEL_CONFIG_FILE`, the `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` variable is ignored, so the handler must be attached manually. It will not be required in a future release. Upstream work is tracked in [open-telemetry/opentelemetry-python#5352](https://github.com/open-telemetry/opentelemetry-python/issues/5352). Point the SDK at your configuration file using `OTEL_CONFIG_FILE`, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml opentelemetry-instrument python app.py ``` # Send Structured Logs with Honeytail Source: https://docs.honeycomb.io/send-data/logs/structured/honeytail Tail, parse, and send existing structured log files to Honeycomb with Honeytail, Honeycomb's lightweight log ingestion agent. We have written a lightweight tool called `honeytail`. Honeytail will tail your existing log files, parse the content, and send it up to Honeycomb. If you already have structured data in an existing log, this is the easiest method to get that data in to Honeycomb. **The quality of your dataset within Honeycomb depends entirely upon the quality of the data going into the log file.** To get the most useful insight out of Honeycomb, you must provide high quality data in your log file. In addition to as much detail about each event as you can include, it is best to always include some host-level information to give each event in the log context. For example, the host on which the log exists. Honeytail is designed to run as a daemon so that it can continuously consume new content as it appears in the log files as well as detect when a log file rotates. It must be configured with your team API key and the name of the Dataset to which you want to write data. You specify one of the available parser modules depending on how your log data is structured. Once running, `honeytail` will take care of uploading all the data in your log file and picking up new data as it comes in. Honeytail is open source—we encourage auditing the software you will run on your servers. We also happily consider pull requests with new log format parsers and other improvements. To see an example of Honeytail in action, try out our [Honeytail Example](https://github.com/honeycombio/honeytail/tree/main/examples/backfill). ## Installation `honeytail` will tail existing log files, parse the content, and send it up to Honeycomb. You can [view its source here](https://github.com/honeycombio/honeytail). Download and install the latest `honeytail` by running: Download the `honeytail_1.10.0_amd64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb ``` Verify the package. ```shell theme={} echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_amd64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0_arm64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb ``` Verify the package. ```shell theme={} echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_arm64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0-1.x86_64.rpm` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm ``` Verify the package. ```shell theme={} echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c ``` Install the package. ```shell theme={} sudo rpm -i honeytail_1.10.0-1.x86_64.rpm ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64 ``` Verify the binary. ```shell theme={} echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64 ``` Verify the binary. ```shell theme={} echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64 ``` Verify the binary. ```shell theme={} echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository. ```shell theme={} git clone https://github.com/honeycombio/honeytail ``` Install from source. ```shell theme={} cd honeytail; go install ``` An [example Dockerfile](https://github.com/honeycombio/honeytail/blob/master/Dockerfile) is also available on GitHub. Then, modify the configuration file. Uncomment and set the following variables: * `ParserName` to the appropriate one of `json`, `nginx`, `mysql`, `arangodb`, `regex` * `WriteKey` to your [API key](/configure/environments/manage-api-keys/#find-api-keys) * `LogFiles` to the path for the log file you want to ingest, or `-` for `stdin` * `Dataset` to the name of the dataset you wish to create with this log file. The docs pages for [JSON](/send-data/logs/structured/json/), [NGINX](/send-data/logs/structured/nginx/), [MySQL](/send-data/logs/structured/mysql/), [PostgreSQL](/send-data/logs/structured/postgresql/) and [regex](/send-data/logs/unstructured/honeytail-regex/) have more detail on additional options to set for each parser. The other available options are all described in the configuration file and below. Launch `honeytail` by hand with `honeytail -c /etc/honeytail/honeytail.conf` or using the standard `sudo initctl start honeytail` (upstart) or `sudo systemctl start honeytail` (systemd) commands. `honeytail` will automatically start back up after rebooting your system. To disable this, put the word `manual` in `/etc/init/honeytail.override` (upstart) or run `systemctl disable honeytail` (systemd). ## Launch the Agent Start up a `honeytail` process using `upstart` or `systemd`, or by launching the process by hand. This will tail the log file specified in the configuration and leave the process running as a daemon. ```shell theme={} sudo initctl start honeytail ``` ```shell theme={} sudo systemctl start honeytail ``` ```shell theme={} honeytail -c /etc/honeytail/honeytail.conf ``` In order to start successfully sending data, you will need to update the configuration file included with these packages to specify the parser (for example, `nginx`), its associated options, such as where the log files are found, and the API key. We enforce a rate limit in order to protect our servers from abuse. This can be raised on a case-by-case basis; please [contact us](https://www.honeycomb.io/support/) to lift your limit. ## Data Formats Honeytail has predefined formats for many different use cases. It has built-in support for many existing log formats, including popular services like: * [MySQL](/send-data/logs/structured/mysql/) * [PostgresSQL](/send-data/logs/structured/postgresql/) * [Nginx](/send-data/logs/structured/nginx/) * Mongo * ArangoDB It also has support for common data formats that many services and applications use or can be configured to use: * [JSON](/send-data/logs/structured/json/) * [Regex](/send-data/logs/unstructured/honeytail-regex/) * CSV * KeyVal * Syslog ## Reading from Standard Input (stdin) Honeytail can read directly from standard input (stdin) by setting the file as `-`. In this example, the `honeytail` invocation uses standard input as the data source: ```shell theme={} honeytail \ -c /etc/honeytail/honeytail.conf \ --file=- ``` ## Backfilling Existing Data If you have events in older log files that you would like to load into Honeycomb, use `honeytail` with the `--backfill` option. Honeytail does not unzip log files, so you will need to do this before backfilling. Here is an example `honeytail` invocation to pull in multiple existing logs and as much as the current log as possible. ```shell theme={} honeytail \ -c /etc/honeytail/honeytail.conf \ --file=/var/log/app/myapp.log.* \ --file=/var/log/app/myapp.log \ --backfill ``` Let us break down the various parts of this command: * `--parser=json`: For the purposes of this example, all logs are already JSON formatted. Take a look at [the timestamp section](/send-data/logs/structured/json/#timestamp-parsing) of the JSON connector to make sure your historical logs have their times interpreted correctly. * `--file=/var/log/app/myapp.log.*`: Honeycomb understands file globs and will ingest all of the files in series. * `--file=/var/log/app/myapp.log`: Specify the `--file` (or its short form, `-f`) as many times as necessary to include additional files that do not match a glob. Ingest as much of the current file as exists. * `--backfill`: This flag tells `honeytail` to read the specified files in their entirety, stop when finished reading, and to respond to rate limited responses ([HTTP 429](https://http.cat/429)) by slowing down the rate at which it sends events. Honeytail will read all the content in all the old logs and then stop. When it finishes, you are ready to send new log lines. By default, `honeytail` will keep track of its progress through a file, and if interrupted, will pick back up where it left off. By launching honeytail pointing at the main app log, it will find the state file it created while reading in the backlog and start up where it left off. Here is the second `honeytail` invocation, where it will tail the current log file and send in recent entries: ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --parser=json \ --dataset='My App' \ --file=/var/log/app/myapp.log ``` ### Globs with Same File Names If using a glob that returns files with the same name in different directories, such as log files within multiple websites, use the `tail.hash_statefile_paths` option to ensure each tailed file gets its own state file. This option prevents a single state file from being used for multiple files, and avoids unnecessary file write contentions and load. ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --parser=json \ --dataset='My App' \ --file=/var/www/*/log/daily/app.log --tail.hash_statefile_paths ``` ## Sampling High Volume Data Let us say you have an incredible volume of log content and your website gets hit frequently enough that you will still get excellent data quality even if you are only looking at 1/20th the traffic. Honeytail can sample the log file and for each 20 lines, only send one of them. It does so randomly, so you will not see every 20th line being sent - instead each line will have a 5% chance of being sent. When these log lines reach Honeycomb, they will include metadata indicating that each one represents 20 similar lines, so all your graphs will show accurate total counts. ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset='Webtier' \ --parser=nginx \ --file=/var/log/nginx/access.log \ --samplerate 20 \ --nginx.conf /etc/nginx/nginx.conf \ --nginx.format main ``` Adjusting the sample rate based on the content of your events can allow you to keep important infrequent events while discarding less important higher volume traffic. Honeytail has a dynamic sampler that will vary the sample rate based on the contents of the fields of your choice - more frequent occurrences of the content of the field will be sampled more heavily. For example, suppose that successful web traffic (HTTP status codes in the 200 range) is much more frequent than errored traffic (status codes in the 500s) - you might want to discard more of the successful traffic and keep more of the errored traffic. Applying the dynamic sampler to the `status` field in your nginx traffic will have this effect. The actual sample rate applied will vary based on the cardinality of the chosen field and the frequency of each value, but it will be in the ballpark of the `samplerate` specified. ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset='Webtier' \ --parser=nginx \ --file=/var/log/nginx/access.log \ --samplerate 20 \ --nginx.conf /etc/nginx/nginx.conf \ --nginx.format main \ --dynsampling status ``` You can specify the `dynsampling` flag multiple times and it will sample traffic based on the frequency and uniqueness of concatenating all the values of the fields you specify. ## Sampling Deterministically In addition to static and dynamic sampling support, Honeytail also has support for sampling data deterministically based on the value of a field. This is useful for making sampling decisions based on properties like a request ID or trace ID. Approximately 1/N events will be sampled (where N is the sample rate), and any events, which have the same value for the field passed to the `--deterministic_sampling` flag, will be sampled consistently. This flag must be used with `--samplerate` to specify the sampling rate. ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset='Deterministically Sampled Nginx Logs' \ --parser=nginx \ --file=/var/log/nginx/access.log \ --deterministic_sampling request_id \ --samplerate 2 ``` For instance, in the above example, about half of the requests would be sampled and if another Honeytail instance were running elsewhere with the same settings for `--samplerate` and `--deterministic_sampling`, it would sample the same subset of requests. ## Adding Extra Information into Your Events It is not unusual for a log to omit interesting information like the name of the machine on which the process is running. After all, you are on that machine, right? Why would you add the hostname? Log transports like `rsyslog` will prepend logs with the hostname sending them, but if you are sending logs from each host, this data may not exist. Honeytail lets you add in extra fields to each event sent up to Honeycomb with the `--add_field` flag. For this example, let us assume that you have nginx running as a web server in both your production and staging environments. Your shell sets `$ENV` with the environment (prod or staging). Here is how to run `honeytail` to consume your nginx log and insert the hostname and environment along with each log line: ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset='Webtier' \ --parser=nginx \ --file=/var/log/nginx/access.log \ --nginx.conf /etc/nginx/nginx.conf \ --nginx.format main \ --add_field hostname=$(hostname) \ --add_field env=$ENV ``` When it comes time to add additional fields based on the **content** of your log file, it is time to invoke the data augmentation flag, `--da_map_file`. As an example, your log file might contain the IP address of the host connecting to this service, but you would really like to include the hostname in your events. Or your log file contains user ID and you would like to add user name and group. If you can build a map of source values to new fields, then you can use the `--da_map_file` flag to augment your data. As our example, let us add hostname and AWS availability zone to a log file that contains a IP addresses. The IP address is stored in a field called `ip_addr` in the events we are processing. The first step is to build a JSON file containing the name of the source column (`ip_addr`), and a map of values to new fields (`10.0.0.6` should add a field `hostname` with the value `app21` and `aws_az` of `c`, `10.0.0.7` has different fields). ```json theme={} { "ip_addr": { "10.0.0.6": { "hostname": "app21", "aws_az": "c" }, "10.0.0.7": { "hostname": "app32", "aws_az": "b" } } } ``` When the log parser comes across an event that has `"ip_addr":"10.0.0.6"`, it will add the two additional fields `"hostname":"app21","aws_az":"c"`. Note that additional source column names may be specified (each with their own translation map) by extending the content of the JSON map file. The recommended method to deploy this is to generate a map of all the values to the new fields that should be added and distribute it to all the hosts that will be running Honeytail. Example use cases: * a mysql slow query log includes the IP address of the client; translate those IP addresses to names and host groups to identify unexpected servers connecting to your database * ssh can log the fingerprint of the key used to authenticate; distribute a mapping of key fingerprint to key owners to make it easier to see who is logging in to what servers * different users are sharded to different backend clusters; use the authenticated user ID to add information about which shard they are on, making it easier to spot problems that are localized to individual shards ## Dropping or Scrubbing Fields Sometimes you will have fields in your log file that you do not want to send to Honeycomb or that you want to obscure before letting them leave your servers. For this example, let us say that you have in your log a large text field with the contents of an email. It is large enough that you do not want it sent up to Honeycomb. Also in this log you have a some sensitive information like a person's birthday. You want to be able to ask questions about the most common birthdays, but you do not want to expose the actual birthdays outside your infrastructure. Honeytail has two flags that will help you accomplish these goals. `--drop_field` will remove a field before sending the event to Honeycomb and `--scrub_field` will subject the value of a field to a SHA256 hash before sending it along. You will still be able to do inclusion and frequency analysis on the hashed fields (as there will be a 1-1 mapping of value to hashed value) but the actual value will be obscured. Here is your honeytail invocation: ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset='My App' \ --parser=json \ --file=/var/log/app/myapp.log \ --drop_field email_content \ --scrub_field birthday ``` ## Versioning `honeytail` Configuration ### Convert Command Lines to YAML Configuration The `honeytail` binary supports reading its configuration from a YAML configuration file, as well as command line arguments. To get started, if you have already been using a few command line arguments, add an additional flag: `--write_current_yaml`. This flag will write your command line configuration as YAML to `STDOUT` so you can use it as a starting point. ```shell theme={} honeytail \ -p mysql \ -k YOUR_API_KEY \ -d YOUR_DATASET \ -f ./mysql-slow.log \ --mysql.host=my.fake.host.com \ --write_current_yaml > ./scrubbed_mysql.yaml ``` Display the output captured in `scrubbed_mysql.yaml`. ```shell theme={} cat ./scrubbed_mysql.yaml apihost: https://api.honeycomb.io/ # or if on our EU instance, https://api.eu1.honeycomb.io samplerate: 1 poolsize: 80 send_frequency_ms: 100 send_batch_size: 50 status_interval: 60 request_parse_query: whitelist dynsample_window: 30 dynsample_minimum: 1 required_options: parsername: mysql writekey: YOUR_API_KEY logfiles: - ./mysql-slow.log dataset: YOUR_DATASET mysql: host: my.fake.host.com user: "" pass: "" queryinterval: 0 ``` This flag can be particularly useful for versioning or productionizing `honeytail` use, or for providing additional configuration when using advanced `honeytail` features, like [scrubbing sensitive fields](#dropping-or-scrubbing-fields) or parsing custom URL structures. ### Convert from an INI configuration to YAML Previous versions of `honeytail` used the INI file format (`.ini`) for configuration, but this is now deprecated. To create a YAML configuration file that is equivalent to an existing INI configuration, use this command with your `.ini` configuration file name: ```shell theme={} honeytail --config EXISTING_CONFIG_FILENAME --write_current_yaml ``` This command will write the YAML for your current configuration file to `STDOUT`. ### Using a Configuration File Once the configuration file is created, run `honeytail` with a `--config_yaml` argument in lieu of all of the other flags: ```shell theme={} honeytail --config_yaml ./scrubbed_mysql.yaml ``` ### Notes on Using YAML Configuration * When using YAML configuration, any configuration flags specified on the command line are ignored. * The names of fields in the YAML file are the same "long" names you see in the Honeytail help file. For example, `honeytail -h` * Any field with help that notes "May have multiple values" is specified in YAML as an array. For example: ```yaml theme={} scrub_field: - field_name_1 - field_name_2 ``` * Any values with dotted names in help are specified as YAML objects, with appropriate indentation. For example, `--mysql.host=my.fake.host.com` becomes: ```yaml theme={} mysql: host: my.fake.host.com user: "" pass: "" queryinterval: 0 ``` ## Parsing URL Patterns `honeytail` can break URLs up into their component parts, storing extra information in additional columns. This behavior is turned on by default for the `request` field on `nginx` datasets, but can become more useful with a little bit of guidance from you. There are several flags that adjust the behavior of `honeytail` as it breaks apart URLs. ### Identifying the URL Field When using the `nginx` parser, `honeytail` looks for a field named `request`. When using a different parser (such as the [JSON](/send-data/logs/structured/json/) parser), you should specify the name of the field that contains the URL with the `--request_shape` flag. Using this flag creates a few generated fields. Given a `request` field containing a value like: ```nginx theme={} GET /alpha/beta/gamma?foo=1&bar=2 HTTP/1.1 ``` ... will produce `nginx` events for Honeycomb that look like: | field name | value | description | | :------------------------- | :------------------------------------------ | :------------------------------------------------------- | | request | GET /alpha/beta/gamma?foo=1\&bar=2 HTTP/1.1 | the full original request | | request\_method | GET | the HTTP method, if it exists | | request\_protocol\_version | HTTP/1.1 | the HTTP version string | | request\_uri | /alpha/beta/gamma?foo=1\&bar=2 | the unmodified URL (not including the method or version) | | request\_path | /alpha/beta/gamma | just the path portion of the URL | | request\_query | foo=1\&bar=2 | just the query string portion of the URL | | request\_shape | /alpha/beta/gamma?foo=?\&bar=? | a normalized version of the URL | | request\_pathshape | /alpha/beta/gamma | a normalized version of the path portion of the URL | | request\_queryshape | foo=?\&bar=? | a normalized version of the query portion of the URL | (The generated fields will all be prefixed by the field name specified by `--request_shape`— in the above example `request`. Use the `--shape_prefix` field to prepend an additional string to these generated fields.) If the URL field contains just the URL, the `request_method` and `request_protocol_version` fields will be omitted. ### URL Normalization The path portion of the URL (from the beginning `/` up to the `?` that separates the path from the query) can be grouped by common patterns, as is common for REST interfaces. For example, given a URL fragments like: ```url theme={} /books/978-0812536362 /books/978-9995788940 ``` We can break the fragments into a field containing the generic endpoint (`/books/:isbn`) and a separate field for the ISBN number itself by specifying a `--request_pattern` flag: ```shell theme={} honeytail ... \ # other arguments --parser=nginx \ --request_pattern=/books/:isbn ``` This will produce, among other fields: | request\_path | request\_shape | request\_path\_isbn | (other fields) | | :-------------------- | :------------- | :------------------ | :------------- | | /books/978-0812536362 | /books/:isbn | 978-0812536362 | ... | | /books/978-9995788940 | /books/:isbn | 978-9995788940 | ... | You can specify multiple `--request_pattern` flags and they will be considered in order. The first one to match a URL will be used. Patterns should represent the entire path portion of the URL - include a "\*" at the end to match arbitrary additional segments. For example, if we have a wider variety of URL fragments, like: ```url theme={} /books/978-0812536362 /books/978-3161484100/borrow /books/978-9995788940 /books/978-9995788940/borrow ``` We can provide our additional `--request_pattern` flags and track a wider variety of `request_shape`s: ```shell theme={} honeytail ... \ # other arguments --parser=nginx \ --request_pattern=/books/:isbn/borrow --request_pattern=/books/:isbn ``` We will see our `request_path_isbn` populated as before, as the `:isbn` parameter is respected in both patterns: | request\_path | request\_shape | request\_path\_isbn | (other fields) | | :--------------------------- | :------------------ | :------------------ | :------------- | | /books/978-0812536362 | /books/:isbn | 978-0812536362 | ... | | /books/978-3161484100/borrow | /books/:isbn/borrow | 978-3161484100 | ... | | /books/978-9995788940 | /books/:isbn | 978-9995788940 | ... | | /books/978-9995788940/borrow | /books/:isbn/borrow | 978-9995788940 | ... | A URL's query string can be broken apart similarly, with the `--request_query_keys` flag, with generated fields named like `_query_`. If, on top of our previous examples, our URL fragments had query strings like: ```url theme={} /books/978-0812536362?borrower_id=23597 ``` Providing `--request_query_keys=borrower_id` would return us a Honeycomb event with a `request_query_borrower_id` field with a value of `23597`. If you would like to automatically create a field for every key in the query string, you can use the flag `--request_parse_query=all`. This will automatically create a new field `_query_` for every query parameter encountered in the query string. For any publicly accessible web server, it is likely that this will quickly create many useless columns because of all the random traffic on the internet. For more detail and examples, see our [urlshaper](https://github.com/honeycombio/urlshaper) package on GitHub. # Send JSON Logs Source: https://docs.honeycomb.io/send-data/logs/structured/json Ingest and backfill JSON log files into Honeycomb using Honeytail. Learn how Honeycomb handles nested JSON structures. Learn how to use Honeytail to send JSON logs directly to Honeycomb. ## Data Expectations Honeycomb expects data with a flat structure. By default, any structure deeper than top level keys will be serialized and a string representation of the content will be used in the field. However, Honeycomb can automatically unpack nested JSON objects and flatten them into unique columns. This is a per-dataset setting, and it is off by default. You must be a team owner to change this setting. If you enable this setting, nested objects will be flattened with new fields and field names created based on the keys. For example, `{"outer": {"inner": 42}}` would become a field `outer.inner` with a value of 42. To tell Honeycomb to automatically unpack JSON objects: 1. In the left navigation menu, select **Manage Data**. 2. In the list, locate and select **Datasets** and select the dataset you want to configure. 3. Select the **Schema** view. 4. Enable the **Automatically unpack non-OpenTelemetry nested JSON** toggle. 5. Select your preferred **Maximum unpacking depth** for your data. Changes to this setting take effect within 60 seconds. If your objects are deeply-nested, unpacking may result in a very large number of columns in Honeycomb. Consider unpacking only to the level of columns you will find useful. Any objects nested more deeply than the depth you select here will be converted to strings under the last unpacked column. In particular, if nested structures in your data can be created/added by your users (for example, HTTP headers), consider not unpacking them to that level. ## Installation Download and install the latest `honeytail` by running: Download the `honeytail_1.10.0_amd64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb ``` Verify the package. ```shell theme={} echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_amd64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0_arm64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb ``` Verify the package. ```shell theme={} echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_arm64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0-1.x86_64.rpm` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm ``` Verify the package. ```shell theme={} echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c ``` Install the package. ```shell theme={} sudo rpm -i honeytail_1.10.0-1.x86_64.rpm ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64 ``` Verify the binary. ```shell theme={} echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64 ``` Verify the binary. ```shell theme={} echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64 ``` Verify the binary. ```shell theme={} echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository. ```shell theme={} git clone https://github.com/honeycombio/honeytail ``` Install from source. ```shell theme={} cd honeytail; go install ``` You should modify the config file and uncomment and set: * `ParserName` to `json` * `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account) * `LogFiles` to the path for the log file you want to ingest, or `-` for stdin * `Dataset` to the name of the dataset you wish to create with this log file. ## Launch the Agent Start up a `honeytail` process using `upstart` or `systemd` or by launching the process by hand. This will tail the log file specified in the config and leave the process running as a daemon. ```shell theme={} sudo initctl start honeytail ``` ```shell theme={} sudo systemctl start honeytail ``` ```shell theme={} honeytail -c /etc/honeytail/honeytail.conf ``` ## Backfilling Archived Logs To backfill existing data, run `honeytail` with `--backfill` the first time: ```shell theme={} honeytail -c /etc/honeytail/honeytail.conf \ --file /var/log/myapp/log12.json \ --backfill ``` This command can also be used at any point to backfill from older, rotated log files. You can read more about our [backfill behavior here](/send-data/logs/structured/honeytail/). If you have chosen to backfill from old JSON logs, do not forget to transition into the default streaming behavior to stream live logs to Honeycomb! ## Timestamp Parsing Honeycomb expects all events to contain a timestamp field; if one is not provided, the server will associate the current time of ingest with the given payload. By default, we look for a few candidate fields based on name (`"timestamp"`, `"time"`) and handle the following time formats: * RFC3339 (`2006-01-02T15:04:05Z07:00`) * RFC3339 with nanoseconds (`2006-01-02T15:04:05.999999999Z07:00`) * Unix string representation (`Mon Jan 2 15:04:05 MST 2006`) * Ruby string representation (`Mon Jan 02 15:04:05 -0700 2006`) * Golang string representation (`2006-01-02 15:04:05.999999999 -0700 MST`) If your timestamps are not correctly handled by the above formats, use the `--json.timefield` and `--json.format` flags to help `honeytail` understand where and how to extract the event's timestamp. For example, given a JSON log file with events like the following: ```javascript theme={} {"color":"orange","size":3,"server_time":"Aug 12 2016, 15:12:06 -0800"} {"color":"blue","server_time":"Sep 01 2016, 06:10:32 -0800","size":4} ``` The command to consume those log lines (while retaining the `"server_time"` field as the event's timestamp) would look something like: ```shell theme={} honeytail --writekey=YOUR_API_KEY --dataset="API Server Logs" --parser=json \ --file=/var/log/api_server.log \ --json.timefield="server_time" --json.format="%b %d %Y, %k:%M:%S %z" ``` The `--json.timefield="server_time"` argument tells `honeytail` to consider the `"server_time"` value to be the canonical timestamp for the events in the specified file. The `--json.format` argument specifies the timestamp format to be used while parsing. (It understands common [`strftime`](https://www.strfti.me/) formats.) Ultimately, the above command would would produce events with the fields (note the times below are represented in UTC; Honeycomb parses time zone information if provided). | time | color | size | | :------------------- | :----- | :--- | | 2016-08-12T23:12:06Z | orange | 3 | | 2016-09-01T14:10:32Z | blue | 4 | # Send MySQL Logs Source: https://docs.honeycomb.io/send-data/logs/structured/mysql Parse MySQL logs with Honeytail and send them to Honeycomb to analyze database query patterns, slow queries, and traffic across your application. Our connector pulls your MySQL logs into Honeycomb for analysis, so you can analyze MySQL traffic on your machines and finally get a quick handle on the database queries triggered by your application logic. It surfaces attributes like: * The **normalized** query shape * Time spent waiting to acquire lock * Number of rows examined to execute the query * Number of rows returned by MySQL * ... and more! Honeycomb is unique in its ability to calculate metrics and statistics on the fly, while retaining the full-resolution log lines (and the original MySQL query that started it all!). Once you have got data flowing, be sure to take a look at our starter queries! Our entry points will help you see how we recommend comparing lock retention by normalized query, scan efficiency by collection, or read vs. write distribution by host. This document is for running **MySQL directly**. If running [MySQL on RDS](https://aws.amazon.com/rds/mysql/), Honeycomb offers support for ingesting RDS MySQL logs via [CloudWatch Logs](/send-data/aws/aws-cloudformation/) with the option to convert these unstructured logs into structured logs. The agent used to translate logs to events and send them to Honeycomb is called `honeytail`. ## Configure MySQL Query Logging Before running `honeytail`, you will want to turn slow query logging on for all queries if possible. To turn on slow query logging for your MySQL host, run the following in your MySQL shell: ```mysql theme={} mysql> SET GLOBAL slow_query_log = 'ON'; ``` Set the threshold for a query to be considered a "slow" query to `0` (the default is `10`): ```mysql theme={} mysql> SET GLOBAL long_query_time = 0; ``` And verify the slow query log's location via: ```mysql theme={} mysql> SELECT @@GLOBAL.slow_query_log_file; ``` If this technique is a problem for you—specifically, you do not want to rely on slow query log output—let us know! We have got something in the works that might satisfy your needs. ## Install and Run Honeytail On your MySQL host, download and install the latest `honeytail` by running: Download the `honeytail_1.10.0_amd64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb ``` Verify the package. ```shell theme={} echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_amd64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0_arm64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb ``` Verify the package. ```shell theme={} echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_arm64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0-1.x86_64.rpm` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm ``` Verify the package. ```shell theme={} echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c ``` Install the package. ```shell theme={} sudo rpm -i honeytail_1.10.0-1.x86_64.rpm ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64 ``` Verify the binary. ```shell theme={} echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64 ``` Verify the binary. ```shell theme={} echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64 ``` Verify the binary. ```shell theme={} echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository. ```shell theme={} git clone https://github.com/honeycombio/honeytail ``` Install from source. ```shell theme={} cd honeytail; go install ``` Make sure you have enabled [MySQL query logging](#configure-mysql-query-logging) before running `honeytail`. To consume the current MySQL slow query log from the beginning, run: ```shell theme={} honeytail --writekey=YOUR_API_KEY --dataset=MySQL --parser=mysql \ --file=/usr/local/var/mysql/myhost-slow.log \ --tail.read_from=beginning ``` ## Troubleshooting Check out [`honeytail` Troubleshooting](/troubleshoot/common-issues/sending-data/#honeytail) for debugging tips. ## Run Honeytail Continuously To run `honeytail` continuously as a daemon process, first modify the config file `/etc/honeytail/honeytail.conf` and uncomment and set: * `ParserName` to `mysql` * `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account) * `LogFiles` to the path for your MySQL slow query log file, often located at `/usr/local/var/mysql/myhost-slow.log` * `Dataset` to the name of the dataset you wish to create with this log file. Then start `honeytail` using `upstart` or `systemd`: ```shell theme={} sudo initctl start honeytail ``` ```shell theme={} sudo systemctl start honeytail ``` ## Backfill Archived Logs You may have archived logs that you would like to import into Honeycomb. If you have a MySQL logfile located at `/usr/local/var/mysql/myhost-slow.16.log`, you can backfill using this command: ```shell theme={} honeytail --writekey=YOUR_API_KEY --dataset=MySQL --parser=mysql \ --file=/usr/local/var/mysql/myhost-slow.16.log \ --backfill ``` This command can be used at any point to backfill from archived log files. You can read more about `honeytail`'s backfill behavior [here](/send-data/logs/structured/honeytail/#backfilling-existing-data). `honeytail` does not unzip log files, so you will need to do this before backfilling. Once you have finished backfilling your old logs, we recommend transitioning to the default streaming behavior to stream live logs to Honeycomb. ## Scrub Personally Identifiable Information While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data, which may contain sensitive user information. With that in mind, we recommend using `honeytail`'s MySQL parser, but adding a `--scrub_field=query` flag to hash the concrete `query` value. The `normalized_query` attribute will still be representative of the **shape** of the query, and identifying patterns including specific queries will still be possible—but the sensitive information will be completely obscured before leaving your servers. More information about dropping or scrubbing sensitive fields can be [found here](/send-data/logs/structured/honeytail/#dropping-or-scrubbing-fields). ## Example Extracted MySQL Fields Ingesting a MySQL log line (resulting from a `SELECT` with a `JOIN`): ```mysql theme={} # Time: 161019 18:30:00 # User@Host: rdsadmin[rdsadmin] @ localhost [127.0.0.1] Id: 1 # Query_time: 1.294391 Lock_time: 0.000119 Rows_sent: 4049 Rows_examined: 4049 SET timestamp=1476901800; SELECT teams.* FROM teams INNER JOIN users_teams ON team_id=teams.id WHERE user_id=21782 AND slug='foobar' LIMIT 1 ``` will produce an event for Honeycomb that looks like: | field name | value | type | | :---------------- | :----- | :------------------------------------------------------------------------------------------------------------------- | | client | string | localhost | | client\_ip | string | 127.0.0.1 | | lock\_time | float | 0.000119 | | normalized\_query | string | `select teams._ from teams inner join users_teams on team_id = teams.id where user_id = ? and slug = ? limit ?` | | query | string | `SELECT teams.* FROM teams INNER JOIN users_teams ON team_id=teams.id WHERE user_id=21782 AND slug='foobar' LIMIT 1` | | query\_time | float | 1.294391 | | rows\_examined | float | 4049 | | rows\_sent | float | 4049 | | statement | string | select | | tables | string | teams users\_teams | | user | string | rdsadmin | Numbers are ingested as floats by default in Honeycomb, though you can coerce a field to integers in the Schema section of your dataset's Overview. You can find more on our MySQL query normalization in [our `mysqltools` repository](https://github.com/honeycombio/mysqltools). ## Open Source [Honeytail](https://github.com/honeycombio/honeytail) is open source and Apache 2.0 licensed. # Send NGINX Logs Source: https://docs.honeycomb.io/send-data/logs/structured/nginx Parse NGINX logs with Honeytail and send them to Honeycomb to analyze web traffic, request patterns, and service activity across your infrastructure. NGINX is one of the most popular web servers today. In a world driven by the web and connected APIs, its logs are a great candidate for surfacing a birds' eye view of activity in your service. Explore the [Honeytail NGINX Example App](https://github.com/honeycombio/honeytail/tree/main/examples/nginx) and read the [blog post with user tips on sending NGINX logs to Honeycomb](https://www.honeycomb.io/blog/sending-nginx-logs-honeycomb). ## Setup Capturing web logs for Honeycomb requires: 1. installing our agent, `honeytail` 2. configuring it to parse your NGINX logs correctly 3. launching honeytail ### Install the Agent Download and install the latest `honeytail` by running: Download the `honeytail_1.10.0_amd64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb ``` Verify the package. ```shell theme={} echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_amd64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0_arm64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb ``` Verify the package. ```shell theme={} echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_arm64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0-1.x86_64.rpm` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm ``` Verify the package. ```shell theme={} echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c ``` Install the package. ```shell theme={} sudo rpm -i honeytail_1.10.0-1.x86_64.rpm ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64 ``` Verify the binary. ```shell theme={} echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64 ``` Verify the binary. ```shell theme={} echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64 ``` Verify the binary. ```shell theme={} echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository. ```shell theme={} git clone https://github.com/honeycombio/honeytail ``` Install from source. ```shell theme={} cd honeytail; go install ``` You should modify the config file and uncomment and set: * `ParserName` to `nginx` * `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account) * `LogFiles` to the path for the log file you want to ingest. For NGINX, this is typically `/var/log/nginx/access.log`. * `Dataset` to the name of the dataset you wish to create with this log file. Honeytail also supports [configuration with YAML](/send-data/logs/structured/honeytail/#convert-command-lines-to-yaml-configuration). ### Identify Log Locations + Formats Make sure to run through [Optional Configuration](#optional-configuration) below before running `honeytail`, in order to get the richest metadata out of your web traffic and into your logs. In addition to the standard configuration captured in `/etc/honeytail/honeytail.conf`, you will want to set the two options in the `Nginx Parser Options` section: * `ConfigFile`: the path to your NGINX config file: whichever part of it contains the definition for the log format * `LogFormatName`: the name of the log format used to produce the NGINX access log file For example, if your nginx config file is at `/etc/nginx/nginx.conf` and has the following snippet: ```nginx theme={} log_format my_favorite_format '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent'; access_log /var/log/nginx/access.log my_favorite_format; ``` ... then `ConfigFile` should be set to `/etc/nginx/nginx.conf` and your `LogFormatName` value should be set to `my_favorite_format`. Or configure `honeytail` to read the nginx logs using command line parameters: ```shell theme={} honeytail \ --parser=nginx \ --dataset=examples.honeytail-nginx \ --writekey=$HONEYCOMB_WRITE_KEY \ --nginx.conf=/etc/nginx/nginx.conf \ --nginx.format=my_favorite_format \ --file=/var/log/nginx/access.log ``` ### Launch the Agent Start up a `honeytail` process using `upstart` or `systemd` or by launching the process by hand. ```shell theme={} sudo initctl start honeytail ``` ```shell theme={} sudo systemctl start honeytail ``` ```shell theme={} honeytail -c /etc/honeytail/honeytail.conf ``` ## Backfilling Archived Logs In addition to getting current logs flowing, you can backfill old logs into Honeycomb to kickstart your dataset. By running `honeytail` from the command line, you can import old logs separate from tailing your current logs. Adding the `--backfill` flag to `honeytail` adjusts a number of settings to make it appropriate for backfilling old data, such as stopping when it gets to the end of the log file instead of the default behavior of waiting for new content (like `tail`). The specific locations on your system may vary from ours, but once you fill in your system's values instead of our examples, you can backfill using this command: ```shell theme={} honeytail --writekey=YOUR_API_KEY --dataset="nginx API logs" --parser=nginx \ --file=/var/log/nginx/access.16.log \ --nginx.conf=/etc/nginx/nginx.conf \ --nginx.format=api_fmt \ --backfill ``` This command can be used at any point to backfill from archived log files. You can read more about our [agent honeytail](/send-data/logs/structured/honeytail/) or its [backfill behavior](/send-data/logs/structured/honeytail/#backfilling-existing-data) here. `honeytail` does not unzip log files, so you will need to do this before backfilling. Easiest way—pipe to STDIN: `zcat *.gz | honeytail --file - --backfill --all-the-other-flags.` ## Troubleshooting Check out [`honeytail` Troubleshooting](/troubleshoot/common-issues/sending-data/#honeytail) for debugging tips. ## Optional Configuration Nginx logs can be an incredibly powerful, high-level view of your system—especially so if they are configured correctly and enriched with custom, application-specific information about each request. Below are two simple ways to pack those logs with more useful metadata. ### Missing Default Options Nginx comes with some fairly powerful optional log fields that are not included by default. This is the `log_format` we recommend for any configuration file (note the extra quotes around some fields): ```nginx theme={} log_format combined '$remote_addr - $remote_user [$time_local] $host ' '"$request" $status $bytes_sent $body_bytes_sent $request_time ' '"$http_referer" "$http_user_agent" $request_length "$http_authorization" ' '"$http_x_forwarded_proto" "$http_x_forwarded_for" $server_name'; access_log /var/log/nginx/access.log combined; ``` You may already have an `access_log` line, but by defining a `log_format` (`combined`, in the example above) and specifying the format name (`--nginx.format=combined`), you will be able to take advantage of all of these additional fields. Make sure that all fields that start `$http_` are quoted in your `log_format`: * `$bytes_sent`: the size of the response sent back to the client, including headers * `$host`: the requested Host header, identifying how your server was addressed * `$http_authorization`: authorization headers, for associating logs with individual users (must be quoted) * `$http_referer`: the referring site, if the client followed a link to your site (must be quoted) * `$http_user_agent`: the User-Agent header, useful in identifying your clients (must be quoted) * `$http_x_forwarded_for`: the origin IP address, if running behind a load balancer (must be quoted) * `$http_x_forwarded_proto`: the origin protocol, if terminating TLS in front of nginx (must be quoted) * `$remote_addr`: the IP address of the host making the connection to nginx * `$remote_user`: the user name supplied when/if using basic authentication * `$request_id`: an nginx-generated unique ID to every request (only available in nginx version 1.11 and later). * `$request_length`: the length of the client's request, including headers and body * `$request_time`: the time (in ms) the server took to respond to the request * `$request`: the HTTP method, request path, and protocol version * `$server_name`: the hostname of the machine accepting the request * `$status`: the HTTP status code returned for this request ### Embedding Custom Response Headers Nginx can also be configured to extract custom request and response headers. Of the two, response headers are the most powerful in this case—they can carry application-specific IDs or timers back through to the nginx log. Having all of the information pertinent to a single request, available in a single log line, can be an incredibly powerful tool in diagnosing the origin of a problem in your system. To include a specific response header in your `access.log`, add an [`$upstream_http_`](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_http_) variable to your `log_format`—the response header values will be written out and ingested by our nginx parser! Make sure to put quotes around these variables to capture any embedded spaces. For example, an `X-RateLimit-Remaining` header can be output by adding `$upstream_http_x_ratelimit_remaining` to the `log_format` line. See the nginx docs for more about [extracting metadata from the HTTP response](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#variables) or [request](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_http_). As with other fields which may output strings (for example,`$http_user_agent`), be careful when logging strings—add an extra set of double quotes around values which might contain spaces, in order to ensure correct parsing. **A final trick**: sometimes, response headers may be set for logging that should not be exposed back to the user. In this case, the [`proxy_hide_header`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header) directive may be used to strip out specific headers by name: ```nginx theme={} log_format combined `... "$upstream_x_internal_top_secret" ...`; # Wrap string values with double quotes access_log /var/log/nginx/access.log combined; location / { proxy_pass http://127.0.0.1:8080; # Expose port 8080 proxy_hide_header X-Internal-Top-Secret; # Strip from client } ``` ## Scrubbing Personally Identifiable Information While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data which may contain sensitive user information. With that in mind, we recommend using `honeytail`'s nginx parser, but adding a `--scrub_field=sensitive_field_name` flag to hash the concrete `sensitive_field_name` value, or `--drop_field=sensitive_field_name` to drop it altogether and prevent it being sent to Honeycomb's servers. Find [more information about dropping or scrubbing sensitive field](/send-data/logs/structured/honeytail/#dropping-or-scrubbing-fields). ## Parsing URL Patterns `honeytail` can break URLs up into their component parts, storing extra information in additional columns. This behavior is turned on by default for the `request` field on `nginx` datasets, but can become more useful with a little bit of guidance from you. See [`honeytail`'s documentation](/send-data/logs/structured/honeytail/#parsing-url-patterns) for details on configuring our agent to parse URL strings. ## Open Source [Honeytail](https://github.com/honeycombio/honeytail) is open source and Apache 2.0 licensed. # Send Other Webserver Logs Source: https://docs.honeycomb.io/send-data/logs/structured/other-webserver Parse Apache, HAProxy, and other webserver logs with Honeytail using a customized NGINX parser configuration and send them to Honeycomb. To support a variety of webserver and related technologies, our [Honeytail agent](/send-data/logs/structured/honeytail/) has an [Nginx parser](/send-data/logs/structured/nginx/) that can be easily tricked in to parsing other webservers' logs. You will create a config that contains the log format of your web server and pass it to the Honeytail nginx parser. As an example, this page describes how to consume [HAProxy](https://www.haproxy.org) and [Apache](https://httpd.apache.org/) logs using the `honeytail` nginx parser. ## Overview To use the nginx parser to consume a non-nginx log file, we will create a config that looks something like nginx config and use it to define the log format. We will then run `honeytail` on the log using the config file containing the format. The config file will have one statement `log_format name '';` (maybe broken up in to multiple lines). The format will be a series of labels identifying each field - the character following each label is the field separator. For example, to collect the HAProxy name, pid, ip address, and port from a log snippet of `haproxy[291] 127.0.0.1:4715`, you would use `$process[$pid] $ip:$port` as your format string. You can use any names you like for the labels—they will be used as the column names in Honeycomb. Below are two examples—HAProxy's http log formats and the default apache log format. You will likely have to tailor these examples to your specific config depending on the version of the web server you are running and other options you may have in their configs. ## HAProxy Http Format The HAProxy's http format for logs has a wealth of detail packed in to a very compact form. Here is a sample log line (from the [HAProxy docs](https://docs.haproxy.org/2.6/configuration.html#8.2.3)): ```log theme={} Feb 6 12:12:56 localhost \ haproxy[14389]: 10.0.1.2:33317 [06/Feb/2009:12:14:14.655] http-in static/srv1 \ 10/0/30/69/109 200 2750 - - ---- 1/1/1/1/0 0/0 {1wt.eu} {}\ "GET /index.html HTTP/1.1" ``` Here is the description of those fields (again, from the haproxy docs): | Field | Format | Extract from the example above | | ----- | --------------------------------------------------------- | ------------------------------ | | 1 | `process_name '[' pid ']:'` | `haproxy[14389]:` | | 2 | `client_ip ':' client_port` | `10.0.1.2:33317` | | 3 | `'[' accept_date ']'` | `[06/Feb/2009:12:14:14.655]` | | 4 | `frontend_name` | `http-in` | | 5 | `backend_name '/' server_name` | `static/srv1` | | 6 | `Tq '/' Tw '/' Tc '/' Tr '/' Tt*` | `10/0/30/69/109` | | 7 | `status_code` | `200` | | 8 | `bytes_read*` | `2750` | | 9 | `captured_request_cookie` | `-` | | 10 | `captured_response_cookie` | `-` | | 11 | `termination_state` | `----` | | 12 | `actconn '/' feconn '/' beconn '/' srv_conn '/' retries*` | `1/1/1/1/0` | | 13 | `srv_queue '/' backend_queue` | `0/0` | | 14 | `'{' captured_request_headers* '}'` | `{haproxy.1wt.eu}` | | 15 | `'{' captured_response_headers* '}'` | `{}` | | 16 | `'"' http_request '"'` | `"GET /index.html HTTP/1.1"` | Here is the config snippet used to match that log line. Because there are two date fields, we are going to use the one in square brackets `[]` because it is easier to parse and it is in the correct format (`d/m/y:h:m:s.sss`). We will stub out the syslog-provided date at the beginning of the line by using dots (to match any character). We can split up the log\_format line into multiple lines for easier editing. Make sure the last line ends with a semicolon. For this example, let us call this file `hny-haproxy.conf` ```nginx theme={} log_format haproxy '... .. ..:..:.. $hostname $process[$pid]: ' '$client_ip:$client_port [$time_local] $frontend $backend/$backend_server ' '$time_client_connect/$time_queued/$time_backend_conn/$time_backend_resp/$time_total ' '$status_code $bytes_read $request_cookie $response_cookie $termination_state ' '$act_conn/$fe_conn/$be_conn/$srv_conn/$retries $srv_queue/$backend_queue ' '{$request_headers} {$response_headers} "$request"'; ``` To use this config, you would run [our agent `honeytail`](/send-data/logs/structured/honeytail/) like this: ```shell theme={} honeytail \ -k YOUR_API_KEY \ -p nginx \ -d haproxy \ -f /path/to/haproxy.log \ --nginx.conf hny-haproxy.conf \ --nginx.format haproxy ``` ## Apache Log Format Apache's configuration can truly go as far as you want to take it. For this example, let us just stick with the default log format. Here is an example log line (split into two for readability) ```log theme={} 207.46.1.2 - - [03/Nov/2016:16:11:43 -0700] "GET /robots.txt HTTP/1.1" 200 334 \ "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" ``` From an Apache logging config like this: ```apache theme={} LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined ``` There is not nearly as much there as in the HAProxy log, but let us pull out what we can, taking a hint from the [Apache docs](https://httpd.apache.org/docs/current/mod/mod_log_config.html) to decipher the fields. Let us call this file `hny-apache.conf` ```nginx theme={} log_format apache '$remote_ip $identd $user [$time_local] "$request" $status_code ' '$bytes_sent "$referrer" "$user_agent"'; ``` To use this config, you would run [our agent `honeytail`](/send-data/logs/structured/honeytail/) like this: ```shell theme={} honeytail \ -k YOUR_API_KEY \ -p nginx \ -d apache \ -f /path/to/apache/access.log \ --nginx.conf hny-apache.conf \ --nginx.format apache ``` ## Details on The Log Format ### Timestamps The nginx parser can only interpret timestamps in one of the two formats that nginx itself uses. The field in the log format description **must be named correctly** in order for Honeycomb to use the timestamp for the event instead of considering it a normal string field. * `$time_local` : Time in the [Common Log Format](https://en.wikipedia.org/wiki/Common_Log_Format), `06/Feb/2009:12:14:14.655` for example * `$time_iso8601`: Time in the [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) standard format, `2009-02-06T12:14:14+00:00` for example ### Case and Allowed Characters The field names in the `log_format` specification have some restrictions - it must only contain characters in the set `[a-z_]`. In other words, they: * Must be all lower case * Must not contain spaces * May only contain letters and underscores (no numbers or other symbols) Any other characters will be considered a field delimiter in the log format. ## Suggested Queries in Honeycomb Just to whet your appetite, we would like to suggest a few graphs to explore with your haproxy dataset: * **Slowest endpoints**: breakdown by `http_request`, calculate p95(`time_duration`), order by p95(`time_duration`) descending * **Average connection time per backend server**: breakdown by `backend_name`, `server_name`, calculate avg(`time_backend_connect`) ## Scrubbing Personally Identifiable Information While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data, which may contain sensitive user information. With that in mind, we recommend using `honeytail`'s nginx parser, but adding a `--scrub_field=sensitive_field_name` flag to hash the concrete `sensitive_field_name` value, or `--drop_field=sensitive_field_name` to drop it altogether and prevent it being sent to Honeycomb's servers. More information about dropping or scrubbing sensitive fields can be [found here](/send-data/logs/structured/honeytail/#dropping-or-scrubbing-fields). ## Parsing URL Patterns `honeytail` can break URLs up into their component parts, storing extra information in additional columns. This behavior is turned on by default for the `request` field on `nginx` datasets, but can become more useful with a little bit of guidance from you. See \[`honeytail`'s documentation]/send-data/logs/structured/honeytail/#parsing-url-patterns) for details on configuring our agent to parse URL strings. # Send PostgreSQL Logs Source: https://docs.honeycomb.io/send-data/logs/structured/postgresql Parse PostgreSQL logs with Honeytail and send them to Honeycomb to analyze database query patterns, slow queries, and traffic across your application. Our connector pulls your PostgreSQL logs into Honeycomb for analysis, so you can analyze PostgreSQL traffic on your machines and finally get a quick handle on the database queries triggered by your application logic. It surfaces attributes like: * The **normalized** query shape * Time spent executing the query * Transaction ID * Client information * ... and more! Honeycomb is unique in its ability to calculate metrics and statistics on the fly, while retaining the full-resolution log lines (and the original query that started it all!). This document is for running **PostgreSQL directly**. If running [PostgreSQL on RDS](https://aws.amazon.com/rds/postgresql/), Honeycomb offers support for ingesting RDS PostgreSQL logs via [CloudWatch Logs](/send-data/aws/aws-cloudformation/) with the option to convert these unstructured logs into structured logs. The agent used to translate logs to events and send them to Honeycomb is called `honeytail`. ## Configure PostgreSQL Query Logging Before running `honeytail`, turn slow query logging on for all queries if possible. To turn on slow query logging, edit your `postgresql.conf` and set: ```sql theme={} log_min_duration_statement = 0 log_statement='none' ``` `log_statement` indicates which types of queries are logged, but is superseded when setting `log_min_duration_statement` to `0`, as this effectively logs all queries. Setting `log_statement` to any other value will change the format of the query logs in a way that is not currently supported by the Honeycomb PostgreSQL parser. Alternatively, you can set this from the `psql` shell by running ```sql theme={} ALTER SYSTEM SET log_min_duration_statement=0; ALTER SYSTEM SET log_statement='none'; SELECT pg_reload_conf(); ``` Finally, take note of the value of the `log_line_prefix` configuration line. It will look something like this: ```sql theme={} log_line_prefix = '%t [%p-%l] %q%u@%d ' ``` ## Install and Run Honeytail On your PostgreSQL host, download and install the latest `honeytail` by running: Download the `honeytail_1.10.0_amd64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb ``` Verify the package. ```shell theme={} echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_amd64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0_arm64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb ``` Verify the package. ```shell theme={} echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_arm64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0-1.x86_64.rpm` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm ``` Verify the package. ```shell theme={} echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c ``` Install the package. ```shell theme={} sudo rpm -i honeytail_1.10.0-1.x86_64.rpm ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64 ``` Verify the binary. ```shell theme={} echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64 ``` Verify the binary. ```shell theme={} echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64 ``` Verify the binary. ```shell theme={} echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository. ```shell theme={} git clone https://github.com/honeycombio/honeytail ``` Install from source. ```shell theme={} cd honeytail; go install ``` Make sure you have enabled [query logging](#configure-postgresql-query-logging) before running `honeytail`. To consume the current slow query log from the beginning, run: ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset=postgres-queries --parser=postgresql \ --postgresql.log_line_prefix=YOUR_LOG_LINE_PREFIX \ --file=/var/log/postgresql/postgresql-9.5-main.log \ --tail.read_from=beginning ``` ## Troubleshooting Check out [`honeytail` Troubleshooting](/troubleshoot/common-issues/sending-data/#honeytail) for debugging tips. ## Run Honeytail Continuously To run `honeytail` continuously as a daemon process, first modify the configuration file `/etc/honeytail/honeytail.conf` and uncomment and set: * `ParserName` to `postgresql` * `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account) * `LogFiles` to the path for your PostgreSQL log file. * `Dataset` to the name of the dataset you wish to create with this log file. Then start `honeytail` using `upstart` or `systemd`: ```shell theme={} sudo initctl start honeytail ``` ```shell theme={} sudo systemctl start honeytail ``` ## Backfill Archived Logs You may have archived logs that you would like to import into Honeycomb. If you have a log file located at `/var/log/postgresql/postgresql-main.log`, you can backfill using this command: ```shell theme={} honeytail \ --writekey=YOUR_API_KEY \ --dataset=PostgreSQL \ --parser=postgresql \ --file=/var/log/postgresql/postgresql-main.log \ --postgresql.log_line_prefix=YOUR_CONFIGURED_LOG_LINE_PREFIX \ --backfill ``` This command can be used at any point to backfill from archived log files. You can read more about `honeytail`'s backfill behavior [here](/send-data/logs/structured/honeytail/#backfilling-existing-data). `honeytail` does not unzip log files, so you will need to do this before backfilling. Once you have finished backfilling your old logs, we recommend transitioning to the default streaming behavior to stream live logs to Honeycomb. ## Scrub Personally Identifiable Information While we believe strongly in the value of being able to track down the precise query causing a problem, we understand the concerns of exporting log data, which may contain sensitive user information. With that in mind, we recommend using `honeytail`'s PostgreSQL parser, but adding a `--scrub_field=query` flag to hash the concrete `query` value. The `normalized_query` attribute will still be representative of the **shape** of the query, and identifying patterns including specific queries will still be possible—but the sensitive information will be completely obscured before leaving your servers. More information about dropping or scrubbing sensitive fields can be [found here](/send-data/logs/structured/honeytail/#dropping-or-scrubbing-fields). ## Open Source [Honeytail](https://github.com/honeycombio/honeytail) is open source and Apache 2.0 licensed. # Send Logs with the OpenTelemetry Collector Source: https://docs.honeycomb.io/send-data/logs/unstructured/collector Use the OpenTelemetry Collector to collect, parse, and structure unstructured logs from Fluentd, AWS, Azure, and other sources before sending them to Honeycomb. Collect and convert logs from different sources to OpenTelemetry's structured log format. The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) can collect logs from many different sources in common or custom formats. You can use a Collector as a logging agent, often as a drop-in replacement for other logging agents. This lets you process your logs, traces, and metrics in one place. ## Use a Receiver to Collect Logs The OpenTelemetry Collector supports a large number of [receivers](https://opentelemetry.io/docs/collector/configuration/#receivers) that can be used to collect logs from a variety of sources. ### Setup 1. Make sure you're using the [Collector Contrib distribution of the OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib), which contains contributions that are not part of the core repository and core distribution of the OpenTelemetry Collector. 2. Prepare your collector configuration file by adding the following boilerplate: ```yaml theme={} receivers: # Add your receiver here # ... processors: batch: # Add any additional processors here # ... exporters: otlp/logs: endpoint: "api.honeycomb.io:443" headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "YOUR_LOGS_DATASET_NAME" service: pipelines: logs: receivers: [receiver1,receiver2,etc] processors: [batch] exporters: [otlp/logs] ``` If you are sending OTLP logs from a service with a `service.name` defined, then the dataset for those logs will be the name of the service, and the `x-honeycomb-dataset` header will not be used. ## Collect any Log with the Filelog Receiver The [Filelog Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) supports reading and parsing any arbitrary log written to a file on a server. The Filelog Receiver is the most flexible receiver, but depending on the shape of your logs, it may require additional configuration to parse your logs correctly. For example, here is a configuration that reads an NGINX access log and parses it into a structured log: ```yaml theme={} receivers: filelog: include: ["/var/log/nginx/access.log"] operators: - type: "regex_parser" regex: "(?P[^ ]*) - - \\[(?P[^\\]]*)\\] \"(?P\\S+)(?: +(?P[^ ]*) +\\S*)?\" (?P\\d+) (?P\\d+) \"(?P[^\"]*)\" \"(?P[^\"]*)\"" timestamp: parse_from: attributes.time layout: "%d/%b/%Y:%H:%M:%S %z" ``` Here's a configuration that reads a JSON log: ```yaml theme={} receivers: filelog: include: [ /var/log/myservice/*.json ] operators: - type: json_parser timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' ``` ### Logs with Mixed Formats Sometimes, logs have a mix of structured and unstructured information, such as an info log with JSON inside. To parse these, you need to parse each piece of the log into a specific element, and also know how to handle the structured information. For example, let's say you have a log at `/var/log/name-service/0.log` that mixes text and JSON: ```txt theme={} 2024-03-19T14:49:51.998-0600 info name-service/main.go:212 Listening on http://localhost:8000/name {"name": "name-service", "function": "main", "featureflag.allow-future": true, "name-format": "lowercase"} ``` You can use a `filelogreceiver` configuration to do the following: 1. Read the log from `/var/log/name-service/*.log`. 2. Parse the full text of the log into a timestamp, severity, file name, message, and "details". 3. Parse the "details" text using a JSON parser, where each key in the JSON object is turned into an attribute for the log. 4. Parse the service name from the file path. 5. Sets the `service.name` resource with the parsed service name. The following `filelogreceiver` configuration completes the required tasks: ```yaml theme={} filelog: include: - /var/log/name-service/*.log include_file_path: true operators: # Parse the text. # Each capture group becomes an attribute on the log. # The original full string becomes the body. - type: regex_parser regex: (?P^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}-\d{4}) (?P\w+) (?P[\w\/\.\:\-]+) (?P[^\{]*) (?P

.*) timestamp: layout: "%Y-%m-%dT%H:%M:%S.%f%z" parse_from: attributes.timestamp severity: parse_from: attributes.severity # The "details" attribute is a json string. # This operator parses the json string to turn it into a map. # Honeycomb will flatten the map on ingest. - type: json_parser parse_from: attributes.details parse_to: attributes.details # We can extract the service name, name-service, from the file path - type: regex_parser regex: \/var\/log\/(?P.*)\/\d+\.log parse_from: attributes["log.file.path"] # Move the extracted attribute to the service name resource attribute - type: move from: attributes.servicename to: resource["service.name"] ``` The configuration produces a structured log that, when exported to Honeycomb, contains the following fields: ```json theme={} { "body": "2024-03-19T14:49:51.998-0600 info name-service/main.go:212 Listening on http://localhost:8000/name {\"name\": \"name-service\", \"function\": \"main\", \"featureflag.allow-future\": true, \"name-format\": \"lowercase\"}", "severity": "info", "severity_code": 9, "service.name": "name-service", "message": "Listening on http://localhost:8000/name", "timestamp": 1679265391998, "details.featureflag.allow-future": true, "details.function": "main", "details.name": "name-service", "details.name-format": "lowercase", "log.file.name": "0.log", "log.file.path": "/var/log/name-service/0.log", "flags": 0, } ``` Use the [Explore Data](/reference/honeycomb-ui/query/query-results/#explore-data) tab in Query Results to view the structured log in Honeycomb. ## Log Sources You can configure many different receivers to collect logs from a specific source. ### AWS Cloudwatch The [AWS Cloudwatch Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver) supports autodiscovery of log groups and log streams in AWS Cloudwatch, with optional filtering of those sources. For example, here is a configuration that autodiscovers only EKS logs from `us-west-1`: ```yaml theme={} receivers: awscloudwatch/: region: us-west-1 logs: poll_interval: 1m groups: autodiscover: limit: 100 prefix: /aws/eks/ ``` ### Azure Blob The [Azure Blob Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azureblobreceiver) reads logs and trace data from Azure Blob Storage. For example, here is a configuration that reads logs from a specific container in Azure Blob Storage: ```yaml theme={} receivers: azureblob: connection_string: DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=;EndpointSuffix=core.windows.net event_hub: endpoint: Endpoint=sb://oteldata.servicebus.windows.net/;SharedAccessKeyName=otelhubbpollicy;SharedAccessKey=;EntityPath=otellhub logs: container_name: name-of-container ``` ### Azure Event Hub The [Azure Event Hub Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/azureeventhubreceiver) pulls logs from an Azure Event Hub and transforms them. For example, here is a configuration that reads logs from a specific parition and group, then structures them a structured JSON log: ```yaml theme={} receivers: azureeventhub: connection: Endpoint=;SharedAccessKeyName=;SharedAccessKey=;EntityPath=hubName partition: my-partition group: my-consumer-group ``` ### Cloudflare The [Cloudflare Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/cloudflarereceiver) accepts logs from CloudFlare's [LogPush Jobs](https://developers.cloudflare.com/logs/about/). For example, here is a configuration that reads logs from a specific LogPush Job: ```yaml theme={} receivers: cloudflare: logs: tls: key_file: some_key_file cert_file: some_cert_file endpoint: secret: timestamp_field: EdgeStartTimestamp attributes: ClientIP: http_request.client_ip ClientRequestURI: http_request.uri ``` ### Fluent Forward The [Fluent Forward Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver) runs a TCP server that accepts logs via the [Fluent Forward protocol](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1), which enables collecting logs from Fluentbit and Fluentd. For example, here is a configuration that reads all logs on port 8006: ```yaml theme={} receivers: fluentforward: endpoint: 0.0.0.0:8006 ``` ### Google Pubsub The [Google Pubsub Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/googlecloudpubsubreceiver) reads logs from a Google Pubsub subscription. For example, here is a configuration that reads raw text logs and wraps them into an OpenTelemetry Log: ```yaml theme={} receivers: googlecloudpubsub: project: otel-project subscription: projects/otel-project/subscriptions/otlp-logs encoding: raw_text ``` ### Journald The [Journald Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver) parses Journald events from the systemd. For example, here is a configuration that reads all logs from Journald from some specific units: ```yaml theme={} receivers: journald: directory: /run/log/journal units: - ssh - kubelet - docker - containerd priority: info ``` ### Kafka The [Kafka Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkareceiver) reads logs, metrics, and traces from Kafka. For example, here is a configuration that reads all Kafka data: ```yaml theme={} receivers: kafka: protocol_version: 2.0.0 ``` ### Kubernetes The OpenTelemetry Collector has several receivers that can be used to collect logs from Kubernetes. To learn more, visit [Kubernetes Log Collection](/send-data/kubernetes/send-logs-filelog-receiver/) and [Kubernetes Event Collection](/send-data/kubernetes/send-events-objects-receiver/). ### Loki The [Loki Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/lokireceiver) allows Promtail instances to send logs to the OpenTelemetry Collector. For example, here is a configuration that reads all logs from an endpoint: ```yaml theme={} receivers: loki: protocols: http: endpoint: 0.0.0.0:3500 grpc: endpoint: 0.0.0.0:3600 use_incoming_timestamp: true ``` ### MongoDB Atlas The [MongoDB Atlas Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mongodbatlasreceiver) reads logs from MongoDB Atlas. For example, here is a configuration that reads all logs from a specific project: ```yaml theme={} receivers: mongodbatlas: logs: enabled: true projects: - name: "project 1" collect_audit_logs: true collect_host_logs: true ``` ### OTLPJson File The [OTLPJson File Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/otlpjsonfilereceiver) reads any existing OTLP Logs, Metrics, or Traces from a file on a server. For example, here is a configuration that reads from a specific directory and excludes a specific file: ```yaml theme={} receivers: otlpjsonfile: include: - "/var/log/*.log" exclude: - "/var/log/example.log" ``` ### Apache Pulsar The [Pulsar Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/pulsarreceiver) collects logs, metrics, and traces from Apache Pulsar. For example, here is a configuration that reads data from a Pulsar cluster: ```yaml theme={} receivers: pulsar: endpoint: pulsar://localhost:6650 topic: otlp-spans subscription: otlp_spans_sub consumer_name: otlp_spans_sub_1 encoding: otlp_proto auth: tls: cert_file: cert.pem key_file: key.pem tls_allow_insecure_connection: false tls_trust_certs_file_path: ca.pem ``` ### SignalFx The [SignalFx Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/signalfxreceiver) reads logs from a SignalFx endpoint. For example, here is a configuration that reads data from a SignalFx endpoint: ```yaml theme={} receivers: signalfx: endpoint: 0.0.0.0:9943 signalfx/advanced: endpoint: 0.0.0.0:9943 access_token_passthrough: true tls: cert_file: /test.crt key_file: /test.key ``` ### Splunk HEC The [Splunk HEC Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver) accepts events in the Splunk HEC format. For example, here is a configuration that reads JSON HEC events and raw log data: ```yaml theme={} receivers: splunk_hec: endpoint: 0.0.0.0:8088 splunk_hec/advanced: endpoint: 0.0.0.0:8088 access_token_passthrough: true tls: cert_file: /test.crt key_file: /test.key raw_path: "/raw" hec_metadata_to_otel_attrs: source: "mysource" sourcetype: "mysourcetype" index: "myindex" host: "myhost" ``` ### Syslog The [Syslog Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/syslogreceiver) parses Syslogs received over UDP or TCP. For example, here is a configuration that reads Syslogs from TCP: ```yaml theme={} receivers: syslog: tcp: listen_address: "0.0.0.0:54526" protocol: rfc5424 ``` ### TCP The [TCP Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver) receives logs over TCP. For example, here is a configuration that reads logs from TCP over a particular address: ```yaml theme={} receivers: tcplog: listen_address: "0.0.0.0:54525" ``` ### UDP The [UDP Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/udplogreceiver) receives logs over UDP. For example, here is a configuration that reads logs from UDP over a particular address: ```yaml theme={} receivers: udplog: listen_address: "0.0.0.0:54525" ``` ### Webhook Event The [Webhook Event Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/webhookeventreceiver) allows for any webhook-style data source to send logs to the OpenTelemetry Collector. For example, here is a configuration that reads logs from a webhook: ```yaml theme={} receivers: webhookevent: endpoint: localhost:8088 read_timeout: "500ms" path: "eventsource/receiver" health_path: "eventreceiver/healthcheck" required_header: key: "required-header-key" value: "required-header-value" ``` ### Windows Log Event The [Windows Log Event Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/windowseventlogreceiver) tails and parses logs from the Windows event log API. For example, here is a configuration that reads logs from a named channel: ```yaml theme={} receivers: windowseventlog: channel: application ``` # Send Unstructured Logs with Honeytail + Custom Regex Source: https://docs.honeycomb.io/send-data/logs/unstructured/honeytail-regex Parse unstructured log files using custom regular expressions and Honeytail, then send the parsed events to Honeycomb for high-cardinality querying. Use custom regular expressions and Honeytail, our lightweight tool to tail your existing log files, parse the content, and send it to Honeycomb. ## Installation Download and install the latest `honeytail` by running: Download the `honeytail_1.10.0_amd64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_amd64.deb ``` Verify the package. ```shell theme={} echo '3db441215f97eaed068aa0531c986cf5405957e3e8e26b22c16b571091caf917 honeytail_1.10.0_amd64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_amd64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0_arm64.deb` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0_arm64.deb ``` Verify the package. ```shell theme={} echo '4220756e5a941cde6a484cb4cfde184eb189aaf29170df301a874eb143e960ed honeytail_1.10.0_arm64.deb' | sha256sum -c ``` Install the package. ```shell theme={} sudo dpkg -i honeytail_1.10.0_arm64.deb ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the `honeytail_1.10.0-1.x86_64.rpm` package. ```shell theme={} wget -q https://honeycomb.io/download/honeytail/v1.10.0/honeytail_1.10.0-1.x86_64.rpm ``` Verify the package. ```shell theme={} echo 'b23215a9301b20b2e2262a0823c9e761e8b57e1a62fd5cec35f697fce41fa863 honeytail_1.10.0-1.x86_64.rpm' | sha256sum -c ``` Install the package. ```shell theme={} sudo rpm -i honeytail_1.10.0-1.x86_64.rpm ``` The packages install `honeytail`, its config file `/etc/honeytail/honeytail.conf`, and some start scripts. Build `honeytail` from source if you need it in an unpackaged form or for ad-hoc use. Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-amd64 ``` Verify the binary. ```shell theme={} echo 'c9cc7dd1aa2b12afeb30b089061870f3407d2df0119e7c2807fec648b603e2d5 honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-linux-arm64 ``` Verify the binary. ```shell theme={} echo '1dd37227788548c4ed44592554e3c90e374c4d796c444dde9f372db8618bc7fa honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Download the 1.10.0 binary. ```shell theme={} wget -q -O honeytail https://honeycomb.io/download/honeytail/v1.10.0/honeytail-darwin-amd64 ``` Verify the binary. ```shell theme={} echo '9a3da0f48fe21b1e610ac6b63130dfb8118a9a0ec16abae13350edba02d85e4d honeytail' | shasum -a 256 -c ``` Set the permissions to allow execution. ```shell theme={} chmod 755 ./honeytail ``` Clone the [Honeytail](https://github.com/honeycombio/honeytail) repository. ```shell theme={} git clone https://github.com/honeycombio/honeytail ``` Install from source. ```shell theme={} cd honeytail; go install ``` You should modify the config file and uncomment and set: * `WriteKey` to your API key, available from [the account page](https://ui.honeycomb.io/account) * `LogFiles` to the path for the log file you want to ingest, or `-` for stdin * `Dataset` to the name of the dataset you wish to create with this log file. * `ParserName` to `regex` * `LineRegex` to a regular expression with named capture groups. ## Launch the Agent Start up a `honeytail` process using `upstart` or `systemd` or by launching the process by hand. This will tail the log file specified in the config and leave the process running as a daemon. ```shell theme={} sudo initctl start honeytail ``` ```shell theme={} sudo systemctl start honeytail ``` ```shell theme={} honeytail -c /etc/honeytail/honeytail.conf ``` ## Backfilling Archived Logs To backfill existing data, run `honeytail` with `--backfill` the first time: ```shell theme={} honeytail -c /etc/honeytail/honeytail.conf \ --file /var/log/myapp/log12.log \ --backfill ``` This command can also be used at any point to backfill from older, rotated log files. You can read more about our [backfill behavior here](/send-data/logs/structured/honeytail/). If you have chosen to backfill from old logs, do not forget to transition into the default streaming behavior to stream live logs to Honeycomb! ## Regexes We use golang's [regexp](https://golang.org/pkg/regexp/) package, which uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax). ### Specifying Regexes Command line: use the **`--regex.line_regex`** flag to tell `honeytail` how to extract data from a log line. You must provide at least one regex. You may optionally specify multiple regexes. Lines will be parsed by the first regex to find a match. Precedence is based on the order you pass in `line_regex`, so specify your regexes from most-specific to least-specific. On the command line, you will need to wrap the regex in quotes. ```shell theme={} honeytail \ --writekey YOUR_API_KEY \ --file PATH/FILE.LOG \ --parser regex \ --dataset "MY_TEST_SET" \ --backfill \ --regex.line_regex="\[(?P\d{2}:\d{2}:\d{2})\] (?P\w+)" \ --regex.line_regex="(?P\w+) (?P\w+)" ``` Equivalent configuration file specification. Note that you should not wrap the regex in quotes here. ```toml theme={} [Regex Parser Options] ; a regular expression with named capture groups representing the fields you want parsed LineRegex = \[(?P\d{2}:\d{2}:\d{2})\] (?P\w+) LineRegex = (?P\w+) (?P\w+) ``` ### Regex Syntax Regexes must contain at least one named capture group. Use the `(?Pre)` syntax for named groups. Example: Log file ```log theme={} [2017/11/07 22:59:46] 200 ... [2017/11/07 22:59:48] 500 ... [2017/11/07 23:01:02] 404 ... ``` with ```bash theme={} --regex.line_regex="\[(?P\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\] (?P\d+)" ``` will yield rows like this: ```json theme={} { time: "2017/11/07 22:59:46", status: "200" } ``` ### Nested Regex Grouping Nested groups are supported. For example: ```bash theme={} --regex.line_regex="(?P[^ ]* (?P[^ ]*) (?P[^ ]*))" ``` will parse a log line "`A B C`" into `{ outer: "A B C", inner1: "B", inner2: "C" }`. ## Timestamp Parsing Honeycomb expects all events to contain a timestamp field; if one is not provided, the server will associate the current time of ingest with the given payload. Use the `--regex.timefield` and `--regex.time_format` flags to help `honeytail` understand where and how to extract the event's timestamp. For example, given a log file like the following: ```log theme={} [08/Oct/2015:00:26:26 +0000] 200 174 0.099 ``` A command to consume those log lines (retaining the `"local_time"` field as the event's timestamp would look like): ```shell theme={} honeytail \ --parser=regex \ --writekey=YOUR_API_KEY \ --file=server.log \ --dataset='MY_DATASET' \ --backfill \ --regex.line_regex=SOME_REGEX \ --regex.timefield="local_time" \ --regex.time_format="%d/%b/%Y:%H:%M:%S %z" ``` The `--regex.timefield="local_time"` argument tells `honeytail` to consider the `"local_time"` value to be the canonical timestamp for the events in the specified file. The `--regex.time_format` argument specifies the timestamp format to be used while parsing. (It understands common [`strftime`](https://www.strfti.me/) formats.) # Send Metrics to Honeycomb Source: https://docs.honeycomb.io/send-data/metrics Send application and system metrics to Honeycomb and query them alongside your traces and logs. Metrics are well suited to giving a picture of the general health of a host or resource. Honeycomb can consume and display metrics in addition to tracing and telemetry data. ## Sending metrics to Honeycomb Honeycomb supports [OpenTelemetry metric types](https://opentelemetry.io/docs/concepts/signals/metrics/), including gauges, delta and cumulative sums, and histograms. There are several ways to send metric data to Honeycomb. In general, we recommend that you send metric data to its own dataset. * Send [host metrics to Honeycomb with an OpenTelemetry Collector](/send-data/metrics/system/) running as an agent on each host * Send metrics from a [Prometheus Client](/integrations/metrics/prometheus/) * Send AWS metrics from [AWS CloudWatch metric streams](/integrations/metrics/aws-cloudwatch/) * An [OpenTelemetry SDK](/send-data/metrics/application/) can send metrics from within an application ## Querying metrics Querying metrics is just like querying any other data in Honeycomb. Metric resources and attributes are stored as fields and can be used in **WHERE** and **GROUP BY** clauses to plot specific timeseries. You can also use [temporal aggregation functions](/investigate/query/temporal-aggregation) in your queries such as `RATE()`, `INCREASE()`, `SUMMARIZE()`, and `LAST()`. Learn more about [writing queries for metrics data](/investigate/query/examples-metrics/). ### Making metrics graphs look less spiky Because metric data arrives at known, regular intervals, you may find that your metrics graphs have a spiky appearance where after each metric, the line graph drops to zero before plotting the next point. Some people prefer a continuous line graph, without the drops to zero, because it is easier to identify trends in the data. There are two ways to achieve this continuous line graph: 1. Modify the Granularity using the time picker in Query Builder. Alternatively, set a [default granularity](/reference/honeycomb-ui/manage-data/datasets/dataset-settings/#default-granularity) for the dataset. 2. Enable the **Omit Missing Values** graph setting on the query. To do this: 1. On the right side of Query Results, select the Settings icon () to display the **Display Settings**. 2. Select **Omit Missing Values**. 3. The graph refreshes and should display a continuous line. ## Correlating query results and metrics data Use the [Correlations tab](/investigate/analyze/correlate/) in Query Builder and compare query results with your metrics data. Looking at your infrastructure metrics alongside query results can give you a better insight into your system. For example, a system running out of memory, CPU, or network resources might be the reason for an [out-of-compliance SLO](/notify/slos/) or an [alerting trigger](/notify/triggers/), and seeing the graph of the problem alongside graphs of relevant system resources could confirm or deny this kind of hypothesis. ## Adjusting metrics event volume Honeycomb stores metrics in events, and it combines as many metrics into a single event as possible. Every captured data point is stored as a value in an event whose timestamp represents the time of capture. All attributes that uniquely identify the metric's timeseries are included on the event. Events that contain metrics are most effective (and cost-effective) when many data points for the same resource, same attributes, and same timestamp arrive together. Learn about factors that affect [metrics event volume and how to manage it](/manage-data-volume/adjust-granularity/metrics-events/). ## Integrations Honeycomb provides the following integrations for sending metrics: Stream metrics stored in AWS CloudWatch to Honeycomb using AWS Kinesis Data Firehose. Configure HashiCorp Consul to send cluster metrics to Honeycomb with an OpenTelemetry Collector. Configure HashiCorp Nomad to send metrics to Honeycomb with an OpenTelemetry Collector. Configure HashiCorp Vault to send server metrics to Honeycomb with an OpenTelemetry Collector. Scrape Prometheus client metrics with an OpenTelemetry Collector and send them to Honeycomb. # Send Application Metrics to Honeycomb Source: https://docs.honeycomb.io/send-data/metrics/application Instrument your application with OpenTelemetry SDKs to create counters, histograms, and gauges, and send metrics to Honeycomb for analysis. ## How It Works OpenTelemetry SDKs are implemented and stable for several languages. They provide an API, which allows your application to create custom metrics with **metrics instruments** like counters, histograms, and gauges. Instrumentation libraries used in different languages can also capture metrics from the application's runtime environment, such as the number of HTTP requests served by an HTTP server. OpenTelemetry SDKs can then be configured to send metrics to Honeycomb over OpenTelemetry Protocol (OTLP). ```mermaid actions={false} theme={} flowchart LR subgraph S["Instrumented App"] A("OpenTelemetry SDK") end A -->|OTLP| B("Honeycomb") ``` ## Instrumenting Your Application Refer to the [OpenTelemetry SDK Documentation for your environment](https://opentelemetry.io/status/) for details on how to instrument your application. You will need to enable an OTLP exporter and it should have: * a destination of: * for our US instance: `api.honeycomb.io:443` * for our EU instance: `api.eu1.honeycomb.io:443` * an `x-honeycomb-team` header with an API key that has write permissions With the current metrics experience, Honeycomb routes all of your metrics to a managed metrics dataset (named `Metrics`) automatically, so you do not need to specify a dataset. The `x-honeycomb-dataset` header applied to the legacy metrics experience, where it named the dataset metrics were sent to. It is no longer used to route metrics and can be omitted. ## Troubleshooting Metrics ### Metrics Sent to `unknown_metrics` `unknown_metrics` was the default destination in the legacy metrics experience when no dataset was specified with the `x-honeycomb-dataset` header. With the current metrics experience, metrics are routed to the managed `Metrics` dataset automatically, so this no longer applies. This troubleshooting scenario is not applicable to [Honeycomb Classic users](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments). # Send System Metrics with the OpenTelemetry Collector Source: https://docs.honeycomb.io/send-data/metrics/system Deploy the OpenTelemetry Collector as a host agent to collect system metrics on a regular interval and send them to Honeycomb for analysis. Get host metrics on a regular interval and send them to Honeycomb with an OpenTelemetry Collector. You can deploy an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) as an agent in your host environment to collect and export diagnostic information about the system. For example, you can get samples of CPU, memory, disk, and network utilization so this data can be queried and visualized in Honeycomb. Follow this guide to learn how to: * Deploy an OpenTelemetry Collector to collect host metrics. * Configure your Collector using our configuration as a starting point. ## Install the Collector Agent Install the latest [OpenTelemetry Collector `otelcol-contrib` binary](https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest) on your host system. If you have not installed a Collector before or need a refresher, follow the instructions in [Install the Collector](https://opentelemetry.io/docs/collector/installation/) on the OpenTelemetry documentation. Download our [`honeycomb-metrics-config.yaml` file](https://github.com/honeycombio/opentelemetry-collector-configs/releases/latest) and use that to configure your Collector. ## Set Environment Variables Set the following environment variables before running the Collector: * `HNY_API_KEY`: Your [API key](/configure/environments/manage-api-keys/). Replace `` with this value in the example below. * `HNY_DATASET`: The name of the dataset you want to send these metrics to. Replace `` with this value in the following example. * `OTEL_RESOURCE_ATTRIBUTES`: These attributes are added to each data point. Replace `hellohostmetrics` with the name of your service and `development` with the name of your environment in the following example. ```bash theme={} export HNY_API_KEY= \ export HNY_DATASET= \ export OTEL_RESOURCE_ATTRIBUTES='service.name=hellohostmetrics,deployment.environment=development' ``` ## Run the Collector Agent After setting your environment variables, run the Collector binary with the following command. ```bash theme={} ./otelcol_hny_linux_amd64 --config ./honeycomb-metrics-config.yaml ``` A process should start and your new Collector agent will send host metrics to your dataset every 10 seconds. ## Customize Your Agent Configuration Our host metrics Collector configuration (`honeycomb-metrics-config.yaml`) keeps your event volume low and gives you as much data as possible in each event. Use it as the basis for your own configuration, making changes to better fit your observability needs and host environment. ### Resource Attributes You can add custom resource attributes to all your data points by changing the [`OTEL_RESOURCE_ATTRIBUTES` environment variable](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md#specifying-resource-information-via-an-environment-variable). Attributes are separated by the comma character (`,`). #### Resource Detection Processor If your host is deployed using Docker, Google Cloud, Google Kubernetes Engine, Amazon EC2, Amazon ECS, Amazon Elastic Beanstalk, Amazon EKS, or Azure, you can use the [resource detection processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#readme) to automatically add metadata about the host to every metric data point. Enable the resource detection processor for a host by adding a value from the list below to `processors.resourcedetection.detectors`: * Docker: `docker` * Google Cloud: `gce` * EC2: `ec2` * ECS: `ecs` * Azure: `azure` ```yaml theme={} processors: resourcedetection/docker: detectors: [env, docker] timeout: 2s override: false ``` ### Metrics Collection Interval Change how often host metrics are collected by setting `receivers.hostmetrics.collection_interval`. ```yaml theme={} receivers: hostmetrics: collection_interval: 10s ``` [Host Metrics Receiver on GitHub](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver#readme). ### Disable a Metrics Scraper You can also disable specific scrapers by removing them from the list in `receivers.hostmetrics.scrapers`. ```yaml theme={} receivers: hostmetrics: collection_interval: 30s scrapers: cpu: memory: # disk: ``` [Host Metrics Receiver on GitHub](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver#readme). ### Debug Output Verbosity Change debug logging **verbosity** by setting `exporters.debug.verbosity` to either `detailed`, `normal`, or `basic`. ```yaml theme={} exporters: debug: verbosity: detailed ``` [Debug Exporter on Github](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter#readme). ### Using a Custom Build of OpenTelemetry Collector It is possible to use a different build of OpenTelemetry Collector to send host metrics to Honeycomb. If you do this, we still recommend using [our `honeycomb-metrics-config.yaml` file](https://github.com/honeycombio/opentelemetry-collector-configs/releases/latest) as a basis for your configuration, since it is designed to keep your event volume low while providing as much data as possible in each event. Refer more about [managing your metrics data volume](/manage-data-volume/adjust-granularity/metrics-events/), and [about using your own Collector with this custom configuration](https://opentelemetry.io/docs/collector/custom-collector/). # Send Data from Applications Source: https://docs.honeycomb.io/send-data/opentelemetry Instrument your applications with OpenTelemetry, the open-source standard for collecting telemetry, and send traces, logs, and metrics to Honeycomb. Honeycomb supports [OpenTelemetry](https://opentelemetry.io/). OpenTelemetry is an open-source framework for collecting and managing telemetry data from distributed systems. It provides a standardized way of instrumenting code to generate telemetry data, such as traces, metrics, and logs, and exporting that data to various backends for analysis and visualization. OpenTelemetry supports multiple programming languages and platforms, making it easy to integrate with a variety of systems. OpenTelemetry comprises two main components: instrumentation libraries and exporters. The instrumentation libraries enable developers to collect telemetry data from their applications, while the exporters allow users to send the data to backends like Honeycomb. For more structured learning, check out the [OpenTelemetry Foundations](https://academy.honeycomb.io/app/courses/3c8c4ded-ce8c-4df1-8150-b73fdf59f2b0) course from Honeycomb Academy. If instrumenting code for the first time, we recommend using OpenTelemetry. If your application is already instrumented with OpenTelemetry, you can send OpenTelemetry Protocol (OTLP) data directly to Honeycomb. To learn more, read [Using the Honeycomb OpenTelemetry Endpoint](#using-the-honeycomb-opentelemetry-endpoint). Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If a system is already instrumented with legacy instrumentation like OpenTracing, Zipkin, or Jaeger, you should use an [OpenTelemetry Collector](/send-data/opentelemetry/collector/) to convert into OTLP and export your data to Honeycomb. ## Instrumenting with OpenTelemetry Select your application's language for instructions on instrumenting with OpenTelemetry: {"Node.js"} } /> {"Go"} } /> {"Python"} } /> {"Ruby"} } /> {"Java"} } /> {".NET"} } /> {"JavaScript"} } /> {"Android"} } /> {"iOS"} } /> {"React Native"} } /> ## Using the Honeycomb OpenTelemetry Endpoint If your language is not listed above, you can send data to Honeycomb directly over OTLP with an OpenTelemetry SDK or the OpenTelemetry Collector. For Rust, we recommend that you use the [`opentelemetry`](https://crates.io/crates/opentelemetry) and [`opentelemetry-otlp`](https://crates.io/crates/opentelemetry-otlp) crates to send data over OTLP to Honeycomb. Follow the instructions below for the required configuration values. An OpenTelemetry integration should set the endpoint and headers. 1. Set the endpoint: * For OTLP/gRPC: * US instance: `export OTEL_EXPORTER_OTLP_ENDPOINT=api.honeycomb.io:443` * EU instance: `export OTEL_EXPORTER_OTLP_ENDPOINT=api.eu1.honeycomb.io:443` * For OTLP/HTTPS: * US instance: `export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io/` * EU instance: `export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.eu1.honeycomb.io/` 2. Set your API key: ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key" ``` 3. Set the service name in the `OTEL_SERVICE_NAME` environment variable: ```shell theme={} export OTEL_SERVICE_NAME=your-service-name ``` **Note**: If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), set the `x-honeycomb-dataset` in the HTTP Headers. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset-name" ``` 1. Set the endpoint in SDK initialization code: * For OTLP/gRPC: * US instance: `api.honeycomb.io:443` * EU instance: `api.eu1.honeycomb.io:443` * For OTLP/HTTPS: * US instance: * `https://api.honeycomb.io/v1/traces` for traces * `https://api.honeycomb.io/v1/metrics` for metrics * `https://api.honeycomb.io/v1/logs` for logs * EU instance: * `https://api.eu1.honeycomb.io/v1/traces` for traces * `https://api.eu1.honeycomb.io/v1/metrics` for metrics * `https://api.eu1.honeycomb.io/v1/logs` for logs 2. For HTTP requests, define the API key in the HTTP headers. For example, in Rust, it may look like this: ```rust theme={} //... const API_KEY: &str = "your-api-key-here"; //... .with_headers(HashMap::from([ ("x-honeycomb-team".into(), API_KEY.into()) //... ])) ``` 3. Set the service name in the `OTEL_SERVICE_NAME` environment variable: ```shell theme={} export OTEL_SERVICE_NAME=your-service-name ``` **Note**: If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), set the `x-honeycomb-dataset` in the HTTP Headers. ## About the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable When using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable with an SDK and an HTTP exporter, the final path of the endpoint is modified by the SDK to represent the specific signal being sent. For example, when exporting trace data, the endpoint is updated to append `v1/traces`. When exporting metrics data, the endpoint is updated to append `v1/metrics`. So, if you were to set the `OTEL_EXPORTER_OTLP_ENDPOINT` to `https://api.honeycomb.io`, traces would be sent to `https://api.honeycomb.io/v1/traces` and metrics would be sent to `https://api.honeycomb.io/v1/metrics`. The same modification is not necessary for gRPC. ```shell theme={} export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.eu1.honeycomb.io # EU instance ``` If the desired outcome is to send data to a different endpoint depending on the signal, use `OTEL_EXPORTER_OTLP__ENDPOINT` instead of the more generic `OTEL_EXPORTER_OTLP_ENDPOINT`. When using a signal-specific environment variable, these paths must be appended manually. Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` for traces, appending the endpoint with `v1/traces`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` for metrics, appending the endpoint with `v1/metrics`. Send both traces and metrics to Honeycomb using this method by setting the following variables: ```shell theme={} export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces # US instance #export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.eu1.honeycomb.io/v1/traces # EU instance export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics # US instance #export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.eu1.honeycomb.io/v1/metrics # EU instance ``` More details about endpoints and signals can be found in the [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md). ## Instrumented Fields With its Automatic Instrumentation, OpenTelemetry automatically tracks trace and span relationships using the following fields: | Field | Description | | ----------------- | -------------------------------------------------------------------------------------------------- | | `name` | Name of the function or method where the span was created | | `service.name` | Name of the instrumented service | | `duration_ms` | Length of time the span took, in milliseconds | | `trace.span_id` | Unique identifier for each span | | `trace.trace_id` | Unique identifier of the trace this span belongs to | | `trace.parent_id` | Unique identifier of this span's parent span, which is the call location this span was called from | # Send Data with the OpenTelemetry Collector Source: https://docs.honeycomb.io/send-data/opentelemetry/collector Deploy the OpenTelemetry Collector as a vendor-agnostic pipeline to gather traces, logs, and metrics from your instrumented applications and send them to Honeycomb. Use the OpenTelemetry Collector to receive, process, and export telemetry from your infrastructure to Honeycomb. For enterprise-scale pipeline management, visit [Honeycomb Telemetry Pipeline](/send-data/telemetry-pipeline/). The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) offers a vendor-agnostic way to gather observability data from a variety of instrumentation solutions and send that data to Honeycomb. Applications instrumented with OpenTelemetry SDKs or with Jaeger or Zipkin can use the OpenTelemetry Collector to send trace data to Honeycomb as events. Additionally, applications instrumented with OpenTelemetry SDKs or with metrics data from Prometheus, StatsD, Influx, and others can use the OpenTelemetry Collector to send metrics data to Honeycomb. A Collector can be used with any number of applications, and by default all OpenTelemetry SDK's export telemetry to the Collector OTLP endpoint. Using a Collector allows for experimenting and setting up new instrumentation, as well as exporting to multiple backends that may include Honeycomb or even another Collector. A Collector can also help by offloading data and processing telemetry before it is sent on, using various processors for batching, transforming, and filtering data. Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. This means you can use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) and its standard OTLP exporter to send data to Honeycomb without any additional exporters or plugins. To send trace data to Honeycomb, configure an OTLP exporter with the Honeycomb API Key as a header, and include the exporter in the relevant pipeline: ```yaml theme={} exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" ``` ## Basic Configuration The Collector consists of three components: [receivers](https://opentelemetry.io/docs/collector/configuration/#receivers), [processors](https://opentelemetry.io/docs/collector/configuration/#processors), and [exporters](https://opentelemetry.io/docs/collector/configuration/#exporters), which are then used to construct telemetry pipelines. To configure the components for the Collector, create a file called `otel-collector-config.yaml`. ### Receivers Add receivers for OTLP trace output from applications instrumented with OpenTelemetry: ```yaml theme={} receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 ``` ### Processors Include a batch processor to batch telemetry before sending to Honeycomb: ```yaml theme={} processors: batch: ``` ### Exporters Configure an OTLP exporter with the Honeycomb API Key as a header: ```yaml theme={} exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" ``` ### Service Pipelines Finally, configure a service pipeline for traces: ```yaml theme={} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp_http] ``` ### Full Configuration File For Basic Usage The final result of adding receivers, processors, exporters, and pipelines will look like this: ```yaml theme={} # otel-collector-config.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp_http] ``` If you are a [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) user, the Dataset also must be specified using the `x-honeycomb-dataset` header in the exporters section, in the line below the `x-honeycomb-team`. A **Dataset** is a bucket where data gets stored in Honeycomb. ```yaml theme={} exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "YOUR_DATASET" ``` ## Metrics and Logs Signals The Collector can handle OTLP [metrics](/send-data/metrics/) and [logs](/send-data/logs/) signals in addition to traces. Metrics require a dataset in the exporter. Logs will use the `service.name` associated with the trace from which it originated, or a dataset if provided. It is possible to send these to the same or to multiple places in Honeycomb. To set a different pipeline with the same type of exporter, append the exporter with `/my-pipeline`. The following example exports OTLP application metrics to a specified dataset, and exports logs to the same place as traces. ```yaml theme={} receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" otlp_http/metrics: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "YOUR_METRICS_DATASET" service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp_http] metrics: receivers: [otlp] processors: [batch] exporters: [otlp_http/metrics] logs: receivers: [otlp] processors: [batch] exporters: [otlp_http] ``` ## Browser Telemetry To receive browser telemetry, the Collector requires an enabled OTLP/HTTP receiver. The `allowed_origins` property is required to enable [Cross-Origin Resource Sharing](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) (CORS) from the browser to the collector. The Collector will need to be accessible by the browser. It is recommended to put an external load balancer in front of the collector, which will also need to be configured to accept requests from the browser origin. You can also configure the Collector to [symbolicate your JavaScript stack traces](/send-data/opentelemetry/collector/symbolication/). This translates obfuscated or minified names from your stack traces into human-readable formats. In the example below, the configuration allows for the OpenTelemetry Collector to accept browser OpenTelemetry tracing, and is required to get data from the browser to Honeycomb. ```yaml theme={} receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 cors: allowed_origins: - "http://*..com" - "https://*..com" processors: batch: exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp_http] ``` More configuration options can be found on the [Collector Github Repository](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#server-configuration). If you are a [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) user, the Dataset also must be specified using the `x-honeycomb-dataset` header in the exporters section, in the line below the `x-honeycomb-team`. A **Dataset** is a bucket where data gets stored in Honeycomb. ```yaml theme={} exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "YOUR_DATASET" ``` Find out more about setting up [browser instrumentation](/send-data/javascript-browser/). ## Symbolicate Web, iOS, Android, and React Native stack traces Honeycomb provides several [Collector processors for symbolicating stack traces](https://github.com/honeycombio/opentelemetry-collector-symbolicator/) from your browser, iOS, Android, and React Native applications. This gives you human-readable errors and stack traces in your frontend launchpad's [Errors UI](/observe/frontend-errors/). * [Sourcemap symbolication](https://github.com/honeycombio/opentelemetry-collector-symbolicator/tree/main#javascript-source-maps) * [dSym symbolication](https://github.com/honeycombio/opentelemetry-collector-symbolicator/tree/main#dsym-symbolication) * [Proguard symbolication](https://github.com/honeycombio/opentelemetry-collector-symbolicator/tree/main#proguard-symbolication) For tutorials on setting up these processors, check out: * [Symbolicate Android Stack Traces with the OpenTelemetry Collector](/send-data/android/symbolicate/) * [Symbolicate iOS Stack Traces with the OpenTelemetry Collector](/send-data/ios/symbolicate/) * [Symbolicate JavaScript Stack Traces with the OpenTelemetry Collector](/send-data/javascript-browser/symbolicate/) ## Advanced Configuration The Collector can receive telemetry from different applications in different formats, and also export to multiple backends. Image of otel collector from opentelemetry.io, showing receivers, extensions, processors, and exporters

Image of otel collector from opentelemetry.io, showing receivers, extensions, processors, and exporters

The following is a complete configuration file example for a Collector instance that accepts Jaeger and OpenTelemetry (over gRPC and HTTP) trace data, as well as Prometheus metrics. This example includes the batch processor and several extensions, and exports both trace and metric data to Honeycomb: ```yaml theme={} receivers: jaeger: protocols: thrift_http: endpoint: "0.0.0.0:14268" otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus: config: scrape_configs: - job_name: "prometheus" scrape_interval: 15s static_configs: - targets: ["0.0.0.0:9100"] processors: batch: exporters: otlp_http: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" otlp_http/metrics: endpoint: "https://api.honeycomb.io:443" # US instance #endpoint: "https://api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" "x-honeycomb-dataset": "YOUR_METRICS_DATASET" extensions: health_check: endpoint: 0.0.0.0:13133 pprof: zpages: service: extensions: [health_check, pprof, zpages] pipelines: traces: receivers: [jaeger, otlp] processors: [batch] exporters: [otlp_http] metrics: receivers: [prometheus] processors: [] exporters: [otlp_http/metrics] ``` See the [Collector documentation](https://opentelemetry.io/docs/collector/configuration/) for more configuration options. ## Using gRPC Instead of HTTP/protobuf The `otlp_http` exporter uses the HTTP/protobuf protocol to send telemetry data. To use gRPC instead of HTTP/protobuf, use the `otlp_grpc` exporter, update the endpoint, and update the pipeline: ```yaml theme={} receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 exporters: otlp_grpc: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" service: pipelines: traces: receivers: [otlp] processors: [] exporters: [otlp_grpc] ``` ## Filtering Span Events and Other Data To filter span events (or any other data) using the `filterprocessor`, use version 0.66.0 or higher of the OpenTelemetry Collector Contrib distribution. The `filterprocessor` component can be especially helpful if using instrumentations that create a lot of noisy, unneeded data. To use, add the `filterprocessor` component as a processor in your OpenTelemetry Collector configuration file, such as `otel-collector-config.yaml`. Configure your OpenTelemetry Collector to filter Span Events, similar to: ```yaml theme={} processors: filter: # tell it to operate on span data traces: # Filter out span events with the 'grpc' attribute, # or have a span event name with 'grpc' in it. spanevent: - 'attributes["grpc"] == true' - 'IsMatch(name, ".*grpc.*") == true' ``` Each individual line in the `spanevent` list is a separate filter. If any of the filters match, the span event will be filtered out. In other words, each line is an `OR` condition. Additionally, Honeycomb currently translates all fields with the `instrumentation_scope.name` field into `library.name`. To filter based on the value of an instrumentation scope, use `instrumentation_scope.name` instead of `library.name`. To require multiple conditions to be true, write a single filter that combines them, similar to: ```yaml theme={} processors: filter: traces: # Filter out only span events with both the 'grpc' attribute and # that have a span event name with 'grpc' in it. spanevent: - 'attributes["grpc"] == true and IsMatch(name, ".*grpc.*") == true' ``` You can filter any OpenTelemetry signals, not only Span Events. This example filters data across spans, metrics, and logs: ```yaml theme={} processors: filter: traces: span: - 'attributes["container.name"] == "app_container_1"' - 'resource.attributes["host.name"] == "localhost"' - 'name == "app_3"' spanevent: - 'attributes["grpc"] == true' - 'IsMatch(name, ".*grpc.*") == true' metrics: metric: - 'name == "my.metric" and attributes["my_label"] == "abc123"' - 'type == METRIC_DATA_TYPE_HISTOGRAM' datapoint: - 'metric.type == METRIC_DATA_TYPE_SUMMARY' - 'resource.attributes["service.name"] == "my_service_name"' logs: log_record: - 'IsMatch(body, ".*password.*") == true' - 'severity_number < SEVERITY_NUMBER_WARN' ``` The filtering language is done with the OpenTelemetry Transformation Language (OTTL). Learn more about [the OTTL Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl#opentelemetry-transformation-language). ## Scrubbing Sensitive Information Sometimes you want to make sure that certain information does not escape your application or service. This could be for regulatory reasons regarding personally identifiable information, or you want to ensure that certain information does not end being stored with a vendor. Scrubbing attributes from a span is possible with the built-in [Attributes Span Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor). Span processors in OpenTelemetry allow you to hook into the lifecycle of a span and modify the span contents before sending to a backend. The Attributes Span Processor can be configured with actions, and specifically the `delete` action can be used to remove span attributes. There are other actions that could be used, such as `upsert` to replace the value or `hash` to encrypt the value with a SHA1 hash. ```yaml theme={} processors: attributes: actions: - key: my.sensitive.data action: delete ``` ## Handling Large Requests If request sizes are too large, there will be an error when trying to send to Honeycomb. The request size limit is 15MB. To help mitigate errors for requests being too large, it is recommended to set a limit on the batch size and use compression when exporting to Honeycomb. ### Configuring Max Batch Size A [Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor) has a configuration option for `send_batch_size` and `send_batch_max_size`. These options specify the number of data points, regardless of size, to include in a batch to be sent. A one-size-fits-all value does not exist for each of these, since different requests will vary in size. However, these are worth tuning to find the right limit to ensure data is being sent reliably. ### Enabling Compression Collector exporters have [configuration options](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md) to enable compression, including for both gRPC and HTTP. Current supported compression types include `gzip`, `snappy`, and `zstd`. ## Replacing the Old Honeycomb-Specific Exporter Because Honeycomb supports OTLP, the old Honeycomb-specific exporter is no longer required and is no longer available in current versions of the Collector. The built-in OTLP exporter should be used instead to send trace data to Honeycomb. ## Choosing between gRPC and HTTP Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind: * Some SDKs default to using gRPC, and it may be easiest to start with the default option. * Some firewall policies are not set up to handle gRPC and require using HTTP. * gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery. gRPC default export uses port 4317, whereas HTTP default export uses port 4318. # Group Logs by Pattern with the OpenTelemetry Drain Processor Source: https://docs.honeycomb.io/send-data/opentelemetry/collector/group-logs-by-pattern Use the OpenTelemetry Collector drain processor to derive a template attribute on every log record, then group, filter, and sample by recurring patterns in Honeycomb. The OpenTelemetry Collector [`drainprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/drainprocessor) applies the [Drain](https://arxiv.org/pdf/1806.04356) online log parsing algorithm to each log record and attaches the derived template as an attribute. A template captures the invariant skeleton of a log message, with variable parts replaced by `<*>` wildcards. For example, the messages `user 1234 logged in from 10.0.0.1` and `user 5678 logged in from 10.0.0.2` both reduce to `user <*> logged in from <*>`. Deriving templates at the collector means a stable template attribute, defaulting to `log.record.template`, lands on every log record in your Honeycomb dataset. You can `GROUP BY` it to find dominant patterns, build SLOs or Triggers against specific templates, and feed template-based conditions into Refinery sampling rules. The drain processor is at **alpha** stability for logs and is distributed in the OpenTelemetry Collector Contrib and Kubernetes builds. The processor operates on log records only; it does not affect traces or metrics. Configuration options and behavior may change between releases. Pin a specific collector version, test in a non-production environment, and read the upstream release notes before upgrading. ## When to use the drain processor The drain processor is the right choice when you need to discover and stabilize log patterns at the collector rather than define them manually. Use it when: * You have high-volume or noisy logs and want to identify the dominant message patterns without manually defining them. * You want a stable template attribute on every event so that SLOs, Triggers, and Refinery rules can reference it directly. * You want to make log volume reduction decisions (sampling, dropping, or routing) at the collector based on pattern rather than message content. Reach for an alternative when: * You only need to template logs for a single ad-hoc query. A [Calculated Field](/configure/environments/calculated-fields/) using regex or string functions is enough and doesn't require collector changes. Calculated Fields are per-query and can't be referenced by SLOs, Triggers, or Refinery. * You already know the exact patterns and just need to tag them. The [`transform`](/send-data/opentelemetry/collector/handle-sensitive-information#transform-processor) processor with OTTL is more precise than a learned algorithm. * You need a stable, generally-available processor and can't adopt alpha components. ## How drain works Drain incrementally builds a fixed-depth parse tree of tokens seen across log messages. Each leaf holds a small set of templates; an incoming message walks the tree to the matching leaf and is merged with the closest template if a sufficient fraction of tokens match. This approach is fast, runs online without retraining, and converges to a stable set of templates after a short warmup. The two parameters that most affect output quality are `tree_depth` and `merge_threshold`. To learn more, refer to the [Tune the Algorithm](#tune-the-algorithm) section. ## Configuration reference This table covers the most common fields. For the authoritative option list at the version of the collector you run, refer to the [upstream `drainprocessor` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/drainprocessor). | Field | Type | Required | Default | Purpose | | --------------------- | ---------- | -------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `tree_depth` | int | No | `4` | Depth of the parse tree. Minimum is `3`. Higher values increase precision but use more memory. | | `merge_threshold` | float | No | `0.4` | Fraction of tokens that must match an existing template (range `0.0`–`1.0`). Lower values merge more aggressively into fewer templates. | | `max_node_children` | int | No | `100` | Maximum children per internal tree node. | | `max_clusters` | int | No | `0` | Maximum number of templates retained. `0` means unlimited. When the cap is hit the least-recently-used template is evicted. | | `extra_delimiters` | `[]string` | No | `[]` | Additional characters to treat as token delimiters, in addition to whitespace. | | `body_field` | string | No | `""` | When the log body is a map, the key whose value should be templated. Empty means template the stringified body. | | `template_attribute` | string | No | `log.record.template` | Attribute name written to each log record. | | `seed_templates` | `[]string` | No | `[]` | Pre-loaded templates the processor starts with, using `<*>` for wildcards. | | `seed_logs` | `[]string` | No | `[]` | Example log lines used to seed templates at startup. | | `warmup_min_clusters` | int | No | `0` | Minimum templates that must exist before the processor begins annotating records. Useful to avoid noisy templates during a cold start. | | `storage` | string | No | in-memory | ID of a [storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage) for persisting templates across collector restarts. | | `save_interval` | duration | No | `0s` | How often to snapshot template state to the storage extension. `0s` disables periodic snapshots. | ## Basic configuration Wire the processor into your logs pipeline. Spans and metric pipelines should not include it. ```yaml theme={} processors: drain: tree_depth: 4 merge_threshold: 0.4 max_clusters: 500 service: pipelines: logs: receivers: [otlp] processors: [drain] exporters: [otlp/honeycomb] ``` After deployment, every log record arriving at Honeycomb has a `log.record.template` attribute set to its derived template. To start with a curated set of templates rather than learning from scratch, seed the processor: ```yaml theme={} processors: drain: seed_templates: - "user <*> logged in from <*>" - "request <*> completed in <*> ms" warmup_min_clusters: 20 ``` `warmup_min_clusters` holds back annotation until the processor has learned the specified number of templates, preventing a noisy first few minutes of data. ## Tune the algorithm The defaults work well for general application logs but produce poor results for very high-volumne logs with many distinct patterns, or very low-volume logs where the algorithm has little data to learn from. Two parameters do most of the work: * **`tree_depth`**: Controls how many leading tokens the tree splits on. Increase it (5–6) for logs where the first few tokens carry less signal, such as logs prefixed with a long timestamp or request ID. Decrease it (3) for short, structured logs. * **`merge_threshold`**: Controls how readily new messages merge into existing templates. Lower values (0.2–0.3) produce fewer, broader templates. Higher values (0.5–0.7) produce more, narrower templates. If you see hundreds of near-duplicate templates, lower the threshold. `max_clusters` is a memory ceiling, not a quality control. Set it to roughly twice the number of stable templates you expect; LRU eviction handles the rest. For most services this is in the low hundreds. If a single source emits structured logs as a map, set `body_field` to the key that holds the human-readable message so the processor templates the right value. ## Use templates in Honeycomb Once log records carry `log.record.template`, you can use it like any other attribute: * **Find dominant patterns:** Run a query over your logs dataset with `COUNT` grouped by `log.record.template`, ordered descending. The top rows are your noisiest patterns. * **Build SLOs and Triggers:** Reference `log.record.template = ""` in the SLI or Trigger condition. A stable template attribute makes these conditions resilient to changes in variable values that would otherwise require regex. * **Drive Refinery sampling:** Configure Refinery rules to drop or downsample high-volume templates while keeping a representative sample of each pattern. * **Identify reduction candidates:** A small number of templates accounting for a large fraction of log volume usually means easy reductions are available, either by dropping at the collector, by pairing with the [log deduplication processor](#reduce-volume-by-pairing-with-the-log-deduplication-processor) downstream, or by addressing the source. ## Reduce volume by pairing with the log deduplication processor A common and effective pattern is to put the [`logdedupprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logdedupprocessor) immediately downstream of the drain processor. The drain processor attaches a stable template attribute to every record; the log deduplication processor then collapses records that share the same template within a time window into a single record annotated with a count. For high-volume, repetitive logs, this can reduce egress to Honeycomb by an order of magnitude or more while preserving the information needed to reason about pattern frequency. By default, the log deduplication processor matches on the full body, attributes, severity, and resource, which prevents two records with the same template but different variable values from collapsing. To deduplicate by template, restrict matching to the template attribute and any resource attributes you care about with `include_fields`. ```yaml theme={} processors: drain: tree_depth: 4 merge_threshold: 0.4 log_dedup: interval: 60s log_count_attribute: log.record.count include_fields: - attributes["log.record.template"] - resource.attributes["service.name"] service: pipelines: logs: receivers: [otlp] processors: [drain, log_dedup] exporters: [otlp/honeycomb] ``` Order matters. The drain processor must run before the log deduplication processor so the template attribute exists when deduplication evaluates its match fields. In Honeycomb, deduplicated records carry the count attribute (renamed to `log.record.count` in the example above) and `first_observed_timestamp`/`last_observed_timestamp` attributes that the log deduplication processor adds. Use `SUM(log.record.count)` rather than `COUNT` when summing volume across templates. The log deduplication processor is also at **alpha** stability for logs. The same upgrade caveats apply. ## Persist templates across restarts By default, the processor's learned templates are in-memory and are lost on collector restart. After a restart, the processor relearns templates and may produce different output until the algorithm converges again. To persist templates, configure a [storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage) and reference it from the processor: ```yaml theme={} extensions: file_storage: directory: /var/lib/otelcol/drain processors: drain: storage: file_storage save_interval: 5m service: extensions: [file_storage] pipelines: logs: receivers: [otlp] processors: [drain] exporters: [otlp/honeycomb] ``` `save_interval` controls how often state is snapshotted. Shorter intervals reduce data loss on crash at the cost of more disk activity. ## Limitations Keep these constraints in mind when planning your pipeline configuration: * The processor operates on logs only. Span and metric pipelines should not include it. * Drain is an online clustering algorithm; output is deterministic only when seeded with `seed_templates` or `seed_logs`. Two collectors without shared state will produce slightly different template inventories. * Without a `storage` extension, templates do not survive collector restarts. * Alpha stability means option names and defaults may change between collector releases. Validate after every upgrade. ## Related topics These resources cover the tools and approaches that work alongside or instead of the drain processor: * [Handle Sensitive Information with the OpenTelemetry Collector](/send-data/opentelemetry/collector/handle-sensitive-information): Covers the `transform` and `redaction` processors, which are the right tools when you know the patterns up front rather than learning them. * [Calculated Fields](/configure/environments/calculated-fields/): Honeycomb-side, per-query templating when collector changes are not an option. * [Refinery](/manage-data-volume/sample/honeycomb-refinery/): Apply template-based sampling rules using the attribute the drain processor produces. * [Upstream `drainprocessor` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/drainprocessor) * [Upstream `logdedupprocessor` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logdedupprocessor) * [Drain: An Online Log Parsing Approach with Fixed Depth Tree (paper)](https://arxiv.org/pdf/1806.04356) # Handle Sensitive Information with the OpenTelemetry Collector Source: https://docs.honeycomb.io/send-data/opentelemetry/collector/handle-sensitive-information Filter, redact, and hash PII, credit card numbers, and other sensitive data from your telemetry before it reaches Honeycomb. Sensitive data, such as personally identifiable information (PII), credit card numbers, and email addresses, can be useful for incident diagnosis, but security and compliance considerations may require you to remove or mask it before it reaches any observability backend. The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) can filter and redact sensitive data before it leaves your environment using [processors](/send-data/opentelemetry/collector/processors), which are components that sit between [receivers](/send-data/opentelemetry/collector/receivers) and [exporters](/send-data/opentelemetry/collector/exporters) in the Collector pipeline and transform, filter, or redact telemetry data in transit. This page covers three processors for handling sensitive data: `attributes`, `redaction`, and `transform`. To learn more about these processors, visit [OpenTelemetry Collector Processors](/send-data/opentelemetry/collector/processors). Each example on this page includes both the processor configuration and the [service pipeline](/send-data/opentelemetry/collector/configure) entry required to activate it. Filtering sensitive data reduces your ability to diagnose and troubleshoot problems. Carefully consider the trade-offs between security and usability before configuring sensitive data filtering. ## Choosing an approach The right approach depends on how much of the value you need to preserve and which signal types you are working with: | Goal | Processor | Approach | Signals | | -------------------------------------------------------------------------------------------------- | ------------ | ------------------- | --------------------- | | [Remove an attribute entirely](#deleting-a-specific-attribute) | `attributes` | `delete` | traces, metrics, logs | | [Replace a value with a static placeholder](#redacting-or-hashing-a-specific-attribute-value) | `attributes` | `upsert` | traces, metrics, logs | | [Hash a value (SHA1)](#redacting-or-hashing-a-specific-attribute-value) | `attributes` | `hash` (SHA1) | traces, metrics, logs | | [Hash a value (SHA256)](#using-sha256-for-stronger-hashing) | `transform` | `SHA256()` | traces, metrics, logs | | [Block values matching a regex pattern](#blocking-values-matching-a-pattern) | `redaction` | `blocked_values` | traces only | | [Restrict which attributes are allowed through](#restricting-which-attributes-are-allowed-through) | `redaction` | `allowed_keys` | traces only | | [Mask part of a value while preserving context](#masking-values-using-pattern-matching) | `transform` | `replace_pattern()` | traces, metrics, logs | | [Anonymize an IP address by truncating the last octet](#anonymizing-ip-addresses) | `transform` | `replace_pattern()` | traces, metrics, logs | | [Keep only specific attributes](#keep-only-specific-attributes) | `transform` | `keep_keys()` | traces, metrics, logs | For metrics and logs, use the `attributes` or `transform` processor. The `redaction` processor supports traces only. ## OpenTelemetry Transformation Language (OTTL) Several examples on this page use [OpenTelemetry Transformation Language (OTTL)](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl), a scripting language used by the `transform` processor (and others) that lets you match and modify telemetry data using expressions. If you encounter OTTL syntax in an example and want to understand it before configuring your own rules, visit OpenTelemetry's GitHub resources on: * [OTTL syntax](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md) * [OTTL functions](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs) * [OTTL contexts](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts) ## Deleting a specific attribute Use the [`attributes` processor](/send-data/opentelemetry/collector/processors#attributes-processor) to remove an attribute entirely from spans, metrics, or logs. This is the simplest approach when an attribute should never reach Honeycomb under any circumstances. ```yaml theme={} processors: attributes: actions: - key: account_password action: delete ... service: pipelines: traces: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] ``` ## Redacting or hashing a specific attribute value Use the [`attributes` processor](/send-data/opentelemetry/collector/processors#attributes-processor) to replace a sensitive value with a static placeholder, remove it entirely, or replace it with a SHA1 hash. Hashing preserves the ability to correlate data (for example, identifying that two spans share the same email address) without exposing the raw value. The following configuration redacts a credit card number using `upsert` to guarantee the placeholder is set whether or not the attribute exists, deletes a password, and hashes an email address: ```yaml theme={} processors: attributes: actions: - key: cc_number value: redacted action: upsert - key: account_password action: delete - key: account_email action: hash ... service: pipelines: traces: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] metrics: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] logs: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] ``` After processing, the attributes appear in Honeycomb as follows: | Attribute | Before | After | | ------------------ | ------------------ | ------------------------------------------ | | `cc_number` | `4111111111111111` | `redacted` | | `account_password` | `mysecretpassword` | *absent* | | `account_email` | `user@example.com` | `63a710569261a24b3766275b7000ce8d7b32e2f7` | The `attributes` processor `hash` action uses SHA1. SHA1 is no longer recommended by NIST and is vulnerable to rainbow table attacks if the set of possible values is small or predictable. For stronger anonymization, use the `transform` processor with SHA256 instead. For an example, refer to the [Using SHA256 for stronger hashing](#using-sha256-for-stronger-hashing) section. ### Using SHA256 for stronger hashing The `transform` processor supports SHA256 via OTTL, which is significantly stronger than the SHA1 used by the `attributes` processor's `hash` action. The following configuration replaces `user.email` with its SHA256 hash and deletes the original: ```yaml theme={} processors: transform: trace_statements: - context: span statements: - set(attributes["user.email"], SHA256(attributes["user.email"])) where attributes["user.email"] != nil ... service: pipelines: traces: receivers: [otlp] processors: [transform, batch] exporters: [otlp_http] ``` | Attribute | Before | After | | ------------ | ------------------ | ------------------------------------------------------------------ | | `user.email` | `user@example.com` | `b4c9a289323b21a01c3e940f150eb9b8c542587f1abfd8f0e1cc1ffc5e475514` | For the full OTTL syntax and function reference, visit the [OpenTelemetry Transformation Language (OTTL)](#opentelemetry-transformation-language-ottl) section. SHA256 is deterministic: the same input always produces the same hash, which preserves correlability. However, if the set of possible values is small or predictable (for example, a fixed list of internal user IDs), hashes can be reversed using a lookup table. For high-sensitivity requirements, consult your security team about whether deterministic hashing meets your compliance obligations. ## Blocking values matching a pattern Use this approach when sensitive data could appear in any attribute value and you can't enumerate every key in advance, such as credit card numbers or IP addresses that may surface across multiple fields. ### For traces Use the [`redaction` processor](/send-data/opentelemetry/collector/processors#redaction-processor) to mask attribute values that match regex patterns. The redaction processor supports traces only. For metrics or logs, use the attributes or transform processor instead. The following configuration blocks credit card numbers for Visa, Amex, and Mastercard, and IP addresses: ```yaml theme={} processors: redaction: allow_all_keys: true blocked_values: - "^4[0-9]{12}(?:[0-9]{3})?$" ## Visa - "^3[47][0-9]{13}$" ## Amex - "^(5[1-5][0-9]{14}|2(22[1-9][0-9]{12}|2[3-9][0-9]{13}|[3-6][0-9]{14}|7[0-1][0-9]{13}|720[0-9]{12}))$" ## Mastercard - "\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.|$)){4}\b" ## IP address ... service: pipelines: traces: receivers: [otlp] processors: [redaction, batch] exporters: [otlp_http] ``` The `redaction` processor only masks string attribute values. Numeric attributes, such as a credit card number stored as an integer rather than a string, won't be caught by `blocked_values` patterns. If your application may store sensitive values as non-string types, use the `attributes` or `transform` processor instead. Matching values are replaced with `***` in Honeycomb. The attribute key is preserved but the value is obscured: | Attribute | Before | After | | -------------- | ------------------ | ----- | | `payment.card` | `4111111111111111` | `***` | | `client.ip` | `192.168.1.42` | `***` | ### For metrics and logs The `redaction` processor supports traces only. For metrics and logs, choose based on how much of the value you need to preserve: * To completely remove or hash a value, use the `attributes` processor. Hashing replaces the value with a SHA1 hash, which is unreadable but consistent, so you can still correlate events without exposing the original value. * When you need partial masking that preserves part of the value for readability, such as keeping an email domain while obscuring the local part, use the `transform` processor. #### Completely removing or hashing a value To block sensitive values in metrics or logs, use the `attributes` processor with the `hash` or `upsert` action to mask values, or `delete` to remove them entirely: ```yaml theme={} processors: attributes: actions: - key: account_email action: hash - key: account_password action: delete ... service: pipelines: metrics: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] logs: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] ``` #### Partially masking a value Use the `transform` processor to mask values using `replace_pattern` for more complex pattern matching: ```yaml theme={} processors: transform: metric_statements: - context: datapoint statements: - replace_pattern(attributes["account_email"], "^(.+)@(.+)$", "****@$2") log_statements: - context: log statements: - replace_pattern(attributes["account_email"], "^(.+)@(.+)$", "****@$2") ... service: pipelines: metrics: receivers: [otlp] processors: [transform, batch] exporters: [otlp_http] logs: receivers: [otlp] processors: [transform, batch] exporters: [otlp_http] ``` The local part of the email address is masked while the domain is preserved: | Attribute | Before | After | | --------------- | ------------------ | ------------------ | | `account_email` | `user@example.com` | `****@example.com` | For the full OTTL syntax and function reference, visit the [OpenTelemetry Transformation Language (OTTL)](#opentelemetry-transformation-language-ottl) section. ## Restricting which attributes are allowed through Use this approach when you need the strictest possible data governance: you know exactly which attributes should reach Honeycomb and want to guarantee that nothing unexpected gets through, including attributes added by future instrumentation changes. ### For traces Use the [`redaction` processor](/send-data/opentelemetry/collector/processors#redaction-processor) in allowlist mode to remove every attribute except those you explicitly permit. The `redaction` processor supports traces only. For metrics or logs, use the `attributes` processor with `delete` actions instead. The following configuration removes all attributes except `description`, `group`, and `id`: ```yaml theme={} processors: redaction: allow_all_keys: false allowed_keys: - description - group - id ... service: pipelines: traces: receivers: [otlp] processors: [redaction, batch] exporters: [otlp_http] ``` ### For metrics and logs The `redaction` processor supports traces only. To restrict attributes on metrics or logs, use the `attributes` processor to explicitly delete each attribute you want to remove. Unlike the `redaction` processor's allowlist approach, you enumerate the attributes to remove rather than the attributes to keep: ```yaml theme={} processors: attributes: actions: - key: account_password action: delete - key: cc_number action: delete - key: user.full_name action: delete ... service: pipelines: metrics: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] logs: receivers: [otlp] processors: [attributes, batch] exporters: [otlp_http] ``` ## Masking values using pattern matching Use the [`transform` processor](/send-data/opentelemetry/collector/processors#transform-processor) with [OpenTelemetry Transformation Language (OTTL)](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl) to mask sensitive values that appear within larger strings, such as passwords embedded in command line arguments or URLs. ### Masking passwords in command line arguments The following configuration masks any password appearing in a command line attribute, such as `$env password=mysecret username=myusername python run-my-app.py`: ```yaml theme={} processors: transform: trace_statements: - context: resource statements: - replace_pattern(attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***") ... service: pipelines: traces: receivers: [otlp] processors: [transform, batch] exporters: [otlp_http] ``` The password value is replaced while the rest of the command line is preserved: | Attribute | Before | After | | ---------------------- | ----------------------------------- | ------------------------------ | | `process.command_line` | `python app.py --password=mysecret` | `python app.py --password=***` | For the full OTTL syntax and function reference, visit the [OpenTelemetry Transformation Language (OTTL)](#opentelemetry-transformation-language-ottl) section. ### Anonymizing IP addresses The following configuration drops the last octet of an IPv4 address in the `client.address` attribute, replacing it with `0` to anonymize the address while preserving the network prefix: ```yaml theme={} processors: transform: trace_statements: - context: span statements: - replace_pattern(attributes["client.address"], "\\.\\d+$", ".0") ... service: pipelines: traces: receivers: [otlp] processors: [transform, batch] exporters: [otlp_http] ``` The last octet is replaced with `0`, anonymizing the host while preserving the network prefix: | Attribute | Before | After | | ---------------- | -------------- | ------------- | | `client.address` | `192.168.1.42` | `192.168.1.0` | For the full OTTL syntax and function reference, visit the [OpenTelemetry Transformation Language (OTTL)](#opentelemetry-transformation-language-ottl) section. ## Keeping only specific attributes Use the [`transform` processor](/send-data/opentelemetry/collector/processors#transform-processor) with the `keep_keys` OTTL function to remove all attributes except those you specify. This is useful when you want to reduce the attribute surface to a known safe set before data reaches Honeycomb. The following configuration keeps only `service.name`, `service.namespace`, `cloud.region`, and `process.command_line`, removing everything else: ```yaml theme={} processors: transform: trace_statements: - context: resource statements: - keep_keys(attributes, "service.name", "service.namespace", "cloud.region", "process.command_line") ... service: pipelines: traces: receivers: [otlp] processors: [transform, batch] exporters: [otlp_http] ``` Any attribute not in the list, such as `http.url` or `db.statement`, is removed before data reaches Honeycomb. For the full OTTL syntax and function reference, visit the [OpenTelemetry Transformation Language (OTTL)](#opentelemetry-transformation-language-ottl) section. # Migrate Telemetry Schemas with the OpenTelemetry Schema Processor Source: https://docs.honeycomb.io/send-data/opentelemetry/collector/migrate-telemetry-schemas Use the OpenTelemetry Collector schema processor to translate telemetry between schema versions, with migration mode to emit both old and new attribute names during a transition. The OpenTelemetry Collector [`schemaprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/schemaprocessor) translates telemetry between versions of a [telemetry schema](https://opentelemetry.io/docs/specs/otel/schemas/). It reads each signal's `schema_url`, fetches the corresponding schema file, and applies schema-defined transformations, such as attribute renames, so signals from instrumentations on different schema versions arrive in Honeycomb with consistent attribute names. The processor works with any schema family that follows the OpenTelemetry telemetry schemas specification. The most common case is migrating between OpenTelemetry semantic convention versions, such as the HTTP semantic conventions stabilization, but the same configuration applies to custom schema families your organization publishes. Rather than writing [transform processor](/send-data/opentelemetry/collector/handle-sensitive-information#transform-processor) OTTL statements by hand, the schema processor reads the schema file and applies the defined renames automatically. This matters for Honeycomb because any feature that references a renamed attribute by name, such as an SLO, Trigger, Calculated Field, Board, saved query, or Refinery rule, stops matching new data until you update that feature. The schema processor is at **alpha** stability for traces, metrics, and logs and is distributed in the OpenTelemetry Collector Contrib build. Configuration options and behavior may change between releases. Pin a specific collector version, test in a non-production environment, and read the upstream release notes before upgrading. ## When to use the schema processor Choosing the right tool upfront saves significant rework later. Use the schema processor when: * You are migrating between schema versions where the schema author publishes a telemetry schema file describing the changes. OpenTelemetry semantic conventions are the canonical example; an internal schema family hosted on your own domain also qualifies. * You want to emit both the old and the new attribute names during a transition window so that existing queries, SLOs, Triggers, Boards, and Refinery rules keep working until you cut over. * You operate a fleet of services on mixed instrumentation versions and want a single collector configuration to converge on one consistent set of attribute names. Use the [`transform`](/send-data/opentelemetry/collector/handle-sensitive-information#transform-processor) processor instead when: * The rename or rewrite is ad-hoc and not described by a telemetry schema file. * You need conditional logic, value rewrites, or metric unit conversions that the schema files do not express. * You need a stable, generally-available processor and can't adopt alpha components. The two processors can coexist in the same pipeline: the schema processor handles schema-driven renames, and the transform processor handles everything else. ## How the schema processor works The processor reads each incoming signal's `schema_url` and, if it matches a known schema family, applies the chain of transformations from the signal's version to the configured `target` version for that family. By default, a rename replaces the old attribute with the new one. In *migration mode* (sometimes called *copy-mode*), the processor instead emits both the old and the new attribute name for renames between a specified `from` version and the `target`. This is the safe-rollout pattern: deploy migration mode, update downstream consumers to the new names at your own pace, then remove migration mode to complete the cutover. ## Configuration reference This table covers the most common fields. For the authoritative option list at the version of the collector you run, refer to the [upstream `schemaprocessor` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/schemaprocessor). | Field | Type | Required | Default | Purpose | | ------------------- | ---------- | -------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `targets` | `[]string` | Yes | — | Schema URLs identifying the target version for each schema family the processor should translate to. One entry per family. | | `prefetch` | `[]string` | No | — | Schema URLs to download and cache at collector startup. Useful for warming the cache or ensuring availability in restricted networks. | | `migration` | `[]object` | No | — | Migration entries. Each entry has a `target` and a `from` schema URL. While the entry is present, renames between `from` and `target` produce both old and new attribute names. | | `storage` | `string` | No | in-memory | ID of a [storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage) for persistent schema file caching across collector restarts. | | `cache_cooldown` | duration | No | `5m` | Cooldown period after the processor hits the retry limit while fetching a schema. | | `cache_retry_limit` | int | No | `5` | Consecutive fetch failures before the processor enters cooldown. | ## Basic configuration The minimum configuration sets one or more `targets` and wires the processor into a pipeline. ```yaml theme={} processors: schema: targets: - https://opentelemetry.io/schemas/1.27.0 service: pipelines: traces: receivers: [otlp] processors: [schema] exporters: [otlp/honeycomb] metrics: receivers: [otlp] processors: [schema] exporters: [otlp/honeycomb] logs: receivers: [otlp] processors: [schema] exporters: [otlp/honeycomb] ``` Once deployed, signals arriving with an older `schema_url` from the same family are translated to the target version's (in theis case, `1.27.0`) attribute names before they leave the collector. To translate multiple schema families at once, including a custom one your organization publishes, list a target URL for each family: ```yaml theme={} processors: schema: targets: - https://opentelemetry.io/schemas/1.27.0 - https://schemas.example.com/telemetry/2.0.0 ``` ## Migration mode for safe rollouts A hard cutover from old to new attribute names means any Honeycomb feature that references the old names stops matching new incoming data until you update that feature. Migration mode avoids that by coalescing old and new attribute names onto the same span, metric, or log record, so queried referencing either name continue to match while you work through the updates. The recommended workflow is: 1. **Set the new target version.** Update `targets` to the new schema URL. 2. **Add a migration entry.** Add a `migration` entry with the same `target` and a `from` pointing at the previous version your fleet is emitting. 3. **Deploy.** Telemetry now carries both old and new names for the renames in that version range. 4. **Migrate downstream consumers.** Update queries, SLOs, Triggers, Calculated Fields, Boards, and Refinery rule conditions to reference the new names. Verify each in Honeycomb before moving on. 5. **Remove the migration entry and redeploy.** The processor stops emitting the old names, and the cutover is complete. This example migrates from 1.26.0 to 1.27.0: ```yaml theme={} processors: schema: prefetch: - https://opentelemetry.io/schemas/1.26.0 targets: - https://opentelemetry.io/schemas/1.27.0 migration: - target: https://opentelemetry.io/schemas/1.27.0 from: https://opentelemetry.io/schemas/1.26.0 ``` After updating downstream consumers, remove the `migration` block: ```yaml theme={} processors: schema: targets: - https://opentelemetry.io/schemas/1.27.0 ``` Migration mode duplicates renames only between the `from` and `target` you specify. Other transformations defined in the schema files still apply normally. ## Verify the migration in Honeycomb Before removing migration mode, confirm that both attribute name sets are present and that the Honeycomb features depending on the renamed attributes still work. For each feature that depends on a renamed attribute, run these checks: * Query the dataset for the new attribute name and confirm recent events appear. * Query the dataset for the old attribute name and confirm recent events still appear. * Fire a Trigger or evalute an SLO that depends on the renamed attribute against a known event and confirm it evaluates correctly. Audit the following Honeycomb features that commonly reference attribute names: * SLOs * Triggers * Calculated Fields * Boards and saved queries * Refinery rule conditions and field lists * Other OpenTelemetry Collector configurations downstream of this one If you use [Refinery](/manage-data-volume/sample/honeycomb-refinery/), confirm rule conditions match. Refinery 2.3 and later support [multi-field conditions](/troubleshoot/product-lifecycle/recommended-migrations/migrate-to-opentelemetry-semantic-conventions/#update-refinery-rules), which complement migration mode for tail sampling during the transition. After you remove the migration entry, repeat the same checks to confirm the old names are no longer appearing and that nothing still depends on them. ## Limitations Keep these constraints in mind when you plan your pipeline configuration: * The processor applies only transformations defined in telemetry schema files. Ad-hoc or organization-specific renames that aren't captured in a schema file require the [transform processor](/send-data/opentelemetry/collector/handle-sensitive-information#transform-processor). * The processor must be able to reach the schema URLs it is configured with. For air-gapped or restricted environments, use `prefetch` with a `storage` extension to cache schemas locally. * Signals without a `schema_url`, or with a `schema_url` from a family the processor isn't configured to handle, pass through untouched. * Alpha stability means option names and defaults may change between collector releases. Validate after every upgrade. ## Related Topics These resources cover the tools and migration paths that work alongside the schema processor: * [Handle Sensitive Information with the OpenTelemetry Collector](/send-data/opentelemetry/collector/handle-sensitive-information): Covers the `transform` processor and OTTL, the imperative alternative for arbitrary attribute rewrites. * [Migrate to Stabilized OpenTelemetry HTTP Semantic Conventions](/troubleshoot/product-lifecycle/recommended-migrations/migrate-to-opentelemetry-semantic-conventions/): The HTTP semantic conventions migration guide with worked OTTL coalescing examples. * [Customize Telemetry Schema](/configure/teams/customize-telemetry-schema): A distinct, Honeycomb-side feature that resolves attributes at query time using overlay schemas. It does not transform data on ingest and is not a substitute for the schema processor. * [Upstream `schemaprocessor` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/schemaprocessor) * [OpenTelemetry Telemetry Schemas specification](https://opentelemetry.io/docs/specs/otel/schemas/) # Send Data to Honeycomb with Python Source: https://docs.honeycomb.io/send-data/python Explore available methods for sending telemetry from your Python application to Honeycomb. When you are working with Python, we recommend these methods of sending data to Honeycomb. When you want to instrument Python applications in a standard, vendor-agnostic, and future-proof way, we recommend using the OpenTelemetry Python SDK to send telemetry data to Honeycomb. The OpenTelemetry Python SDK allows you to send traces, logs, and metrics. When you need to create and send structured logs to Honeycomb, use Libhoney for Python, our structured logging library for Python applications. # Send Logs with Libhoney for Python Source: https://docs.honeycomb.io/send-data/python/libhoney Send structured events to Honeycomb from your Python application using Libhoney, Honeycomb's low-level structured logging library for the Events API. Libhoney for Python is Honeycomb's structured logging library for Python applications. It is a low-level library that helps you send structured events to Honeycomb's [Events API](/api/events/). If you are instrumenting a new application for tracing, we recommend that you use [OpenTelemetry](/send-data/opentelemetry/) instead. ## Installation ```shell pip theme={} pip install libhoney ``` ```shell poetry theme={} poetry add libhoney ``` ## Links * [Changelog](https://github.com/honeycombio/libhoney-py/blob/main/CHANGELOG.md) * [Source Code](https://github.com/honeycombio/libhoney-py) * [Examples](https://github.com/honeycombio/libhoney-py/tree/main/examples) ## Initialization Initialize the library by passing in your Team API key and the default dataset name to which it should send events. ```python theme={} import libhoney libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-python-example", debug=True) # ... Do work and capture events ``` ### Working With Proxies Using a proxy requires overriding the default `Transmission` implementation when initializing `libhoney`: ```python theme={} import libhoney libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-python-example", debug=True, transmission_impl=libhoney.transmission.Transmission(proxies={'https': 'https://myproxy.example.com'})) ``` The `proxies` map passed in is documented in the [requests](https://requests.readthedocs.io/en/latest/user/advanced/#proxies) documentation. Note that if you override `transmission_impl`, if you also have non-default values for options such as `max_concurrent_batches` and `max_batch_size` in `libhoney.init`, they will need to be specified in the new `Transmission` object. Further configuration options can be found in [the **init**.py file](https://github.com/honeycombio/libhoney-py/blob/main/libhoney/__init__.py#L45). ## Building And Sending Events Once initialized, `libhoney` is ready to send events. Events go through three phases: ```python theme={} # Creation event = libhoney.new_event() # Adding Fields event.add_field("key", "val") dataMap = {"key2": "val2"} event.add(dataMap) # Enqueuing event.send() ``` Upon calling `.send()`, the event is dispatched to be sent to Honeycomb. Events are queued to be transmitted asynchronously, allowing your application to continue without delay. The queue is limited in size, however, and when creating events faster than they can be sent, overflowed events will be dropped instead of backing up and slowing down your application. This behavior can be configured at initialization by adjusting `block_on_send`, as described [here](https://github.com/honeycombio/libhoney-py/blob/main/libhoney/__init__.py#L54). In its simplest form, you can add a single field to an event with the `libhoney.add_field(k, v)` method. If you add the same key multiple times, only the last value added will be kept. More complex structures (dicts and objects—things that can be serialized into a JSON object) can be added to an event with the `.add(data)` method. Events can have metadata associated with them that is not sent to Honeycomb. This metadata is used to identify the event when processing the response. More detail about metadata is below in the Response section. ## Handling Responses Sending an event is an asynchronous action and will avoid blocking by default. `.send()` will enqueue the event to be batched and sent as soon as possible (thus, the return value doesn't indicate that the event was successfully sent). You can monitor the queue returned by `.responses()` to check whether events were successfully received by Honeycomb's servers. The responses queue will receive responses for each batch of events sent to the Honeycomb API. Before sending an event, you have the option to attach metadata to that event. This metadata is not sent to Honeycomb; instead, it is used to help you match up individual responses with sent events. When sending an event, `libhoney` will take the metadata from the event and attach it to the response object for you to consume. Add metadata by calling `.add_metadata({"key": "value"})` on an event. Responses are represented as dicts with the following keys: * **metadata**: the metadata you attached to the event to which this response corresponds * **status\_code**: the HTTP status code returned by Honeycomb when trying to send the event. `2xx` indicates success. * **duration**: the number of milliseconds it took to send the event. * **body**: the body of the HTTP response from Honeycomb. On failures, this body contains some more information about the failure. * **error**: when the event does not even get to create a HTTP attempt, the reason will be in this field. For example, when sampled or dropped because of a queue overflow. You do not have to process responses if you are not interested in them—simply ignoring them is perfectly safe. Unread responses will be dropped. ## What to Send? Honeycomb events are composed of fields of four basic types: `string`, `int`, `float`, and `bool`. You should send any data that will help you provide context to an event in your application, such as user ids, timers, measurements, system info, build ID, and metadata. For more guidance about instrumenting your application, check our [instrumentation introduction guide](/get-started/basics/observability/concepts/instrumentation/). ## Examples ### Simple: Send a Blob of Data ```python theme={} import libhoney libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-python-example", debug=True) # create a new event ev = libhoney.new_event() # add data up front ev.add({ "method": "get", "hostname": "appserver15", "payload_length": 27 }) # do some work, maybe take some measurements # add another field # you can send raw measurements - Honeycomb can help us compute averages, percentiles, etc # when you query ev.add_field("duration_ms", 153.12) ev.send() ``` ### Intermediate: Override Some Attributes ```python theme={} # ... Initialization code ... params = { "hostname": "foo.local", "built": false, "user_id": -1 } # add the above parameters to the global client - these will get added # to every new event, but can be overridden libhoney.add(params) # Spawn a new event and override the user_id event = libhoney.new_event() # overrides a previously set field event.add_field("user_id", 15) # new fields are also welcome event.add_field("latency_ms", datetime.datetime.now() - start) event.send() ``` Further examples can be found [on GitHub](https://github.com/honeycombio/libhoney-py). ## Middleware Examples: Django [Django](https://www.djangoproject.com/) is a widely-used, high-level Python Web framework. Each inbound HTTP request as received by a framework like Django maps nicely to Honeycomb events, representing "a single thing of interest that happened" in a given system. Django middlewares are simply classes that have access to the request object (and optionally the response object) in the application's request-response chain. As such, you can define a simple `honey_middleware.py` file as in the following: ```python theme={} import os import time import libhoney class HoneyMiddleware(object): def __init__(self): libhoney.init(writekey=os.environ["YOUR_API_KEY"], dataset="django-requests", debug=True) def process_request(self, request): request.start_time = time.time() return None def process_response(self, request, response): response_time = time.time() - request.start_time ev = libhoney.Event(data={ "method": request.method, "scheme": request.scheme, "path": request.path, "query": request.GET, "isSecure": request.is_secure(), "isAjax": request.is_ajax(), "isUserAuthenticated": request.user.is_authenticated(), "username": request.user.username, "host": request.get_host(), "ip": request.META['REMOTE_ADDR'], "responseTime_ms": response_time * 1000, }) ev.send() return response ``` See the [`examples/` directory on GitHub](https://github.com/honeycombio/libhoney-py/tree/main/examples) for more sample code demonstrating how to use events, builders, fields, and dynamic fields, specifically in the context of Django middleware. ## Advanced Usage: Global Fields Some fields are interesting to have on every event: the build ID of the application, the application server's hostname, or any fields you would like. Rather than remembering to create these for each event you create, you can add them globally at any time in your application. ```python theme={} libhoney.add_field("build_id", 12345) # can also add a dictionary of fields libhoney.add({ "server_hostname": "myHostname", "server_ip": "1.2.3.4", }) # now any event I create has these fields ev = libhoney.new_event() ev.send() ``` ## Advanced Usage: Utilizing Builders Global fields can be useful, but what if you want to propagate common fields that are only relevant to one component of your application? Builders are objects that generate new events, but that also propagate any fields added to it to the created events. You can also clone builders—the cloned builder will have a copy of all the fields and dynamic fields in the original. As your application forks down into more and more specific functionality, you can create more detailed builders. The final event creation in the leaves of your application's tree will have all the data you have added along the way in addition to the specifics of this event. The global scope is essentially a specialized builder, for capturing attributes that are likely useful to all events (for example, hostname or environment). Adding this kind of peripheral and normally unavailable information to every event gives you enormous power to identify patterns that would otherwise be invisible in the context of a single request. ```python theme={} libhoney.add_field("name", "fruit stand") # this part of the application deals with apple sales b1 = libhoney.NewBuilder() b1.add_field("fruit", "apple") ev1 = b1.new_event() # will have field { "name": "fruit stand", "fruit": "apple" } ev1.send() # ... # this component is specific to medium fuji apple sales - clone the previous builder # and add more context to events from this component b2 = b1.clone() b2.add_field("variety", "fuji") b2.add_field("size", "medium") # ... ev2 = b2.new_event() ev2.add_field("qty_sold", 5) # will have field { "name": "fruit stand", "fruit": "apple", "variety": "fuji", "size": "medium", "qty_sold": 5 } ev2.send() ``` ## Advanced Usage: Dynamic Fields The top-level `libhoney` and Builders support `.add_dynamic_field(func)`. Adding a dynamic field to a Builder or top-level `libhoney` ensures that each time an event is created, the provided function is executed and the returned value is added to the event. The key is the `__name__` attribute of the function. This may be useful for including dynamic process information such as memory used, number of threads, concurrent requests, and so on to each event. Adding this kind of dynamic data to an event makes it easy to understand the application's context when looking at an individual event or error condition. ```python theme={} import random colors = ['red', 'blue', 'green', 'yellow'] def color_category(): return colors[random.randint(0,3)] libhoney.add_dynamic_field(color_category) ev = libhoney.new_event() # will contain field 'color_category' with one value in 'red|blue|green|yellow' ev.send() ``` ## Advanced Usage: Responses Queue If you would like to view the API response for events you have sent, if you need to verify receipt of an event, you can use the responses queue to do so: ```python theme={} ev = libhoney.new_event() # metadata that is attached to an event is preserved in the responses queue ev.add_metadata({"my": "metadata", "event_id": 12345}) ev.send() def read_responses(resp_queue): '''read responses from the libhoney queue, print them out.''' while True: resp = resp_queue.get() # libhoney will enqueue a None value after we call libhoney.close() if resp is None: break status = "sending event with metadata {} took {}ms and got response code {} with message \"{}\" and error message \"{}\"".format( resp["metadata"], resp["duration"], resp["status_code"], resp["body"].rstrip(), resp["error"]) print(status) # ... read_responses(libhoney.responses()) ``` Note that responses, if unread, will begin to fill the queue. Once full, new responses will be dropped. To change this behavior, set `block_on_response=True` when initializing the SDK with `libhoney.init()` - this will prevent dropped responses, but will also prevent new batches from being sent if the queue is not being read from. To continuously process the responses queue, you can run your processing function in another thread. ```python theme={} t = threading.Thread(target=read_responses, args=(libhoney.responses(),)) t.start() ``` ## Customizing Event Transmission By default, events are sent to the Honeycomb API. It is possible to override the default transmission implementation by specifying `transmission_impl` to `init`. A couple of alternative implementations ship with the SDK. * FileTransmission: writes events to a file handle, defaulting to stderr * TornadoTransmission: sends events using Tornado's AsyncHTTPClient rather than using a thread pool To override the default transmission and write events out to stderr: ```python theme={} import libhoney from libhoney.transmission import FileTransmission libhoney.init(writekey='YOUR_API_KEY', transmission_impl=FileTransmission(output=sys.stderr)) ``` If you have unique requirements, you can write your own implementation. See the [transmission source](https://github.com/honeycombio/libhoney-py/blob/main/libhoney/transmission.py) to get started. ## Flushing Events The Python SDK has no mechanism for sending an individual event immediately - events are enqueued and sent asynchronously in batches. You can, however, trigger a flush of the send queue. There are two ways to do this. ```python theme={} libhoney.flush() ``` `.flush()` instructs the transmission to send all of its events and will block until all events in the queue have been sent (or an attempt has been made - see the responses queue if you need to verify success). ```python theme={} libhoney.close() ``` `.close()` flushes all events, but prevents transmission of new events. This is best called as part of your application's shutdown logic to ensure that all events are sent before the program terminates. ## Troubleshooting Refer to [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#libhoney). ## Contributions Features, bug fixes and other changes to `libhoney` are gladly accepted. Please open issues or a pull request with your change [via GitHub](https://github.com/honeycombio/libhoney-py). Remember to add your name to the CONTRIBUTORS file! All contributions will be released under the Apache License 2.0. # Send Data with the OpenTelemetry Python SDK Source: https://docs.honeycomb.io/send-data/python/opentelemetry-sdk Instrument your Python application with the OpenTelemetry Python SDK and send traces, logs, and metrics to Honeycomb. Use the [OpenTelemetry Python SDK](https://github.com/open-telemetry/opentelemetry-python) to instrument Python applications and send telemetry data to Honeycomb. In this guide, you'll instrument your Python application with the OpenTelemetry Python SDK, including automatic instrumentation. ## Before You Begin Before you can set up automatic instrumentation for your Python application, you will need to do a few things. ### Prepare Your Development Environment To complete the required steps, you will need: * A working Python environment * An application written in Python ### Get Your Honeycomb API Key To send data to Honeycomb, you'll need to [sign up for a free Honeycomb account](https://ui.honeycomb.io/signup) and [create a Honeycomb Ingest API Key](/configure/environments/manage-api-keys/#create-api-key). To get started, you can create a key that you expect to swap out when you deploy to production. Name it something helpful, perhaps noting that it's a getting started key. Make note of your API key; for security reasons, you will not be able to see the key again, and you will need it later! For setup, make sure you check the "Can create datasets" checkbox so that your data will show up in Honeycomb. Later, when you replace this key with a permanent one, you can uncheck that box. If you want to use an API key you previously stored in a secure location, you can also [look up details for Honeycomb API Keys](/configure/environments/manage-api-keys/#find-api-keys) any time in your Environment Settings, and use them to retrieve keys from your storage location. ## Add Automatic Instrumentation Automatic instrumentation is provided by an agent, a command line tool `opentelemetry-instrument` that is used to run your application. Add additional instrumentation to your application manually using the included OpenTelemetry Python SDK. ### Acquire Dependencies To add instrumentation, you should install required OpenTelemetry packages and instrumentation libraries. 1. Install the OpenTelemetry Python packages: ```shell theme={} python -m pip install "opentelemetry-sdk[file-configuration]" \ opentelemetry-instrumentation \ opentelemetry-distro \ opentelemetry-exporter-otlp ``` The `[file-configuration]` extra installs the optional dependencies needed to load a YAML configuration file. 2. Install instrumentation libraries for the packages used by your application. We recommend using the `opentelemetry-bootstrap` tool that comes with the OpenTelemetry SDK to scan your application packages and print out a list of available instrumentation libraries. You should then add these libraries to your `requirements.txt` file: ```shell theme={} opentelemetry-bootstrap >> requirements.txt pip install -r requirements.txt ``` If you do not use a `requirements.txt` file, you can install the libraries directly in your current environment: ```shell theme={} opentelemetry-bootstrap --action=install ``` 1. Install the OpenTelemetry Python packages: ```shell theme={} poetry add opentelemetry-instrumentation \ opentelemetry-distro \ opentelemetry-exporter-otlp ``` 2. Install instrumentation libraries for the packages used by your application. We recommend using the `opentelemetry-bootstrap` tool that comes with the OpenTelemetry SDK to scan your application packages to get a list of available libraries. ```shell theme={} poetry run opentelemetry-bootstrap ``` The tool does not support installing packages directly when using Poetry, so you must install them manually. For example, to install the Flask instrumentation library: ```shell theme={} poetry add opentelemetry-instrumentation-flask ``` ### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces # Use the endpoint below for EU # endpoint: https://api.eu1.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Set the following environment variables before running your application: | Environment Variable | Value | | :------------------- | :----------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. The OpenTelemetry declarative configuration is stable at the specification level. Individual fields still under active development are marked with a `/development` suffix in the YAML (see [configuration versioning](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md#experimental-features)). Check the [language support status](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/language-support-status.md) for per-SDK maturity. Add `meter_provider` and `logger_provider` sections to the same file to export metrics and logs. This version also enables resource detectors, which add attributes such as `host.*` and `process.*` automatically: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} detection/development: detectors: - host: - os: - process: - service: tracer_provider: # traces processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # metrics readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/metrics headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} # Legacy metrics only; omit with the current metrics experience: # - name: x-honeycomb-dataset # value: ${HONEYCOMB_METRICS_DATASET} logger_provider: # logs processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` For the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` throughout the file. If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must also specify the Dataset for traces using the `x-honeycomb-dataset` header: ```yaml theme={} headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} - name: x-honeycomb-dataset value: your-dataset ``` ### Enable Python Standard Library Logging If your application uses Python's standard library `logging` module and you want those log records to reach Honeycomb, attach the OpenTelemetry `LoggingHandler` to the root logger during application startup: ```python theme={} import logging from opentelemetry._logs import get_logger_provider from opentelemetry.sdk._logs import LoggingHandler logging.getLogger().addHandler( LoggingHandler(logger_provider=get_logger_provider()) ) ``` This step is a temporary workaround. It will not be required in a future release of the OpenTelemetry Python SDK. Upstream work to honor `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` alongside `OTEL_CONFIG_FILE` is tracked in [open-telemetry/opentelemetry-python#5352](https://github.com/open-telemetry/opentelemetry-python/issues/5352). ### Run Your Application ```shell pip theme={} OTEL_CONFIG_FILE=./otelconfig.yaml opentelemetry-instrument python YOUR_APPLICATION_NAME.py ``` ```shell poetry theme={} OTEL_CONFIG_FILE=./otelconfig.yaml poetry run opentelemetry-instrument python YOUR_APPLICATION_NAME.py ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. ## Add Custom Instrumentation Automatic instrumentation is the easiest way to get started with instrumenting your code. To get additional insight into your system, you should also add custom, or manual, instrumentation where appropriate. Follow the instructions below to add custom instrumentation to your code. To learn more about custom, or manual, instrumentation, visit the comprehensive set of topics covered by [Manual Instrumentation for Python](https://opentelemetry.io/docs/languages/python/instrumentation/) in OpenTelemetry's documentation. ### Acquire Dependencies To start adding custom instrumentation, ensure that the `opentelemetry-api` package exists as a direct dependency in your project. This package provides access to the high-level instrumentation APIs, which gives the ability to retrieve the current span to enrich with additional attributes, to create new spans, and to generate metrics. ```shell pip theme={} python -m pip install opentelemetry-api ``` ```shell poetry theme={} poetry add opentelemetry-api ``` ### Add Attributes to Spans Adding context to a currently executing span in a trace can be useful. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your dataset in Honeycomb. To do this, get the current span from the context and set an attribute with the user ID: ```python theme={} from opentelemetry import trace # ... span = trace.get_current_span() span.set_attribute("user.id", user.id()) ``` ### Acquire a Tracer To create spans, you need to acquire a `Tracer`. ```python theme={} from opentelemetry import trace tracer = trace.get_tracer("tracer.name.here") ``` When you create a `Tracer`, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter. When traces are sent to Honeycomb, the name of the `Tracer` is turned into the `library.name` field, which can be used to show all spans created from a particular tracer. In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different "layers" of your application, then use the name that corresponds to that "layer". The `library.name` field is also used with traces created from instrumentation libraries. ### Create Spans To get the full picture of what is happening, you can leverage manual instrumentation to create custom spans that describe what is happening in your application. To do this, create or re-use a `Tracer` instance and start a span. ```python theme={} from opentelemetry import trace tracer = trace.get_tracer("tracer.name.here") with tracer.start_as_current_span("some-long-running-handler"): # prepare to auth with tracer.start_as_current_span("what-it-takes-to-authorize-handling"): # do the auth # do the authorized long running handling of cool stuff ``` #### Creating Spans Around Methods You can use a decorator to wrap the execution of a method with a span. The span will be automatically closed once the method has completed. ```python theme={} from opentelemetry import trace tracer = trace.get_tracer("tracer.name.here") # ... @tracer.start_as_current_span("do_work") def do_work(): # do some work ... ``` ### Add Multi-Span Attributes Sometimes you want to add the same attribute to many spans within the same trace. This attribute may include variables calculated during your program, or other useful values for correlation or debugging purposes. To add this attribute, leverage the OpenTelemetry concept of [baggage](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#baggage-signal). Baggage allows you to add a `key` with a `value` as an attribute to every subsequent child span of the current application context. In Python, baggage is configured as part of the OpenTelemetry trace context. Modifications to the trace context must be both attached and detached when no longer used to make sure context state is disposed of correctly. 1. Install the `opentelemetry-processor-baggage` package: ```shell pip theme={} python -m pip install opentelemetry-processor-baggage ``` ```shell poetry theme={} poetry add opentelemetry-processor-baggage ``` 2. Configure the OpenTelemetry SDK tracer provider to add a `BaggageSpanProcessor`: ```python theme={} from opentelemetry.processor.baggage import BaggageSpanProcessor, ALLOW_ALL_BAGGAGE_KEYS tracer_provider = TracerProvider() tracer_provider.add_span_processor(BaggageSpanProcessor(ALLOW_ALL_BAGGAGE_KEYS)) ``` 3. Add a baggage entry for the current trace. For example,add a user ID attribute to multiple spans in a trace. ```python theme={} from opentelemetry import baggage from opentelemetry.context import attach, detach ... token = attach( baggage.set_baggage("user.id", user.id()) ) with tracer.start_as_current_span(name="hello"): span.set_attribute("message", "hello world!") detach(token) ``` If you do not detach a modified trace context, you will see runtime errors as the OpenTelemetry SDK will detect un-detached contexts in its internal stack. Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do **NOT** put sensitive information in the Baggage attributes. ## Sampling You can configure the OpenTelemetry SDK to [sample the data](/manage-data-volume/sample/guidelines/) it generates. Honeycomb [weights sampled data based on sample rate](/manage-data-volume/sample/sampled-data-in-honeycomb/), so you must set a resource attribute containing the sample rate. Use a [`TraceIdRatioBased` sampler](https://opentelemetry.io//docs/specs/otel/trace/sdk/#traceidratiobased), with a ratio expressed as `1/N`. Then, also create a resource attribute called `SampleRate` with the value of `N`. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data. In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5). ```shell theme={} export OTEL_TRACES_SAMPLER="traceidratio" export OTEL_TRACES_SAMPLER_ARG=0.5 export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2" ``` The value of `SampleRate` **must** be a positive integer. ## Choosing between gRPC and HTTP Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind: * Some SDKs default to using gRPC, and it may be easiest to start with the default option. * Some firewall policies are not set up to handle gRPC and require using HTTP. * gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery. gRPC default export uses port 4317, whereas HTTP default export uses port 4318. ## Troubleshooting To explore common issues when sending data, visit [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#opentelemetry-sdks-and-honeycomb-distributions). # Instrument React Native Applications with OpenTelemetry Source: https://docs.honeycomb.io/send-data/react-native Instrument your React Native application with the Honeycomb OpenTelemetry React Native SDK to detect slow renders and send telemetry to Honeycomb. The [Honeycomb OpenTelemetry React Native SDK](https://github.com/honeycombio/honeycomb-opentelemetry-react-native) is Honeycomb's distribution of OpenTelemetry for React Native applications. It includes instrumentation for things like detecting slow event loops or unhandled exceptions, and provides a component to simplify navigation instrumentation. This page covers basic usage of the SDK. If you just want to see some code, check out the [example on GitHub](https://github.com/honeycombio/honeycomb-opentelemetry-react-native/tree/main/example). ## Installing the SDK Before you can use Honeycomb’s OpenTelemetry React Native SDK, you need to install it and configure platform-specific dependencies. 1. Configure Metro to recognize `package.json` exports. To do this, create a new file, `metro.config.js`, in your application's root directory. [Enable `package.json` exports](https://reactnative.dev/blog/2023/06/21/package-exports-support) in your Metro configuration. ```js theme={} config.resolver.unstable_enablePackageExports = true; ``` Here's an [example Metro configuration](https://github.com/honeycombio/honeycomb-opentelemetry-react-native/blob/main/example/metro.config.js): ```js theme={} const path = require('path'); const { getDefaultConfig } = require('@react-native/metro-config'); const { getConfig } = require('react-native-builder-bob/metro-config'); const pkg = require('../package.json'); const root = path.resolve(__dirname, '..'); const config = getConfig(getDefaultConfig(__dirname), { root, pkg, project: __dirname, }); // Required to use @opentelemetry package.json "exports" field config.resolver.unstable_enablePackageExports = true; module.exports = config; ``` To learn more about Metro configuration, visit [Configuring Metro](https://metrobundler.dev/docs/configuration/). 2. Install Honeycomb’s OpenTelemetry React Native SDK in your application’s root directory using your preferred package manager. ```bash theme={} npm install @honeycombio/opentelemetry-react-native ``` ```bash theme={} yarn add @honeycombio/opentelemetry-react-native ``` 3. Install the necessary dependencies for each mobile platform your application supports. Add the following dependencies to your application's `build.gradle`. ```gradle theme={} dependencies { //... implementation "io.honeycomb.android:honeycomb-opentelemetry-android:0.0.20" implementation "io.opentelemetry.android:android-agent:0.11.0-alpha" } ``` If your application's `minSDK` version is lower than 26, add [core library desugaring](https://developer.android.com/studio/write/java8-support#library-desugaring) to your `android/app/build.gradle`. ```gradle theme={} android { compileOptions { // Enable support for the new language APIs coreLibraryDesugaringEnabled true } dependencies { coreLibraryDesugaring "com.android.tools:desugar_jdk_libs:2.1.5" } } ``` Add the `use_frameworks!` option to your application's `Podfile`. ```podfile theme={} platform :ios, min_ios_version_supported prepare_react_native_project! use_frameworks! ``` From the `ios` directory, run `pod install` to install the required dependencies. ## Initializing Initialize the SDK at the start of your React Native application. This ensures that events such as startup time and early view loads are captured. ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; import { DiagLogLevel } from '@opentelemetry/api'; const sdk = new HoneycombReactNativeSDK({ // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", apiKey: "YOUR-API-KEY", serviceName: "YOUR-SERVICE-NAME", logLevel: DiagLogLevel.DEBUG, }); sdk.start(); ``` Initialize the SDK at the start of the `onCreate()` method in your main `Application` class. ```kotlin theme={} // MainApplication.kt override fun onCreate() { val options = HoneycombOpentelemetryReactNativeModule.optionsBuilder(this) // Uncomment the line below to send to EU instance. Defaults to US. // .setApiEndpoint("https://api.eu1.honeycomb.io:443") .setApiKey("YOUR-API-KEY") .setServiceName("YOUR-SERVICE-NAME") HoneycombOpentelemetryReactNativeModule.configure(this, options) super.onCreate() } ``` Initialize the SDK at the start of the `application()` method in your `AppDelegate` class. ```swift theme={} // AppDelegate.swift override func application( _ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil ) -> Bool { let options = HoneycombReactNative.optionsBuilder() // Uncomment the line below to send to EU instance. Defaults to US. // .setAPIEndpoint("https://api.eu1.honeycomb.io:443") .setAPIKey("YOUR-API-KEY") .setServiceName("YOUR-SERVICE-NAME") HoneycombReactNative.configure(options) } ``` ## Configuring The Honeycomb React Native SDK can be configured using many of the [configuration options from the Honeycomb Web SDK](https://github.com/honeycombio/honeycomb-opentelemetry-web/tree/main/packages/honeycomb-opentelemetry-web#sdk-configuration). ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; import { DiagLogLevel } from '@opentelemetry/api'; import { ExampleSpanProcessor } from "@/services/ExampleSpanProcessor"; const sdk = new HoneycombReactNativeSDK({ apiKey: "YOUR-API-KEY", // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", serviceName: "YOUR-SERVICE-NAME", resourceAttributes: { "app.environment": "development", }, logLevel: DiagLogLevel.DEBUG, fetchInstrumentationConfig: { propagateTraceHeaderCorsUrls: [ // Regex to match your backend URLs. // Update to the domains you wish to include. /.+/g, ] }, slowEventLoopInstrumentationConfig: { enabled: true, }, spanProcessors: [ new ExampleSpanProcessor(); ] }); sdk.start(); ``` ### Adding resource attributes Resource attributes are available on every span your instrumentation emits. Adding custom, application-specific attributes makes it easier to correlate your data to important business information. You can set resource attributes using the `resourceAttributes` configuration option. ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; const sdk = new HoneycombReactNativeSDK({ apiKey: "YOUR-API-KEY", // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", serviceName: "YOUR-SERVICE-NAME", resourceAttributes: { "app.team": "team name", }, }); sdk.start(); ``` ### Enabling sampling The SDK includes optional [deterministic head sampling](/manage-data-volume/sample/). The sample rate is `1` by default, meaning every trace is exported. The example below sets a `sampleRate` of `40`, meaning 1 in 40 traces will be exported. This sample rate will only apply to JavaScript/TypeScript traces. ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; const sdk = new HoneycombReactNativeSDK({ apiKey: "YOUR-API-KEY", // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", serviceName: "YOUR-SERVICE-NAME", sampleRate: 40, }); sdk.start(); ``` You can configure a sample rate for Android traces with `.setSampleRate()`: ```kotlin theme={} // MainApplication.kt override fun onCreate() { val options = HoneycombOpentelemetryReactNativeModule.optionsBuilder(this) // Uncomment the line below to send to EU instance. Defaults to US. // .setApiEndpoint("https://api.eu1.honeycomb.io:443") .setApiKey("YOUR-API-KEY") .setServiceName("YOUR-SERVICE-NAME") .setSampleRate(40) HoneycombOpentelemetryReactNativeModule.configure(this, options) super.onCreate() } ``` You can configure a sample rate for iOS traces with `.sampleRate()`: ```swift theme={} // AppDelegate.swift override func application( _ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil ) -> Bool { let options = HoneycombReactNative.optionsBuilder() // Uncomment the line below to send to EU instance. Defaults to US. // .setAPIEndpoint("https://api.eu1.honeycomb.io:443") .setAPIKey("YOUR-API-KEY") .setServiceName("YOUR-SERVICE-NAME") .sampleRate(40) HoneycombReactNative.configure(options) } ``` ### Sending to OpenTelemetry Collector In production, we recommend running an [OpenTelemetry Collector](/send-data/opentelemetry/collector/). Your application sends telemetry to your Collector instead of directly to Honeycomb. Your Collector then forwards the telemetry data to Honeycomb, keeping your API key stored securely in the Collector's configuration. Configure your Collector's URL by setting the `endpoint` option when initializing the Honeycomb React Native SDK: ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; const sdk = new HoneycombReactNativeSDK({ endpoint: "http(s)://YOUR-COLLECTOR-URL", serviceName: "YOUR-SERVICE-NAME", }); sdk.start(); ``` This example works for all platforms and shows the core configuration needed to connect your application to a Collector. For Android, use `.setApiEndpoint()`: ```kotlin theme={} // MainApplication.kt override fun onCreate() { val options = HoneycombOpentelemetryReactNativeModule.optionsBuilder(this) .setApiEndpoint("http(s)://YOUR-COLLECTOR-URL") .setServiceName("YOUR-SERVICE-NAME") HoneycombOpentelemetryReactNativeModule.configure(this, options) super.onCreate() } ``` For iOS, use `.setAPIEndpoint()`: ```swift theme={} // AppDelegate.swift override func application( _ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil ) -> Bool { let options = HoneycombReactNative.optionsBuilder() .setAPIEndpoint("http(s)://YOUR-COLLECTOR-URL") .setServiceName("YOUR-SERVICE-NAME") HoneycombReactNative.configure(options) } ``` ### Sending to Honeycomb To send telemetry data directly to Honeycomb, set the `apiKey` option with your [Ingest API Key](/configure/environments/manage-api-keys/#create-api-key). ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; const sdk = new HoneycombReactNativeSDK({ apiKey: "YOUR-API-KEY", // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", serviceName: "YOUR-SERVICE-NAME", }); sdk.start(); ``` This example works for all platforms and shows the core configuration needed to send data directly to Honeycomb. For Android, use `.setApiKey()`: ```kotlin theme={} // MainApplication.kt override fun onCreate() { val options = HoneycombOpentelemetryReactNativeModule.optionsBuilder(this) // Uncomment the line below to send to EU instance. Defaults to US. // .setApiEndpoint("https://api.eu1.honeycomb.io:443") .setApiKey("YOUR-API-KEY") .setServiceName("YOUR-SERVICE-NAME") HoneycombOpentelemetryReactNativeModule.configure(this, options) super.onCreate() } ``` For iOS, use `.setAPIKey()`: ```swift theme={} // AppDelegate.swift override func application( _ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil ) -> Bool { let options = HoneycombReactNative.optionsBuilder() // Uncomment the line below to send to EU instance. Defaults to US. // .setAPIEndpoint("https://api.eu1.honeycomb.io:443") .setAPIKey("YOUR-API-KEY") .setServiceName("YOUR-SERVICE-NAME") HoneycombReactNative.configure(options) //... } ``` ## Adding automatic instrumentation The Honeycomb OpenTelemetry React Native SDK includes [auto-instrumentation options](https://github.com/honeycombio/honeycomb-opentelemetry-react-native/tree/main?tab=readme-ov-file#auto-instrumentation) for: * Application startup time, measured from when the native SDKs start to when the JavaScript SDK finishes initializing * Errors or uncaught exceptions * [Fetch](https://developer.mozilla.org/en-US/docs/Web/API/Window/fetch) instrumentation, using the [opentelemetry-instrumentation-fetch](https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-instrumentation-fetch) package * Slow event loop detection Automatic instrumentation is enabled by default. You can enable or disable individual auto-instrumentation libraries in your configuration. ```ts theme={} import { HoneycombReactNativeSDK } from '@honeycombio/opentelemetry-react-native'; import { DiagLogLevel } from '@opentelemetry/api'; const sdk = new HoneycombReactNativeSDK({ // Uncomment the line below to send to EU instance. Defaults to US. // endpoint: "https://api.eu1.honeycomb.io:443", apiKey: "YOUR-API-KEY", serviceName: "YOUR-SERVICE-NAME", logLevel: DiagLogLevel.DEBUG, // auto-instrumentation options reactNativeStartupInstrumentationConfig: { enabled: true, }, uncaughtExceptionInstrumentationConfig: { enabled: true, }, fetchInstrumentationConfig: { enabled: true, }, slowEventLoopInstrumentationConfig: { enabled: true, }, }); sdk.start(); ``` ## Adding custom instrumentation Automatic instrumentation is a fast way to instrument your code, but you get more insight into your application by adding custom, otherwise known as manual, instrumentation. ### Adding attributes to an active span You can retrieve the currently active span in a trace and add attributes to it. This lets you add more context to traces and gives you more ways to group or filter traces in your queries. ```ts theme={} import { trace } from '@opentelemetry/api'; function handleUser(user) { let currentActiveSpan = trace.getActiveSpan(); currentActiveSpan.setAttribute('user.id', user.getId()); } ``` ### Acquiring a tracer For manual tracing, you need to get a tracer: ```ts theme={} import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('example-tracer', '0.1.0'); ``` ### Creating spans Create custom spans to get a clear view of the critical parts in your application. ```ts theme={} import { trace } from '@opentelemetry/api'; function trackWork() { const tracer = trace.getTracer('example-tracer', '0.1.0'); const span = tracer.startActiveSpan('do work'); console.log('performing work'); span.end(); } ``` ### Adding instrumentation to navigation You can instrument React NativeRouter or Expo Router navigation in your application using the `` component. #### React NativeRouter Add a `` component as a child of your `` component. Pass the [container ref](https://reactnavigation.org/docs/navigation-container/#ref) to your navigation container and your navigation instrumentation components. ```ts theme={} import { NavigationInstrumentation } from '@honeycombio/opentelemetry-react-native'; import { useNavigationContainerRef, NavigationContainer } from '@react-navigation/native'; export default function App() { const navigationRef = useNavigationContainerRef(); return ( {/* Navigation/UI code*/} ); } ``` #### Expo Router Wrap your navigation code with a `` component and pass the container ref to the `` component. ```ts theme={} import { NavigationInstrumentation } from '@honeycombio/opentelemetry-react-native'; import { useNavigationContainerRef } from 'expo-router'; export default function App() { const navigationRef = useNavigationContainerRef(); return ( {/* Navigation/UI code*/} ); } ``` ## Troubleshooting Running into issues? Here are some common problems and ways to fix them. To explore common issues when sending data, visit [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#opentelemetry-sdks-and-honeycomb-distributions). ### File not found error on iOS If you see an error like this when running your application on iOS: ```text theme={} 'HoneycombOpentelemetryReactNative/HoneycombOpentelemetryReactNative-Swift.h' file not found when trying to run for iOS ``` It usually means your application's `Podfile` is missing the `use_frameworks!` line. To resolve this error: 1. Add `use_frameworks!` immediately below `prepare_react_native_project!` in your `Podfile`: ```swift theme={} prepare_react_native_project! use_frameworks! ``` 2. Install the iOS dependencies by navigating to your `ios` directory and running `pod install`: ```bash theme={} cd ios pod install ``` ### Not receiving native telemetry data Unlike JavaScript code, native code does not hot reload on changes. So if you updated your `AppDelegate.swift` or `MainApplication.kt` files and still not getting native telemetry you'll need to rebuild your application. Stop metro, the simulator, and restart the build. If this still doesn’t work try uninstalling the app and reinstalling it. # Send Data to Honeycomb with Ruby Source: https://docs.honeycomb.io/send-data/ruby Explore available methods for sending telemetry from your Ruby application to Honeycomb. When you are working with Ruby, we recommend these methods of sending data to Honeycomb. When you want to instrument Ruby applications in a standard, vendor-agnostic, and future-proof way, we recommend using the OpenTelemetry Ruby SDK to send telemetry data to Honeycomb. The OpenTelemetry Ruby SDK allows you to send traces to Honeycomb. Logs and metrics are in development. When you need to create and send structured logs to Honeycomb, use Libhoney for Ruby, our structured logging library for Ruby applications. # Send Logs with Libhoney for Ruby Source: https://docs.honeycomb.io/send-data/ruby/libhoney Send structured events to Honeycomb from your Ruby application using Libhoney, Honeycomb's low-level structured logging library for the Events API. Libhoney for Ruby is Honeycomb's structured logging library for Ruby applications. It is a low-level library that helps you send structured events to Honeycomb's [Events API](/api/events/). If you are instrumenting a new application for tracing, we recommend that you use [OpenTelemetry](/send-data/opentelemetry/) instead. ## Installation ```ruby theme={} gem 'libhoney' # or, to follow the bleeding edge: # gem 'libhoney', git: 'https://github.com/honeycombio/libhoney-rb.git' ``` This gem has some native dependencies, so if you see an error along the lines of "Failed to build gem native extension", you may need to install the Ruby development headers and a C++ compiler. For example, on Ubuntu: ```shell theme={} sudo apt-get install build-essential ruby-dev ``` Note that `libhoney` requires Ruby 2.3 or greater. ## Links * [API Reference](https://www.rubydoc.info/gems/libhoney/) * [Source Code](https://github.com/honeycombio/libhoney-rb) * [Examples](https://github.com/honeycombio/libhoney-rb/tree/main/examples) ## Initialization Initialize the library by passing in your Team API key and the default dataset name to which it should send events. ```ruby theme={} require 'libhoney' libhoney = Libhoney::Client.new(writekey: 'YOUR_API_KEY', dataset: 'honeycomb-ruby-example') # ... Do work and capture events # Call close to flush any pending calls to Honeycomb libhoney.close(true) ``` Further configuration options can be found [in the API reference](https://www.rubydoc.info/gems/libhoney/). The `Libhoney::Client` initialization params may contain an `api_host` key, which defaults to Honeycomb's API server. Overriding this with an empty string is a good way to drop events in a test environment. ## Building and Sending Events Once initialized, `libhoney` is ready to send events. Events go through three phases: * Creation `event = builder.event()` * Adding fields `event.add_field('key', 'val')`, `event.add(dataMap)` * Transmission `event.send()` Upon calling `.send()`, the event is dispatched to be sent to Honeycomb. All libraries set defaults that will allow your application to function as smoothly as possible during error conditions. When creating events faster than they can be sent, overflowed events will be dropped instead of backing up and slowing down your application. In its simplest form, you can add a single attribute to an event with the `.add_field(k, v)` method. If you add the same key multiple times, only the last value added will be kept. More complex structures, such as hashes and objects that can be serialized into a JSON object, can be added to an event with the `.add(data)` method. Events can have metadata associated with them that is not sent to Honeycomb. This metadata is used to identify the event when processing the response. More detail about metadata is below in the [Response](#handling-responses) section. ## Handling Responses Sending an event is an asynchronous action and will avoid blocking by default. `.send()` will enqueue the event to be sent as soon as possible, and thus, the return value does not indicate that the event was successfully sent. Use the queue returned by `.responses` to check whether events were successfully received by Honeycomb's servers. Before sending an event, you have the option to attach metadata to that event. This metadata is not sent to Honeycomb; instead, it is used to help you match up individual responses with sent events. When sending an event, `libhoney` will take the metadata from the event and attach it to the response object for you to consume. Add metadata by calling `.add_metadata(k, v)` on an event. Responses are represented as hashes with the following keys: * **metadata**: the metadata you attached to the event to which this response corresponds * **status\_code**: the HTTP status code returned by Honeycomb when trying to send the event. `2xx` indicates success. * **duration**: the number of seconds it took to send the event. * **error**: when the event does not even get to create a HTTP attempt, the reason will be in this field. For example, when sampled or dropped because of a queue overflow. You do not have to process responses if you are not interested in them. Ignoring them is perfectly safe. Unread responses will be dropped. ## Examples Honeycomb can calculate all sorts of statistics, so send the data you care about and let us crunch the averages, percentiles, lower/upper bounds, cardinality—whatever you want—for you. ### Simple: Send an Event ```ruby theme={} require 'libhoney' libhoney = Libhoney::Client.new(writekey: 'YOUR_API_KEY', dataset: 'honeycomb-ruby-example') ev = libhoney.event ev.add({ 'duration_ms': 153.12, 'method': 'get', 'hostname': 'appserver15', 'payload_length': 27 }) ev.send # Call close to flush any pending calls to Honeycomb libhoney.close(true) ``` ### Intermediate: Override Some Attributes ```ruby theme={} # ... Initialization code ... params = { 'hostname': 'foo.local', 'built': false, 'user_id': -1 } libhoney.add(params) builder = libhoney.builder({ 'builder': true }) # Spawn a new event and override the timestamp event = builder.event() event.add_field('user_id', 15) event.add_field('latency_ms', Time.now() - start) event.timestamp = Time.utc(2016, 2, 29, 1, 1, 1) # Timestamp is a special field that must be set via the timestamp method, it cannot be set by the add_field method event.send() ``` Check out the [documentation](https://www.rubydoc.info/gems/libhoney) for `Libhoney::Client` for more detailed API documentation. Further examples can be found [on GitHub](https://github.com/honeycombio/libhoney-rb/tree/main/examples). ## Proxy Configuration Libhoney will use any `https_proxy`, `http_proxy`, and `no_proxy` environment variables set for the process for all event traffic to Honeycomb. Proxy configuration via environment variables was added in Libhoney version 1.18.0. Proxy support was added in Libhoney version 1.14.0 and was configured via a `proxy_config` Array passed during client initialization. Array-style proxy configuration will be deprecated in Libhoney 2.0. ```ruby theme={} require 'libhoney' libhoney = Libhoney::Client.new(writekey: 'YOUR_API_KEY', dataset: 'honeycomb-ruby-example' proxy_config: ['myproxy.example.com', 8080]) # ... or if your proxy requires authentication libhoney = Libhoney::Client.new(writekey: 'YOUR_API_KEY', dataset: 'honeycomb-ruby-example' proxy_config: ['myproxy.example.com', 8080, 'username', 'password']) # ... Do work and capture events # Call close to flush any pending calls to Honeycomb libhoney.close(true) ``` ## Advanced Usage: Utilizing Builders Builders are, at their simplest, a convenient way to avoid repeating common attributes that may not apply globally. Creating a builder for a given component allows a variety of different events to be spawned and sent within the component, without having to repeat the component name as an attribute for each. You can clone builders. The cloned builder will have a copy of all the fields and dynamic fields in the original. As your application forks down into more and more specific functionality, you can create more detailed builders. The final event creation in the leaves of your application's tree will have all the data you have added along the way in addition to the specifics of this event. The global scope is essentially a specialized builder, for capturing attributes that are likely useful to all events, such as hostname, environment, and so on. Adding this kind of peripheral and normally unavailable information to every event gives you enormous power to identify patterns that would otherwise be invisible in the context of a single request. ## Advanced Usage: Dynamic Fields The top-level `libhoney` and Builders support `.add_dynamic_field(func)`. Adding a dynamic field to a Builder or top-level `libhoney` ensures that each time an event is created, the provided function is executed and the returned key/value pair is added to the event. This may be useful for including dynamic process information such as memory used, number of threads, concurrent requests, and so on to each event. Adding this kind of dynamic data to an event makes it easy to understand the application's context when looking at an individual event or error condition. ## Debugging Instrumentation If you have instrumented your code to send events to Honeycomb, you may want to verify that you are sending the events you expected at the right time with the desired fields. To support this use case, `libhoney` provides a `LogClient` that outputs events to standard error, which you can swap in for the usual `Client`. Example usage: ```ruby theme={} honeycomb = Libhoney::LogClient.new my_app = MyApp.new(..., honeycomb, ...) my_app.do_stuff # should output events to standard error ``` Note that this change will disable sending events to Honeycomb, so you will want to revert this change once you have verified that the events are coming through appropriately. ## Testing Instrumented Code Once you have instrumented your code to send events to Honeycomb, you may want to consider writing tests that verify your code is producing the events you expect, annotating them with the right information, and so on. That way, if your code changes and breaks the instrumentation, you will find out straight away, instead of at 3 a.m. when you need that data available for debugging! To support this use case, `libhoney` provides a `TestClient`, which you can swap in for the usual `Client`. Example usage: ```ruby theme={} fakehoney = Libhoney::TestClient.new my_app = MyApp.new(..., fakehoney, ...) my_app.do_stuff expect(fakehoney.events.size).to eq 3 first_event = fakehoney.events[0] expect(first_event.data['hovercraft_contents']).to eq 'Eels' ``` For more details, see the [docs](https://www.rubydoc.info/gems/libhoney) for `TestClient` and `Event`. ## Troubleshooting Refer to [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#libhoney). ## Contributions We gladly accept features, bug fixes, and other changes to `libhoney`. Please open issues or a pull request with your change. Remember to add your name to the `CONTRIBUTORS` file. All contributions will be released under the Apache License 2.0. # Send Data with the OpenTelemetry Ruby SDK Source: https://docs.honeycomb.io/send-data/ruby/opentelemetry-sdk Instrument your Ruby application with the OpenTelemetry Ruby SDK and send traces to Honeycomb. Use the OpenTelemetry Ruby SDK to instrument Ruby applications in a standard, vendor-agnostic, and future-proof way and send telemetry data to Honeycomb. In this guide, we will walk you through instrumenting with [OpenTelemetry for Ruby](https://github.com/open-telemetry/opentelemetry-ruby), which will include adding both automatic and custom, or manual, instrumentation to your applications. ## Before You Begin Before you can set up automatic and custom instrumentation for your Ruby service, you will need to do a few things. ### Prepare Your Development Environment To complete the required steps, you will need: * A working Ruby environment with Ruby 2.5 or higher * A service written in Ruby ### Get Your Honeycomb API Key To send data to Honeycomb, you'll need to [sign up for a free Honeycomb account](https://ui.honeycomb.io/signup) and [create a Honeycomb Ingest API Key](/configure/environments/manage-api-keys/#create-api-key). To get started, you can create a key that you expect to swap out when you deploy to production. Name it something helpful, perhaps noting that it's a getting started key. Make note of your API key; for security reasons, you will not be able to see the key again, and you will need it later! For setup, make sure you check the "Can create datasets" checkbox so that your data will show up in Honeycomb. Later, when you replace this key with a permanent one, you can uncheck that box. If you want to use an API key you previously stored in a secure location, you can also [look up details for Honeycomb API Keys](/configure/environments/manage-api-keys/#find-api-keys) any time in your Environment Settings, and use them to retrieve keys from your storage location. ## Add Automatic Instrumentation To enable automatic instrumentation, you will add [instrumentation packages](https://github.com/open-telemetry/opentelemetry-ruby-contrib/tree/main/instrumentation#opentelemetry-instrumentation-libraries) to your application. ### Acquire Dependencies Add these gems to your Gemfile: ```ruby theme={} gem 'opentelemetry-sdk' gem 'opentelemetry-exporter-otlp' gem 'opentelemetry-instrumentation-all' ``` | Gem | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `opentelemetry-sdk` | Required to create spans | | `opentelemetry-exporter-otlp` | An exporter to send data in the OTLP format | | `opentelemetry-instrumentation-all` | A meta package that provides instrumentation for Rails, Sinatra, several HTTP libraries, [and more](https://github.com/open-telemetry/opentelemetry-ruby#instrumentation-libraries) | Install the gems using your terminal: ```shell theme={} bundle install ``` ### Initialize Initialize OpenTelemetry early in your application lifecycle. For Rails applications, we recommend that you use a Rails initializer. For other Ruby services, initialize as early as possible in the startup process. ```ruby theme={} # config/initializers/opentelemetry.rb require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' OpenTelemetry::SDK.configure do |c| c.use_all() # enables all instrumentation! end ``` ### Configure the OpenTelemetry SDK Use environment variables to configure OpenTelemetry to send events to Honeycomb: ```ruby theme={} export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io" # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu1.honeycomb.io" # EU instance export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key" export OTEL_SERVICE_NAME="your-service-name" ``` | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Base endpoint to which you want to send your telemetry data. | | `OTEL_EXPORTER_OTLP_HEADERS` | List of headers to apply to all outgoing telemetry data. Place your API Key generated in Honeycomb in the `x-honeycomb-team` header. [Learn how to find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys). | | `OTEL_SERVICE_NAME` | Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. | If you are a [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) user, the Dataset also must be specified using the `x-honeycomb-dataset` header. A **Dataset** is a bucket where data gets stored in Honeycomb. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset" ``` ### Run Run your Ruby application. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. ## Add Custom Instrumentation Automatic instrumentation is the easiest way to get started with instrumenting your code, but to get the most insight into your system, you should add custom, or manual, instrumentation where appropriate. To do this, use the OpenTelemetry SDK to access the currently executing span and add attributes to it, and/or to create new spans. To add custom instrumentation, you need to add the OpenTelemetry SDK gem to your Gemfile: ```ruby theme={} gem 'opentelemetry-sdk' ``` ### Acquire a Tracer To create spans, you need to get a `Tracer`. ```ruby theme={} require "opentelemetry/sdk" tracer = OpenTelemetry.tracer_provider.tracer("tracer.name.here") ``` When you create a `Tracer`, OpenTelemetry requires you to give it a name as a string. This string is the only required parameter. When traces are sent to Honeycomb, the name of the `Tracer` is turned into the `library.name` field, which can be used to show all spans created from a particular tracer. In general, pick a name that matches the appropriate scope for your traces. If you have one tracer for each service, then use the service name. If you have multiple tracers that live in different "layers" of your application, then use the name that corresponds to that "layer". The `library.name` field is also used with traces created from instrumentation libraries. ### Add Attributes to Spans It is often helpful to add information to a currently executing span in a trace. For example, you may have an application or service that handles users, and you want to associate the user with the span when querying your service in Honeycomb. To do this, get the current span from the context and set an attribute with the user ID: ```ruby theme={} # somewhere within the service, the SDK has been required and configured require 'opentelemetry/sdk' OpenTelemetry::SDK.configure do ... end # ... def handle_user(user) current_span = OpenTelemetry::Trace.current_span current_span.set_attribute("user.id", user.id) end ``` This will add a `user.id` field to the current span so that you can use the field in `WHERE`, `GROUP BY`, or `ORDER` clauses in the Honeycomb query builder. ### Create New Spans Automatic instrumentation can show the shape of requests to your system, but only you know the really important parts. To get the full picture of what is happening, you will have to add custom instrumentation and create some custom spans. To do this, grab the tracer from the OpenTelemetry API, and use it to create spans and add attributes. ```ruby theme={} # somewhere within the service, the SDK has been required and configured require 'opentelemetry/sdk' OpenTelemetry::SDK.configure do ... end # ... def run_query tracer = OpenTelemetry.tracer_provider.tracer('my-tracer') tracer.in_span("expensive-query") do |span| # ... cool stuff span.set_attribute('coolness', 100) end end ``` ### Add Multi-Span Attributes Sometimes you want to add the same attribute to many spans within the same trace. This attribute may include variables calculated during your program, or other useful values for correlation or debugging purposes. To add this attribute to multiple spans, leverage the OpenTelemetry concept of [baggage](https://opentelemetry.io/docs/concepts/signals/baggage/). Baggage allows you to add a `key` with a `value` as an attribute to every subsequent child span of the current application context, as long as you configured a `BaggageSpanProcessor` when you [initialized OpenTelemetry](#initialize). 1. Install the baggage span processor gem using your terminal: ```shell theme={} gem install opentelemetry-processor-baggage ``` 2. When configuring the OpenTelemetry SDK tracer provider, add the baggage span processor: ```ruby theme={} require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/processor/baggage' OpenTelemetry::SDK.configure do |c| # Add the BaggageSpanProcessor to the collection of span processors c.add_span_processor(OpenTelemetry::Processor::Baggage::BaggageSpanProcessor.new) # Because the span processor list is no longer empty, the SDK will not use the # values in OTEL_TRACES_EXPORTER to instantiate exporters. # You'll need to declare your own values here in the configure block. # # These lines setup the default: a batching OTLP exporter. c.add_span_processor( # these constructors without arguments will pull config from the environment OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new( OpenTelemetry::Exporter::OTLP::Exporter.new() ) ) end ``` 3. Add a baggage entry for the current trace and replace `key` and `value` with your desired key-value pair: ```ruby theme={} require 'opentelemetry/sdk' context_with_baggage = OpenTelemetry::Baggage.set_value('key', 'value') tracer.in_span('test-span', with_parent: context_with_baggage) do |span| # ... cool stuff end ``` Any Baggage attributes that you set in your application will be attached to outgoing network requests as a header. If your service communicates to a third party API, do **NOT** put sensitive information in the Baggage attributes. ## Sampling You can configure the OpenTelemetry SDK to [sample the data](/manage-data-volume/sample/guidelines/) it generates. Honeycomb [weights sampled data based on sample rate](/manage-data-volume/sample/sampled-data-in-honeycomb/), so you must set a resource attribute containing the sample rate. Use a [`TraceIdRatioBased` sampler](https://opentelemetry.io//docs/specs/otel/trace/sdk/#traceidratiobased), with a ratio expressed as `1/N`. Then, also create a resource attribute called `SampleRate` with the value of `N`. This allows Honeycomb to reweigh scalar values, like counts, so that they are accurate even with sampled data. In the example below, our goal is to keep approximately half (1/2) of the data volume. The resource attribute contains the denominator (2), while the OpenTelemetry sampler argument contains the decimal value (0.5). ```shell theme={} export OTEL_TRACES_SAMPLER="traceidratio" export OTEL_TRACES_SAMPLER_ARG=0.5 export OTEL_RESOURCE_ATTRIBUTES="SampleRate=2" ``` If you have multiple services that communicate with each other, it is important that they have the same sampling configuration. Otherwise, each service might make a different sampling decision, resulting in incomplete or broken traces. You can sample using a standalone proxy as an alternative, like [Honeycomb Refinery](/manage-data-volume/sample/honeycomb-refinery/), or when you have more robust sampling needs. ## Choosing between gRPC and HTTP Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind: * Some SDKs default to using gRPC, and it may be easiest to start with the default option. * Some firewall policies are not set up to handle gRPC and require using HTTP. * gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery. gRPC default export uses port 4317, whereas HTTP default export uses port 4318. ## Endpoint URLs for OTLP/HTTP When using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable with an SDK and an HTTP exporter, the final path of the endpoint is modified by the SDK to represent the specific signal being sent. For example, when exporting trace data, the endpoint is updated to append `v1/traces`. When exporting metrics data, the endpoint is updated to append `v1/metrics`. So, if you were to set the `OTEL_EXPORTER_OTLP_ENDPOINT` to `https://api.honeycomb.io`, traces would be sent to `https://api.honeycomb.io/v1/traces` and metrics would be sent to `https://api.honeycomb.io/v1/metrics`. The same modification is not necessary for gRPC. ```shell theme={} export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.eu1.honeycomb.io # EU instance ``` If the desired outcome is to send data to a different endpoint depending on the signal, use `OTEL_EXPORTER_OTLP__ENDPOINT` instead of the more generic `OTEL_EXPORTER_OTLP_ENDPOINT`. When using a signal-specific environment variable, these paths must be appended manually. Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` for traces, appending the endpoint with `v1/traces`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` for metrics, appending the endpoint with `v1/metrics`. Send both traces and metrics to Honeycomb using this method by setting the following variables: ```shell theme={} export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces # US instance #export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.eu1.honeycomb.io/v1/traces # EU instance export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics # US instance #export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.eu1.honeycomb.io/v1/metrics # EU instance ``` More details about endpoints and signals can be found in the [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md). ## Distributed Trace Propagation When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace. Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header. Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment. ## Troubleshooting To explore common issues when sending data, visit [Common Issues with Sending Data in Honeycomb](/troubleshoot/common-issues/sending-data/#opentelemetry-sdks-and-honeycomb-distributions). # Add Context Source: https://docs.honeycomb.io/send-data/standardize/add-context Add metadata to your traces using span annotations to capture additional context, attach performance metrics, and correlate data across distributed systems. To provide additional context and insight into what is happening in your data, you may want to enhance your traces by adding metadata using span annotations. Adding span annotations makes it easier to understand what’s happening inside a distributed system and lets you troubleshoot performance issues or failures more efficiently. ## Annotating Spans A span annotation is a type of event used within tracing data to add granular observability without creating additional spans. You can use span annotations to: * Attach additional context to a span at a particular moment in time * Establish relationships between spans that are part of different trace hierarchies In Honeycomb, span annotations appear as tabbed entries in a selected span's tracing sidebar (rather than in the trace's waterfall graph). ### Attach Context at a Particular Moment in Time Span annotations offer a lightweight way to provide more detailed insights into specific moments within a span, which can help you troubleshoot, analyze performance, and improve your overall understanding of complex systems. A span event is a type of span annotation that lets you attach additional context or metadata to a span at a particular moment in time. More specifically, you can use span events to record data that correlates with logs, metrics, and other error monitoring tools. #### Uses You may want to use span event annotations to: * Capture significant moments or key milestones. For example, record the start and end of a sub-operation (such as a database query or external API call), business logic transitions (such as customer authentication or payment verification), or state changes in an application (such as cache hit or miss). * Improve debugging and troubleshooting. For example, when an error occurs, log the event with detailed information about the failure, such as exception messages, stack traces, or other diagnostic data. * Add more granular observability without creating extra spans. For example, when processing a request, instead of creating a new span for every minor step, such as reading headers or validating a token, use span events to mark these steps. * Perform time-based analysis. For example, measure latency between events, the duration of intermediate steps in a span, or the sequence of execution. * Enhance context and logging. For example, add HTTP status codes at the time of a response or add user IDs relevant to an action. * Correlate with external systems or actions. For example, record external service responses, notifications received, or security events. **Scenario: Record errors produced inside a loop** **Scenario:** You have a span that represents a specific operation in your service. That operation has a loop, inside of which a non-fatal error could occur. You want to record any errors that are produced inside of the loop. **Solution:** Attach each error event as a span event annotation on the span that represents the operation. Doing so will let you capture all of the errors. Conversely, if you write the error to an error field on the span that represents the operation, then you will overwrite any errors from previous loop iterations already recorded in the field. #### Adding Span Events **General Rules** * Span events are timestamped, structured events without a duration (aka logs). * Associate span events with a timestamp (just as you do with other events) * Attach span events only to spans that are regular trace spans * Span events are not counted towards the total span count for the trace to which they belong. * To appear in the [Trace Sidebar](/investigate/analyze/explore-traces/#trace-sidebar), span events must belong to a dataset that has been configured. To learn how to configure datasets for traces, visit [Dataset Definitions: Configure](/configure/datasets/definitions/). To add a span event annotation, create an event that includes the following key/value pairs: | Honeycomb Field Name | Description | | ------------------------- | ------------------------------------------------------------------------------------- | | `meta.annotation_type` | Type of annotation. To identify an annotation as a span event, set to `"span_event"`. | | `trace.parent_id` | ID of the span to which you want to attach this span event. | | `trace.trace_id` | ID of the trace to which you want this span event to belong. | | `name` | Name of the span event. | | `service.name` (optional) | Name of the service in which the span event occurred. | Remember to [map the source fields in your sent data to Honeycomb's standard fields](/send-data/standardize/map-data/). ### Establish Relationships Between Spans in Different Trace Hierarchies Although the general tracing data model, which focuses on the parent-child relationship between spans, can adequately describe most spans, in special cases, you may need to describe a less direct, causal relationship between spans. A span link is a type of span annotation that lets you establish relationships between spans that are part of different trace hierarchies. #### Uses You may want to use span link annotations to: * Handle asynchronous or parallel operations in distributed systems. For example, in a service that initiates multiple background tasks (each of which may create its own trace), use span links to associate the traces created by background tasks back to the initiating span. * Correlate related traces across complex systems, in which traces are split across different services or organizations. For example, when integrating with external APIs or microservices, links spans from different traces to connect requests and responses that exist in separate distributed tracing systems. * Track the relationship between fan-out and fan-in phases. For example, in a data processing pipeline where different stages run in parallel, use span links to show how the final aggregation step is related to earlier individual processing spans. * Track the flow of work in event-driven architectures where spans may not have direct parent-child relationships. For example, in a message queue system, where one service produces a message and another service consumes it later, use span links to relate the decoupled producer and consumer spans to give context about the entire message lifecycle. * Accurately model the relationships in complex workflows with multiple initiators. For example, in a system where multiple services can trigger a batch processing job and each service has its own trace, use span links to link the batch processing job's span to all of the initiating spans to show how multiple inputs led to the same job. * Merge traces from different requests that logically contribute to the same end result. For example, if two separate user actions trigger the same downstream process, use span links to link those separate user request traces to the same result span. * Track and correlate tracing and retry/fallback logic. For example, if an HTTP request fails and the system retries, use span links to connect the retry spans to the original span, so you get full visibility into the retry logic. #### Adding Span Links **General Rules** * Span links are optional. * Span links can point to spans within the same trace or spans in different traces. * Each span link points to one other span. * Each span can have one or many span links attached to it. To add a span link annotation, create an event that includes the following key/value pairs: | Honeycomb Field Name | Description | | ---------------------- | ----------------------------------------------------------------------------------- | | `meta.annotation_type` | Type of annotation. To identify an annotation as a Span Link, set to `"link"`. | | `trace.parent_id` | ID of the span from which you wish to link. The link will be attached to this span. | | `trace.trace_id` | ID of the trace from which you wish to link. The link will belong to this trace. | | `trace.link.span_id` | ID of the span to which you wish to link. | | `trace.link.trace_id` | ID of the trace to which you wish to link. | Remember to [map the source fields in your sent data to Honeycomb's standard fields](/send-data/standardize/map-data/). # Add Custom Instrumentation Source: https://docs.honeycomb.io/send-data/standardize/add-custom-instrumentation Extend automatic OpenTelemetry instrumentation with custom spans, attributes, and events to capture the specific application behavior you need in Honeycomb. The quickest way to start seeing your trace data in Honeycomb is to use [OpenTelemetry](https://opentelemetry.io/), an open-source collection of tools, APIs, and SDKs, to automatically inject instrumentation code into your application without requiring explicit changes to your codebase. But to get the most insight into your system, you should also add custom instrumentation. For example, if you have an application or service that handles users, you probably want to associate the user with the span created by automatic instrumentation. Additionally, because automatic instrumentation doesn't know your business logic--it only knows about frameworks and languages--you'll want to explicitly add instrumentation code surrounding your business logic. These instructions will guide you through the process of customizing your instrumentation to get additional visibility into the inner workings of your business logic. To learn more about adding automatic instrumentation to your application, visit [Start Building: Honeycomb for Applications](/get-started/start-building/application/). When deciding what to instrument, we recommend thinking of this as an exploratory process and part of the [core analysis loop](/get-started/honeycomb/core-analysis-loop). You don't need to add your custom instrumentation to everything all at once. Consider your top interests and instrument them. Then analyze your results and add additional code as necessary. ## Acquiring Dependencies Before you can customize your instrumentation, you must install any required dependencies. These will vary depending on your chosen programming language. To continue, you must install OpenTelemetry's API package. Once you have installed the API package, you can use it to add instrumentation for everything you care about in your application. This Quick Start uses the `npm` dependency manager. For instructions with `yarn` or if using TypeScript, read our [OpenTelemetry Node.js documentation](/send-data/javascript-nodejs/opentelemetry-sdk/#add-custom-instrumentation). ```shell theme={} npm install --save @opentelemetry/api ``` To continue, you must install OpenTelemetry's API package. Once you have installed the API package, you can use it to add instrumentation for everything you care about in your application. This Quick Start uses the `pip` package manager. For instructions with `poetry`, read our [OpenTelemetry Python documentation](/send-data/python/opentelemetry-sdk/#add-custom-instrumentation). ```shell theme={} python -m pip install opentelemetry-api ``` To continue, you must install the OpenTelemetry API. Once you have installed the API, you can use it to add instrumentation for everything you care about in your application. This Quick Start uses the Gradle build tool. ```groovy theme={} dependencies { // Replace '{opentelemetry_java_instrumentation.version}' below with the version of the OTel JavaAgent in use. implementation(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom:{opentelemetry_java_instrumentation.version}")) implementation("io.opentelemetry:opentelemetry-api") implementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations") } ``` This Quick Start uses ASP.NET Core. For other .NET configuration options, visit our [OpenTelemetry for .NET SDK](/send-data/dotnet/) documentation. Because you already installed all necessary dependencies when implementing automatic instrumentation, you can skip this step. Because you already installed all necessary dependencies when implementing automatic instrumentation, you can skip this step. Because you already installed all necessary dependencies when implementing automatic instrumentation, you can skip this step. If your preferred language is not covered here, you can find relevant instrumentation information in the [OpenTelemetry community documentation](https://opentelemetry.io/docs/languages/). ## Adding Contextual Information to Existing Spans To enhance the understanding of your telemetry data, you can attach contextual information to the spans generated by automatic instrumentation. Contextual information includes attributes, which are arbitrary key-value pairs that you can add to provide additional data about the span, such as a specific database query being executed or an HTTP request being made. In this example, you will enhance your automatic instrumentation by associating a span created automatically when a user queries your service with the user's ID, which will be saved in an attribute called `user.id`. Once this attribute is added, you will be able to use it to query your data in Honeycomb. ```js theme={} const { trace } = require("@opentelemetry/api"); //... function handleUser(user) { // Gets the active span from the context let activeSpan = trace.getActiveSpan(); // Sets an attribute that contains the user ID activeSpan.setAttribute("user.id", user.getId()); } ``` ```python theme={} from opentelemetry import trace #... # Gets the active span from the context span = trace.get_current_span() # Sets an attribute that contains the user ID span.set_attribute("user.id", user.id()) ``` ```java theme={} import io.opentelemetry.api.trace.Span; //... // Gets the active span from the context Span span = Span.current(); // Sets an attribute that contains the user ID span.setAttribute("user.id", user.getId()); ``` ```csharp theme={} using OpenTelemetry.Trace; //... // Gets the active span from the context var currentSpan = Tracer.CurrentSpan; // Sets an attribute that contains the user ID currentSpan.SetAttribute("user.id", User.GetUserId()) ``` This example assumes you are writing a web application with the `net/http` package. ```go theme={} import ( // ... "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" // ... ) // ... handler := func(w http.ResponseWriter, r *http.Request) { // Gets the currently logged in user user := someServiceCall() ctx := r.Context() // Gets the active span from the context span := trace.SpanFromContext(ctx) // Sets an attribute that contains the user ID span.SetAttributes(attribute.Int("user.id", user.getID())) } // ... ``` ```ruby theme={} # Somewhere within the service, the SDK has been required and configured require 'opentelemetry/sdk' OpenTelemetry::SDK.configure do ... end # ... def handle_user(user) # Gets the active span from the context current_span = OpenTelemetry::Trace.current_span # Sets an attribute that contains the user ID current_span.set_attribute("user.id", user.id) end ``` If your preferred language is not covered here, you can find relevant instrumentation information in the [OpenTelemetry community documentation](https://opentelemetry.io/docs/languages/). ## Creating Custom Spans To further enhance the understanding of your telemetry data, you'll want to generate custom spans. Automatic instrumentation can show the shape of requests to your system, but only you know the truly important parts. To get the full picture, you should manually add spans to areas of interest. In OpenTelemetry, this is called manual instrumentation. ### Initialize a Tracer Before you can create a custom span, you must initialize a tracer. In OpenTelemetry, a tracer is used to instrument your code and generate telemetry data. The tracer provides the interface for capturing spans, which represent individual units of work in your code. ```js theme={} const { trace } = require("@opentelemetry/api"); // Creates the tracer and names it. // Choose a name that matches the appropriate scope for your trace. // For example, if you have one tracer for each service, then use // the service name. If you have multiple tracers that live in // different "layers" of your application, then user the name that // corresponds to the appropriate "layer". const tracer = trace.getTracer("your.tracer.name"); ``` ```python theme={} from opentelemetry import trace #... # Creates the tracer and names it. # Choose a name that matches the appropriate scope for your trace. # For example, if you have one tracer for each service, then use # the service name. If you have multiple tracers that live in # different "layers" of your application, then user the name that # corresponds to the appropriate "layer". tracer = trace.get_tracer("tracer.name.here") ``` ```java theme={} import io.opentelemetry.api.GlobalOpenTelemetry; import io.opentelemetry.api.trace.Tracer; //... // Creates the tracer and names it. // Choose a name that matches the appropriate scope for your trace. // For example, if you have one tracer for each service, then use // the service name. If you have multiple tracers that live in // different "layers" of your application, then user the name that // corresponds to the appropriate "layer". Tracer tracer = GlobalOpenTelemetry.getTracer("tracer.name.here"); ``` ```csharp theme={} using OpenTelemetry.Trace; //... // Creates the tracer and names it. // Choose a name that matches the appropriate scope for your trace. // For example, if you have one tracer for each service, then use // the service name. If you have multiple tracers that live in // different "layers" of your application, then user the name that // corresponds to the appropriate "layer". var tracer = TracerProvider.Default.GetTracer("tracer.name.here"); ``` Then, inject the `Tracer` instance with ASP.NET Core dependency injection or manage its lifecycle manually. ```go theme={} import ( // ... "go.opentelemetry.io/otel" // ... ) //... // Creates the tracer and names it. // Choose a name that matches the appropriate scope for your trace. // For example, if you have one tracer for each service, then use // the service name. If you have multiple tracers that live in // different "layers" of your application, then user the name that // corresponds to the appropriate "layer". tracer := otel.Tracer("tracer.name.here") ``` ```ruby theme={} require "opentelemetry/sdk" #... # Creates the tracer and names it. # Choose a name that matches the appropriate scope for your trace. # For example, if you have one tracer for each service, then use # the service name. If you have multiple tracers that live in # different "layers" of your application, then user the name that # corresponds to the appropriate "layer". tracer = OpenTelemetry.tracer_provider.tracer("tracer.name.here") ``` If your preferred language is not covered here, you can find relevant instrumentation information in the [OpenTelemetry community documentation](https://opentelemetry.io/docs/languages/). ### Create Your Span Get the tracer from the OpenTelemetry API and use it to create your custom span. ```js theme={} const { trace } = require("@opentelemetry/api"); const tracer = trace.getTracer("my-service-tracer"); function runQuery() { tracer.startActiveSpan("expensive-query", (span) => { // Do cool stuff span.end(); }); } ``` ```python theme={} from opentelemetry import trace tracer = trace.get_tracer("tracer.name.here") with tracer.start_as_current_span("some-long-running-handler"): # Prepare to auth with tracer.start_as_current_span("what-it-takes-to-authorize-handling"): # Perform auth # Do the authorized long-running handling of cool stuff ``` ```java theme={} import io.opentelemetry.api.GlobalOpenTelemetry; import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; ... Tracer tracer = GlobalOpenTelemetry.getTracer("my-service"); Span span = tracer.spanBuilder("expensive-query").startSpan(); // ... Do cool stuff span.end(); ``` ```csharp theme={} using OpenTelemetry.Trace; //... using var span = TracerProvider.Default.GetTracer("my-service").StartActiveSpan("expensive-query") // ... Do cool stuff ``` ```go theme={} import ( // ... "go.opentelemetry.io/otel" // ... ) // ... tracer := otel.Tracer("my-app") // If not already in scope ctx, span := tracer.Start(ctx, "expensive-operation") defer span.End() // ... ``` ```ruby theme={} # Somewhere within the service, the SDK has been required and configured require 'opentelemetry/sdk' OpenTelemetry::SDK.configure do ... end # ... def run_query tracer = OpenTelemetry.tracer_provider.tracer('my-tracer') tracer.in_span("expensive-query") do |span| # ... Do cool stuff span.set_attribute('coolness', 100) end end ``` If your preferred language is not covered here, you can find relevant instrumentation information in the [OpenTelemetry community documentation](https://opentelemetry.io/docs/languages/). ## Analyzing Your Results Now that you have customized your instrumentation, take a look at your trace. We generated the following examples after adding additional custom spans (a note to indicate where we are in the process, plus `person name` and `message`) to our simple greeting service application written in Node.js. You can [see the code in our GitHub repository](https://github.com/brianlangbecker/best-practices-blog). Here's our example trace: Trace showing custom spans implemented in our example application

Trace showing custom spans implemented in our example application

In this trace, you can see the additional spans: * note indicating we are preparing the greeting * call name * call message If you look in the **Fields** section of the UI, you would also see new attributes, such as the name of the person and the message that was sent. ## What's Next? Excellent work! If you made it this far, you should now have additional visibility into the inner workings of your business logic. But there is so much more to explore! To learn more about what you can do with Honeycomb, check out: * [Ways to Explore Your Data](/observe/): Get a quick run-through of the different ways you can explore your data in Honeycomb. * [Board Templates](/observe/boards/): Get key insights with one-click with out-of-the-box Board Templates. * [Honeycomb's Sandbox](https://play.honeycomb.io/sandbox/tours): Explore common scenarios with real data. * [DevRel Office Hours](https://www.honeycomb.io/devrel/observability-office-hours/): Join in on observability talk with world-class experts. * [Integrations](/integrations/): Learn about other types of data you can explore with Honeycomb. # Map Your Data Source: https://docs.honeycomb.io/send-data/standardize/map-data Map your dataset fields to Honeycomb's standard fields so trace waterfall views, log visualizations, and other features work correctly. Honeycomb automatically maps source data fields for you if you follow Honeycomb field naming conventions. Otherwise, you must map your dataset fields manually. Honeycomb automatically maps source data fields if you follow Honeycomb field naming conventions. If your field names differ from the standard names, or if you have already sent data and can't change field names in your source, you can map fields manually using Dataset Definitions. Unmapped data is the most common cause of empty visualizations. If you see an unpopulated trace waterfall or log view, check your dataset definitions first. ## Trace dataset fields Honeycomb identifies a dataset as a trace dataset if you send all of these fields to Honeycomb: * `trace.trace_id` * `trace.span_id` * `duration_ms` If your field names differ, [map them manually](#mapping-fields-manually) in **Dataset Definitions**. | Dataset Field | Description | Allowed Type(s) | Source Field Name | Can Group By? | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------- | ------------- | | **Error** | Value that indicates an error occurred. Used to identify errors when calculating the "Total Spans with Errors" visualizations. | boolean, string, [calculated field](/configure/environments/calculated-fields/) | `error` | Yes | | **HTTP Status Code** | Code that indicates the success, failure, or other status of a request. | string, integer, float, [calculated field](/configure/environments/calculated-fields/) | `response.status_code` | Yes | | **Metadata: Annotation Type** | Type of the span annotation, which controls how Honeycomb visualizes this event in a trace. Possible values include `span_event` (indicates the span is a [span event](/send-data/standardize/add-context/#attach-context-at-a-particular-moment-in-time)) and `link` (indicates the span is a [span link](/send-data/standardize/add-context/#establish-relationships-between-spans-in-different-trace-hierarchies)). | string, [calculated field](/configure/environments/calculated-fields/) | `meta.annotation_type` | No | | **Metadata: Link Span ID** | Unique ID of the span you want to link to the current span. Indicates the span is a span event. Use with **Metadata: Link Trace ID**. | string, [calculated field](/configure/environments/calculated-fields/) | `trace.link.span_id` | No | | **Metadata: Link Trace ID** | Unique ID of the trace you want to link to the current span. Use with **Metadata: Link Span ID**. | string, [calculated field](/configure/environments/calculated-fields/) | `trace.link.trace_id` | No | | **Name** | Name of the function or method in which the span was created. | string, [calculated field](/configure/environments/calculated-fields/) | `name` | Yes | | **Parent span ID** | ID of the span's parent span, which is the location from which the span was called. | string, [calculated field](/configure/environments/calculated-fields/) | `trace.parent_id` | No | | **Route** | HTTP URL or equivalent route processed by the request. | string, [calculated field](/configure/environments/calculated-fields/) | `http.route` | Yes | | **Service name** | Name of the instrumented service. | string, [calculated field](/configure/environments/calculated-fields/) | `service.name` | No | | **Span duration** | Length of time the span took in milliseconds (ms). Used to calculate the "95th Percentile Latency" visualizations. | float, integer, [calculated field](/configure/environments/calculated-fields/) | `duration_ms` or `durationMs` | No | | **Span ID** | Unique ID of the span. | string, [calculated field](/configure/environments/calculated-fields/) | `trace.span_id` | No | | **Trace ID** | Unique ID of the trace to which the span belongs. | string, [calculated field](/configure/environments/calculated-fields/) | `trace.trace_id` | No | | **User** | User making the request in the system. | float, string, integer, [calculated field](/configure/environments/calculated-fields/) | `user.id` | Yes | Let Honeycomb detect and map dataset fields automatically by naming your data fields using the values in the **Source Field Name** column. Fields without a source field name must be [mapped manually](#mapping-fields-manually). ## Log dataset fields Honeycomb identifies a dataset as a logs dataset if you send a field named `body` to Honeycomb. If your field name differs, [map it manually](#mapping-fields-manually) in **Dataset Definitions**. | Dataset Field | Description | Allowed Type(s) | Source Field Name | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ----------------- | | **Log Message** | Value containing the log event message. May be a human-readable string (including multiline) describing the event in free form. | string, [calculated field](/configure/environments/calculated-fields/) | `body` | | **Log Severity** | Severity level of the event (also known as log level). Supported values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `unspecified`. | string, [calculated field](/configure/environments/calculated-fields/) | `severity` | Let Honeycomb detect and map dataset fields automatically by naming your data fields using the values in the **Source Field Name** column. Fields without a source field name must be [mapped manually](#mapping-fields-manually). ## Mapping visualization components Honeycomb visualizations rely on fields in your source data being mapped to Honeycomb standard fields. If you ever see an unpopulated visualization in Honeycomb, check your dataset definitions first! Unmapped data is the main cause of visualizations missing data. ### Trace dropdowns Each trace chart visualization's accompanying dropdown is populated with source fields mapped to Honeycomb dataset fields. You can [map your source fields](#mapping-fields-manually) to the selected dataset's fields using [Dataset Definitions](/configure/datasets/definitions/). Dataset fields you can map include: * **Error**: Value that indicates an error occurred. * **HTTP Status Code**: Code that indicates the success, failure, or other status of a request. * **Name**: Name of the function or method in which the span was created. * **Route**: HTTP URL or equivalent route processed by the request. * **User**: User making the request in the system. For datasets with Kubernetes metadata, the following fields are also included in the dropdown if available: * `k8s.pod.name`, `source.k8s.pod.name`, or `destination.k8s.pod.name`: Name of the Kubernetes pod. * `k8s.container.name`, `source.k8s.container.name`, or `destination.k8s.container.name`: Name of the Kubernetes container. * `k8s.node.name`, `source.k8s.node.name`, or `destination.k8s.node.name`: Name of the Kubernetes node. * `k8s.namespace.name`, `source.k8s.namespace.name`, or `destination.k8s.namespace.name`: Name of the Kubernetes namespace. * `source.k8s.service.name` or `destination.k8s.service.name`: Name of the Kubernetes service. To learn more about sending Kubernetes data to Honeycomb, visit [Send Data from Kubernetes](/send-data/kubernetes/). ### Exception fields Exception visualizations are populated by [OpenTelemetry Exception](https://opentelemetry.io/docs/specs/otel/trace/exceptions/) source fields. If your exception visualizations are empty, verify that your instrumentation includes the `exception.type` and `exception.message` fields. ## Mapping fields manually If you have already sent data into a Honeycomb dataset or you are unable to change field names in your source data, use [Dataset Definitions](/configure/datasets/definitions/) to manually map the standard fields in your Honeycomb dataset to the source fields in your data: 1. Select **Manage Data** () from the navigation menu, and choose **Datasets**. 2. Select the name of your target dataset to access its settings. 3. Select the **Definitions** view. 4. Locate the appropriate Honeycomb **Dataset Field**, then select the source field in your data from the corresponding **Field name** dropdown. 5. Select **Update**. # Transform Your Data Source: https://docs.honeycomb.io/send-data/standardize/transform-data Transform your data without making permanent changes to your dataset by defining calculated fields in Honeycomb. In Honeycomb, you can define unique calculated fields that help you flexibly transform your data without making permanent changes to your dataset. ## What are Calculated Fields? Otherwise known as Derived Columns, calculated fields are computed properties that are calculated by a formula. The field's value is based on the result of the formula, which can contain functions, mathematical and/or logical operations on other field values, and constants and literals--similar to expressions defined in a spreadsheet. To learn more about syntax and available functions, visit [Calculated Field Formula Reference](/reference/calculated-field-expression/). ## Transforming Data with Calculated Fields In Honeycomb, you can use calculated fields to create new data fields based on existing ones, so you can perform advanced data transformation. Then, you can use calculated fields in Honeycomb just as you would use other fields in your data, such as when [building queries](/investigate/query/build/), defining [Triggers](/notify/triggers/), and creating [Service Level Objectives (SLOs)](/notify/slos/). When transforming data, you can use calculated fields to: * cleanse data * create boolean flags and indicators * categorize data * calculate a new value, so you can express an event detail in a more human-readable way * transform your source data values to Honeycomb standard values (for example, [log severities](#transform-source-severities-into-honeycomb-standard-severities)) Honeycomb calculates calculated fields dynamically whenever you run a query, which ensures the value of your field is always based on the latest data. When planning your transformations, remember that Honeycomb does not allow you to nest calculated fields. ### Cleansing Data If you have non-standard data, you can use calculated fields to cleanse or format fields values. This example standardizes text in a field by converting it to all lowercase: ```ruby theme={} TO_LOWER($service.name) ``` ### Creating Boolean Flags or Indicators You can use calculated fields to create flags for easier filtering and analysis. This example creates a flag for HTTP statuses greater than or equal to `400`: ```ruby theme={} IF(GTE($http_status, 400), 1, 0) ``` ### Categorizing Data You can use calculated fields to classify data based on conditions for easier segmentation. This example classifies API version based on URL fragment: ```ruby theme={} IF(CONTAINS($url, "/v1/"), "api_v1", "api_v2") ``` ### Calculating New Values You can use calculated fields to calculate values based on existing fields to create new metrics. This example calculates event latency and converts the unit from seconds to minutes. ```ruby theme={} DIV(SUB(INGEST_TIMESTAMP(),EVENT_TIMESTAMP()),60) ``` ### Transforming Source Data Values into Honeycomb Standard Values In Honeycomb, certain visualizations rely on Honeycomb-specific, standard values. For example, the full experience of log visualization relies on Honeycomb standard severity values being populated. You can use calculated fields to transform your source data values to Honeycomb standard values without making permanent changes to your dataset. #### Transform Source Severities into Honeycomb Standard Severities If you are sending log source data to Honeycomb and the data you are sending contains severities other than the standard log severities expected by Honeycomb, Honeycomb visualizations that depend on severity may not populate. To solve this issue, use a calculated field to transform any non-standard severities to Honeycomb standard log severities. The standard Honeycomb log severities include `info`, `error`, `warn`, `fatal`, `trace`, `debug`, and `unspecified`. **Scenario:** You have a source field named `my.severity` that contains values of `a`, `b`, `c`, `d`, `e`, `f`, and `g`. You want the values to map like so: | Your Severity | Honeycomb Standard Log Severity | | ------------- | ------------------------------- | | `a` | `info` | | `b` | `error` | | `c` | `warn` | | `d` | `fatal` | | `e` | `trace` | | `f` | `debug` | | `g` | `unspecified` | Any other severity values should be recognized as non-standard and labeled as `Other` by Honeycomb. **Solution:** Use a Calculated Field to parse your severity values and transform them to Honeycomb standard log severity values. 1. [Create a Calculated Field](/configure/datasets/calculated-fields/#creating-calculated-fields), and enter the following function in the **Editor**: ```ruby theme={} IF( REG_MATCH($my.severity, "a"), "info", REG_MATCH($my.severity, "b"), "error", REG_MATCH($my.severity, "c"), "warn", REG_MATCH($my.severity, "d"), "fatal", REG_MATCH($my.severity, "e"), "trace", REG_MATCH($my.severity, "f"), "debug", REG_MATCH($my.severity, "g"), "unspecified", "Other" ) ``` 2. Use your Calculated Field as the source field, and map it to the **Logs: Severity** dataset field when you [map your data](/send-data/standardize/map-data/). Your severities will now map to Honeycomb standard log severities, and visualizations that depend on severity should populate. ### Parsing Unstructured Logs Because of the shape of unstructured log data in Honeycomb, unstructured logs can be hard to query. To improve the experience and make unstructured logs more useful, use calculated fields to parse meaningful information out of the message attribute. 1. [Create a calculated field](#creating-calculated-fields). 2. Leverage the [`REG_VALUE` operation](/reference/calculated-field-expression/string#reg_value) to parse log data using regular expressions. ## Creating Calculated Fields You can create calculated fields at both the Environment and the Dataset level. All calculated fields are applied at the Dataset level, but Environment-level calculated fields are shared and available for you to use across all Datasets in your Environment. To create a calculated field: 1. In the Honeycomb UI, select the **Environments** label from the navigation sidebar, then select **Manage Environments**. 2. In the list, locate the Environment to which you would like to add a Calculated Field, and select its name to view the available settings. 3. Navigate to the **Schema** view. 4. Select **Add New Calculated Field**. 5. In the modal, enter a **Display Name**, which will show as the field name in the Query Builder. Enter a display name that is unique across the containing Environment. Your Calculated Field name should not match the name of any other Calculated Field or any other field in any Dataset contained within the Environment. Although Honeycomb tries to prevent duplicate field names, they can still occur. To learn more about behaviors related to name collision and solutions, visit [Common Issues with Queries: Calculated Fields](/troubleshoot/common-issues/queries/#a-link-points-to-the-wrong-location). 6. In the editor, define the formula for your Calculated Field. To learn more about syntax and available functions, and to explore some example formulas, visit [Calculated Field Formula Reference](/reference/calculated-field-expression/). You can also generate a calculated field expression using natural language. Enter your prompt and select **Generate**. Hover over any syntax errors (red underlines or red triangles) for assistance with correcting them. 7. Select **Save**. # Honeycomb Telemetry Pipeline Source: https://docs.honeycomb.io/send-data/telemetry-pipeline Collect, refine, and route metrics, logs, and traces to any destination with the Honeycomb Telemetry Pipeline, an OpenTelemetry-native pipeline tool. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. For an open-source alternative, visit [OpenTelemetry Collector](/send-data/opentelemetry/collector/). The Honeycomb Telemetry Pipeline (HTP) is a powerful, OpenTelemetry-native platform designed to ingest, process, and transform observability data at scale. Built for the complex needs of modern, high-volume systems, HTP provides flexible and efficient telemetry management. With HTP, your team can extract meaningful insights from your data and control costs without losing visibility. HTP is composed of three primary components: * [Pipeline Manager](#pipeline-manager) * [Enhance](#enhance) * [Pipeline health monitoring](#pipeline-health) You can use any or all of these components. ## Pipeline Manager Pipeline Manager is an enterprise-grade platform for managing telemetry pipelines at scale. Pipeline Manager is ideal for teams needing a reliable, agent-based data collection tool with centralized fleet management capabilities. Learn about the Honeycomb Telemetry Pipeline Manager. ## Enhance Enhance enables you to store full-fidelity data in your own AWS S3 buckets and rehydrate it into Honeycomb on-demand within the UI. It is built to address pain points like sampling gaps, log proliferation, and costly third-party archival tools. Store telemetry data in Amazon S3 and rehydrate it on-demand in Honeycomb. Use indexed fields for faster queries at lower cost. ## Pipeline Health Free, non-billable environment ($pipeline-telemetry$) included with HTP for monitoring operational telemetry for Refinery and Collectors. Requires E\&S (not Classic) * How to send data to it: * Environment is enabled by account team on HTP purchase * Customer gets an ingest key for their Pipeline Telemetry environment * Send operational data from Refinery * Set REFINERY\_HONEYCOMB\_API\_KEY env var in Refinery config to that ingest key * How to observe * Refinery 3.0 board template for refinery * OTel Collector Ops board template ### Pipeline Telemetry Environment The Pipeline Telemetry Environment is a full-featured Honeycomb Environment that does not count against your event quotas. Because it is a full Honeycomb Environment, you can: * Create custom queries and Boards to analyze pipeline telemetry. * Select any chart title to explore the underlying query. * Set up SLOs to monitor pipeline performance. * Configure Triggers to alert on pipeline issues. * Use Canvas to ask questions about your pipeline data. For example, you might create an SLO that alerts when your pipeline stops receiving data or set up a Trigger to notify you when error rates increase. Access the Pipeline Telemetry Environment from the **Environment** selector in the navigation menu, just as you would any other Environment. # Pipeline Manager Destinations Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations Browse available destinations for the Honeycomb Telemetry Pipeline Manager to route telemetry data to Honeycomb, S3, and other targets. Enterprise add-on. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The following destinations are available for the Honeycomb Telemetry Pipeline Manager: | Destination | Logs | Metrics | Traces | Persistent Queuing | Proxy | | :------------------------------------------------------------------------------------------------------------- | :--- | :------ | :----- | :----------------- | :---- | | [AWS S3](/send-data/telemetry-pipeline/destinations/aws-s3/) | ✓ | ✓ | ✓ | | ✓ | | [Azure Blob](/send-data/telemetry-pipeline/destinations/azure-blob/) | ✓ | ✓ | ✓ | ✓ | ✓ | | [Honeycomb](/send-data/telemetry-pipeline/destinations/honeycomb/) | ✓ | ✓ | ✓ | ✓ | ✓ | | [Honeycomb Enhance Indexing AWS S3](/send-data/telemetry-pipeline/destinations/honeycomb-enhance-indexing-s3/) | ✓ | ✓ | | ✓ | ✓ | | [Honeycomb Refinery](/send-data/telemetry-pipeline/destinations/honeycomb-refinery/) | ✓ | | ✓ | ✓ | ✓ | | [Bindplane Gateway](/send-data/telemetry-pipeline/destinations/bindplane-gateway) | ✓ | ✓ | ✓ | ✓ | ✓ | # AWS S3 Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations/aws-s3 Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | The AWS S3 destination saves telemetry into timestamped JSON files in an S3 bucket.\ ## Configuration | Parameter | Type | Default | Description | | :----------------- | :------------------ | :------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | telemetry\_types\* | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | Specifies which types of telemetry to export. | | region\* | `enum` | us-east-1 | AWS region of the bucket to export telemetry to. | | bucket\* | `string` | "" | Name of the S3 Bucket to export telemetry into. | | prefix | `string` | "" | The root directory of the bucket to export telemetry into. | | file\_prefix | `string` | "" | Prefix for the name of exported telemetry files. | | granularity\* | `enum` | minute | The granularity of the timestamps in the S3 key, either "minute" or "hour". | | compression | `string` | "" | Whether or not data should be uploaded and stored in S3 as compressed data. Valid values are "gzip" or no value set. | | storage\_class | `enum` | "STANDARD" | The storage class to store telemetry data in. Read this AWS [documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html#sc-compare) for more info. | | endpoint | `string` | "" | The endpoint to use for the S3 destination. Overrides the endpoint constructed from the region and bucket values. Generally not needed. | | marshaler\_logs | `enum` | "otlp\_json" | Determines the format of data sent to AWS S3. See [Marshaler Configuration](#marshaler-configuration) for more details. | | marshaler\_metrics | `enum` | "otlp\_json" | Determines the format of data sent to AWS S3. See [Marshaler Configuration](#marshaler-configuration) for more details. | | marshaler\_traces | `enum` | "otlp\_json" | Determines the format of data sent to AWS S3. See [Marshaler Configuration](#marshaler-configuration) for more details. | \* *required field* ### Marshaler Configuration The destinations supports configuring the marshaler for each telemetry type. The marshaler determines how the data is formatted before being sent to AWS S3. By default, the `otlp_json` marshaler is used for all telemetry types. | Marshaler Type | Metrics | Logs | Traces | Description | | :------------- | :------ | :--- | :----- | :----------------------------------------------------------------- | | `otlp_json` | ✓ | ✓ | ✓ | Uses the OpenTelemetry Protocol (OTLP) in JSON format. | | `otlp_proto` | ✓ | ✓ | ✓ | Uses the OpenTelemetry Protocol (OTLP) in Protocol Buffers format. | | `sumo_ic` | | ✓ | | Uses the Sumo Logic Ingestion format. | | `body` | | ✓ | | Export the log body as string. | Note that `sumo_ic` and `body` marshaler types are only applicable for logs. ## Credentials With AWS S3, users are required to provide some form of authentication. There are two ways to configure this: either entering profile credentials manually or using the CLI, or environment variables that specify access keys for user accounts.\ In the [AWS CLI getting started guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) it will instruct you to install for your current user or all users.\ The observIQ OTel Collector runs [as root](https://github.com/observIQ/observiq-otel-collector/blob/main/docs/installation-linux.md#configuring-the-collector) by default, meaning the AWS CLI and credentials should be installed under the collector system's root account. ### CLI The easiest way to configure this is using the `aws` CLI program provided by AWS. The [AWS CLI Getting Started](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) guide describes how to install the CLI and configure it with credentials. The AWS S3 destination uploads telemetry to the specified bucket, so the credentials configured should be associated with an account that has `s3:PutObject` permissions for that bucket. ### Environment Variables Alternatively, [AWS Environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) can be specified to override a credentials file. You can modify the collector's environment variables by configuring a `systemd` override.\ Run `sudo systemctl edit observiq-otel-collector` and add your access key, secret key, and region: ```text theme={} [Service] Environment=AWS_ACCESS_KEY_ID=****** Environment=AWS_SECRET_ACCESS_KEY=****** Environment=AWS_DEFAULT_REGION=us-east-1 ``` After making that change, reload Systemd and restart the collector service. ```bash theme={} sudo systemctl daemon-reload sudo systemctl restart observiq-otel-collector ``` # Azure Blob Storage Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations/azure-blob Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | The Azure Blob Storage destination saves telemetry as [OTLP JSON](https://github.com/open-telemetry/opentelemetry-proto#otlpjson) files in Azure Blob Storage. ## Configuration | Parameter | Type | Default | Description | | :------------------- | :------------------ | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types\* | `telemetrySelector` | Logs, Metrics, Traces | Specifies which types of telemetry to export. | | connection\_string\* | `string` | "" | The connection string for the Azure Storage account. More information can be found [here](https://learn.microsoft.com/en-gb/azure/storage/common/storage-configure-connection-string). | | container\* | `string` | "" | Name of the Azure Storage container to export telemetry into. | | prefix | `string` | "" | The root directory of the blob path to export telemetry into. | | blob\_prefix | `string` | "" | Prefix for the name of the exported telemetry files. | | partition\* | `enum` | minute | The granularity of the timestamps in the blob path, either "minute" or "hour". | | compression | `enum` | `gzip` | The compression algorithm to use when exporting telemetry, either "none" or "gzip" | \* *required field* ## Supported Retry and Queuing Settings This destination supports the following retry and queuing settings: | Sending Queue | Persistent Queue | Retry on Failure | | :------------ | :--------------- | :--------------- | | ✓ | ✓ | ✓ | ## Example Configuration ### Basic Configuration For basic configuration, we specify the `connection_string` as well as a `container`, `prefix`, and `blob_prefix`. By default, we are still using a `partition` of `minute` and `gzip` for the `compression` value. This will create a blob path that looks like the following inside the `otel` container: ```text theme={} telemetry/year=2021/month=01/day=01/hour=01/minute=00/bindplanemetrics_{random_id}.json.gz telemetry/year=2021/month=01/day=01/hour=01/minute=00/bindplanelogs_{random_id}.json.gz telemetry/year=2021/month=01/day=01/hour=01/minute=00/bindplanetraces_{random_id}.json.gz ``` #### Web Interface Honeycomb Docs - Azure Blob Storage - image 1

Honeycomb Docs - Azure Blob Storage - image 1

Honeycomb Docs - Azure Blob Storage - image 2

#### Standalone Destination ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: id: azure_blob name: azure_blob spec: type: azure_blob parameters: - name: telemetry_types value: ['Logs', 'Metrics', 'Traces'] - name: connection_string value: 'DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=+idLkHYcL0MUWIKYHm2j4Q==;EndpointSuffix=core.windows.net' - name: container value: 'otel' - name: prefix value: 'telemetry' - name: blob_prefix value: bindplane - name: partition value: 'hour' - name: compression value: 'none' ``` # Bindplane Gateway Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations/bindplane-gateway Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Bindplane Gateway destination is an OTLP destination meant to be used for nodes sending to a gateway. When using this destination in conjunction with a Telemetry Pipeline Gateway source from another configuration, telemetry traveling through this destination will not be double counted in the Summary view. ## Supported Types | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.52.0`+ | ## Configuration Table | Field | Description | | :--------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Hostname | Hostname or IP address where the exporter will send OTLP data. | | Port | TCP port to which the exporter is going to send OTLP data. | | Protocol | The OTLP protocol to use when sending OTLP telemetry. Can be gRPC or HTTP. | | Compression | Compression algorithm to use when sending data to the OTLP server. Ensure that the server supports the compression algorithm selected. Kinds of compression depend on `Protocol`. | | Additional Headers | Add additional headers to be attached to each request. | | Enable TLS | Whether or not to use TLS. | | Skip TLS Certificate Verification | Enable to skip TLS certificate verification. | | TLS Certificate Authority File | Certificate authority used to validate TLS certificates. | | Server Name Override | Optional virtual hostname. Indicates the name of the server requested by the client. This option is generally not required. Read more [here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#client-configuration). | | Mutual TLS | Whether or not to use mutual TLS authentication. | | Mutual TLS Client Certificate File | A TLS certificate used for client authentication. | | Mutual TLS Client Private Key File | A TLS private key used for client authentication. | ## Supported Retry and Queuing Settings This destination supports the following retry and queuing settings: | Sending Queue | Persistent Queue | Retry on Failure | | :------------ | :--------------- | :--------------- | | ✓ | ✓ | ✓ | ## Example Configuration In this configuration, we specify the `hostname` of the Telemetry Pipeline Gateway server telemetry is going to be sent to, as well as what `protocol` will be used. ### Web Interface Honeycomb Docs - BindPlane Gateway Destination - image 1

Honeycomb Docs - BindPlane Gateway Destination - image 1

### Standalone Destination ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: id: bindplane_gateway name: bindplane_gateway spec: type: bindplane_gateway parameters: - name: hostname value: '0.0.0.0' - name: protocol value: 'grpc' - name: grpc_port value: '4317' - name: enable_tls value: 'false' ``` # Honeycomb Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations/honeycomb Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description Sends logs, metrics, and traces to Honeycomb using the OTLP exporter. ## Prerequisites A Honeycomb account, team, environment, and API key will need to be created before being able to send telemetry with this destination. See our [Honeycomb Quick Start guide](/get-started/start-building/application/) for more information. With a Honeycomb environment and API key setup, you can configure this destination to send to Honeycomb. Note that the API key will need the "Send Events" permission. If configuring this destination with a dataset that does not exist, the API key will also need the "Create Dataset" permission to work properly. ## Supported Types | Logs | Metrics | Traces | HTP Agent | | :--- | :------ | :----- | :--------- | | ✓ | ✓ | ✓ | `v1.36.0`+ | ## Configuration | Field | Description | | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Choose Telemetry Type | The kind of telemetry that should be sent to Honeycomb. | | API Key | The API key to use for sending telemetry to Honeycomb. Make sure the key has the "Send Events" permission. If the provided dataset(s) do not exist, use the "Create Dataset" permission as well. See our [Honeycomb documentation](/configure/environments/manage-api-keys/) for more. | | Metrics Dataset | Honeycomb dataset that metrics will be sent to. Will be created if it does not exist. See our [Honeycomb documentation](/send-data/opentelemetry/collector/#metrics-and-logs-signals) for more. | | Logs Dataset | Honeycomb dataset that logs will be sent to. Will be created if it does not exist. If not set, the "service.name" log resource attribute will be used. See our [Honeycomb documentation](/send-data/opentelemetry/collector/#metrics-and-logs-signals) for more. | | Protocol | The OTLP protocol to use when sending to Honeycomb. Can be `HTTP` or `gRPC`. See our [Honeycomb documentation](/send-data/opentelemetry/collector/#choosing-between-grpc-and-http) for more. | | Compression | Compression algorithm to use when sending data to Honeycomb. Available options are `none` or `gzip`. | | Additional Headers | Add additional headers to be attached to requests. | This destination supports the following retry and queuing settings: | Sending Queue | Persistent Queue | Retry on Failure | | :------------ | :--------------- | :--------------- | | ✓ | ✓ | ✓ | ## Honeycomb Datasets The Honeycomb destination can be configured to route telemetry to specific Honeycomb datasets, however, Refinery will respect the `service.name` resource field when determining which dataset the telemetry should belong to. You can route telemetry to datasets dynamically by using the [Add Fields](/send-data/telemetry-pipeline/processors/add-fields/) processor to set `service.name` resource field. ## Example Configuration For this configuration we'll configure to send logs, metrics, and traces. We specify an API key, Metrics Dataset, and Logs Dataset. In the `Advanced` section we specify gRPC protocol and gzip compression. We'll also enable Retry on Failure, Sending Queue, and Persistent Queueing. ### Web Interface Honeycomb Docs - Honeycomb Destination - image 1

Honeycomb Docs - Honeycomb Destination - image 1

Honeycomb Docs - Honeycomb Destination - image 3

### Standalone Destination ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: id: honeycomb name: honeycomb spec: type: honeycomb parameters: - name: telemetry_types value: ['Logs', 'Metrics', 'Traces'] - name: api_key value: 'some_api_key' - name: metrics_dataset value: 'bindplane_metrics' - name: logs_dataset value: 'bindplane_logs' - name: protocol value: 'grpc' - name: compression value: 'gzip' - name: retry_on_failure_enabled value: true - name: sending_queue_enabled value: true - name: persistent_queue_enabled value: true ``` # Honeycomb Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations/honeycomb-enhance-indexing-s3 Export logs and traces to AWS S3 with automatic field indexing for efficient rehydration and querying in Honeycomb. ## Supported Telemetry Types | Telemetry Type | Supported | | -------------- | --------- | | Traces | ✓ | | Metrics | | | Logs | ✓ | ### Prerequisites * A Honeycomb account with a Management API key and secret (must have `enhance:write` scope) * An AWS S3 bucket with appropriate write permissions * AWS credentials configured on the collector host * Network connectivity from the collector to both S3 and the Honeycomb API endpoint ### Configuration #### Basic Configuration The minimum configuration requires Honeycomb API credentials, an S3 bucket, and an AWS region. | Parameter | Type | Default | Description | Required | | ---------------- | -------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------- | | Telemetry Types | Selector | `Logs`, `Traces` | Select which types of telemetry to export. | Yes | | API Key | String | | Honeycomb Management API key with `enhance:write` scope. | Yes | | API Secret | String | | Honeycomb Management API secret. | Yes | | API Endpoint | String | `https://api.honeycomb.io` | Honeycomb API endpoint URL. | Yes | | Region | Enum | `us-east-1` | AWS region where the S3 bucket is located. | Yes | | Bucket | String | | Name of the S3 bucket to export telemetry into. | Yes | | Folder Prefix | String | | Root directory of the bucket to export telemetry into. | No | | Partition Format | String | `year=%Y/month=%m/day=%d/hour=%H/minute=%M` | Time-based partition format using ctime format codes. Must contain year, month, day, hour, and minute placeholders. | Yes | #### Advanced Configuration **Data Format and Compression** | Parameter | Type | Default | Description | Required | | ----------- | ---- | --------------- | --------------------------------------------------------------------------------------------------------------------------------- | -------- | | Data Format | Enum | `otlp_protobuf` | Format of telemetry data written to S3. Options: `otlp_protobuf` (smaller files, more efficient) or `otlp_json` (human-readable). | No | | Compression | Enum | `gzip` | Compression applied to data stored in S3. Options: `gzip` or `none`. | No | **S3 Connection Options** | Parameter | Type | Default | Description | Required | | ---------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------- | -------- | | Endpoint | String | | Overrides the S3 endpoint URL. Use this for S3-compatible services like MinIO. | No | | Force Path Style | Boolean | `false` | Force path-style S3 addressing instead of virtual-host style. Required for some S3-compatible services. | No | | Disable SSL | Boolean | `false` | Disable SSL verification for S3 connections. | No | **Indexing** | Parameter | Type | Default | Description | Required | | --------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------ | -------- | | Custom Indexed Fields | Strings | | Additional fields to index beyond the automatic ones (`trace.trace_id`, `service.name`, `session.id`). | No | To minimize Collector processing compute and time, Honeycomb recommends indexing no more than 5 custom fields. **General Settings** | Parameter | Type | Default | Description | Required | | --------------- | ------- | ---------- | ---------------------------------------------------------------------------------- | -------- | | Drop Raw Copy | Boolean | `true` | Drop the raw copy of the log record stored in `log.record.original`. | No | | Timeout | Integer | `30` | Maximum time in seconds to wait for a batch to be sent. | No | | Retry Mode | Enum | `standard` | S3 client retry strategy. Options: `standard` or `adaptive`. | No | | Maximum Retries | Integer | `3` | Maximum number of retry attempts for S3 uploads. Set to `0` for unlimited retries. | No | **Retry and Queuing** This destination supports the [retry settings](https://docs.bindplane.com/configuration/bindplane-otel-collector/retry-on-failure), the [sending queue settings](https://docs.bindplane.com/configuration/bindplane-otel-collector/sending-queue), and the [persistent queue settings](https://docs.bindplane.com/configuration/bindplane-otel-collector/persistent-queue). ### Configuration Tips #### S3 Bucket and Credentials * AWS credentials must be available to the collector via IAM role, environment variables, or shared credentials file. The destination does not accept inline AWS credentials. * Make sure the S3 bucket policy grants `s3:PutObject` permission to the collector's identity. * Use the Folder Prefix to isolate telemetry data within a shared bucket. #### Custom Indexed Fields * Fields like `user.id` or `environment` can be added to Custom Indexed Fields for faster rehydration queries in Honeycomb. * Each additional indexed field increases S3 storage usage due to extra index files written alongside data files. #### Honeycomb API Endpoint * Use `https://api.honeycomb.io` for US-based Honeycomb accounts. * Use `https://api.eu1.honeycomb.io` for EU-based Honeycomb accounts. ## Troubleshooting ### S3 Upload Failures Errors referencing `AccessDenied`, `NoSuchBucket`, or `InvalidAccessKeyId` in collector logs. 1. Verify the S3 bucket exists in the configured region and the collector host has valid AWS credentials with `s3:PutObject` permission. 2. If using a custom Endpoint for S3-compatible storage, make sure Force Path Style is enabled. ### Honeycomb API Authentication Errors Errors referencing `401 Unauthorized` or `403 Forbidden` from the Honeycomb API endpoint. 1. Confirm the API Key and API Secret are for a Management API key with the `enhance:write` scope. Standard ingest keys will not work. 2. Verify the API Endpoint matches your Honeycomb account region (US vs EU). ### Missing or Incomplete Index Files Data appears in S3 but index files are missing or incomplete. Rehydration queries in Honeycomb return fewer results than expected. 1. Check that the partition format contains all required placeholders (`%Y`, `%m`, `%d`, `%H`, `%M`). Missing placeholders will cause validation failures. 2. Verify that custom indexed field names match the exact attribute keys present in your telemetry data. ## See also * [Collector Export to S3 Archive](/send-data/telemetry-pipeline/enhance/collector-export-s3/) * [Managing API Keys](/configure/teams/manage-api-keys/) * [Enhance Indexing S3 Exporter (GitHub)](https://github.com/honeycombio/enhance-indexing-s3-exporter) # Honeycomb Refinery Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/destinations/honeycomb-refinery Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description Sends logs and traces to [Honeycomb Refinery](/manage-data-volume/sample/honeycomb-refinery/). ## Prerequisites This destination requires network access to a [Honeycomb Refinery](/manage-data-volume/sample/honeycomb-refinery/) deployment. If Refinery is configured to forward telemetry to Honeycomb.io, an API key is required. The API key should have "Send Events" and "Create Dataset" permissions. See the [Honeycomb Refinery](/manage-data-volume/sample/honeycomb-refinery/) and [Honeycomb quick start guide](/get-started/start-building/application/traces/) for more information. ## Supported Types | Logs | Metrics | Traces | HTP Agent | | :--- | :------ | :----- | :--------- | | ✓ | | ✓ | `v1.60.0`+ | ## Configuration | Field | Description | | :--------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Choose Telemetry Type | The kind of telemetry that should be sent to Honeycomb. | | Hostname | Hostname or IP address where the exporter will send OTLP data. | | Port | TCP port to which the exporter is going to send OTLP data. | | API Key | The API key to use for sending telemetry to Honeycomb. Make sure the key has the "Send Events" permission. If the provided dataset(s) do not exist, use the "Create Dataset" permission as well. See this [Honeycomb documentation](/configure/environments/manage-api-keys/) for more. | | Compression | Compression algorithm to use when sending data to Honeycomb. Available options are `none` or `gzip`. | | Enable TLS | Whether or not to use TLS. | | Skip TLS Certificate Verification | Enable to skip TLS certificate verification. | | TLS Certificate Authority File | Certificate authority used to validate TLS certificates. | | Server Name Override | Optional virtual hostname. Indicates the name of the server requested by the client. This option is generally not required. Read more [here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#client-configuration). | | Mutual TLS | Whether or not to use mutual TLS | | Mutual TLS Client Certificate File | A TLS certificate used for client authentication. | | Mutual TLS Client Private Key File | A TLS private key used for client authentication. | This destination supports the following retry and queuing settings: | Sending Queue | Persistent Queue | Retry on Failure | | :------------ | :--------------- | :--------------- | | ✓ | ✓ | ✓ | ## Honeycomb Datasets Your Refinery server can be configured to route telemetry to specific Honeycomb datasets, however, Refinery will respect the `service.name` resource field when determining which dataset the telemetry should belong to. You can route telemetry to datasets dynamically by using the [Add Fields](/send-data/telemetry-pipeline/processors/add-fields/) processor to set `service.name` resource field. ## Example Configuration ### Web Interface Honeycomb Docs - Honeycomb Refinery Destination - image 1

Honeycomb Docs - Honeycomb Refinery Destination - image 1

### Standalone Destination ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Destination metadata: name: honeycomb-refinery spec: type: honeycomb_refinery_otlp parameters: - name: telemetry_types value: ['Logs', 'Traces'] - name: hostname value: refinery.default.svc.cluster.local - grpc_port: value: 4317 - name: api_key value: your_api_key - name: compression value: gzip ``` # Enhance with Archive Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/enhance Store full-fidelity, unsampled telemetry in Amazon S3 and rehydrate it on-demand in Honeycomb for deep-dive investigations beyond your retention window. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. Honeycomb supports rehydrating archived, unsampled OpenTelemetry (OTel) trace and log data from your own Amazon S3 bucket. With Enhance, you can: * Rehydrate unsampled traces or archived logs for deep-dive investigations. * Avoid expensive up-front storage costs by keeping data in cold storage until needed. * Use indexed fields to filter and retrieve only the data you need, reducing rehydration time and costs. * [Query Archived Data](/investigate/query/archive/) to automatically fetch and query relevant data directly in the Honeycomb UI. ## Set Up an Archive Use the AWS Management Console to configure your Amazon S3 bucket for trace and log rehydration. Use Terraform to configure your Amazon S3 bucket for trace and log rehydration. ## Export Unsampled Data to Your Archive Export unsampled telemetry data to your Amazon S3 archive using an OpenTelemetry Collector with field-based indexing. Export logs and traces to Amazon S3 using the Pipeline Manager. # Set Up an Amazon S3 Archive with AWS Console Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/enhance/aws-console-setup Use the AWS Management Console to configure your Amazon S3 bucket for trace and log rehydration in Honeycomb. Ent+ Use the AWS Management Console to create the IAM policy and role that Honeycomb uses to access your S3 archive for trace and log rehydration. This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. To enable trace and log rehydration from your Amazon S3 archive, you will need an AWS IAM role that includes: * Permissions to list and retrieve objects from your S3 bucket. * A trust policy that allows the Honeycomb IAM role to assume this role when interacting with your archive. This guide walks you through creating these resources in the AWS Management Console. If you prefer, you can [use Terraform instead](/send-data/telemetry-pipeline/enhance/terraform-setup/). After you create the IAM policy and role, share the details with your Honeycomb account team, so they can complete your S3 archive setup. ## Create an IAM Policy First, create an IAM policy with permissions for listing and retrieving objects from your Amazon S3 bucket: 1. Sign in to the AWS Management Console and [open the IAM console](https://console.aws.amazon.com/iam/). 2. In the navigation pane, choose **Policies**. 3. Select **Create policy**. 4. In the Policy editor, choose the JSON option. 5. Copy the JSON policy that corresponds to your encryption method, then paste it into the editor: Replace the placeholder ARNs in the `Resource` element with the actual ARNs for your bucket. ```json theme={} { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAccessToArchiveBucket", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::/*", "arn:aws:s3:::" ] } ] } ``` Replace the placeholder ARNs in the `Resource` elements with the actual ARNs for your bucket and AWS KMS key. For the KMS key, specify the full key, not an alias. ```json theme={} { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAccessToArchiveBucket", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::/*", "arn:aws:s3:::" ] }, { "Sid": "AllowKMSDecryptionOfArchiveBucket", "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": [ "arn:aws:kms:::key/" ] } ] } ``` 6. Fix any warnings or errors if necessary, then select **Next**. 7. Provide a name and optional description for the policy. Remember the policy name; you will use it when you create the IAM role. 8. Select **Create Policy**. ## Create the IAM Role Next, create an IAM role that can access your Amazon S3 bucket, and authorize Honeycomb to assume that role when interacting with your bucket: 1. Sign in to the AWS Management Console and [open the IAM console](https://console.aws.amazon.com/iam/). 2. In the navigation pane, select **Roles**, and then choose **Create role**. 3. For the role type, select **Custom trust policy**. 4. In the Custom trust policy section, use the appropriate trust policy, depending on your team's Honeycomb instance: **US:** ```json theme={} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::702835727665:role/production-eks-bulk-ingest-role" }, "Action": "sts:AssumeRole" } ] } ``` **EU:** ```json theme={} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::919259170365:role/production-eu1-eks-bulk-ingest-role" }, "Action": "sts:AssumeRole" } ] } ``` 5. Select **Next**. 6. In the **Permissions policies** section, search for the name of the IAM policy that you created in the previous section. 7. Select the checkbox for that IAM policy in the results. 8. Skip the permissions boundary: Screenshot of the optional set permission boundaries step.

Screenshot of the optional set permission boundaries step.

9. Select **Next**. 10. Provide a name and optional description for the IAM role. 11. Select **Create Role**. **KMS Users: Check Your Key Policy** If the data in your S3 bucket is encrypted with an AWS KMS encryption key, ensure that the corresponding [KMS Key Policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) allows this IAM Role to use the key to perform `kms:Decrypt` actions. If the IAM Role does not have permission to decrypt using the KMS key, rehydration will fail. ## Share with Your Honeycomb Account Team To complete your setup, share the following details with your Honeycomb account team: * The name of your AWS S3 bucket * The ARN of the IAM role that will interact with the bucket To locate the IAM Role's ARN: 1. In the IAM service's navigation pane in the console, choose **Roles**. 2. Search for the name of the IAM role you created, then choose the role from the results. 3. In the **Summary** section, locate **ARN**, and use the copy icon to copy the ARN value to your clipboard. * `s3_prefix` from your OpenTelemetry Collector exporter configuration, if configured * `s3_partition_format` from your OpenTelemetry Collector exporter configuration, if configured * `indexed_fields` from your OpenTelemetry Collector exporter configuration, if you configured custom indexed fields # Collector Export to Amazon S3 Archive Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/enhance/collector-export-s3 Configure an OpenTelemetry Collector to export unsampled trace and log data to Amazon S3 with field-based indexing for fast, cost-effective rehydration in Honeycomb. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. Honeycomb supports rehydrating archived, unsampled OpenTelemetry (OTel) trace and log data stored in an Amazon S3 bucket. In this guide, you will learn how to configure an OpenTelemetry Collector to export unsampled trace and log data to an Amazon S3 bucket using Honeycomb's [Enhance Indexing S3 Exporter](https://github.com/honeycombio/enhance-indexing-s3-exporter/tree/main/enhanceindexings3exporter/README.md). This exporter extends the standard [OpenTelemetry AWS S3 Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awss3exporter/README.md) by adding field-based indexing, so you can rehydrate data more quickly and cost-effectively. Adding AWS Credentials requires assistance from your Honeycomb account team. To learn more, visit [Set Up an Archive](/send-data/telemetry-pipeline/enhance/#set-up-an-archive). ## How Indexing Works When you send data to your S3 archive, the exporter automatically indexes these fields: * `trace.trace_id`: Unique identifier of the trace * `service.name`: Name of the service * `session.id`: Unique identifier of the session You can also index additional fields that you frequently query, such as `user.id`, `customer.id`, or `environment`. When you run a query that requires rehydration, you can use these indexes to locate and retrieve only the relevant subset of your archived data. This gives you: * **Faster rehydration**: Retrieve only the events that match your query. * **Lower costs**: Reduce the amount of archived data you need to ingest. * **Better performance**: Investigate with a smaller, more targeted dataset. ## Configuring the Enhance Indexing S3 Exporter The [Honeycomb OpenTelemetry Collector distribution](https://github.com/honeycombio/honeycomb-collector-distro) includes the [Enhance Indexing S3 Exporter](https://github.com/honeycombio/enhance-indexing-s3-exporter/tree/main/enhanceindexings3exporter/README.md) by default. If you're using another collector distribution or building your own, make sure it is built with the Enhance Indexing S3 Exporter. The exporter requires configuration in these areas. ### Honeycomb API Credentials The exporter uses a Honeycomb Management API key with the `enhance:write` scope for authentication and usage tracking. Configure these fields in your exporter configuration: * `api_key`: Your Honeycomb Management API key. Must have the `enhance:write` scope. * `api_secret`: Your Honeycomb Management API secret. * `api_endpoint`: URL of the Honeycomb API endpoint. * US: `https://api.honeycomb.io` * EU: `https://api.eu1.honeycomb.io` To learn how to create a Management API key, visit [Managing API Keys](/configure/teams/manage-api-keys/). ### S3 Uploader Settings Set these attributes in the `s3uploader` section: * `s3_bucket`: Amazon S3 bucket in which to store the data. Not required when `endpoint` is set. * `region`: AWS region for your bucket. Default: `us-east-1`. * `s3_prefix`: (Optional) Directory-like prefix for organizing files in your S3 bucket. Example: `traces-logs-directory`. * `s3_partition_format`: (Optional) Partition format to use when writing files to S3. Default: `"year=%Y/month=%m/day=%d/hour=%H/minute=%M"` (minute-level resolution). * `compression`: Compression algorithm for S3 files. Options: * `gzip` * `none` * `retry_mode`: (Optional) Retry behavior for failed requests. Default: `standard`. Options: * `standard` (fixed intervals) * `adaptive` (adjusts based on server response) * `retry_max_attempts`: (Optional) Maximum number of times to retry a failed attempt. Default: `3`. * `endpoint`: (Optional) Custom S3 endpoint for S3-compatible services, such as MinIO or on-premises object storage. Overrides the default AWS S3 endpoint. Example: `http://localhost:9000`. * `s3_force_path_style`: (Optional) Use path-style rather than virtual-hosted style URLs. Required for MinIO and some S3-compatible services. Default: `false`. * `disable_ssl`: (Optional) Allow unencrypted HTTP connections. Use only for local development. Default: `false`. The `file_prefix` attribute is not supported and will cause validation to fail. ### Data Format Settings Choose how telemetry is encoded before being written to S3: * `marshaler`: Format for encoding telemetry data. Default: `otlp_proto`. Options: * `otlp_proto` (smaller files, better performance) * `otlp_json` (human-readable output). ### Batching and Queue Settings Batch data before sending to S3 by configuring the `sending_queue` section: * `flush_timeout`: Maximum time before a batch is sent to S3, even if not full. Must be a non-zero value. * `min_size`: Minimum size of a batch. Measured in units defined by `sizer`. Default: `50000` * `max_size`: Maximum size of a batch. Enables batch splitting. Must be greater than or equal to `min_size`. Measured in units defined by `sizer`. Default: `50000` * `queue_size`: Maximum number the queue can accept. Measured in units defined by `sizer`. Default: `500000`. * `sizer`: Unit used to measure the queue and batch size. Default: `items`. Options: * `requests`: Number of incoming batches of traces and logs (the most performant option). * `items`: Number of the smallest parts of each signal (spans, log records). * `bytes`: Size of serialized data in bytes (the least performant option). You can also set a custom timeout for the exporter: * `timeout`: Maximum time to wait for each S3 send attempt. Default: `30s`. ### Retry Settings Control how the exporter retries failed send attempts by configuring the `retry_on_failure` section: * `enabled`: Allow the exporter to retry failed sends. * `initial_interval`: Amount of time to delay before the first retry. Example: `5s` * `max_interval`: Maximum amount of time to delay. Retries use exponential backoff, so each retry waits longer than the previous one. Example: `30s` ### Custom Indexed Fields Add indexed fields beyond the built-in ones (`trace.trace_id`, `service.name`, `session.id`) using the `indexed_fields` section. Choose fields that are frequently used in your queries or that help narrow down your investigations. High-cardinality fields are especially useful because they make rehydration more selective. Examples: * `user.id` * `customer.id` * `environment` * `deployment.version` To minimize Collector processing compute and time, we recommend indexing no more than 5 custom fields. ```yaml theme={} indexed_fields: - "user.id" - "customer.id" - "environment" ``` Share your indexed fields with your Honeycomb account team, along with your S3 bucket information and IAM role ARN. This ensures that rehydration is configured correctly. ## Examples These examples show common configurations, so you can choose the setup that best fits your needs. ### Basic Configuration A basic example of the Enhance Indexing S3 Exporter configuration: ```yaml theme={} exporters: enhance_indexing_s3_exporter: # Honeycomb API credentials api_key: ${env:HONEYCOMB_MANAGEMENT_API_KEY} api_secret: ${env:HONEYCOMB_MANAGEMENT_API_SECRET} api_endpoint: https://api.honeycomb.io # S3 configuration s3uploader: region: 'us-east-1' s3_bucket: 'my-test-bucket' s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H/minute=%M" compression: 'gzip' # Data format marshaler: 'otlp_proto' ``` ### Pipeline Configuration Honeycomb supports processing both trace and log OpenTelemetry signal types for data archival and rehydration. Set up a separate pipeline configuration block for each signal type. In this example, pipelines for logs and traces are labeled to indicate pipelines intended for object storage (storage in Amazon S3): ```yaml theme={} service: pipelines: traces: // [...] logs: // [...] logs/objectstorage: exporters: - enhance_indexing_s3_exporter receivers: - otlp traces/objectstorage: exporters: - enhance_indexing_s3_exporter receivers: - otlp ``` ### Full Configuration The example below shows a simple, but complete OpenTelemetry (OTel) Collector configuration for exporting both log and trace OTel signal types through the Enhance Indexing S3 Exporter: * Trace and log telemetry is exported to an Amazon S3 bucket called `my-test-bucket`. * An S3 prefix is used at the root of the bucket called `telemetry-data`. * Data will be partitioned into subdirectories underneath the prefix with minute resolution. * Files uploaded to S3 will be compressed with gzip compression. * Custom indexed fields are configured for `user.id`, `customer.id`, `environment`, and `deployment_version`. * Queue batching is configured. ```yaml theme={} receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: enhance_indexing_s3_exporter: # Honeycomb API credentials api_key: ${env:HONEYCOMB_MANAGEMENT_API_KEY} api_secret: ${env:HONEYCOMB_MANAGEMENT_API_SECRET} api_endpoint: https://api.honeycomb.io #S3 configuration s3uploader: region: 'us-east-1' s3_bucket: 'my-test-bucket' s3_prefix: 'telemetry-data' s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H/minute=%M" compression: 'gzip' retry_mode: 'adaptive' retry_max_attempts: 5 # Data format marshaler: 'otlp_proto' # Custom indexed fields indexed_fields: - "user.id" - "customer.id" - "environment" - "deployment.version" # Batching, timeout, and retry configuration sending_queue: batch: flush_timeout: 30s max_size: 50000 min_size: 50000 enabled: true queue_size: 500000 sizer: items timeout: 30s # Pipeline configuration service: pipelines: logs: receivers: - otlp exporters: - enhance_indexing_s3_exporter traces: receivers: - otlp exporters: - enhance_indexing_s3_exporter ``` ### Local Development with MinIO For local development and testing, you can use MinIO as an S3-compatible object storage service: ```yaml theme={} exporters: enhance_indexing_s3_exporter: # Honeycomb API credentials (required even for local development) api_key: ${env:HONEYCOMB_MANAGEMENT_API_KEY} api_secret: ${env:HONEYCOMB_MANAGEMENT_API_SECRET} api_endpoint: https://api.honeycomb.io # MinIO configuration s3uploader: region: 'us-east-1' endpoint: 'http://localhost:9000' s3_bucket: 'telemetry-bucket' s3_force_path_style: true disable_ssl: true s3_partition_format: "year=%Y/month=%m/day=%d/hour=%H/minute=%M" compression: 'gzip' # Data format marshaler: 'otlp_json' # Custom indexed fields indexed_fields: - "user.id" - "customer.id" ``` # Set Up an Amazon S3 Archive with Terraform Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/enhance/terraform-setup Use Terraform to configure an Amazon S3 bucket for trace and log rehydration in Honeycomb. Ent+ Use Terraform to create the IAM policy and role that Honeycomb uses to access your S3 archive for trace and log rehydration. This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. To enable trace and log rehydration, you will need an AWS IAM role that includes: * Permissions to list and retrieve objects from your S3 bucket. * A trust policy that allows the Honeycomb IAM role to assume this role when interacting with your archive. This guide walks you through creating these resources using Terraform. If you prefer, you can [use the AWS Management Console instead](/send-data/telemetry-pipeline/enhance/aws-console-setup/). After you create the IAM policy and role, share the details with your Honeycomb account team, so they can complete your S3 archive setup. ## Create an IAM Policy First, create an IAM policy that grants permissions to list and retrieve objects from your Amazon S3 bucket. Use the configuration that matches your encryption method: Replace the placeholder ARNs in the `resources` argument with the actual ARNs for your bucket. ```hcl theme={} data "aws_iam_policy_document" "s3_bucket_access" { statement { effect = "Allow" actions = [ "s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation" ] resources = [ "arn:aws:s3:::/*", "arn:aws:s3:::" ] } } resource "aws_iam_policy" "s3_bucket_access" { name = "${var.env}_s3_bucket_access" policy = data.aws_iam_policy_document.s3_bucket_access.json } ``` Replace the placeholder ARNs in the `resources` arguments with the actual ARNs for your bucket and AWS KMS key. For the KMS key, specify the full key rather than an alias. ```hcl theme={} data "aws_iam_policy_document" "s3_bucket_access" { statement { effect = "Allow" actions = [ "s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation", ] resources = [ "arn:aws:s3:::/*", "arn:aws:s3:::" ] } statement { effect = "Allow" actions = [ "kms:Decrypt", ] resources = [ "arn:aws:kms:::key/", ] } } resource "aws_iam_policy" "s3_bucket_access" { name = "${var.env}_s3_bucket_access" policy = data.aws_iam_policy_document.s3_bucket_access.json } ``` ## Create the IAM Role Next, create an IAM role that can access your Amazon S3 bucket, and authorize Honeycomb to assume that role when interacting with your bucket: Replace the `honeycomb_role_arn` placeholder with the appropriate ARN, depending on your team's Honeycomb instance: * **US:** `"arn:aws:iam::702835727665:role/production-eks-bulk-ingest-role"` * **EU:** `"arn:aws:iam::919259170365:role/production-eu1-eks-bulk-ingest-role"` ```hcl theme={} data "aws_iam_policy_document" "hny_assume_role_policy" { statement { effect = "Allow" actions = [ "sts:AssumeRole" ] principals { type = "AWS" identifiers = [honeycomb_role_arn] } } } resource "aws_iam_role" "hny_s3_bucket_access_role" { name = "${var.env}-hny-s3-bucket-access-role" assume_role_policy = data.aws_iam_policy_document.hny_assume_role_policy.json } ``` ## Attach the Policy to the Role Attach the `"s3_bucket_access"` policy to your new IAM role: ```hcl theme={} resource "aws_iam_role_policy_attachment" "s3_bucket_access" { role = aws_iam_role.s3_bucket_access_role.name policy_arn = aws_iam_policy.s3_bucket_access.arn } ``` **KMS Users: Check Your Key Policy** If the data in your S3 bucket is encrypted with an AWS KMS encryption key, ensure that the corresponding [KMS Key Policy](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) allows this IAM Role to use the key to perform `kms:Decrypt` actions. If the IAM Role does not have permission to decrypt using the KMS key, rehydration will fail. ## Share with Your Honeycomb Account Team To complete your setup, share the following information with your Honeycomb account team: * The name of your Amazon S3 bucket * The ARN of the IAM role that will interact with your bucket To locate the IAM Role's ARN: 1. In the IAM service's navigation pane in the console, choose **Roles**. 2. Search for the name of the IAM role you created, then choose the role from the results. 3. In the **Summary** section, locate **ARN**, and use the copy icon to copy the ARN value to your clipboard. * `s3_prefix` from your OpenTelemetry Collector exporter configuration, if configured * `s3_partition_format` from your OpenTelemetry Collector exporter configuration, if configured * `indexed_fields` from your OpenTelemetry Collector exporter configuration, if you configured custom indexed fields # Install Pipeline Manager Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/install Set up the Honeycomb Telemetry Pipeline Manager to start collecting and routing telemetry from your infrastructure. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. Pipeline Manager runs as a hosted service powered by Bindplane. A Honeycomb license key activates Pipeline Manager with Honeycomb-specific branding, capabilities, and configuration. ## Before you begin To set up Pipeline Manager, you'll need a Honeycomb license key provided by your Honeycomb account team. If you don't have one, contact your account team to get started. ## Set up your account 1. Navigate to [app.bindplane.com/signup](https://app.bindplane.com/signup) and create an account. 2. Verify your email address if prompted, then sign in. 3. On the **Create a New Organization** screen, enter your Honeycomb team account name for the organization name. 4. Select the **I have a license key** link and enter the license key provided by your Honeycomb account team. 5. Accept the End User License Agreement and select **Get Started**. 6. The Honeycomb branding appears, confirming your account is activated. If you skip the license key step during organization creation, you can apply it later. Navigate to **Organization Settings** and select the **Have a license key?** link. Enter your key and save. You may need to refresh the page or sign out and back in for the Honeycomb branding to appear. ## Next steps After activating your account, Pipeline Manager displays a Quick Start Guide that walks you through installing your first agent, setting up a configuration, and rolling it out. Follow the steps in the Quick Start Guide to start collecting telemetry. You can also explore [Sources](/send-data/telemetry-pipeline/sources/), [Processors](/send-data/telemetry-pipeline/processors/), and [Destinations](/send-data/telemetry-pipeline/destinations/) to configure how your data is collected, refined, and routed. To add teammates, invite them from Organization Settings or [set up SSO](https://docs.bindplane.com/feature-guides/saas-single-sign-on) for centralized access management. # Pipeline Manager Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/manager Collect, refine, and route metrics, logs, and traces to any destination with Pipeline Manager, Honeycomb's enterprise-grade OpenTelemetry pipeline tool. Ent+ Pipeline Manager is an OpenTelemetry-native telemetry pipeline designed to collect, refine, and export metrics, logs, and traces from any source to any destination. Pipeline Manager is ideal for teams needing a reliable, agent-based data collection tool with centralized fleet management capabilities. Learn how to install the Pipeline Manager Available sources for the Honeycomb Telemetry Pipeline Manager. Available processors for the Honeycomb Telemetry Pipeline Manager. Available destinations for the Honeycomb Telemetry Pipeline Manager. ## Features * Manage the lifecycle of telemetry agents, starting with the Honeycomb Telemetry Pipeline Agent, a distribution of OpenTelemetry * Build, deploy, and manage telemetry configurations for different [Sources](/send-data/telemetry-pipeline/sources/) and deploy them to your agents * Send trace, log, and metric data to one or many [Destinations](/send-data/telemetry-pipeline/destinations/) * Use flow controls to adjust the flow of your data in real-time ## Architecture Honeycomb Telemetry Pipeline Architecture Diagram

Honeycomb Telemetry Pipeline Architecture Diagram

The Honeycomb Pipeline Manager is composed of the following components: * **GraphQL Server**: Provides configuration and agent details via GraphQL * **REST Server**: The Honeycomb Telemetry Pipeline CLI and UI make requests to the server via REST * **WebSocket Server**: Agents connect to receive configuration updates via [OpAMP](https://github.com/open-telemetry/opamp-spec) * **Store**: Pluggable storage manages configuration and Agent state * **Manager**: Dispatches configuration changes to Agents # Pipeline Manager Processors Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors Browse processors for the Honeycomb Telemetry Pipeline Manager to transform, filter, and enrich telemetry before it reaches its destination. Enterprise add-on. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## What are Processors? Processors can be inserted into your telemetry pipeline to transform your data before it arrives at your destination. Adding attributes, filtering, and converting logs to metrics are all the types of transformations we can do using processors in the Honeycomb Telemetry Pipeline Manager. ## Available Processors for the Honeycomb Telemetry Pipeline Manager ### Data Transformation Processors | Processor | Metrics | Logs | Traces | Description | | ------------------------------------------------------------------------------------ | :-----: | :--: | :----: | --------------------------------------------------- | | [Add Fields](/send-data/telemetry-pipeline/processors/add-fields/) | ✓ | ✓ | ✓ | Add attributes, resources, and log record body keys | | [Copy Field](/send-data/telemetry-pipeline/processors/copy-field/) | ✓ | ✓ | ✓ | Copy values from one field to another | | [Delete Empty Values](/send-data/telemetry-pipeline/processors/delete-empty-values/) | ✓ | ✓ | ✓ | Remove fields with empty or null values | | [Delete Fields](/send-data/telemetry-pipeline/processors/delete-fields/) | ✓ | ✓ | ✓ | Remove specific fields from telemetry | | [Move Field](/send-data/telemetry-pipeline/processors/move-field/) | ✓ | ✓ | ✓ | Move values between different field types | | [Rename Fields](/send-data/telemetry-pipeline/processors/rename-field/) | ✓ | ✓ | ✓ | Rename fields and attributes | | [Rename Metric](/send-data/telemetry-pipeline/processors/rename-metric/) | ✓ | | | Rename metric names | | [Split](/send-data/telemetry-pipeline/processors/split/) | ✓ | ✓ | | Split telemetry records into multiple records | | [Transform](/send-data/telemetry-pipeline/processors/transform/) | ✓ | ✓ | ✓ | Transform telemetry using custom OTTL statements | | [Unroll](/send-data/telemetry-pipeline/processors/unroll/) | | ✓ | | Expand arrays into separate log records | ### Parsing Processors | Processor | Metrics | Logs | Traces | Description | | ---------------------------------------------------------------------------- | :-----: | :--: | :----: | ------------------------------------------------ | | [Parse CSV](/send-data/telemetry-pipeline/processors/parse-csv/) | ✓ | ✓ | ✓ | Parse CSV-formatted strings into structured data | | [Parse JSON](/send-data/telemetry-pipeline/processors/parse-json/) | ✓ | ✓ | ✓ | Parse JSON strings into structured data | | [Parse Key Value](/send-data/telemetry-pipeline/processors/parse-key-value/) | ✓ | ✓ | ✓ | Parse key-value pairs from strings | | [Parse Severity](/send-data/telemetry-pipeline/processors/parse-severity/) | | ✓ | | Extract and standardize log severity levels | | [Parse Timestamp](/send-data/telemetry-pipeline/processors/parse-timestamp/) | ✓ | ✓ | ✓ | Parse timestamps from various formats | | [Parse with Regex](/send-data/telemetry-pipeline/processors/parse-regex/) | ✓ | ✓ | ✓ | Extract data using regular expressions | | [Parse XML](/send-data/telemetry-pipeline/processors/parse-xml/) | ✓ | ✓ | ✓ | Parse XML strings into structured data | ### Filtering & Sampling Processors | Processor | Metrics | Logs | Traces | Description | | ------------------------------------------------------------------------------------- | :-----: | :--: | :----: | -------------------------------------- | | [Filter by Condition](/send-data/telemetry-pipeline/processors/filter-by-condition/) | ✓ | ✓ | ✓ | Filter based on OTTL conditions | | [Filter by Field](/send-data/telemetry-pipeline/processors/filter-by-field/) | ✓ | ✓ | ✓ | Filter based on field values | | [Filter by Regex](/send-data/telemetry-pipeline/processors/filter-by-regex/) | | ✓ | | Filter using regular expressions | | [Filter HTTP Status](/send-data/telemetry-pipeline/processors/filter-by-http-status/) | | ✓ | | Filter logs based on HTTP status codes | | [Filter Metric Name](/send-data/telemetry-pipeline/processors/filter-by-metric-name/) | ✓ | | | Filter metrics by name patterns | | [Filter Severity](/send-data/telemetry-pipeline/processors/filter-by-severity/) | | ✓ | | Filter logs by severity level | | [Log Sampling](/send-data/telemetry-pipeline/processors/log-sampling/) | | ✓ | | Sample logs to reduce volume | | [Deduplicate Logs](/send-data/telemetry-pipeline/processors/deduplicate-logs/) | | ✓ | | Remove duplicate log entries | ### Data Enrichment Processors | Processor | Metrics | Logs | Traces | Description | | ---------------------------------------------------------------------------------------- | :-----: | :--: | :----: | ------------------------------------------------- | | [Coalesce](/send-data/telemetry-pipeline/processors/coalesce/) | ✓ | ✓ | ✓ | Combine multiple fields into a single field | | [Group by Attributes](/send-data/telemetry-pipeline/processors/group-by-attributes/) | ✓ | ✓ | ✓ | Group telemetry by attribute values | | [Lookup Fields](/send-data/telemetry-pipeline/processors/lookup-fields/) | ✓ | ✓ | ✓ | Enrich data using lookup tables | | [Resource Detection](/send-data/telemetry-pipeline/processors/resource-detection/) | ✓ | ✓ | ✓ | Automatically detect and add resource information | | [Redact Sensitive Data](/send-data/telemetry-pipeline/processors/redact-sensitive-data/) | ✓ | ✓ | ✓ | Redact sensitive data in telemetry | | [Rewrite Timestamp](/send-data/telemetry-pipeline/processors/rewrite-timestamp/) | ✓ | ✓ | ✓ | Modify timestamp values | ### Aggregation & Statistics Processors | Processor | Metrics | Logs | Traces | Description | | ------------------------------------------------------------------------------------------- | :-----: | :--: | :----: | -------------------------------------------- | | [Batch](/send-data/telemetry-pipeline/processors/batch/) | ✓ | ✓ | ✓ | Batch telemetry for efficient processing | | [Compute Metric Statistics](/send-data/telemetry-pipeline/processors/compute-metric-stats/) | ✓ | | | Calculate statistics from metric data | | [Count Telemetry](/send-data/telemetry-pipeline/processors/count-telemetry/) | ✓ | ✓ | ✓ | Count telemetry records and generate metrics | | [Extract Metric](/send-data/telemetry-pipeline/processors/extract-metric/) | | ✓ | | Extract metrics from log data | ### Specialized Processors | Processor | Metrics | Logs | Traces | Description | | -------------------------------------------------------------------------------------------------------- | :-----: | :--: | :----: | ----------------------------------------------- | | [Concat](/send-data/telemetry-pipeline/processors/concat/) | ✓ | ✓ | ✓ | Concatenate multiple fields into a single field | | [Custom](/send-data/telemetry-pipeline/processors/custom/) | ✓ | ✓ | ✓ | Custom OpenTelemetry processor configuration | | [Google SecOps Standardization](/send-data/telemetry-pipeline/processors/google-secops-standardization/) | | ✓ | | Standardize logs for Google Security Operations | | [Marshal](/send-data/telemetry-pipeline/processors/marshal/) | | ✓ | | Marshal data into specific formats | # Add Fields Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/add-fields Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Add Fields processor can be used to add attributes, resources, and log record body keys from telemetry in the pipeline. ## Use The Add Fields processor is utilized for enriching telemetry data by appending or modifying attributes, resources, and log record body keys in metrics, logs, and traces based on specified conditions. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration | Field | Description | | :-------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | An OTTL condition that must evaluate to true to apply this processor. By default, the processor applies to all telemetry. | | Field Type | Indicates the context in which the processor should operate: `Attributes`, `Body`, or `Resource`. Note: `Body` fields are applicable only for logs. | | Action | `Insert`: Add field if it does not exist. `Update`: Update existing value. `Upsert`: Insert or update. | | Key | Key to add or modify to the telemetry. | | Value | Value associated with the defined key to add or modify to the telemetry. | ## Example Configuration ### Append Resource to Metrics for Categorization by Source In this configuration, additional resource data is appended to the metrics to categorize and identify them based on their source. This is particularly useful for differentiating and filtering metrics that are generated from various environments or locations. * environment: dev * location: us-east1-b ### Add Custom Attributes to Logs for Enhanced Searchability Based on Error Status This configuration will add custom resource (application\_name) to logs that have a log level of 'ERROR'. This can help in quickly identifying and tracing critical issues within a specific part of the application. Example log body: ```JSON theme={} { "error": "incorrect username or password", "level": "error", "message": "failed to authenticate user: test", "timestamp": "2023-07-05T13:40:17.549-0400" } ``` The log condition we use to determine what logs to append the resource to: ```expr theme={} body["level"] == error ``` # Batch Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/batch Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The batch processor accepts spans, metrics, or logs and places them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. This processor supports both size and time based batching. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :----- | :------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | send\_batch\_size | `int` | `8192` | Number of spans, metric data points, or log records after which a batch will be sent regardless of the timeout. | | send\_batch\_max\_size | int | 0 | The upper limit of the batch size. 0 means no upper limit of the batch size. This property ensures that larger batches are split into smaller units. It must be greater than or equal to send batch size. | | timeout | string | 200ms | Time duration after which a batch will be sent regardless of size. Example: 2s (two seconds). | ## Example Configuration Batch telemetry with the following options: * Send batches of size 200 * Max batch size of 1000 * Build batches for up to two seconds ### Web Interface Honeycomb Docs - Batch - image 1

# Coalesce Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/coalesce Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Coalesce processor can be used to consolidate many field names into a single field name. ## Use The Coalesce processor is utilized for consolidating telemetry fields in metrics, logs, and traces based on specified conditions. While this is similar on concept to SQL Coalesce, it has some key differences. Especially around the order of precedence. See the behavior section. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration | Field | Description | | :-------------- | :-------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | A condition that determines when this processor is applied. | | Action | The action to take (insert, update, upsert) when coalescing telemetry | | Coalesce From | The telemetry fields to coalesce from. | | Coalesce To | The telemetry field to coalesce to. | ## Behavior The behavior of this processor is dependent on the selected action. When insert is selected, the target field only gets coalesced to if it doesn't exist. Precedence: First item in list. When update is selected, the target field gets coalesced to only if it already exists. Precedence: Last item in list. When upsert it selected, the target field gets coalesces to regardless whether it existed or not. Precedence: Last item in list. ## Example Configuration(s) ### Coalesce timestamp fields This configuration will coalesce the body fields `ts`, `time`, and `timestamp` to an attribute named `time` for later parsing. Upsert is used for the action. #### Web Interface Honeycomb Docs - Coalesce Field - Timestamps

Honeycomb Docs - Coalesce Field - Timestamps

### Coalesce severity fields This configuration will coalesce the body fields `sev`, `severity`, and `level` to an attribute named `severity` for later parsing. Upsert is used for the action. #### Web Interface Honeycomb Docs - Coalesce Field - Severity

Honeycomb Docs - Coalesce Field - Severity

### Coalesce datacenter fields This configuration will coalesce the body fields `dc`, `datacenter`, and `location` to an attribute named `datacenter` for later parsing. Upsert is used for the action. #### Web Interface Honeycomb Docs - Coalesce Field - Datacenter

Honeycomb Docs - Coalesce Field - Datacenter

# Compute Metric Statistics Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/compute-metric-stats Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Compute Metric Statistics processor can be used to calculate statistics for metrics over fixed time intervals to reduce metric throughput. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | | | ## Configuration Table | Parameter | Type | Default | Description | | :-------- | :--------- | :---------------------- | :--------------------------------------------------------------------------------------------------------------------------------- | | interval | `int` | `60` | The interval in seconds on which to aggregate metrics. Calculated metrics will be emitted after the interval passes. | | include | `regex` | `.*` | A regex that must match against the metric's name in order to calculate statistics from it. The default value matches all metrics. | | stats | `[]string` | `["min", "max", "avg"]` | A list of statistics to calculate on matched metrics. Valid values are: `min`, `max`, `avg`, `first`, and `last`. | ## Example Configuration ### Calculate Average, Minimum, and Maximum Calculate the average, minimum, and maximum values of each incoming metric, and emit them with a `.avg`, `.min`, and `.max` suffix, respectively. #### Web Interface Honeycomb Docs - Metric Statistics - image 1

Honeycomb Docs - Metric Statistics - image 1

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: calculate-avg-min-max name: calculate-avg-min-max spec: type: metric_stats parameters: - name: interval value: 60 - name: include value: '.*' - name: stats value: ['min', 'max', 'avg'] ``` ### Take the Most Recent Value Take the last value of a metric over a 60 second interval, and emit the metric with a suffix of `.last`. #### Web Interface Honeycomb Docs - Metric Statistics - image 2

Honeycomb Docs - Metric Statistics - image 2

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: calculate-last-value name: calculate-last-value spec: type: metric_stats parameters: - name: interval value: 60 - name: include value: '.*' - name: stats value: ['last'] ``` # Concat Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/concat Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Concat processor can be used to concatenate fields together into a single string. ## Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ## Configuration | Field | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | | Condition | A condition that determines when this processor is applied. | | Source Field Type | The type of telemetry that contains the fields to be concatenated. | | Source Fields | The telemetry fields to concatenate together. | | Target Field Type | The telemetry field type where the concatenation result will be placed. | | Target Field | The target field to place the concatenation result into. May only be blank if Body is selected for the Target Field Type. | | Delimiter | The delimiter to place between the source fields when concatenating. Defaults to a space, but may also be empty. | # Copy Field Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/copy-field Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Copy Field processor can be used to copy telemetry from one field to another. ### Use The Copy Field processor is utilized for copying telemetry fields in metrics, logs, and traces based on specified conditions. ## Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ## Configuration | Field | Description | | --------------- | ----------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | A condition that determines when this processor is applied. | | Copy From | The telemetry field to copy from. | | Copy To | The telemetry field to copy to. | # Count Telemetry Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/count-telemetry Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.27.0\`+ | ## Description The Count Telemetry Processor can count the number of logs, metric data points, or trace spans matching some filter, and create a metric with that value. Both the name and units of the created metric can be configured. Additionally, fields from matching logs can be preserved as metric attributes. ## Use Count Telemetry is especially useful as a way to convert your logs to metrics, allowing you to drop logs you don't need while still capturing signal from them. A frequent use case is to count how many logs you're getting from your web server by http status code. This lets you see if you're getting 500s, without paying to store logs for your 200s. See below for specific configuration examples. ## Configuration | Field | Description | | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Telemetry Types | The types of telemetry to apply the processor to. | | Match Expression | [OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl#readme) expression to find matching logs. Uses the [log context](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md) for logs, .[datapoint context](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottldatapoint/README.md) for metrics, and [span context](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspan/README.md) for traces. | | Metric Name | The name of the metric created. | | Metric Units | The unit of the metric created. See [Unified Code for Units of Measure](https://unitsofmeasure.org/ucum#section-Alphabetic-Index-By-Name) for available units. | | Metric Attributes | The mapped attributes of the metric created. Each key is an attribute name. Each value is an [expression](https://github.com/expr-lang/expr/blob/master/docs/language-definition.md) that extracts data from the log. | ## Example Configurations ### Count all telemetry By default, enabling metrics, traces, or logs will count all of their respective telemetry types. Below is an example of what this looks like when we want to count all logs. Honeycomb Docs - Count Telemetry - image 1

Honeycomb Docs - Count Telemetry - image 1

### Count HTTP Requests by Status (logs) In this configuration, we want to parse our HTTP server logs to count how many requests were completed, broken down by status code. Our logs are JSON with the following structure: ```JSON theme={} { "level": "warn", "host": "10.0.10.0", "datetime":"2022-12-07T13:21", "method": "POST", "request": "/api/create", "protocol": "HTTP/1.1", "status": 500 } ``` The match expression will exclude all logs without a status code in its body: ```expr theme={} body["status"] != nil ``` We'll name this metric `http.request.count`, then we'll use the status code for the `status_code` metric attribute on the created metric: ```yaml theme={} log_attributes: status_code: body["status"] ``` Honeycomb Docs - Count Telemetry - image 2

Honeycomb Docs - Count Telemetry - image 2

# Custom Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/custom Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Custom Processor The Custom processor can be used to inject a custom processor configuration into a pipeline. A list of supported processors can be found [here](https://github.com/observIQ/observiq-otel-collector/blob/main/docs/processors.md). The Custom processor is useful for solving use cases not covered by the Honeycomb Telemetry Pipeline's other processor types. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | The custom processor type can support all telemetry types, however, it is up to the user to enable / disable the correct types\ based on the processor being used. ## Configuration Table | Parameter | Type | Default | Description | | :--------------- | :------------------ | :------------------------------ | :---------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | The list of telemetry types the processor will act on. | | configuration | `yaml` | required | Enter any supported Processor and the YAML will be inserted into the configuration. | ## Example Configuration Inject the following [resource processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor) configuration: ```yaml theme={} resource: attributes: - action: upsert key: custom value: true ``` ### Web Interface Honeycomb Docs - Custom - image 1

### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: custom name: custom spec: type: custom parameters: - name: configuration value: | resource: attributes: - action: upsert key: custom value: true - name: telemetry_types value: - Metrics - Traces - Logs ``` ### Configuration with Embedded Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Configuration metadata: id: custom name: custom labels: platform: linux spec: sources: - type: journald parameters: - name: units value: [] - name: directory value: '' - name: priority value: info - name: start_at value: end processors: - type: custom parameters: - name: configuration value: | resource: attributes: - action: upsert key: custom value: true - name: telemetry_types value: - Metrics - Traces - Logs selector: matchLabels: configuration: custom ``` # Deduplicate Logs Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/deduplicate-logs Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Deduplicate Logs processor can be used to deduplicate logs over a time range and emit a single log with the count of duplicate logs. Logs are considered duplicates if the following match: * Severity * Log Body * Resource Attributes * Log Attributes ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :---------------------- | :-------- | :---------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | interval\* | `int` | `10` | The interval in seconds on which to aggregate logs. An aggregated log will be emitted after the interval passes. | | log\_count\_attribute\* | `string` | `log_count` | The name of the count attribute of deduplicated logs that will be added to the emitted log. | | timezone\* | `string` | `UTC` | The timezone of the `first_observed_timestamp` and `last_observed_timestamp` log attributes that are on the emitted log. | | include\_fields | `strings` | | A list of fields to include in duplicate matching. Fields can be from the log `body` or `attributes`. This option is mutually exclusive with exclude\_fields. More details can be found [here](#include_fields-parameter). | | exclude\_fields | `strings` | | A list of fields to exclude from duplicate matching. Fields can be excluded from the log `body` or `attributes`. These fields will not be present in the emitted log. This option is mutually exclusive with include\_fields. More details can be found [here](#exclude_fields-parameter). | \* *required field* ### `include_fields` Parameter The `include_fields` parameter allows the user to remove fields from being considered when looking for duplicate logs. Fields can be included from either the `body` or `attributes` of a log. Though the entire `body` cannot be included. Nested fields can be specified by delimiting each part of the path with a `.`. If a field contains a `.` as part of its name it can be escaped by using `\.`. Below are a few examples and how to specify them: * Include `timestamp` field from the body -> `body.timestamp` * Include a `log.file.name` field from the log attributes -> `attributes.log\.file\.name` * Include a nested `ip` field inside a `src` attribute -> `attributes.src.ip` ### `exclude_fields` Parameter The `exclude_fields` parameter allows the user to remove fields from being considered when looking for duplicate logs. Fields can be excluded from either the `body` or `attributes` of a log. Though the entire `body` cannot be excluded. Nested fields can be specified by delimiting each part of the path with a `.`. If a field contains a `.` as part of its name it can be escaped by using `\.`. Below are a few examples and how to specify them: * Exclude `timestamp` field from the body -> `body.timestamp` * Exclude a `host.name` field from the log attributes -> `attributes.host\.name` * Exclude a nested `ip` field inside a `src` attribute -> `attributes.src.ip` ## Example Configuration ### Basic Configuration Setting a custom `log_count_attribute` and `timezone` while deduplicating logs on a 60 second `interval`. #### Web Interface Honeycomb Docs - Deduplicate Logs - image 1

Honeycomb Docs - Deduplicate Logs - image 1

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: log-dedup name: log-dedup spec: type: log_dedup parameters: - name: interval value: 60 - name: log_count_attribute value: 'dedup_count' - name: timezone value: 'America/Los_Angeles' ``` ### Exclude Fields This example shows the addition of `exclude_fields`. More information on exclude\_fields can be found [here](#exclude_fields-parameter). #### Web Interface Honeycomb Docs - Deduplicate Logs - image 2

Honeycomb Docs - Deduplicate Logs - image 2

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: exclude-fields name: exclude-fields spec: type: log_dedup parameters: - name: interval value: 10 - name: log_count_attribute value: 'log_count' - name: timezone value: 'UTC' - name: exclude_fields value: - 'attributes.timestamp' - 'body.time' - 'attributes.log\.file\.name' ``` # Delete Empty Values Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/delete-empty-values Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Delete Empty Values processor can be used to delete null and other empty values from telemetry resource attributes, telemetry attributes, or a log record's body. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :----------------------- | :------------------ | :------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | The list of telemetry types the processor will act on. | | deleted\_values | `enums` | `["Null Values"]` | List of values types to delete. May include "Null Values", "Empty Lists", and "Empty Maps". Enabling "Null Values" will remove both empty (zero-length) strings and entirely empty values. Enabling "Empty Lists" will delete empty (no element) list values. Enabling "Empty Maps" will delete empty (no key) map values. | | exclude\_resource\_keys | `strings` | `[]` | List of resource keys to exclude from deletion. | | exclude\_attribute\_keys | `strings` | `[]` | List of attribute keys to exclude from deletion. | | exclude\_body\_keys | `string` | `[]` | List of body keys to exclude from deletion. | | empty\_string\_values | `strings` | `[]` | List of string values that are considered "empty". String fields will be deleted if they match any of the strings in this list. | ## Example Configuration This example configuration removes empty values from NGINX logs, where "-" is used to denote an empty field. ### Web Interface Honeycomb Docs - Delete Empty Values - image 1

Honeycomb Docs - Delete Empty Values - image 1

### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: remove-empty-attributes name: remove-empty-attributes spec: type: remove_empty_values parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: empty_string_values value: - '-' ``` ### Configuration with Embedded Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Configuration metadata: id: remove-empty-attributes name: remove-empty-attributes labels: platform: linux spec: sources: - type: nginx parameters: - name: telemetry_types value: - Logs - Metrics - name: endpoint value: http://localhost:80/status - name: disable_metrics value: [] - name: enable_tls value: false - name: insecure_skip_verify value: false - name: ca_file value: '' - name: cert_file value: '' - name: key_file value: '' - name: collection_interval value: 60 - name: data_flow value: high - name: log_format value: default - name: access_log_paths value: - /var/log/nginx/access.log* - name: error_log_paths value: - /var/log/nginx/error.log* - name: start_at value: end processors: - type: delete_empty_values parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: deleted_values value: - Null Values - name: exclude_resource_keys value: [] - name: exclude_attribute_keys value: [] - name: exclude_body_keys value: [] - name: empty_string_values value: - '-' disabled: false disabled: false selector: matchLabels: configuration: remove-empty-attributes ``` # Delete Fields Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/delete-fields Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Delete Fields processor can be used to remove attributes, resource attributes, and log record body keys from telemetry in the pipeline. Deleting Metric attributes may be unsound. Be careful when deleting metric attributes. Deleting attributes on metrics may cause multiple data points to have the same set of attributes, causing a datapoint collision. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :------------------------------ | :------------------------------------------------------------------------------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | The list of telemetry types the processor will act on. | | condition | `string` | `true` | An OTTL condition that must evaluate to true to apply this processor. By default, the processor applies to all telemetry. | | body\_keys | `string` | `[]` | One or more body key names to remove from telemetry data. Note: Body fields are applicable only for logs. | | attributes | `string` | `[]` | One or more attribute names to remove from telemetry data. | | resource\_attributes | `string` | `[]` | One or more resource attribute names to remove from telemetry data. | ## Example Configuration This example configuration removes the "spid" body field, the "log.file.name" attribute, and the "host.id" resource attribute from any log record. ### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: delete-fields-v2 name: delete-fields-v2 spec: type: delete_fields-v2 parameters: - type: delete_fields-v2 parameters: - name: telemetry_types value: ['Logs'] - name: condition value: 'true' - name: body_keys value: - spid - name: attributes value: - log.file.name - name: resource_attributes value: - host.id ``` ### Configuration with Embedded Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Configuration metadata: id: delete-fields-v2 name: delete-fields-v2 labels: platform: linux spec: sources: - type: journald parameters: - name: units value: [] - name: directory value: '' - name: priority value: info - name: start_at value: end processors: - type: delete_fields-v2 parameters: - name: telemetry_types value: ['Logs'] - name: condition value: 'true' - name: body_keys value: - spid - name: attributes value: - file.name - name: resource_attribute value: - host.id selector: matchLabels: configuration: delete-fields-v2 ``` # Resource Detection Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/detect-resource Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The resource detection processor can be used to detect resource information from the host, in a format that conforms to the [OpenTelemetry resource semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/), and append or override the resource value in telemetry data with this information. ## Supported Types | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.40.0`+ | ## Configuration Table | Field | Description | | :---------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Detector | Detector(s) to use for resource detection. Available detectors include `aks`, `azure`, `consul`, `ec2`, `eks`, `gcp`, `k8snode`, `openshift`, and `system`. | | Override | Whether or not to overwrite existing resource attributes. | | Timeout | Time duration after which a resource detector request will timeout. | | Hostname Source | When the `system` detector is in use, specifies the source used to detect the system hostname. Options include `os`, `dns`, `cname`, and `lookup`. | | EC2 Tags | Specifies a list of regex's to match to EC2 instance tag keys that will be added as resource attributes to processed data. | | K8s Node Environment Variable | The K8s node environment variable that has the node name to retrieve metadata for. Defaults to "KUBE\_NODE\_NAME" because the Telemetry Pipeline Agent has this automatically set. | | Consul Address | The address of the Consul server. If unset, the environment variable "CONSUL\_HTTP\_ADDR" will be used if it exists. | | Consul Authentication | The type of authentication to use with Consul. One of 'Token' or 'Token File' is required if Consul's ACL System is enabled. Options are "None", "Token", or "Token File". | | Consul Token | Token is used to provide a per-request ACL token which overrides the agent's default token. If unset, the environment variable "CONSUL\_HTTP\_TOKEN" will be used if it exists. | | Consul Token File | Token File is a file containing the current token to use for this client. If provided, it is read once at startup and never again. If unset, the environment variable "CONSUL\_HTTP\_TOKEN\_FILE" will be used if it exists. | | Consul Datacenter | Optional Consul Datacenter to use. If not provided, the default agent datacenter is used. | | Consul Namespace | Optional namespace to attach to each Consul request. If unset, the environment variable "CONSUL\_NAMESPACE" will be used if it exists. | | Consul Metadata Labels | Allowlist of Consul Metadata keys to use as resource attributes. | Multiple detectors may be selected on a single processor. However, if multiple processors use a common attribute name, the first detector will have precedent. For more info, see this OTel [documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#ordering). ## Example Configurations ### Google Compute Engine (GCE) In this example, the Resource Detection Processor is configured to use the [GCP detector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#gce-metadata) to detect GCE resource attributes. resource-detection-v2-1

### Amazon EC2 In this example, the Resource Detection Processor is configured to use the [EC2 detector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#aws-ec2). resource-detection-v2-3

### Azure Compute Instance In this example, the Resource Detection Processor is configured to use the [Azure detector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#azure) to detect Azure Virtual Machine resource attributes. resource-detection-v2-5

### Kubernetes The Resource Detection Processor can detect Kubernetes resources on the following platforms: GKE, Amazon EKS, Azure AKS. Using the GCP detector, you can detect cloud-based Kubernetes resources. resource-detection-v2-1

# Extract Metric Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/extract-metric Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Extract Metric Processor creates new metrics based on log telemetry. For logs matching a filter, the processor will extract a numerical value from a field and then create a metric with that value. The name, unit, and type of the created metric can be configured. Additionally, fields from matching logs can be preserved as metric attributes. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | | ✓ | | ## Supported Collector Versions `v1.14.0`+ ## Configuration The configuration of an Extract Metric processor consists of a number of sub-metrics that are defined using fields described below. | Field | Description | | :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric Name | The name of the metric that will be created. | | Match | The log context the source field is located in. Options are `Body`, `Attributes`, and `Resource`. | | Metric Field | The name of the source field containing a numeric value that will become the new metric value. | | Metric Type | The type of metric that will be created. Options are `gauge_double`, `gauge_int`, `counter_double`, and `counter_int`. | | Metric Unit | The unit of the created metric. Some default choices provided with the ability to create a custom unit. | | Attributes | Existing attributes on the source log that should be carried over. Can also specify new metric attributes. Each value is an OTTL [path](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/LANGUAGE.md#paths) expression that extracts data from the log. | ## Example Configuration In this configuration we are creating a new latency metric based on a field in the log body. Here is the new `latency` metric created as a `gauge_double` with seconds set as the unit. Honeycomb Docs - Extract Metric - image 1

Honeycomb Docs - Extract Metric - image 1

Now we can see the new metric in the snapshot view. Honeycomb Docs - Extract Metric - image 2

# Filter by Condition Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/filter-by-condition Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Filter by Condition Processor The Filter by Condition processor can be used to include or exclude telemetry based on a condition that is evaluated against the telemetry data. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration | Field | Description | | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------- | | Choose Telemetry Type | The types of telemetry to filter. | | Action | The action to take when the condition is met. Include will retain matching telemetry. Exclude will remove matching telemetry. | | Condition | The condition to match against telemetry to include or exclude based on the action. | ## Example Configuration ### Excluding matching log records In this example, we exclude logs that have **all** of the following: * A `host.name` resource attribute that equals `dev-server` * An `environment` attribute that equals `dev` * A `remote-ip` log body field that equals `127.0.0.1` ### Web Interface Honeycomb Docs - Filter by Condition

## API Reference This processor can be defined as yaml and applied using the CLI or API. ### Type filter-by-condition ### Parameters | Parameter | Type | Default | Description | | :--------------- | :------------------ | :------------------------------ | :---------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | The types of telemetry to filter. | | action | `enum` | `exclude` | The action to take when the condition is met. Include will retain matching telemetry. Exclude will remove matching telemetry. | | condition | `condition` | `{"ottl":""}` | The condition to match against telemetry to include or exclude based on the action. | ### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: name: filter-by-condition-example spec: type: filter-by-condition parameters: - name: telemetry_types value: - Logs - name: action value: exclude - name: condition value: ottl: (resource.attributes["host.name"] == "dev-server" and attributes["environment"] == "dev" and body["remote-ip"] == "127.0.0.1") ui: statements: - key: host.name match: resource operator: Equals value: dev-server - key: environment match: attributes operator: Equals value: dev - key: remote-ip match: body operator: Equals value: 127.0.0.1 ``` # Filter by Field Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/filter-by-field Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Filter by Field Processor The Filter by Field processor can be used to include or exclude telemetry based on matched resource attributes, attributes, or log body fields. This processor has been deprecated and replaced with a new [Filter by Condition](/send-data/telemetry-pipeline/processors/filter-by-condition) processor. While it will continue to function, it will no longer receive any enhancements and you should migrate to the new processor. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :--------------- | :------------------ | :------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | The list of telemetry types the processor will act on. | | action | `enum` | `exclude` | Whether to `include` (retain) or `exclude` (drop) matches. | | match\_type | `enum` | `strict` | Method for matching values. Strict matching requires that 'value' be an exact match. Regexp matching uses [re2](https://github.com/google/re2/wiki/Syntax) to match a value. | | attributes | `map` | `[]` | Attribute key value pairs to filter on. Telemetry is filtered if all attribute, resource, and body key pairs are matched. | | resources | `map` | `[]` | Resource key value pairs to filter on. Telemetry is filtered if all attribute, resource, and body key pairs are matched. | | bodies | `map` | `[]` | Log body key value pairs to filter on. Log records filtered if all attribute, resource, and body key pairs are matched. | ## Example Configuration ### Excluding matching log records In this example, we exclude logs that have **all** of the following: * A `host.name` resource attribute that equals `dev-server` * An `environment` attribute that equals `dev` * A `remote-ip` log body field that equals `127.0.0.1` #### Web Interface Honeycomb Docs - Filter by Field - image 1

Honeycomb Docs - Filter by Field - image 1

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: filter_field name: filter_field spec: type: filter_field parameters: - name: telemetry_types value: - Metrics - Logs - Traces - name: action value: exclude - name: match_type value: strict - name: attributes value: environment: dev - name: resources value: host.name: dev-server - name: bodies value: remote-ip: 127.0.0.1 ``` # Filter HTTP Status Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/filter-by-http-status Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The HTTP Status processor can be used to filter out logs that contain a status code between a minimum and a maximum status code. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :-------- | :----- | :------ | :------------------------------------------------------------------------------ | | minimum | `enum` | `100` | Minimum Status to match. Log entries with lower status codes will be filtered. | | maximum | `enum` | `599` | Maximum Status to match. Log entries with higher status codes will be filtered. | Valid Minimum Status Codes: * 100 * 200 * 300 * 400 * 500 Valid Maximum Status Codes: * 199 * 299 * 399 * 499 * 599 ## Example Configuration Filter out all 1xx status codes and 2xx status codes. ### Web Interface Honeycomb Docs - Filter HTTP Status - image 1

Honeycomb Docs - Filter HTTP Status - image 1

### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: http-status name: http-status spec: type: filter_http_status parameters: - name: maximum value: 599 - name: minimum value: 300 ``` ### Configuration with Embedded Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Configuration metadata: id: http-status name: http-status labels: platform: linux spec: sources: - type: journald parameters: - name: units value: [] - name: directory value: '' - name: priority value: info - name: start_at value: end processors: - type: filter_http_status parameters: - name: maximum value: 599 - name: minimum value: 300 selector: matchLabels: configuration: http-status ``` # Filter Metric Name Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/filter-by-metric-name Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Metric Name Filter Processor The Metric Name Filter processor can be used to include or exclude metrics based on their name. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | | | ## Configuration Table | Parameter | Type | Default | Description | | :------------ | :-------- | :-------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | action | `enum` | `exclude` | `exclude` or `include` metrics that match. | | match\_type | `enum` | `strict` | Method for matching values. Strict matching requires that 'value' be an exact match. Regexp matching uses [re2](https://github.com/google/re2/wiki/Syntax) to match a value. | | metric\_names | `strings` | required | One or more metric names to match on. | ## Example Configuration ### Web Interface Honeycomb Docs - Filter Metric Name - image 1

Honeycomb Docs - Filter Metric Name - image 1

### Exclude Regexp Filter out (exclude) metrics that match the expression `k8s.node.*`. ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: filter-name-regexp name: filter-name-regexp spec: type: filter_metric_name parameters: - name: action value: exclude - name: match_type value: regexp - name: metric_names value: - k8s.node.* ``` ### Include Strict Include metrics that match, drop all other metrics. ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: include-name-strict name: include-name-strict spec: type: filter_metric_name parameters: - name: action value: include - name: match_type value: strict - name: metric_names value: - k8s.container.cpu - k8s.pod.memory ``` # Filter by Regex Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/filter-by-regex Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Filter by Regex Processor The Filter by Regex processor can be used to include or exclude logs based on Regex matching body fields. This processor is exclusive to the Google license. Enterprise and Free users should use the [Filter By Condition](/send-data/telemetry-pipeline/processors/filter-by-condition) processor, which has more robust filtering. | Metrics | Logs | Traces | | :------ | :--- | :----- | | | ✓ | | | Field | Description | | :----------------- | :--------------------------------------------------------------------------------------------------------------------------------------------- | | Action | The action to take when the filter condition is met. Include will retain matching logs. Exclude will remove matching logs. | | Regex | The regular expression (Regex) that logs will be evaluated against. | | Match | The type containing the field the Regex will be evaluated against. Options are "Body" and "Attributes". | | Field (Body) | If `Field Type` is set to "body", this is the name of the body field Regex will be evaluated against. Leave empty to apply to the entire body. | | Field (Attributes) | If `Field Type` is set to "attributes", this is the name of the attribute field Regex will be evaluated against. | ## Example Configuration In this example, we exclude logs that have the body field "path" matching this Regex: `.+(?:ql)`. ### Web Interface Honeycomb Docs - Filter by Regex Processor - image 1

Honeycomb Docs - Filter by Regex Processor - image 1

### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: filter_regex name: filter_regex spec: type: google_filter_regex parameters: - name: action value: exclude - name: regex value: '.+(?:ql)' - name: field_type value: 'Body' - name: body_field value: 'path' ``` # Filter Severity Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/filter-by-severity Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Severity Filter processor can be used to filter out logs that do not meet a given severity threshold. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :-------- | :------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | severity | `enum` | `TRACE` | Minimum severity to match. Log entries with lower severities will be filtered. | | condition | `string` | `true` | An [OTTL] expression used to match which log records to sample from. All paths in the [log context] are available to reference. All [converters] are available to use. | [OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.109.0/pkg/ottl#readme [converters]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.109.0/pkg/ottl/ottlfuncs/README.md#converters [log context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.109.0/pkg/ottl/contexts/ottllog/README.md Valid severity levels: * TRACE * INFO * WARN * ERROR * FATAL ## Example Configuration Filter out INFO and TRACE logs where Attribute `ID` is less than 3. ### Web Interface Honeycomb Docs - Filter Severity - image 1

Honeycomb Docs - Filter Severity - image 1

### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: severity-filter name: severity-filter spec: type: filter_severity parameters: - name: severity value: WARN - name: condition value: ottl: (attributes["ID"] < 3) ui: operator: '' statements: - key: ID match: attributes operator: < value: '3' ``` ### Configuration with Embedded Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Configuration metadata: id: severity-filter name: severity-filter labels: platform: linux spec: sources: - type: journald parameters: - name: units value: [] - name: directory value: '' - name: priority value: info - name: start_at value: end processors: - type: filter_severity parameters: - name: severity value: WARN - name: condition value: ottl: (attributes["ID"] < 3) ui: operator: '' statements: - key: ID match: attributes operator: < value: '3' selector: matchLabels: configuration: severity-filter ``` # Google SecOps Standardization Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/google-secops-standardization Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. This processor requires collector version 1.64.0 or newer to send fields to Google SecOps. In older collector versions, namespace and ingestion label fields will be added to telemetry but not parsed in Google SecOps. ## Description The Google SecOps Standardization processor can be used to add the `log_type` [ingestion label](https://cloud.google.com/chronicle/docs/ingestion/parser-list/supported-default-parsers), which specifies the appropriate SecOps Parser for your logs. ### Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | | ✓ | | ### Configuration | Field | Description | | ---------------- | ------------------------------------------------------------------------------------------- | | Log Type | The type of log that will be sent. | | Namespace | User-configured environment namespace to identify the data domain the logs originated from. | | Ingestion Labels | Key-value pairs of labels to be applied to the logs when sent to chronicle. | ## Example Configuration ### Configure Google SecOps for Windows events This example configuration sets `logType` to "WINEVTLOG", `namespace` to "security", and `ingestionLabels` to a key-value pair: "environment" and "production". Standalone Processor: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: google_secops_standardization name: google_secops_standardization spec: type: google_secops_standardization parameters: - name: telemetry_types value: ['Logs'] - name: condition value: 'true' - name: googleSecOpsStandardization value: condition: true logType: WINEVTLOG namespace: status ingestionLabels: environment: production ``` # Group by Attributes Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/group-by-attributes Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.0.0+` | ## Description The Group by Attributes Processor groups telemetry with shared attribute values into the same resource. ## Use 1. "Promote" attributes so telemetry with those attributes gets grouped under a *resource* of that value. 2. [Compact](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbyattrsprocessor#compaction) telemetry data that shares a resource after batching. ## Configuration | Field | Description | | :-------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Attribute Keys | The attribute keys to group by and move to the resource. When no keys are specified, resources with the same attributes are compacted into the one resource. | ## Example Configurations ### 1. Promote an attribute An example of promoting the `user` attribute on logs: Honeycomb Docs - Group by Attributes - image 1

Honeycomb Docs - Group by Attributes - image 1

### Before processor Honeycomb Docs - Group by Attributes - image 2

### After

Honeycomb Docs - Group by Attributes - image 3

### 2. Compact telemetry with the same resource By default, the processor will [compact](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbyattrsprocessor#compaction) telemetry that shares the same resource. Using this processor after the batch processor can reduce the amount of data sent to some destinations. Honeycomb Docs - Group by Attributes - image 4

Honeycomb Docs - Group by Attributes - image 4

# Log Sampling Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/log-sampling Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Log Sampling processor can be used to filter out logs with a configured "drop ratio". ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :---------- | :----- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | drop\_ratio | `enum` | `"0.50"` | The probability an entry is dropped (used for sampling). A value of 1.0 will drop 100% of matching entries, while a value of 0.0 will drop 0%. | | condition | string | `true` | An [OTTL] expression used to match which log records to sample from. All paths in the [log context] are available to reference. All [converters] are available to use. | [OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.109.0/pkg/ottl#readme [converters]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.109.0/pkg/ottl/ottlfuncs/README.md#converters [log context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.109.0/pkg/ottl/contexts/ottllog/README.md Use of the `condition` parameter requires HTP Agent `v1.61.0` or above. Value drop ratio's range from "0.0" (0%) to "1.00" (100%) with 5% increments. Note that the drop ratio value is a string. ## Example Configuration Filter out 75% of logs where Attribute "ID" == 1. ### Web Interface Honeycomb Docs - Log Sampling - image 1

### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: sampling name: sampling spec: type: sampling parameters: - name: drop_ratio value: '0.75' - name: condition value: '(attributes["ID"] == 1)' ``` # Lookup Fields Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/lookup-fields Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Lookup Fields processor can be used to add matching telemetry fields from a CSV file. ## Use The Lookup Fields processor is used to dynamically add fields based on an existing telemetry value. For each unit of telemetry processed, this processor will grab the value of a field and perform a lookup using a CSV file. If that value exists, the processor will add all other fields associated with that CSV row. The processor re-reads the CSV file every 60 seconds, allowing updates to be made without restarting the agent. ## Supported Types | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.45.0\`+ | ## Configuration | Field | Description | | :------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CSV | The CSV file used to perform a lookup operation. | | Context | The context of the lookup operation. The source field must exist here in order to perform a lookup. If a lookup succeeds, all matching fields are also added to this location. | | Field | The field to lookup in the specified context. A lookup operation is performed if the name and value of this field match a header and value in the CSV file. | ## Example Configuration(s) ### Adding Fields Based on Host In this configuration, the processor will perform a lookup on the `host.name` value of all incoming metrics. If this field exists on the resource of a metric and the specified CSV file contains a matching value with that header, all other fields in that CSV row will be added to the metric. For example, in this particular configuration, if a metric has a value of `MacBook-Pro-4.local` for `host.name`, the corresponding values for `region` and `env` will automatically be added to the same context. #### Web Interface Honeycomb Docs - Lookup Fields - image 1

Honeycomb Docs - Lookup Fields - image 1

#### Example CSV ```csv theme={} host.name,region,env MacBook-Pro-4.local,us-east,prod MacBook-Pro-3.local,us-west,dev MacBook-Pro-2.local,us-central,dev MacBook-Pro-1.local,us-central,prod ``` # Marshal Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/marshal Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Marshal Processor moves fields onto the body and turns them into a JSON or Key Value string. To use this processor, the input body must not be a string; it must contain one or more fields. ## Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :----------------------------: | :--------: | :------: | :--------------------------------------------------------------------------------------------: | | format | enum | None | Which format to marshal to. Can be JSON, KV, or None. | | log\_fields\_to\_include | ottlFields | \[] | Which log fields to include. | | log\_fields\_to\_exclude | ottlFields | \[] | Which log fields to exclude. | | log\_selection | enum | Include | Whether to use include or exclude mode for log field selection. | | body\_fields\_to\_include | ottlFields | \[] | Which body fields to include. | | body\_fields\_to\_exclude | ottlFields | \[] | Which body fields to exclude. | | body\_selection | enum | Exclude | Whether to use include or exclude mode for body field selection. | | attribute\_fields\_to\_include | ottlFields | \[] | Which attribute fields to include. | | attribute\_fields\_to\_exclude | ottlFields | \[] | Which attribute fields to exclude. | | attribute\_selection | enum | Exclude | Whether to use include or exclude mode for attribute field selection. | | resource\_fields\_to\_include | ottlFields | \[] | Which resource fields to include. | | resource\_fields\_to\_exclude | ottlFields | \[] | Which resource fields to exclude. | | resource\_selection | enum | Exclude | Whether to use include or exclude mode for resource field selection. | | flatten | bool | false | Whether to flatten fields after moving to body. | | log\_field | string | bp.log | The name of the body field to move log fields into. If empty, moves fields to top level. | | attribute\_field | string | bp.attrs | The name of the body field to move attribute fields into. If empty, moves fields to top level. | | resource\_field | string | bp.res | The name of the body field to move resource fields into. If empty, moves fields to top level. | | kv\_delimiter | string | = | The delimiter to use between key and value. | | kv\_pair\_delimiter | string | | The delimiter to use between key value pairs. | | sort\_by\_keys | bool | false | Ensure deterministic ordering of keys before marshaling. | ### Basic Configuration Below is an example of configuration using the defaults. It will select all body, attributes, and resource fields but will not flatten or marshal them. Standalone Processor: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: marshal-default name: marshal-default spec: type: marshal parameters: - name: format value: None - name: log_fields_to_exclude value: [] - name: log_fields_to_include value: [] - name: log_selection value: Include - name: body_fields_to_exclude value: [] - name: body_fields_to_include value: [] - name: body_selection value: Exclude - name: attribute_fields_to_exclude value: [] - name: attribute_fields_to_include value: [] - name: attribute_selection value: Exclude - name: resource_fields_to_exclude value: [] - name: resource_fields_to_include value: [] - name: resource_selection value: Exclude - name: flatten value: false - name: log_field value: bp.log - name: attribute_field value: bp.attrs - name: resource_field value: bp.res - name: kv_delimiter value: = - name: kv_pair_delimiter value: ' ' - name: sort_by_keys value: false ``` ### Key Value Example The configuration below will flatten and marshal the body into a string like this: `name=test bp.log.severity_number=5 bp.attrs.baba=you bp.res.field1=val1 bp.res.field2=val2` In the advanced section, the KV delimiters can be customized and the `bp.log`, `bp.attrs`, and `bp.res` fields can be renamed or ignored, putting fields directly onto the body. Standalone Processor: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: marshal-kv name: marshal-kv spec: type: marshal parameters: - name: format value: KV - name: log_fields_to_exclude value: [] - name: log_fields_to_include value: - severity_number - name: log_selection value: Include - name: body_fields_to_exclude value: - severity - name: body_fields_to_include value: [] - name: body_selection value: Exclude - name: attribute_fields_to_exclude value: - host - name: attribute_fields_to_include value: [] - name: attribute_selection value: Exclude - name: resource_fields_to_exclude value: - field2 - field1 - name: resource_fields_to_include value: - field1 - field2 - name: resource_selection value: Include - name: flatten value: true - name: log_field value: bp.log - name: attribute_field value: bp.attrs - name: resource_field value: bp.res - name: kv_delimiter value: = - name: kv_pair_delimiter value: ' ' - name: sort_by_keys value: false ``` # Mask Sensitive Data Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/mask-sensitive-data Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Mask Sensitive Data processor can be used to detect and mask sensitive data. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :----------------------- | :-------- | :----------------------------------------- | :-------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `strings` | `[Metrics, Logs, Traces]` | Which types of telemetry to apply masking rules to. | | default\_rules | `enums` | See [default rules](#default-rules-values) | Commonly used masking rules. | | custom\_rules | `map` | See [custom rules](#custom-rules-values) | Create custom rules with the key being the rule name and the value being a regular expression to match against. | | exclude\_resource\_keys | `strings` | | A list of resource keys to exclude from masking. | | exclude\_attribute\_keys | `strings` | | A list of attribute keys to exclude from masking. | | exclude\_body\_keys | `strings` | | A list of log body keys to exclude from masking. | \* *required field* ### Default Rules Values * Credit Card: `\b(?:(?:(?:\d{4}[- ]?){3}\d{4}|\d{15,16}))\b` * Date of Birth: `\b(0?[1-9]|1[0-2])\/(0?[1-9]|[12]\d|3[01])\/(?:\d{2})?\d{2}\b` * Email: `\b[a-zA-Z0-9._\/\+\-—|]+@[A-Za-z0-9.\-—|]+\.?[a-zA-Z|]{0,6}\b` * International Bank Account Number (IBAN): `\b[A-Z]{2}\d{2}[A-Z\d]{1,30}\b` * IPv4 Address: `\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b` * IPv6 Address: `\b(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}\b` * MAC Address: `\b([0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}\b` * Phone Number: `\b((\+|\b)[1l][\-\. ])?$?\b[\dOlZSB]{3,5}([\-\. ]|$ ?)[\dOlZSB]{3}[\-\. ][\dOlZSB]{4}\b` * Social Security Number (SSN): `\b\d{3}[- ]\d{2}[- ]\d{4}\b` * US City, State: `\b[A-Z][A-Za-z\s\.]+,\s{0,1}[A-Z]{2}\b` * US Street Address: `\b\d+\s[A-z]+\s[A-z]+(\s[A-z]+)?\s*\d*\b` * US Zipcode: `\b\d{5}(?:[-\s]\d{4})?\b` * UUID/GUID: `\b[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-4[a-fA-F0-9]{3}-[8|9|aA|bB][a-fA-F0-9]{3}-[a-fA-F0-9]{12}\b` ## Example Configuration ### Basic Configuration Below is an example of configuration using the defaults. #### Web Interface Honeycomb Docs - Mask Sensitive Data - image 1

Honeycomb Docs - Mask Sensitive Data - image 1

Honeycomb Docs - Mask Sensitive Data - image 2

##### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: mask_sensitive_data name: mask_sensitive_data spec: type: mask_sensitive_data parameters: - name: telemetry_types value: ["Metrics", "Logs", "Traces"] - name: custom_rules value: birth_date_dash: \b(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])-(19|20)\d{2}\b ``` ### Exclusions You can exclude fields from being masked based on their key by specifying excluded keys in the body, resources, or attributes respectively. #### Web Interface Honeycomb Docs - Mask Sensitive Data - image 3

Honeycomb Docs - Mask Sensitive Data - image 3

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: mask_sensitive_data name: mask_sensitive_data spec: type: mask_sensitive_data parameters: - name: telemetry_types value: ['Metrics', 'Logs', 'Traces'] - name: exclude_resource_keys value: ['excluded_resource_key'] - name: exclude_attribute_keys value: ['excluded_attribute_key'] - name: exclude_body_keys value: ['excluded_body_key'] ``` # Move Field Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/move-field Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Move Field processor can be used to move a telemetry field. ## Use The Move Field processor is utilized for moving telemetry fields in metrics, logs, and traces based on specified conditions. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration | Field | Description | | :-------------- | :---------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | A condition that determines when this processor is applied. | | Move From | The telemetry field to move from. | | Move To | The telemetry field to move to. | ## Example Configuration(s) ### Moving An Entire Log Body In this configuration, the entire body field is moved to a field on attributes called `body_nested`. This method is useful for destinations that ignore or use the body field in an undesired manner. #### Web Interface Honeycomb Docs - Move Field - image 1

### Nesting A Field This configuration will nest an attributes field named `time_local` within another field named simply `time`. This is useful for simplifying or standardizing the data structure of incoming logs. In this example, note the use of bracket notation to create nested fields. #### Web Interface Honeycomb Docs - Move Field - image 2

# Parse CSV Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-csv Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.45.0`+ | ## Description The Parse CSV Processor is utilized to parse CSV strings from specified fields within log, metric, or trace data. It's particularly useful when your telemetry data contains serialized CSV strings, and you need to convert them into a structured format for easier analysis and filtering. The processor supports specifying the source field and the target field for the parsed CSV data, offering flexibility in handling diverse data structures. ## Use When dealing with telemetry data that includes CSV strings embedded within logs, metrics, or traces, the Parse CSV Processor becomes instrumental. For instance, logs from certain applications or systems might contain CSV strings representing specific attributes or metadata. By utilizing the Parse CSV Processor, these CSV strings can be parsed and converted into structured data, enhancing readability and facilitating more complex queries and analyses. ## Configuration | Field | Description | | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Type | The type of telemetry to apply the processor to. | | Condition | The condition to apply the CSV parsing. It supports OTTL expressions for logs, metrics, and traces. This field determines which telemetry data entries are processed based on their content and attributes. | | Source Field Type | Determines the type of source field for logs, metrics, or traces. This can be Resource, Attribute, Body, or Custom for logs and Resource, Attribute, or Custom for metrics and traces. It defines where the processor should look to find the CSV string to parse. | | Source Field | Specifies the exact field where the CSV string is located, based on the selected Source Field Type. For instance, if the Source Field Type is Attribute, this field should specify the particular attribute containing the CSV string. | | Target Field Type | Like the Source Field Type, this field determines the type of target field for logs, metrics, or traces where the parsed CSV data will be stored. The options are similar, allowing users to store the parsed data as a resource, attribute, body, or in a calculated field. | | Target Field | Specifies the exact field where the parsed CSV data will be stored, based on the selected Target Field Type. This allows users to organize and structure the parsed data in a manner that facilitates easy querying and analysis. | | Header Field Type | Like the Source Field Type, this field determines the type of header field for parsing the CSV line. The default option, Static String, allows you to specify the CSV headers as a fixed string. The other options are similar to Source Field, allowing users to select dynamic headers from a resource, attribute, body, or in a calculated field. | | Headers | Only relevant when Header Field Type is set to Static String. This is the static CSV header row to use when parsing. | | Header Field | Specifies the exact field where the CSV header row is located. This header will be used to determine the fields to use when parsing the CSV string. | | Delimiter | Specifies the delimiter to be used as the separator between fields. By default, "," is used. | | Header Delimiter | Specifies the delimiter to be used for the header row, if it differs from the delimiter used in the CSV row. If unspecified, Delimiter is used as the header delimiter. | | Mode | Specifies the mode to use when parsing. Strict mode follows normal CSV parsing rules. Lazy Quotes allows bare quotes in the middle of an unquoted field. Ignore Quotes ignores all quoting rules for CSV, splitting purely based on the delimiter. | ## Example Configurations ### Parse CSV from Logs In this example, we are looking to parse CSV strings from a log's body field and store the parsed data into the attributes field. The logs contain CSV strings detailing a web request, and we want to make this data more accessible. Honeycomb Docs - Parse CSV - image 1

Here is a sample log entry: ```json theme={} { "body": "10.0.0.1\tGET\t200", "attributes": { "log.file.name": "example.log", "log_type": "file" } } ``` We want to parse the CSV string from the Body and store it as structured data within the log entry. The configuration for the Parse CSV Processor would be: * Condition: `true` * Source Field Type: `Body` * Source Field: Left empty * Target Field Type: `Attribute` * Target Field: Left empty * Header Field Type: `Static String` * Headers: `ip,method,status` * Delimiter: `\t` * Header Delimiter: `ip,method,status` * Mode: `Strict` The resulting log entry after processing would be: ```json theme={} { "body": "10.0.0.1\tGET\t200", "attributes": { "log.file.name": "example.log", "log_type": "file", "ip": "10.0.0.1", "method": "GET", "status": "200" } } ``` This structured format makes it easier to filter and analyze the log data based on the ip, method and status fields. # Parse JSON Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-json Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.36.0`+ | ## Description The Parse JSON Processor is utilized to parse JSON strings from specified fields within log, metric, or trace data. It's particularly useful when your telemetry data contains serialized JSON strings, and you need to convert them into a structured format for easier analysis and filtering. The processor supports specifying the source field and the target field for the parsed JSON data, offering flexibility in handling diverse data structures. ## Use When dealing with telemetry data that includes JSON strings embedded within logs, metrics, or traces, the Parse JSON Processor becomes instrumental. For instance, logs from certain applications or systems might contain JSON strings representing specific attributes or metadata. By utilizing the Parse JSON Processor, these JSON strings can be parsed and converted into structured data, enhancing readability and facilitating more complex queries and analyses. ## Configuration | Field | Description | | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | The condition to apply the JSON parsing. It supports OTTL expressions for logs, metrics, and traces. This field determines which telemetry data entries are processed based on their content and attributes. | | Source Field Type | Determines the type of source field for logs, metrics, or traces. This can be Resource, Attribute, Body, or Custom for logs and Resource, Attribute, or Custom for metrics and traces. It defines where the processor should look to find the JSON string to parse. | | Source Field | Specifies the exact field where the JSON string is located, based on the selected Source Field Type. For instance, if the Source Field Type is Attribute, this field should specify the particular attribute containing the JSON string. | | Target Field Type | Like the Source Field Type, this field determines the type of target field for logs, metrics, or traces where the parsed JSON data will be stored. The options are similar, allowing users to store the parsed data as a resource, attribute, body, or in a calculated field. | | Target Field | Specifies the exact field where the parsed JSON data will be stored, based on the selected Target Field Type. This allows users to organize and structure the parsed data in a manner that facilitates easy querying and analysis. | ## Example Configurations ### Parse JSON from Logs In this example, we are looking to parse JSON strings from a log's attribute field and store the parsed data back into another attribute field. The logs contain JSON strings detailing additional information about log events, and we want to make this data more accessible. Honeycomb Docs - Parse JSON - image 1

Here is a sample log entry: ```JSON theme={} { "level": "info", "eventDetails": "{\"action\": \"login\", \"status\": \"success\"}" } ``` We want to parse the JSON string from the `eventDetails` attribute and store it as structured data within the log entry. The configuration for the Parse JSON Processor would be: * Condition: `"attributes['eventDetails'] != nil"` * Source Field Type: `Attribute` * Source Field: `eventDetails` * Target Field Type: `Attribute` * Target Field: `parsedEventDetails` The resulting log entry after processing would be: ```JSON theme={} { "level": "info", "eventDetails": "{\"action\": \"login\", \"status\": \"success\"}", "parsedEventDetails": { "action": "login", "status": "success" } } ``` This structured format makes it easier to filter and analyze the log data based on the action and status fields. # Parse Key Value Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-key-value Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.45.0`+ | ## Description The Parse Key Value processor is utilized to parse key value pairs from specified fields within log, metric, or trace data. The processor supports specifying the source field and the target field for the parsed key value data, offering flexibility in handling diverse data structures. ## Use When dealing with telemetry data that includes key value pairs embedded within logs, metrics, or traces, the Parse Key Value Processor becomes instrumental. For instance, logs from certain applications or systems might contain key value pairs representing specific attributes or metadata. By utilizing the Parse Key Value Processor, these key value pairs can be parsed and converted into structured data, enhancing readability and facilitating more complex queries and analyses. ## Configuration | Field | Description | | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | The condition to apply the Key Value parsing. It supports OTTL expressions for logs, metrics, and traces. This field determines which telemetry data entries are processed based on their content and attributes. | | Source Field Type | Determines the type of source field for logs, metrics, or traces. This can be Resource, Attribute, Body, or Custom for logs and Resource, Attribute, or Custom for metrics and traces. It defines where the processor should look to find the key value pairs to parse. | | Source Field | Specifies the exact field where the key value pairs are located, based on the selected Source Field Type. For instance, if the Source Field Type is Attribute, this field should specify the particular attribute containing the key value pairs. | | Target Field Type | Like the Source Field Type, this field determines the type of target field for logs, metrics, or traces where the parsed key value pairs will be stored. The options are similar, allowing users to store the parsed data as a resource, attribute, body, or in a calculated field. | | Target Field | Specifies the exact field where the parsed key value pairs data will be stored, based on the selected Target Field Type. This allows users to organize and structure the parsed data in a manner that facilitates easy querying and analysis. | | Delimiter | Specifies the string that should be used to split a key value pair. The default is `=`. | | Pair Delimiter | Specifies the string that should be used to separate multiple pairs from each other. The default is a single space(`" "`). | Parsing from the "Body" or "Attributes" field into the "Resource" field is not supported. ## Example Configurations ### Parse Key Value Pairs from Logs In this example, we are looking to parse key value pairs from a log's attribute field and store the parsed data in another attribute field. The logs contain key value pairs detailing additional information about log events, and we want to make this data more accessible. Honeycomb Docs - Parse Key Value - image 1

Honeycomb Docs - Parse Key Value - image 1

```JSON theme={} { "level": "info", "eventDetails": "action:login!status:success" } ``` We want to parse the key value pairs from the `eventDetails` attribute and store them as structured data within the log entry. The configuration for the Parse Key Value Processor would be: * Condition: `"attributes['eventDetails'] != nil"` * Source Field Type: `Attribute` * Source Field: `eventDetails` * Target Field Type: `Attribute` * Target Field: `parsedEventDetails` * Delimiter: `:` * Pair Delimiter: `!` The resulting log entry after processing would be: ```JSON theme={} { "level": "info", "eventDetails": "action:login!status:success", "parsedEventDetails": { "action": "login", "status": "success" } } ``` This structured format makes it easier to filter and analyze the log data based on the action and status fields. # Parse with Regex Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-regex Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.36.0`+ | ## Description The Parse with Regex Processor is designed to extract and transform telemetry data—including logs, metrics, and traces—using regular expressions (regex) with named capture groups. This enables users to define specific regex patterns with named capture groups to parse and reformat data from different source fields, enhancing data analysis and insights. ## Use This processor is invaluable when users need to extract and categorize specific elements from unstructured or semi-structured data. Users can employ regex patterns with named capture groups to classify extracted data, making it easily identifiable and accessible for further analysis, monitoring, or alerting. ## Configuration | Field | Description | | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | The condition, expressed in OTTL, that must be met for the processor to be applied. Allows users to apply specific criteria to select the data entries to be processed. | | Source Field Type | Indicates the type of the source field where the regex will be applied. It can be Resource, Attribute, Body, or Custom for logs; Resource, Attribute, or Custom for metrics and traces. | | Source Field | Specifies the specific field where the regex is applied, pertinent to the selected Source Field Type. | | Target Field Type | Indicates the type of the target field where the parsed data will be stored. It can be Resource, Attribute, Body, or Custom for logs; Resource, Attribute, or Custom for metrics and traces. | | Regex Pattern | The regex pattern with a named capture group used for parsing the data, essential for extracting or transforming specific data elements within the telemetry data. | ## Example Configurations ### Extract Error Codes from Log Messages In this example, the Parse with Regex Processor is configured to extract error codes embedded within log messages. Given the unstructured nature of these messages, the use of a regex pattern with a named capture group is crucial for efficient extraction and categorization. Honeycomb Docs - Parse with Regex - image 1

Honeycomb Docs - Parse with Regex - image 1

Here is a sample log entry divided into body and attributes: **Body:** ```JSON theme={} { "message": "2023-06-20 14:32:10 Error: An error occurred. ErrorCode: ER1023" } ``` **Attributes:** ```JSON theme={} { "timestamp": "2023-06-20 14:32:10" } ``` The objective is to extract the error code "ER1023" and assign it to a new attribute for enhanced analysis. The configuration for the Parse with Regex Processor is as follows: * Condition: `"body contains 'ErrorCode:'"` * Source Field Type: `Body` * Source Field: `message` * Target Field Type: `Attribute` * Regex Pattern: `"ErrorCode: (?P\w+)"` With this setup, the named capture group "errorCode" is employed to categorize the extracted error code. The processed log entry would appear with an updated attributes section as follows: **Attributes After Processing:** ```JSON theme={} { "timestamp": "2023-06-20 14:32:10", "errorCode": "ER1023" } ``` Now, the error code is not only extracted but also categorized under the "errorCode" attribute, facilitating effortless filtering and analysis. This structured format allows for precise monitoring and troubleshooting, especially when dealing with specific error codes. # Parse Severity Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-severity Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Collector | | :------ | :--- | :----- | :--------------------------- | | - | ✓ | - | `v1.36.0`+ | ## Description The Parse Severity Processor is designed to normalize severity fields in log data into user-defined values, enhancing the consistency and readability of log data. By allowing users to map existing severity values to standard levels, it aids in the uniform analysis and visualization of logs across varied sources. ## Use The processor is essential in environments where logs from different sources use varied severity naming conventions. By mapping these diverse severity indicators to standard values, it ensures that the severity data remains consistent, making it easier to filter, analyze, and generate insights from the log data. ## Configuration | Field | Description | | :------------- | :---------------------------------------------------------------------------------------------------------------------------------- | | Condition | An OTTL condition that must evaluate to true for the processor to be applied to the logs, allowing selective processing of entries. | | Match | Specifies the location of the severity value in the log entry: body, resource, or attributes. | | Severity Field | The specific field that contains the severity value to be parsed and normalized. | | Trace | A list of values that 'TRACE' severity should be mapped to. | | Debug | A list of values that 'DEBUG' severity should be mapped to. | | Info | A list of values that 'INFO' severity should be mapped to. | | Warn | A list of values that 'WARN' severity should be mapped to. | | Error | A list of values that 'ERROR' severity should be mapped to. | | Fatal | A list of values that 'FATAL' severity should be mapped to. | ## Example Configurations ### Available Parsing Formats In addition to simple string matching, this processor supports some unique value mapping options. For example, HTTP status code ranges can easily be assigned using notation such as `2xx`, seen below. Available HTTP status code ranges include `1xx`, `2xx`, `3xx`, `4xx`, and `5xx`. Honeycomb Docs - Parse Severity - HTTP Status Range

Honeycomb Docs - Parse Severity - HTTP Status Range

Another unique value mapping is a range of numbers, such as `8-12`. This will map any number in that range, such as 9, to the log level this range is assigned to. Honeycomb Docs - Parse Severity - Number Range

Honeycomb Docs - Parse Severity - Number Range

### Normalize Severity Levels in Log Data In this example, the Parse Severity Processor is configured to normalize severity levels from the "level" field in the log body into user-defined standard levels. Honeycomb Docs - Parse Severity - image 1

Honeycomb Docs - Parse Severity - image 1

Here is a sample log entry: **Body:** ```JSON theme={} { "message": "An error occurred during processing", "level": "err" } ``` The objective is to map the "err" severity level to a standard "error" level for consistency across all log entries. The configuration for the Parse Severity Processor is as follows: * Condition: `"true"` (applies to all logs) * Log Body, Resource, or Attributes: `Body` * Severity Field: `level` * Severity Mappings: ```JSON theme={} { "err": "error", "warning": "warn", "information": "info", "debugging": "debug", "critical": "fatal" } ``` With this setup, when the log entry is processed, the "severity" field is updated as follows: **Log After Processing:** ```JSON theme={} { "severity": "error" } ``` The severity level "err" is now normalized to "error," allowing for a uniform representation of severity levels across all log entries. This normalization facilitates more straightforward log analysis, filtering, and alerting, especially when dealing with logs from multiple sources with different severity naming conventions. # Parse Timestamp Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-timestamp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.36.0`+ | ## Description The Parse Timestamp Processor is designed to extract and standardize timestamps from specified fields in log, metric, or trace data. It ensures uniformity of timestamp data across different sources and formats, facilitating seamless analysis and correlation of time-series data. ## Use In environments where telemetry data comes in various timestamp formats or from different fields, uniformity in timestamp data is crucial for accurate analysis and monitoring. The Parse Timestamp Processor addresses this by allowing users to specify the source field and format, enabling the extraction and standardization of timestamps across diverse data types and sources. ## Configuration | Field | Description | | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The types of telemetry to apply the processor to. | | Condition | A condition expressed in OTTL that must be true for the processor to be applied. It supports dynamic evaluation, allowing for conditional application of the processor based on the content or attributes of the telemetry data. | | Source Field Type | Determines the type of source field to parse the timestamp from. It can be either Resource, Attribute, or Body for logs, and Resource or Attribute for metrics and traces. | | Source Field | The specific field from which to parse the timestamp. | | Time Format | The format of the timestamp in the source field. Supported formats include RFC3339, ISO8601, Epoch, and Manual, catering to a wide range of timestamp formats encountered in real-world data. | | Epoch Layout | Applicable when the Time Format is set to Epoch. It determines the layout of the epoch timestamp, ensuring accurate parsing of timestamps represented as epoch time. | | Manual Layout | Required when the Time Format is set to Manual. It defines the strptime layout for parsing timestamps, offering flexibility to handle custom timestamp formats beyond the standard RFC3339 and ISO8601 formats. | ## Example Configurations ### Standardizing Log Timestamps In this example, we configure the Parse Timestamp Processor to extract and standardize timestamps embedded within log messages. The log entries contain timestamps in various formats, and the goal is to normalize them for consistent analysis. Honeycomb Docs - Parse Timestamp - image 1

Honeycomb Docs - Parse Timestamp - image 1

Sample log entry with a non-standard timestamp format: ```JSON theme={} { "level": "info", "timestamp": "20/06/2023 14:32:10" } ``` The configuration for the Parse Timestamp Processor is set as follows: * Condition: `"attributes['timestamp'] != nil"` * Source FieldType: `Attribute` * Source Field: `timestamp` * Time Format: `Manual` * Manual Layout: `%d/%m/%Y %H:%M:%S` As a result, the log entry is processed to extract and standardize the timestamp, transforming it into a consistent, machine-readable format for enhanced querying and analysis. Processed log entry: ```JSON theme={} { "level": "info", "timestamp": "2023-06-20T14:32:10Z" } ``` This setup ensures that all timestamps, regardless of their original format, are standardized to facilitate accurate and efficient data analysis. # Parse XML Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/parse-xml Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Metrics | Logs | Traces | Telemetry Pipeline Agent | | :------ | :--- | :----- | :----------------------- | | ✓ | ✓ | ✓ | `v1.46.0`+ | ## Description The Parse XML Processor is utilized to parse XML document strings from specified fields within log, metric, or trace data. It's particularly useful when your telemetry data contains a serialized XML document, and you need to convert them into a structured format for easier analysis and filtering. The processor supports specifying the source field and the target field for the parsed XML data, offering flexibility in handling diverse data structures. ## Use When dealing with telemetry data that includes an XML document embedded within logs, metrics, or traces, the Parse XML Processor becomes instrumental. For instance, logs from certain applications or systems might contain XML documents representing specific attributes or metadata. By utilizing the Parse XML Processor, these XML documents can be parsed and converted into structured data, enhancing readability and facilitating more complex queries and analyses. **Multiline XML** It's common for XML to be formatted to span multiple lines. When reading XML logs from a file, make sure to configure the multiline section of the File source to properly read the whole XML document. The parsed XML is structured as follows: 1. All character data for an XML element is trimmed and placed in the `content` field. 2. The tag for an XML element is trimmed and placed in a `tag` field. 3. The attributes for an XML element is placed as a mapping of attribute name to attribute value in the `attribute` field. 4. Processing instructions, directives, and comments are ignored and not represented in the parsed XML. 5. All child XML elements are parsed as above, and placed in an array in a `children` field. As an example, see the following XML: ```xml theme={} 00001 Joe joe.smith@example.com User fired alert A ``` This XML, when parsed, becomes: ```json theme={} { "tag": "Log", "children": [ { "tag": "User", "children": [ { "tag": "ID", "content": "00001" }, { "tag": "Name", "content": "Joe", "attributes": { "type": "first" } }, { "tag": "Email", "content": "joe.smith@example.com" } ] }, { "tag": "Text", "content": "User fired alert A" } ] } ``` ## Configuration | Field | Description | | :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Telemetry Type | The type of telemetry to apply the processor to. | | Condition | The condition to apply the XML parsing. It supports OTTL expressions for logs, metrics, and traces. This field determines which telemetry data entries are processed based on their content and attributes. | | Source Field Type | Determines the type of source field for logs, metrics, or traces. This can be Resource, Attribute, Body, or Custom for logs and Resource, Attribute, or Custom for metrics and traces. It defines where the processor should look to find the XML document to parse. | | Source Field | Specifies the exact field where the XML document is located, based on the selected Source Field Type. For instance, if the Source Field Type is Attribute, this field should specify the particular attribute containing the XML document. | | Target Field Type | Like the Source Field Type, this field determines the type of target field for logs, metrics, or traces where the parsed XML data will be stored. The options are similar, allowing users to store the parsed data as a resource, attribute, body, or in a calculated field. | | Target Field | Specifies the exact field where the parsed XML data will be stored, based on the selected Target Field Type. This allows users to organize and structure the parsed data in a manner that facilitates easy querying and analysis. | ## Example Configurations ### Parse XML from Logs In this example, we have a basic log that details an action and the user that triggered the action, like an audit log. This log is in XML format, and we'd like to parse the content into a structured log. Honeycomb Docs - Parse XML - image 1

Here is a sample log record: ```json theme={} { "body": "00001JoeUser did a thing" } ``` In order to parse the body of the log record, and store it on the `parsed_xml` attribute, we can configure the Parse XML processor as follows: * Telemetry: `Logs` * Condition: `true` * Source Field Type: `Body` * Source Field: Left empty * Target Field Type: `Attribute` * Target Field: `parsed_xml` After parsing, the log record looks like this: ```json theme={} { "body": "00001JoeUser did a thing", "attributes": { "parsed_xml": { "children": [ { "children": [ { "content": "00001", "tag": "ID" }, { "children": [ { "content": "Joe", "tag": "First" } ], "tag": "Name" } ], "tag": "User" }, { "content": "User did a thing", "tag": "Text" } ], "tag": "Log" } } } ``` # Processor Impact on Agent Performance Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/performance Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. Processors are a very powerful tool in the Honeycomb Telemetry Pipeline. They allow you to transform your data before it arrives at your destination. This can be useful for adding attributes, filtering, and converting logs to metrics. The question is, how do processors impact agent performance? We'll look at some common uses cases for processors and see how they can impact agent performance. ## Benchmarking Setup For our benchmarking setup we have two agents running on a GCP [n2-standard-2](https://cloud.google.com/compute/docs/general-purpose-machines#n2_machine_types) Compute Engine instance, which has 2 vCPUs and 8GB of memory. One agent will be the test agent that we will be applying processors to. The other agent will be acting as our destination. It is configured with an OTLP Source and custom destination to log the output of the data it receives. For the log generation tests static JSON logs were generated at a rate of 1000 logs per second, across 10 files. This roughly equates to 140MiB of logs per minute. The log source is the [Filelog](/send-data/telemetry-pipeline/sources/filelog) source with JSON parsing configured. For metric generation the [Host Metrics](/send-data/telemetry-pipeline/sources/host-metrics) source was used with the default configuration. ## Control Runs A one hour control run was done for both the log and metric generation tests. This was to establish a baseline for the agent's performance without any processors applied. ### Logs Control The control run for logs was conducted collecting from 10 log files with logs generating at 1000 logs per second to each file. This results in a throughput of \~140MiB/m. The agent's average CPU usage was 27% and average memory usage was 2.8% (\~230MB). This is the topology view of the control configuration: Logs Control Topology

These are the graphs showing CPU and memory usage over the hour long run: Logs Control Results

### Metrics Control The control run for metrics had a data throughput of \~99KiB/m. The agent's average CPU usage was 0.2% and average memory usage was 2.1% (\~16MB). This is the topology view of the control configuration: Logs Control Topology

These are the graphs showing CPU and memory usage over the hour long run: Logs Control Results

## Deduplicating Logs Deduplicating logs is a common data reduction use case for logs. It can help reduce the number of redundant logs that are sent to a destination. For our tests we used the [Deduplicate Logs](/send-data/telemetry-pipeline/processors/deduplicate-logs) processor to deduplicate logs. The processor is configured to deduplicate logs excluding the timestamp and `log.file.name` attributes. The benchmark was run for 30 minutes. The agent's average CPU usage was 16% and average memory usage was 2.8% (\~230MB). This is actually lower than the control run. This is due to the fact that the logs are being deduplicated and the agent is processing less data. This is the topology view of the benchmark configuration: Logs DeDup Topology

These are the graphs showing CPU and memory usage over the 30 minute run: (The benchmark takes place in the section between the blue lines.) Logs DeDup Results

### Benchmark Conclusion The above images show that apply a log deduplication processor does not negatively impact the agent's performance. It actually reduces CPU usage as it reduces the amount of data the agent has to process. ## Converting Logs to Metrics Converting logs to metrics is a great way to reduce cost and data ingestion of logs if you just want to see some metric based off of the logs. For this benchmark we will be solving the use case of wanting to know the number of logs for each severity and discard the logs themselves. We will use the [Count Telemetry](/send-data/telemetry-pipeline/processors/count-telemetry) processor to count the number of info, warning, and error logs. We will use the [Filter by Field](/send-data/telemetry-pipeline/processors/filter-by-field) processor to filter out the logs and only send the metrics. Here is the order in which the processors are applied: Logs To Metric Processors

The benchmark was run for 30 minutes. The agent's average CPU usage was 20% and average memory usage was 2.5% (\~205MB). Like the Log DeDuplication Benchmark, this is actually lower than the control run. This is the topology view of the benchmark configuration: (It shows 0B/m after the processors as they are all being filtered out.) Logs To Metrics Topology

These are the graphs showing CPU and memory usage over the 30 minute run: (The benchmark takes place in the section between the blue lines.) Logs to Metrics Results

### Benchmark Conclusion The above images show that apply even with four processors in use it does not negatively impact the agent's performance. Both CPU and memory usage are lower than the control run. ## Lookup Fields The [Lookup Fields](/send-data/telemetry-pipeline/processors/lookup-fields) processor allows adding additional fields to telemetry based on existing fields. This can be useful for adding additional context before sending to a destination. This benchmark we add an additional field to the logs based on the `env` field in the body. The benchmark was run for 30 minutes. The agent's average CPU usage was 25% and average memory usage was 2.5% (\~205MB). This is fairly close to the control run. This is the topology view of the benchmark configuration: Logs Lookup Topology

These are the graphs showing CPU and memory usage over the 30 minute run: (The benchmark takes place in the section between the blue lines.) Logs Lookup Results

### Benchmark Conclusion The results are fairly close to the control run. We don't see a performance improvement like the previous runs as we are adding a small amount of metadata to the logs rather than reducing overall throughput. ## Delete Fields The [Delete Fields](/send-data/telemetry-pipeline/processors/delete-fields/) processor allows removing fields from telemetry. This can be useful for removing extraneous or sensitive data from telemetry before sending it to a destination. This benchmark was run against both logs and metrics deleting fields from both. ### Logs The benchmark was run for 30 minutes. The agent's average CPU usage was 24% and average memory usage was 2.4% (\~196MB). This is fairly close to the control run. This is the topology view of the benchmark configuration: Delete Fields Log Topology

These are the graphs showing CPU and memory usage over the 30 minute run: (The benchmark takes place in the section between the blue lines.) Delete Fields Log Results

#### Benchmark Conclusion The results are fairly close to the control run with using slightly less CPU and memory as the logs are slightly smaller due to the fields being removed. ### Metrics The benchmark was run for 30 minutes. The agent's average CPU usage was 0.2% and average memory usage was 2.1% (\~172MB). This is identical to the metrics control run. This is the topology view of the benchmark configuration: Delete Fields Metrics Topology

These are the graphs showing CPU and memory usage over the 30 minute run: (The benchmark takes place in the section between the blue lines.) Delete Fields Metrics Results

#### Benchmark Conclusion The results were identical to the metrics control run. Since volume of metrics and the rate at which they are processed is low, the impact of the processor is negligible. ## Takeaways The benchmarks show that using processors do not negatively impact the agent's performance. In fact, in some cases it can reduce the amount of data the agent has to process and reduce CPU usage. Data reduction can significantly reduce the resource usage of an agent. Ordering processors so data reduction happens earlier in the pipeline can help reduce the overall resource usage of the agent. The impact of a processor is dependent on the volume of data being processed. The log benchmarks showed a greater impact from processors than the metric benchmarks. This is due to the volume of logs being processed being much higher than the volume of metrics. # Redact Sensitive Data Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/redact-sensitive-data Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Redact Sensitive Data processor detects and redacts sensitive data. ## Supported Types | Metrics | Logs | Traces | Bindplane Collector | | ------- | ---- | ------ | ------------------- | | ✓ | ✓ | ✓ | `v1.81.0`+ | ## Configuration Table

Parameter	Type	Default	Description
telemetry\_types	`strings`	`\[Metrics, Logs, Traces]`	Which types of telemetry to apply redaction rules to.
redaction\_strategy\*	`enum`	`Censor with Asterisks`	The approach used to redact sensitive data. `Censor with Asterisks` will replace all detected sensitive data with asterisks (e.g. `\\\\\\`). `Hashing` will replace all detected sensitive data with their hashed values based on the `hash\_function` parameter.
hash\_function	`enum`	`sha3`	The hash function used to redact sensitive data. Only relevant if `redaction\_strategy` is set to `Hashing`. Must be one of `sha3`, `sha1`, or `md5`. `sha3` is highly recommended when dealing with sensitive data as it is secure and future-proof. `sha1` and `md5` are faster hashing algorithms, but are insecure and susceptible to attacks.
redaction\_rules	`enums`	See default rules	Commonly used redaction rule presets.
custom\_rules	`strings`		A list of regular expressions representing custom redaction rules. Any data that matches any rule will be redacted according to the `redaction\_strategy` parameter.
ignored\_values	`strings`		A list of regex patterns. Data that matches will not be redacted regardless of its value.
redacted\_keys	`strings`		A list of regex patterns corresponding to keys of resource attributes, attributes, or log body fields. Keys that match will have their values redacted regardless of their value.
ignored\_keys	`strings`		A list of names (not regex patterns) corresponding to keys of resource attributes, attributes, or log body fields. Keys that match will not have their values redacted regardless of their value.
added\_summary\_attributes\*	`enum`	`None`	The summary attributes added to the telemetry. Must be one of `None`, `Redacted Field Counts`, or `Redacted Field Counts and Names`. `Redacted Field Counts` will add attributes to the outgoing telemetry with the count of redacted telemetry. `Redacted Field Counts and Names` will also add attributes with the name of the attribute keys that were redacted. Note: Does not have an effect when configured within a Google SecOps Stream.

\* *required field* ### Default Rules Values * Credit Card: `\b(?:3[47][ -]?\d{4}[ -]?\d{6}[ -]?\d{3}|4\d{3}(?:[ -]?\d{4}){3}|5[1-5]\d{2}(?:[ -]?\d{4}){3}|6(?:011|5\d{2})(?:[ -]?\d{4}){3}|35(?:2[89]|[3-8]\d)(?:[ -]?\d{4}){3}|3(?:0[0-5]|[68]\d)(?:[ -]?\d){11}|62(?:[ -]?\d){14,17})\b` * Date of Birth: `\b(?:(?:19|20)?\d{2}[-/])?(?:0?[1-9]|1[0-2])-/(?:-/?\d{2})?\b` * Email: `\b[a-zA-Z0-9._/+-—|]+@[A-Za-z0-9-—|]+.[a-zA-Z|]{2,6}\b` * International Bank Account Number (IBAN): `\b[A-Z]{2}\d{2}(?: ?[A-Z0-9]){11,31}(?:\s[A-Z0-9])*\b` * IPv4 Address: `\b(?:(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9]).){3}(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\b` * IPv6 Address: `\b(?:[0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}\b` * MAC Address: `\b([0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}\b` * US Phone Number: `\b((+|\b)[1l][-. ])?(?\b[\dOlZSB]{3,5}([-. ]|) ?)[\dOlZSB]{3}[-. ][\dOlZSB]{4}\b` * International Phone Number: `+[1-9]\d{0,2}(?:[-.\s]?(?\d+)?(?:[-.\s]?\d+)*)\b` * Social Security Number (SSN): `\b\d{3}[- ]\d{2}[- ]\d{4}\b` * US City, State: `\b[A-Z][A-Za-z\s\.]+,\s{0,1}[A-Z]{2}\b` * US Street Address: `\b\d+\s[A-z]+\s[A-z]+(\s[A-z]+)?\s*\d*\b` * US Zip Code: `\b\d{5}(?:[-\s]\d{4})?\b` * UUID/GUID: `\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}\b` ## Basic Example Configuration Below is an example of a basic Redact Sensitive Data processor configuration using a subset of the preset redaction rules. Standalone Processor: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor spec: parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: redaction_strategy value: Censor with Asterisks - name: redaction_rules value: - Social Security Number (SSN) - Credit Card - IPv4 Address - US Street Address - US City, State - US Zip Code ``` ## Advanced Example Configuration Below is an example of an advanced Redact Sensitive Data configuration using custom redaction rules, and allowing/ignoring specific values & keys. Standalone Processor: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor spec: parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: redaction_strategy value: Hashing - name: hash_function value: sha3 - name: redaction_rules value: [] - name: custom_rules value: - ^\d{9}$ - ^[a-zA-Z]+@[a-zA-Z]+.com$ - name: ignored_values value: - ^[a-zA-Z]+@example.com$ - name: redacted_keys value: - ^service.*$ - name: ignored_keys value: - service.name - service.env - name: added_summary_attributes value: Redacted Field Counts and Names ``` ### Redaction Strategy This configuration sets the `redaction_strategy` parameter to `Hashing` and the `hash_function` parameter to `sha3`. This replaces any matching sensitive data with a `sha3` hash of its value as seen in the Web Interface screenshot above. ### Custom Rules This example does not select any of the preset redaction rules, and instead configures two custom rules to match passport IDs (nine digit numbers), and alphabetical emails that end in `.com`. ### Ignored Values This example configures the `ignored_values` parameter to process example emails (ending in "example.com") without redaction, even though they will match the custom email rule. ### Redacted Keys This example configures the `redacted_keys` parameter to always redact the value of any keys that begin with "service." Note that this applies to all levels of keys (resource attributes, attributes, and log body keys). ### Ignored Keys The above example configures two keys to be ignored by the redaction processor: "service.name" and "service.env". These two values would have normally been redacted due to the *Redacted Keys* configuration, but will now be ignored. Note that this applies to all levels of keys (resource attributes, attributes, and log body keys). ### Added Summary Attributes The above example sets the value of the `added_summary_attributes` parameter to `Redacted Field Counts and Names`. This adds various metrics in the form of attributes (at both the resource & span level). These metrics track the count of redacted attributes and the names of each attribute that is redacted, as seen in the Web Interface screenshot above. # Rename Field Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/rename-field Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Rename Fields processor can be used to rename resource, attribute, and log body fields. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :--------------- | :------------------ | :------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | The list of telemetry types the processor will act on. | | Condition | `string` | `true` | An OTTL condition that must evaluate to true to apply this processor. By default, the processor applies to all telemetry. | | Field Type | `string` | `attributes` | Determines whether the field is renamed in the body, attributes, or resource fields of the telemetry. Note: Body fields are applicable only for logs. | | Old Name | `string` | `""` | Specifies the existing field in the telemetry that the processor will rename. | | New Name | `string` | `""` | Indicates the new field name that will replace the old field name in the telemetry data. | ## Example Configuration ### Renaming log body fields In this example, we rename the `status` field in a log body to `status_code`. Since this change only applies to logs, we have disabled metrics and traces for this processor. #### Web Interface Honeycomb Docs - Rename Field - image 1

#### Standalone Processor ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor metadata: id: rename_fields_v2 name: rename_fields_v2 spec: type: rename_fields_v2 parameters: - name: telemetry_types value: ['Logs'] - name: condition value: 'true' - name: fields value: - fieldType: body key: status value: status_code ``` # Rename Metric Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/rename-metric Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Rename Metric processor can be used to rename metrics. ## Use The Rename Metric processor is utilized for renaming metrics. It supports renaming either the entire name of a single metric or the prefix of multiple metrics. ## Supported Types | Metrics | Logs | Traces | | :------ | :--- | :----- | | ✓ | | | ## Configuration | Field | Description | | :--------- | :--------------------------------------------------------------------------------------------------------------------------------- | | Operation | The operation to perform when renaming a metric. `Name`: Rename an entire metric. `Prefix`: Rename the prefix of multiple metrics. | | Old Name | The name of an incoming metric to rename. Required when `Operation` is set to `Name`. | | New Name | The new name of an incoming metric. Required when `Operation` is set to `Name`. | | Old Prefix | The prefix of incoming metrics to rename. Required when `Operation` is set to `Prefix`. | | New Prefix | The new prefix of incoming metrics. Required when `Operation` is set to `Prefix`. | ## Example Configuration(s) ### Renaming A Single Metric In this configuration, the `system.network.packets` metric is renamed to just `network.traffic`. #### Web Interface Honeycomb Docs - Rename Metric - image 1

Honeycomb Docs - Rename Metric - image 1

### Renaming A Metric Prefix In this configuration, the `system` prefix for host metrics is replaced with `macos`. #### Web Interface Honeycomb Docs - Rename Metric - image 2

Honeycomb Docs - Rename Metric - image 2

# Resource Detection Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/resource-detection Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The resource detection processor can be used to detect resource information from the host, in a\ format that conforms to the [OpenTelemetry resource semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/), and append or override the resource value in telemetry data with this information. ## Supported Types | Metrics | Logs | Traces | Bindplane Collector | | ------- | ---- | ------ | ------------------- | | ✓ | ✓ | ✓ | `v1.40.0`+ | ## Configuration Table

Field	Description
Detector	Detector(s) to use for resource detection. Available detectors include `aks`, `azure`, `consul`, `ec2`, `eks`, `gcp`, `k8snode`, `openshift`, and `system`.
Override	Whether or not to overwrite existing resource attributes.
Timeout	Time duration after which a resource detector request will timeout.
Hostname Source	When the `system` detector is in use, specifies the source used to detect the system hostname. Options include `os`, `dns`, `cname`, and `lookup`.
EC2 Tags	Specifies a list of regex's to match to EC2 instance tag keys that will be added as resource attributes to processed data.
K8s Node Environment Variable	The K8s node environment variable that has the node name to retrieve metadata for. Defaults to "KUBE\_NODE\_NAME" because the Bindplane Collector has this automatically set.
Consul Address	The address of the Consul server. If unset, the environment variable "CONSUL\_HTTP\_ADDR" will be used if it exists.
Consul Authentication	The type of authentication to use with Consul. One of 'Token' or 'Token File' is required if Consul's ACL System is enabled. Options are "None", "Token", or "Token File".
Consul Token	Token is used to provide a per-request ACL token which overrides the collector's default token. If unset, the environment variable "CONSUL\_HTTP\_TOKEN" will be used if it exists.
Consul Token File	Token File is a file containing the current token to use for this client. If provided, it is read once at startup and never again. If unset, the environment variable "CONSUL\_HTTP\_TOKEN\_FILE" will be used if it exists.
Consul Datacenter	Optional Consul Datacenter to use. If not provided, the default collector datacenter is used.
Consul Namespace	Optional namespace to attach to each Consul request. If unset, the environment variable "CONSUL\_NAMESPACE" will be used if it exists.
Consul Metadata Labels	Allowlist of Consul Metadata keys to use as resource attributes.

Multiple detectors may be selected on a single processor. However, if multiple processors use a common attribute name, the first detector will have precedent. For more info, see this OTel [documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#ordering). # Rewrite Timestamp Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/rewrite-timestamp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Rewrite Timestamp processor can be used to write the value from the Time field of a telemetry entry into a string value. ## Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ## Configuration

Field	Description
Condition	A condition that determines when this processor is applied.
Target Field Type	The telemetry field type where the newly formatted timestamp will be placed.
Target Field	The target field to place the result into. May only be blank if Body is selected for the Target Field Type.
Timestamp Format	The ctime variable based timestamp format to write.

# Split Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/split Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The `Split` processor splits a string into a list of strings using a specified delimiter. ## Use The `Split` processor divides a string into a list of strings using a delimiter. Common delimiters include commas (`,`) and newlines (`\`). ### Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | | ## Configuration

Field	Description
Telemetry Types	Specifies which types of telemetry the processor should be applied to.
Delimiter	The character or string used to split the field.
Condition	Defines when the processor should be applied. If not specified, the processor will attempt to process all telemetry.
Split From	The source field to split.
Split To	The destination field that will store the split values.

# Transform Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/transform Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Transform processor allows for configuring advanced conditional transformation rules on incoming telemetry. ## Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ## Configuration Table

Parameter	Type	Default	Description
telemetry\_type\*	`enum`	`Logs`	Specifies the type of telemetry to which transformation statements are applied. One of `Logs` , `Metrics` , or `Traces`
statements	`strings`		The OTTL statements to execute on matching telemetry. The statements are executed in order from top to bottom.
conditions	`strings`		OTTL conditions to determine if statements should run. If any condition evaluates to true on a given log, metric, or trace, the OTTL statements will be executed on that telemetry. Individual statements may still add their own `where` clauses.
error\_mode\*	`enum`	`ignore`	One of `ignore` , `silent` , or `propagate` . Determines how error handling will be performed during processing. `Ignore` will ignore errors, log the error, and continue processing. `Silent` will ignore errors without logging them, and continue processing. `Propagate` will halt processing when an error occurs and return errors up the pipeline.
metrics\_context	`enum`	`datapoint`	Either `datapoint` or `metric` . This determines the context at which the transform statements are executed on metrics. Read about OTTL contexts here.
traces\_context	`enum`	`span`	Either `span` or `spanevent` . This determines the context at which the transform statements are executed on traces. Read about OTTL contexts here.

\* *required field* ### Basic Example Configuration Below is an example of a basic Transform processor configuration using multiple statements and one condition. Standalone Processor: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Processor spec: parameters: - name: telemetry_type value: Logs - name: statements value: - set(attributes["myNewFieldKey"], "myNewFieldValue") - delete_key(attributes, "myUnwantedKey") - name: conditions value: - attributes["service.name"] = "my-service-name" - name: error_mode value: ignore ``` # Unroll Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/processors/unroll Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The `Unroll` processor operates on log entries that contain a list/array in their `body` field. It expands the data into multiple log entries, with each item in the list becoming a separate entry while preserving the original attributes, resource attributes, and timestamp. The unroll processor may cause UI rendering slowdowns when processing log entries that expand into a large number of entries. ## Use The `Unroll` processor expands log entries that contain multiple logical entries within a single record. ## Supported Types | Metrics | Logs | Traces | Bindplane Agent | | ------- | ---- | ------ | --------------- | | | ✓ | | `v1.66.0+` | ## Configuration The unroll processor has no configurable options and only processes log entries with a list/array in their `body` field. # Pipeline Manager Sources Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources Browse available sources for the Honeycomb Telemetry Pipeline Manager to connect infrastructure, logs, metrics, and traces to your pipeline. Enterprise add-on. Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The following sources are available for the Honeycomb Telemetry Pipeline Manager. ### Infrastructure & Cloud Sources | Source | Metrics | Logs | Traces | Description | | ----------------------------------------------------------------------------------------------------------- | ------- | ---- | ------ | ------------------------------------------------ | | [AWS CloudWatch](/send-data/telemetry-pipeline/sources/aws-cloudwatch/) | | ✓ | | Collect logs from AWS CloudWatch | | [AWS S3 Event](/send-data/telemetry-pipeline/sources/aws-s3-event/) | | ✓ | | Collect events from AWS S3 | | [Azure Blob Storage](/send-data/telemetry-pipeline/sources/azure-blob-storage/) | | ✓ | | Collect data from Azure Blob Storage | | [Azure Event Hub](/send-data/telemetry-pipeline/sources/azure-event-hub/) | ✓ | ✓ | | Collect data from Azure Event Hub | | [Google Cloud PubSub](/send-data/telemetry-pipeline/sources/google-cloud-pubsub/) | ✓ | ✓ | ✓ | Collect data from Google Cloud Pub/Sub | | [Google Cloud Storage Rehydration](/send-data/telemetry-pipeline/sources/google-cloud-storage-rehydration/) | | ✓ | | Rehydrate data from Google Cloud Storage | | [Kubernetes Cluster Events](/send-data/telemetry-pipeline/sources/kubernetes-cluster-events/) | | ✓ | | Collect Kubernetes cluster events | | [Kubernetes Cluster Metrics](/send-data/telemetry-pipeline/sources/kubernetes-cluster-metrics/) | ✓ | | | Collect Kubernetes cluster metrics | | [Kubernetes Container Logs](/send-data/telemetry-pipeline/sources/kubernetes-container-logs/) | | ✓ | | Collect Kubernetes container logs | | [Kubernetes Kubelet Metrics](/send-data/telemetry-pipeline/sources/kubernetes-kubelet-metrics/) | ✓ | | | Collect Kubernetes kubelet metrics | | [Kubernetes Prometheus Node](/send-data/telemetry-pipeline/sources/kubernetes-prometheus-metrics/) | ✓ | | | Collect Prometheus metrics from Kubernetes nodes | | [Host Metrics](/send-data/telemetry-pipeline/sources/host-metrics/) | ✓ | | | Collect system metrics | | [macOS](/send-data/telemetry-pipeline/sources/macos/) | ✓ | | | Collect macOS system metrics | | [VMware ESXi](/send-data/telemetry-pipeline/sources/vmware-esxi/) | ✓ | | | Monitor VMware ESXi hosts | | [VMware vCenter](/send-data/telemetry-pipeline/sources/vmware-vcenter/) | ✓ | | | Monitor VMware vCenter | ### Application & Database Sources | Source | Metrics | Logs | Traces | Description | | ----------------------------------------------------------------------------------- | ------- | ---- | ------ | ------------------------------------ | | [Apache Combined](/send-data/telemetry-pipeline/sources/apache-combined/) | | ✓ | | Collect Apache combined format logs | | [Apache Common](/send-data/telemetry-pipeline/sources/apache-common/) | | ✓ | | Collect Apache common format logs | | [Apache HTTP](/send-data/telemetry-pipeline/sources/apache-http/) | ✓ | ✓ | | Monitor Apache web servers | | [Apache Spark](/send-data/telemetry-pipeline/sources/apache-spark/) | ✓ | | | Monitor Apache Spark | | [Aerospike](/send-data/telemetry-pipeline/sources/aerospike/) | ✓ | | | Monitor Aerospike database | | [Cassandra](/send-data/telemetry-pipeline/sources/cassandra/) | ✓ | | | Monitor Apache Cassandra | | [CockroachDB](/send-data/telemetry-pipeline/sources/cockroachdb/) | ✓ | ✓ | | Monitor CockroachDB | | [Couchbase](/send-data/telemetry-pipeline/sources/couchbase/) | ✓ | | | Monitor Couchbase | | [CouchDB](/send-data/telemetry-pipeline/sources/couchdb/) | ✓ | ✓ | | Monitor Apache CouchDB | | [Elasticsearch](/send-data/telemetry-pipeline/sources/elasticsearch/) | ✓ | | | Monitor Elasticsearch | | [HBase](/send-data/telemetry-pipeline/sources/hbase/) | ✓ | | | Monitor Apache HBase | | [Hadoop](/send-data/telemetry-pipeline/sources/hadoop/) | ✓ | | | Monitor Apache Hadoop | | [JBoss](/send-data/telemetry-pipeline/sources/jboss/) | ✓ | | | Monitor JBoss application server | | [JVM](/send-data/telemetry-pipeline/sources/jvm/) | ✓ | | | Monitor Java Virtual Machine metrics | | [MongoDB](/send-data/telemetry-pipeline/sources/mongodb/) | ✓ | | | Monitor MongoDB | | [MongoDB Atlas](/send-data/telemetry-pipeline/sources/mongodb-atlas/) | ✓ | | | Monitor MongoDB Atlas | | [Microsoft IIS](/send-data/telemetry-pipeline/sources/microsoft-iis/) | ✓ | ✓ | | Monitor Microsoft IIS web server | | [Microsoft SQL Server](/send-data/telemetry-pipeline/sources/microsoft-sql-server/) | ✓ | | | Monitor Microsoft SQL Server | | [Microsoft 365](/send-data/telemetry-pipeline/sources/microsoft-365/) | | ✓ | | Collect Microsoft 365 logs | | [MySQL](/send-data/telemetry-pipeline/sources/mysql/) | ✓ | ✓ | | Monitor MySQL database | | [Oracle Database](/send-data/telemetry-pipeline/sources/oracle-database/) | ✓ | ✓ | | Monitor Oracle Database | | [PostgreSQL](/send-data/telemetry-pipeline/sources/postgresql/) | ✓ | ✓ | | Monitor PostgreSQL databases | | [PgBouncer](/send-data/telemetry-pipeline/sources/pgbouncer/) | ✓ | | | Monitor PgBouncer connection pooler | | [Redis](/send-data/telemetry-pipeline/sources/redis/) | ✓ | ✓ | | Monitor Redis instances | | [SAP HANA](/send-data/telemetry-pipeline/sources/sap-hana/) | ✓ | | | Monitor SAP HANA | | [SAP NetWeaver](/send-data/telemetry-pipeline/sources/sap-netweaver/) | ✓ | | | Monitor SAP NetWeaver | | [Solr](/send-data/telemetry-pipeline/sources/solr/) | ✓ | | | Monitor Apache Solr | | [SQL Query](/send-data/telemetry-pipeline/sources/sql-query/) | ✓ | | | Execute SQL queries for metrics | | [Tomcat](/send-data/telemetry-pipeline/sources/tomcat/) | ✓ | | | Monitor Apache Tomcat | | [WildFly](/send-data/telemetry-pipeline/sources/wildfly/) | ✓ | | | Monitor WildFly application server | | [ZooKeeper](/send-data/telemetry-pipeline/sources/zookeeper/) | ✓ | | | Monitor Apache ZooKeeper | ### Message Queue & Streaming Sources | Source | Metrics | Logs | Traces | Description | | --------------------------------------------------------------------- | ------- | ---- | ------ | ------------------------------------- | | [Kafka Cluster](/send-data/telemetry-pipeline/sources/kafka-cluster/) | ✓ | ✓ | | Monitor Kafka clusters | | [Kafka Node](/send-data/telemetry-pipeline/sources/kafka-node/) | ✓ | | | Monitor Kafka nodes | | [Kafka OTLP](/send-data/telemetry-pipeline/sources/kafka-otlp/) | ✓ | ✓ | ✓ | Collect OpenTelemetry data from Kafka | | [RabbitMQ](/send-data/telemetry-pipeline/sources/rabbitmq/) | ✓ | ✓ | | Monitor RabbitMQ message broker | | [HAProxy](/send-data/telemetry-pipeline/sources/haproxy/) | ✓ | | | Monitor HAProxy load balancer | ### Network & Security Sources | Source | Metrics | Logs | Traces | Description | | --------------------------------------------------------------------------- | ------- | ---- | ------ | ----------------------------------------------- | | [Active Directory](/send-data/telemetry-pipeline/sources/active-directory/) | ✓ | ✓ | | Monitor Active Directory | | [Cisco ASA](/send-data/telemetry-pipeline/sources/cisco-asa/) | | ✓ | | Collect logs from Cisco ASA | | [Cisco Catalyst](/send-data/telemetry-pipeline/sources/cisco-catalyst/) | | ✓ | | Collect logs from Cisco Catalyst | | [Cisco Meraki](/send-data/telemetry-pipeline/sources/cisco-meraki/) | | ✓ | | Collect logs from Cisco Meraki | | [Cloudflare](/send-data/telemetry-pipeline/sources/cloudflare/) | | ✓ | | Collect Cloudflare logs | | [CrowdStrike FDR](/send-data/telemetry-pipeline/sources/crowdstrike-fdr/) | | ✓ | | Collect CrowdStrike Falcon Data Replicator logs | | [F5 BIG-IP](/send-data/telemetry-pipeline/sources/f5-big-ip/) | ✓ | | | Monitor F5 BIG-IP load balancers | | [NetFlow](/send-data/telemetry-pipeline/sources/netflow/) | | ✓ | | Collect NetFlow data | | [Okta](/send-data/telemetry-pipeline/sources/okta/) | | ✓ | | Collect logs from Okta | | [Packet Capture](/send-data/telemetry-pipeline/sources/packet-capture/) | | ✓ | | Collect network packets | | [Ubiquiti](/send-data/telemetry-pipeline/sources/ubiquiti/) | ✓ | | | Monitor Ubiquiti devices | | [Windows DHCP](/send-data/telemetry-pipeline/sources/windows-dhcp/) | | ✓ | | Collect Windows DHCP logs | ### Logging & Monitoring Sources | Source | Metrics | Logs | Traces | Description | | ------------------------------------------------------------------------------------------------------------------- | ------- | ---- | ------ | -------------------------------------------------------- | | [Honeycomb Telemetry Pipeline](/send-data/telemetry-pipeline/sources/honeycomb-telemetry-pipeline/) | ✓ | | | Monitor Honeycomb Telemetry Pipeline itself | | [Honeycomb Telemetry Pipeline Collector](/send-data/telemetry-pipeline/sources/honeycomb-telemetry-pipeline-agent/) | ✓ | | | Monitor Honeycomb Telemetry Pipeline agents & collectors | | [Honeycomb Telemetry Pipeline Gateway](/send-data/telemetry-pipeline/sources/honeycomb-telemetry-pipeline-gateway/) | ✓ | ✓ | | Monitor Honeycomb Telemetry Pipeline gateway | | [Common Event Format](/send-data/telemetry-pipeline/sources/common-event-format/) | | ✓ | | Collect Common Event Format logs | | [CSV](/send-data/telemetry-pipeline/sources/csv/) | ✓ | | | Collect metrics from CSV files | | [Custom](/send-data/telemetry-pipeline/sources/custom/) | ✓ | ✓ | ✓ | Custom OpenTelemetry receiver configuration | | [File Stats](/send-data/telemetry-pipeline/sources/filestats/) | ✓ | | | Collect file statistics | | [Fluent Forward](/send-data/telemetry-pipeline/sources/fluent-forward/) | | ✓ | | Collect logs via Fluentd forward protocol | | [HTTP Check](/send-data/telemetry-pipeline/sources/http-check/) | ✓ | | | Monitor HTTP endpoints | | [HTTP Log](/send-data/telemetry-pipeline/sources/http-log/) | | ✓ | | Collect logs via HTTP | | [Journald](/send-data/telemetry-pipeline/sources/journald/) | | ✓ | | Collect systemd journal logs | | [Logstash](/send-data/telemetry-pipeline/sources/logstash/) | | ✓ | | Collect logs from Logstash | | [macOS Unified Logging](/send-data/telemetry-pipeline/sources/macos-unified-logging/) | | ✓ | | Collect Unified Logging logs | | [OpenTelemetry OTLP](/send-data/telemetry-pipeline/sources/opentelemetry/) | ✓ | ✓ | ✓ | Standard OpenTelemetry protocol | | [Prometheus](/send-data/telemetry-pipeline/sources/prometheus/) | ✓ | | | Scrape Prometheus metrics | | [Splunk HEC](/send-data/telemetry-pipeline/sources/splunk-hec/) | | ✓ | | Collect logs via Splunk HTTP Event Collector | | [Splunk Search API](/send-data/telemetry-pipeline/sources/splunk-search-api/) | | ✓ | | Collect logs via Splunk Search API | | [Splunk TCP](/send-data/telemetry-pipeline/sources/splunk-tcp/) | | ✓ | | Collect logs via Splunk TCP | | [StatsD](/send-data/telemetry-pipeline/sources/statsd/) | ✓ | | | Collect StatsD metrics | | [Syslog](/send-data/telemetry-pipeline/sources/syslog/) | | ✓ | | Collect Syslog messages | | [TCP](/send-data/telemetry-pipeline/sources/tcp/) | | ✓ | | Collect logs via TCP | | [Telemetry Generator](/send-data/telemetry-pipeline/sources/telemetry-generator/) | ✓ | ✓ | ✓ | Generate test telemetry data | | [UDP](/send-data/telemetry-pipeline/sources/udp/) | | ✓ | | Collect logs via UDP | | [W3C](/send-data/telemetry-pipeline/sources/w3c/) | | ✓ | | Collect W3C format logs | | [Windows Events](/send-data/telemetry-pipeline/sources/windows-events/) | | ✓ | | Collect Windows Event logs | | [Windows Event Trace ETW](/send-data/telemetry-pipeline/sources/windows-event-trace-etw/) | | ✓ | | Collect Windows ETW traces | ### File-based Sources | Source | Metrics | Logs | Traces | Description | | --------------------------------------------------------------------------------------- | ------- | ---- | ------ | -------------------------------------- | | [Filelog](/send-data/telemetry-pipeline/sources/filelog/) | | ✓ | | Collect logs from files | | [AWS S3 Rehydration](/send-data/telemetry-pipeline/sources/aws-s3-rehydration/) | | ✓ | | Rehydrate logs from AWS S3 | | [Azure Blob Rehydration](/send-data/telemetry-pipeline/sources/azure-blob-rehydration/) | | ✓ | | Rehydrate logs from Azure Blob Storage | # Active Directory Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/active-directory Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Windows | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :-------------------- | :---------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | # Aerospike Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/aerospike Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------ | :------------------ | :-------------------- | :--------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | hostname\* | `string` | localhost | The hostname or IP address of the Aerospike system. | | port | `int` | 3000 | The TCP port of the Aerospike system. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | collect\_cluster\_metrics | `bool` | false | Whether discovered peer nodes should be collected. | | aerospike\_enterprise | `bool` | false | Enable Aerospike enterprise authentication. | | username\* | `string` | | The username to use when connecting to Aerospike. | | password\* | `string` | | The password to use when connecting to Aerospike. | | start\_at | `enum` | end | Start reading Aerospike Journald logs from 'beginning' or 'end'. | \* *required field* # Apache Combined Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/apache-combined Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :-------- | :--------------------------------- | :-------------------------------------------- | | file\_path | `strings` | \["/var/log/apache\_combined.log"] | Paths to Apache combined formatted log files. | | start\_at | `enum` | end | Start reading logs from 'beginning' or 'end'. | # Apache Common Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/apache-common Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :-------- | :------------------------------- | :-------------------------------------------- | | file\_path | `strings` | \["/var/log/apache2/access.log"] | Path to Apache common formatted log file. | | start\_at | `enum` | end | Start reading logs from 'beginning' or 'end'. | # Apache HTTP Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/apache-http Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | hostname\* | `string` | localhost | The hostname or IP address of the Apache HTTP system. | | port | `int` | 3000 | The TCP port of the Apache HTTP system. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS when connecting to the Apache HTTP server. | | strict\_tls\_verify | `bool` | false | Enable to require TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate TLS certificates. It's not required if the collector's operating system already trusts the certificate authority. | | mutual\_tls | `bool` | false | Enable to require TLS mutual authentication. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | | start\_at | `enum` | end | Start reading logs from 'beginning' or 'end'. | | access\_log\_path | `strings` | \["/var/log/apache2/access.log"] | Access Log File paths to tail for logs. | | error\_log\_path | `strings` | \["/var/log/apache2/error.log"] | Error Log File paths to tail for logs. | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | \* *required field* # Apache Spark Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/apache-spark Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Configuration Table | Field | Description | | :--------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------ | | Telemetry Types | The types of telemetry to gather with this receiver. | | Endpoint | The endpoint of the Apache Spark REST API. | | Enable TLS | Whether to use TLS when connecting to Apache Spark. | | Skip TLS Certificate Verification | Enable to skip TLS certificate verification. This is relevant if TLS is enabled. | | TLS Certificate Authority File | Certificate authority used to validate TLS certificates. This is relevant if TLS is enabled. | | Mutual TLS Client Certificate File | A TLS certificate used for client authentication. This is relevant if TLS is enabled. | | TLS Client Private Key File | A TLS private key used for client authentication. This is relevant if TLS is enabled. | | Collection Interval | Sets how often (seconds) to scrape for metrics. | | Allowed Spark Application Names | Filters that define which Spark applications are scraped for metrics. If undefined, all applications at the endpoint will be scraped. | | Spark Metrics | A list of Cluster, Job, Executor, and Stage metrics to be included or excluded. | # AWS Cloudwatch Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/aws-cloudwatch Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites While installing AWS CLI is not required in order to collect logs using the AWS Cloudwatch source type, it still provides an easier means of authentication. With AWS Cloudwatch, users are required to provide some form of authentication. This can be either profile credentials or environment variables that provide access keys for user accounts. Setting these credentials up can prove tedious and confusing. Luckily, AWS CLI can generate the user's profile/credentials with the `aws configure` command. Here is the [AWS CLI Getting Started](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) Guide, which outlines a few prerequisites. Some prerequisites: 1. Creating an IAM user account * Required Permissions * `logs:GetLogEvents` * `logs:DescribeLogGroups` * `logs:DescribeLogStreams` * The user does not require console access 2. Create an access key ID and secret access key *** In the [AWS CLI getting started guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), it will instruct you to install it for your current user or all users.\ The observIQ OTEL Collector runs [as root](https://github.com/observIQ/observiq-otel-collector/blob/main/docs/installation-linux.md#configuring-the-collector) by default, meaning the AWS CLI and credentials should be installed under the collector system's root account. *** ### Credentials ### Credential and Config Files AWS Authentication utilizes a user profile specified in the user's home directory at `.aws/credentials`. Each profile's credentials should include, at minimum, the profile name, access key, and secret access key. ```text theme={} [profile_name] aws_access_key_id=****** aws_secret_access_key=****** ``` In addition to the credentials file, there is also a `.aws/config`. This includes less sensitive configuration options such as the region, output format, etc. A typical entry in the `config` file should look as such. ```text theme={} [profile_name] region=us-west-2 output=json ``` More information on [AWS Configuration and Credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) ### Environment Variables Alternatively, [AWS Environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) can be specified to override a credentials file.\ You can modify the collector's environment variables by configuring a Systemd override.\ Run `sudo systemctl edit observiq-otel-collector` and add your access key, secret key, and region. ```text theme={} [Service] Environment=AWS_ACCESS_KEY_ID=****** Environment=AWS_SECRET_ACCESS_KEY=****** Environment=AWS_DEFAULT_REGION=us-west-2 ``` After making that change, reload Systemd and restart the collector service. ```bash theme={} sudo systemctl daemon-reload sudo systemctl restart observiq-otel-collector ``` ## Setup 1. Once logged in, Select the `Configs` tab at the top of the Honeycomb Telemetry Pipeline Home page. 2. Select a pre-existing config or create a new one. 3. Add a new source and select `AWS Cloudwatch`. 4. After configuring credentials using the AWS CLI on the collector system, using the default values in the source form should enable the collector to collect logs from Cloudwatch. If credentials were configured using environment variables, you will need to leave the `Profile` field blank 5. Click save configuration. 6. Add the destination type of your choosing. 7. Apply configuration to the desired agent. 8. Voila. Logs should be collecting ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------ | :------------------------ | :----------- | :---------------------------------------------------------------------------------------------------------------- | | region\* | `enum` | us-east-1 | The AWS recognized region string. | | profile \* | `string` | "default" | The AWS profile is used to authenticate; if none is specified, the default is chosen from the list of profiles. | | credential\_type | `enum` | profile | Determines whether to pull credentials from a `credentials` file or use environment variables for authentication. | | discovery\_type | `enum` | AutoDiscover | Configuration for Log Groups. By default, all Log Groups and Log Streams will be collected. | | limit | `int` | 50 | Limits the number of discovered log groups. | | prefix | `string` | "" | A prefix for log groups to limit the number of log groups discovered. | | names | `strings` | \[] | A list of full log stream names to filter the discovered log groups to collect from. | | prefixes | `strings` | \[] | A list of prefixes to filter the discovered log groups to collect from. | | named\_groups | `awsCloudwatchNamedField` | \[] | Configuration for Log Groups. By default, all Log Groups and Log Streams will be collected. | | imds\_endpoint | `string` | "" | A way of specifying a custom URL to be used by the EC2 IMDS client to validate the session. | | poll\_interval | `int` | 1 | The duration of waiting in between requests (minutes). | | max\_events\_per\_request | `int` | 50 | The maximum number of events to process per request to Cloudwatch. | \* *required field* ## Discovery Type ### Default Settings When starting with an AWS Cloudwatch set to its default values, you should see log collection from all log groups with no filtering of log streams. The default polling interval for Cloudwatch is 1 minute, so there may be a delay before seeing any logs coming through. ### AutoDiscover When using Discovery Type `AutoDiscover`, there are some optional parameters that can be added to / filter the amount of logs collected. * `limit`: limits the number of discovered log groups(default = 50). * `prefix`: Prefix for log groups to limit the number of log groups discovered * `prefix: /aws/eks/` * If omitted, all log groups up to the limit will be collected. * `names`: A list of full log stream names to filter the discovered log groups to collect from. * `names: [kube-apiserver-ea9c831555adca1815ae04b87661klasdj]` * `prefixes`: A list of log stream prefixes to filter the discovered log groups to collect from. * `prefixes: [kube-api-controller]` ### Named This Discovery Type filters logs by listing only the desired log groups to collect from and omitting any other log groups. When selecting this Discovery Type, at least one log group is required otherwise no logs would be collected. When listing log groups the `ID` field of each log group instance should match the full name of the group. Additionally, `Named` also provides `prefixes` and `names` parameters for each listed log group that filters out the listed log streams. These parameters should be listed underneath each log group's IDs as they are unique to each individual log group. # AWS S3 Event Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/aws-s3-event Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The AWS S3 Event Receiver consumes S3 event notifications for object creation events (`s3:ObjectCreated:*`) and emits the S3 object as the string body of a log record. ## Supported Platforms | Platform | Supported | | -------- | --------- | | Linux | ✓ | | Windows | ✓ | | macOS | ✓ | Available in the Bindplane Distro for OpenTelemetry Collector `v1.74.0+`. ## Prerequisites * An AWS account with access to S3 and SQS. * An SQS queue configured to receive S3 event notifications. 1. Ensure the collector has permission to read and delete messages from the SQS queue. 2. Ensure the collector has permission to read objects from the S3 bucket. ## How It Works 1. The receiver polls an SQS queue for S3 event notifications. 2. When an object creation event (`s3:ObjectCreated:*`) is received, the receiver downloads the S3 object. 3. The receiver reads the object into the body of a new log record. 4. Non-object creation events are ignored but removed from the queue. 5. If an S3 object is not found (404 error), the corresponding SQS message is preserved for retry later. ## Configuration Fields

Field	Type	Default	Required	Description
sqs\_queue\_url	string		`true`	The URL of the SQS queue to poll for S3 event notifications.
standard\_poll\_interval	duration	15	`false`	The interval (in seconds) at which the SQS queue is polled for messages.
max\_poll\_interval	duration	2	`false`	The maximum interval (in seconds) at which the SQS queue is polled for messages.
polling\_backoff\_factor	float	2	`false`	The factor by which the polling interval is multiplied after an unsuccessful poll.
workers	int	5	`false`	The number of workers to process events.
visibility\_timeout	duration	300	`false`	How long (in seconds) messages received from the queue will be invisible to other consumers.
visibility\_extension\_interval	duration	1m	`false`	How often to extend message visibility during processing. Should be less than visibility\_timeout. Minimum is 10s.
max\_visibility\_window	duration	1h	`false`	Maximum total time a message can remain invisible before becoming visible to other consumers. Must be less than SQS's 12-hour limit.
max\_log\_size	int	1048576	`false`	The maximum size of a log record in bytes. Logs exceeding this size will be split
max\_logs\_emitted	int	1000	`false`	The maximum number of log records to emit in a single batch. A higher number will result in fewer batches, but more memory.
bucket\_name\_filter	string		`false`	When set, the source will only emit logs for bucket names that match the specified regex.
object\_key\_filter	string		`false`	When set, the source will only emit logs for object names that match the specified regex.
notification\_type	enum	`s3`	`false`	Options are `s3` and `sns`. S3 notifications are the default for S3 to SQS. SNS is for S3 notifications sent via SNS to SQS.
enable\_offset\_storage	bool	`false`	`false`	When enabled, the current position into an object will be saved to Redis, and reading will resume from where it left off after a collector restart.
redis\_hostname	string		`false`	The hostname or IP address of the Redis server used for offset storage.
redis\_port	int	6379	`false`	The port number of the Redis server used for offset storage.
redis\_password	string		`false`	The password for the Redis server used for offset storage.
redis\_database	string		`false`	The Redis database number to use for offset storage.
redis\_expiration	int	0	`false`	The expiration time (in seconds) for offset storage in Redis.
redis\_tls\_enabled	bool	`false`	`false`	Whether to enable TLS for the Redis connection.
redis\_tls\_ca\_file	string		`false`	The path to the CA file for Redis TLS connection.
redis\_tls\_cert\_file	string		`false`	The path to the client certificate file for Redis TLS connection.
redis\_tls\_key\_file	string		`false`	The path to the client key file for Redis TLS connection.

## Component Telemetry This component emits telemetry that can provide insight into how it is performing. The collector is configured to emit these metrics to `localhost:8888/metrics` by default.

Metric Name	Type	Description
otelcol\_s3event\_batch\_size	histogram	The number of logs in a batch.
otelcol\_s3event\_objects\_handled\_total	counter	The number of S3 objects processed by the receiver.
otelcol\_s3event\_failures\_total	counter	The number of failures encountered while processing S3 objects.

# AWS S3 Rehydration Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/aws-s3-rehydration Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Object Format Rehydrated data must be in [OTLP format JSON](https://github.com/open-telemetry/opentelemetry-proto) for correct processing. Ensure your data adheres to this format. The [AWS S3 Destination](/send-data/telemetry-pipeline/destinations/aws-s3/) supports `gzip` compression. This receiver will gracefully detect and handle if any data is gzipped or uncompressed. ## Supported Platforms | Platform | Supported | | :------- | :-------- | | Linux | ✓ | | Windows | ✓ | | macOS | ✓ | ## Configuration Fields | Field | Description | | :---------------- | :----------------------------------------------------------------------------------------------------------------------------------------- | | Region | The AWS recognized region string | | Bucket | Name of the S3 Bucket to rehydration telemetry from. | | Folder Prefix | Root directory of the bucket to rehydration telemetry from. | | Poll Interval | The interval for checking new objects, e.g., '1m' for one minute. | | Starting Time | UTC start time for rehydration in the format `YYYY-MM-DDTHH:MM`. | | Ending Time | UTC end time for rehydration in the format `YYYY-MM-DDTHH:MM`. | | Delete on Read | If true, objects are deleted after rehydration. | | Enable Storage | Enable to specify a storage extension for tracking rehydration progress. | | Storage Directory | Directory for storing rehydration state, useful for maintaining state and resuming operations. (Only relevant if `Enable Storage` is true) | ## Example Configuration ### Basic Configuration This configuration sets up AWS S3 Rehydration with necessary details such as Region, Bucket, and time range for rehydration. Honeycomb Docs - AWS S3 Rehydration

Adjust the Region, Bucket, Starting Time, and Ending Time to match your configuration. # Azure Blob Rehydration Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/azure-blob-rehydration Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites Ensure you have access to an Azure Blob Storage account. Set up your Azure Blob Storage to store OTLP data in the required format for rehydration. More information on how to set this up can be found in the Azure documentation [here](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview). ## Blob Format Rehydrated data must be in [OTLP format](https://github.com/open-telemetry/opentelemetry-proto) for correct processing. Ensure your data adheres to this format. ## Supported Platforms | Platform | Supported | | :------- | :-------- | | Linux | ✓ | | Windows | ✓ | | macOS | ✓ | ## Configuration Fields | Field | Description | | :---------------- | :----------------------------------------------------------------------------------------------------------------------------------------- | | Connection String | The connection string for the Azure Blob Storage account. | | Container | Name of the Azure Storage container from which to rehydrate data. | | Poll Interval | The interval for checking new blobs, e.g., '1m' for one minute. | | Starting Time | UTC start time for rehydration. | | Ending Time | UTC end time for rehydration. | | Delete on Read | If true, blobs are deleted after rehydration. | | Enable Storage | Enable to specify a storage extension for tracking rehydration progress. | | Storage Directory | Directory for storing rehydration state, useful for maintaining state and resuming operations. (Only relevant if `Enable Storage` is true) | ## Example Configuration ### Basic Configuration This configuration sets up Azure Blob Rehydration with necessary details such as connection string, container, and time range for rehydration. Honeycomb Docs - Azure Blob Rehydration

Adjust the Connection String, Container, Starting Time, and Ending Time to match your configuration. # Azure Blob Storage Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/azure-blob-storage Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites Setup an Event Hub to listen for `Blob Create` events. More information on how to set this up can be found in the Azure documentation [here](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview). ## Blob Format Stored logs and traces must be in [OTLP JSON format](https://github.com/open-telemetry/opentelemetry-proto#otlpjson) in order to be correctly parsed by the receiver. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | ✓ | | Windows | | ✓ | ✓ | | macOS | | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Traces"]` | Choose Telemetry Type. | | connection\_string\* | `string` | "" | The connection string for the Azure Storage account. Information can be found [here](https://learn.microsoft.com/en-gb/azure/storage/common/storage-configure-connection-string). | | event\_hub\_endpoint\* | `string` | "" | The Azure Event Hub endpoint triggering on the `Blob Create` events. Information can be found [here](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview). | | logs\_container\* | `string` | "logs" | Name of the Azure Storage container where logs are stored. | | traces\_container\* | `string` | "traces" | Name of the Azure Storage container where traces are stored | \* *required field* ## Example Configuration ### Basic Configuration For basic configuration only, the `connection_string` for Azure Storage and the `event_hub_endpoint` need to be supplied. In the example below, we are using a fake `connection_string` and `event_hub_endpoint` in the form Azure expects. #### Web Interface Honeycomb Docs - Azure Blob Storage - image 1

#### Standalone Source ```yaml YAML theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: azure_blob name: azure_blob spec: type: azure_blob parameters: - name: connection_string value: 'DefaultEndpointsProtocol=https;AccountName=accountName;AccountKey=+idLkHYcL0MUWIKYHm2j4Q==;EndpointSuffix=core.windows.net' - name: event_hub_endpoint value: 'Endpoint=sb://oteldata.servicebus.windows.net/;SharedAccessKeyName=otelhubbpollicy;SharedAccessKey=mPJVubIK5dJ6mLfZo1ucsdkLysLSQ6N7kddvsIcmoEs=;EntityPath=otellhub' ``` # Azure Event Hub Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/azure-event-hub Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites The source must have access to an Azure Event Hub with the necessary permissions, the minimum permission being `Listen`. You can configure resources' Diagnostic settings to send logs to the Azure Event Hub. Read more here: [Azure Event Hub](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings). Each event hub should only accept one telemetry type. ## Supported Platforms HTP Agent: `v1.39.0`+ | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration | Field | Description | | :------------- | :-------------------------------------------------------------------------------------------- | | Telemetry Type | The type of telemetry to gather with this receiver. | | Connection | A string describing the connection to an Azure event hub. | | Consumer Group | The Consumer Group to read from. Defaults to `$Default` if empty. | | Partition | The partition to watch. If empty, it will watch all partitions. | | Offset | The offset at which to start watching the event hub. If empty, starts with the latest offset. | ## Example Configuration ### Basic Configuration For basic configuration, the `connection` parameter is required. Optionally, specify consumer group, `partition`, and `offset`. #### Web Interface Honeycomb Docs - Azure Event Hub Source - image 1

Honeycomb Docs - Azure Event Hub Source - image 1

#### Standalone Source ```yaml YAML theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: azure_event_hub name: azure_event_hub spec: type: azure_event_hub parameters: - name: telemetry_types value: Logs - name: connection value: '' - name: group value: '$Default' # optional - name: partition value: '1' # optional - name: offset value: '-1' # optional ``` # Cassandra Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/cassandra Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports Apache Cassandra versions 3.11 and 4.0. ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :------------------------------------------------ | :------------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | address | string | localhost | IP address or hostname to scrape for Cassandra metrics. | | port | int | 7199 | Port to scrape for Cassandra metrics. | | jar\_path | string | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | | collection\_interval | int | 60 | How often (seconds) to scrape for metrics. | | enable\_system\_logs | bool | true | Enables collecting system logs. | | system\_log\_path | strings | \["/var/log/cassandra/system.log"] | File paths to system logs. | | enable\_debug\_logs | bool | true | Enables the collection of debug logs. | | debug\_log\_path | strings | \["/var/log/cassandra/debug.log"] | File paths to debug logs. | | enable\_gc\_logs | bool | true | Enables collection of garbage collector logs. | | gc\_log\_path | strings | \["/var/log/cassandra/gc.log"] | File paths to garbage collection logs. | | timezone | timezone | UTC | The timezone to use when parsing timestamps. | | start\_at | enum | end | Start reading logs from 'beginning' or 'end'. | # Cisco ASA Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/cisco-asa Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :----------- | :------- | :-------- | :------------------------------------------------------------------------------ | | listen\_port | `int` | 5140 | A TCP port, which the agent will listen for syslog messages. | | listen\_ip | `string` | "0.0.0.0" | An IP address for the agent to bind. Typically 0.0.0.0 for most configurations. | # Cisco Catalyst Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/cisco-catalyst Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :----------- | :--------- | :-------- | :------------------------------------------------------------------------------ | | listen\_port | `int` | 5140 | A UDP port, which the agent will listen for syslog messages. | | listen\_ip | `string` | "0.0.0.0" | An IP address for the agent to bind. Typically 0.0.0.0 for most configurations. | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | # Cisco Meraki Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/cisco-meraki Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :----------- | :------- | :-------- | :------------------------------------------------------------------------------ | | listen\_port | `int` | 5140 | A UDP port, which the agent will listen for syslog messages. | | listen\_ip | `string` | "0.0.0.0" | An IP address for the agent to bind. Typically 0.0.0.0 for most configurations. | # Cloudflare Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/cloudflare Receive logs from Cloudflare LogPush jobs Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites * Cloudflare Enterprise plan * Publicly signed CA certificate * Follow the [OpenTelemetry receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/cloudflarereceiver#getting-started) getting started section for help configuring Cloudflare LogPush jobs. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------------- | :------- | :----------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | | listen\_address | `string` | 0.0.0.0 | The IP address to listen on. The collector must be available on the internet in order to receive logs from Cloudflare. | | https\_port | `int` | 8443 | TCP port to receive incoming uploads from the LogPush job(s). | | cert\_file\* | `string` | | A TLS certificate used to encrypt communications on the listening server. Cloudflare requires that this certificate be signed by a public certificate authority. | | key\_file\* | `string` | | A TLS private key used to encrypt communications on the listening server. Cloudflare requires TLS. | | secret | `string` | | String used to validate that messages are coming from an expected source. | | attributes | `map` | | Any Cloudflare LogPush field names provided as keys will be mapped to attributes using the map value as the attribute name. | | timestamp\_field | `string` | EdgeStartTimestamp | The name of the field that should be parsed to represent the Timestamp of the log record. | \* *required field* ## External Documentation * [Cloudflare LogPush documentation](https://developers.cloudflare.com/logs/about/) # CockroachDB Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/cockroachdb Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------------ | :------------------ | :-------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | hostname\* | `string` | "localhost" | The hostname or IP address of the CockroachDB system. Required: true. | | port | `int` | 8080 | The port to listen on for DB Console HTTP requests. [Read more](https://www.cockroachlabs.com/docs/stable/cockroach-start#networking) | | username | `string` | "" | The username to use when connecting to CockroachDB. [Read more](https://www.cockroachlabs.com/docs/stable/create-user#create-a-user) | | password | `string` | "" | The password to use when connecting to CockroachDB. TLS must be configured in the Advanced section if this field is set. Sensitive: true. [Read more](https://www.cockroachlabs.com/docs/stable/create-user#create-a-user-with-a-password) | | enable\_tls | `bool` | false | Whether or not to use TLS. | | ca\_file\_path | `string` | "" | File path for CA certification file for CockroachDB (only needed if you have a secure cluster). [Read more](https://www.cockroachlabs.com/docs/stable/cockroach-cert#client-key-and-certificates) | | cert\_file\_path | `string` | "" | A TLS certificate used for client authentication, if mutual TLS is enabled. [Read more](https://www.cockroachlabs.com/docs/stable/cockroach-cert#client-key-and-certificates) | | key\_file\_path | `string` | "" | A TLS private key used for client authentication, if mutual TLS is enabled. [Read more](https://www.cockroachlabs.com/docs/stable/cockroach-cert#client-key-and-certificates) | | server\_name | `string` | "" | The name of the server. [Read more](https://datatracker.ietf.org/doc/html/rfc4366#section-3.1) | | insecure\_skip\_verify | `bool` | false | Disable validation of the server certificate. | | collection\_interval | `int` | 60 | How often (seconds) to scrape for metrics. | | enable\_health\_log\* | `bool` | true | Enable to collect health logs. [Read more](https://www.cockroachlabs.com/docs/stable/logging#health) | | health\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach-health.log"] | The absolute path to the CockroachDB health logs. | | enable\_dev\_log\* | `bool` | true | Enable to collect general developer logs. [Read more](https://www.cockroachlabs.com/docs/stable/logging#dev) | | dev\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach.log"] | The absolute path to the CockroachDB Dev Logs. | | enable\_error\_log\* | `bool` | true | Enable to collect stderr logs. | | error\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach-stderr.log"] | The absolute path to the CockroachDB stderr logs. | | enable\_sql\_schema\_log\* | `bool` | true | Enable to collect sql schema logs. [Read more](https://www.cockroachlabs.com/docs/stable/logging#sql_schema) | | sql\_schema\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach-sql-schema.log"] | The absolute path to the CockroachDB sql schema logs. | | enable\_telemetry\_log\* | `bool` | true | Enable to collect telemetry logs. [Read more](https://www.cockroachlabs.com/docs/stable/logging#telemetry) | | telemetry\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach-telemetry.log"] | The absolute path to the CockroachDB telemetry logs. | | enable\_kv\_distribution\_log\* | `bool` | true | Enable to collect kv distribution logs. [Read more](https://www.cockroachlabs.com/docs/stable/logging#kv_distribution) | | kv\_distribution\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach-kv-distribution.log"] | The absolute path to the CockroachDB kv distribution logs. | | enable\_pebble\_log\* | `bool` | true | Enable to collect cockroachdb pebble logs. [Read more](https://www.cockroachlabs.com/docs/stable/logging#storage) | | pebble\_log\_path\* | `strings` | \["/var/log/cockroach-data/logs/cockroach-pebble.log"] | The absolute path to the CockroachDB pebble logs. | | offset\_storage\_dir\* | `string` | "\$OIQ\_OTEL\_COLLECTOR\_HOME/storage" | The directory the offset storage file will be created in. | | timezone\* | `timezone` | "UTC" | The timezone to use when parsing timestamps. | | start\_at\* | `enum` | end | Start reading logs from 'beginning' or 'end'. | | parse\_to\* | `enum` | body | Parse structured log parts to either body or attributes. | | retain\_raw\_logs\* | `bool` | false | Preserve the original log message in a raw\_log key. | ## Example Configuration This configuration shows the default values for the CockroachDB Source. Honeycomb Docs - CockroachDB -1

# Common Event Format Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/common-event-format Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :----------------------- | :--------- | :------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | file\_log\_path\* | `strings` | | Specify a single path or multiple paths to read one or many files. You may also use a wildcard (\*) to read multiple files within a directory. | | exclude\_file\_log\_path | `strings` | "" | Specify a single path or multiple paths to exclude one or many files from being read. You may also use a wildcard (\*) to exclude multiple files from being read within a directory. | | log\_type | `string` | "cef" | Adds the specified 'Type' as a log record attribute to each log message. | | location | `timezone` | "UTC" | The geographic location (timezone) to use when parsing logs that contain a timestamp | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # Couchbase Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/couchbase Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :-------------------------------- | :------------------ | :----------------------------------------------------------------- | :----------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | hostname\* | `string` | "localhost" | The hostname or IP address of the Couchbase API. | | port | `int` | 8091 | The TCP port of the Couchbase API. | | username\* | `string` | | Username used to authenticate. | | password\* | `string` | | Password used to authenticate. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_error\_log | `bool` | true | Enable to read error logs. | | error\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/error.log" | Log File paths to tail for error logs. | | enable\_info\_log | `bool` | false | Enable to read info logs. | | info\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/info.log" | Log File paths to tail for info logs. | | enable\_debug\_log | `bool` | false | Enable to read debug logs. | | debug\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/debug.log" | Log File paths to tail for debug logs. | | enable\_access\_log | `bool` | false | Enable to read http access logs. | | http\_access\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/http\_access.log" | Log File paths to tail for http access logs. | | enable\_internal\_access\_log | `bool` | false | Enable to read internal access logs. | | http\_internal\_access\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/http\_access\_internal.log" | Log File paths to tail for internal access logs. | | enable\_babysitter\_log | `bool` | false | Enable to read babysitter logs. | | babysitter\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/babysitter.log" | Log File paths to tail for babysitter logs. | | enable\_xdcr\_log | `bool` | false | Enable to read xdcr logs. | | xdcr\_log\_path | `strings` | "/opt/couchbase/var/lib/couchbase/logs/goxdcr.log" | Log File paths to tail for xdcr logs. | | start\_at | `enum` | end | Start reading logs from 'beginning' or 'end'. | # CouchDB Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/couchdb Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | hostname\* | `string` | | The hostname or IP address of the CouchDB system. | | port | `int` | 5984 | The TCP port of the CouchDB system. | | username\* | `string` | | The username to use when connecting to CouchDB. | | password\* | `string` | | The password to use when connecting to CouchDB. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS when connecting to CouchDB. | | strict\_tls\_verify | `bool` | false | Enable to require TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate TLS certificates. It's not required if the collector's operating system already trusts the certificate authority. | | mutual\_tls | `bool` | false | Enable to require TLS mutual authentication. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | | log\_paths | `strings` | "/var/log/couchdb/couchdb.log" | Path to CouchDB log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # Crowdstrike FDR Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/crowdstrike-fdr Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The Crowdstrike FDR source consumes S3 event notifications for object creation events (`s3:ObjectCreated:*`) and emits the S3 object as the string body of a log record. This source is similar to the generic [S3 Event source](/send-data/telemetry-pipeline/sources/aws-s3-event/). It expects SQS notifications sent from the Crowdstrike FDR platform. ## Supported Platforms | Platform | Supported | | ------------------ | --------- | | Linux | ✓ | | Windows | ✓ | | macOS | ✓ | | Kubernetes Gateway | ✓ | Available in the Bindplane Distro for OpenTelemetry Collector `v1.76.4+`. ## Prerequisites * An AWS account with access to S3 and SQS. * An SQS queue configured to receive S3 event notifications. * You have followed your [Crowdstrike FDR documentation](https://www.crowdstrike.com/en-us/resources/data-sheets/falcon-data-replicator/) for configuring replication to S3. 1. Configure an S3 bucket to send event notifications to an SQS queue for object creation events. * Configure your S3 event notifications with `BatchSize: 1` to ensure each SQS message contains only one S3 event. * This setting is crucial because if an object cannot be accessed (e.g., 404 error), the entire SQS message is preserved for retry. * If a message contains multiple objects and one fails, all objects will be reprocessed on retry, causing unnecessary duplication. 2. Ensure the collector has permission to read and delete messages from the SQS queue. 3. Ensure the collector has permission to read objects from the S3 bucket. ## How It Works 1. The receiver polls an SQS queue for S3 event notifications. 2. When an object creation event (`s3:ObjectCreated:*`) is received, the receiver downloads the S3 object. 3. The receiver reads the object into the body of a new log record. 4. Non-object creation events are ignored but removed from the queue. 5. If an S3 object is not found (404 error), the corresponding SQS message is preserved for retry later. ## Configuration Fields

Field	Type	Default	Required	Description
sqs\_queue\_url	string		`true`	The URL of the SQS queue to poll for S3 event notifications.
standard\_poll\_interval	duration	15	`false`	The interval (in seconds) at which the SQS queue is polled for messages.
max\_poll\_interval	duration	2	`false`	The maximum interval (in seconds) at which the SQS queue is polled for messages.
polling\_backoff\_factor	float	2	`false`	The factor by which the polling interval is multiplied after an unsuccessful poll.
workers	int	5	`false`	The number of workers to process events.
visibility\_timeout	duration	300	`false`	How long (in seconds) messages received from the queue will be invisible to other consumers.

# CSV Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/csv Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------ | :-------- | :------ | :------------------------------------------------------------------------------------------------------------------ | | header\* | `string` | | A comma-delimited list of keys assigned to each of the columns. | | file\_path\* | `strings` | | File or directory paths to tail for logs. | | exclude\_file\_path | `strings` | | File or directory paths to exclude. | | log\_type | `string` | csv | A friendly name that will be added to each log entry as an attribute. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | encoding | `enum` | utf-8 | The encoding of the file being read. Valid values include: `nop`, `utf-8`, `utf-16le`, `utf-16be`, `ascii`, `big5`. | # Custom Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/custom Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description This Custom source can be used to directly configure an OpenTelemetry Receiver. The Custom source is useful for testing new receivers or for fulfilling a use case that is not supported by Telemetry Pipeline natively. The Custom Source can only be used with components that are present in the Telemetry Pipeline Agent. See the [Included Components](https://github.com/observIQ/bindplane-agent/tree/main#included-components) documentation for a list of supported components. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :--------------------------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | ✓ | | Windows | ✓ | ✓ | ✓ | | macOS | ✓ | ✓ | ✓ | | Kubernetes Node (DaemonSet) | ✓ | ✓ | ✓ | | Kubernetes Deployment | ✓ | ✓ | ✓ | | Kubernetes Gateway | ✓ | ✓ | ✓ | | OpenShift 4 Node (DaemonSet) | ✓ | ✓ | ✓ | | OpenShift 4 Deployment | ✓ | ✓ | ✓ | The supported platforms and types will be limited to those allowed by the specific receiver used in the configuration. ## Configuration | Field | Description | | :-------------- | :---------------------------------------------------------------------------------------------------------------- | | Telemetry Types | The kind of telemetry that will be gathered by the receiver. Can be any combination of metrics, logs, and traces. | | Configuration | The YAML configuration for the receiver. | ## Example Configuration ### SQL Server Receiver The [SQL Server Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sqlserverreceiver) is already configurable via the [Microsoft SQL Server](https://observiq.com/docs/integrations/sources/microsoft-sql-server) source, but the custom source can be used to access configuration options that are not exposed in Telemetry Pipeline. #### Web Interface Honeycomb Docs - Custom Source - image 1

Honeycomb Docs - Custom Source - image 1

#### Standalone Source ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: custom name: custom spec: type: custom parameters: - name: telemetry_types value: - Logs - Metrics - Traces - name: configuration value: |- sqlserver: collection_interval: 10s computer_name: CustomServer instance_name: CustomInstance resource_attributes: sqlserver.computer.name: enabled: true sqlserver.instance.name: enabled: true ``` # Elasticsearch Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/elasticsearch Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This receiver supports Elasticsearch versions 7.9+. If Elasticsearch security features are enabled, you must have either the `monitor` or `manage` cluster privilege. See the [Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/authorization.html) for more information on authorization and [Security privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html). ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | hostname\* | `string` | "localhost" | The hostname or IP address of the Elasticsearch API. | | port | `int` | 9200 | The TCP port of the Elasticsearch API. | | username | `string` | | Username used to authenticate. | | password | `string` | | Password used to authenticate. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | nodes | `strings` | \_node | Filters that define which nodes are scraped for node-level metrics. It should be set to '\_node' if the collector is installed on all nodes. '\_all' if a single collector is scraping the entire cluster. `https://www.elastic.co/guide/en/elasticsearch/reference/7.9/cluster.html#cluster-nodes`. | | skip\_cluster\_metrics | `bool` | false | Enable to disable the collection of cluster-level metrics. | | json\_log\_paths | `strings` | `- \"/var/log/elasticsearch/__server.json\"` \n`- \"/var/log/elasticsearch/__deprecation.json\"` \n`- \"/var/log/elasticsearch/__index_search_slowlog.json\"` \n`- \"/var/log/elasticsearch/__index_indexing_slowlog.json\"` \n`- \"/var/log/elasticsearch/*_audit.json\"` | File paths for the JSON formatted logs. | | gc\_log\_paths | `strings` | `- \"/var/log/elasticsearch/gc.log*\"` | File paths for the garbage collection logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # F5 BIG-IP Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/f5-big-ip Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------- | :------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | address\* | `string` | | The hostname or IP address of the Big-IP environment. | | port | `int` | 443 | The TCP port of the Big-IP environment. | | username\* | `string` | | Username used for authenticating with Big-IP. | | password\* | `string` | | Password used for authenticating with Big-IP. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | strict\_tls\_verify | `bool` | false | Enable to require TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. It's not required if the collector's operating system already trusts the certificate authority. | | mutual\_tls | `bool` | false | Enable to require TLS mutual authentication. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | # File Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/filelog Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. This source offers a delete\_after\_read option that can be hazardous. When this option is combined with file globbing, it will delete every file that matches the globbing pattern. Use with caution and care. *** ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------------ | :--------- | :----------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | file\_path\* | `strings` | | File or directory paths to tail for logs. | | exclude\_file\_path | `strings` | "" | File or directory paths to exclude. | | log\_type | `string` | "file" | A friendly name that will be added to each log entry as an attribute. | | parse\_format | `enum` | none | Method to use when parsing. Valid values are `none`, `json`, and `regex`. When regex is selected, 'Regex Pattern' must be set. | | regex\_pattern | `string` | | The regex pattern that is used when parsing log entries. | | multiline\_line\_start\_pattern | `string` | | Regex pattern that matches the beginning of a log entry for handling multiline logs. | | multiline\_line\_end\_pattern | `string` | | Regex pattern that matches the end of a log entry, useful for terminating parsing of multiline logs. | | parse\_timestamp | `bool` | false | Whether to parse the timestamp from the log entry. | | timestamp\_field | `string` | timestamp | The field containing the timestamp in the log entry. | | parse\_timestamp\_format | `enum` | ISO8601 | The format of the timestamp in the log entry. Choose a common format, or specify a custom format. Options include "ISO8601", "RFC3339", "Epoch", and "Manual". | | epoch\_timestamp\_format | `enum` | s | The layout of the epoch-based timestamp. It's required when parse\_timestamp\_format is set to "Epoch". Options include "s", "ms", "us", "ns", "s.ms", "s.us", "s.ns". | | manual\_timestamp\_format | `string` | '%Y-%m-%dT%H:%M:%S.%f%z' | The strptime layout of the timestamp. It's used when parse\_timestamp\_format is set to "Manual". | | timezone | `timezone` | UTC | The timezone to use if the Timestamp Format doesn't include a timezone. Otherwise, the timezone in the Timestamp Format will be respected. **NOTE: This is also required to parse timezone abbreviations due to their ambiguity.** | | parse\_severity | `bool` | false | Whether to parse severity from the log entry. | | severity\_field | `string` | severity | The field containing the severity in the log entry. | | encoding | `enum` | utf-8 | The encoding of the file being read. Valid values are `nop`, `utf-8`, `utf-16le`, `utf-16be`, `ascii`, and `big5`. | | include\_file\_name\_attribute | `bool` | true | Whether to add the file name as the attribute `log.file.name`. | | include\_file\_path\_attribute | `bool` | false | Whether to add the file path as the attribute `log.file.path`. | | include\_file\_name\_resolved | `bool` | false | Whether to add the file name after symlinks resolution as the attribute `log.file.name_resolved`. | | include\_file\_path\_resolved | `bool` | false | Whether to add the file path after symlinks resolution as the attribute `log.file.path_resolved`. | | delete\_after\_read | `bool` | false | Whether to delete the file(s) after reading. Only valid in combination start\_at: beginning. | | offset\_storage\_dir | `string` | \$OIQ\_OTEL\_COLLECTOR\_HOME/storage | The directory where the offset storage file will be created. It is okay if multiple receivers use the same directory. By default, the [observIQ Distro for OpenTelemetry Collector](https://github.com/observIQ/observiq-otel-collector) sets `$OIQ_OTEL_COLLECTOR_HOME` in its runtime. | | poll\_interval | `int` | 200 | The duration of time in milliseconds between filesystem polls. | | max\_concurrent\_files | `int` | 1024 | The maximum number of log files from which logs will be read concurrently. If the number of files matched exceeds this number, then files will be processed in batches. | | parse\_to | `string` | body | The [field](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/field.md) that the log will be parsed to. Some exporters handle logs favorably when parsed to `attributes` over `body` and vice versa. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # Filestats Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/filestats Ent+ ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Configuration Table

Parameter	Type	Default	Description
include\*	`string`		The glob path for files to watch.
initial\_delay	`int`	1	The time (seconds) to wait before starting the collection.
collection\_interval	`int`	60	How often (seconds) to scrape for metrics.
metrics	`metrics`		Which metrics to collect for the included file(s).

### Metrics

Name	Unit	Type	Monotonic	Description
file.mtime	`seconds`	`Sum`	false	Elapsed time since the last modification of the file or folder, in seconds since Epoch.
file.size	`bytes`	`Gauge`		The size of the file or folder, in bytes.
file.atime	`seconds`	`Sum`	false	Elapsed time since last access of the file or folder, in seconds since Epoch.
file.count	`int`	`Gauge`		The number of files matched.
file.ctime	`seconds`	`Sum`	false	Elapsed time since the last change of the file or folder, in seconds since Epoch. In addition to `file.mtime`, this metric tracks metadata changes such as permissions or renaming the file.

\* *required field* # Fluent Forward Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/fluent-forward Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Fluent Forward source type can receive logs from Fluentd and Fluentbit agents or software capable of emitting logs using the [Fluent Forward Protocol](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#fluentd-forward-protocol-specification-v1).\ \ The Fluent Forward source type is useful for integrating Open Telemetry-based collectors into an existing environment where Fluentd or Fluentbit is the primary collection system for logs. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :-------------- | :------- | :-------- | :------------------------------------------------------------------------------- | | listen\_address | `string` | `0.0.0.0` | The IP address to listen on and receive logs from Fluent Forward capable agents. | | port | `int` | `24224` | TCP port to listen on and receive logs from Fluent Forward capable agents. | See the Open Telemetry [Fluent Forward Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver) documentation for more information. # Google Cloud PubSub Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/google-cloud-pubsub Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description Read messages from a Google Cloud PubSub topic and generate logs, metrics, or traces from the result. The OpenTelemetry receiver this source uses is in beta and is subject to change. ## Supported Platforms | Platform | Supported | | -------- | --------- | | Linux | ✓ | | Windows | ✓ | | macOS | ✓ | ## Configuration

Field	Description
Telemetry Types	The types of telemetry to read from PubSub.
Project ID	The Google Cloud project ID where the PubSub client will connect to.
Subscription Name	Name of the pre-existing PubSub subscription to read messages from.
Payload Encoding	Use "none" for attribute based detection. If the "content-type" attribute is missing from data, this is used to determine the encoding. Metrics and traces can only be OTLP Proto. See Attribute Labels for more information.
Payload Compression	Use "none" for attribute based detection. If the "content-encoding" attribute is missing from data, this is used to determine the compression. See Attribute Labels for more information.
Client ID	The client ID to be used by PubSub to make load balancing decisions.
User Agent	The user agent that will be used by the PubSub client to connect to the service.
Timeout	Timeout for all API calls, default is 12 seconds if unset.
Endpoint	Override of the default PubSub endpoint. Leave empty for default.
Insecure	If using a custom endpoint, determines if connection should be insecure.

### Authentication The service account used by the collector should have the `Pub/Sub Subscriber` permission to read from the subscription. Additionally, make sure the subscription channel being read from is configured with `Pull` delivery. Authentication is handled via the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. This should point to a file containing the service account key for the project. To set this for the collector, you need to modify the service file depending on your OS. ### Linux Use a [Systemd override](https://wiki.archlinux.org/title/systemd): ```bash theme={} sudo systemctl edit observiq-otel-collector ``` Then run the following command to reload the systemd configuration: ```bash theme={} sudo systemctl daemon-reload sudo systemctl restart observiq-otel-collector ``` ### Windows Start powershell as administrator and run the following command: ```powershell theme={} reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\observiq-otel-collector" /v Environment /t REG_MULTI_SZ /d "GOOGLE_APPLICATION_CREDENTIALS=" /f ``` Then restart the collector service: ```powershell theme={} Restart-Service observiq-otel-collector ``` ### macOS Add `GOOGLE_APPLICATION_CREDENTIALS` to the `EnvironmentVariables` dict in the launchd service file `/Library/LaunchDaemons/com.observiq.collector.plist` (other values are shown for context): ```xml theme={} EnvironmentVariables GOOGLE_APPLICATION_CREDENTIALS OIQ_OTEL_COLLECTOR_HOME /opt/observiq-otel-collector/ OIQ_OTEL_COLLECTOR_STORAGE /opt/observiq-otel-collector/storage ``` Then restart the collector: ```bash theme={} sudo launchctl unload /Library/LaunchDaemons/com.observiq.collector.plist sudo launchctl load /Library/LaunchDaemons/com.observiq.collector.plist ``` ### Attribute Labels The Google Cloud PubSub receiver looks for the following attributes on the message to determine the message encoding and compression: * `content-type` - How the message content should be interpreted. * If this value is set to `application/protobuf`, another attribute, `ce-type`, is expected to determine telemetry type. This attribute should be one of `org.opentelemetry.otlp.traces.v1`, `org.opentelemetry.otlp.metrics.v1`, or `org.opentelemetry.otlp.logs.v1`. * `content-encoding` - Despite being called the "content-encoding" this controls the compression of the message. If these attributes are present on the message data, the `Payload Encoding` and `Payload Compression` source parameters are ignored and should be set to "none". If these attributes are not present on the message data, the receiver will use the `Payload Encoding` and `Payload Compression` parameters to determine the encoding and compression. Metrics and traces can only be OTLP Proto. Logs can be OTLP JSON, OTLP Proto, or Raw Text. For more information, see the [OpenTelemetry Receiver Documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/googlecloudpubsubreceiver#encoding). ### Example Configuration In this example, we are parsing log data from the given PubSub subscription. The expected attributes are not present on the PubSub message data, so the `Payload Encoding` and `Payload Compression` parameters are used to determine the encoding and compression. If these attributes were present, then the `Payload Encoding` and `Payload Compression` parameters should be set to "none". Standalone Source: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: googlecloudpubsub name: googlecloudpubsub spec: type: googlecloudpubsub parameters: - name: telemetry_types value: ['Logs'] - name: project value: 'my_project_id' - name: subscription value: 'my_subscription_name' - name: encoding value: 'raw_text' - name: compression value: 'gzip' ``` # Google Cloud Storage Rehydration Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/google-cloud-storage-rehydration Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Types | Metrics | Logs | Traces | | ------- | ---- | ------ | | ✓ | ✓ | ✓ | ## How It Works 1. This source rehydrates data previously stored by Google Cloud Storage. 2. It will process both uncompressed JSON objects and objects compressed with gzip. 3. You can authenticate to Google Cloud using the provided `credentials`, `credentials_file`, or by using Application Default Credentials. 4. Your authentication credentials must have the Storage Admin permission to read and delete objects. ### Notes This is not a traditional source that continually produces data. Instead, it rehydrates all objects found within a specified time range. Once all objects in that time range have been rehydrated, the source will stop producing data. ## Configuration

Parameter	Type	Default	Description
telemetry\_types\*	`telemetrySelector`	Logs, Metrics, Traces	Specifies which types of telemetry to rehydrate.
bucket\_name\*	`string`	""	The name of the bucket to rehydrate from.
project\_id	`string`	""	The ID of the Google Cloud project the bucket belongs to. Will be read from credentials if not configured.
auth\_type	`enum`	auto	The method used for authenticating to Google Cloud. Valid values are "auto", "json", or "file".
credentials	`string`	""	JSON value from a Google Service Account credential file. Required if auth\_type is "json".
credentials\_file	`string`	""	Path to a Google Service Account credential file. Required if auth\_type is "file".
starting\_time\*	`dateTime`	""	The UTC start time for rehydration. Must be in the format "YYYY-MM-DDTHH:MM".
ending\_time\*	`dateTime`	""	The UTC end time for rehydration. Must be in the format "YYYY-MM-DDTHH:MM".
folder\_name	`string`	""	Restricts rehydration to objects in a specific folder within the bucket.
batch\_size	`int`	30	The number of objects to download at once. This impacts performance by controlling the number of concurrent object downloads.
delete\_on\_read	`bool`	false	If true, objects will be deleted after being rehydrated.
storage\_enable	`bool`	true	Enable to specify a storage extension for rehydration progress.
storage\_directory	`string`	\$OIQ\_OTEL\_COLLECTOR\_HOME/storage	Directory for storing rehydration state. Useful for maintaining state and resuming operations after disruptions.

\* *required field* ## Example Configurations ### Basic Configuration This configuration authenticates using Application Default Credentials and rehydrates data in the specified bucket, folder, and time range. Standalone Source: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: google_cloud_storage_rehydration name: google_cloud_storage_rehydration spec: type: google_cloud_storage_rehydration parameters: - name: telemetry_types value: ['Logs', 'Metrics', 'Traces'] - name: bucket_name value: 'my-bucket' - name: auth_type value: 'auto' - name: starting_time value: '2025-03-03T16:00' - name: ending_time value: '2025-03-03T17:00' - name: folder_name value: 'my-folder-name' - name: batch_size value: 30 - name: storage_enable value: false ``` ### Complete Configuration This configuration demonstrates all available options for the Google Cloud Storage Rehydration source, including authentication, storage settings, and delete on read functionality. Standalone Source: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: google_cloud_storage_rehydration name: google_cloud_storage_rehydration spec: type: google_cloud_storage_rehydration parameters: - name: telemetry_types value: ['Logs', 'Metrics', 'Traces'] - name: bucket_name value: 'my-bucket' - name: project_id value: 'my-project' - name: auth_type value: 'file' - name: credentials_file value: '/path/to/googlecloud/credentials/file' - name: starting_time value: '2025-03-03T16:00' - name: ending_time value: '2025-03-03T17:00' - name: folder_name value: 'my-folder' - name: batch_size value: 30 - name: delete_on_read value: true - name: storage_enable value: true - name: storage_directory value: '/custom/storage/path' ``` # Hadoop Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/hadoop Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------ | :------------------ | :-------------------------------------------------- | :---------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | address | `string` | localhost | IP address or hostname to scrape for JMX metrics. | | port | `int` | 8004 | Port to scrape for JMX metrics. | | jar\_path | `string` | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | | enable\_datanode\_logs | `bool` | true | Enable to collect datanode logs. | | datanode\_log\_path | `strings` | "/usr/local/hadoop/logs/hadoop-*-datanode-*.log" | File paths to tail for datanode logs. | | enable\_resourcemgr\_logs | `bool` | true | Enable to collect resource manager logs. | | resourcemgr\_log\_path | `strings` | "/usr/local/hadoop/logs/hadoop-*-resourcemgr-*.log" | File paths to tail for resource manager logs. | | enable\_namenode\_logs | `bool` | true | Enable to collect namenode logs. | | namenode\_log\_path | `strings` | "/usr/local/hadoop/logs/hadoop-*-namenode-*.log" | File paths to tail for namenode logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # HAProxy Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/haproxy Ent+s This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :--------- | :--------------------------- | :------------------------------------------------ | | file\_path | `strings` | /var/log/haproxy/haproxy.log | Log File paths to tail for logs. | | start\_at | `enum` | true | Start reading logs from the 'beginning' or 'end'. | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | # HBase Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/hbase Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :---------------------------------------------------- | :---------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | address | `string` | localhost | IP address or hostname to scrape for JMX metrics. | | jar\_path | `string` | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | | enable\_master\_jmx | `bool` | true | Enable to scrape the master server's JMX port. | | master\_jmx\_port | `int` | 10101 | Master server's JMX Port. | | enable\_region\_jmx | `bool` | true | Enable to scrape the region server's JMX port. | | region\_jmx\_port | `int` | 10102 | Region server's JMX Port. | | enable\_master\_log | `bool` | true | Enable to read master logs. | | master\_log\_path | `strings` | "/usr/local/hbase\*/logs/hbase\*-master-\*.log" | File paths to tail for master logs. | | enable\_region\_log | `bool` | true | Enable to read region server logs. | | region\_log\_path | `strings` | "/usr/local/hbase\*/logs/hbase\*-regionserver-\*.log" | File paths to tail for region server logs. | | enable\_zookeeper\_log | `bool` | false | Enable to read zookeeper logs. | | zookeeper\_log\_path | `strings` | "/usr/local/hbase\*/logs/hbase\*-zookeeper-\*.log" | File paths to tail for zookeeper logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # Honeycomb Telemetry Pipeline Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/honeycomb-telemetry-pipeline Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description Monitor the logs of a Honeycomb Telemetry Pipeline (HTP) server. Requires an HTP Agent be installed on the same system as the server. ## Supported Platforms HTP Agent: `v1.40.0`+ | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | macOS | | ✓ | | | Windows | | | | ### Configuration | Field | Description | | :------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------- | | Honeycomb Telemetry Pipeline Log Path | The absolute path to the HTP log. Default is `/var/log/bindplane/bindplane.log`. | | Enable File Offset Storage | When enabled, the current position into a file will be saved to disk, and reading will resume from where it left off after a collector restart. | | Offset Storage Directory | The directory that the offset storage file will be created. Default is `$OIQ_OTEL_COLLECTOR_HOME/storage`. | | Start At | Where in the log file to begin reading logs from. Can be `beginning` or `end`. | ## Example Configuration ### Basic Configuration For basic configuration, the defaults are used. #### Web Interface Honeycomb Docs - HTP Agent Source - image 1

Honeycomb Docs - HTP Agent Source - image 1

#### Standalone Source ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: bindplane-agent name: bindplane-agent spec: type: bindplane-agent parameters: - name: log_path value: '$OIQ_OTEL_COLLECTOR_HOME/log/collector.log' - name: enable_offset_storage value: true - name: offset_storage_dir value: '$OIQ_OTEL_COLLECTOR_HOME/storage' - name: start_at value: 'end' ``` # Honeycomb Telemetry Pipeline Gateway Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/honeycomb-telemetry-pipeline-gateway Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Honeycomb Telemetry Pipeline (HTP) Gateway source is an OTLP source meant to be used for gateway nodes. When using this source in conjunction with an HTP Gateway destination from another configuration, telemetry traveling through this source will not be double counted in the Summary view. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :--------------------------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | ✓ | | Windows | ✓ | ✓ | ✓ | | macOS | ✓ | ✓ | ✓ | | Kubernetes Node (DaemonSet) | ✓ | ✓ | ✓ | | Kubernetes Gateway | ✓ | ✓ | ✓ | | OpenShift 4 Node (DaemonSet) | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | Choose Telemetry Type. | | listen\_address | `string` | "0.0.0.0" | The IP address to listen on. | | grpc\_port | `int` | 4317 | TCP port to receive OTLP telemetry using the gRPC protocol. The port used must not be the same as the HTTP port. Set to 0 to disable. | | http\_port | `int` | 4318 | TCP port to receive OTLP telemetry using the HTTP protocol. The port used must not be the same as the gRPC port. Set to 0 to disable. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | ## Example Configuration The HTP Gateway source type does not have any required fields. By default, the HTP Gateway source will listen on ports 4317/gRPC and 4318/HTTP on all IP addresses without TLS. Honeycomb Docs - HTP Gateway - image 1

Honeycomb Docs - HTTP Check Source - image 1

Honeycomb Docs - HTTP Check Source - image 2

#### Standalone Source ```yaml YAML theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: http name: http spec: type: http parameters: - name: hostname value: 'localhost' - name: port value: '8080' - name: path value: 'metrics' - name: method value: 'GET' - name: enable_tls value: 'false' - name: initial_delay value: '1' - name: collection_interval value: '60' ``` # HTTP Log Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/http-log Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites The log source should be able to send its logs to an endpoint, commonly called "LogPush". The log source should also be able to reach the agent over the network, so any firewall rules must be adjusted to allow TCP and HTTP traffic to flow to the configured IP address and port. ## Request Format The request body should be JSON for requests made to the HTTP log receiver. ## Supported Platforms HTP Agent: `v1.39.0`+ | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ### Configuration | Field | Description | | :----------------- | :------------------------------------------------------------------------------------------------------------------------------------------------ | | Listen Address | Specifies what IP address the receiver should listen on for logs being sent as POST requests. | | HTTP Port | Specifies what port the receiver should use for listening for logs. | | Path | Specifies a path the receiver should be listening to for logs. Useful when the log source also sends other data to the endpoint, such as metrics. | | Enable TLS | Option to configure the receiver's HTTP server to use TLS. | | Cert File Location | Local path to the TLS cert file. | | Key File Location | Local path to the TLS key file. | ## Example Configuration ### Basic Configuration For basic configuration, only the `listen_address` and `http_port` parameters are needed. #### Web Interface Honeycomb Docs - HTTP Log Source - image 1

Honeycomb Docs - HTTP Log Source - image 1

#### Standalone Source ```yaml YAML theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: http name: http spec: type: http parameters: - name: listen_address value: '0.0.0.0' - name: http_port value: '12345' ``` # JBoss Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/jboss Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :--------- | :-------------------------------------- | :---------------------------------------------------- | | file\_path | `strings` | /usr/local/JBoss/EAP-*/*/log/server.log | File paths to tail for logs. | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # Journald Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/journald Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :-------- | :-------- | :------ | :--------------------------------------------------------------------------------------------------------- | | units | `strings` | | Service Units to filter on. If not set, all units will be read. | | directory | `string` | | The directory containing Journald's log files. If not set, /run/log/journal and /run/journal will be used. | | priority | `enum` | "info" | Set log level priority. Valid values are: `trace`, `info`, `warn`, `error`, and `fatal`. | | start\_at | `enum` | end | Start reading the journal from the 'beginning' or 'end'. | # JVM Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/jvm Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Prerequisites This source supports Java versions 11 and 16. ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------- | :------------------------------------------------ | :------------------------------------------------ | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | address | `string` | localhost | IP address or hostname to scrape for JMX metrics. | | port | `int` | 9999 | Port to scrape for JMX metrics. | | jar\_path | `string` | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | # Kafka Cluster Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/kafka-cluster Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Configuration Table | Parameter | Type | Default | Description | | :---------------------- | :-------- | :------------------------- | :---------------------------------------------------------------------------------------------------------------------------------- | | cluster\_name\* | `string` | | Friendly name used for the resource kafka.cluster.name. | | protocol\_version | `enum` | "2.0.0" | The Kafka protocol version to use when communicating with brokers. Valid values are: `"2.2.1"`, `"2.2.0"`, `"2.0.0"`, or `"1.0.0"`. | | brokers | `strings` | localhost:9092 | List of brokers to scrape for metrics. | | client\_id | `string` | otel-metrics-receiver | The consumer client ID that the receiver will use. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_auth | `bool` | false | | | auth\_type | `enum` | basic | `basic`, `sasl`, or `kerberos` | | basic\_username | `string` | | | | basic\_password | `string` | | | | sasl\_username | `string` | | | | sasl\_password | `string` | | | | sasl\_mechanism | `enum` | SCRAM-SHA-256 | `SCRAM-SHA-256`, `SCRAM-SHA-512`, or `PLAIN` | | kerberos\_service\_name | `string` | | | | kerberos\_realm | `string` | | | | kerberos\_config\_file | `string` | /etc/krb5.conf | | | kerberos\_auth\_type | `enum` | keytab | `keytab` or `basic` | | kerberos\_keytab\_file | `string` | /etc/security/kafka.keytab | | | kerberos\_username | `string` | | | | kerberos\_password | `string` | | | # Kafka Node Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/kafka-node Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------- | :------------------ | :------------------------------------------------ | :---------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | enable\_server\_log | `bool` | true | | | server\_log\_path | `strings` | /home/kafka/kafka/logs/server.log | File paths to tail for server logs. | | enable\_controller\_log | `bool` | true | | | controller\_log\_path | `strings` | /home/kafka/kafka/logs/controller.log | File paths to tail for controller logs. | | enable\_state\_change\_log | `bool` | true | | | state\_change\_log\_path | `strings` | /home/kafka/kafka/logs/state-change.log | File paths to tail for stage change logs. | | enable\_log\_cleaner\_log | `bool` | true | | | log\_cleaner\_log\_path | `strings` | /home/kafka/kafka/logs/state-cleaner.log | File paths to tail for log cleaner logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | address | `string` | localhost | IP address or hostname to scrape for JMX metrics. | | port | `int` | 9999 | Port to scrape for JMX metrics. | | jar\_path | `string` | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | # Kafka OTLP Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/kafka-otlp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | ✓ | | Windows | ✓ | ✓ | ✓ | | macOS | ✓ | ✓ | ✓ | ## Configuration Table No encoding field for metric events is available because the only option, `otlp_proto`, is set by default. | Parameter | Type | Default | Description | | :---------------------- | :------------------ | :------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | Choose Telemetry Type. | | protocol\_version | `enum` | "2.0.0" | The Kafka protocol version to use when communicating with brokers. Valid values are: `"2.2.1"`, `"2.2.0"`, `"2.0.0"`, or `"1.0.0"`. | | brokers | `strings` | localhost:9092 | List of brokers to connect and subscribe to for metrics, traces, and logs. | | group\_id | `string` | otel-collector | Consumer group to consume messages from. | | client\_id | `string` | otel-collector | The consumer client ID that the receiver will use. | | log\_topic | `string` | otlp\_logs | The topic name for subscribing to log events. | | log\_encoding | `enum` | otlp\_proto | The encoding of the log event pulled from the Kafka topic. `otlp_proto`, `raw`, `text`, or `json` | | metric\_topic | `string` | otlp\_metrics | The topic name for subscribing to metric events. | | trace\_topic | `string` | otlp\_spans | The topic name for subscribing to trace events. | | trace\_encoding | `enum` | otlp\_proto | The encoding of the trace event pulled from the Kafka topic. `otlp_proto`, `jaeger_proto`, `jaeger_json`, `zipkin_proto`, `zipkin_json`, or `zipkin_thrift` | | enable\_auth | `bool` | false | | | auth\_type | `enum` | basic | `basic`, `sasl`, or `kerberos` | | basic\_username | `string` | | | | basic\_password | `string` | | | | sasl\_username | `string` | | | | sasl\_password | `enum` | | | | sasl\_mechanism | `string` | SCRAM-SHA-256 | `SCRAM-SHA-256`, `SCRAM-SHA-512`, or `PLAIN` | | kerberos\_service\_name | `string` | | | | kerberos\_realm | `string` | | | | kerberos\_config\_file | `string` | /etc/krb5.conf | | | kerberos\_auth\_type | `enum` | keytab | `keytab` or `basic` | | kerberos\_keytab\_file | `string` | /etc/security/kafka.keytab | | | kerberos\_username | `string` | | | | kerberos\_password | `string` | | | # Kubernetes Cluster Events Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/kubernetes-cluster-events Retrieve cluster events as logs Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :--------------------- | :------ | :--- | :----- | | Kubernetes Deployment | | ✓ | | | OpenShift 4 Deployment | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :-------------- | :-------- | :------ | :-------------------------------------------------------------------------------- | | cluster\_name\* | `string` | | The cluster name that will be added as the `k8s.cluster.name` resource attribute. | | namespaces | `strings` | | Namespaces to collect events from. Default to all namespaces. | ## Example Configuration The Kubernetes Events source has two options: * Cluster Name: The name of the cluster, which will be inserted as the `k8s.cluster.name` resource attribute * Namespaces: List of namespaces to collector events from. This is optional and defaults to all namespaces. Honeycomb Docs - Kubernetes Cluster Events - image 1

Honeycomb Docs - Kubernetes Cluster Events - image 1

Once running on an agent, events will look like this: Honeycomb Docs - Kubernetes Cluster Events - image 2

Honeycomb Docs - Kubernetes Cluster Metrics - image 1

Once running on an agent, metrics will look like this: Honeycomb Docs - Kubernetes Cluster Metrics - image 2

# Kubernetes Container Logs Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/kubernetes-container-logs Read container logs from Kubernetes nodes Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :-------------------- | :------ | :--- | :----- | | Kubernetes DaemonSet | | ✓ | | | OpenShift 4 DaemonSet | | ✓ | | ## Configuration Table | Field | Default | Description | | :---------------- | :---------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cluster Name\* | | The cluster name that will be added as the `k8s.cluster.name` resource attribute. | | Log Source | File | Where to read logs from. Generally, this is `file`. `file` source supports Docker json-file and Containerd cri-o log formats. Options include `file` and `journald`. | | File Path(s) | `/var/log/containers/\*.log` | When log\_source is file. File or directory paths to tail for logs. Defaults to all container logs. | | Journald Path\* | | When log\_source is journald. The directory containing Journald's log files. | | Exclude File Path | `/var/log/containers/bindplane-*-agent-*` | File or directory paths to exclude. Generally, the collector's own log should be excluded. | | Start At | end | Start reading the logs from the 'beginning' or 'end'. | | Recombine Logs | | Options for configuring multi line logging. See [Multi Line Logging](#multi-line-logging). | ## Example Configuration The Kubernetes Container logs source has one required parameter: * Cluster Name: The name of the cluster, which will be inserted as the `k8s.cluster.name` resource attribute Honeycomb Docs - Kubernetes Container Logs - image 1

Honeycomb Docs - Kubernetes Container Logs - image 1

Once running on an agent, logs will look like this: Honeycomb Docs - Kubernetes Container Logs - image 2

## Multi Line Logging Multi line logging is useful for re-assembling multi line logs into a single log entry. Multi-line log re-assembly requires that all logs emitted by the application are consistent in structure. For example, the logs must start or end in a consistent way, in order to reliably match on the beginning or end of the log. If your application has inconsistent logging, multi-line log re-assembly will behave in irregular ways, such as combining two unique logs into one. Multi line logging is supported by configuring a selector, selector match expression, and a recombine match expression. | Field | Description | | -------------------------- | ----------------------------------------------------------------------------------- | | Selector | The OTTL path to match on. | | Selector Match Expression | A regular expression used to match the selector. | | Recombine Type | Whether or not to recombine logs by matching the first or last line in the log. | | Recombine With | The delimiter used to recombine logs. Defaults a single space or newline character. | | Recombine Match Expression | The regular expression used to recombine the multi-line log. | ### Example In this example, there are two Deployments. One is logging XML while the other is logging JSON. ```bash theme={} NAME READY STATUS RESTARTS AGE json-7d4fd9fd99-f29sl 1/1 Running 1 (38m ago) 3h49m xml-b44858b84-hdbqp 1/1 Running 1 (38m ago) 3h49m ``` The XML logs are a combination of multi-line and single line logs. Each log has a timestamp prefix indicating the start of the log. ```xml theme={} 2024-07-01 18:49:15 John Doe 30 2024-07-01 18:49:15 John Doe30 ``` The JSON logs are a combination of multi-line and single line logs. Each log has a trailing `}` without a comma, indicating the end of the log. ```json theme={} { "timestamp": "2024-07-01 18:51:56", "name": "John Doe", "user": { "name": "John Doe", "age": 25 }, "location": "New York", "type": "Checking", "severity": "warn" } {"timestamp":"2024-07-01 18:51:56","name":"John Doe","user":{"name":"John Doe","age":25},"location":"New York","type":"Checking","severity":"warn"} ``` Multi-line logging can be configured by matching on the First Entry of the XML logs and the Last Entry of the JSON logs. The `k8s.pod.name` resource attribute is used to select the XML and JSON pods. Honeycomb Docs - Kubernetes Container Multi Line Logs - image

Once configured, logs will be re-assembled into a single line. Honeycomb Docs - Kubernetes Container Multi Line Logs - image 2

Honeycomb Docs - Kubernetes Kubelet Metrics - image 1

Once running on an agent, metrics will look like this: Honeycomb Docs - Kubernetes Kubelet Metrics - image 2

# Kubernetes Prometheus Node Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/kubernetes-prometheus-metrics Retrieve metrics from Kubernetes pods using service discovery Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :-------------------- | :------ | :--- | :----- | | Kubernetes DaemonSet | ✓ | | | | OpenShift 4 DaemonSet | ✓ | | | ## Configuration Table | Field | Description | | :-------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------ | | Cluster Name\* | The cluster name that will be added as the `k8s.cluster.name` resource attribute. | | Relabel Configs | Enable or disable relabel configurations. See [Relabel Configs](#relabel-configs). | | Scrapers | Enable or disable HTTP and HTTPS scrapers. | | Collection Interval | Sets how often (seconds) to scrape for metrics. | | Skip TLS Certificate Verification | Enable to skip TLS certificate verification. | | TLS Certificate Authority File | Certificate authority used to validate the exporters TLS certificate. See [Transport Layer Security](#transport-layer-security). | | TLS Client Certificate File | A TLS certificate used for client authentication if mutual TLS is enabled. See [Transport Layer Security](#transport-layer-security). | | TLS Client Private Key File | A TLS private key used for client authentication if mutual TLS is enabled. See [Transport Layer Security](#transport-layer-security). | ### Relabel Configs Relabel configs are used to control how detected pods are scraped. There are four options. #### prometheus.io/scrape When enabled, only pods with the `prometheus.io/scrape: "true"` annotation will be considered for scraping. This option is enabled by default, to prevent the receiver from scraping all pods. #### prometheus.io/path The default HTTP path is `/metrics`. The path can be overridden by enabling this option and configuring the `prometheus.io/path` annotation. #### prometheus.io/scheme When this option is enabled, the HTTP scraper (configured with the "Scrapers" option) will only scrape pods that have the `prometheus.io/scheme: "http"` annotation set. Similarly, the HTTPS scraper will only scrape pods that have the `prometheus.io/scheme: "https"` annotation set. This option is recommended when using both HTTP and HTTPS scrapers. #### prometheus.io/job-name When this option is enabled, the `service.name` resource attribute will be set to the value of the pod annotation `prometheus.io/job-name`. This allows you to dynamically set `service.name`, which defaults to `kubernetes-pod-http` and `kubernetes-pod-https`, depending on which scraper is in use. #### Example Configuration When using Relabel configs, make sure to annotate your pods. Pod annotations are set at `spec.template.metadata.annotations`, not to be confused with `metadata.annotations`. ```yaml theme={} apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx replicas: 2 template: metadata: annotations: + prometheus.io/path: /metrics + prometheus.io/port: '9113' + prometheus.io/scrape: 'true' + prometheus.io/scheme: http + prometheus.io/job-name: nginx ``` Updating pod annotations will cause your pods to be re-deployed. ### Transport Layer Security When using TLS, if you need to configure a TLS certificate authority or a client key pair, update your Telemetry Pipeline Agent YAML manifest to include a `volumeMount` that will mount your TLS files into the container. You can find documentation for mounting secrets into a container [here](https://kubernetes.io/docs/concepts/configuration/secret/). ## Example Configuration By default, the Prometheus source is configured to use the HTTP scraper and the `prometheus.io/scrape` relabel config is enabled. This means the receiver will only scrape pods that have `prometheus.io/scrape: "true"` set in their annotations. A cluster name is required, and will be set as `k8s.cluster.name`. You can use a placeholder value if you intend to use [Resource Detection](/send-data/telemetry-pipeline/processors/detect-resource/) or [Add Fields](/send-data/telemetry-pipeline/processors/add-fields/) processors. Honeycomb Docs - Kubernetes Prometheus Node - image 1

Honeycomb Docs - Kubernetes Prometheus Node - image 1

Once running on an agent, some notable resource attributes are: * k8s.cluster.name * k8s.node.name * k8s.container.name * k8s.pod.name * service.name: The name Prometheus job name Honeycomb Docs - Kubernetes Prometheus Node - image 2

Honeycomb Docs - Kubernetes Prometheus Node - image 2

# Logstash Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/logstash Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------------ | :--------- | :----------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | listen\_port\* | `int` | 2255 | Port to listen on. | | listen\_ip | `string` | "0.0.0.0" | IP Address to listen on. | | log\_type | `string` | logstash | Arbitrary for attribute 'log\_type'. Useful for filtering between many logstash sources. | | parse\_format | `enum` | none | Method to use when parsing. Valid values are `none`, `json`, and `regex`. When regex is selected, 'Regex Pattern' must be set. | | regex\_pattern | `string` | | The regex pattern used when parsing log entries. | | multiline\_line\_start\_pattern | `string` | | Regex pattern that matches the beginning of a log entry, for handling multiline logs. | | multiline\_line\_end\_pattern | `string` | | Regex pattern that matches the end of a log entry, useful for terminating parsing of multiline logs. | | parse\_timestamp | `bool` | false | Whether to parse the timestamp from the log entry. | | timestamp\_field | `string` | timestamp | The field containing the timestamp in the log entry. | | parse\_timestamp\_format | `enum` | ISO8601 | The format of the timestamp in the log entry. Choose a common format, or specify a custom format. Options include "ISO8601", "RFC3339", "Epoch", and "Manual". | | epoch\_timestamp\_format | `enum` | s | The layout of the epoch-based timestamp. Required when parse\_timestamp\_format is set to "Epoch".. Options include "s", "ms", "us", "ns", "s.ms", "s.us", "s.ns". | | manual\_timestamp\_format | `string` | '%Y-%m-%dT%H:%M:%S.%f%z' | The strptime layout of the timestamp. Used when parse\_timestamp\_format is set to "Manual". | | timezone | `timezone` | UTC | The timezone to use if the Timestamp Format doesn't include a timezone. Otherwise, the timezone in the Timestamp Format will be respected. **NOTE: This is also required to parse timezone abbreviations, due to their ambiguity.** | | parse\_severity | `bool` | false | Whether to parse severity from the log entry. | | severity\_field | `string` | severity | The field containing the severity in the log entry. | | parse\_to | `string` | body | The [field](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/field.md) that the log will be parsed to. Some exporters handle logs favorably when parsed to `attributes` over `body` and vice versa. | | preserve\_original | `bool` | false | When this option is set to true, the original event will be preserved under the attributes | | enable\_tls | `bool` | false | Whether or not to use TLS. | | tls\_certificate\_path | `string` | | Path to the TLS cert to use for TLS-required connections. | | tls\_private\_key\_path | `string` | | Path to the TLS key to use for TLS-required connections. | | tls\_min\_version | `enum` | "1.2" | The minimum TLS version to support. 1.0 and 1.1 should not be considered secure. Valid values include: `1.3`, `1.2`, `1.1`, `1.0`. | # macOS Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/macos Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------- | :------------------ | :--------------------- | :---------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | enable\_system\_log | `bool` | true | Enable to collect macOS system logs. | | system\_log\_path | `string` | "/var/log/system.log" | The absolute path to the System log. | | enable\_install\_log | `bool` | true | Enable to collect macOS install logs. | | install\_log\_path | `string` | "/var/log/install.log" | The absolute path to the Install log. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | host\_collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | # macOS Unified Logging Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/macos-unified-logging Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The macOS Unified Logging Receiver collects logs from macOS systems using the native `log` command. This receiver supports both live system logs and archived log files (`.logarchive`). ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------------- | ------- | ---- | ------ | | macOS (10.12+) | | ✓ | | Available in the Bindplane Distro for OpenTelemetry Collector `v1.87.0+` . ## Prerequisites * macOS 10.12 (Sierra) or later. * The `log` command must be available in PATH. * For archive mode: Read access to the `.logarchive` directory. * For stream mode: Appropriate permissions to read system logs. ## Configuration Table | Parameter | Type | Default | Description | | -------------- | ----------- | ------- | ---------------------------------------------------------------------------------------------- | | mode | `enum` | stream | The collection mode to run the collector in. | | format | `enum` | default | The format of the logs to collect. | | archive\_path | `string` | "" | The path to the `.logarchive` directory to collect logs from. Only applicable in archive mode. | | predicate | `predicate` | "" | The expression to use to filter logs. | | start\_time | `datetime` | "" | The time to start collecting logs from. | | end\_time | `datetime` | "" | The time to stop collecting logs at. Only applicable in archive mode. | | poll\_interval | `int` | 30 | The interval to poll for new logs in seconds. Only applicable in stream mode. | | max\_log\_age | `int` | 24 | The maximum age of a log to collect in hours. Only applicable in stream mode. | ## Output Format The receiver converts macOS logs to OpenTelemetry log records: * **Body**: Contains the entire log line as a string * **Attributes**: Not set ### Format Options #### `ndjson` and `json` formats When using JSON formats, each log line is captured as a complete JSON string in the body, with timestamp and severity extracted: * **Timestamp**: Parsed from the `timestamp` field in the JSON * **Severity**: Mapped from `messageType` (Error, Fault, Default, Info, Debug) #### `default`, `syslog`, and `compact` formats When using plain text formats, each log line is captured as plain text in the body: * **Timestamp**: Set to observed time (when the log was received) * **Severity**: Not set ## Security Note Since the source passes values to the built-in `log` command on the agent, there are security risks in using this source if malicious code is passed in by a user. The source uses the following validation to mitigate these risks: * The following predicate values are not allowed: * Command separators: `;` * Pipes: `|` * Variable expansion: `$` * Backticks: `` ` `` * Redirects: `>>`, `<<` * Control characters: newlines, carriage returns * Valid predicate operators like `&&` (logical AND), `<`, `>` (comparison) are allowed. * The `>` operator is allowed for comparisons (e.g., processID > 100) but blocked when followed by file paths. * The `&&` operator is automatically normalized to `AND` to prevent command chaining. * The `||` operator is normalized to `OR` to prevent piping. * Archive file paths are sanitized and validated. # Microsoft 365 Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/microsoft-365 Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Note Instructions for [configuring](#configuring-microsoft-365) your Microsoft 365 tenant are included at the bottom of this document, but here are a few important notes to acknowledge. After configuring this source, **both metrics and logs will take some time to become available.** * Metrics are reported by the API every 24 hours, with each report containing the metrics that were created **two days prior**. **This means metrics generated on June 25 will appear to be from June 27**. The receiver scrapes every hour, but note that data points collected within the same 24-hour reporting interval will be duplicates. * Logs require an extra step the first time the tenant is configured for logs. Instructions are included at the bottom of this document. After this configuration step, it will take up to 60 minutes for logs to be enabled - **do not run the receiver until this finishes**. It may take up to 12 hours for logs to be made available by the API, after they have been enabled. After that point, logs will typically take 0-3 hours to be reported after being generated, but there is no guarantee given by the API. Finally, note that the client secret used to connect will expire (recommended 180 days) and needs to be re-generated, and your source will need to be reconfigured with the new client secret. The initial delay for logs will not be repeated upon reconfiguration. ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites * Created instance of Microsoft 365 with the following subscriptions: Microsoft 365 Business Basic, Microsoft 365 E5 Compliance, and Microsoft 365 E3 (Works with the respective "upgraded" versions as well.) * Access to an Admin account for the instance of 365 to be monitored. ## Metrics | Metric | Unit | Description | Attribute Values | | ----------------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | m365.onedrive.files.active.count | `{files}` | The number of active files across the OneDrive in the last seven days. | | | m365.onedrive.files.count | `{files}` | The number of total files across the OneDrive for the last seven days. | | | m365.onedrive.user\_activity.count | `{users}` | The number of users who have interacted with a OneDrive file, by action in the last seven days. | activity: `view_edit`, `synced`, `internal_share`, `external_share` | | m365.outlook.app.user.count | `{users}` | The number of unique users per app over the period of time in the organization Outlook in the last seven days. | app: `pop3`, `imap4`, `smtp`, `windows`, `mac`, `web`, `mobile`, `other_mobile` | | m365.outlook.email\_activity.count | `{emails}` | The number of email actions by members over the period of time in the organization Outlook. | activity: `read`, `sent`, `received` | | m365.outlook.mailboxes.active.count | `{mailboxes}` | The number of mailboxes that have been active each day in the organization for the last seven days. | | | m365.outlook.quota\_status.count | `{mailboxes}` | The number of mailboxes in the various quota statuses over the period of time in the org in the last seven days. | state: `under_limit`, `warning`, `send_prohibited`, `send_receive_prohibited`, `indeterminate` | | m365.outlook.storage.used | By | The amount of storage used in Outlook by the organization in the last seven days. | | | m365.sharepoint.files.active.count | `{files}` | The number of active files across all sites in the last seven days. | | | m365.sharepoint.files.count | `{files}` | The number of total files across all sites in the last seven days. | | | m365.sharepoint.pages.unique.count | `{views}` | The number of unique views of pages across all sites in the last seven days. | | | m365.sharepoint.pages.viewed.count | `{pages}` | The number of unique pages viewed across all sites in the last seven days. | | | m365.sharepoint.site.storage.used | By | The amount of storage used by all sites across SharePoint in the last seven days. | | | m365.sharepoint.sites.active.count | `{sites}` | The number of active sites across SharePoint in the last seven days. | | | m365.teams.calls.count | `{calls}` | The number of MS Teams calls from users in the organization in the last seven days. | | | m365.teams.device\_usage.users | `{users}` | The number of unique users by device/platform that used Teams in the last seven days. | device: `Android`, `iOS`, `Mac`, `Windows`, `Chrome OS`, `Linux`, `Web` | | m365.teams.meetings.count | `{meetings}` | The number of MS Teams meetings for users in the organization in the last seven days. | | | m365.teams.messages.private.count | `{messages}` | The number of MS Teams private-messages sent by users in the organization in the last seven days. | | | m365.teams.messages.team.count | `{messages}` | The number of MS Teams team-messages sent by users in the organization in the last seven days. | | ## Source Configuration Table | Parameter | Type | Default | Description | | -------------------- | ------------------- | --------------------- | ------------------------------------------------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | tenant\_id\* | `string` | | Identifies the instance of Microsoft 365 to be monitored. | | client\_id\* | `string` | | Identifier this receiver will use when monitoring. | | client\_secret\* | `string` | | The private key this receiver will use when monitoring must belong to the given Client ID. | | poll\_interval | `duration` | 5m | Sets how often (minutes) to collect logs. | | collection\_interval | `duration` | 1h | Sets how often (hours) to scrape for metrics. | ## Example Source Configuration Honeycomb Docs - Microsoft 365 - image 1

Honeycomb Docs - Microsoft 365 - image 1

Honeycomb Docs - Microsoft 365 - image 2

Honeycomb Docs - Microsoft 365 - image 3

**Once running on an agent, metrics will look like this:** Honeycomb Docs - Microsoft 365 - image 4

Honeycomb Docs - Microsoft 365 - image 5

**And logs will look like this:** Honeycomb Docs - Microsoft 365 - image 6

Honeycomb Docs - Microsoft 365 - image 7

Honeycomb Docs - Microsoft 365 - image 8

## Configuring Microsoft 365 The steps below outline how to configure Microsoft 365 to allow the receiver to collect metrics from the tenant to be monitored. 1. **Login to Azure:** Log in to Microsoft Azure under an Admin account for the instance of 365 to be monitored. 2. **Register the receiver in Azure AD:** Navigate to Azure Active Directory. Then go to "App Registrations" and select "New Registration".\ Give the app a descriptive name like "365 Receiver". For "Supported account types", select the Single Tenant option and leave the Redirect URL empty. Honeycomb Docs - Microsoft 365 - image 9

Honeycomb Docs - Microsoft 365 - image 9

Honeycomb Docs - Microsoft 365 - image 10

Honeycomb Docs - Microsoft 365 - image 11

3. **Add API Permissions:** Select "View API Permissions" beneath the general application info and click "Add Permissions". The permissions needed for metrics and logs differ, so whichever monitoring is necessary, the respective permissions are outlined below. * **Metrics:** Select "Microsoft Graph", then "Application Permissions". Find the "Reports" tab and select "Reports.Read.All". Click "Add Permissions" at the bottom of the panel. Honeycomb Docs - Microsoft 365 - image 12

Honeycomb Docs - Microsoft 365 - image 12

Honeycomb Docs - Microsoft 365 - image 13

Honeycomb Docs - Microsoft 365 - image 14

Honeycomb Docs - Microsoft 365 - image 15

* **Logs:** Select "Office 365 Management APIs", then "Application Permissions". Now select the "ActivityFeed.Read", "ActivityFeed.ReadDlp", and "ServiceHealth.Read" permissions. Click "Add Permissions" at the bottom of the panel. Honeycomb Docs - Microsoft 365 - image 16

Honeycomb Docs - Microsoft 365 - image 16

Honeycomb Docs - Microsoft 365 - image 17

Honeycomb Docs - Microsoft 365 - image 18

1. **Grant Admin Consent:** Select the "Grant admin consent for `{organization}`" button and confirm the pop-up. This will allow the application to access the data returned by the Microsoft Graph and Office 365 Management APIs. Honeycomb Docs - Microsoft 365 - image 19

Honeycomb Docs - Microsoft 365 - image 19

Honeycomb Docs - Microsoft 365 - image 20

2. **Generate Client Secret:** Select the "Certificates & secrets" tab in the left panel. Under the "Client Secrets" tab, select "New Client Secret." Give it a meaningful description and select the recommended period of 180 days. Save the text in the "Value" column since this is the only time the value will be accessible. The receiver needs be reconfigured with a newly generated Client Secret once the initial one expires. Honeycomb Docs - Microsoft 365 - image 21

Honeycomb Docs - Microsoft 365 - image 21

Honeycomb Docs - Microsoft 365 - image 22

Honeycomb Docs - Microsoft 365 - image 23

Honeycomb Docs - Microsoft 365 - image 24

3. **Save Client ID and Tenant ID values:** You will also need the "client\_id" value found on the information page for the application that was created. The value will be listed as "Application (client) id." You will also need the tenant value, which will be listed as "Directory (tenant) id." Save these values for later. Honeycomb Docs - Microsoft 365 - image 25

Honeycomb Docs - Microsoft 365 - image 25

The first time an instance of Microsoft 365 is set up for monitoring, an extra step for collecting logs is required. 4. Log into [Microsoft Purview Compliance Portal](https://compliance.microsoft.com) with an admin account. 5. Navigate to "Solutions" then ["Audit"](https://compliance.microsoft.com/auditlogsearch). 6. If auditing is not turned on for your organization, a banner is displayed prompting you to start recording user and admin activity. 7. Select "Start recording user and admin activity".\ **It will take up to 60 minutes for the change to take effect, so until that point, do not run the receiver with logs turned on, or else it will fail.** # Microsoft IIS Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/microsoft-iis Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Windows | ✓ | ✓ | | ## Prerequisites This source supports IIS versions 8.5 and 10.0. ## Configuration Table | Parameter | Type | Default | Description | | :----------------------- | :------------------ | :--------------------------------------------- | :--------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | file\_path | `strings` | `["C:/inetpub/logs/LogFiles/W3SVC_/**/_.log"]` | File or directory paths to tail for logs. | | exclude\_file\_log\_path | `strings` | | File or directory paths to exclude. | | timezone | `enum` | UTC | RFC3164 only. The timezone to use when parsing timestamps. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | ## Metrics | Metric | Unit | Description | | :--------------------------- | :-------------- | :--------------------------------------------------- | | iis.connection.active | `{connections}` | Number of active connections. | | iis.connection.anonymous | `{connections}` | Number of connections established anonymously. | | iis.connection.attempt.count | `{attempts}` | Total number of attempts to connect to the server. | | iis.network.blocked | By | Number of bytes blocked due to bandwidth throttling. | | iis.network.file.count | `{files}` | Number of transmitted files. | | iis.network.io | By | Total amount of bytes sent and received. | | iis.request.count | `{requests}` | Total number of requests of a given type. | | iis.request.queue.age.max | ms | Age of oldest request in the queue. | | iis.request.queue.count | `{requests}` | Current number of requests in the queue. | | iis.request.rejected | `{requests}` | Total number of requests rejected. | | iis.thread.active | `{threads}` | Current number of active threads. | | iis.uptime | s | The amount of time the server has been up. | # Microsoft SQL Server Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/microsoft-sql-server Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Windows | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :-------------------- | :---------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | # MongoDB Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/mongodb Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports MongoDB versions 2.6, 3.x, 4.x, and 5.0. ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :--------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | log\_paths | `strings` | `- \"/var/log/mongodb/mongodb.log_\"` \n`- \"/var/log/mongodb/mongod.log_\"` | Path to Mongodb log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | hosts | `strings` | "localhost:27017" | List of host:port or unix domain socket endpoints. \n \n- For standalone MongoDB deployments this is the hostname and port of the mongod instance.\n- For replica sets specify the hostnames and ports of the mongod instances that are in the replica set configuration. If the replica\_set field is specified, nodes will be auto discovered.\n- For a sharded MongoDB deployment, please specify a list of the mongos hosts. | | username | `string` | | If authentication is required, specify a username with "clusterMonitor" permission. | | password | `string` | | The password user's password. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | # MongoDB Atlas Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/mongodb-atlas Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------ | :------------------ | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | log\_project\_name\* | `string` | "" | Project to collect logs for. | | collect\_audit\_logs | `bool` | false | Enable to collect Audit Logs. It must be enabled on the project, and the API Key must have Organization Owner permissions. | | log\_filter\_mode\* | `enum` | All | Mode of filtering clusters. Either collect from all clusters or specify an inclusive list or exclusive list. Valid values: `All`, `Inclusive`, `Exclusive` | | log\_include\_clusters | `strings` | | Clusters in the project to collect logs from. Applicable if `log_filter_mode` is `Inclusive` | | log\_exclude\_clusters | `strings` | | Clusters in the project to exclude from log collection. Applicable if `log_filter_mode` is `Exclusive` | | public\_key\* | `string` | "" | API Public Key with at least Organization Read Only permissions. | | private\_key\* | `string` | "" | API Private Key. | | collection\_interval | `int` | 180 | Sets how often (seconds) to scrape for | | granularity | `enum` | PT1M | Duration interval between measurement data points. Read more [here](https://www.mongodb.com/docs/api/doc/atlas-admin-api-v2/operation/operation-gethostmeasurements#operation-gethostmeasurements-granularity). Valid values: `PT1M`, `PT5M`, `PT1H`, `P1D` | | enable\_alerts | `bool` | false | Enable to collect alerts. | | alert\_collection\_mode\* | `enum` | poll | Method of collecting alerts. In `poll` mode alerts are scraped from the API. In `listen` mode a server is set up to listen for incoming alerts. Valid values: `poll`, `listen`. | | alert\_project\_name\* | `string` | "" | Project to collect alerts from. Applicable if `alert_collection_mode` is `poll`. | | alert\_filter\_mode\* | `enum` | All | Mode of filtering clusters. Either collect from all clusters or specify an inclusive list or exclusive list. Applicable if `alert_collection_mode` is `poll`. Valid values: `All`, `Inclusive`, `Exclusive`. | | alert\_include\_clusters | `strings` | | Clusters in the project to collect alerts from. Applicable if `log_filter_mode` is `Inclusive` and `alert_collection_mode` is `poll`. | | alert\_exclude\_clusters | `strings` | | Clusters in the project to exclude from the alert collection. Applicable if `log_filter_mode` is `Exclusive` and `alert_collection_mode` is `poll`. | | page\_size | `int` | 100 | The number of alerts to collect per API request. Applicable if `alert_collection_mode` is `poll`. | | max\_pages | `int` | 10 | The limit of how many pages of alerts will be requested per project. Applicable if `alert_collection_mode` is `poll`. | | listen\_secret\* | `string` | "" | Secret key configured for push notifications. Applicable if `alert_collection_mode` is `listen`. | | listen\_endpoint\* | `string` | "0.0.0.0:4396" | Local "ip:port" to bind to, to listen for incoming webhooks. Applicable if `alert_collection_mode` is `listen`. | | enable\_listen\_tls | `bool` | false | Enable TLS for alert webhook server. Applicable if `alert_collection_mode` is `listen`. | | listen\_tls\_key\_file | `string` | "" | Local path to the TLS key file. Applicable if `enable_listen_tls` is true and `alert_collection_mode` is `listen`. | | listen\_tls\_cert\_file | `string` | "" | Local path to the TLS cert file. Applicable if `enable_listen_tls` is true and `alert_collection_mode` is `listen`. | # MySQL Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/mysql Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports MySQL versions 5.7 and 8.0. ## Configuration Table | Parameter | Type | Default | Description | | :---------------------- | :------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | enable\_general\_log | `bool` | false | Enable to read and parse the general log file. | | general\_log\_paths | `strings` | `- \"/var/log/mysql/general.log\"` | Path to the general log file(s). | | enable\_slow\_log | `bool` | true | Enable to read and parse the slow query log. | | slow\_query\_log\_paths | `strings` | `- \"/var/log/mysql/slow*.log\"` | Path to the slow query log file(s). | | enable\_error\_log | `bool` | true | Enable to read and parse the error log. | | error\_log\_paths | `strings` | For CentOS / RHEL: `- \"/var/log/mysqld.log\"` \nFor SLES: `- \"/var/log/mysql/mysqld.log\"` \nFor Debian / Ubuntu: `- \"/var/log/mysql/error.log\"` | Path to the error log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | username\* | `string` | | Username used to authenticate. | | password\* | `string` | | Password used to authenticate. | | endpoint | `string` | localhost:3306 | The endpoint of the MySQL server. | | transport | `enum` | tcp | The transport protocol used to connect to MySQL. | | database | `string` | | The database name. If not specified, metrics will be collected for all databases. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | # NetFlow Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/netflow Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Data Support * Process [Template Records](https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html) if present * Process Netflow V5, V9, and IPFIX messages * Mapping of custom fields is not yet supported ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Fields

Field	Description	Default
Scheme	The scheme to use for the NetFlow source. Can be `netflow` or `sflow`.	`netflow`
Hostname	The hostname or IP address to bind to.	`0.0.0.0`
Port	The port to use for the NetFlow source.	`2055`
Sockets	The number of sockets to use for the NetFlow source.	`1`
Workers	The number of workers to use for the NetFlow source.	`1`

## Example Configuration This configuration sets up a basic NetFlow source with necessary details such as host, port, and version. Standalone Source: ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: netflow name: netflow spec: type: netflow parameters: - name: scheme value: 'netflow' - name: hostname value: 'localhost' - name: port value: 2055 - name: sockets value: 1 - name: workers value: 1 ``` # Nginx Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/nginx Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports nginx versions 1.18 and 1.20. ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :------------------------------- | :---------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | data\_flow | `enum` | high | Enable high flow or reduced low flow. | | log\_format | `enum` | default | | | access\_log\_paths | `strings` | `- "/var/log/nginx/access.log*"` | Path to NGINX access log file(s). | | error\_log\_paths | `strings` | `- "/var/log/nginx/error.log*"` | Path to NGINX error log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | endpoint\* | `string` | "http\://localhost:80/status" | The endpoint of the NGINX server. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | # Okta Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/okta Collect System Logs from Okta Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms HTP Agent: `v1.59.0`+ | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------- | :------- | :------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | okta\_domain\* | `string` | | The Okta domain to collect logs from (Excluding "https\://").
[Find your Okta Domain](https://developer.okta.com/docs/guides/find-your-domain/main/) | | api\_token\* | `string` | | An Okta API Token generated from the above Okta domain.
[How to Create an Okta API Token](https://support.okta.com/help/s/article/How-to-create-an-API-token?language=en_US) | | poll\_interval | `string` | 1m | The rate at which this receiver will poll Okta for logs. This value must be in the range \[1 second - 24 hours] and must be a string readable by Golang's [time.ParseDuration](https://pkg.go.dev/time#ParseDuration).
Okta recommends between 60s - 300s. | Depending on your Okta plan, setting the poll\_interval below 10 seconds risks your API Token getting rate limited. You can increase the rate limit allocated to your API Token to minimize the chances of getting rate limited while using a short poll\_interval. [Okta - Set Token Rate Limits](https://help.okta.com/en-us/content/topics/security/api.htm#Set) ### Example Configuration #### Web Interface Honeycomb Docs - Okta Source Example Configuration

Honeycomb Docs - Okta Source Example Configuration

#### Standalone Source ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: okta name: okta spec: type: okta parameters: - name: okta_domain value: example.okta.com - name: api_token value: (sensitive) sensitive: true - name: poll_interval value: 60s ``` # OpenTelemetry (OTLP) Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/opentelemetry Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :--------------------------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | ✓ | | Windows | ✓ | ✓ | ✓ | | macOS | ✓ | ✓ | ✓ | | Kubernetes Node (DaemonSet) | ✓ | ✓ | ✓ | | Kubernetes Gateway | ✓ | ✓ | ✓ | | OpenShift 4 Node (DaemonSet) | ✓ | ✓ | ✓ | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics", "Traces"]` | Choose Telemetry Type. | | listen\_address | `string` | "0.0.0.0" | The IP address to listen on. | | grpc\_port | `int` | 4317 | TCP port to receive OTLP telemetry using the gRPC protocol. The port used must not be the same as the HTTP port. Set to 0 to disable. | | http\_port | `int` | 4318 | TCP port to receive OTLP telemetry using the HTTP protocol. The port used must not be the same as the gRPC port. Set to 0 to disable. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the OTLP server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication if mutual TLS is enabled. | ## Example Configuration The OTLP source type does not have any required fields. By default, the OTLP source will listen on ports 4317/gRPC and 4318/HTTP on all IP addresses without TLS. Honeycomb Docs - OpenTelemetry (OTLP) - image 1

Honeycomb Docs - OpenTelemetry (OTLP) - image 1

## Kubernetes The OTLP source type supports Kubernetes, OpenShift Node (DaemonSet), and Gateway agents. Applications within the cluster can forward metrics, logs, and traces to the agents using the clusterIP services. ### Prerequisites * Honeycomb Telemetry Pipeline v1.31.0 or newer ### Configuration The OTLP source type does not require additional configuration. It can be attached to any Kubernetes, OpenShift Node (DaemonSet), or Gateway configuration. The following endpoints can forward telemetry to the managed Node (DaemonSet) agents. | Protocol | Service | Endpoint | | :------- | :----------------- | :--------------------------------------------------------------------- | | gRPC | clusterIP | `bindplane-node-agent.bindplane-agent.svc.cluster.local:4317` | | gRPC | headless clusterIP | `bindplane-node-agent-headless.bindplane-agent.svc.cluster.local:4317` | | HTTP | clusterIP | `http://bindplane-node-agent.bindplane-agent.svc.cluster.local:4318` | The following endpoints can forward telemetry to the managed Gateway agents. | Protocol | Service | Endpoint | | :------- | :----------------- | :------------------------------------------------------------------------ | | gRPC | clusterIP | `bindplane-gateway-agent.bindplane-agent.svc.cluster.local:4317` | | gRPC | headless clusterIP | `bindplane-gateway-agent-headless.bindplane-agent.svc.cluster.local:4317` | | HTTP | clusterIP | `http://bindplane-gateway-agent.bindplane-agent.svc.cluster.local:4318` | It is a matter of preference if you should forward telemetry to the DaemonSet or Gateway agents. It is recommended to use the Gateway agent, if DaemonSet resource consumption is a concern, as the Gateway agent can scale independent of cluster size. # Oracle Database Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/oracle-database Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Metrics Requirements To collect metrics from OracleDB, a user with `SELECT` access to the relevant views is required. To create a new user with those permissions, run the following SQL script as a user with sufficient permissions connected to the Oracle DB instance as SYSDBA or SYSOPER. ```sql theme={} -- Create the monitoring user "bindplane" CREATE USER bindplane IDENTIFIED BY ; -- Grant the "bindplane" user the required permissions GRANT CONNECT TO bindplane; GRANT SELECT ON SYS.GV_$DATABASE to bindplane; GRANT SELECT ON SYS.GV_$INSTANCE to bindplane; GRANT SELECT ON SYS.GV_$PROCESS to bindplane; GRANT SELECT ON SYS.GV_$RESOURCE_LIMIT to bindplane; GRANT SELECT ON SYS.GV_$SYSMETRIC to bindplane; GRANT SELECT ON SYS.GV_$SYSSTAT to bindplane; GRANT SELECT ON SYS.GV_$SYSTEM_EVENT to bindplane; GRANT SELECT ON SYS.V_$RMAN_BACKUP_JOB_DETAILS to bindplane; GRANT SELECT ON SYS.V_$SORT_SEGMENT to bindplane; GRANT SELECT ON SYS.V_$TABLESPACE to bindplane; GRANT SELECT ON SYS.V_$TEMPFILE to bindplane; GRANT SELECT ON SYS.DBA_DATA_FILES to bindplane; GRANT SELECT ON SYS.DBA_FREE_SPACE to bindplane; GRANT SELECT ON SYS.DBA_TABLESPACE_USAGE_METRICS to bindplane; GRANT SELECT ON SYS.DBA_TABLESPACES to bindplane; GRANT SELECT ON SYS.GLOBAL_NAME to bindplane; ``` ## Configuration Table | Parameter | Type | Default | Description | | :-------------------- | :------------------ | :-------------------------------------------------------------------------- | :---------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | audit\_log\_path | `strings` | "/u01/app/oracle/product/*/dbhome\_1/admin/*/adump/\*.aud" | File paths to audit logs. | | enable\_alert\_log | `bool` | true | | | alert\_log\_path | `strings` | "/u01/app/oracle/product/*/dbhome\_1/diag/rdbms/*/\_/trace/alert\_\_.log" | File paths to alert logs. | | enable\_listener\_log | `bool` | true | | | listener\_log\_path | `strings` | "/u01/app/oracle/product/*/dbhome\_1/diag/tnslsnr/*/listener/alert/log.xml" | File paths to listener logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | host | `string` | localhost | Host to scrape metrics from. | | port | `int` | 1521 | Port of the host to scrape metrics from. | | username\* | `string` | | Database user to run metric queries with. | | password | `string` | | Password for user. | | sid | `string` | | Site Identifier. One or both of sid or service\_name must be specified. | | service\_name | `string` | | OracleDB Service Name. One or both of sid or service\_name must be specified. | | wallet | `string` | | OracleDB Wallet file location (must be URL encoded). | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_audit\_log | `bool` | true | Enable to collect audit logs. | # Packet Capture Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/packet-capture Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The Packet Capture (PCAP) source captures network packets and emits them as OpenTelemetry logs. It uses system-native tools (`tcpdump` on macOS/Linux, `Npcap` on Windows) to capture packets directly from a network interface. ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | macOS | | ✓ | | | Linux | | ✓ | | | Windows | | ✓ | | ## Prerequisites ### macOS/Linux **Tool**: `tcpdump` is pre-installed on macOS and most Linux distributions. To verify: ```bash theme={} tcpdump --version ``` ### Windows **Tool**: Requires Npcap driver (included with Wireshark, or install standalone from [Npcap](https://npcap.com/)). * Install [Npcap](https://npcap.com/) (or install Wireshark which includes Npcap) * List interfaces using PowerShell or the Npcap SDK tools * Interface names on Windows use Npcap device paths (e.g., `\Device\NPF_{GUID}`) ## Configuration Table | Parameter | Type | Default | Description | | ------------------ | -------- | ------- | -------------------------------------------------------------------------------- | | network\_interface | `string` | "" | Network interface to capture packets from. | | filter | `string` | "" | BPF (Berkeley Packet Filter) expression to filter packets. | | parse\_attributes | `bool` | true | The path to the `dumpcap` executable. Windows only (ignored on other platforms). | | snaplen | `int` | 65535 | Maximum bytes to capture per packet (64-65535). | | promiscuous | `bool` | true | Enable promiscuous mode to capture all network traffic. | ### Interface Names To list available interfaces on macOS/Linux: ```bash theme={} tcpdump -D ``` To list available interfaces on Windows: If you have Wireshark installed, use the `dumpcap` executable: ```powershell theme={} C:\\path-to-wireshark-installation\dumpcap.exe -D ``` Otherwise, use `Get-NetAdapter`: ```powershell theme={} Get-NetAdapter | Select-Object DeviceName ``` This result will have the interface names, but not in the Npcap format that the receiver expects. To convert it to the correct format, insert `\NPF_` ```txt theme={} \Device\{1D5B8F34-3D34-47E7-960B-E18EBC729A13} -> \Device\NPF_{1D5B8F34-3D34-47E7-960B-E18EBC729A13} ``` ### BPF Filters BPF filters allow you to capture only specific traffic. Examples: ```yaml theme={} # Capture only HTTPS traffic filter: "tcp port 443" # Capture DNS queries and responses filter: "udp port 53" # Capture HTTP and HTTPS filter: "tcp port 80 or tcp port 443" # Capture traffic to/from specific IP filter: "host 192.168.1.100" # Complex filter with multiple conditions filter: "(tcp port 80 or tcp port 443) and not src 192.168.1.1" ``` BPF filter syntax reference: [tcpdump manual](https://www.tcpdump.org/manpages/pcap-filter.7.html) # PgBouncer Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/pgbouncer Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :-------- | :------------------------------- | :---------------------------------------------------- | | file\_path | `strings` | /var/log/pgbouncer/pgbouncer.log | Path to log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # PostgreSQL Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/postgresql Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports PostgreSQL versions 10.18 and higher. The monitoring user must be granted `SELECT` on `pg_stat_database`. ## Configuration Table | Parameter | Type | Default | Description | | :-------------------------------- | :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | postgresql\_log\_path | `strings` | For CentOS / RHEL: `- \"/var/log/postgresql/postgresql_.log\"` \nFor SLES: `- \"/var/lib/pgsql/data/log/postgresql_.log\"` \nFor Debian / Ubuntu: `- \"/var/lib/pgsql/_/data/log/postgresql_.log\"` | Path to Postgres log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | username\* | `string` | | Username used to authenticate. | | password\* | `string` | | Password used to authenticate. | | endpoint | `string` | localhost:5432 | The endpoint of the Postgres server. If transport is set to unix, the endpoint will internally be translated from host:port to /host.s.PGSQL.port. | | transport | `enum` | tcp | The transport protocol used to connect to Postgres. Valid values are `tcp`, or `unix`. | | databases | `strings` | | The list of databases for which the receiver will attempt to collect statistics. If an empty list is provided, the receiver will attempt to collect statistics for all databases. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | enable\_tlsinsecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication, if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication, if mutual TLS is enabled. | # Prometheus Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/prometheus Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :-------- | :--------- | :----------------------------------------------------------------------- | | job\_name\* | `string` | | The name of the scraper job. Will be set as service.name resource label. | | static\_targets\* | `strings` | | List of endpoints to scrape. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | metrics\_path | `string` | "/metrics" | HTTP Resource path on which to fetch metrics from targets. | # RabbitMQ Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/rabbitmq Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites Supports RabbitMQ versions 3.8 and 3.9. The RabbitMQ Management Plugin must be enabled by following the [official instructions](https://www.rabbitmq.com/management.html#getting-started). Also, a user with at least [monitoring](https://www.rabbitmq.com/management.html#permissions) level permissions must be used for monitoring. ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :----------------------------------- | :---------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | daemon\_log\_paths | `strings` | `- "/var/log/rabbitmq/rabbit@*.log"` | Path to Rabbitmq log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | username\* | `string` | | Username used to authenticate. | | password\* | `string` | | Password used to authenticate. | | endpoint | `string` | `http://localhost:15672` | The endpoint of the Rabbitmq server. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication, if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication, if mutual TLS is enabled. | # Redis Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/redis Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports Redis version 6.2. ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | file\_path | `strings` | One-click installer: `- \"/var/log/redis/redis_6379.log\"` \nUbuntu / Debian: `- \"/var/log/redis/redis-server.log\"` \nsrc: `- \"/var/log/redis_6379.log\"` \nCentOS / RHEL: `- \"/var/log/redis/redis.log\"` \nSLES: `- \"/var/log/redis/default.log\"` | Path to Redis log file(s). | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | endpoint | `string` | "localhost:6379" | The endpoint of the Redis server. | | transport | `enum` | tcp | The transport protocol being used to connect to Redis. Valid values are `tcp` or `unix`. | | password | `string` | | The password used to access the Redis instance; must match the password specified in the requirepass server configuration option. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication, if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication, if mutual TLS is enabled. | # SAP HANA Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/sap-hana Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :--------- | :------------------------------ | :---------------------------------------------------- | | file\_path | `strings` | "/usr/sap/*/HDB*/*/trace/*.trc" | File paths to logs. | | timezone | `timezone` | "UTC" | The hostname or IP address of the Elasticsearch API. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # SAP NetWeaver Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/sap-netweaver Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | | | | | macOS | | | | ## Prerequisites Supports SAP Netweaver 7.10+. SAP Netweaver requires a valid OS user and password via HTTP basic auth and unix write permission. The receiver must run on the host to execute OS executables in order to collect certificate, rfc, and session metrics. ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------- | :-------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hostname\* | `string` | localhost | The hostname or IP address of the SAP Netweaver system. | | port | `int` | 50013 | The TCP port of the SAP Netweaver system (for HTTP, use port 50013. for HTTPS use port 50014). | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | username\* | `string` | | The username to use when connecting to SAP Netweaver. | | password\* | `string` | | The password to use when connecting to SAP Netweaver. | | profile | `string` | | The profile path in the form of /sapmnt/SID/profile/SID\_INSTANCE\_HOSTNAME to collect sapnetweaver.abap.rfc.count and sapnetweaver.abap.session.count metrics | | enable\_tls | `bool` | false | Whether or not to use TLS. | | insecure\_skip\_verify | `bool` | false | Enable to skip TLS certificate verification. | | ca\_file | `string` | | Certificate authority used to validate the database server's TLS certificate. | | cert\_file | `string` | | A TLS certificate used for client authentication, if mutual TLS is enabled. | | key\_file | `string` | | A TLS private key used for client authentication, if mutual TLS is enabled. | # Solr Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/solr Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :------------------------------------------------ | :------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | address | `string` | localhost | IP address or hostname to scrape for JMX metrics. | | port | `int` | 9012 | Port to scrape for JMX metrics. | | jar\_path | `string` | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | # Splunk (HEC) Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/splunk-hec Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description The Splunk HTTP Event Collector source can be used to receive events (logs) from applications that emit events in the Splunk HEC format. Events are converted to OTLP format and can be sent to any destination. The HEC source can be combined with the [Splunk HEC Destination](https://observiq.com/docs/integrations/destinations/splunk-hec). This allows Telemetry Pipeline's agent to sit in the middle of a Splunk pipeline, giving you the ability to leverage Telemetry Pipeline's processing capabilities. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :----------------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | | Kubernetes Gateway | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------- | :------- | :-------- | :----------------------------------------------------------------------------------------------------------------------- | | listen\_port | `int` | 8888 | Port to listen on. | | listen\_ip | `string` | "0.0.0.0" | IP Address to listen on. | | access\_token\_passthrough | `string` | false | Whether to preserve incoming access token (Splunk header value) as "com.splunk.hec.access\_token" metric resource label. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | tls\_certificate\_path | `string` | | Path to the TLS cert to use for TLS-required connections. | | tls\_private\_key\_path | `string` | | Path to the TLS key to use for TLS-required connections. | ## Example Configuration The HEC source type has two required parameters: * Listen IP Address * Listening Port It is recommended to enable the Access Token Passthrough option if you wish to preserve the Splunk access token header as a resource attribute `com.splunk_hec.access_token`. Honeycomb Docs - Splunk (HEC) - image 1

Once configured, incoming events will be displayed as logs like this: Honeycomb Docs - Splunk (HEC) - image 2

## Kubernetes The Splunk HEC source type supports Kubernetes Gateway agents. Splunk HEC forwarders can send logs to the agents using the clusterIP services. ### Prerequisites * Honeycomb Telemetry Pipeline v1.49.0 or newer ### Configuration Add the Splunk HEC source to your Gateway agent configuration. Set "Listen Address" to `0.0.0.0` and Listen Port to `8088`. The Splunk forwarders should be configured to forward telemetry to `bindplane-gateway-agent.bindplane-agent.svc.cluster.local` on port `8088`. If the Splunk forwarders live outside of the cluster, you must make the `bindplane-gateway-agent` service in the `bindplane-agent` namespace available using TCP ingress or by defining your own service that can receive traffic from outside of the cluster. See the Kubernetes [service documentation](https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/) for more information. # Splunk Search API Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/splunk-search-api Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. The Splunk Search API source uses the [Splunk Search API](https://docs.splunk.com/Documentation/Splunk/9.3.1/RESTREF/RESTsearch) to collect past events. ## Supported Platforms | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Prerequisites * Splunk admin credentials ## Use Case Unlike other sources, the SSAPI source is not built to stream live data. Instead, it collects a finite set of event data and transfers it to a destination, preserving the timestamp from the original Splunk event. **Note**: Once the source has started collecting events for a search, allow it to complete unless it is absolutely necessary to stop progress. If a search must be interrupted, run that search again to pick up where progress was interrupted. If you switch to a different search instead, the checkpoint for the search will be wiped from storage. ## Configuration Table

Parameter	Type	Default	Description
hostname	`string`		Splunk search head hostname.
port	`string`	8089	Splunk instance endpoint port.
auth\_mode	`enum`	"basic"	Authentication mode to use when connecting to the Splunk REST API. Valid values are "basic" and "token".
username	`string`		Username used to authenticate to the Splunk REST API.
password	`string`		Password used to authenticate to the Splunk REST API.
auth\_token	`string`		Auth token used to authenticate to the Splunk REST API.
token\_type	`enum`		Type of token used to authenticate to the Splunk REST API. Valid values are "Bearer" and "Splunk".
job\_poll\_interval	`int`	5	How many seconds to wait between polling for search job completion.
searches.query	`string`		Splunk search to run to retrieve the desired events. Queries must start with search and should not contain additional commands, nor any time fields (e.g. earliesttime).
searches.earliest\_time	`string`		Earliest timestamp to collect logs (inclusive).
searches.latest\_time	`string`		Latest timestamp to collect logs (inclusive).
searches.event\_batch\_size	`int`	100	Amount of events to query from Splunk for a single request.
enable\_tls	`bool`	true	Whether or not to use TLS.
tls\_certificate\_path	`string`		Path to the TLS cert to use for TLS-required connections.
tls\_private\_key\_path	`string`		Path to the TLS key to use for TLS-required connections.
enable\_storage	`bool`	true	Whether or not to use a storage extension. Should be enabled in all environments.
storage\_directory	`string`	`\$OIQ\_OTEL\_COLLECTOR\_HOME/storage`	The directory where the storage file will be created.

### Configuration Instructions 1. Identify the Splunk index to migrate events from. Create a Splunk search to capture the events from that index. This will be the query you pass to the source. * Example: `search index=my_index` * Note: queries must begin with the explicit `search` command, and must not include additional commands, nor any time fields (e.g. earliesttime) 2. Determine the timeframe you want to migrate events from, and set the ‘Earliest Time’ and ‘Latest Time’ config fields accordingly. * To migrate events from December 2024, EST (UTC-5): * Earliest Time: "2024-12-01T05:00" * Latest Time: "2025-01-01T04:59” * Note: By default, GCL will not accept logs with a timestamp older than 30 days. Contact Google to modify this rule. 3. Repeat steps 1 & 2 for each index you wish to collect from * This is not a requirement. The receiver can migrate multiple searches at once, but doing one at a time will allow for easier testing and debugging of the source. 4. Configure the rest of the source fields according to your Splunk environment. # Splunk (TCP) Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/splunk-tcp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :----------------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | | Kubernetes Gateway | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------------ | :--------- | :----------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | listen\_ip | `string` | "0.0.0.0" | IP Address to listen on. | | listen\_port\* | `int` | | Port to listen on. | | log\_type | `string` | splunk\_tcp | Arbitrary for attribute 'log\_type'. Useful for filtering between many log sources. | | parse\_format | `enum` | none | Method to use when parsing. Valid values are `none`, `json`, and `regex`. When regex is selected, 'Regex Pattern' must be set. | | regex\_pattern | `string` | | The regex pattern used when parsing log entries. | | multiline\_line\_start\_pattern | `string` | | Regex pattern that matches the beginning of a log entry, for handling multiline logs. | | multiline\_line\_end\_pattern | `string` | | Regex pattern that matches the end of a log entry, useful for terminating parsing of multiline logs. | | parse\_timestamp | `bool` | false | Whether to parse the timestamp from the log entry. | | timestamp\_field | `string` | timestamp | The field containing the timestamp in the log entry. | | parse\_timestamp\_format | `enum` | ISO8601 | The format of the timestamp in the log entry. Choose a common format, or specify a custom format. Options include "ISO8601", "RFC3339", "Epoch", and "Manual". | | epoch\_timestamp\_format | `enum` | s | The layout of the epoch-based timestamp. Required when parse\_timestamp\_format is set to "Epoch".. Options include "s", "ms", "us", "ns", "s.ms", "s.us", "s.ns". | | manual\_timestamp\_format | `string` | '%Y-%m-%dT%H:%M:%S.%f%z' | The strptime layout of the timestamp. Used when parse\_timestamp\_format is set to "Manual". | | timezone | `timezone` | UTC | The timezone to use if the Timestamp Format doesn't include a timezone. Otherwise, the timezone in the Timestamp Format will be respected. **NOTE: This is also required to parse timezone abbreviations, due to their ambiguity.** | | parse\_severity | `bool` | false | Whether to parse severity from the log entry. | | severity\_field | `string` | severity | The field containing the severity in the log entry. | | parse\_to | `string` | body | The [field](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/field.md) that the log will be parsed to. Some exporters handle logs favorably when parsed to `attributes` over `body` and vice versa. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | tls\_certificate\_path | `string` | | Path to the TLS cert to use for TLS-required connections. | | tls\_private\_key\_path | `string` | | Path to the TLS key to use for TLS-required connections. | | tls\_min\_version | `enum` | "1.2" | The minimum TLS version to support. 1.0 and 1.1 should not be considered secure. Valid values include: `1.3`, `1.2`, `1.1`, `1.0`. | ## Kubernetes The Splunk TCP source type supports Kubernetes Gateway agents. Splunk forwarders can send logs to the agents using the clusterIP services. ### Prerequisites * Honeycomb Telemetry Pipeline v1.46.0 or newer ### Configuration Add the Splunk TCP source to your Gateway agent configuration. Set "Listen Address" to `0.0.0.0` and Listen Port to `9997`. The Splunk forwarders should be configured to forward telemetry to `bindplane-gateway-agent.bindplane-agent.svc.cluster.local` on port `9997`. If the Splunk forwarders live outside of the cluster, you must make the `bindplane-gateway-agent` service in the `bindplane-agent` namespace available using TCP ingress or by defining your own service that can receive traffic from outside of the cluster. See the Kubernetes [service documentation](https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/) for more information. Below is an example Splunk forwarder outputs configuration. ```ini theme={} [tcpout] defaultGroup = bindplane-gateway-agent [tcpout:bindplane-gateway-agent] server = bindplane-gateway-agent.bindplane-agent.svc.cluster.local:9997 compressed = false useACK = false sendCookedData = false ``` # SQL Query Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/sql-query Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Description Write an SQL query to execute on a compatible database server and generate logs from the result. ## Supported Platforms HTP Agent: `v1.40.0`+ | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ### Configuration | Field | Description | | :-------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Driver | Which database driver should be used. Typically indicates which kind of database is being queried. Options include "postgres", "mysql", "snowflake", "sqlserver", "sap-hana", and "oracle". | | Database Connection Options | A driver specific string specifying how to connect to the database. Usually contains information like host, port, authorization credentials, TLS configuration, and other connection options. | | Query | The SQL query to run. The results of the query are used to generate the telemetry specified below. | | Log Body Column | Defines the name of the column whose value will become the body for the generated log. | | Tacking Column | Used for parameterized queries. Defines the name of the column to retrieve for the parameter value on subsequent query runs. See this [OTel Documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sqlqueryreceiver#tracking-processed-results) for more information. | | Tracking Start Value | Used for parameterized queries. Defines the initial value of the tracking column to compare against on subsequent query runs. See this [OTel Documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sqlqueryreceiver#tracking-processed-results) for more information. | | Collection Interval | How frequently to execute queries to retrieve log data. Default is '10s'. | | Enable Tracking Storage | If using tracking values, enable this to persist those values when the collector is restarted. Directory will be "\$OIQ\_OTEL\_COLLECTOR\_HOME/storage". See this [OTel Documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sqlqueryreceiver#tracking-processed-results) for more information. | | Enable Query Logging | Whether or not the collector should log the SQL query with associated parameters when the query is ran. | ### Example Configuration In this example, we are connecting to a postgres database using a postgres driver specific connection string. We are using a simple query which is retrieving rows of logs from a table. We are tracking the `id` column to avoid creating duplicate logs. #### Web Interface Honeycomb Docs - SQL Query Source - image 1

Honeycomb Docs - SQL Query Source - image 1

#### Standalone Source ```yaml theme={} apiVersion: bindplane.observiq.com/v1 kind: Source metadata: id: sqlquery name: sqlquery spec: type: sqlquery parameters: - name: driver value: 'postgres' - name: datasource value: 'postgresql://postgres:password@localhost:5432/production?sslmode=disable' - name: query value: 'select data, id from log_data where id > $1' - name: body_column value: 'data' - name: tracking_column value: 'id' - name: tracking_start_value value: '0' - name: interval value: '10s' - name: enable_storage value: true - name: enable_log_query_logging value: true ``` # StatsD Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/statsd Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | | | | Windows | ✓ | | | | macOS | ✓ | | | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------- | :------- | :-------- | :------------------------------------------------------------------------ | | listen\_ip | `string` | "0.0.0.0" | IP Address to listen on. | | listen\_port | `int` | 8125 | Port to listen on and receive metrics from statsd clients. | | aggregation\_interval | `int` | 60 | The aggregation time in seconds that the receiver aggregates the metrics. | | enable\_metric\_type | `bool` | false | Enable the statsd receiver to be able to emit the metric type as a label. | | is\_monotonic\_counter | `bool` | false | Set all counter-type metrics the statsd receiver received as monotonic. | # Syslog Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/syslog Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------------------- | :------- | :-------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | protocol\* | `enum` | "rfc3164" | The RFC protocol to use when parsing incoming syslog. Valid values are `rfc3164` or `rfc5424`. | | connection\_type | `enum` | udp | The transport protocol to use. Valid values are `udp` or `tcp`. | | data\_flow | `enum` | high | Enable high flow or reduced low flow. | | listen\_port\* | `int` | 5140 | The port to bind to and receive syslog. Collector must be running as root (Linux) or Administrator (windows) when binding to a port below 1024. | | listen\_ip\* | `string` | "0.0.0.0" | The IP address to bind to and receive syslog. | | timezone | `enum` | UTC | RFC3164 only. The timezone to use when parsing timestamps. | | enable\_octet\_counting | `bool` | false | Whether or not to parse using a trailer character. This is a special character that will be the termination character for syslog messages. This is only applicable to `tcp` and `rfc5424` configurations. | | non\_transparent\_framing\_trailer | `enum` | LF | Whether or not to enable octet counting on syslog framing. This framing allows for the transmission of all characters inside a syslog message. This is only applicable to `tcp` and `rfc5424` configurations. | | enable\_mutual\_tls | `bool` | false | Whether or not to use mutual TLS. | | cert\_file | `string` | | Path to the TLS cert to use for TLS-required connections. | | key\_file | `string` | | Path to the TLS key to use for TLS-required connections. | | ca\_file | `string` | | When set, enforces mutual TLS authentication and verifies client certificates. | | tls\_min\_version | `enum` | "1.2" | The minimum TLS version to support. 1.0 and 1.1 should not be considered secure. | \* *required field* # TCP Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/tcp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------------ | :--------- | :----------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | listen\_port\* | `int` | | Port to listen on. | | listen\_ip | `string` | "0.0.0.0" | IP Address to listen on. | | log\_type | `string` | tcp | Arbitrary for attribute 'log\_type'. Useful for filtering between many tcp sources. | | encoding | `enum` | utf-8 | The encoding of the data being read. See the [list of supported encodings](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/tcplogreceiver#supported-encodings). | | parse\_format | `enum` | none | Method to use when parsing. Valid values are `none`, `json`, and `regex`. When regex is selected, 'Regex Pattern' must be set. | | regex\_pattern | `string` | | The regex pattern used when parsing log entries. | | multiline\_line\_start\_pattern | `string` | | Regex pattern that matches the beginning of a log entry, for handling multiline logs. | | multiline\_line\_end\_pattern | `string` | | Regex pattern that matches the end of a log entry, useful for terminating parsing of multiline logs. | | parse\_timestamp | `bool` | false | Whether to parse the timestamp from the log entry. | | timestamp\_field | `string` | timestamp | The field containing the timestamp in the log entry. | | parse\_timestamp\_format | `enum` | ISO8601 | The format of the timestamp in the log entry. Choose a common format, or specify a custom format. Options include "ISO8601", "RFC3339", "Epoch", and "Manual". | | epoch\_timestamp\_format | `enum` | s | The layout of the epoch-based timestamp. Required when parse\_timestamp\_format is set to "Epoch".. Options include "s", "ms", "us", "ns", "s.ms", "s.us", "s.ns". | | manual\_timestamp\_format | `string` | '%Y-%m-%dT%H:%M:%S.%f%z' | The strptime layout of the timestamp. Used when parse\_timestamp\_format is set to "Manual". | | timezone | `timezone` | UTC | The timezone to use if the Timestamp Format doesn't include a timezone. Otherwise, the timezone in the Timestamp Format will be respected. **NOTE: This is also required to parse timezone abbreviations, due to their ambiguity.** | | parse\_severity | `bool` | false | Whether to parse severity from the log entry. | | severity\_field | `string` | severity | The field containing the severity in the log entry. | | parse\_to | `string` | body | The [field](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/field.md) that the log will be parsed to. Some exporters handle logs favorably when parsed to `attributes` over `body` and vice versa. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | tls\_certificate\_path | `string` | | Path to the TLS cert to use for TLS-required connections. | | tls\_private\_key\_path | `string` | | Path to the TLS key to use for TLS-required connections. | | tls\_min\_version | `enum` | "1.2" | The minimum TLS version to support. 1.0 and 1.1 should not be considered secure. Valid values include: `1.3`, `1.2`, `1.1`, `1.0`. | # Telemetry Generator Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/telemetry-generator Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | ✓ | | Windows | ✓ | ✓ | ✓ | | macOS | ✓ | ✓ | ✓ | ## Description The Telemetry Generator is a source that generates random telemetry data for testing purposes. This source is useful for testing the load and pipeline configurations. ## Minimum Agent Versions * Introduced: [v1.46.0](https://github.com/observIQ/bindplane-agent/releases/tag/v1.46.0) * Updated to include Host Metrics & Windows Events: [v1.47.0](https://github.com/observIQ/bindplane-agent/releases/tag/v1.47.0) ## Supported Pipelines * Logs * Metrics * Traces ## Configuration for all generators | Field | Default | Required | Description | | ------------------- | ------- | -------- | -------------------------------------------------------------- | | Payloads per second | `1` | `true` | The number of payloads this receiver will generate per second. | ### Logs Generator Configuration | Field | Description | | ------------------- | -------------------------------------------------------------------------------------------- | | Resource Attributes | A map of resource attributes to be included in the generated telemetry. Values can be `any`. | | Attributes | A map of attributes to be included in the generated telemetry. Values can be `any`. | | body | The body of the log | | severity | The severity of the log message | ### OTLP Replay Generator The OTLP Replay Generator replays JSON-formatted telemetry. It adjusts the timestamps of the telemetry relative the current time, with the most recent record moved to the current time, and the previous records the same relative duration in the past. The text in the `OTLP JSON` box should be valid [JSON-formatted OTLP](https://github.com/open-telemetry/opentelemetry-proto#otlpjson), such as the JSON created by `plog.JSONMarshaler`,`ptrace.JSONMarshaler`, or `pmetric.JSONMarshaler`. | Field | Description | | --------- | ---------------------------------------------------------------- | | Type | The type of telemetry to replay: `logs`, `metrics`, or `traces`. | | OTLP JSON | A string of JSON encoded OTLP telemetry | ### Host Metrics Generator The host metrics generator creates synthetic host metrics, from a list of pre-defined metrics. The metrics resource attributes can be set in the `Resource Attributes` section of the configuration. ### Windows Events Generator The Windows Events Generator replays a sample of recorded Windows Event Log data. It has no additional configuration. # Tomcat Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/tomcat Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This source supports Apache Tomcat versions 9.0.x and 10.x. ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :------------------------------------------------ | :------------------------------------------------ | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | address | `string` | localhost | IP address or hostname to scrape for JMX metrics. | | port | `int` | 9012 | Port to scrape for JMX metrics. | | jar\_path | `string` | "/opt/opentelemetry-java-contrib-jmx-metrics.jar" | Full path to the JMX metrics jar. | # Ubiquiti Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/ubiquiti Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :----------- | :--------- | :-------- | :------------------------------------------------------------------------------ | | listen\_port | `int` | 5140 | A UDP port which the agent will listen for syslog messages. | | listen\_ip | `string` | "0.0.0.0" | An IP address for the agent to bind. Typically 0.0.0.0 for most configurations. | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | # UDP Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/udp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------------ | :--------- | :----------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | listen\_port\* | `int` | | Port to listen on. | | listen\_ip | `string` | "0.0.0.0" | IP Address to listen on. | | log\_type | `string` | udp | Arbitrary for attribute 'log\_type'. Useful for filtering between many udp sources. | | parse\_format | `enum` | none | Method to use when parsing. Valid values are `none`, `json`, and `regex`. When regex is selected, 'Regex Pattern' must be set. | | regex\_pattern | `string` | | The regex pattern used when parsing log entries. | | multiline\_line\_start\_pattern | `string` | | Regex pattern that matches the beginning of a log entry, for handling multiline logs. | | multiline\_line\_end\_pattern | `string` | | Regex pattern that matches the end of a log entry, useful for terminating parsing of multiline logs. | | parse\_timestamp | `bool` | false | Whether to parse the timestamp from the log entry. | | timestamp\_field | `string` | timestamp | The field containing the timestamp in the log entry. | | parse\_timestamp\_format | `enum` | ISO8601 | The format of the timestamp in the log entry. Choose a common format, or specify a custom format. Options include "ISO8601", "RFC3339", "Epoch", and "Manual". | | epoch\_timestamp\_format | `enum` | s | The layout of the epoch-based timestamp. Required when parse\_timestamp\_format is set to "Epoch".. Options include "s", "ms", "us", "ns", "s.ms", "s.us", "s.ns". | | manual\_timestamp\_format | `string` | '%Y-%m-%dT%H:%M:%S.%f%z' | The strptime layout of the timestamp. Used when parse\_timestamp\_format is set to "Manual". | | timezone | `timezone` | UTC | The timezone to use if the Timestamp Format doesn't include a timezone. Otherwise, the timezone in the Timestamp Format will be respected. **NOTE: This is also required to parse timezone abbreviations, due to their ambiguity.** | | parse\_severity | `bool` | false | Whether to parse severity from the log entry. | | severity\_field | `string` | severity | The field containing the severity in the log entry. | | parse\_to | `string` | body | The [field](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/field.md) that the log will be parsed to. Some exporters handle logs favorably when parsed to `attributes` over `body` and vice versa. | # VMware ESXi Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/vmware-esxi Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------- | :------- | :-------- | :---------------------------------------------------------------------------------------------------------------------------------------------- | | listen\_port\* | `int` | 5140 | The port to bind to and receive syslog. Collector must be running as root (Linux) or Administrator (windows) when binding to a port below 1024. | | listen\_ip | `string` | "0.0.0.0" | The IP address to bind to and receive syslog. | | enable\_tls | `bool` | false | Whether or not to use TLS. | | cert\_file | `string` | | Path to the x509 PEM certificate. | | key\_file | `string` | | Path to the x509 PEM private key. | # VMware vCenter Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/vmware-vcenter Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Prerequisites This receiver has been built to support ESXi and vCenter versions: * 7.5 * 7.0 * 6.7 A “Read Only” user assigned to a vSphere with permissions to the vCenter server, cluster and all subsequent resources being monitored must be specified in order for the receiver to retrieve information about them. ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | endpoint\* | `string` | | Endpoint to the vCenter Server or ESXi host that has the sdk path enabled. Required. The expected format is `://` \n \ni.e: `https://vcsa.hostname.localnet` | | username\* | `string` | | Username used to authenticate. | | password\* | `string` | | Password used to authenticate. | | tls | `string` | | Not Required. Will use defaults for [configtls.TLSClientSetting](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md). By default insecure settings are rejected and certificate verification is on. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | # W3C Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/w3c Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | | macOS | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------------------------------------- | :-------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | file\_path | `strings` | | File or directory paths to tail for logs. | | exclude\_file\_log\_path | `strings` | | File or directory paths to exclude. | | delimiter | `enum` | tab | Delimiter character used between the fields of the W3C log line. Valid values include: `tab`, `space`. | | encoding | `enum` | utf-8 | The encoding of the file being read. Valid values include: `nop`, `utf-8`, `utf-16le`, `utf-16be`, `ascii`, `big5`. | | header\_delimiter | `enum` | default | Delimiter character used between fields in the W3C Field header. The value of the "Delimiter" parameter is used by default. Valid values include: `tab`, `space`, `default`. | | include\_file\_name\_attribute | `bool` | true | Whether to add the file name as the attribute "log.file.name". | | include\_file\_path\_attribute | `bool` | false | Whether to add the file path as the attribute "log.file.path". | | include\_file\_name\_resolved\_attribute | `bool` | false | Whether to add the file name after symlinks resolution as the attribute "log.file.name\_resolved". | | include\_file\_path\_resolved\_attribute | `bool` | false | Whether to add the file path after symlinks resolution as the attribute "log.file.path\_resolved". | # WildFly Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/wildfly Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | | ✓ | | | Windows | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------------- | :--------- | :---------------------------------------------- | :---------------------------------------------------- | | standalone\_file\_path | `strings` | /opt/wildfly/standalone/log/server.log | File paths to tail for standalone server logs. | | enable\_domain\_server | `bool` | true | Enable to read domain server logs. | | domain\_server\_path | `strings` | '/opt/wildfly/domain/servers/\*/log/server.log' | File paths to tail for domain server logs. | | enable\_domain\_controller | `bool` | true | Enable to read domain controller logs. | | domain\_controller\_path | `strings` | '/opt/wildfly/domain/log/\*.log' | File paths to tail for domain controller logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | | timezone | `timezone` | "UTC" | The timezone to use when parsing timestamps. | # Windows DHCP Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/windows-dhcp Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Windows | | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :--------- | :-------- | :------------------------------------------- | :---------------------------------------------------- | | file\_path | `strings` | "C:/Windows/System32/dhcp/DhcpSrvLog-\*.log" | File or directory paths to tail for logs. | | start\_at | `enum` | end | Start reading the file from the 'beginning' or 'end'. | # Windows Event Trace (ETW) Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/windows-event-trace-etw Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Prerequisites This source is built on top of Windows ETW, which has resource limitations and was not initially designed for long-term monitoring. Please use with caution on your host systems. While the source supports reading events from ETL logs, using them for continuous event collection may result in dropped events and increased resource utilization on your hosts if not tuned to your specific environment (the default settings may not work out of the box). This source creates a Real-Time ETW Session to ingest Event Tracing Logs from Windows using the [Event Tracing API](https://learn.microsoft.com/en-us/windows-hardware/test/weg/instrumenting-your-code-with-etw). Currently, only real-time monitoring sessions are supported. ## System Requirements * Windows system with Event Tracing for Windows (ETW) enabled * Familiarity with ETW provider GUIDs for configuration * Administrative privileges for ETW session creation For more information on ETW, refer to Microsoft's [ETW Documentation](https://docs.microsoft.com/en-us/windows/win32/etw/event-tracing-portal). ## Supported Platforms | Platform | Supported | | -------- | --------- | | Windows | ✓ | Available in Bindplane Distro for OpenTelemetry Collector `v1.75.0+`. ## Configuration Fields

Field	Description
Session Name	Name of the ETW session to create and monitor. Default: "Bindplane-ETW-Session"
Providers	Provider names or GUIDs to monitor (e.g., `Microsoft-Windows-Kernel-File`)
Enable Raw Logs (XML)	When enabled, logs are saved as raw XML strings instead of parsed objects. Useful for XML analysis in some destinations.
Level	Maximum event level to ingest. Options: `none`, `verbose`, `informational`, `warning`, `error`, `critical`.
Session Buffer Size	Buffer size for the ETW session. Default: 64 KiB
Require All Providers	When enabled, source only starts if all providers are available. Default: false

## Common ETW Providers | Provider Name | | ----------------------------- | | Microsoft-Windows-Kernel-File | | Microsoft-Windows-DNS-Client | ## Discovering Available Providers To list all registered ETW providers on your system, run this command in an administrative PowerShell session: ```powershell theme={} logman query providers ``` ## Best Practices 1. Start with a minimal set of providers and expand gradually based on monitoring needs 2. Monitor resource usage when enabling multiple providers 3. Use Performance Monitor to view session settings under `Event Trace Sessions` 4. Consider the impact on system performance when enabling verbose logging levels # Windows Events Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/windows-events Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. | Platform | Metrics | Logs | Traces | | -------- | ------- | ---- | ------ | | Windows | | ✓ | | #### Prerequisites for Remote Configuration ##### Supported Versions * Windows Vista or later ##### Minimum Setup Requirements * **User Permissions:** * The user must be a member of the `Event Log Readers` group. * The user must have DCOM and WMI permissions for remote access. * **Firewall Configuration:** * Ensure the firewall rules allow the necessary ports: TCP 135, 445, and dynamic RPC ports (49152-65535). * **Windows Firewall Exception:** * Enable the "Remote Event Log Management" exception on the remote machine. #### Configuration Table ##### Windows Event Log Receiver | Parameter | Type | Default | Description | | ---------------------- | ------- | ------- | ------------------------------------- | | system\_event\_input | bool | true | Enable the System event channel. | | app\_event\_input | bool | true | Enable the Application event channel. | | security\_event\_input | bool | true | Enable the Security event channel. | | custom\_channels | strings | | Custom channels to read events from. | ##### Remote Configuration Options | Parameter | Type | Default | Description | | --------------- | ------ | ------- | ----------------------------------------------- | | remote.server | string | | The server to connect to for remote event logs. | | remote.username | string | | The username to authenticate with the server. | | remote.password | string | | The password to authenticate with the server. | | remote.domain | string | | The domain of the server (optional). | # ZooKeeper Source: https://docs.honeycomb.io/send-data/telemetry-pipeline/sources/zookeeper Ent+ This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/). Please contact your Honeycomb account team for details. ## Supported Platforms | Platform | Metrics | Logs | Traces | | :------- | :------ | :--- | :----- | | Linux | ✓ | ✓ | | | Windows | ✓ | ✓ | | | macOS | ✓ | ✓ | | ## Configuration Table | Parameter | Type | Default | Description | | :------------------- | :------------------ | :-------------------- | :---------------------------------------------- | | telemetry\_types | `telemetrySelector` | `["Logs", "Metrics"]` | Choose Telemetry Type. | | collection\_interval | `int` | 60 | Sets how often (seconds) to scrape for metrics. | | address | `string` | localhost | IP address or hostname of the ZooKeeper system. | | port | `int` | 2181 | Port of the ZooKeeper system. | # Instrumenting AI Agents Source: https://docs.honeycomb.io/send-data/use-cases/agents Track input prompts, responses, token usage, tool calls, and agent invocations using OpenTelemetry. AI agents may invoke tools, call other agents, or prompt a Generative AI (GenAI) model with multi-step user input. The distributed and non-deterministic nature of agentic workflows makes monitoring and debugging difficult. Instrument your AI agents with OpenTelemetry (OTel) GenAI semantic conventions to get full visibility into agent sessions and explore your agents in the [Agent Timeline](/investigate/observe/agent-timeline). * Refer to [Fast AI Feedback Loops with Honeycomb and OpenTelemetry](https://www.honeycomb.io/blog/fast-ai-feedback-loops-honeycomb-opentelemetry) for an example of agent instrumentation using [Pydantic](https://ai.pydantic.dev/logfire/#setting-opentelemetry-sdk-providers). * Visit the [example storechat](https://github.com/honeycombio/devrel-opentelemetry-demo/tree/main/src/storechat) application repo for further agent instrumentation examples. ## Enriching your traces with GenAI context Add the following [OTel GenAI attributes](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/) to your agent spans to give Honeycomb the context it needs to group spans into conversations, identify agents, and surface meaningful data in the Agent Timeline. Unique identifier for the conversation or session. Used to group all traces and spans belonging to the same agent conversation. All spans in a conversation must have a `gen_ai.conversation.id`. Name of the agent emitting the span. In multi-agent workflows [each agent should have a unique name](#use-unique-names-for-each-agent). Type of agentic operation occurring: * `chat` * `create_agent` * `embeddings` * `execute_tool` * `generate_content` * `invoke_agent` * `invoke_workflow` * `retrieval` * `text_completion` For span naming conventions, refer to the [Naming generative AI operation spans](#naming-generative-ai-operation-spans) section. Number of tokens used in the GenAI input prompt. Number of tokens used in the GenAI response. Name of the model requested. Name of the model that generated the response. This can differ from the requested model. Why the model stopped generating tokens. Examples: `["stop"]`, `["tool_calls"]`, `["stop", "length"]` Name of the tool called by the agent. Unique identifier for the tool call. Parameters passed to the tool call. Result returned by the tool call (if any). To learn how to handle failed tool calls, refer to the [Recording errors and exceptions](#recording-errors-and-exceptions) section. ### Span events for input prompts, completions, and evaluations Full prompts, chat history, and completion responses may be too large or contain personally identifiable information (PII) or other sensitive data. Store them in span events where your OTel Collector can filter them before they reach Honeycomb. Chat history or input prompts provided to the model. GenAI prompts or chats may contain PII or other sensitive data. Messages returned by the model. Each message represents a specific model response. GenAI responses may contain PII or other sensitive data. Attach `gen_ai.evaluation.result` events to the GenAI operation span to [review evaluations in the GenAI tab](/investigate/analyze/explore-traces#gen-ai-tab). ## Keeping agent names unique The [Agent Timeline](/investigate/observe/agent-timeline) groups spans by `gen_ai.agent.name`, so duplicate or missing names make it impossible to distinguish between agents during an investigation. Each agent should have its own unique `gen_ai.agent.name`. Sub-agents should use their own distinct name, instead of inheriting the parent agent's name. If `gen_ai.agent.name` is omitted on a span, it will show up as `"Unknown"` on the Agent Timeline. ## Instrumenting agent-to-agent calls In multi-agent workflows, correctly attributing spans to the right agent is what makes the Agent Timeline useful. The calling agent should emit the `invoke_agent` span, not the agent being called. The called agent then emits its own spans (`chat`, `execute_tool`, and so on) under its own unique `gen_ai.agent.name`, keeping each agent's work distinct and traceable in the timeline. To learn more about agent invocation spans, visit the OTel documentation: * [Invoke agent client span (remote invocation)](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/#invoke-agent-client-span) * [Invoke agent internal span (same process invocation)](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/#invoke-agent-internal-span) ## Handling errors and exceptions Agentic workflows can fail in subtle ways: a tool call that silently drops its result, or an error that surfaces in a child span but never reaches the parent. Recording errors consistently gives you the full picture when something goes wrong. Record errors or exceptions following [the OTel specification](https://opentelemetry.io/docs/specs/semconv/general/recording-errors/) and include as many attributes as apply: * `error.type` / `exception.type` * `error.message` / `exception.message` * `error.stacktrace` / `exception.stacktrace` For tool call failures, propagate the error status to the parent span. ## Naming generative AI operation spans Generative AI operation spans should follow these naming conventions. Naming your spans this way ensures the [Agent Timeline](/investigate/observe/agent-timeline) can understand the operation type and display it meaningfully. | Operation | `gen_ai.operation.name` | Span Name Pattern | | ----------------------------- | ----------------------- | --------------------------- | | Chat | `chat` | `chat {model}` | | Create GenAI agent | `create_agent` | `create_agent {agent_name}` | | Tool execution | `execute_tool` | `execute_tool {tool_name}` | | Agent invocation | `invoke_agent` | `invoke_agent {agent_name}` | | Embeddings | `embeddings` | `embeddings {model}` | | RAG retrieval | `retrieval` | `retrieval {data_source}` | | Multimodal content generation | `generate_content` | `generate_content {model}` | | Text completions | `text_completion` | `text_completion {model}` | ## Remapping existing telemetry In some cases, your existing telemetry may not conform to OpenTelemetry semantic conventions. The [`transform` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) lets you remap those attributes before they reach Honeycomb. For example, [Claude Code can emit telemetry](https://code.claude.com/docs/en/monitoring-usage) using the OpenTelemetry protocol, but does not yet use the OpenTelemetry semantic conventions. The following configuration remaps Claude Code's emitted traces to match the OpenTelemetry semantic conventions, making them visible in the Agent Timeline. ```yml theme={} processors: # Detailed Beta names spans `claude_code.*` and uses bare-namespace # attribute keys (session.id, input_tokens, …). The transform processor # remaps both names and attributes to the GenAI semconv on the way # through. Statements run in order, per OTTL context. transform: error_mode: ignore trace_statements: - context: resource statements: # Set service.name to identify your agent. - set(attributes["service.name"], "") - context: span statements: # ---- Identity attributes added to every span ---- # The AI Conversations viewer and Agent Timeline both key on # gen_ai.conversation.id; Detailed Beta carries the same value # under `session.id`. - set(attributes["gen_ai.agent.name"], "") - set(attributes["gen_ai.conversation.id"], attributes["session.id"]) where attributes["session.id"] != nil # ---- claude_code.interaction → invoke_agent claude ---- # Order matters. Rename the span first, then set # gen_ai.operation.name keyed off the new name. Detailed Beta # emits exactly one interaction span per Claude invocation. - set(name, "invoke_agent ") where name == "claude_code.interaction" - set(attributes["gen_ai.operation.name"], "invoke_agent") where name == "invoke_agent " # ---- claude_code.llm_request → chat {model} ---- # gen_ai.request.model is already populated by Detailed Beta. # If it is missing on some build, leave the original name in # place rather than renaming to "chat ". - set(name, Concat(["chat ", attributes["gen_ai.request.model"]], "")) where name == "claude_code.llm_request" and attributes["gen_ai.request.model"] != nil - set(attributes["gen_ai.operation.name"], "chat") where IsMatch(name, "^chat ") # ---- claude_code.tool → execute_tool {tool_name} ---- # Detailed Beta names the wrapping tool span `claude_code.tool` # and carries the tool identity in the `tool_name` attribute. - set(name, Concat(["execute_tool ", attributes["tool_name"]], "")) where name == "claude_code.tool" and attributes["tool_name"] != nil - set(attributes["gen_ai.operation.name"], "execute_tool") where IsMatch(name, "^execute_tool ") - set(attributes["gen_ai.tool.name"], attributes["tool_name"]) where IsMatch(name, "^execute_tool ") # ---- Token alias (Detailed Beta llm_request) ---- # Detailed Beta puts some tokens in gen_ai.* and others in the # bare namespace. Alias the bare ones up so the GenAI consumers # see the full set on each chat {model} span. - set(attributes["gen_ai.usage.input_tokens"], attributes["input_tokens"]) where attributes["input_tokens"] != nil - set(attributes["gen_ai.usage.output_tokens"], attributes["output_tokens"]) where attributes["output_tokens"] != nil - set(attributes["gen_ai.usage.cache_read_input_tokens"], attributes["cache_read_tokens"]) where attributes["cache_read_tokens"] != nil - set(attributes["gen_ai.usage.cache_creation_input_tokens"], attributes["cache_creation_tokens"]) where attributes["cache_creation_tokens"] != nil # ---- Tool args / result remap ---- # With OTEL_LOG_TOOL_DETAILS=1 + OTEL_LOG_TOOL_CONTENT=1 # claude_code.tool spans carry two content attributes: # * `tool_input` — the tool arguments JSON, prefixed with # "[TOOL INPUT: ]\n" # * `new_context` — the tool result JSON, prefixed with # "[TOOL RESULT: ]\n" # Alias them onto gen_ai.tool.call.arguments / .result. # gen_ai.tool.call.id has no Detailed Beta equivalent, so it # stays absent. - set(attributes["gen_ai.tool.call.arguments"], attributes["tool_input"]) where IsMatch(name, "^execute_tool ") and attributes["tool_input"] != nil - set(attributes["gen_ai.tool.call.result"], attributes["new_context"]) where IsMatch(name, "^execute_tool ") and attributes["new_context"] != nil ``` # Monitor Anthropic Usage & Cost Source: https://docs.honeycomb.io/send-data/use-cases/anthropic-usage-monitoring Monitor Anthropic API token consumption, feature usage, and cost attribution in real time using a custom OpenTelemetry collector and Honeycomb Board Templates. Monitor your Anthropic API usage and costs in real-time by collecting detailed metrics and logs with a custom OpenTelemetry collector. This integration provides comprehensive visibility into token consumption, feature usage, and cost attribution across models, workspaces, and API keys. ## Overview The Anthropic Usage & Cost Monitoring integration uses an OpenTelemetry receiver to collect detailed usage metrics and cost data from the Anthropic Admin API, transforming them into structured telemetry for monitoring and cost tracking in Honeycomb. **What's Collected:** * **Usage Metrics** (minute-level granularity): Token consumption (input, output, cache), feature usage (web searches), grouped by model, tier, workspace, and API key * **Cost Data** (daily logs): Detailed cost breakdowns with workspace attribution and human-readable descriptions ## Prerequisites * [Anthropic Admin API key](https://console.anthropic.com/settings/keys) with organization admin access * [OpenTelemetry Collector Builder](https://github.com/open-telemetry/opentelemetry-collector/releases) v0.130.0+ * Go 1.23+ (for building custom collector) ## Custom Collector Setup The Anthropic Usage Receiver is designed to be integrated into a custom OpenTelemetry Collector using the OpenTelemetry Collector Builder (OCB). Download the appropriate OCB binary for your platform: ```bash theme={} # For Linux/macOS (adjust version and platform as needed) curl -L -O https://github.com/open-telemetry/opentelemetry-collector/releases/download/cmd%2Fbuilder%2Fv0.131.0/ocb_0.131.0_linux_amd64 chmod +x ocb_0.131.0_linux_amd64 mv ocb_0.131.0_linux_amd64 ocb ``` Create a `builder-config.yaml` file that includes the Anthropic Usage Receiver: ```yaml theme={} dist: name: otelcol-anthropic description: Custom OpenTelemetry Collector with Anthropic Usage Receiver output_path: ./dist otelcol_version: 0.131.0 exporters: - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.131.0 - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.131.0 - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.131.0 processors: - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.131.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.131.0 receivers: - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.131.0 - gomod: github.com/honeycombio/anthropic-usage-receiver/anthropicusagereceiver v0.1.0 extensions: - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.131.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.131.0 ``` ```bash theme={} ./ocb --config builder-config.yaml ``` This generates a custom collector binary in the `./dist` directory. ## Deployment Options ### Stateless Deployment For environments where persistence isn't required (e.g., short-lived containers, testing): ```yaml theme={} receivers: anthropicusage: admin_api_key: ${env:ANTHROPIC_ADMIN_API_KEY} endpoint: https://api.anthropic.com collection_interval: 1m usage: enabled: true bucket_width: 1m group_by: [model, service_tier, context_window, workspace_id, api_key_id] cost: enabled: true collection_interval: 24h bucket_width: 1d group_by: [workspace_id, description] exporters: otlp_http: endpoint: https://api.honeycomb.io:443 headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" service: pipelines: metrics: receivers: [anthropicusage] processors: [batch] exporters: [otlp_http] logs: receivers: [anthropicusage] processors: [batch] exporters: [otlp_http] ``` ### Stateful Deployment For production environments where checkpoint persistence is important to prevent data loss during restarts: ```yaml theme={} extensions: file_storage: directory: /var/lib/otel/anthropic-storage timeout: 10s receivers: anthropicusage: admin_api_key: ${env:ANTHROPIC_ADMIN_API_KEY} endpoint: https://api.anthropic.com collection_interval: 1m storage: file_storage # Enable checkpoint persistence usage: enabled: true bucket_width: 1m group_by: [model, service_tier, context_window, workspace_id, api_key_id] cost: enabled: true collection_interval: 24h bucket_width: 1d group_by: [workspace_id, description] retry_on_failure: enabled: true initial_interval: 5s max_interval: 60s max_elapsed_time: 5m exporters: otlp_http: endpoint: https://api.honeycomb.io:443 headers: "x-honeycomb-team": "${env:HONEYCOMB_API_KEY}" processors: batch: timeout: 10s send_batch_size: 512 service: extensions: [file_storage] pipelines: metrics: receivers: [anthropicusage] processors: [batch] exporters: [otlp_http] logs: receivers: [anthropicusage] processors: [batch] exporters: [otlp_http] ``` **Stateful vs Stateless:** * **Stateful**: Maintains checkpoint state across restarts, preventing data loss and duplicate collection. Recommended for production. * **Stateless**: Simpler setup but may miss data during restarts. Suitable for testing or environments with external state management. ### Run the Collector ```bash theme={} export ANTHROPIC_ADMIN_API_KEY="sk-ant-admin-..." export HONEYCOMB_API_KEY="your-honeycomb-api-key" ./dist/otelcol-anthropic --config=config.yaml ``` ## Using the Anthropic Usage Board Template Once your data is flowing into Honeycomb, use the pre-built Anthropic Usage & Cost Monitoring Board Template to instantly visualize your usage and costs: 1. Navigate to **Boards** in the Honeycomb UI 2. Select **Templates** or **Create Boards** → **From Template** 3. Choose **Anthropic Usage & Cost Monitoring** template 4. Review the template preview with your data 5. If needed, map any fields in the **Setup** view to match your data structure 6. Select **Use Template** to create your board The template includes pre-configured visualizations for: * Token usage trends and patterns * Cost monitoring and attribution * Cache utilization analysis * Usage distribution across models and workspaces The board template automatically works with data from the Anthropic Usage Receiver. If some queries show "Unable to display", verify your collector is running and data is flowing to Honeycomb. ## Configuration Reference ### Usage Metrics | Metric | Description | | --------------------------------------------- | ------------------------------------------------- | | `anthropic.usage.input_tokens` | Number of uncached input tokens consumed | | `anthropic.usage.output_tokens` | Number of output tokens generated | | `anthropic.usage.cache_creation.ephemeral_1h` | Tokens used for 1-hour ephemeral cache creation | | `anthropic.usage.cache_creation.ephemeral_5m` | Tokens used for 5-minute ephemeral cache creation | | `anthropic.usage.cache_read_input_tokens` | Input tokens read from cache | | `anthropic.usage.web_search_requests` | Number of web search requests | ### Cost Metrics | Metric | Description | | ----------------------- | -------------------------------------------------------------------- | | `anthropic.cost.amount` | Cost amount in minor currency units (e.g., fractional cents for USD) | ### Key Attributes | Attribute | Description | Example | | ---------------- | ----------------------------------------------------------- | ------------------------------------------------------------------ | | `model` | Claude model used | `claude-3-7-sonnet-20250219`, `claude-opus-4-20250514` | | `service_tier` | API service tier | `standard`, `batch`, `priority` | | `context_window` | Context window size | `0-200k`, `200k-1M` | | `workspace_id` | Anthropic workspace identifier (null for default workspace) | `wrkspc_01JwQvzr7rXLA5AGx3HKfFUJ` | | `api_key_id` | API key identifier (null for workbench usage) | `apikey_01Rj2N8SVvo6BePZj99NhmiT` | | `description` | Human-readable cost description | `Claude Sonnet 4 Usage - Input Tokens, Cache Write 1 Hour - Batch` | | `currency` | ISO 4217 currency code | `USD` | ## Advanced Configuration ### Filtering Data Collection Limit data collection to specific models, tiers, or workspaces: ```yaml theme={} receivers: anthropicusage: usage: # Collect only specific models models: ["claude-3-7-sonnet-20250219", "claude-3-haiku-20240307"] # Filter by service tiers service_tiers: ["standard", "priority"] # Filter by context window sizes context_window: ["0-200k", "200k-1M"] # Filter by workspaces workspace_ids: ["wrkspc_01JwQvzr7rXLA5AGx3HKfFUJ", "wrkspc_013jtUYxRJqPEXaTUxQVmmJa"] # Customize grouping group_by: [model, service_tier, context_window, workspace_id] ``` ### Multiple Exporters Send data to multiple destinations: ```yaml theme={} service: pipelines: metrics: receivers: [anthropicusage] processors: [batch] exporters: [otlp, prometheus, debug] logs: receivers: [anthropicusage] processors: [batch] exporters: [otlp, debug] ``` ## Troubleshooting ### Common Issues **No data appearing in Honeycomb:** * Verify your Anthropic Admin API key has proper permissions * Check collector logs for authentication errors * Ensure your Honeycomb API key is configured correctly **Missing cost data:** * Cost data is only available at daily granularity * Ensure `cost.enabled: true` in your configuration * Check that your Admin API key has cost visibility permissions **Checkpoint persistence not working:** * Verify the storage directory is writable * Check that the file\_storage extension is properly configured * Ensure the storage extension is referenced in the service section **High API usage:** * Increase `collection_interval` to reduce API call frequency (minimum recommended: 1 minute) * Use filtering to collect only necessary data * Monitor the `anthropic.scraper.up` metric for health status * Note: Usage and cost data appears within 5 minutes of API request completion **Rate limiting:** * The API is designed for polling once per minute maximum * Reduce polling frequency if you encounter rate limit errors * Consider caching results for frequently updated dashboards ### Getting Help * View detailed configuration options in the [Anthropic Usage Receiver repository](https://github.com/honeycombio/anthropic-usage-receiver) * Check the [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/) for general collector issues * For Honeycomb-specific questions, consult the [Honeycomb documentation](/troubleshoot/) or contact support # Monitor LLM-Powered Applications Source: https://docs.honeycomb.io/send-data/use-cases/llm Instrument LLM-powered applications with OpenTelemetry and send telemetry to Honeycomb to monitor model behavior, latency, and output quality in production. Observability is a critical part of building and maintaining applications that use large language models (LLMs). Learn how Observability can help you understand how your LLMs are behaving in production, and how to get started with Honeycomb. ## Why Observability for LLMs Converting user input (and some contextual data for that user) into a useful output is something that LLMs are great at. However, it can often be difficult to debug then when they fail at their task. The reasons why they fail are varied: * Natural language inputs can be ill-specified, ambiguous, or simply unexpected. * You have little to no hope of predicting what users will input. * Similarly, you have little to no hope of predicting how the LLM will respond to a given input, let alone how useful that output is for users. * When users are presented with a natural language input, they may try things they would have otherwise not thought to try with other systems. * Small changes to the prompt can have a large impact on the output, making regressions easy. * Depending on the model you're using or its settings, outputs are nondeterministic. Furthermore, this may be by design. All of the above reasons make it difficult to understand how your LLM is behaving in production, and it's not tractable to try and predict all of the ways it might fail before you go live. So, how do you debug your LLMs in production? The best way today is to observe them and systematically analyze which user inputs, and which contextual pieces of data that are combined with those inputs, lead to particular outputs. ## LLM Observability Needs Traces [Trace data](/get-started/basics/observability/concepts/distributed-tracing/) is essential to understanding the lifecycle of a system end-to-end as requests flow through it. For LLMs, it's critical to use OpenTelemetry traces for two reasons: 1. Traces let you represent several operations that perform meaningful work before or after a call to an LLM. 2. OpenTelemetry lets you correlate the behavior tracked in an LLM request with all other behavior in your application. Most production applications that use LLMs perform several meaningful operations before and after a call to an LLM, particularly for retrieving relevant context for an input, or [RAG (Retrieval-Augmented Generation)](https://arxiv.org/abs/2005.11401). For example, it's common to use an [Embedding model](https://platform.openai.com/docs/guides/embeddings) to calculate vector embeddings for a user's input, and then use those embeddings to query an index or database of other embeddings for semantically relevant context. Retrieval can sometimes be quite complex, and it's important to understand how it behaves in production, because the results of retrieval directly impact how an LLM behaves. Similarly, it's also common to perform several operations on an LLM's output rather than returning that output to the user directly. For example, there may be ways to not only validate that the output has a particular structure or contents, but some outputs can be programmatically modified to be correct based on other known information. Understanding how validation can fail, or how many outputs had to be programmatically corrected (and which corrections were involved), can inform how to improve your prompts or fine-tuning, if applicable. Tracing is what ties all of these things together, and it's also what ties the behavior of an LLM to the rest of your application. By tying the LLM request behavior to the rest of an application, you can get a complete picture of how your application is behaving for your users. For example, you may notice a spike in latency for your system that involves a request to the LLM. Is this because the LLM is taking longer to respond, or is it because some other network calls you're making are taking longer? By looking at trace data that correlates the rest of your request flow with the LLM request, you can understand exactly what's affecting a poor user experience. ## Getting Started To explore Observability for LLMs, try our [Text to JSON Quick Start](/send-data/llm/text-to-json-quickstart/) guide, which requires an application that uses an LLM. # Text to JSON Quick Start Source: https://docs.honeycomb.io/send-data/use-cases/llm/text-to-json-quickstart Instrument a text-to-JSON LLM workflow with OpenTelemetry and send telemetry to Honeycomb to observe model inputs, outputs, and validation steps. An extremely common use case for large language models (LLMs) is to convert text to JSON where the LLM: * generates a JSON object from user input * parses that object * validates its contents * uses the content elsewhere in the application Collecting observability data for this process is a great fit as clear and measurable outcomes are present for each step. This data answers questions like: * Do some user inputs result in higher latency or more errors? * Was the JSON object structurally valid as per the JSON schema? * Did the JSON object contain expected fields? * Did the new version of our prompt result in more invalid JSON objects? * Are there particular patterns in the user input that result in invalid JSON objects? Adding telemetry to your LLM and then analyzing with Honeycomb provides a flexible way to contribute towards and measure the overall user experience. Use our Quick Start to instrument your LLM application. ## Before You Begin Before instrumenting your LLM application, you'll need to do a few things: 1. **[Create a Honeycomb Account](https://ui.honeycomb.io/signup)**. Signup is free! 2. **Create a Honeycomb Team**. Complete your account creation by giving us a team name. Honeycomb uses teams to organize groups of users, grant them access to data, and create a shared work history in Honeycomb. We recommend using your company or organization name as your Honeycomb team name. 3. **Get Your Honeycomb API Key**. To send data to Honeycomb, you'll need your Honeycomb API Key. Once you create your team, you will be able to view or copy your API key. Make note of it; you will need it later! You can also [find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys) any time in your Environment Settings. ## Send Telemetry Data to Honeycomb Once you have your Honeycomb API key and your LLM application to instrument, it's time to send telemetry data to Honeycomb! To instrument your LLM, you will [add automatic instrumentation to your code](#add-automatic-instrumentation-to-your-code) for standard trace data telemetry, and then [add custom instrumentation specifically for your LLM](#add-custom-instrumentation-for-llms). ### Add Automatic Instrumentation to Your Code The quickest way to start seeing your trace data in Honeycomb is to use [OpenTelemetry](https://opentelemetry.io/), an open-source collection of tools, APIs, and SDKs, to automatically inject instrumentation code into your application without requiring explicit changes to your codebase. Automatic instrumentation works slightly differently within each language, but the general idea is that it attaches hooks into popular tools and frameworks and "watches" for certain functions to be called. When they're called, the instrumentation automatically starts and completes trace spans on behalf of your application. When you add automatic instrumentation to your code, OpenTelemetry will inject spans, which represent units of work or operations within your application that you want to capture and analyze for observability purposes. This Quick Start uses the `npm` dependency manager. For instructions with `yarn` or if using TypeScript, read our [OpenTelemetry Node.js documentation](/send-data/javascript-nodejs/opentelemetry-sdk/#add-automatic-instrumentation). #### Acquire Dependencies Open your terminal, navigate to the location of your project on your drive, and install OpenTelemetry's automatic instrumentation meta package and OpenTelemetry's Node.js SDK package: ```shell theme={} npm install --save \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-node ``` | Module | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `auto-instrumentations-node` | OpenTelemetry's meta package that provides a way to add automatic instrumentation to any Node application to capture telemetry data from a number of popular libraries and frameworks, like `express`, `dns`, `http`, and more. | | `sdk-node` | OpenTelemetry's Node.js distribution package that streamlines configuration and allows you to instrument as quickly and easily as possible. | Alternatively, install [individual instrumentation packages](https://github.com/open-telemetry/opentelemetry-js#instrumentations). If using TypeScript, install `ts-node` to run the code: ```shell theme={} npm install --save-dev ts-node ``` #### Initialize Create an initialization file, commonly known as the `tracing.js` file: ```javascript theme={} // Example filename: tracing.js 'use strict'; const { startNodeSDK } = require('@opentelemetry/sdk-node'); const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); startNodeSDK({ instrumentations: [ getNodeAutoInstrumentations(), ], }); ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces # Use the endpoint below for EU # endpoint: https://api.eu1.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Set the following environment variables before running your application: | Environment Variable | Value | | :------------------- | :----------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. The OpenTelemetry declarative configuration is stable at the specification level. Individual fields still under active development are marked with a `/development` suffix in the YAML (see [configuration versioning](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md#experimental-features)). Check the [language support status](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/language-support-status.md) for per-SDK maturity. Add `meter_provider` and `logger_provider` sections to the same file to export metrics and logs. This version also enables resource detectors, which add attributes such as `host.*` and `process.*` automatically: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} detection/development: detectors: - host: - os: - process: - service: - env: tracer_provider: # traces processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # metrics readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/metrics headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} # Legacy metrics only; omit with the current metrics experience: # - name: x-honeycomb-dataset # value: ${HONEYCOMB_METRICS_DATASET} logger_provider: # logs processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` For the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` throughout the file. If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must also specify the Dataset using the `x-honeycomb-dataset` header. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset" ``` If you are sending data directly to Honeycomb, you must configure the API key and service name. If you are using an [OpenTelemetry Collector](/send-data/opentelemetry/collector/), configure your API key at the Collector level instead. #### Run Your Application Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run the Node.js app with the initialization file: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml node -r ./tracing.js YOUR_APPLICATION_NAME.js ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. Alternatively, you can import the initialization file as the first step in your application lifecycle. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. This Quick Start uses the `pip` package manager. For instructions with `poetry`, read our [OpenTelemetry Python documentation](/send-data/python/opentelemetry-sdk/#add-automatic-instrumentation). #### Acquire Dependencies 1. Install the OpenTelemetry Python packages: ```shell theme={} python -m pip install "opentelemetry-sdk[file-configuration]" \ opentelemetry-instrumentation \ opentelemetry-distro \ opentelemetry-exporter-otlp ``` The `[file-configuration]` extra installs the optional dependencies needed to load a YAML configuration file. 2. Install instrumentation libraries for the packages used by your application. We recommend using the `opentelemetry-bootstrap` tool that comes with the OpenTelemetry SDK to scan your application packages and print out a list of available instrumentation libraries. You should then add these libraries to your `requirements.txt` file: ```shell theme={} opentelemetry-bootstrap >> requirements.txt pip install -r requirements.txt ``` If you do not use a `requirements.txt` file, you can install the libraries directly in your current environment: ```shell theme={} opentelemetry-bootstrap --action=install ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces # Use the endpoint below for EU # endpoint: https://api.eu1.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Set the following environment variables before running your application: | Environment Variable | Value | | :------------------- | :----------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. The OpenTelemetry declarative configuration is stable at the specification level. Individual fields still under active development are marked with a `/development` suffix in the YAML (see [configuration versioning](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md#experimental-features)). Check the [language support status](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/language-support-status.md) for per-SDK maturity. Add `meter_provider` and `logger_provider` sections to the same file to export metrics and logs. This version also enables resource detectors, which add attributes such as `host.*` and `process.*` automatically: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} detection/development: detectors: - host: - os: - process: - service: tracer_provider: # traces processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # metrics readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/metrics headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} # Legacy metrics only; omit with the current metrics experience: # - name: x-honeycomb-dataset # value: ${HONEYCOMB_METRICS_DATASET} logger_provider: # logs processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` For the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` throughout the file. If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must also specify the Dataset using the `x-honeycomb-dataset` header. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset" ``` If you are sending data directly to Honeycomb, you must configure the API key and service name. If you are using an [OpenTelemetry Collector](/send-data/opentelemetry/collector/), configure your API key at the Collector level instead. #### Run Your Application Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run your Python application with the OpenTelemetry Python automatic instrumentation tool `opentelemetry-instrument`: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml opentelemetry-instrument python YOUR_APPLICATION_NAME.py ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. #### Acquire Dependencies The automatic instrumentation agent for OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is run alongside your app. In order to use the automatic instrumentation agent, you must first download it: ```shell theme={} curl -L -O https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces # Use the endpoint below for EU # endpoint: https://api.eu1.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Set the following environment variables before running your application: | Environment Variable | Value | | :------------------- | :----------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. The OpenTelemetry declarative configuration is stable at the specification level. Individual fields still under active development are marked with a `/development` suffix in the YAML (see [configuration versioning](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md#experimental-features)). Check the [language support status](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/language-support-status.md) for per-SDK maturity. Add `meter_provider` and `logger_provider` sections to the same file to export metrics and logs. This version also enables resource detectors, which add attributes such as `host.*` and `process.*` automatically: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} detection/development: detectors: - host: - process: - service: tracer_provider: # traces processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # metrics readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/metrics headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} # Legacy metrics only; omit with the current metrics experience: # - name: x-honeycomb-dataset # value: ${HONEYCOMB_METRICS_DATASET} logger_provider: # logs processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` For the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` throughout the file. #### Run Your Application Point the agent at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml java -javaagent:opentelemetry-javaagent.jar -jar /path/to/myapp.jar ``` In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. This Quick Start uses ASP.NET Core. #### Acquire Dependencies Install the OpenTelemetry .NET packages. For example, with the .NET CLI, use: ```shell theme={} dotnet add package OpenTelemetry dotnet add package OpenTelemetry.Extensions.Hosting dotnet add package OpenTelemetry.Instrumentation.AspNetCore dotnet add package OpenTelemetry.Instrumentation.Http ``` #### Initialize Initialize the TracerProvider during application setup. ```csharp theme={} services.AddOpenTelemetry().WithTracing(builder => builder .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddOtlpExporter()); ``` #### Configure Use environment variables to configure the OpenTelemetry SDK: ```shell theme={} export OTEL_SERVICE_NAME="your-service-name" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443" # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu1.honeycomb.io:443" # EU instance export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=" ``` | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_SERVICE_NAME` | Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The data format that the SDK uses to send telemetry to Honeycomb. For more on data format configuration options, read [Choosing between gRPC and HTTP](/send-data/dotnet/#choosing-between-grpc-and-http). | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Honeycomb endpoint to which you want to send your data. | | `OTEL_EXPORTER_OTLP_HEADERS` | Adds your Honeycomb API Key to the exported telemetry headers for authorization. [Learn how to find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys). | #### Run Run your application. You will see the incoming requests and outgoing HTTP calls generate traces. ```shell theme={} dotnet run ``` In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. #### Acquire Dependencies Install OpenTelemetry Go packages: ```shell theme={} go get \ go.opentelemetry.io/contrib/otelconf/x \ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp ``` #### Initialize Prepare your application to send spans to Honeycomb. Open or create a file called `main.go`: ```go theme={} package main import ( "context" "fmt" "log" "net/http" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" otelconf "go.opentelemetry.io/contrib/otelconf/x" "go.opentelemetry.io/otel" ) // Implement an HTTP Handler function to be instrumented func httpHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello, World") } func main() { // Use otelconf/x to set up the OpenTelemetry SDK. The `/x` package // adds support for experimental spec fields such as resource detectors. sdk, err := otelconf.NewSDK() if err != nil { log.Fatalf("error setting up OTel SDK - %e", err) } defer sdk.Shutdown(context.Background()) otel.SetTracerProvider(sdk.TracerProvider()) otel.SetTextMapPropagator(sdk.Propagator()) // Initialize HTTP handler instrumentation handler := http.HandlerFunc(httpHandler) wrappedHandler := otelhttp.NewHandler(handler, "hello") http.Handle("/hello", wrappedHandler) // Serve HTTP server log.Fatal(http.ListenAndServe(":3030", nil)) } ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces # Use the endpoint below for EU # endpoint: https://api.eu1.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Set the following environment variables before running your application: | Environment Variable | Value | | :------------------- | :----------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | When `OTEL_CONFIG_FILE` is set, the configuration file is the single source of truth for the SDK. Other `OTEL_*` environment variables are ignored by design, so set all SDK options in the YAML file. You can still reference environment variables from inside the YAML using `${VAR_NAME}` substitution. The OpenTelemetry declarative configuration is stable at the specification level. Individual fields still under active development are marked with a `/development` suffix in the YAML (see [configuration versioning](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/VERSIONING.md#experimental-features)). Check the [language support status](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/language-support-status.md) for per-SDK maturity. Add `meter_provider` and `logger_provider` sections to the same file to export metrics and logs. This version also enables resource detectors, which add attributes such as `host.*` and `process.*` automatically: ```yaml theme={} file_format: "1.1" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} detection/development: detectors: - host: - container: - process: - service: tracer_provider: # traces processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/traces headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # metrics readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/metrics headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} # Legacy metrics only; omit with the current metrics experience: # - name: x-honeycomb-dataset # value: ${HONEYCOMB_METRICS_DATASET} logger_provider: # logs processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io/v1/logs headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` For the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` throughout the file. #### Run Your Application Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml go run YOUR_APPLICATION_NAME.go ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. #### Acquire Dependencies Add these gems to your Gemfile: ```ruby theme={} gem 'opentelemetry-sdk' gem 'opentelemetry-exporter-otlp' gem 'opentelemetry-instrumentation-all' ``` | Gem | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `opentelemetry-sdk` | Required to create spans | | `opentelemetry-exporter-otlp` | An exporter to send data in the OTLP format | | `opentelemetry-instrumentation-all` | A meta package that provides instrumentation for Rails, Sinatra, several HTTP libraries, [and more](https://github.com/open-telemetry/opentelemetry-ruby#instrumentation-libraries) | Install the gems using your terminal: ```shell theme={} bundle install ``` #### Initialize Initialize OpenTelemetry early in your application lifecycle. For Rails applications, we recommend that you use a Rails initializer. For other Ruby services, initialize as early as possible in the startup process. ```ruby theme={} # config/initializers/opentelemetry.rb require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' OpenTelemetry::SDK.configure do |c| c.use_all() # enables all instrumentation! end ``` #### Configure the OpenTelemetry SDK Use environment variables to configure OpenTelemetry to send events to Honeycomb: ```ruby theme={} export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io" # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu1.honeycomb.io" # EU instance export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key" export OTEL_SERVICE_NAME="your-service-name" ``` | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Base endpoint to which you want to send your telemetry data. | | `OTEL_EXPORTER_OTLP_HEADERS` | List of headers to apply to all outgoing telemetry data. Place your API Key generated in Honeycomb in the `x-honeycomb-team` header. [Learn how to find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys). | | `OTEL_SERVICE_NAME` | Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. | #### Run Your Application Run your Ruby application. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. If your preferred language is not covered here, you can find relevant instrumentation information in the [OpenTelemetry community documentation](https://opentelemetry.io/docs/languages/). For Rust, we recommend you use the [opentelemetry](https://crates.io/crates/opentelemetry) and [opentelemetry-otlp](https://crates.io/crates/opentelemetry-otlp) crates to send data to Honeycomb over OTLP. For any required configuration values, see [Using the Honeycomb OpenTelemetry Endpoint](/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint). ### Generate Automated Data Now that you have added automatic instrumentation to your application and have it running in your development environment, interact with your application by making a few requests. Making requests to your service will generate telemetry data and send it to Honeycomb where it will appear in the Honeycomb UI within seconds. ### Add Custom Instrumentation for LLMs With OpenTelemetry's automatic instrumentation now installed in your application, trace data is being sent to Honeycomb. The next step is to add custom instrumentation that tracks all relevant information related to your LLM feature. In OpenTelemetry, custom instrumentation is called manual instrumentation. To get the most out of your traces, you must use OpenTelemetry APIs to instrument. To add custom instrumentation, create a single span that tracks all relevant information related to your LLM feature. Specifically, the minimum information to track must include: * User ID * Prompt version * User input * Full prompt text * Full LLM response * Any error including parsing JSON and/or validating it * Error message * Token count #### Examples The following code examples show how to capture the correct information on an OpenTelemetry span: ```ts Node.js theme={} import { trace, Span, SpanStatusCode } from "@opentelemetry/api"; const tracer = trace.getTracer("llm.tracer"); function getJsonFromText( userInput: string, userId: string, promptTemplate: string, promptVersion: string ) { return tracer.startActiveSpan("app.get_json_from_text", (span: Span) => { span.setAttribute("app.user_id", userId); span.setAttribute("app.llm.prompt_version", promptVersion); span.setAttribute("app.llm.user_input", userInput); try { // Programmatically build the full prompt. // The output is the entire prompt you'd send to the LLM, // after RAG or any other context-building operations. const fullPrompt = buildFullPrompt(promptTemplate, userInput); span.setAttribute("app.llm.prompt_text", fullPrompt); // Call the LLM and get back the text of the result // and the number of tokens used. const { response, tokenCount } = callLLM(full_prompt); span.setAttribute("app.llm.response", response); span.setAttribute("app.llm.token_count", tokenCount); // Parse the JSON object and validate it, // capturing any errors you might encounter. const result = parseAndValidateResponse(response); return result; } catch (ex) { // Track any unexpected errors. span.setStatus({ code: SpanStatusCode.ERROR }); span.recordException(ex); } finally { span.end(); } }); } function parseAndValidateResponse(llmResult: string) { const currentSpan = trace.getActiveSpan(); // Extract and parse the JSON object from the LLM response. const { extracted, result, extractionError } = extractAndParseJson(llmResult); if (!extracted) { currentSpan.setAttribute("error.message", extractionError); currentSpan.setStatus({ code: SpanStatusCode.ERROR }); return null; } // Validate the structure of the result, capturing // any validation errors you might encounter. const { validated, validationError } = validateResult(result); if (!validated) { currentSpan.setAttribute("error.message", validationError); } return result; } ``` ```python Python theme={} from opentelemetry import trace from opentelemetry.trace import Status, StatusCode tracer = trace.get_tracer("llm.tracer") def get_json_from_text(user_input, user_id, prompt_template, prompt_version): with tracer.start_as_current_span("app.get_json_from_text") as span: span.set_attribute("app.user_id", user_id) span.set_attribute("app.llm.prompt_version", prompt_version) span.set_attribute("app.llm.user_input", user_input) try: # Programmatically build the full prompt. # The output is the entire prompt you'd send to the LLM, # after RAG or any other context-building operations. full_prompt = build_full_prompt(prompt_template, user_input) span.set_attribute("app.llm.prompt_text", full_prompt) # Call the LLM and get back the text of the result # and the number of tokens used. llm_result, token_count = call_llm(full_prompt) span.set_attribute("app.llm.response", response) span.set_attribute("app.llm.token_count", token_count) # Parse the JSON object and validate it, # capturing any errors you might encounter. result = parse_and_validate_response(llm_result) return result except Exception as ex: # Track any unexpected errors. current_span.set_status(Status(StatusCode.ERROR)) current_span.record_exception(ex) def parse_and_validate_response(llm_result): current_span = trace.get_current_span() # Extract and parse the JSON object from the LLM response. extracted, result, extraction_error = extract_and_parse_json(llm_result) if not extracted: current_span.set_attribute("error.message", extraction_error) current_span.set_status(Status(StatusCode.ERROR)) return None # Validate the structure of the result, capturing # any validation errors you might encounter. validated, validation_error = validate_result(result) if not validated: current_span.set_attribute("error.message", validation_error) return result ``` ## Explore Your Data With your app running and telemetry being sent to Honeycomb, it's time to explore your data. ### Create a Board For quick reference over time, you should create a [Board](/observe/boards/) to show LLM-specific queries of interest. We recommend creating a Board first before [trying queries](#create-queries), so you can save with ease later. To create a Board: 1. In the Honeycomb UI's left navigation menu, select **Boards** ()n. When the left navigation menu is compact, only the icon appears. 2. Select **New Board**. 3. In the modal that appears, name your new board, such as "LLM Dashboard." Optionally, give your Board a description to help others find and use it. Determine the board's Sharing setting - **Public to the Team** or **Limited to Collaborators**. 4. Select **Create** to finish. Your new board appears next. #### Next Steps 1. Select **Add Query** to go to the Query Builder display. 2. Use the example queries in the next section to populate your LLM Board. 3. Follow the directions to [add queries to an existing Board](/observe/boards/#adding-queries). ### Create Queries Now it's time to create your first queries for LLMs! Use the query examples below to explore the performance and behavior of your LLM application. The specific attributes should exist in your data and environment if you added [custom instrumentation for LLMs](#add-custom-instrumentation-for-llms) in the previous step. Enter each example query using the [Query Builder](/investigate/query/build/). These example queries use two to three of the `VISUALIZE`, `WHERE`, and `GROUP BY` clauses, located at the top of the Query Builder. * `VISUALIZE` - Performs a calculation and displays a corresponding graph over time. Most `VISUALIZE` queries return a line graph while the `HEATMAP` visualization shows the distribution of data over time * `WHERE` - Filters based on attribute parameter(s) * `GROUP BY` - Groups fields by attribute parameter(s) Screenshot of Visualize, Where, and Group by clauses in Query Builder

Screenshot of Visualize, Where, and Group by clauses in Query Builder

#### Track Overall Latency This query tracks overall latency of all LLM-related operations and the slowest requests. | VISUALIZE | WHERE | | ---------------------------------------------- | -------------------------------- | | HEATMAP(duration\_ms)
MAX(duration\_ms) | name = app.get\_json\_from\_text | Use to identify any spikes in latency, or if latency is increasing over time. In the event of a spike, you can investigate what happened by using [BubbleUp](/investigate/analyze/identify-outliers/) to find outliers. #### Track Invalid JSON Objects This query shows each instance that a user input led to a bad JSON object, whether that was because of a parsing error or a validation error. | VISUALIZE | WHERE | GROUP BY | | --------- | ---------------------------------------------------- | ---------------------------------------------------------- | | COUNT | name = app.get\_json\_from\_text
error exists | app.llm.input
app.llm.response
error.message | Use to identify exactly which inputs lead to bad behavior, which makes it easier to identify specific bugs to solve. #### Track all User Inputs Grouped by Response and Errors This query shows groups of all inputs and LLM outputs that succeeded. | VISUALIZE | WHERE | GROUP BY | | --------- | ------------------------------------------------------------- | ------------------------------------------- | | COUNT | name = app.get\_json\_from\_text
error does-not-exists | app.llm.user\_input
app.llm.response | Use to understand general user behavior, and to identify any patterns in user input that leads to a particularly useful response. It's just as helpful to understand **what's working** as it is to understand what isn't. #### Show Token Usage Over Time This query tracks token use over time, grouped by user ID. | VISUALIZE | WHERE | GROUP BY | | ----------------------------- | -------------------------------- | ------------ | | HEATMAP(app.llm.token\_count) | name = app.get\_json\_from\_text | app.user\_id | Use to understand how many tokens being used over time, but also to identify that usage down to specific users, as often a small number of users is responsible for the majority of usage. ### Investigate Specific Traces The queries on our LLM Board act as a starting point. If curious about specific behavior, you can view a specific trace that represents one request. Select any point on a graph, and in the menu that appears, select **View trace**. The next screen displays a [trace detail view](/reference/honeycomb-ui/query/trace-waterfall/) that lets you see what happened step by step. ## Next Steps The queries on our LLM board act as a referenceable starting point. If you're curious about specific behavior(s), start with any query and: 1. Add additional fields in the **GROUP BY** clause to slice your data into revealing interesting field values. 2. Use [BubbleUp](/investigate/analyze/identify-outliers/) to find outlier behavior and identify its contributing characteristics. 3. Select a specific trace that represents one request to see what happened step-by-step. # Troubleshoot: Overview Source: https://docs.honeycomb.io/troubleshoot Find solutions to common issues, get customer support, track recommended migrations, and stay up to date on product lifecycle and release stages. Honeycomb provides resources to help you find solutions when troubleshooting your Honeycomb implementation, and recommended migrations to ensure you get the most out of your investment. ## Get Support Explore the support resources available to you when you are experiencing a challenge with Honeycomb. Join Pollinators, Honeycomb's extended community, to expand your network, find opportunities for speaking and writing about observability, and connect on using Honeycomb in the workplace. Pollinators get help with Honeycomb from our staff and other customers, and regularly are given the opportunity to provide feedback to inform new features. ## Common Issues Explore common issues with sending data to Honeycomb. Explore common issues with data ingest in Honeycomb. Explore common issues with Query Builder, Query Assistant, and Calculated Fields. Explore common issues with visualization, including Honeycomb Home, Board Templates, and Service Map. Explore common issues when investigating and finding outliers with BubbleUp. Explore common issues with configuring Honeycomb, especially when configuring team access through Google, Okta SAML, and Microsoft Entra ID SAML SSO. Explore common issues with sampling and Refinery. Explore common issues with integrations. ## Honeycomb Product Lifecycle Understand how Honeycomb stages, releases, and retires product functionality. Identify the product versions that Honeycomb supports. Explore available beta and early access experiences that you can participate in. Understand Honeycomb's migration process and learn about what you can expect from us to support you during your migration. Identify migrations that Honeycomb recommends, understand their impact, and access resources to help you migrate. # Common Issues with Honeycomb Alerts Source: https://docs.honeycomb.io/troubleshoot/common-issues/alerts Troubleshoot common problems with Honeycomb SLOs, Triggers, and Anomaly Detection, including misconfigured thresholds, missing notifications, and burn alert behavior. If you need to troubleshoot alerts issues in Honeycomb, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## General Troubleshoot general alert issues. ### Relational fields do not show up when working with SLOs or Triggers We do not support relational fields in SLOs, Triggers, or Calculated Fields. To emulate the functionality of relational fields, propagate the data you would like to see on to the spans so you can filter on them. For example, if you have data on a root span that you would like to be available to a child span, propagate that data to the child spans. If you are using OpenTelemetry, you can enable field propagation from root spans to child spans by using OpenTelemetry concept of baggage. For an example using the OpenTelemetry for JavaScript SDK, visit [OpenTelemetry Javascript SDK: Add Multi-Span Attributes](/send-data/javascript-nodejs/opentelemetry-sdk/#add-multi-span-attributes). To see examples for other programming languages, [check out our documentation for other OpenTelemetry SDKs](/send-data/opentelemetry/#instrumenting-with-opentelemetry). ## SLOs Troubleshoot alert issues related to [Service Level Objectives (SLOs)](/notify/slos/). ### Exhaustion Time Burn Alert Because an incident can dramatically deplete your SLO budget, an [Exhaustion Time burn alert](/notify/slos/monitor/) may take a long time to resolve even after addressing the incident and deploying a fix. It takes time for this data to age out and recovery to occur. This means that an Exhaustion Time burn alert will **remain in a fired state** after triggering. If your budget stabilizes, and then starts burning again without ever going back above zero percent (`0%`), you will not be alerted a second time. If this scenario is not ideal, you can do one of the following: * [Configure a Budget Rate Alert](/notify/slos/monitor/#budget-rate-burn-alerts): Budget Rate Alerts fire regardless of your current budget. * [Reset your SLO Budget](/reference/honeycomb-ui/slos/slo-detail-view/#reset-your-remaining-slo-budget). This will erase past failed events in your SLO window, which may impact your charts. ## Anomaly Detection Troubleshoot issues related to [Anomaly Detection](/notify/anomaly-detection/). ### Monitoring status #### A service shows as Ineligible Honeycomb bases eligibility on a rolling data coverage window. A service that recently changed traffic patterns, or that sends data intermittently, can drop below the coverage Honeycomb needs for a reliable baseline. Monitoring resumes automatically once the service's data coverage recovers; you don't need to re-enable it. To check why a service is ineligible: 1. Select **Anomalies** () from the navigation menu. 2. Select the service. 3. Review the coverage stat next to the service's status, which shows its current data coverage against the threshold Honeycomb requires. #### A service stays in Onboarding longer than expected Onboarding reflects how much historical data Honeycomb has accumulated for the service so far. A service with sparse or irregular data takes longer to build a reliable baseline than one with continuous traffic. To check a service's onboarding progress: 1. Select **Anomalies** () from the navigation menu. 2. Select the service. 3. Select the **Error Rate** or **Presence** view to explore its current state and how much data Honeycomb has collected so far. #### Anomaly Detection flags a normal, expected traffic pattern The baseline assumes relatively stable traffic. A service with a strong daily or weekly cycle, such as a predictable weekday spike or weekend drop, can trigger false positives, since the current algorithm doesn't yet model seasonality. To reduce false positives until seasonality support ships, lower the service's sensitivity threshold: 1. Select **Anomalies** () from the navigation menu. 2. Select the service. 3. Select the **Error Rate** view. 4. Select the sensitivity dropdown and choose **Low**. 5. Select **Save**. ### Notifications #### I am not getting notified about an anomaly On the service's detail page, check that: * The signal is enabled * At least one recipient is configured * The recipient isn't muted Honeycomb still shows the anomaly in the Anomalies list and on the service's detail page even when no recipient is configured. #### A signal didn't resume monitoring after a service became eligible again Honeycomb preserves your on/off intent for each signal. If you turned a signal off before the service became ineligible, Honeycomb keeps it off rather than resuming it automatically. To resume monitoring: 1. Select **Anomalies** () from the navigation menu. 2. Select the service. 3. Locate the **Enable** toggle and turn monitoring back on. # Common Issues with Configuring Honeycomb Source: https://docs.honeycomb.io/troubleshoot/common-issues/configuring-honeycomb Troubleshoot common problems with configuring Honeycomb Environments, Datasets, and Teams, including access issues, missing data, and SSO configuration errors. If you need to troubleshoot configuring your Honeycomb Environments and Teams, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## Environments Troubleshoot issues related to [managing Environments](/configure/environments/manage/). ### When you try to delete an Environment, the delete button is disabled Check that you have typed in the correct unique identifier of the environment you want to delete. It should be listed in parentheses in the prompt for the field. ## API Keys Troubleshoot issues related to [API Keys](/configure/environments/manage-api-keys/). ### You cannot see the value of your API key For security reasons, the full, 64-character API key is only available when you first create the API key. Only the non-sensitive first 32 characters of the API Key (a region-specific prefix and the key's identifier) are available in the Honeycomb UI thereafter. If you have misplaced the full API key, we recommend that you disable or delete the key, and create a new one. ## Team Access Troubleshoot issues related to [configuring access for Teams](/configure/teams/configure-access/). ### Google SSO Troubleshoot team member authentication access that uses [Google Single Sign-on (SSO)](/configure/teams/configure-access/sso-google/). #### Team members encounter a message stating that no user accounts match their credentials When attempting to log in to Honeycomb using Google SSO, users may encounter a message stating that no accounts match their user credentials: Honeycomb login screen with message stating that no accounts match the entered user credentials

Honeycomb login screen with message stating that no accounts match the entered user credentials

This message appears when the user's team has configured Google SSO authentication and the user already has a Honeycomb account associated with email address/password as credentials. To solve this issue, the user must [link their Honeycomb account to their Google account](/configure/teams/configure-access/sso-google/#enabling-google-sso-for-an-existing-honeycomb-account). After they have linked their Honeycomb and Google accounts, they can [log in to Honeycomb using their Google account](/configure/teams/configure-access/sso-google/#logging-in-to-honeycomb-using-your-google-account). ### SAML SSO Troubleshoot team member authentication access that uses [SAML Single Sign-on (SSO)](/configure/teams/configure-access/sso-okta-saml/). #### You receive a SAML assertion error when converting to a SAML SSO team in Honeycomb If a "SAML Assertion" error appears, verify that your SSO Identity Provider **Audience** and **Recipient** fields match your Honeycomb team settings SSO **Service Provider Issuer/Entity ID** and **Service Provider ACS URL** fields. The **Audience** field should contain: `https://ui.honeycomb.io/saml/`, while the **Recipient** field should contain: `https://ui.honeycomb.io/auth/callback/saml/`. #### You receive a SAML assertion error when updating your SAML certificate If you receive a "SAML Assertion" error while updating to a new Service Provider certificate, your Identity Provider may still be configured with the older certificate. Follow the certificate rotation process to properly update your Identity Provider before updating your team's configuration. For detailed steps, see [SAML Certificate Rotation](/configure/teams/configure-access/saml-certificate-rotation/). ### Microsoft Entra ID SSO Troubleshoot team member authentication access that uses [Microsoft Entra ID Single Sign-on (SSO)](/configure/teams/configure-access/sso-microsoft-entra-id-saml/). Honeycomb requires the `FirstName`, `LastName`, and `Email` claims to be populated. If your Microsoft Entra accounts are configured to use a value other than `user.givenname` or `user.surname` to represent a user's first or last name, you may use those other values to populate the corresponding claims instead. # Common Issues with Data in Honeycomb Source: https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb Troubleshoot common data ingest problems in Honeycomb, including missing events, unexpected field values, and data that does not appear as expected. If you need to troubleshoot data ingest issues in Honeycomb, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## Trace Waterfall Display Troubleshoot data ingest issues related to the [trace waterfall display](/investigate/analyze/explore-traces/). ### Child spans appear to start before parent spans When collapsing the trace waterfall tree, it can be difficult to see the extent of the child spans of a parent. For example, an async process might have a parent that returns in a few milliseconds -- but it starts a database query that runs for several minutes. Honeycomb draws a thin line to the right of the parent span to show the duration of the child process. For example, in this figure, the span with `name` `head.startQuery` runs 2.533 ms -- but it has children that run much longer. (The three subprocesses called `FetchPartial` all run over 2.5 seconds). Async spans

Clock skew, temporal anomalies, or time travel can cause a child to start **before** the parent span. Honeycomb renders these with similar lines to the left of the main body. To learn more about how to resolve this issue, visit our [Troubleshooting Traces documentation](/troubleshoot/common-issues/data-in-honeycomb/#spans-in-a-trace-appear-out-of-order-or-have-strange-time-offsets). ### Your very large trace displays only 32,000 spans Honeycomb displays traces with a maximum of 32,000 spans. If you load a trace larger than 32,000 spans into the trace viewer, Honeycomb will display the 32,000 spans closest to the root, using a breadth-first search, and will display a warning message. Honeycomb will omit the remainder of the spans. In addition, Honeycomb will display placeholder spans showing where the omitted spans would go. To work with very large traces, use the `CONCURRENCY` aggregate to visualize the structure of the trace, as seen in the [Trace Summary](/investigate/query/build/visualize-concurrency/#trace-summary) example. ### Traces have a missing root span or missing spans Sometimes, a trace has some spans that claim to have a `trace.parent_id`--but no span claims to have that `trace.span_id`. Honeycomb also expects one span to have no parent; that is the root span. If there is no root span, then this is understood as a **missing root**; in that case, the root span is drawn as missing. In the trace view, a message about missing spans or a missing root span may appear. Alternatively, Honeycomb renders a missing span in that slot: a placeholder that shows the span with all the children that point to that parent. Instead of a span, "(missing)" may appear in a display. This section explains the concepts of missing root spans and missing spans, and how to troubleshoot these issues.

#### Missing Root Span A trace is made of many spans, all with the same **Trace ID**. They connect to each other with their **Span ID** and **Parent Span ID** fields. Every trace has one span that represents the trace as a whole, or a **root span**. This root span has no Parent Span, or Parent Span ID, because it is the root of the whole trace. When Honeycomb finds a span without a trace ID and a parent span ID, then the root span is missing. This is disruptive because it is common to query for overall response times, error rates, and other properties of the customer experience by looking only at root spans. When the root span is missing, queries that include the filter `is_root` does not count the associated trace at all. For this reason, "missing root span" is the most troubling case of missing spans. #### Missing Spans If you rehydrated data in this trace from your S3 archive (available with the Enhance Enterprise add-on) and spans are still missing, you may need to rehydrate additional data. To learn more, visit [Enhancing Traces with Missing Spans](/investigate/query/archive/#enhancing-traces-with-missing-spans). In the trace view, Honeycomb can sometimes detect that not all of the spans describing a trace appear in the query results. Here is how Honeycomb can tell this: A trace is made of many spans, all with the same **Trace ID**. They connect to each other with their **Span ID** and **Parent Span ID** fields. For example, examine these three spans: | Trace ID | Span ID | Parent Span ID | Name | | -------------------------------- | ---------------- | ---------------- | ----------- | | 0123456789abcdef0123456789abcdef | aaaaaaaaaaaaaaaa | | root span | | 0123456789abcdef0123456789abcdef | bbbbbbbbbbbbbbbb | aaaaaaaaaaaaaaaa | second span | | 0123456789abcdef0123456789abcdef | cccccccccccccccc | bbbbbbbbbbbbbbbb | third span | When Honeycomb assembles the trace, it connects each span under its parent span, using the values in Span ID and Parent Span ID. Sometimes, the Parent Span does not exist. In this situation, the data Honeycomb retrieves may look similar to this example, which duplicates the earlier example without the named second span: | Trace ID | Span ID | Parent Span ID | Name | | -------------------------------- | ---------------- | ---------------- | ---------- | | 0123456789abcdef0123456789abcdef | aaaaaaaaaaaaaaaa | | root span | | 0123456789abcdef0123456789abcdef | cccccccccccccccc | bbbbbbbbbbbbbbbb | third span | In this case, Honeycomb finds the third span's parent span ID of `bbbbbbbbbbbbbbbb`, but no span with span ID `bbbbbbbbbbbbbbbb`. This is a missing span situation. Therefore, Honeycomb makes a guess, and in the trace view, places two rows underneath the trace's root span: a row labeled "(missing)" and then a row for the named third span. To view a trace in table form similar to above, within your [Query Results](/reference/honeycomb-ui/query/query-results/), either: * Select a Trace ID to view its [Trace Waterfall view](/reference/honeycomb-ui/query/trace-waterfall/#waterfall-representation) and use the Fields button to modify the displayed fields. * Navigate to the Explore Data tab and [modify the Fields list](/reference/honeycomb-ui/query/query-results/#fields-list) with similar field names based on your data. #### Why Do I Have Missing Spans? Spans have a long journey from your application to your screen, and they can go missing at any stage. This section starts with an overview, and then goes through several possibilities, starting with the easiest to fix. ##### Possible Areas to Troubleshoot Spans may be lost at any step, as labeled in the diagram below: * in the service (E, C, S) * in the pipeline (C or S) * at ingest or arrival to Honeycomb (I) * in the Honeycomb query results (Q) Spans may be lost at any step: in the service (E, C, S) or in the pipeline (C, S) or arriving at Honeycomb (I) or in the query results (Q)

Spans may be lost at any step: in the service (E, C, S) or in the pipeline (C, S) or arriving at Honeycomb (I) or in the query results (Q)

where each letter represents a step's potential component: * E for [Exporter](#applications-exporter-drops-spans) * C for [Configuration](#services-send-spans-to-wrong-location) * S for [Sampling](#sampling-causes-missing-spans) * I for [Ingest](#spans-rejected-at-ingest-or-arrival-to-honeycomb) * Q for [Query results](#query-results-do-not-include-all-spans) Spans are emitted by every application that handles a request. Typically, spans go through a pipeline inside your network, which may include an [OpenTelemetry Collector](/send-data/opentelemetry/collector/). From there, the spans cross the internet to Honeycomb's ingestion endpoint. Once accepted, they are stored in Honeycomb's database. Then, the Honeycomb UI retrieves the spans with a query to that database. At every step in this process, at least one way exists for spans to be lost. Let's begin at the last point in the process, because it is the easiest to fix. ##### Query Results Do Not Include All Spans The trace view is backed by a query for all spans with a particular trace ID within a time range. Every Honeycomb query includes a time range, including the trace view. If any spans have a starting timestamp outside that time range, then the database would not return them, and these spans will be missing in the Honeycomb UI. First, if a restrictive time range is the cause, determine the timespan that the query uses. How you reached the trace view determines the underlying time range for the trace view. * If accessing the trace from **Recent Traces** tab in [Home](/observe/honeycomb-home/), the default timespan is the last two hours. * If accessing the trace from a [direct trace link](/investigate/collaborate/share-trace/), then the timestamp is specified in the query parameters or defaults to the last two hours. It is possible that the missing spans started before this determined time span begins. Then, try constructing a [direct trace link](/investigate/collaborate/share-trace/) with a larger range of specified start and end times. Alternatively, test for "spans are in Honeycomb, but not in this time range." Sometimes a clock's time is incorrect or timestamps are in a misunderstood format, and spans arrive with a timestamp in the distant past. In [Query Builder](/investigate/query/build/): 1. Use the WHERE clause to filter for your trace ID and enter `trace.trace_id = your_trace_id_here` 2. Use the time picker to modify the selected timespan and set a custom timespan to the year 1969, since "1970-01-01" is the earliest timestamp Honeycomb can hold. ##### Spans Rejected at Ingest or Arrival to Honeycomb Honeycomb's ingestion endpoint will **rate limit**, or reject too many events in a second, and it will **throttle**, or discard 9 out of 10 spans after receiving too many spans for several days. If your account is throttled, all traces will have missing spans. To troubleshoot, refer to the [Usage Center](/get-started/manage-costs/how-honeycomb-calculates-usage/) to find out whether these conditions apply to your situation. Resolve the rate limit or throttle conditions to solve for missing spans. ##### Sampling Causes Missing Spans [Sampling](/manage-data-volume/sample/guidelines/) reduces your total volume of data while ensuring a statistically representative set of data exists to analyze. We recommend sampling as the main method to control the amount of data sent to Honeycomb. Sampling should always be done per trace, so that Honeycomb receives either all of the spans in a trace, or none of them. Consider the possibility of spans being dropped when sampling if sampling your trace data. It is possible that a misconfiguration can lead to child spans in a trace getting sampled, but the root span dropped. This results in Honeycomb interpreting that the root span is missing. It is also possible that a root span and most of its children are sampled, but spans within that trace are dropped. This kind of misconfiguration leads Honeycomb into thinking there are missing spans. Sampling can happen anywhere in the pipeline or in the application sending spans. Troubleshooting is specific to each tool and language. ##### Application's Exporter Drops Spans Libraries for telemetry, including OpenTelemetry instrumentation, are designed to drop data before they interfere with your application's function. There is a limit to how many spans they will hold in memory, how frequently they will transmit batches of spans, and how large a batch of spans they will send. If too many spans are created, some spans will be dropped. Some of these drops cause "(missing span)" gaps in traces. In this situation, their appearance will be common when the application is very busy, and rare when application traffic is slow. Some OpenTelemetry SDKs let you configure the batch size, batch frequency, and memory limits. Also consider removing some instrumentation. ##### Services Send Spans to Wrong Location If your traces are consistently missing root spans or other spans at a consistent place in each trace, then it is likely that those spans are not reaching Honeycomb. This may be because of configuration issues. To troubleshoot: 1. Check each service that you expect to participate in the trace. 2. Determine if any events arrive to Honeycomb with the expected service name. 3. Use the [Query Builder](/investigate/query/build/) to summarize available services with **GROUP BY** `service.name`, or filter for a specific service name with the **WHERE** operator. While `service.name` is a common standard format, refer to the set field for "Service name" in [Dataset Settings](/configure/datasets/definitions/) for the correct query format. 4. If no events arrive with the expected service name, then check: 1. That service's configuration 2. The configuration of any pipeline elements, such as the [OpenTelemetry Collector](/send-data/opentelemetry/collector/) or [Refinery](/manage-data-volume/sample/honeycomb-refinery/) 3. That the Honeycomb API key resolves to the correct environment. An [API call](/api/auth/) can confirm this. ##### Your System Continues Someone Else's Trace Missing root spans can happen when the trace began outside of your system. It is possible that external requests arrive with a **[trace context](https://www.w3.org/TR/trace-context/)** header, and your service picks that up and continues the trace. In this case, no data is missing. Instead, what should be your root span is referencing a span that you never should receive. Whenever your system receives a request from outside, it should start a new trace. Resolve this issue by configuring your gateway or ingest layer to strip trace context headers when a request arrives from a system you do not own. ##### You Are Using AWS Lambda If you are using AWS Lambda with API Gateway (or another service that governs traffic), missing root spans for your traces in Honeycomb are likely. This is because API Gateway often generates the initial request but may not propagate tracing headers correctly to your Lambda function. Check out our dedicated troubleshooting section for the AWS-managed OpenTelemetry Lambda layer to find solutions for: * [Missing root spans in traces](/send-data/aws/lambda/opentelemetry/#missing-root-spans-in-traces) * [Missing other spans in traces](/send-data/aws/lambda/opentelemetry/#missing-non-root-spans-in-traces) ### Honeycomb is not recognizing my Traces Depending on your configuration, field names may be slightly different. In addition, you can manually select columns as tracing fields, as explained in [changing the trace schema](/configure/datasets/definitions/). | Standard Tracing Fields | Legacy Zipkin Fields | | ----------------------- | -------------------- | | trace.span\_id | id | | trace.trace\_id | traceId | | trace.parent\_id | parentId | | duration\_ms | durationMs | | service.name | serviceName | | name | name | When Honeycomb is not recognizing your traces, it is typically caused by a few common pitfalls. First, to verify that Honeycomb is correctly detecting your trace, check that the `trace.trace_id` field is hyperlinked. To do this, select `trace.trace_id` in the GROUP BY window and COUNT in the VISUALIZE window. If the **traceID** field looks like it does below - as a quoted string - Honeycomb is not properly recognizing your traces. Screenshot of the trace view where the trace.trace_id field appears as a quoted string

Screenshot of the trace view where the trace.trace_id field appears as a quoted string

#### Properly Use the Column `trace.parent_id` Honeycomb recognizes a dataset as a tracing dataset by the existence of the column `trace.parent_id`. You must send at least one event that has the column `trace.parent_id` in it or Honeycomb will not recognize the dataset as a tracing dataset and will not show traces. But, remember that root spans should not have the column `trace.parent_id` in order to be recognized as root spans. This likely raises a question for you: what if I am sending a traces that only have a single span? As root spans, they are not allowed to have a `trace.parent_id` column if they are to be detected properly, so how will my dataset get a trace.parent\_id column? If this is your situation (and it very well may be if you are experimenting with tracing), just send in any event with a column trace.parent\_id. All you need is one event to be sent into Honeycomb with `trace.parent_id` as a field and the column will be created. Then, Honeycomb will properly recognize your dataset as a tracing dataset. Alternately, by [manually changing the trace schema](/configure/datasets/definitions/), you can ensure that the trace schema is recognized. #### Verify Field Formats Every trace event must have a defined trace identifier (`trace.trace_id`), a span identifier (`trace.span_id`), and a duration (`duration_ms`). The `trace.trace_id` and `trace.span_id` must both be strings; the duration must be a number, measuring milliseconds. ### Spans in a Trace appear out of order or have strange time offsets If the tracing view appears to show out of order spans, then there are gaps in the layout of the spans or the root span is missing. There are a number of reasons why spans in a trace may appear out of order or have strange time offsets: * Make sure you are providing the start time of the span When sending trace events into Honeycomb, you must include a `timestamp` field in the event that represents the start time of the span. When a `timestamp` field does not exist, Honeycomb sets the timestamp for the event to the date/time when the event was received by our collectors. * Verify that `trace.parent_id`, `trace.trace_id`, and `trace.span_id` are set properly The root span for any given trace must have no field for `trace.parent_id` in its event. If all of the spans in a trace have a `trace.parent_id`, Honeycomb will not show a root span for that trace. Honeycomb identifies which span is the root span by its absence of the column `trace.parent_id`. If all of your span events have a `trace.parent_id` column, then Honeycomb assumes the root span is missing. Ensure that your root spans have no `trace.parent_id` field, and validate that your child spans have their `trace.parent_id` and `id` set properly. * Verify that duration is set properly You must include the duration of the span, in milliseconds and in the field `durationMs`. Verify that this duration is being set properly. ### Honeycomb is not recognizing your services If you can not find your service in Honeycomb, check the following areas: * Verify that your API key has write permissions To create Service Datasets, your API key must have `sent events` and `create dataset` permissions. To learn more about API keys and their permissions, visit [API Key permissions](/configure/environments/manage-api-keys/#api-key-permissions). * Verify that your Team has not hit the Service Dataset maximum Every Environment has a Service Dataset limit. If you have reached this limit, check for any inaccurate Service Datasets, or join our [Pollinators Community Slack](/troubleshoot/community/#join-pollinators-community-slack) to ask questions and learn more. * Verify that your Event has defined a service Depending on your instrumentation approach, events with undefined services receive a default value. In OpenTelemetry and Beelines, it may start with `unknown_service`. You can query this dataset in Honeycomb to see if your missing events are in here. * Verify OTLP Protobuf definitions Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK's protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK. * Verify Endpoint URLs for OTLP/HTTP When using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable with an SDK and an HTTP exporter, the final path of the endpoint is modified by the SDK to represent the specific signal being sent. For example, when exporting trace data, the endpoint is updated to append `v1/traces`. When exporting metrics data, the endpoint is updated to append `v1/metrics`. So, if you were to set the `OTEL_EXPORTER_OTLP_ENDPOINT` to `https://api.honeycomb.io`, traces would be sent to `https://api.honeycomb.io/v1/traces` and metrics would be sent to `https://api.honeycomb.io/v1/metrics`. The same modification is not necessary for gRPC. ```shell theme={} export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.eu1.honeycomb.io # EU instance ``` If the desired outcome is to send data to a different endpoint depending on the signal, use `OTEL_EXPORTER_OTLP__ENDPOINT` instead of the more generic `OTEL_EXPORTER_OTLP_ENDPOINT`. When using a signal-specific environment variable, these paths must be appended manually. Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` for traces, appending the endpoint with `v1/traces`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` for metrics, appending the endpoint with `v1/metrics`. Send both traces and metrics to Honeycomb using this method by setting the following variables: ```shell theme={} export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces # US instance #export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.eu1.honeycomb.io/v1/traces # EU instance export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics # US instance #export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.eu1.honeycomb.io/v1/metrics # EU instance ``` More details about endpoints and signals can be found in the [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md). * Enable debug mode Most SDKs include an option to enable a debugging mode. For some SDKs, including the Honeycomb OpenTelemetry SDK Distributions, it may be enabled by setting the `DEBUG` environment variable to `true`: ```shell theme={} export DEBUG=true ``` Refer to the Troubleshooting sections for each language to find the proper configuration as needed. * Export to console Most SDKs include an option to export telemetry to the console to verify the data is instrumented as expected. For each signal supported by the SDK, a signal-specific exporter likely also exists: * For tracing, try a `ConsoleSpanExporter`. * For metrics, try a `ConsoleMetricExporter`. * For logs, try a `ConsoleLogExporter`. Refer to [OpenTelemetry documentation](https://opentelemetry.io/docs/languages/) or the Troubleshooting sections in Honeycomb documentation for each language to find the proper configuration as needed. * Choose between gRPC and HTTP Most OpenTelemetry SDKs have an option to export telemetry as OTLP either over gRPC or HTTP/protobuf, with some also offering HTTP/JSON. If you are trying to choose between gRPC and HTTP, keep in mind: * Some SDKs default to using gRPC, and it may be easiest to start with the default option. * Some firewall policies are not set up to handle gRPC and require using HTTP. * gRPC may improve performance, but its long-lived connections may cause problems with load balancing, especially when using Refinery. gRPC default export uses port 4317, whereas HTTP default export uses port 4318. # Common Issues with Integrations Source: https://docs.honeycomb.io/troubleshoot/common-issues/integrations Troubleshoot common problems with Honeycomb integrations, including AWS, Terraform, webhook delivery, and third-party tool connectivity issues If you need to troubleshoot your integrations with Honeycomb, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## PagerDuty Troubleshoot issues related to [PagerDuty](/notify/pagerduty/). ### You do not receive alerts from PagerDuty when a trigger fires If there are gaps in your escalation policy schedule, PagerDuty will not create an incident if no responder is on-call during the moment the trigger fires. Examine your PagerDuty escalation policy to ensure that a responder is always on-call in PagerDuty. # Common Issues when Investigating Source: https://docs.honeycomb.io/troubleshoot/common-issues/investigating Troubleshoot common problems with Honeycomb's investigation tools, including trace views, BubbleUp, Correlations, and Canvas not behaving as expected. If you need to troubleshoot tools used to investigate your data in Honeycomb, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## BubbleUp Troubleshoot issues related to [BubbleUp](/investigate/analyze/identify-outliers/). ### Your data is not ordered, grouped, or limited as you expect BubbleUp ignores the `GROUP BY`, `ORDER BY`, and `LIMIT` clauses in Query Builder. ### Your field is represented as a measure Fields stored as numbers, like `user_id`, are represented as measures when they are dimensions. In **Dataset Settings** > **[Schema](/reference/honeycomb-ui/manage-data/datasets/dataset-settings/)**, expand **Unique Fields** and adjust the field's data type to `string`. ### Your Calculated Field is represented as a dimension [Calculated Fields](/configure/environments/calculated-fields/) that extract a numeric value from text are represented as dimensions rather than measures. Use the `FLOAT()` function to coerce the regular expression in a Calculated Field, like `REG_VALUE('$field', '*([0-9]+).*')`, to a float. ### Your field value displays "(no value)" A field's value displays `(no value)` when the field is not populated in the data. Make sure the field is populated in the data you sent to Honeycomb. ### Your field value displays "(empty)" Most commonly, a field's value displays `(empty)` in `string` typed columns when a string is empty. ## Searching Query Results Troubleshoot issues related to the [Explore Data query results search](/reference/honeycomb-ui/query/query-results/#search-within-results). ### When you search query results, field names that don't match a search term are highlighted When you submit a search in the **Explore Data** view, Honeycomb searches each field, which consists of a name-value pair, for your search term. When a field value matches your search term, the field name will be highlighted, even if the field name does not match your search term. To learn more about the **Explore Data** search feature, visit our [UI Reference](/reference/honeycomb-ui/query/query-results/#search-within-results). # Common Issues with Queries Source: https://docs.honeycomb.io/troubleshoot/common-issues/queries Troubleshoot common problems with building queries, including Relational Fields, Query Assistant, and Calculated Fields, unexpected results, slow queries, and formula errors. If you need to troubleshoot querying in Honeycomb, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## Query Builder Troubleshoot issues related to [Query Builder](/investigate/query/build/). ### You don't know how to write a query that answers your question The Canvas agent can help write the query: 1. Select **Canvas** () from the navigation menu. 2. Tell the agent what you want to do. ### You receive a notice that fields were removed from your query When you select a new dataset, if the fields in the existing query do not exist in the new dataset, Honeycomb notifies you about the field(s) that were removed from your query. Screenshot illustrating the informational message

Screenshot illustrating the informational message

If the fields should be available, make sure the fields in the old dataset exist in the new dataset. ### Random items in the Explore Data table display and fields list are highlighted When you run a search in the **Explore Data** view, Honeycomb highlights relevant field names and values in the table display and fields list. To clear highlighted areas, cancel your search by selecting the close icon () in the search bar. To learn more about the **Explore Data** search feature, visit our [UI Reference](/reference/honeycomb-ui/query/query-results/#search-within-results). ## Relational fields Troubleshoot issues related to [Relational Fields](/investigate/query/build#relational-fields-2). These include `parent.`, `child.`, `root.` and `any[1-3].` prefixes. ### You cannot add a relational field to SELECT Honeycomb does not allow relational fields in the **SELECT** clause. You can get the same effect by swapping the relational field prefix and the primary span (the one without a relational field prefix) in your query and including the primary span in the **SELECT** clause instead. For example, if you want to aggregate data to find how many database queries exist on average for transaction X, you might want to write a query that looks at all of transaction X and calculates the SUM of the duration of the database query spans: `VISUALIZE SUM (child.duration_ms) WHERE is_root GROUP BY name`. Instead, try: `VISUALIZE SUM(duration_ms) WHERE name=db.query GROUP BY root.name`. ### Query Builder takes a long time to produce results Relational field queries can take a long time to run. Add more filters to your **WHERE** clause or reduce your time window. ### Query results are missing some traces Relational field queries may be missing some traces. Often, this is because you have many datasets within your environment and relational fields have a built-in joinable time range. When traces exceed the joinable time range for a relational field, they are excluded from the query results. ### Query results do not show missing spans With relational fields, Honeycomb only returns spans that have a matching span to join with. For example, if you want to find traces that are missing their root span, you might query with a filter of `root.name` `does-not-exist`. This query will only return traces that do have a root span, but the root span is missing a `name` field. If the root span is missing entirely, nothing will appear in the results. ### Query results are unexpected Relational field queries can return results that are not quite what you expect. * Make sure all column names defined in your dataset are distinct from the relational field prefixes. For example, you should avoid having a column defined in your dataset called `root.name` because this will collide with the `root.` relational field prefix, and Honeycomb will try to query the `name` column on the `root` span. * Make sure your instrumentation is not accidentally sending duplicate spans. When Honeycomb finds multiple matching spans for a given prefix, it will choose one to join and ignore the rest. If a trace has two root spans, only one will be joined for the `root.` prefix. If your supplied filters match multiple spans in the same trace and any fields are used in the **GROUP BY** clause, you may receive confusing results. Unexpected results can also come from how Honeycomb approaches querying: * If you are running a query that contains multiple `anyX.` (`any.`, `any2.`, `any3.`) relational fields, Honeycomb applies them all to the same span. * `anyX.` and `child.` relational fields find the first match randomly; they are not ordered as they search through a trace. * When grouping by relational fields, your **GROUP BY** clause can sometimes function as a filter. For example, if your query on `name` returns 100 results that are all root spans, then grouping by `parent.name` could return 0 results. * Because Query Builder's main purpose is to produce accurate aggregate numbers, we implement a strict single-match join rather than using the approach taken by traditional relational databases. Each aggregate will only count each primary span (the one without a relational field prefix) in your query once and will only join it with a single other span for each of the available prefixes. To illustrate how this works, imagine you have a trace with ten leaf spans and another with just one. You want to get the sum of root span durations for traces that contain leaf spans, so you query with `SUM(root.duration_ms) WHERE name=leaf`. An SQL-style implementation would count the first trace's root duration ten times. Suppressing duplicates to avoid this would immediately run into trouble if you wanted to add a **GROUP BY** clause. Instead, in Honeycomb, you would query with `SUM(duration_ms) WHERE trace.parent_id does-not-exist AND any.name=leaf`. ## Query Assistant Troubleshoot issues related to [Query Assistant](/investigate/query/build/#query-assistant). ### Query Assistant does not produce a query We do our best to produce queries for a broad collection of input, but we can't guarantee a query for all inputs. Try rephrasing your input. You can also use [Canvas](/investigate/canvas) to ask questions about your data in natural language and explore answers conversationally. ### Query Assistant takes a long time to produce a query Query Assistant relies on OpenAI's API, which usually responds in under 5 seconds. If you receive no response to your request after 10 seconds, compute resources for OpenAI's API are likely unavailable for your request. We recommend waiting for a few seconds and then trying your request again. ### Query Assistant returns different answers when you give it the same input Because Large Language Models (LLMs) are nondeterministic, we cannot guarantee that you will always receive the same query fo the same input, even though we try to be consistent. To ensure you have a consistent query to run, we recommend saving your query to a [Board](/observe/boards/) for later use. ## Calculated Fields Troubleshoot issues related to [Calculated Fields](/configure/environments/calculated-fields/). ### Relational fields do not show up when working with Calculated Fields We do not support relational fields in Calculated Fields, SLOs, or Triggers. To emulate the functionality of relational fields, propagate the data you would like to see on to the spans so you can filter on them. For example, if you have data on a root span that you would like to be available to a child span, propagate that data to the child spans. If you are using OpenTelemetry, you can enable field propagation from root spans to child spans by using OpenTelemetry concept of baggage. For an example using the OpenTelemetry for JavaScript SDK, visit [OpenTelemetry Javascript SDK: Add Multi-Span Attributes](/send-data/javascript-nodejs/opentelemetry-sdk/#add-multi-span-attributes). To see examples for other programming languages, [check out our documentation for other OpenTelemetry SDKs](/send-data/opentelemetry/#instrumenting-with-opentelemetry). ### A Calculated Field displays a null or unexpected value Null or unexpected values in Calculated Fields are usually related to: * Mismatched field types. Many functions are sensitive to the input's value type. Review the field types in the [schema](/reference/honeycomb-ui/manage-data/datasets/dataset-settings/) to ensure they align with the inputs of the functions. If necessary, coerce the fields with a [cast operator function](/reference/calculated-field-expression/cast). * Name collisions. If data from an event sent to Honeycomb uses the same name as a Calculated Field, the Calculated Field takes priority over the data sent to Honeycomb. We recommend that you rename the field in your instrumentation or rename the Calculated Field. ### A link points to the wrong location When a name collision occurs for a Calculated Field, it may cause confusing behavior in Honeycomb. For example, links that use a duplicate field name may point to the wrong location. Honeycomb requires that Calculated Fields names be unique across their containing Environment. Calculated Field names should not match the name of any other Calculated Field or any other field in any Dataset contained within the Environment. When you create a Calculated Field in Honeycomb, we try to check that you have entered a display name that is unique to your Environment (although duplicate names can still occur). If the Calculated Field exists before Honeycomb receives a field name with the same name, we allow both the pre-existing Calculated Field and the new field with the duplicate name to exist, and we display a warning message in the **Dataset Settings** section of the UI under the **Define Schema** view. To resolve this issue, check that your Calculated Field names are unique and not already in use by a field in any Dataset in your Environment. If you find a duplicate name, we recommend renaming the Calculated Field as soon as possible, making sure that the new name is unique to your Environment. If you unable to change the name of the Calculated Field, then change your instrumentation to rename the field instead. If you rename the field in your instrumentation, you must also delete it in Honeycomb. To do this, use the [Column Deletion API](/api/columns/update-a-column). This method will prevent the column from being queried and remove the data in question from new query results, but it will not remove permalinks to existing queries that may contain this data. If you are a Pro/Enterprise user, support can help you with deletion. To request help, visit [Honeycomb Support](/troubleshoot/customer-support/) to create a ticket for our support team, or email [support@honeycomb.io](mailto:support@honeycomb.io). ### Error: "Please specify at least one field using \$ syntax" when generating a Calculated Field with AI This error message means your entered Calculated Field prompt did not reference at least one field using the required `$` syntax. To resolve this issue, start each field with the `$` symbol, such as `$field_name`. When using natural language to create a Calculated Field in Honeycomb, you must explicitly reference schema fields in your Dataset or Environment using the `$` syntax. For example, within the prompt: * `Extract the directory after "/api" in $url_path` — `url_path` is the field name * `if $span.kind = "server" and $trace.parent_id does not exist, return whether $duration_ms < 500 or $http_status_code < 400` — `span.kind`, `trace.parent_id` and `duration_ms` are the field names * `if $error exists and contains "trace", return $error` — `error` is the field name As you type the `$` symbol in the prompt display, Honeycomb shows a dropdown of available fields to select. Use this feature to select exactly which fields to use in your prompt expression. ### Error: "The following fields do not exist:" when generating a Calculated Field with AI This error message means the AI could not find one or more entered fields with the `$` syntax in your Calculated Field prompt. To resolve this issue: * double-check that each column name you reference (for example, `$field_name`) exactly matches a field in your schema. * avoid typos by typing `$` in the prompt display and selecting the target field from the dropdown that appears. This ensures the use of a valid field name. Calculated field selector

# Common Issues with Honeycomb Refinery Source: https://docs.honeycomb.io/troubleshoot/common-issues/refinery Troubleshoot common Refinery problems, including configuration errors, sampling rule issues, cluster connectivity, and data not reaching Honeycomb as expected. Troubleshoot common issues with [Honeycomb Refinery](/manage-data-volume/sample/honeycomb-refinery/). To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## Debug Logs The default logging level is `warn`. The `debug` level emits too much data to be used in production, but contains excellent information in a pre-production environment. Setting the logging level to `debug` during initial configuration will help you understand what is working and what is not, but when traffic volumes increase it should be set to `warn`. ## Restarts Refinery does not yet buffer traces or sampling decisions to disk. When you restart the process, all in-flight traces will be flushed and sent upstream to Honeycomb, but you will lose the record of past trace decisions. When started back up, Refinery will start with a clean slate. ## Configuration Configuration file formats (TOML and YAML) can be confusing to read and write. There is an option to check the loaded configuration by using one of the `/query` endpoints from the command line, from a server that can access a Refinery host. The `/query` endpoints are protected and can be enabled by specifying `QueryAuthToken` in the configuration file or specifying `REFINERY_QUERY_AUTH_TOKEN` in Refinery's environment. All requests to any `/query` endpoint must include the header `X-Honeycomb-Refinery-Query` set to the value of the specified token. ### Checking All Rules Retrieve the entire rules configuration in the desired format from Refinery: ```shell theme={} curl --get $REFINERY_HOST/query/allrules/$FORMAT --header "x-honeycomb-refinery-query: my-local-token" ``` where: * `$REFINERY_HOST` should be the url of your refinery. * `$FORMAT` can be one of `json`, `yaml`, or `toml`. ### Checking Rules for a Given Context Retrieve the rule set that Refinery uses for the environment (or dataset in Classic Mode) defined in the variable `$ENVIRON`: ```shell theme={} curl --get $REFINERY_HOST/query/rules/$FORMAT/$ENVIRON --header "x-honeycomb-refinery-query: my-local-token" ``` where: * `$REFINERY_HOST` should be the URL of your refinery. * `$FORMAT` can be one of `json`, `yaml`, or `toml`. * `$ENVIRON` is the name of the environment (or dataset, in Classic Mode). The response contains a map of the sampler type to its rule set. ### Checking Configuration Metadata Retrieve information about the configurations currently in use, including the timestamp when the configuration was last loaded: ```bash theme={} curl --include --get $REFINERY_HOST/query/configmetadata --header "x-honeycomb-refinery-query: my-local-token" ``` where: * `$REFINERY_HOST` should be the URL of your refinery. The response contains a JSON blob of information about Refinery's configurations. It will look something like this: ```json theme={} [ { "type": "config", "id": "tools/loadtest/config.yaml", "hash": "1047bb6140b487ecdb0745f3335b6bc3", "loaded_at": "2022-11-08T22:24:18-05:00" }, { "type": "rules", "id": "tools/loadtest/rules.yaml", "hash": "2d88389e1ff6530fba53466973e591e0", "loaded_at": "2022-11-08T22:24:18-05:00" } ] ``` For file-based configurations (the only type currently supported), the `hash` value is identical to the value generated by the `md5sum` command available in major operating systems. ## Sampling Decisions Refinery can send telemetry that includes information that can help debug the sampling decisions that are made. To enable it, in the configuration file, set `AddRuleReasonToTrace` to `true`. This will cause traces that are sent to Honeycomb to include a field `meta.refinery.reason`. This field will contain text indicating which rule was evaluated that caused the trace to be included. ## Rules Datatypes The rules comparisons in Refinery's Rules-Based Sampler take the datatype of the fields into account. In particular, a rule that compares `status_code` to `200` (an integer) will fail if the status code is actually `"200"` (a string), and vice-versa. In a mixed environment where either datatype may be included in the telemetry, you should create a separate rule for each case. This situation can be hard to diagnose, because Honeycomb's backend converts all the values of a given field to the datatype specified in the [dataset schema](/reference/honeycomb-ui/manage-data/datasets/dataset-settings/). Inspection of the data in Honeycomb will not give any indication that this has happened. If you see rules that appear to not execute when they should have, please consider this possibility of incorrect datatype. ## Health Checks Use health check API endpoints to determine if an instance is bootstrapped. The Refinery cluster machines respond to two different health check endpoints via HTTP: ### `/alive` This `/alive` API call will return a `200` JSON response. It does not perform any checks beyond the web server's response to requests. ```json theme={} { "source": "refinery", "alive": "yes" } ``` ### `/x/alive` This `/x/alive` API call will return a `200` JSON response that has been proxied from the Honeycomb API. This can be used to determine if the instance is able to communicate with Honeycomb. ```json theme={} { "alive": "yes" } ``` ### gRPC Health Check If gRPC is configured, Refinery also responds to [a standard gRPC Health Probe](https://github.com/grpc/grpc/blob/592bfb9c0040fba382e5dc106e2ebbf0c96465fb/doc/health-checking.md). # Common Issues with Sending Data to Honeycomb Source: https://docs.honeycomb.io/troubleshoot/common-issues/sending-data Troubleshoot common problems when sending data to Honeycomb, including authentication errors, missing telemetry, and SDK or Collector configuration issues. If you need to troubleshoot sending data to Honeycomb, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## OpenTelemetry Collector Troubleshoot issues related to the [OpenTelemetry Collector](/send-data/opentelemetry/collector/). ### Export Debug Logging or to a File If data is not arriving in Honeycomb as expected, add a debug-level logger to emit the data to the console for review. In the exporters section of your config file, add a `debug` exporter with `verbosity` of `detailed`. The `debug` exporter should also be added to the `service` section, either replacing or accompanying the `otlp` exporter. If the collector is running in Docker or otherwise difficult to parse via the console, you can also send the data to a specific file for review. Add an additional `file` exporter with a path to the file that should contain the output. This example includes an `otlp` exporter for sending to Honeycomb, a `debug` exporter for debug-level logging to the console, and a `file` exporter for storing the data logged. ```yaml theme={} receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: otlp: endpoint: "api.honeycomb.io:443" # US instance #endpoint: "api.eu1.honeycomb.io:443" # EU instance headers: "x-honeycomb-team": "YOUR_API_KEY" debug: verbosity: detailed file: # optionally export data to a file path: /var/lib/data.json # optional file to store exported data service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp, debug, file] # only add file if added above ``` ### CORS Errors Sometimes a [CORS error](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors) may occur when setting up browser telemetry. Confirm the receiver's setup has the correct port defined. The default port for `http` is `4318`. If this port or endpoint is overwritten in the collector configuration file, ensure it matches the endpoint set in the application sending telemetry. Confirm the `allowed_origins` list in the receivers matches the origin of the browser telemetry. If there is a load balancer in front of the Collector, it should also be configured to accept requests from the browser origin. One way to determine whether the issue is rooted in how the application is exporting as opposed to network connectivity is to issue a `curl` command to the server from the browser origin. For example, if the application was running on `http://localhost:3000`, and the collector was listening on port `4318` at `http://otel-collector.com/v1/traces`: ```sh theme={} curl -i http://otel-collector.com/v1/traces -H "Origin: http://localhost:3000" -H "Access-Control-Request-Method: POST" -H "Access-Control-Request-Headers: X-Requested-With" -H "Content-Type: application/json" -X OPTIONS --verbose ``` The response from the server should include `Access-Control-Allow-Credentials: true`. ### Verify OTLP Protobuf Definitions Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK's protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK. ## OpenTelemetry SDKs and Honeycomb Distributions Troubleshoot issues related to the OpenTelemetry SDKs and Honeycomb Distributions. ### Browser JavaScript #### CORS Errors Sometimes a [CORS error](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors) may occur when setting up browser telemetry. Confirm the receiver's setup has the correct port defined. The default port for `http` is `4318`. If this port or endpoint is overwritten in the collector configuration file, ensure it matches the endpoint set in the application sending telemetry. Confirm the `allowed_origins` list in the receivers matches the origin of the browser telemetry. If there is a load balancer in front of the Collector, it should also be configured to accept requests from the browser origin. One way to determine whether the issue is rooted in how the application is exporting as opposed to network connectivity is to issue a `curl` command to the server from the browser origin. For example, if the application was running on `http://localhost:3000`, and the collector was listening on port `4318` at `http://otel-collector.com/v1/traces`: ```sh theme={} curl -i http://otel-collector.com/v1/traces -H "Origin: http://localhost:3000" -H "Access-Control-Request-Method: POST" -H "Access-Control-Request-Headers: X-Requested-With" -H "Content-Type: application/json" -X OPTIONS --verbose ``` The response from the server should include `Access-Control-Allow-Credentials: true`. #### FID is Missing As of v0.18.0, [Honeycomb OpenTelemetry Web SDK](/send-data/javascript-browser/) does not support `fid` (First Input Delay) vital. To continue capturing meaningful responsiveness metrics, we recommend [migrating from `fid` to `inp` (Interaction to Next Paint) in the Honeycomb Web Instrumentation Package](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-fid-to-inp-in-the-honeycomb-web-instrumentation-package). ### JavaScript #### My Traces have Duplicate Spans If a `This trace has multiple spans sharing the same non-null span ID` error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once. One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the [Tracing API](https://opentelemetry.io/docs/specs/otel/trace/api/) throughout the application runtime to add manual instrumentation. #### Debug Mode To enable debugging when running the Honeycomb OpenTelemetry Node SDK, set the `DEBUG` environment variable to `true`: ```shell theme={} export DEBUG=true ``` To set the debug option in code instead of an environment variable, add `debug` to the `HoneycombSDK`: ```javascript theme={} const sdk = new HoneycombSDK({ apiKey: "your-api-key", serviceName: "your-service-name", instrumentations: [getNodeAutoInstrumentations()], sampleRate: 5, localVisualizations: true, debug: true, }) ``` When the debug setting is enabled, the Honeycomb SDK configures a [DiagConsoleLogger](https://github.com/open-telemetry/opentelemetry-js/blob/main/api/src/diag/consoleLogger.ts) that logs telemetry to the console with the log level of Debug. The debug setting in the Honeycomb SDK will also output to the console the full options configuration, including but not limited to protocol, API Key, and endpoint. If you are not using the Honeycomb SDK, or if you wish to change the logging level, you can still use the logger directly: ```javascript theme={} const { diag, DiagConsoleLogger, DiagLogLevel } = require("@opentelemetry/api"); diag.setLogger(new DiagConsoleLogger(),DiagLogLevel.DEBUG); ``` Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes. #### OTLP Protobuf Definitions Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK's protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK. #### Receiving 464 Errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. #### Missing `.proto` Files Error If your application builds using a bundler, like Webpack or ESBuild, or if your application uses TypeScript, you will see an error that says that a specific `.proto` file is not found: ```bash theme={} opentelemetry/proto/collector/trace/v1/trace_service.proto not found in any of the include paths /protos ``` This error appears because the `.proto` files are not imported or required by the library that uses them, so bundlers do not know to include them in the final build. There are two ways to work around this error: #### Copy the `/protos` Directory to the Correct Location Modify the bundler configuration to copy the `/protos` directory from the library to the same level as the build directory: ```js theme={} const CopyPlugin = require("copy-webpack-plugin"); module.exports = { ... other config ... plugins: [ new CopyPlugin({ patterns: [ { from: "./node_modules/@opentelemetry/otlp-grpc-exporter-base/build/protos/**/*", to: "./protos" } ], }), ], }; ``` ```js theme={} const { copy } = require('esbuild-plugin-copy') require('esbuild').build({ ... other config ... plugins: [ copy({ resolveFrom: 'cwd', _assets: { from: ['./node_modules/@opentelemetry/otlp-grpc-exporter-base/build/protos/**/*'], to: ['./protos'], keepStructure: true }, }), ], }).catch(() => process.exit(1)); ``` TypeScript configuration currently cannot specify which extra files to copy for your build. In this case, use a `postbuild` npm script to copy the proto files to the correct location. ### Python #### My Traces have Duplicate Spans If a `This trace has multiple spans sharing the same non-null span ID` error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once. One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the [Tracing API](https://opentelemetry.io/docs/specs/otel/trace/api/) throughout the application runtime to add manual instrumentation. #### Typechecking Errors With MyPy Using MyPy requires turning on support for [namespace packages](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-namespace-packages). To turn on support from the command line, run: ```shell theme={} mypy --namespace-packages ``` Or to turn on support from your project configuration file, add: ```toml theme={} [tool.mypy] namespace_packages = true ``` #### Exporting to the Console The OpenTelemetry Python SDK typically shows errors in the console when applicable. If no errors appear but your data is not in Honeycomb as expected, you can enable debug mode, which prints all spans to the console. This will help confirm whether your app is being instrumented with the data you expect. Set the `DEBUG` environment variable: ```shell theme={} export DEBUG=true ``` Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes. #### Flask Debugging If the application uses Flask, instrumentation will not work if Flask debugging is enabled. Unset the `FLASK_DEBUG` variable or set it to `false`. If `DEBUG` is enabled and `FLASK_DEBUG` is disabled, the output in the console will show: ```shell theme={} DEBUG:sitecustomize:Instrumented flask * Debug mode: off ``` #### Django Instrumentation If the application uses Django, instrumentation requires the `DJANGO_SETTINGS_MODULE` and the `--noreload` flag. 1. Set the environment variable for `DJANGO_SETTINGS_MODULE` based on the name of your settings file. ```shell theme={} export DJANGO_SETTINGS_MODULE=myapp.settings ``` 2. Run the application with the `--noreload` flag to avoid Django running `main` twice. For example, the command to run the application may look like this: ```shell theme={} opentelemetry-instrument python manage.py runserver --noreload ``` #### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. #### Visualize Traces Locally Honeycomb's OpenTelemetry Distribution for Python can create a link to a trace visualization in the Honeycomb UI for local traces. Local visualizations enables a faster feedback cycle when adding, modifying, or verifying instrumentation. To enable local visualizations: 1. Set the `HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS` environment variable to `true`: ```shell theme={} export HONEYCOMB_ENABLE_LOCAL_VISUALIZATIONS=true ``` 2. Run your application: ```shell theme={} opentelemetry-instrument python myapp.py ``` The output displays the name of the root span and a link to Honeycomb that shows its trace. For example: ```text theme={} Trace for Honeycomb link: ``` 3. Select the link to view the trace in detail within the Honeycomb UI. In production, disable local visualization as it creates additional overhead to create the link to a trace in Honeycomb and print it to the console. #### Bootstrap with Pip Running the `opentelemetry-bootstrap` command with `--action=install` does not add packages to a `requirements.txt` file, and instead only adds to the current environment. This can result in inadvertently missing dependencies in checked-in code or container images. If using `pip`, our recommendation is as listed in [Acquire Dependencies](/send-data/python/opentelemetry-sdk/#acquire-dependencies): 1. Generate the list of packages with `opentelemetry-bootstrap`. 2. Add the packages to the `requirements.txt` file manually or using `opentelemetry-bootstrap >> requirements.txt`. 3. Install the packages with `pip install -r requirements.txt`. If you do install with the bootstrap command, run `pip freeze` and manually add the necessary packages to the `requirements.txt` file. #### Bootstrap with Poetry The `opentelemetry-bootstrap` command does not include the option to install packages when using Poetry. Refer to the outputted list of packages to manually install each package and add to the `pyproject.toml` file. To confirm the installed packages in your poetry environment, run `poetry show`. #### Versions of Core and Contrib Upgrading OpenTelemetry packages generally work best when they are all upgraded together. The [Releases for `opentelemetry-python`](https://github.com/open-telemetry/opentelemetry-python/releases) and the [Releases for `opentelemetry-python-contrib`](https://github.com/open-telemetry/opentelemetry-python-contrib/releases) list the compatible versions for each. For example, `Version 1.20.0/0.41b0` means core packages versioned `1.20.0` are compatible with Contrib packages versioned `0.41b0` #### OTLP Protobuf Definitions Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK's protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK. #### Receiving 464 Errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. ### Java #### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. #### My Traces have Duplicate Spans If a `This trace has multiple spans sharing the same non-null span ID` error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once. One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the [Tracing API](https://opentelemetry.io/docs/specs/otel/trace/api/) throughout the application runtime to add manual instrumentation. #### Debug Mode To enable debugging when running with the OpenTelemetry Java Agent, set the `otel.javaagent.debug` system property or `OTEL_JAVAAGENT_DEBUG` environment variable to `true`. When this setting is provided, the Agent configures a [LoggingSpanExporter](https://github.com/open-telemetry/opentelemetry-java/tree/main/exporters/logging) that logs traces & metrics data. If you are not using the OpenTelemetry Java Agent, you can add a `LoggingSpanExporter` to your builder configuration. This will require adding another dependency on `io.opentelemetry:opentelemetry-exporter-logging`. ```java theme={} import io.honeycomb.opentelemetry.OpenTelemetryConfiguration; import io.opentelemetry.exporter.logging.LoggingSpanExporter; // for debugging import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor; // for debugging public OpenTelemetry honeycomb() { return OpenTelemetryConfiguration.builder() .setApiKey("your-api-key") .setServiceName("your-service-name") .addSpanProcessor(SimpleSpanProcessor.create(new LoggingSpanExporter())) // for debugging .buildAndRegisterGlobal(); } ``` Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes. #### gRPC Transport Customization A gRPC transport is required to transmit OpenTelemetry data. HoneycombSDK includes `grpc-netty-shaded`. If you are using another gRPC dependency, version conflicts can come up with an error like this: ```cmd theme={} java.lang.NoSuchMethodError: io/grpc/ClientStreamTracer$StreamInfo$Builder.setPreviousAttempts(I)Lio/grpc/ClientStreamTracer$StreamInfo$Builder; (loaded from file:/app.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51) called from class io.grpc.internal.GrpcUtil (loaded from file:/io.grpc/grpc-core/1.41.0/882b6572f7d805b9b32e3993b1d7d3e022791b3a/grpc-core-1.41.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51). ``` If you would like to use another gRPC transport, you can exclude the `grpc-netty-shaded` transitive dependency: ```groovy theme={} dependencies { implementation('io.honeycomb:honeycomb-opentelemetry-sdk:{honeycomb_opentelemetry_java.version}') { exclude group: 'io.grpc', module: 'grpc-netty-shaded' } } ``` ```xml theme={} io.honeycomb honeycomb-opentelemetry-sdk {honeycomb_opentelemetry_java.version} io.grpc grpc-netty-shaded ``` #### Receiving 464 Errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. Additionally, older JVMs may not have sufficient gRPC support and may attempt to send telemetry using HTTP1. To resolve this, either update to a newer JVM or use `http/protobuf` as the transfer protocol. ### .NET #### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. #### My Traces have Duplicate Spans If a `This trace has multiple spans sharing the same non-null span ID` error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once. One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the [Tracing API](https://opentelemetry.io/docs/specs/otel/trace/api/) throughout the application runtime to add manual instrumentation. #### Error Log By default, the .NET SDK does not emit error information to the console. The SDK does include a [self-diagnostics feature](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry#troubleshooting) to help with troubleshooting by writing errors to a log file. Enable self-diagnostics by creating a file in the current working directory called `OTEL_DIAGNOSTICS.json`: ```json theme={} { "LogDirectory": ".", "FileSize": 1024, "LogLevel": "Error" } ``` Running the app with the above configuration in the `OTEL_DIAGNOSTICS.json` file will generate a file named for `ExecutableName.ProcessId.log`, such as `console.30763.log`. The newly generated log file will contain any applicable errors for the app. The `LogDirectory` represents the directory in which the log file will be stored, and can be changed to output to a different location. The `FileSize` is the maximum size the log file can grow in `KiB`, and can be adjusted if a larger size is needed to prevent overwriting of logging output. Adjust the log level as needed for more or less verbose logging, using the [fields available](https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.tracing.eventlevel?view=net-6.0#fields) with System Diagnostics. To disable this error log, delete the `OTEL_DIAGNOSTICS.json` file. #### Exporting to the Console If you are using the Honeycomb OpenTelemetry Distribution, warnings will appear in the console if you are missing an API Key or Dataset. If no errors appear but your data is not in Honeycomb as expected, use a `ConsoleExporter` to print your spans to the console. This will help confirm whether your app is being instrumented with the data you expect. 1. Import the `ConsoleExporter`: `dotnet add package OpenTelemetry.Exporter.Console --prerelease` 2. Add the `ConsoleExporter` to your configuration: ```csharp theme={} using var tracerProvider = OpenTelemetry.Sdk.CreateTracerProviderBuilder() .AddHoneycomb(options) .AddConsoleExporter() // for debugging .Build(); ``` Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes. #### Receiving 464 Errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. #### Instrumentation Package Rename Previous versions of the Honeycomb OpenTelemetry Distribution had a package called `Honeycomb.OpenTelemetry.AutoInstrumentations`, which was renamed to `Honeycomb.OpenTelemetry.CommonInstrumentations`. The name change of the instrumentation package is intended to better reflect the purpose of the package, but is a potential upgrade step to ensure the appropriate package is being used. Refer to our [Releases](https://github.com/honeycombio/honeycomb-opentelemetry-dotnet/releases) page on GitHub as needed. #### OpenTelemetry Tracing and Metrics Rename Previous versions of OpenTelemetry used `AddOpenTelemetryTracing` and `AddOpenTelemetryMetrics` to add tracing and metrics. `AddOpenTelemetryTracing` has been replaced with `AddOpenTelemetry().WithTracing`. `AddOpenTelemetryMetrics` has been replaced with `AddOpenTelemetry().WithMetrics`. ### Go #### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. #### My Traces have Duplicate Spans If a `This trace has multiple spans sharing the same non-null span ID` error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once. One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the [Tracing API](https://opentelemetry.io/docs/specs/otel/trace/api/) throughout the application runtime to add manual instrumentation. #### Printing to the Console If no errors appear but your data is not in Honeycomb as expected, you can set the `DEBUG` environment variable, which will both log the distribution configuration to `stdout` and configure spans to be output to `stdout`. This will help confirm whether your app is being instrumented with the data you expect. ```shell theme={} export DEBUG=true ``` Keep in mind that printing to the console should only be used for debugging purposes. It is not recommended for production. #### Exporting to an Insecure Endpoint By default, the Honeycomb Distribution uses a secure exporter. To export to an insecure endpoint, such as a local collector on the same network, set the Insecure option for the exporter with an environment variable: ```shell theme={} export OTEL_EXPORTER_OTLP_INSECURE=true ``` To set the Insecure option in code instead of an environment variable: ```go theme={} otelconfig.ConfigureOpenTelemetry(otelconfig.WithExporterInsecure(true)) ``` #### OTLP Protobuf Definitions Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK's protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK. #### Receiving 464 Errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. ### Ruby If no errors appear in the console but your data is not in Honeycomb as expected, use a `ConsoleSpanExporter` to print your spans to the console. This will help confirm whether your app is being instrumented with the data you expect. As shown in the [OpenTelemetry SDK for Ruby](https://github.com/open-telemetry/opentelemetry-ruby/blob/main/sdk/README.md), configuration options can be set via environment variables or programmatically. To use an environment variable to print to the console, set the variable before calling `configure`: ```ruby theme={} require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' ENV['OTEL_TRACES_EXPORTER'] = 'console' # for debugging OpenTelemetry::SDK.configure do |c| c.use_all() end ``` Alternatively, add the `ConsoleSpanExporter` to your configuration: ```ruby theme={} OpenTelemetry::SDK.configure do |c| c.add_span_processor( OpenTelemetry::SDK::Trace::Export::SimpleSpanProcessor.new( OpenTelemetry::SDK::Trace::Export::ConsoleSpanExporter.new ) ) ... end ``` Keep in mind that printing to the console is not recommended for production and should only be used for debugging purposes. #### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. #### My Traces have Duplicate Spans If a `This trace has multiple spans sharing the same non-null span ID` error appears in Honeycomb, it is likely that your application is not instrumented correctly and is sending the same trace to Honeycomb more than once. One possible misconfiguration is initializing OpenTelemetry more than once. Make sure to only initialize OpenTelemetry once when the application starts, and then use the [Tracing API](https://opentelemetry.io/docs/specs/otel/trace/api/) throughout the application runtime to add manual instrumentation. #### Apple Mac M1 Protobuf Library Errors The OpenTelemetry libraries may produce an error on Apple Mac M1 computers. ```bash theme={} /rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- google/protobuf_c (LoadError) ``` Rubygems, by default, installs this gem with native extensions. Reinstallation of the gem without these extensions may remove this error. First, uninstall the native version of the `google-protobuf` library. ```shell theme={} gem uninstall google-protobuf ``` Then, install the `google-protobuf` library for the ruby platform. ```shell theme={} gem install google-protobuf --platform=ruby ``` #### OTLP Protobuf Definitions Honeycomb supports receiving telemetry data via OpenTelemetry's native protocol, [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md), over gRPC, HTTP/protobuf, and HTTP/JSON. The minimum supported versions of OTLP [protobuf definitions](https://github.com/open-telemetry/opentelemetry-proto) are 1.0 for traces, metrics, and logs. If the protobuf version in use by the SDK does not match a supported version by Honeycomb, a different version of the SDK may need to be used. If the SDK's protobuf version is older than the minimum supported version, and telemetry is not appearing as expected in Honeycomb, upgrade the SDK to a version with the supported protobuf definitions. If using an added dependency on a proto library, ensure the version of protobuf definitions matches the supported version of the SDK. #### Receiving 464 Errors You may receive a `464` error response from the Honeycomb API when sending telemetry using gRPC and HTTP1. The gRPC format depends on using HTTP2 and any request over HTTP1 will be rejected by the Honeycomb servers. ## Libhoney Troubleshoot issues related to [LibHoney](/send-data/logs/structured/libhoney/). ### JavaScript If you do not specify a dataset, event data will be sent to a dataset called `unknown_dataset`. If using [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) without specifying a dataset, you will get an error at runtime and no data will be sent. ### Python #### Debug Mode Our Python SDK supports an optional **debug mode**. Simply pass `debug=True` to the `init` function at startup to get verbose logging of events sent to and responses from the Honeycomb API. #### Using the Python SDK with Python Pre-Fork Models Popular servers like [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html) and [Gunicorn](https://gunicorn.org/) utilize a pre-fork model where requests are delegated to separate Python processes. Initializing the SDK **before** the fork happens can lead to a state where events cannot be sent. To initialize the SDK correctly, you will need to run your init code inside a post-fork hook. ##### uWSGI Users of uWSGI can use a [postfork decorator](https://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html#uwsgidecorators.postfork). Simply add the `@postfork` decorator to the function that initializes the Python Beeline, and it will be executed post-fork. ```python theme={} import logging import os import libhoney from uwsgidecorators import postfork @postfork def init_libhoney(): logging.info(f'libhoney initialization in process pid {os.getpid()}') libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-uwsgi-example", debug=True) ``` ##### Gunicorn Gunicorn users can define a `post_worker_init` function in the Gunicorn configuration, and initialize the SDK there. ```python theme={} # conf.py import logging import os import libhoney def post_worker_init(worker): logging.info(f'libhoney initialization in process pid {os.getpid()}') libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-gunicorn-example", debug=True) ``` Then start gunicorn with the `-c` option: ```python theme={} gunicorn -c /path/to/conf.py ``` ##### Celery [Celery](https://github.com/celery/celery) uses a pre-fork approach to create worker processes. You can specify a `worker_process_init` decorated function to initialize the Python SDK after each worker has started. ```python theme={} import logging import os import libhoney from celery.signals import worker_process_init @worker_process_init.connect def initialize_honeycomb(**kwargs): logging.info(f'libhoney initialization in process pid {os.getpid()}') libhoney.init(writekey="YOUR_API_KEY", dataset="honeycomb-celery-example", debug=True) ``` ### Java If you do not specify a dataset, event data will be sent to a dataset called `unknown_dataset`. If using [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) without specifying a dataset, you will get an error at runtime and no data will be sent. ### Go If you do not specify a dataset, event data will be sent to a dataset called `unknown_dataset`. If using [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) without specifying a dataset, you will get an error at runtime and no data will be sent. ### Ruby If you do not specify a dataset, event data will be sent to a dataset called `unknown_dataset`. If using [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments) without specifying a dataset, you will get an error at runtime and no data will be sent. ## Honeytail Troubleshoot issues related to [Honeytail](/send-data/logs/structured/honeytail/). ### General Troubleshooting Below, find some general debugging tips when trying to send data to Honeycomb. #### New Data Does Not Show Up in Honeycomb, and New Dataset Does not Appear on Dashboard "Datasets" are created when we first begin receiving data under a new "Dataset Name" (used/specified by all of our SDKs and agents). If you do not see an expected dataset yet, our servers mostly likely have not yet received anything from you. To figure out why, the simplest step is to add a `--debug` flag to your `honeytail` call. This should output information about whether lines are being parsed, failing to send to our servers, or whether `honeytail` is receiving any input at all. Another useful thing to try may be to add `--status_interval=1` to your flags, which will output a line like the below, each second (newlines added for legibility): ```log theme={} INFO[0002] Summary of sent events avg_duration=295.783µs count_per_status=map[400:10] errors=map[] fastest=259.689µs response_bodies=map[request body is too large:10] slowest=348.297µs total=10 ``` The `total` here is the number of events sent to Honeycomb; the rest are stats characterizing how those events were sent and received. (A `total=0` value would clue us into the fact that `honeytail` is not sending any events at all.) In the line above, we see that events were, in fact, invalid and being rejected by the server. #### New Events do not Appear in an Existing Dataset When using `honeytail`, the `--dataset` (`-d` for short) argument will determine the name of the dataset created on Honeycomb's servers. If you are writing into an existing dataset, the quickest way to check for new data is to run a `COUNT` query over the last 30 minutes: If your new events do not appear, try the `--debug` or `--status_interval=1`. (Change the status interval from `1` to `5` to see the summary every 5 seconds). #### `honeytail` does not seem to be progressing on my log file Are you trying to send data from an existing file? `honeytail`'s default behavior is to watch files and process newly-appended data. If you are attempting to send data from an existing file, make sure to use the `--backfill` flag. This flag will make sure `honeytail` begins reading the file from the beginning and exits when finished. #### Existing Timestamps Values are not Respected Our JSON parser makes a best-effort attempt to parse and understand timestamps in your JSON logs. Take a look at the [Timestamp parsing](/send-data/logs/structured/json/#timestamp-parsing) section of the JSON docs to see timestamp formats understood by default. If you suspect your timestamp format is unconventional, or the time field is keyed by an unconventional field name, providing `--json.timefield` and `--json.format` arguments will nudge `honeytail` in the right direction. ### NGINX First, check out `honeytail` [General Troubleshooting section](#general-troubleshooting) for general debugging tips. #### Exiting with error message like `log_format ` not found in given config Make sure the file referenced by `--nginx.conf` contains your log format definitions. The log format definition should look something like the example below, and should contain whatever format name you are passing to `--nginx.format`: ```nginx theme={} log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; ``` ... which defines the output log format for the log format name "combined." Note that, in more advanced nginx setups, it is possible for the log format to be defined in an overall `nginx.conf` file, while a different config file (maybe under, say, `/etc/nginx/sites-enabled/api.conf`) tells nginx how to output the `access_log` and which format to use. In this case, you will want to make sure and use the config file containing the `log_format` line for the `--nginx.conf` argument. #### No data is being sent, and `--debug` reveals `failed to parse nginx log line` messages If your log format has fields that are likely to have spaces in them, make sure to surround that field with single quotes. For example, if `$my_upstream_var` is likely to contain spaces, you will want to change this: ```nginx theme={} log_format main '$remote_addr $host $my_upstream_var $request $other_field'; ``` to a `log_format` with quotes: ```nginx theme={} log_format main '$remote_addr $host "$my_upstream_var" $request $other_field'; ``` You can make sure that your quotes had the right effect by peeking at the nginx logs your server is outputting to make sure that the `$my_upstream_var` value is correctly surrounded by quotes. It is good practice to put **any** variable that comes from an HTTP header in double quotes, because you are depending on whomever is sending you traffic to put only one string in the header. Some headers also default to multiple words. For example, the `$http_authorization` header is represented by a `-` if it is absent and is two words (`Basic abcdef123456`) when present. ### MySQL First, check out `honeytail` [General Troubleshooting section](#general-troubleshooting) for general debugging tips. #### No data is being sent, and `--debug` does not seem to show anything useful Take a look at the `--file` being handed to `honeytail` and make sure they look like MySQL slow query logs, with blocks of comments containing metadata alternating with the MySQL commands issued. An example excerpt from a MySQL slow query log might look like: ```mysql theme={} # Time: 151008 0:31:03 # User@Host: rails[rails] @ [10.252.10.158] # Query_time: 0.000547 Lock_time: 0.000019 Rows_sent: 1 Rows_examined: 938 use rails; SET timestamp=1444264263; SELECT `app_data`.* FROM `app_data` WHERE (`app_data`.user_id = 69213) LIMIT 1; ``` #### Only Some Queries Seem to Appear in Honeycomb Did you remember to `SET` the `GLOBAL long_query_time`? Our parser relies on reading your server's slow query logs, which contain much more valuable metadata than the general log—and the default slow query threshold is 10 seconds. Try checking the output of: ```mysql theme={} mysql> SELECT @@GLOBAL.long_query_time; ``` If it is not `0`, take another look at the steps to [Configure MySQL Query Logging](/send-data/logs/structured/mysql/#configure-mysql-query-logging). ### PostgreSQL First, check out `honeytail` [General Troubleshooting section](#general-troubleshooting) for general debugging tips. #### No data is being sent, and `--debug` does not seem to show anything useful Take a look at the `--file` being handed to `honeytail` and make sure it contains PostgreSQL query statements. An example excerpt from a PostgreSQL log file might look like: ```sql theme={} 2017-11-10 23:24:01 UTC [1998-1] LOG: autovacuum launcher started 2017-11-10 23:24:01 UTC [2000-1] [unknown]@[unknown] LOG: incomplete startup packet 2017-11-10 23:24:02 UTC [2003-1] postgres@postgres LOG: duration: 4.356 ms statement: SELECT d.datname as "Name", pg_catalog.pg_get_userbyid(d.datdba) as "Owner", pg_catalog.pg_encoding_to_char(d.encoding) as "Encoding", d.datcollate as "Collate", d.datctype as "Ctype", pg_catalog.array_to_string(d.datacl, E'\n') AS "Access privileges" FROM pg_catalog.pg_database d ORDER BY 1; ``` Also check that the value you are passing in the `--postgresql.log_line_prefix` flag matches PostgreSQL's configured value, which you can find using `SHOW log_line_prefix` at a `psql` prompt: ```sql theme={} # SHOW log_line_prefix; log_line_prefix --------------------- %t [%p-%l] %q%u@%d ``` # Common Issues with Visualization Source: https://docs.honeycomb.io/troubleshoot/common-issues/visualization Troubleshoot common problems with Honeycomb Home, Board Templates, Service Map, and other visualization features not displaying data as expected. If you need to troubleshoot your visualizations, explore these solutions to common issues. To ask questions and learn more, visit our [Support Knowledge Base](https://support.honeycomb.io/) or [join our Pollinators Community](/troubleshoot/community/). ## Home Troubleshoot visualizations in [Honeycomb Home](/observe/honeycomb-home/). ### Traces View Troubleshoot visualizations on the **Traces** view in [Honeycomb Home](/observe/honeycomb-home/). #### Traces view is empty If your dataset is not a tracing dataset, then the **Traces** view will not populate. To ensure the **Traces** view is populated, either: * send a source field named `trace.trace_id` to Honeycomb, which will let Honeycomb detect and map dataset fields automatically, or * manually map a source field to the **Trace ID** dataset field when you [map your data](/send-data/standardize/map-data/) #### Empty Span Visualizations "Total Spans", "Span Volume", "Spans by Type", "p95 of Span Duration", and "Top Spans" rely on a source field being mapped to the **Span Id** dataset field. To ensure the visualizations are populated, either: * send a source field named `span.id` to Honeycomb, which will let Honeycomb detect and map dataset fields automatically, or * manually map a source field to the **Span Id** dataset field when you [map your data](/send-data/standardize/map-data/) #### Empty Duration Visualizations The "P95 of Span Duration" visualization relies on a source field being mapped to the `duration` dataset field. To ensure the visualizations are populated, either: * send a source field named `duration_ms` to Honeycomb, which will let Honeycomb detect and map dataset fields automatically, or * manually map a source field to the **Span Duration** dataset field when you [map your data](/send-data/standardize/map-data/) #### Empty Total Exceptions and Top Exceptions Visualizations "Total Exceptions" and "Top Exceptions" visualizations rely on the `exception.message` field being available. To learn how to record `exception` with OpenTelemetry, visit [OpenTelemetry's documentation on exceptions](https://opentelemetry.io/docs/specs/otel/trace/exceptions/). #### Empty Exception Volume By Type Visualizations "Exception Volume By Type" relies on the `exception.type` field being available. To learn how to record `exception` with OpenTelemetry, visit [OpenTelemetry's documentation on exception spans](https://opentelemetry.io/docs/specs/semconv/exceptions/exceptions-spans/). ### Logs View Troubleshoot visualizations on the **Logs** view in [Honeycomb Home](/observe/honeycomb-home/). #### Logs view is empty If your dataset is not a logs dataset, the **Logs** view will not populate. To ensure the **Logs** view is populated, either: * send a source field named `body` to Honeycomb, which will let Honeycomb detect and map dataset fields automatically, or * manually map a source field to the **Logs: Message** dataset field when you [map your data](/send-data/standardize/map-data/) #### Severity visualizations are empty Severity visualizations rely on a source field being mapped to the **Logs: Severity** dataset field. To ensure the Severity visualizations are populated, either: * send a source field named `severity` to Honeycomb, which will let Honeycomb detect and map dataset fields automatically, or * manually map a source field to the **Logs: Severity** dataset field when you [map your data](/send-data/standardize/map-data/) #### "Logs by Severity" chart shows only non-standard (Other) severities The Logs by Severity Chart supports standard severities, which include `fatal`, `error`, `warn`, `info`, `trace`, `debug`, and `unspecified`. If you are sending non-standard severities, the chart will label them as `Other`. To ensure the "Logs by Severity" chart shows your severities, use Calculated Fields to [parse your severity values into Honeycomb standard severity values](/send-data/standardize/transform-data/#transform-source-severities-into-honeycomb-standard-severities). ### Explore Data View #### Log lines are not color-coded by severity Severity visualizations rely on a source field being mapped to the **Logs: Severity** dataset field. To ensure the Severity visualizations are populated, either: * send a source field named `severity` to Honeycomb, which will let Honeycomb detect and map dataset fields automatically, or * manually map a source field to the **Logs: Severity** dataset field when you [map your data](/send-data/standardize/map-data/) In addition, if you are using non-standard severities, use Calculated Fields to [parse your severity values into Honeycomb standard severity values](/send-data/standardize/transform-data/#transform-source-severities-into-honeycomb-standard-severities). ## Board Templates Troubleshoot visualizations in [Board Templates](/observe/boards/templates/). ### A Board template is missing a query Visualizations in board templates depend on specific fields being available in your data. Add additional insights and eliminate missing queries by configuring or sending the required fields. Sending the necessary data populates the templates with more queries. You can also choose to select different fields to populate each query within the **Setup** view. For more information, refer to our [available templates](/observe/boards/templates/#board-templates-at-a-glance) and their required fields. ## Service Map Troubleshoot visualizations in [Service Map](/observe/service-map/). ### My map is empty 1. Confirm that your [Dataset fields in Dataset Definitions](/configure/datasets/definitions/) are defined with Field name values. Service Map generates automatically from trace data sent to Honeycomb and, with the exception of **timestamp**, is based on these defined Tracing fields: | Role | OpenTelemetry/Honeycomb field names | Description | | -------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Service Name | `service.name`, `serviceName`, `service_name` | Name of the instrumented service. Becomes the map node identity. | | Span ID | `trace.span_id`, `id` | Unique identifier of the span. | | Timestamp | `time` (not mapped via Dataset Definitions) | Start time of the span. Places the dependency in the selected time range. With OpenTelemetry, your SDK sets this automatically. If a span arrives without a timestamp, Honeycomb defaults to the time of arrival. | | Span Duration | `duration_ms`, `durationMs` | Amount of time in milliseconds that the span took to complete. Used for latency display. With OpenTelemetry, Honeycomb usually populates `duration_ms` from the span's start and end timestamps during ingest. If you send spans through another path, confirm a duration field is present and mapped. | | Parent Span ID | `trace.parent_id`, `parentId` | Unique identifier of the span's parent span or the call location the current span was called from. Any child span must include a parent ID to be mapped properly; a span with no parent ID is treated as a root span rather than dropped. | | Trace ID | `trace.trace_id`, `traceId` | Unique identifier of the trace that the span belongs to. Groups spans into one trace. | If any of this data is missing or undefined, the map will not display. 2. Check back in a few minutes to confirm if the Service Map has generated. Service Map may take up to a few minutes to generate after you start sending data to Honeycomb. ### My map contains only one disconnected service Make sure you are sending traces that include more than one service. Service Map generates based on trace data sent to Honeycomb and visualize edge connections between services. If your trace includes only one service, that singular service will display as one disconnected node. ### My map contains multiple services, but one service is disconnected Disconnected services have little or no cross-service dependency in the selected window. Common causes: | Cause | What to check | | ------------------------- | ----------------------------------------------------------------------------- | | Incomplete traces | Child spans reference a `trace.parent_id` whose parent span never arrived | | Missing parent IDs | Non-root spans lack `trace.parent_id`, so Honeycomb can't join the call chain | | Same-service-only traffic | Parent and child share the same `service.name`, so no edge is created | | Sampling | Quiet paths weren't selected into the dependency sample | | Gateway collapse | Intermediate services are folded into gateway hops | | Narrow time range | Cross-service calls happened outside the selected window | The diagrams below illustrate the most common cause: an incomplete trace. In a complete trace, each span's `trace.parent_id` resolves to a real parent span, so Honeycomb can walk the full chain and record the cross-service edges. In an incomplete trace, a child span's `trace.parent_id` points to a parent span that never arrived. Honeycomb can't join that child to its parent, so the child appears disconnected, an island, instead of connected to the rest of the trace. Complete trace: ```mermaid theme={} graph TD P[Parent span - Service A] --> C[Child span - Service B] C --> GC[Grandchild span - Service C] ``` Incomplete trace, missing parent: ```mermaid theme={} graph TD M[Missing parent span - never arrived] C2[Child span - Service B] -.->|trace.parent_id points here| M C2 --> GC2[Grandchild span - Service C] ``` #### Validating parent-child linkage in your traces In Query Builder or your own scripts, investigate: * **Required fields present.** Count spans missing `service.name`, `trace.trace_id`, `trace.span_id`, or `duration_ms`. * **Orphaned children.** Spans with `trace.parent_id` set where no span in the same `trace.trace_id` has that ID as `trace.span_id`. * **Cross-service children.** For services that should be connected, confirm child spans exist whose parent span has a different `service.name`. * **Roots that never call out.** Root spans with no different-service descendants appear as disconnected or entry-only nodes. ### A service is missing from my map Confirm the service has spans in the time range. In [Query Builder](/investigate/query/build/), run: ``` VISUALIZE COUNT WHERE service.name = ``` If the returned count is 0, the map is correctly omitting it for that window. If the returned count is greater than zero: * Wait a few minutes for recently ingested traffic to appear * Confirm Dataset Definitions for service name, trace ID, span ID, parent span ID, and duration * Expand the time range to view a larger sample; sampling may hide low-volume services * Check whether the service is classified as a gateway (`net.component: proxy` or `component: proxy` attributes). In [Query Builder](/investigate/query/build/), run: ``` VISUALIZE COUNT WHERE service.name = GROUP BY `component:proxy`, `net.component:proxy` ``` * Confirm the span's timestamp reflects when the call actually happened; clock skew or delayed export can push a span outside the selected time range even though the service has current trace volume ### My instrumented gateways do not appear on my map If you [instrumented gateways](/observe/service-map/#instrumenting-for-gateways) and they still do not appear on your map, your gateway requests may not occur between known services. Service Map only displays gateways on an edge, which is formed when one service send requests to another service. If an edge does not exist, either because a service does not send requests to another service or because the service sends requests to another service through multiple gateways, then gateways may not display on your map. If either of these scenarios are common in your architecture, join #discuss-service-map in our [Pollinators Community Slack](/troubleshoot/community/#join-pollinators-community-slack) and let us know. ### All services point to the gateway service Make sure you have [instrumented for gateways](/observe/service-map/#instrumenting-for-gateways), so Service Map can represent them appropriately. ### The map shows too many services Apply service filters, isolate a service, or filter traces to a journey attribute. If you routinely exceed hundreds of distinct service names, revisit how `service.name` is assigned in instrumentation. ### My map is dense and hard to read Narrow your map to a specific set of services using the [**Services**](/reference/honeycomb-ui/service-map/#select-services) or [**Filter Traces**](/reference/honeycomb-ui/service-map/#filter-traces) dropdowns. After narrowing down to a smaller diagram, hover over a specific service on the map and select **Isolate** to display all dependencies (incoming and outgoing services) for the selected service. ### When I apply filters, an expected service or path does not display When filter parameters are applied to the Service Map, the results return a maximum of 10,000 traces. These traces are then displayed on the Service Map when using Filters and/or selectively displayed when using the Service Filter. Because of the applied limit, sometimes very low volume traces may not be represented in the returned sample. ### When I filter my map, a "No Service Map data found" error appears, but I found a trace that matches the filter set Filters find traces with at least one span that matches all filters you have entered. For example, if the filters of `app.cart.items > 5` AND `app.user.currency = USD` are set in [**Filter Traces**](/reference/honeycomb-ui/service-map/#filter-traces), then the map displays traces that have at least one span that matches both filters. If the fields used in your filters exist across different spans in a trace, a result is not returned. If possible, try propagating context throughout your traces to enable filters to return matching results. ### Sample traces are empty when I select a service The right panel's sample traces for a service are based on dependencies where that service appears as the callee (child) of an edge. You may see a node with no sample traces when: * The service only appears as a caller in the selected window; it calls others, but nothing calls it in the sampled dependencies * Active UI filters exclude all matching traces * Sampling or time range left no child-side dependencies for that service Try: * Clearing trace filters * Widening the time range * Selecting an edge into or out of the service and inspecting edge sample traces * Using Isolate on the service to inspect neighbors ### p95 latency for services and edges in Service Map do not match p95(duration\_ms) in Query Builder This difference is expected. Service Map calculates the `p95` duration for services and edges differently from the `p95(duration_ms)` query in Query Builder. Service Map latency calculations include synchronous spans, which communicate between services. Query Builder latency calculations includes both synchronous and asynchronous, or internal, spans. # Honeycomb Community Source: https://docs.honeycomb.io/troubleshoot/community Expand your network, find opportunities for speaking and writing about observability, and connect on using Honeycomb in the workplace. We call our extended community "the Pollinators". Just as our employees are like the bees working in the hive, our Pollinators spread the practices of observability and bring back local knowledge from their organizations, so we can improve Honeycomb. ## Join Pollinators Community Slack Pollinators Community Slack is a great place to expand your network, to find opportunities for speaking and writing about observability, and to connect on using Honeycomb in the workplace. The primary value of Pollinators is having an opportunity for help with Honeycomb--both from our staff and from other customers, who may have faced challenges similar to yours. We also regularly collect feedback to inform new features, and grant early access to those features. ### Join Pollinators with a Honeycomb Account After you sign up for a Honeycomb account, we will automatically send you an email inviting you to join Pollinators. If you have not received your email invitation, contact [Honeycomb Support](https://www.honeycomb.io/support/). ### Request an invitation without a Honeycomb Account Even if you do not use Honeycomb, you may find value in joining and participating in our Slack channels--community members regularly have conversations about OpenTelemetry implementation. If you would like to join Pollinators without a Honeycomb account, send a request to [Honeycomb Support](https://www.honeycomb.io/support/). ## Connect with us on Social Media Connect with us on Bluesky [@honeycombio](https://bsky.app/profile/honeycomb.io) and on [LinkedIn](https://www.linkedin.com/company/honeycomb.io/). ## Come to DevRel Office Hours Interested in talking observability with our world-class experts? To get some specialized advice, [schedule a 30-minute office hour session](https://www.honeycomb.io/devrel/observability-office-hours/). # Customer Support Source: https://docs.honeycomb.io/troubleshoot/customer-support Get help with Honeycomb through the Support Knowledge Base, the Pollinators Community, or by contacting the Honeycomb support team directly. When you need support for Honeycomb, we have a few different options for you to try! ## Visit our Knowledge Base If you have questions, you can visit our [Honeycomb's Support Knowledge Base](https://support.honeycomb.io/). We try to keep it updated with quick solutions to the questions that our users most ask our support team. ## Join our Community We would love to see you in our Community, which we call Pollinators! Our employees are the bees working in the hive, and our Pollinators spread the practices of observability and bring back local knowledge from their organizations, so that we can improve Honeycomb. Members of Pollinators get help with Honeycomb from our staff and from other customers who may have faced similar challenges. To learn more about Pollinators, including how to join, visit [Honeycomb Community](/troubleshoot/community/). ## Open a Ticket - Pro/Enterprise Plans If you are a Pro/Enterprise user, you can visit [Honeycomb Support](https://support.honeycomb.io/) to create a ticket for our support team, or email [support@honeycomb.io](mailto:support@honeycomb.io). We would love to hear from you! # Early Access & Beta Experiences Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/experimental-features Try experimental features before general release through Honeycomb's Early Access and Beta programs, and give feedback that shapes the final product. Get an early look at features in development. Our Early Access experiences let you try alpha-stage features that are still evolving, while our Beta programs offer more polished experiences that are almost ready for general release. Your feedback at both stages helps shape the final product. ## Current Experiences Try features in development and share feedback to guide their final release. ### Beta As part of our [beta program](/troubleshoot/product-lifecycle/release-stages/#beta), you can explore newly developed product capabilities while providing feedback. Beta experiences are stable enough for everyday use but still evolving. Beta releases may be restricted to a select number of Honeycomb users (private) or open to all Honeycomb users (public). If you would like to participate in a beta, get in touch with your Honeycomb point of contact. If you are not sure who that is, [Honeycomb Support](https://support.honeycomb.io/) can help you connect with them. #### Anomaly Detection **Status:** Beta
**Type:** Public
**Dates:** In progress Anomaly Detection represents a fundamentally different philosophy in Honeycomb. Rather than requiring teams to predict what might go wrong and configure alerts accordingly, our system learns your service patterns and proactively notifies you when behavior deviates from normal. When you combine this with BubbleUp, which shows you the outliers around an anomalous event, you can get a very detailed picture of both the what and why on your way to understanding the root cause. The system is designed with opinionated defaults that work out of the box, while still providing the tuning capabilities teams need. It is optimized for accuracy, ensuring that when you get an alert, it is worth investigating. To learn more, visit [Anomaly Detection](/notify/anomaly-detection/). ### Early Access Try experimental features at the earliest stage of development and help shape them from the ground up. Our [early access program](/troubleshoot/product-lifecycle/release-stages/#early-access) provides alpha-stage features to a small group of users for feedback and validation. Early Access features may be incomplete, change significantly before release, or be removed based on what we learn. Participation is typically by invitation or request. If you're interested in joining an early access program, get in touch with your Honeycomb point of contact. If you are not sure who that is, [Honeycomb Support](https://support.honeycomb.io/) can help you connect with them. ## Past Experiences Explore which features we've tested in the past and how they shaped our product today. ### Beta An archive of our past beta programs. These features were nearly production-ready and were refined based on user feedback before graduating to general availability. #### Activity Logs **Type:** Public
**Dates:** Graduated to general availability Activity Logs give you visibility into changes made to key resources in your environment—whether by a person or an automated process. To learn more, visit [Investigate Team Activity](/configure/teams/investigate-activity/). #### Chat About a Page **Type:** Public
**Dates:** Graduated to general availability Chat about this page lets you ask questions about the Honeycomb page you are on in natural language. To learn more, visit [Chat About a Page](/investigate/canvas/chat-about-a-page/). #### Honeycomb Canvas Slack App **Type:** Public
**Dates:** Graduated to general availability Use Honeycomb Canvas directly in Slack to ask questions of your data, explore your system, and investigate problems without switching contexts. Collaborate with your teammates by mentioning `@Honeycomb Canvas` in any channel. To learn more, visit [Ask Canvas in Slack](/investigate/canvas/slack-app). #### Metrics **Type:** Private
**Dates:** Graduated to general availability Metrics is a major update to Honeycomb's legacy metrics experience, built to align with the OpenTelemetry specification for metric types and behaviors. This upgrade introduces a more expressive query model, advanced temporal functions, and full support for alerting via Triggers. To learn more, visit [Metrics in Honeycomb](/get-started/honeycomb/metrics-in-honeycomb). #### Model Context Protocol (MCP) **Type:** Public
**Dates:** Graduated to general availability The Honeycomb Model Context Protocol (MCP) enables AI assistants to directly interact with your Honeycomb observability data through natural language. This open standard allows seamless communication between AI models and Honeycomb, transforming how you investigate issues and analyze system behavior. To learn more, visit [Honeycomb Model Context Protocol (MCP)](/integrations/mcp/). #### Text Panels on Boards **Type:** Public
**Dates:** Graduated to general availability Text Panels let you add narrative context directly to a board, alongside your queries. Use them to explain what the data means, highlight key insights, or link to runbooks and playbooks so your team knows what action to take. To learn more, visit [What is a Board?: Panels](/investigate/observe/boards#panels). # Migration Process Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/migration-process Understand Honeycomb's migration process and find out what to expect from Honeycomb during a migration. Although we know that deprecations and breaking changes can be disruptive, they are necessary to allow us to upgrade technology, improve security and service quality, and continue to invest in resources that provide the most value for our users. As part of our commitment to transparency, we try to proactively notify users when changes to features or behaviors result in breaking changes that alter their use of Honeycomb. We also try to provide accompanying recommendations for migration and replacement capabilities where available. ## Migration Guides Honeycomb recommended migrations usually involve replacing old functionality with substantially comparable and improved functionality. To help you migrate to new functionality, we publish migration guides, which detail any necessary changes and inform you of any other required actions. To learn more about current migrations that we recommend and access their accompanying resources, visit [Recommended Migrations](/troubleshoot/product-lifecycle/recommended-migrations/). # Recommended Migrations Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations Review Honeycomb's recommended migrations, understand their impact, and access resources to help you migrate. We recommend that customers migrate to the new behaviors listed below. If you have questions, visit our [Support Knowledge Base](/troubleshoot/customer-support/) or [join our Pollinators Community](/troubleshoot/community/). ## Migrate from metrics-as-events to time series metrics Honeycomb is migrating Free and Pro customers from the original metrics-as-events implementation to native time series metrics. Time series metrics store data points in their intended format, which enables more accurate queries, proper support for counters and histograms, and separate billing that does not count against your event allotment. On June 10, 2026, Honeycomb will begin the rollout: * If you are not currently sending metrics as events, Honeycomb moves you directly to time series metrics. No action is required. * If you are on a Free or Pro plan and sending metrics-as-events, your account enters dual ingest mode, during which your metrics will write to both the metrics-as-events dataset and the new time series metrics dataset simultaneously. After 60 days, you will have the same retention window of data written in both formats, but you can start working with your time series metrics today. You have approximately six months to complete your migration. At the close of that window, metrics-as-events will be deprecated for Free and Pro plans. ### Resources * To understand what is changing and how to update your queries, Boards, and Triggers, visit the [Migrate to Time Series Metrics FAQ](/troubleshoot/product-lifecycle/recommended-migrations/migrate-to-time-series-metrics-faq). * To learn about time series metrics and how they work in Honeycomb, visit [Metrics in Honeycomb](/get-started/honeycomb/metrics-in-honeycomb). ## Migrate to Refinery 3.0 If you use Refinery, we recommend that you migrate to Refinery 3.0 or later to take advantage of new and improved Refinery features. Benefits of upgrading your Refinery instance from 2.0 to 3.0 include: * Efficient CPU and memory utilization * Enhanced cluster stability and reliability ### Resources To learn how to migrate to Refinery 3.0 and later, visit [Upgrading Refinery](/troubleshoot/product-lifecycle/recommended-migrations/upgrade-refinery/). ## Migrate to Flexible Boards Flexible Boards are a new Honeycomb Board format that supports dynamic panel resizing and rearrangement. To take advantage of these improvements and ensure your Boards continue to work as expected, we recommend migrating all legacy Boards to the Flexible Boards format. On August 15, 2025, the Honeycomb API will stop supporting legacy Boards. Any [Board API](/api/boards/) calls that use `"type": "classic"` or omit the `type` field will fail. By August 29, 2025, Honeycomb will automatically migrate any remaining legacy Boards to the Flexible Boards format. During this process, query captions will move into the corresponding query's description field. If you manage Boards using the Honeycomb UI, Honeycomb API or Terraform, update your configurations to use the Flexible Board format. You can continue managing Boards in your preferred tool after completing the migration. ### Resources To learn how to convert your legacy Boards to Flexible Boards, visit [Migrate to Flexible Boards](/troubleshoot/product-lifecycle/recommended-migrations/migrate-to-flexible-boards/). ## Migrate from FID to INP in the Honeycomb Web Instrumentation package The [Honeycomb Web Instrumentation package](/send-data/javascript-browser/) for the Honeycomb OpenTelemetry Web SDK now uses `web-vital` v5.0.0, which removes support for the `fid` (First Input Delay) vital. To continue capturing meaningful responsiveness metrics, we recommend switching to `inp` (Interaction to Next Paint). While FID only measures the delay before the browser processes a user's first interaction, INP captures the latency of the slowest qualifying interaction on the page. It provides a more complete view of user-perceived responsiveness over the full page lifecycle. ### Configuration changes To migrate, replace `fid` with `inp` in your `WebVitalsInstrumentationConfig`. The configuration structure remains the same, though `inp` uses a slightly different type: `VitalOptsWithTimings`, which includes additional timing details. | From | To | Type | Description | | --------------------------- | --------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fid` | `inp` | `VitalOpts` to `VitalOptsWithTimings` | Configuration options passed through to `web-vitals`. Visit Google Chrome's [ReportOpts](https://github.com/GoogleChrome/web-vitals?tab=readme-ov-file#reportopts). | | `fid.applyCustomAttributes` | `inp.applyCustomAttributes` | `function` to `function` | Function that adds custom attributes to core web vitals spans. | ### Deprecated FID attributes If you’re still using FID, these are the span attributes it produces. Each has a direct INP equivalent with a similar structure. | Attribute | Description | Example | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | `fid.id` | Unique identifier for the metric. | `v4-1724856546003-8941918093361` | | `fid.delta` | Change in value since the last report. | `100` | | `fid.value` | Time in milliseconds between a user's first interaction with the page and when the browser is able to begin processing event handlers for that interaction. | `100` | | `fid.rating` | Performance rating based on thresholds. | `good`, `needs-improvement`, `poor` | | `fid.navigation_type` | Type of navigation that triggered the page load. | `restore`, `navigate`, `reload`, `back-forward`, `prerender`, `back-forward-cache` | | `fid.element` | CSS selector of the element the user interacted with. The element will be the `target` of the dispatched `event`. | `button#submit` | | `fid.event_type` | Type of interaction event. | `click` | | `fid.load_state` | Page load state at the time of interaction. | `loading`, `dom-content-loading`, `complete`, `dom-interactive` | ## Migrate from Global Search As of January 2, 2025, the Global Search feature, which returns results from multiple Honeycomb features in one interface, is no longer available. Instead of using Global Search, you can search for specific terms using the search box in each feature's interface for the following Honeycomb features: * Boards with Team permissions - Navigate to Boards using the left navigation menu. * Datasets - Navigate to the Datasets list. * Dataset fields - Navigate to the dataset's Dataset Settings > Schema tab > Unique Fields section. * Calculated Fields - Navigate to the dataset's Dataset Settings > Schema tab > Calculated Fields section. * Environments - Navigate to the Environments page. * Triggers: Navigate to Triggers using the left navigation menu. * SLOs: Navigate to SLOs using the left navigation menu. ## Migrate from Honeycomb OpenTelemetry distribution to OpenTelemetry If you are using a Honeycomb OpenTelemetry Distribution, an older set of Honeycomb wrappers for instrumenting code with OpenTelemetry, we recommend migrating your instrumentation to the official [OpenTelemetry](/send-data/opentelemetry/) tooling that is supported by many vendors, including Honeycomb. Honeycomb OpenTelemetry Distributions have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. If you are not yet ready to migrate to OpenTelemetry, we recommend instrumenting any new observability efforts with OpenTelemetry while maintaining your old code instrumented with Honeycomb OpenTelemetry Distribution. ### Resources To learn how to migrate from Honeycomb OpenTelemetry Distribution to OpenTelemetry, visit [Migrate from Honeycomb OpenTelemetry Distribution to OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-distributions/). To learn more about how Honeycomb OpenTelemetry Distributions were installed, configured, and used, visit the appropriate reference: * [Honeycomb OpenTelemetry Distribution for Go Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-distributions/go/honeycomb-distribution/) * [Honeycomb OpenTelemetry Distribution for Java Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-distributions/java/honeycomb-distribution/) * [Honeycomb OpenTelemetry Distribution for Node.js Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-distributions/javascript-nodejs/honeycomb-distribution/) * [Honeycomb OpenTelemetry Distribution for Python Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-distributions/python/honeycomb-distribution/) * [Honeycomb OpenTelemetry Distribution for .NET Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-distributions/dotnet/honeycomb-distribution/) ## Migrate to stabilized OpenTelemetry semantic conventions OpenTelemetry HTTP semantic conventions (SemConv) became stable in 2023. As part of the breaking changes related to stabilization, 17 attributes in the HTTP semantic conventions were renamed (for example, `http.status_code` changed to `http.request.status_code`). These changes are likely to impact several areas within Honeycomb. We recommend that you review the complete list of attribute changes in OpenTelemetry's [Summary of Changes](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/non-normative/http-migration.md#summary-of-changes). With these attribute changes in mind, we also recommend that you review your Honeycomb: * SLOs * Triggers * Calculated Fields * Boards and saved queries * Refinery rule conditions and field lists * OpenTelemetry Collector configurations If any of these elements depend on attributes with HTTP semantic conventions, then you will be impacted by these breaking changes. ### Resources To find impacted instrumentation packages, as published under the OpenTelemetry GitHub organization, and the current status of their compliance with the stabilized HTTP semantic conventions, visit [OpenTelemetry HTTP Semantic Conventions Compatibility](/troubleshoot/product-lifecycle/recommended-migrations/migrate-to-opentelemetry-semantic-conventions/compatibility/). To learn how to migrate to stabilized OpenTelemetry semantic conventions, visit [Migrate to Stabilized OpenTelemetry Semantic Conventions](/troubleshoot/product-lifecycle/recommended-migrations/migrate-to-opentelemetry-semantic-conventions/). ## Migrate from Honeycomb Kubernetes Agent If you are using the Honeycomb Kubernetes Agent, we recommend that you choose a new path from our [Kubernetes Overview](/send-data/kubernetes/) and migrate accordingly. The Honeycomb Kubernetes Agent is in maintenance, and we will notify users when a deprecation date is set. ### Resources To learn more about how the Honeycomb Kubernetes Agent was installed, configured, and used, visit [Honeycomb Kubernetes Agent Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-honeycomb-kubernetes-agent/honeycomb-kubernetes-agent/). ## Migrate from Beelines to OpenTelemetry If you are using Beelines, an older set of Honeycomb SDKs for instrumenting code, we recommend migrating your instrumentation to [OpenTelemetry](/send-data/opentelemetry/), a cross-industry standard that is supported by many vendors, including Honeycomb. Beelines have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. If you are not yet ready to migrate to OpenTelemetry, we recommend instrumenting any new observability efforts with OpenTelemetry while maintaining your old code instrumented with Beelines. Beelines and OpenTelemetry work well in a mixed environment and using both will allow you to get trace visualizations that include them both. ### Resources To learn how to migrate from Beelines to OpenTelemetry, visit [Migrate from Beelines to OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/). To learn more about how Beelines were installed, configured, and used, visit the appropriate reference: * [Beeline for Go Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/go/) * [Beeline for Java Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/java/) * [Beeline for Node.js Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/nodejs/) * [Beeline for Python Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/python/) * [Beeline for Ruby Reference](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/ruby/) ## Migrate from Honeycomb Classic to Honeycomb Environments If you are using Honeycomb Classic, we recommend that you migrate to Honeycomb Environments and Services, so you can take advantage of its expanded data model that includes environments and services groupable, relational structures. Benefits of migrating from Honeycomb Classic to Honeycomb Environments include: * Simplified, organized dataset schemas : The groupable, relational structures of datasets in an Environment allow for focused datasets with relevant fields and values rather than an all-in-one dataset. Each service in an Environment gets its own Dataset, and each Dataset is less likely to exceed the 20,000 field limit. * Query across all datasets or a specific dataset : [Query](/investigate/query/build/) all the datasets in the Environment, as if it were in a single Dataset, or scope your query to a specific Dataset. The Query Builder provides auto-complete suggestions based on the scope of the query. * Scale and add new services : Services added to an Environment create new Datasets instead of impacting existing ones. * Dataset-specific Home Page : A dataset dedicated to one service presents relevant information on its [Home page](/observe/honeycomb-home/) as the [Dataset Definitions](/configure/datasets/definitions/) map more meaningfully to the service's fields. * Define Markers for an Environment : Annotate an important moment across all of its Datasets with a [Marker](/configure/environments/manage-markers/). For example, use a Marker to indicate a new deployment in the Production Environment. * Better security : [API keys](/configure/environments/manage-api-keys/) are scoped per Environment instead of per Team, which allows securing critical instances, like your Production environment. Future product updates will support only the Environments and Services model. For example, [Service Map](/observe/service-map/) only supports datasets within an Environment. ### Am I using Honeycomb Classic? If your Honeycomb team existed before the [Honeycomb Environments and Services change](https://changelog.honeycomb.io/preview-new-environments-and-services-functionality-in-honeycomb!-227356), you have a Classic environment. To be sure, look for a label below Honeycomb logo in the Honeycomb UI. Honeycomb Classic environments have a **Classic** label with a gray background. The environment selector set to Classic (screenshot)

The environment selector set to Classic (screenshot)

## Migrate from meta.span\_type in span events and attributes The use of the field `meta.span_type` to identify Span Events and Span Links (hereafter called Span Annotations) was removed as of **2020-09-30**. Note that this only applies to using the field `meta.span_type` to identify Span Annotations. This change only applies to you if you are using the field named `meta.span_type` for annotations. If you are using the field for your own use, you can disregard this. ### What this change means Datasets that use the field `meta.span_type` to identify Span Annotations will continue to work as they do now. Teams are encouraged to migrate their systems to use `meta.annotation_type` to identify Span Annotations. Honeycomb will no longer provide feature enhancements or bug fixes for Span Annotations that use `meta.span_type` as their identifying field. ### How to migrate your instrumentation if you use span annotations If you do not use Span Annotations, no migration is required. #### OpenTelemetry Exporter users Upgrade to the latest version of your OpenTelemetry Exporter. #### Users of other instrumentation types For example, Beelines, OpenTracing, and so on. Update your instrumentation code to replace the field name `meta.span_type` with the field name `meta.annotation_type`. OR If you use your own field name to identify Span Annotations, you do not need to change your instrumentation code. Instead: 1. Go to your Dataset Settings > Definitions tab 2. Set the field `Metadata: Annotation Type` to your field name. 3. Set the field `Metadata: Kind` to blank by clicking the X to remove assignment on this field. 4. Save your changes. Read more on the [Definitions tab](/configure/datasets/definitions/). # Environments & Services Migration FAQ Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations/environments-and-services-in-place Learn what changes when Honeycomb migrates your existing Classic environment into an Environments & Services (E&S) environment. ## What is an in-place migration? An in-place migration converts your existing Classic environment into an Environments & Services (E\&S) environment through behind-the-scenes, database-level changes. It unlocks all of the newer E\&S-gated features without requiring you to re-instrument, re-send data, or restructure anything. Everything keeps working the way it does today - you simply gain access to the new capabilities. This is different from the older E\&S migration path, which involved dual-sending data and rebuilding your SLOs, triggers, and boards in a new environment. The in-place approach skips all of that. ## Is there any downtime, and do I need to do anything? No downtime, and no action required on your side. There's no dual ingest, no data backfill, and nothing to reconfigure. Your data stays exactly where it is - nothing moves or is copied. The migration itself typically takes less than a minute. ## What stays the same? Everything you rely on today continues to work, untouched: * All existing datasets remain exactly as they are * All boards, SLOs, triggers, and saved queries stay in place and keep pointing at the same data * All API keys remain unchanged — no need to rotate keys or update your collector, SDKs, or Terraform * All existing permalinks are automatically redirected to the new environment (for very large teams this redirect can take a little longer to fully propagate) ## What changes, and what do I gain? The main visible change is that your environment is no longer labeled "Classic" - we give it a new name of your choosing during the migration. The big win is access to E\&S-gated features, including: * Honeycomb Intelligence * Canvas and the MCP server * Service Maps * Relational fields * Environment-wide queries To use certain AI-powered features, you may be prompted to accept the Honeycomb Intelligence terms of service. ## How is my data routed into datasets after migration? This is the one important behavioral detail to understand. In a standard E\&S environment, the `service.name` attribute on your telemetry determines which dataset data lands in. In an in-place migrated environment, your existing routing is preserved: the `x-honeycomb-dataset` header (or the dataset specified in your API path) still takes priority. In practice this means your dataset structure stays exactly as it is today. Note: any *new* environments you create after migrating will behave like normal E\&S environments (routing by `service.name`). Only this migrated environment keeps the header-priority behavior. ## Can I test it first, or migrate just a few datasets? The migration applies to the whole Classic environment at once - individual datasets can't be migrated one at a time. However, there's a clean rollback available, so if you do find any issues after being migrated, you can work with either support or your account team to roll back the change and we will work with engineering to fix the source of the problem before attempting the migration again. We have used this migration tooling for some of our largest customers and had no issues. We have very high confidence in the effectiveness and minimal impact of this process. ## Is there a rollback if something goes wrong? Yes. There's a rollback function that cleanly reverts the environment back to Classic. Because the migration is a set of database-level changes (and doesn't touch your stored telemetry), rolling back is just as quick as the original migration. ## What happens to my metrics? A few things worth knowing: * Metrics have always required the dataset header for routing — they've never used `service.name` — so metrics routing is unaffected by the migration. * Customers are getting communications on migrating to Time Series Metrics instead of Metrics as Events. That is not a part of this Classic environment upgrade. Time Series Metrics will be sent to the `Metrics` dataset, while Metrics as Events will be sent wherever the `x-honeycomb-dataset` header indicates. * With Time Series enabled, OTLP metrics route into a dedicated metrics dataset regardless of any header. If you have multiple environments (dev, QA, prod) feeding metrics, they'll share that dataset, **so make sure your metrics carry resource attributes that let you filter by environment or service.** ## Will Service Map still work? Yes. Service Map is driven by your dataset's service name definition (the attribute you map in your dataset settings), not by the dataset header or your dataset list. As long as that definition is in place, Service Map works as expected. ## Will Terraform or my API tokens need changes? No. API keys don't change, and Terraform continues to use the same keys to identify the environment. You'd only need new keys if you later decide to route telemetry into a brand-new environment. ## What if I want to fully adopt the E\&S model (a dataset per service) later? That's entirely optional and can be done on your own timeline after migrating. The typical path is: 1. Stand up separate new environments for Production, dev, QA, staging, etc… 2. Update API keys to send environment appropriate data to the new environments 3. Recreate any SLOs, triggers, and boards to point at the new per-service datasets. 1. Triggers and Board Queries can be environment wide to make it easier to recreate these in a standard E\&S environment 2. SLOs can have as many as 10 datasets attached to them, to target them at the appropriate services for each SLO. There's no requirement to ever do this — the migration gives you all E\&S features permanently regardless. ## Build boards with queries spanning multiple environment-scoped datasets In-place migrated environments keep the ability to build boards with queries spanning multiple environment-scoped datasets — for example, comparing QA against prod on a single board. This is a capability many standard E\&S customers have asked for. Questions about your specific environment? Please reach out to Support or your Honeycomb Account Team for assistance. # Migrate from Beelines to OpenTelemetry Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines Replace Honeycomb Beelines with OpenTelemetry instrumentation to avoid end-of-life SDKs and gain vendor-agnostic, standards-based telemetry collection. If you are using Beelines, an older set of Honeycomb SDKs for instrumenting code, we recommend migrating your instrumentation to [OpenTelemetry](/send-data/opentelemetry/), a cross-industry standard that is supported by many vendors, including Honeycomb. Beelines have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. ## Beelines vs. OpenTelemetry OpenTelemetry is a cross-industry standard that is supported by many vendors, including Honeycomb. Data generated by OpenTelemetry can be used with Honeycomb and other OpenTelemetry-compatible tools. Honeycomb Beelines were written before OpenTelemetry. They use a format and API that can only be used with Honeycomb. While we recommend migration all of your instrumentation to OpenTelemetry, you can instrument some services in the same environment with Beelines and some with OpenTelemetry. You will need to keep some things in mind regarding tracing interoperability, though. To learn more, check out [Mixing Beelines and OpenTelemetry](#mixing-beelines-and-opentelemetry). ## Migrating to OpenTelemetry You can incrementally migrate your services from Beeline to OpenTelemetry. At a high level, this will require that you configure each Beeline instance to use the W3C trace header format, and then migrate to OpenTelemetry instrumentation. Using the W3C trace header format lets Beeline and OpenTelemetry instrumentation share trace context at the service/process level, which allows you to migrate one service at a time rather than rewriting everything at once. To maintain continuity of your traces, you must complete each step across all services before moving on to the next step. For example, you can complete the first step for each service individually, but you must finish the first step for all of your services before moving on to the second step. If you have already configured every service to use W3C headers via [Interoperability with OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/go/#interoperability-with-opentelemetry), go directly to the final step of migrating instrumentation to OpenTelemetry. 1. Upgrade to Beeline [version 1.4.0](https://github.com/honeycombio/beeline-go/releases/tag/v1.4.0) or above. This version of Beeline will parse trace context from incoming headers by default. The order of preference when parsing these headers is: parser hook > honeycomb format > W3C format. 2. Configure each Beeline to propagate trace context using the W3C format. You can do this by [configuring a propagation hook](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/go/#propagation-hook). 3. Migrate instrumentation to OpenTelemetry, paying attention to the information in [Mixing Beelines and OpenTelemetry](#mixing-beelines-and-opentelemetry). If you have already configured every service to use W3C headers via [Interoperability with OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/java/#interoperability-with-opentelemetry), go directly to the final step of migrating instrumentation to OpenTelemetry. 1. Upgrade to Beeline [version 1.7.0](https://github.com/honeycombio/beeline-java/releases/tag/v1.7.0) or above. This version of Beeline will parse trace context from incoming headers by default. The order of preference when parsing these headers is: parser hook > honeycomb format > W3C format. 2. Configure each Beeline to propagate trace context using the W3C format. You can do this by [configuring a propagation hook](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/java/#custom-trace-propagation-hook). 3. Migrate instrumentation to OpenTelemetry, paying attention to the information in [Mixing Beelines and OpenTelemetry](#mixing-beelines-and-opentelemetry). If you have already configured every service to use W3C headers via [Interoperability with OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/nodejs/#interoperability-with-opentelemetry), go directly to the final step of migrating instrumentation to OpenTelemetry. 1. Upgrade to Beeline [version 3.2.2](https://github.com/honeycombio/beeline-nodejs/releases/tag/v3.2.2) or above. This version of Beeline will parse trace context from incoming headers by default. The order of preference when parsing these headers is: parser hook > honeycomb format > W3C format. 2. Configure each Beeline to propagate trace context using the W3C format. ```javascript theme={} const beeline = require('honeycomb-beeline'); beeline({ writeKey: 'YOUR_API_KEY', serviceName: 'your-service-name', httpTracePropagationHook: beeline.w3c.httpTracePropagationHook }); ``` 3. Migrate instrumentation to OpenTelemetry, paying attention to the information in [Mixing Beelines and OpenTelemetry](#mixing-beelines-and-opentelemetry). If you have already configured every service to use W3C headers via [Interoperability with OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/python/#interoperability-with-opentelemetry), go directly to the final step of migrating instrumentation to OpenTelemetry. 1. Upgrade to Beeline [version 2.18.0](https://github.com/honeycombio/beeline-python/releases/tag/v2.18.0) or above. This version of Beeline will parse trace context from incoming headers by default. The order of preference when parsing these headers is: parser hook > honeycomb format > W3C format. 2. Configure each Beeline to propagate trace context using the W3C format. ```python theme={} import beeline import beeline.propagation.w3c as w3c beeline.init( writekey='YOUR_API_KEY', service_name='YOUR_SERVICE', debug=True, http_trace_propagation_hook=w3c.http_trace_propagation_hook ) ``` 3. Migrate instrumentation to OpenTelemetry, paying attention to the information in [Mixing Beelines and OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/#mixing-beelines-and-opentelemetry). If you have already configured every service to use W3C headers via [Interoperability with OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/ruby/#interoperability-with-opentelemetry), go directly to the final step of migrating instrumentation to OpenTelemetry. 1. Upgrade to Beeline [version 2.8.0](https://github.com/honeycombio/beeline-ruby/releases/tag/v2.8.0) or above. This version of Beeline will parse trace context from incoming headers by default. The order of preference when parsing these headers is: parser hook > honeycomb format > W3C format. 2. Configure each Beeline to propagate trace context using the W3C format. ```ruby theme={} require "honeycomb-beeline" require "honeycomb/propagation/w3c" Honeycomb.configure do |config| ... config.http_trace_propagation_hook do |env, context| Honeycomb::W3CPropagation::MarshalTraceContext.parse_faraday_env env, context end ... end ``` 3. Migrate instrumentation to OpenTelemetry, paying attention to the information in [Mixing Beelines and OpenTelemetry](#mixing-beelines-and-opentelemetry). ## Mixing Beelines and OpenTelemetry If you are already using Beelines to instrument some services, but need to instrument a new service, you do not need to continue with Beelines. You can instrument any new services with OpenTelemetry. We call this a mixed environment, and you can get trace visualizations from both Beeline and OpenTelemetry instrumentation. You will need to keep some things in mind regarding tracing interoperability, though. ### Tracing Interoperability Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers. The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used. To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services. Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed. To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration. You can see an [example of this configuration in the Go Beeline](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/go/#interoperability-with-opentelemetry). To learn more about ongoing development for W3C trace context propagation, visit the [W3C Trace Context GitHub repository](https://github.com/w3c/trace-context/). ### Do Not Mix Beelines and OpenTelemetry Instrumentation in The Same Process If you instrument each service or process with OpenTelemetry or a Beeline, then your data will be coherent. The granularity with which you can mix Beelines and OpenTelemetry without additional, more advanced work, is at the process- or service-level. Do not mix them in the same process, as this will result in bad data unless you adopt a wrapper and a strategy for incrementally switching your instrumentation calls to OpenTelemetry. ### Field Name Differences The names of fields differ on events produced by the Beeline instrumentations and the OpenTelemetry instrumentations. For example, the Beelines' `service_name` field has an equivalent of `service.name` in [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/semantic-conventions.md) events. To use a single column for this value in queries for the name of a service, [define a calculated field](/configure/datasets/calculated-fields/) in the dataset and use the [`COALESCE`](/reference/calculated-field-expression/conditional/#coalesce) function to determine the name: ```honeycomb theme={} COALESCE($service.name, $service_name, "unknown") ``` ### Reserved `type` Field When Honeycomb ingests OpenTelemetry data, it automatically creates a `type` field that records the [span kind](https://opentelemetry.io/docs/concepts/signals/traces/#span-kind) value (for example, `server`, `client`, `producer`, `consumer`, or `internal`). If your Beeline instrumentation already sends a custom field named `type`, it will collide with Honeycomb's `type` field after you migrate to OpenTelemetry. The `type` field will contain a mix of your custom values and span kind values, making it unreliable for filtering or grouping. To avoid this collision, rename your custom `type` field to a different name before or during your migration. For example, if you use `type` to represent a request type, rename it to something like `request.type` or `request_type` in your OpenTelemetry instrumentation. After migrating, you can use a [calculated field](/configure/datasets/calculated-fields/) with [`COALESCE`](/reference/calculated-field-expression/conditional/#coalesce) to unify historical Beeline data and new OpenTelemetry data: ```txt theme={} COALESCE($request.type, $type, "unknown") ``` ### Sampling To achieve trace-aware sampling across a mixed environment, all instrumented services need to [send data to Honeycomb via Refinery](/manage-data-volume/sample/honeycomb-refinery/). ## Questions and Support Do you still have questions about OpenTelemetry and Honeycomb Beelines, or have a specific use case for your architecture? The [Honeycomb Pollinators Community](/troubleshoot/community/) is a great place to ask your questions and share your experience. # Beeline for Go Reference Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/go Reference documentation for Beeline for Go, Honeycomb's legacy auto-instrumentation SDK. Beelines reached end of life on August 12, 2025. Migrate to OpenTelemetry. Beelines have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. * Adding new instrumentation? We recommend that you use [OpenTelemetry for Go](/send-data/go/) instead. * Already using a Beeline? We recommend that you [migrate to OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/). * Concerned about mixing Beelines and OpenTelemetry? Learn more about [Mixing Beelines and OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/#mixing-beelines-and-opentelemetry). The Go Beeline for Honeycomb is quick and easy way to instrument your Go application. It includes several optional wrappers that automatically instrument HTTP requests and database queries. It also supports tracing out of the box, linking database queries to the HTTP request from which they originated. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey you may find you would like to add new events or traces to add more details specific to your app. The Go Beeline provides simple interfaces for adding both. To see an example of the Go Beeline in action, try one of the [Example Apps](https://github.com/honeycombio/beeline-go/tree/main/examples). If you would like to see more options in the Go Beeline, please [file an issue](https://github.com/honeycombio/beeline-go/issues) or vote up an already filed one! If you prefer to use only Structured Logs, check out our [Libhoney library](/send-data/go/libhoney/). ## Requirements * A Go app, preferably one that listens for HTTP requests or makes SQL calls * [Go](https://golang.org/doc/install) 1.18+ and [uuid](https://github.com/google/uuid) 0.2+ * A Honeycomb API key You can [find your API key](/configure/environments/manage-api-keys/#find-api-keys) in your Environment Settings. If you do not have an API key yet, sign up for a [free Honeycomb account](https://ui.honeycomb.io/signup). ## Quick Installation If using the **dataset-only** data model, refer to the **Honeycomb Classic** tab for instructions. Not sure? Learn more about [Honeycomb versus Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments). The Go Beeline has rich automatic instrumentation for HTTP applications. To install the Go Beeline for your application: 1. Fetch the `beeline-go` package: ```shell theme={} go mod init go get github.com/honeycombio/beeline-go ``` 2. In your code, import the `beeline-go` package and initialize it with your API key and service name: ```go theme={} import beeline "github.com/honeycombio/beeline-go" func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", // The name of your app/service is a good choice ServiceName: "MyGoApp", }) defer beeline.Close() ... } ``` 3. Add a Beeline wrapper in your `http.ListenAndServe` call. This adds basic instrumentation for each request at the outermost layer of the call stack. ```go theme={} import "github.com/honeycombio/beeline-go/wrappers/hnynethttp" ... http.ListenAndServe(":8080", hnynethttp.WrapHandler(muxer)) ``` The Go Beeline has rich automatic instrumentation for HTTP applications. To install the Go Beeline for your application: 1. Fetch the `beeline-go` package: ```shell theme={} go mod init go get github.com/honeycombio/beeline-go ``` 2. In your code, import the `beeline-go` package and initialize it with your API key and dataset name: ```go theme={} import beeline "github.com/honeycombio/beeline-go" func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", // The name of your app is a good choice to start with Dataset: "MyGoApp", }) defer beeline.Close() ... } ``` 3. Add a Beeline wrapper in your `http.ListenAndServe` call. This adds basic instrumentation for each request at the outermost layer of the call stack. ```go theme={} import "github.com/honeycombio/beeline-go/wrappers/hnynethttp" ... http.ListenAndServe(":8080", hnynethttp.WrapHandler(muxer)) ``` 4. Augment the data with interesting information from your app and extra attributes, such as errors, so that you can see rich information about your app in Honeycomb. The `context.Context` (containing the automatically constructed HTTP event) to use should be present on the request if the Beeline HTTP wrapper was used. ```go theme={} func handleEndpoint(w http.ResponseWriter, r *http.Request) { ctx := r.Context() if err := beeline.AddField(ctx, "user_id", userIDFromSession(r)); err != nil { beeline.AddField(ctx, "get_session_error", err) } // ... do the rest of work for the request ... } ``` 5. Add additional spans and turn a series of events into a trace: ```go theme={} func slowOp(ctx context.Context) { ctx, span := beeline.StartSpan(ctx, "slowOp") defer span.Send() // ... go on and do the slow opp, add more data along the way beeline.AddField(ctx, "interesting_thing", "banana") } ``` 6. Make sure to call `beeline.Close()` to send any pending events to Honeycomb before your program exits. In this example, we are using [`hnynethttp`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnynethttp). See the [Optional configuration](#optional-configuration) section for more about the available wrappers. ## Adding Fields to Events The middleware wrapper creates a Honeycomb event in the request context as a span in the overall trace. This span exists throughout the request's lifecycle, allowing you to add as many additional custom fields as you like. Here is an example of adding a custom field: ```go theme={} func calcBigNum(ctx context.Context) { // ... do work, find a BigNum beeline.AddField(ctx, "big_num", result) // ... do thing with BigNum, maybe something fails if err != nil { beeline.AddField(ctx, "error", err) } } ``` Additional fields are added under the `app.` namespace. For example, the field above would appear in your event as `app.big_num`. The namespace groups your fields together to make them easy to find and examine. If you prefer to avoid the `app.` namespace, for example in order to overwrite an automatically-populated field, you can extract the current `span` instance and call `AddField` directly on that instance. ```go theme={} import "github.com/honeycombio/beeline-go/trace" func calcOtherBigNum(ctx context.Context) { // ... do work, find another BigNum span := trace.GetSpanFromContext(r.Context()) span.AddField(ctx, "handler.name", "calc_other_big_num") } ``` These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per-user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred. It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user's ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request. If you are interested in adding custom fields to all spans, use `AddFieldToTrace` instead. `AddFieldToTrace` adds the field to both the currently active span and all other spans that have yet to be sent and are involved in this trace that occur within this process. Additionally, these fields are packaged up and passed along to downstream processes if they are also using a Beeline. This function is good for adding fields that is better scoped to the request than this specific unit of work. For example: user IDs, globally relevant feature flags, errors, and so on. Fields added here are also prefixed with `app.` ## Adding Spans to a Trace We encourage people to think about instrumentation in terms of "units of work". As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder. Adding spans with the Go Beeline is easy! Here is an example, where calls to `slowOp` get their own span within the trace: ```go theme={} func slowOp(ctx context.Context) { ctx, span := beeline.StartSpan(ctx, "slowOp") defer span.Send() // ... go on and do the slow opp, add more data along the way beeline.AddField(ctx, "interesting_thing", "banana") } ``` Spans always get a few fields: * a name - in this case `slowOp` * a duration - how much time elapsed between when the span was started and sent * a service\_name - generally configured during Beeline initialization * several IDs - trace, span, and parent identifiers (UUIDs) You are always welcome (and encouraged!) to add additional fields to spans using the `beeline.AddField` function. ## Wrappers and Other Middleware After the router has parsed the request, more fields specific to that router are available, such as the specific handler matched or any request parameters that might be attached to the URL. You can use a wrapper to capture additional fields specific to a particular type of request. The available wrappers are listed below. You can find additional information and detailed instructions in the GoDoc links for each subpackage. ### HTTP Wrappers * net/http: [`hnynethttp`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnynethttp) * goji/mux: [`hnygoji`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnygoji) (Goji v2.0+) * gorilla/mux: [`hnygorilla`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnygorilla) * httprouter: [`hnyhttprouter`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnyhttprouter) ### Database Wrappers * database/sql: [`hnysql`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnysql) * github.com/jmoiron/sqlx: [`hnysqlx`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnysqlx) * github.com/gobuffalo/pop: [`pop`](https://godoc.org/github.com/honeycombio/beeline-go/wrappers/hnypop) ## Optional Configuration If you would like to use another framework that supports middleware, you may be able to adapt one of the standard wrappers. We recommend starting with the [hnynethttp](https://github.com/honeycombio/beeline-go/tree/master/wrappers/hnynethttp) wrapper, as it expects a function that takes a `http.Handler` and returns a `http.Handler`. In order to have traces connect HTTP wrapped packages all the way down to the database (using the sql or sqlx wrappers), you must pass the context from the `*http.Request` through to the SQL package using the appropriate Context-enabled function calls. This context ties the SQL calls back to specific HTTP requests, so you can include additional details such as how much time was spent in the DB. It also connects the request IDs from separate events so you can see exactly which DB calls were triggered by a given event. For very high throughput services, you can send only a portion of the events flowing through your service by setting the `SampleRate` during initialization. This sample rate will send 1/n events, so a sample rate of `5` would send 20% of all events. For high throughput services, a sample rate of `100` is a good start. ### Augmenting or Scrubbing Spans If you have some transformations you would like to make on every span before it leaves the process for Honeycomb, the `PresendHook` is your opportunity to make these changes. Examples are scrubbing sensitive data -- for example, you may want to ensure specific fields are dropped or hashed -- or augmenting data -- for example, making out-of-band translations between an ID and a more human readable version. Similar to the `SamplerHook` discussed below, you pass the `PresendHook` a function that will be called on every span with the fields of that span as an argument. The function is free to mutate the map passed in and those mutations will be what finally gets sent to Honeycomb. As an example, say we have some database calls that insert secret values into a table. Normally, we want to see all the query arguments and DB queries that come from our instrumented `sql` package, but we specifically want to protect things that go into the secrets table. This code will examine all spans just before they are sent to Honeycomb and replace the arguments to any SQL statement that involves the `secrets` table with a `sha1` hash of those arguments. ```go theme={} // hashSecretsArguments will replace the query arguments to SQL queries against // the secrets table with a hash of those arguments, so that the contents of the // secrets inserted into a table are not passed along. func hashSecretsArguments(fields map[string]interface{}) { // if the span is a DB span and has a query if query, ok := fields["db.query"]; ok { // and the query accesses the secrets table if strings.Contains(query, "secrets") { // scrub the arguments passed in to the query // replace them with a sha1 hash of the original args args, ok := fields["db.query_args"]; ok { hashedArgs := sha1.Sum([]byte(args)) fields["db.query_args"] = hashedArgs } } } } func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", ServiceName: "MyGoApp", Debug: true, PresendHook: hashSecretsArguments, }) } ``` ## Sampling Events To [sample](/manage-data-volume/sample/guidelines/) a portion of events for very high throughput services, include an integer `SampleRate` in the initialization of the Go Beeline. This sends 1/n of all events, so a sample rate of `5` would send 20% of your events. ```go theme={} func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", ServiceName:"MyGoApp", Debug: true, SampleRate: 5, // This sends 1/5 (20%) of events }) } ``` The value of `SampleRate` **must** be a positive integer. Sampling is performed by default on a per-trace level in the Go Beeline, so adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent. The Honeycomb engine [weights query results based on sample rate](/manage-data-volume/sample/sampled-data-in-honeycomb/) to ensure that sampled computations return correct results. ### Customizing Sampling Logic Our Beeline lets you define a [`SamplerHook`](https://godoc.org/github.com/honeycombio/beeline-go#Config.SamplerHook) in order to customize the logic used for deterministic per-trace sampling. For example, assume you have instrumented an HTTP server. You would like to keep all requests to login, skip all health checks, and sample the rest at a default rate. You could define a sampler function like so: ```go theme={} import ( "crypto/sha1" "math" ) // Deterministic shouldSample taken from https://github.com/honeycombio/beeline-go/blob/7df4c61d91994bd39cc4c458c2e4cc3c0be007e7/sample/deterministic_sampler.go#L55-L57 func shouldSample(traceId string, sampleRate int) bool { upperBound := math.MaxUint32 / uint32(sampleRate) sum := sha1.Sum([]byte(traceId)) // convert last 4 digits to uint32 b := sum[:4] v := uint32(b[3]) | (uint32(b[2]) << 8) | (uint32(b[1]) << 16) | (uint32(b[0]) << 24) return v < upperBound } func sampler(fields map[string]interface{}) (bool, int) { // default sample rate of 10 sampleRate = 10 switch fields["request.url"] { case "/x/alive": // drop health checks return false, 0 case "/login": // keep all login requests return true, 1 } // all the rest of the traffic falls through to the default case if shouldSample(fields["trace.trace_id"].(string), sampleRate) { return true, sampleRate } return false, 0 } func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", ServiceName: "MyGoApp", Debug: true, SamplerHook: sampler, }) } ``` Defining a sampling hook overrides the deterministic sampling behavior for trace IDs. Unless you take `trace.trace_id` into account (as we did above by taking the `sha1.Sum` of the trace ID), you will get incomplete traces. ## Distributed Trace Propagation When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace. Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header. Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment. If all of your services are using a Honeycomb beeline, and you want to include trace propagation headers in all outbound HTTP calls, you just have to use the `http.RoundTripper` provided by the `github.com/honeycombio/beeline-go/wrappers/hnynethttp` package: ```go theme={} import ( "net/http" "time" "github.com/honeycombio/beeline-go/wrappers/hnynethttp" ) func main() { client := &http.Client{ Transport: hnynethttp.WrapRoundTripper(http.DefaultTransport), Timeout: time.Second * 5, } req, _ := http.NewRequest("GET", "https://int-srv.local/", nil) res, err := client.Do(req) } ``` In the above example, the outgoing call will include an HTTP header containing information about the current trace. The downstream service, if instrumented with a Honeycomb beeline, will automatically parse the information in this header and use it to construct a span with the appropriate trace and parent ids. ### Interoperability with OpenTelemetry Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers. The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used. To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services. Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed. To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration. #### Parser Hook An `HTTPParserHook` is a function that takes an `http.Request` as an argument and returns a `propagation.PropagationContext`. The `http.Request` is provided to the function so that the author can make decisions about whether to trust the incoming headers based on information contained in the request. For example, perhaps you do not want to accept headers from the public internet. To specify an `HTTPParserHook` for incoming W3C headers, use the `WrapHandlerWithConfig` function in the `github.com/honeycombio/beeline-go/wrappers/hnynethttp` package. ```go theme={} import ( "fmt" "log" "net/http" "github.com/honeycombio/beeline-go/propagation" "github.com/honeycombio/beeline-go/wrappers/config" "github.com/honeycombio/beeline-go/wrappers/hnynethttp" ) func traceHeaderParserHook(r *http.Request) *propagation.PropagationContext { headers := map[string]string{ "traceparent": r.Header.Get("traceparent"), } ctx := r.Context() ctx, prop, err := propagation.UnmarshalW3CTraceContext(ctx, headers) if err != nil { // ... } return prop } log.Fatal( mux := http.NewServeMux() mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello!") }) http.ListenAndServe(":8080", hnynethttp.WrapHandlerWithConfig(mux, config.HTTPIncomingConfig{ HTTPParserHook: traceHeaderParserHook, })) ) ``` #### Propagation Hook An `HTTPPropagationHook` is a function that takes an `http.Request` and a `propagation.PropagationContext` as arguments and returns a map of name, value pairs representing serialized headers. Similar to the parsing hook described above, the HTTP request and Propagation Context objects are passed to this function so that the author can make decisions about whether to include trace context in the outgoing request. For example, you may not wish to send trace context headers when calling a 3rd party API. To specify an `HTTPPropagationHook` for outgoing W3C headers, use the `WrapRoundTripperWithConfig` function in the `github.com/honeycombio/beeline-go/wrappers/hnynethttp` package when creating an HTTP client. ```go theme={} import ( "net/http" "time" "github.com/honeycombio/beeline-go/propagation" "github.com/honeycombio/beeline-go/wrappers/config" "github.com/honeycombio/beeline-go/wrappers/hnynethttp" ) func propagateTraceHook(r *http.Request, prop *propagation.PropagationContext) map[string]string { ctx := r.Context() ctx, headers := propagation.MarshalW3CTraceContext(ctx, prop) return headers } func main() { client := &http.Client{ Transport: hnynethttp.WrapRoundTripperWithConfig( http.DefaultTransport, config.HTTPOutgoingConfig{HTTPPropagationHook: propagateTraceHook} ), Timeout: time.Second * 5, } req, _ := http.NewRequest("GET", "https://int-srv.local/", nil) res, err := client.Do(req) // ... } ``` Because we specified an `HTTPPropagationHook` that returns a serialized header in W3C trace context format, the outgoing request will include the appropriate trace context header. ## Troubleshooting the Beeline ### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. ### The Events I am Generating Do Not Contain the Content I Expect Set `STDOUT` to `true` in the initialization of the Go Beeline. This setting will print the JSON representation of events to the terminal instead of sending them to Honeycomb. This output lets you quickly see what is getting sent and allows you to modify your code accordingly. When `STDOUT` is set to `true`, events will only be sent to the terminal, not to Honeycomb itself. Remove it when you are finished inspecting the events to start sending to the Honeycomb API again. ```go theme={} func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", ServiceName: "MyGoApp", STDOUT: true, }) } ``` ### The Events I am Sending Are Not Being Accepted by Honeycomb Set `Debug` to `true` in the initialization of the Go Beeline. This will print the responses that come back from Honeycomb to the terminal when sending events. These responses will have extra detail explaining why events are being rejected (or that they are being accepted) by Honeycomb. ```go theme={} func main() { beeline.Init(beeline.Config{ WriteKey: "YOUR_API_KEY", ServiceName: "MyGoApp", Debug: true, }) } ``` ### My Traces Are Showing Missing Root Spans There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause "missing span" gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service's beeline configuration that only Honeycomb propagation headers should be parsed. To override undesired W3C trace header propagation behavior, configure the Beeline to use a `traceParserHook`: ```go theme={} func traceParserHook(r *http.Request) *propagation.PropagationContext { headers := map[string]string{ "traceparent": r.Header.Get("traceparent"), } ctx := r.Context() ctx, prop, err := propagation.UnmarshalHoneycombTraceContext(ctx, headers) if err != nil { fmt.Println("Error unmarshaling header") fmt.Println(err) } return prop } ``` The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See [Distributed Trace Propagation](#distributed-trace-propagation) for more details. ## Example Event Here is a sample event created by the Go Beeline. All events contain `Timestamp`, `duration_ms`, and `meta.type` fields, but each wrapper adds a different set of additional fields appropriate for its type of request. ```json theme={} { "Timestamp": "2018-03-20T00:47:25.339Z", "app.interesting_thing": "banana", "duration_ms": 772.446625, "handler.name": "main.hello", "handler.pattern": "/hello/", "handler.type": "http.HandlerFunc", "meta.beeline_version": "0.2.0", "meta.local_hostname": "cobbler.local", "meta.span_type": "root", "meta.type": "http_request", "name": "main.hello", "request.content_length": 0, "request.header.user_agent": "curl/7.54.0", "request.host": "localhost:8080", "request.http_version": "HTTP/1.1", "request.method": "GET", "request.path": "/hello/", "request.remote_addr": "127.0.0.1:60379", "response.status_code": 200, "service_name": "sample app", "trace.span_id": "9e4fe697-3ea9-48c9-b673-72d7ddf118a6", "trace.trace_id": "b64c89a9-7671-4732-bef1-9ef75ab831f6" } ``` ## Queries to Try Here are some examples to get you started querying your app's behavior: ### Which Endpoints Are the Slowest? * `GROUP BY`: `request.path` * `VISUALIZE`: `P99(duration_ms)` * `WHERE`: `meta.type = http request` * `ORDER BY`: `P99(duration_ms) DESC` ### Which Endpoints Are the Most Frequently Hit? * `GROUP BY`: `request.path` * `VISUALIZE`: `COUNT` * `WHERE`: `meta.type = http request` ### Which Users Are Using the Endpoint That I Would Like to Deprecate? (Using a Custom Field `user.email`) * `GROUP BY`: `app.user.email` * `VISUALIZE`: `COUNT` * `WHERE`: `request.url == ` ## Contributions Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change [via GitHub](https://github.com/honeycombio/beeline-go). Remember to add your name to the CONTRIBUTORS file! All contributions will be released under the Apache License 2.0. # Beeline for Java Reference Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/java Reference documentation for Beeline for Java, Honeycomb's legacy auto-instrumentation SDK. Beelines reached end of life on August 12, 2025. Migrate to OpenTelemetry. Beelines have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. * Adding new instrumentation? We recommend that you use [OpenTelemetry for Java](/send-data/java/) instead. * Already using a Beeline? We recommend that you [migrate to OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/). * Concerned about mixing Beelines and OpenTelemetry? Learn more about [Mixing Beelines and OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/#mixing-beelines-and-opentelemetry). The Java Beeline for Honeycomb is a quick and easy way to instrument your Java application. If you are using Spring Boot 2, it automatically instruments HTTP requests and provides tracing information out of the box. It also provides methods for adding additional information and metadata to your generated events, and creating custom spans to fine-tune your instrumentation. If you do not have a Spring Boot v2 application, you can still take advantage of the Java Beeline to add tracing to your app. Only a small amount of additional setup code is required. This is detailed below. If you would like to see more options in the Java Beeline, please [file an issue](https://github.com/honeycombio/beeline-java/issues) or vote for an existing issue! If you prefer to use only Structured Logs, check out our [Libhoney library](/send-data/java/libhoney/). ## Requirements To take advantage of automatic HTTP instrumentation and tracing support, you will need a Java Spring Boot v2 app. Otherwise, any Java application will suffice. You will just have a bit more configuration code to get traces. You wil also need a Honeycomb API key. You can [find your API key](/configure/environments/manage-api-keys/#find-api-keys) in your Environment Settings. If you do not have an API key yet, sign up for a [free Honeycomb account](https://ui.honeycomb.io/signup). ## Quick Installation With Java Spring Boot v2 If using the **dataset-only** data model, refer to the **Honeycomb Classic** tab for instructions. Not sure? Learn more about [Honeycomb versus Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments). To install the Java Beeline for your Spring Boot application: 1. The Java Beeline is available from Maven Central Repository. To get started, add this to your Maven build's `pom.xml`: ```xml theme={} io.honeycomb.beeline beeline-spring-boot-starter 2.0.0 ``` or in Gradle: ```groovy theme={} implementation 'io.honeycomb.beeline:beeline-spring-boot-starter:2.0.0' ``` 2. In your Spring app, add the following configuration to your `application.properties` file in `src/main/resources`: ```ini theme={} # (Required) Give your application a name to identify the origin of your events honeycomb.beeline.service-name= # (Required) Your Honeycomb account API key honeycomb.beeline.write-key= ``` To install the Java Beeline for your Spring Boot application: 1. The Java Beeline is available from Maven Central Repository. To get started, add this to your Maven build's `pom.xml`: ```xml theme={} io.honeycomb.beeline beeline-spring-boot-starter 2.0.0 ``` or in Gradle: ```groovy theme={} implementation 'io.honeycomb.beeline:beeline-spring-boot-starter:2.0.0' ``` 2. In your Spring app, add the following configuration to your `application.properties` file in `src/main/resources`: ```ini theme={} # (Required) Give your application a name to identify the origin of your events honeycomb.beeline.service-name= # (Required) Dataset to send your data to # The name of your app or your environment [development, production] are good choices to start with honeycomb.beeline.dataset= # (Required) Your Honeycomb account API key honeycomb.beeline.write-key= ``` 3. In your Spring app, add the following configuration to your `application.properties` file in `src/main/resources`: ```ini theme={} # (Required) Give your application a name to identify the origin of your Honeycomb Events/Spans, # and to define where trace data is sent. honeycomb.beeline.service-name= # (Required) Your honeycomb account API key honeycomb.beeline.write-key= # (Optional) Sets the global sample rate of traces. honeycomb.beeline.sample-rate=1 # (Optional) Allows the entire Beeline AutoConfiguration to be disabled completely. honeycomb.beeline.enabled=true # (Optional) Allows to switch off automatic instrumentation of the RestTemplate. honeycomb.beeline.rest-template.enabled=true # (Optional) Allows overriding of the beeline's servlet filter precedence # - in case you want your own filters to come before it. #honeycomb.beeline.filter-order= # (Optional) Enables a form of debug logging of responses from Honeycomb's server honeycomb.beeline.log-honeycomb-responses=true # (Optional) List of paths that should be subject to tracing (ant path pattern) honeycomb.beeline.include-path-patterns=/** # (Optional) List of paths that should not be subject to tracing (ant path pattern) honeycomb.beeline.exclude-path-patterns=/exclude-this-path # (Optional) For testing you can override Honeycomb's hostname and redirect Events/Spans. #honeycomb.beeline.api-host=http://localhost:8089 # (Optional) For using the W3C trace context format instead of Honeycomb's #honeycomb.beeline.propagators=w3c ``` After you build and run your app with this configuration, the beeline will automatically start sending events to Honeycomb. 4. Augment the data with interesting information from your app with `beeline.getActiveSpan().addField` so that you can see rich information about your app in Honeycomb. ```java theme={} import io.honeycomb.beeline.tracing.Beeline; @Controller class OwnerController { private final Beeline beeline; private static final String VIEWS_OWNER_CREATE_OR_UPDATE_FORM = "owners/createOrUpdateOwnerForm"; private final OwnerRepository owners; public OwnerController(OwnerRepository clinicService, Beeline beeline) { this.owners = clinicService; this.beeline = beeline; } @GetMapping("/owners/new") public String initCreationForm(Map model) { Owner owner = new Owner(); this.beeline.getActiveSpan().addField("new_owner", owner); model.put("owner", owner); return VIEWS_OWNER_CREATE_OR_UPDATE_FORM; } } ``` 5. Add child spans with the `@ChildSpan` annotation and turn a series of events into a trace. `@ChildSpan` will only work on methods called in classes **other than** the root controller instrumented by the Beeline. This is due to [Spring AOP using a proxy mechanism](https://honeycombio.github.io/beeline-java/io/honeycomb/beeline/spring/beans/aspects/ChildSpan.html). ```java theme={} import io.honeycomb.beeline.spring.beans.aspects.ChildSpan; @ChildSpan("find an owner") @GetMapping("/owners/find") public String initFindForm(Map model) { model.put("owner", new Owner()); return "owners/findOwners"; } ``` ### Tracing Across Services With Spring Boot v2 Beeline The Beeline includes a [`RestTemplateInterceptor`](https://honeycombio.github.io/beeline-java/io/honeycomb/beeline/spring/beans/BeelineRestTemplateInterceptor.html), which can be used to propagate tracing information across HTTP requests to ensure the requests can be tied together in queries in the Honeycomb UI. If a Spring Boot v2 app, which has been instrumented with the Java Beeline, uses `RestTemplate` to call another Beeline instrumented service, the `X-Honeycomb-Trace` header will be set accordingly and the app on the other side will understand this context to set the tracing information correctly. The Interceptor is on by default when Beeline automatic instrumentation has been added to a Spring Boot v2 app and can be turned off using the `honeycomb.beeline.rest-template.enabled` setting in `application.properties`. ### Testing With Spring Boot v2 Beeline When testing a Spring Boot controller class that uses the beeline, mock the beeline using Mockito's `@MockBean` annotation with Mockito's `Answers.RETURNS_DEEP_STUBS` option: ```java theme={} @RunWith(SpringRunner.class) @WebMvcTest(OwnerController.class) public class OwnerControllerTests { private static final int TEST_OWNER_ID = 1; @Autowired private MockMvc mockMvc; @MockBean private OwnerRepository owners; @MockBean(answer = Answers.RETURNS_DEEP_STUBS) private Beeline beeline; private Owner george; @Before public void setup() { ... } ``` ## Installation With Any Java Application (Simple) 1. The Java Beeline is available from Maven Central Repository. To get started, add this to your Maven build's pom.xml: ```xml theme={} io.honeycomb.beeline beeline-core 2.0.0 ``` or in Gradle: ```groovy theme={} implementation 'io.honeycomb.beeline:beeline-core:2.0.0' ``` 2) In your Java app, import the `DefaultBeeline` and `BeelineBuilder`, supplying `getInstance` with builder, service name, and API key: ```java theme={} package com.example.MyApp; import io.honeycomb.beeline.DefaultBeeline; import io.honeycomb.beeline.builder.BeelineBuilder; /// ... public static void main( String[] args ) { DefaultBeeline beeline = DefaultBeeline.getInstance(new BeelineBuilder().writeKey("WRITE_KEY"), "myServiceName"); // ... // Call close at program termination to ensure all pending // spans are sent. beeline.close(); } ``` If your servers are behind an http proxy, you will need to initialize the beeline with additional transport options: ```java theme={} URI honeycombAPIEndpoint; try { honeycombAPIEndpoint = new URI("https://api.honeycomb.io"); // US instance //honeycombAPIEndpoint = new URI("https://api.eu1.honeycomb.io"); // EU instance } catch (URISyntaxException e) { e.printStackTrace(); } DefaultBeeline beeline = DefaultBeeline.getInstance("test-dataset", "my-app", "WRITE_KEY", honeycombAPIEndpoint, LibHoney.transportOptions().setProxy(HttpHost.create("https://myproxy.example.com")).build()); ``` 2. In your Java app, import the `DefaultBeeline` by supplying `getInstance` with your dataset name, service name, and API key: ```java theme={} package com.example.MyApp; import io.honeycomb.beeline.DefaultBeeline; /// ... public static void main( String[] args ) { DefaultBeeline beeline = DefaultBeeline.getInstance("test-dataset", "myServiceName", "WRITE_KEY"); // ... // Call close at program termination to ensure all pending // spans are sent. beeline.close(); } ``` If your servers are behind an http proxy, you will need to initialize the beeline with additional transport options: ```java theme={} URI honeycombAPIEndpoint; try { honeycombAPIEndpoint = new URI("https://api.honeycomb.io"); // US instance //honeycombAPIEndpoint = new URI("https://api.eu1.honeycomb.io"); // EU instance } catch (URISyntaxException e) { e.printStackTrace(); } DefaultBeeline beeline = DefaultBeeline.getInstance("test-dataset", "my-app", "WRITE_KEY", honeycombAPIEndpoint, LibHoney.transportOptions().setProxy(HttpHost.create("https://myproxy.example.com")).build()); ``` 3. Start adding trace spans at interesting points in your code. For example, add a span at the start of an incoming request or the beginning of an asynchronous job. Start more spans as needed, such as when doing a third party API call or an expensive computation. You can add useful context to spans with `addField`. ```java theme={} // startSpan will begin a trace if there is no active trace. // Otherwise, it will create child span of the currently active span. Span rootSpan = beeline.startSpan("request start"); // Add context by calling addField on spans. // This is useful for describing what's happening // in your system. rootSpan.addField("user_id", userId); rootSpan.addField("endpoint", requestEndpoint); // You can also call addField directly on the DefaultBeeline // to add fields to the current 'active' span. beeline.addField("success", true); // When needed, add more spans: Span callDataAPISpan = beeline.startSpan("call data API"); // These child spans can be annotated with // yet more fields callDataAPISpan.addField("user_id", userId); callDataAPISpan.addField("action", "store result"); // Once the work you're wrapping with a span is finished, // send it to the Honeycomb API by using sendActiveSpan() // or close() on the span itself. callDataAPI(); // do the actual work callDataAPISpan.close(); // finish timing the work // Don't forget to send the root too. rootSpan.close(); // Once all spans have been closed end the trace. beeline.getBeeline().getTracer().endTrace(); ``` ## Installation With Any Java Application (Advanced) 1. The Java Beeline is available from Maven Central Repository. To get started, add this to your Maven build's pom.xml: ```xml theme={} io.honeycomb.beeline beeline-core 2.0.0 ``` or in Gradle: ```groovy theme={} implementation 'io.honeycomb.beeline:beeline-core:2.0.0' ``` 2. In your Java app, first set up the Beeline and its collaborator classes: ```java theme={} package com.mycompany.services.Honeycomb; import io.honeycomb.beeline.tracing.Beeline; import io.honeycomb.beeline.tracing.Span; import io.honeycomb.beeline.tracing.SpanBuilderFactory; import io.honeycomb.beeline.tracing.SpanPostProcessor; import io.honeycomb.beeline.tracing.Tracer; import io.honeycomb.beeline.tracing.Tracing; import io.honeycomb.beeline.tracing.sampling.Sampling; import io.honeycomb.libhoney.HoneyClient; import io.honeycomb.libhoney.LibHoney; public class TracerSpans { private static final String WRITE_KEY = "test-write-key"; private static final String DATASET = "test-dataset"; private static final HoneyClient client; private static final Beeline beeline; static { client = LibHoney.create(LibHoney.options().setDataset(DATASET).setWriteKey(WRITE_KEY).build()); SpanPostProcessor postProcessor = Tracing.createSpanProcessor(client, Sampling.alwaysSampler()); SpanBuilderFactory factory = Tracing.createSpanBuilderFactory(postProcessor, Sampling.alwaysSampler()); Tracer tracer = Tracing.createTracer(factory); beeline = Tracing.createBeeline(tracer, factory); } } ``` 3. Augment the data with interesting information from your app, so that you can see rich information about your app in Honeycomb. Use the `spanBuilderFactory` to create a root span. Add calculated fields to that span with `beeline.getActiveSpan().addField()`. ```java theme={} import io.honeycomb.beeline.tracing.Beeline; import io.honeycomb.beeline.tracing.Span; import io.honeycomb.beeline.tracing.SpanBuilderFactory; import io.honeycomb.beeline.tracing.SpanPostProcessor; import io.honeycomb.beeline.tracing.Tracer; import io.honeycomb.beeline.tracing.Tracing; import io.honeycomb.beeline.tracing.sampling.Sampling; import io.honeycomb.libhoney.HoneyClient; import io.honeycomb.libhoney.LibHoney; public static void main(String... args) { TracerSpans example = new TracerSpans(); try { HttpRequest request = new HttpRequest(); startTrace(request); example.acceptRequest(request); } finally { beeline.getTracer().endTrace(); client.close(); // close to flush events and release its thread pool } } private DatabaseService db = new DatabaseService(); private static void startTrace(HttpRequest request) { Span rootSpan = beeline.getSpanBuilderFactory().createBuilder() .setSpanName("get-customer-data") .setServiceName("customer-db-traced") .build(); beeline.getTracer().startTrace(rootSpan); } public void acceptRequest(HttpRequest request) { Span span = beeline.getActiveSpan(); try { db.queryDb(request.getParameter("customer-id")); span.addField("result", "OK"); } catch (Exception e) { span.addField("result", "Bad Request") .addField("exception-message", e.getMessage()); } } ``` In this example, when the parent context is set in `SpanBuilderFactory`, the `traceId` in the context becomes the id for the trace initialized with `startTrace`. The `spanId` in the context becomes the `parentSpanId` of the next span in the trace. Effectively `startTrace` is more like "continue trace" when there is parent context involved. This is how you can achieve cross-service traces using the Java Beeline. 4. Add additional spans with `beeline.startChildSpan()` and turn a series of spans into a trace: ```java theme={} public static class DatabaseService { public void queryDb(String id) { Span childSpan = beeline.startChildSpan("customer-db-query"); try { String data = getCustomerDataById(id); childSpan.addField("customer-data", data); } catch (Exception e) { childSpan.addField("db.error", e.getMessage()); } finally { childSpan.close(); } } public String getCustomerDataById(String id) { return "customer-0123"; } } ``` ## Instrumenting HTTP Clients and Servers The Beeline Spring Boot Starter automatically instruments the HTTP server and client inside your Spring Boot application. If you are not using Spring Boot, you can still instrument your HTTP server and client code using Beeline Core's HTTP instrumentation utilities. ### Instrumenting HTTP Clients Beeline Core contains high-level utility classes to help you instrument the HTTP client of your choice. Using this pattern will open and close a child span for each HTTP client call and add a standard set of fields to this span based on the data in the HTTP request and response. The class that does the work is [HttpClientPropagator](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/HttpClientPropagator.java). You pass implementations to it of [HttpClientRequestAdapter](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/HttpClientRequestAdapter.java) and [HttpClientResponseAdapter](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/HttpClientResponseAdapter.java), which [adapt](https://en.wikipedia.org/wiki/Adapter_pattern) the HTTP request and response abstractions used in your HTTP client library. This allows the `HttpClientPropagator` to read and write to the request and response so the client call can be traced. The general pattern is as follows: ```java theme={} public class HttpClientCallInterceptor { // Initialized in constructor private final HttpClientPropagator clientPropagator; ///... public HttpResponse instrumentedClientCall(final HttpRequest actualHttpRequest) throws Exception { // Adapt your actual HTTP request so it can be read by the HttpClientPropagator HttpClientRequestAdapter adaptedHttpRequest = new MyHttpClientRequestAdapter(actualHttpRequest); // These two variables are so we can reference the adapted response and error (if they exist) when closing the child span using #endPropagation HttpClientResponseAdapter adaptedHttpResponse = null; Throwable error = null; // Call #startPropagation just before you execute the actual HTTP request; this will start the child span and add standard HTTP span fields. Span span = clientPropagator.startPropagation(adaptedHttpRequest); try { // Execute the actual HTTP request HttpResponse actualHttpResponse = makeClientCall(actualHttpRequest); // Adapt your actual HTTP response so it can be read by the HttpClientPropagator adaptedHttpResponse = new MyHttpClientResponseAdapter(actualHttpResponse); //Return the actual HTTP response for processing by your code return actualHttpResponse; } catch (Exception ex) { // Assign the error if present error = ex; throw ex; } finally { // Call #endPropagation when request has finished. This closes the child span and adds the standard HTTP span fields based on whether the error/response are non-null. clientPropagator.endPropagation(adaptedHttpResponse, error, span); } } private HttpResponse makeClientCall(HttpRequest httpRequest) { // Make the actual HTTP client call /// ... } class MyHttpClientRequestAdapter implements HttpClientRequestAdapter { private final HttpRequest httpRequest; public MyHttpClientRequestAdapter(HttpRequest httpRequest) { this.httpRequest = httpRequest; } //implement methods ///... } class MyHttpClientResponseAdapter implements HttpClientResponseAdapter { private final HttpResponse httpResponse; public MyHttpClientResponseAdapter(HttpResponse httpResponse) { this.httpResponse = httpResponse; } //implement methods ///... } } ``` It is likely that you would implement this in an interceptor that is added to your HTTP client. An example of this is found in the [instrumentation](https://github.com/honeycombio/beeline-java/blob/main/beeline-spring-boot-starter/src/main/java/io/honeycomb/beeline/spring/beans/BeelineRestTemplateInterceptor.java) Beeline adds to Spring's `RestTemplate` interceptor. This also contains an example of how to adapt an HTTP client request and response. ### Instrumenting HTTP Servers Beeline Core contains high-level utility classes to help you instrument the HTTP server of your choice. Using this pattern will open and close a span for each HTTP request received by the server and add a standard set of fields to this span based on the data in the HTTP request and response. If the HTTP request is being made by a service that is also using Beeline tracing, then the new span will be the child of the HTTP client's request span. The class that does the work is [HttpServerPropagator](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/HttpServerPropagator.java). You pass implementations to it of [HttpServerRequestAdapter](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/HttpServerRequestAdapter.java) and [HttpServerResponseAdapter](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/HttpServerResponseAdapter.java) which [adapt](https://en.wikipedia.org/wiki/Adapter_pattern) your framework's HTTP request and response abstractions. This allows the `HttpServerPropagator` to read and write to the request/response so the request handling can be traced. The general pattern is as follows: ```java theme={} // Inside your HTTP server handler class, e.g. a Servlet Filter public class ServerHandler { // Initialized in constructor private final HttpServerPropagator serverPropagator; ///... public HttpResponse instrumentedRequest(final HttpRequest actualHttpRequest) throws Exception { // Adapt the actual received HTTP request so it can be read by the HttpServerPropagator HttpServerRequestAdapter adaptedHttpRequest = new MyHttpServerRequestAdapter(actualHttpRequest); // These two variables are so we can reference the adapted response and error (if they exist) when closing the child span using #endPropagation HttpServerResponseAdapter adaptedHttpResponse = null; Throwable error = null; // Call #startPropagation before you start handling the actual HTTP request; this will start the child span and add standard HTTP span fields. Span span = serverPropagator.startPropagation(adaptedHttpRequest); try { // Handle the HTTP request HttpResponse actualHttpResponse = handleRequest(actualHttpRequest); // Adapt the actual HTTP response that you will return to the client so it can be read by the HttpServerPropagator adaptedHttpResponse = new MyHttpServerResponseAdapter(actualHttpResponse); //Return the actual HTTP response to the client return actualHttpResponse; } catch (Exception ex) { // Assign the error if an exception occurred when your code was processing the HTTP request error = ex; throw ex; } finally { // Call #endPropagation when the server has finished processing the request. This closes the child span and adds the standard HTTP span fields based on whether the error/response are non-null. serverPropagator.endPropagation(adaptedHttpResponse, error, span); } } private HttpResponse handleRequest(HttpRequest actualHttpRequest) { ///... handle the actual HTTP request } class MyHttpServerResponseAdapter implements HttpServerResponseAdapter { private final HttpResponse httpResponse; public MyHttpServerResponseAdapter(HttpResponse httpResponse) { this.httpResponse = httpResponse; } //implement methods ///... } class MyHttpServerRequestAdapter implements HttpServerRequestAdapter { private final HttpRequest httpRequest; public MyHttpServerRequestAdapter(HttpRequest httpRequest) { this.httpRequest = httpRequest; } //implement methods ///... } } ``` Beeline Core already uses this approach to add server instrumentation via a [Java Servlet Filter](https://github.com/honeycombio/beeline-java/blob/main/beeline-core/src/main/java/io/honeycomb/beeline/tracing/propagation/BeelineServletFilter.java). This also contains an example of how to adapt an HTTP server request and response. ### Distributed Trace Propagation When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace. Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header. Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment. #### Interoperability With OpenTelemetry Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers. The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used. To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services. Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed. To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration. #### Custom Trace Propagation Hook Outgoing HTTP requests can contain trace context formatted in HTTP headers. By default, the `HttpClientRequestAdapter` will be configured to propagate trace context using Honeycomb's `X-Honeycomb-Trace` header. The beeline also includes a propagation codec for the W3C trace. The `HttpClientRequestAdapter` can be configured to use a custom propagation hook that can write trace context data in other propagation formats. The propagation hook receives both the HTTP request and the propagation context as parameters that can be used in the decision making of what (if any) propagation format is used. The following example writes propagation context using the built-in `Propagation.w3c()` propagator. ```java theme={} final HttpClientPropagator propagator = new HttpClientPropagator.Builder(tracer, r -> "span-name") .setTracePropagationHook((request, propagationContext) -> { return Propagation.w3c().encode(propagationContext); }) .build(); ``` #### Custom Trace Parser Hook Incoming HTTP requests may contain trace context formatted in HTTP headers. By default, the `HttpServerPropagator` will be configured to parse trace context in Honeycomb's `X-Honeycomb-Trace` header. The beeline also includes a propagation codec for the W3C trace format. The `HttpServerPropagator` can be configured to use a custom parse hook that can be used to read trace context information from other propagation formats. The parse hook receives the HTTP request, which can be used to determine which (if any) propagation format is used. The following example tries to extract a propagation context using the built-in W3C propagation format and then falls back to trying to extract context from the Honeycomb format. ```java theme={} final HttpServerPropagator propagator = new HttpServerPropagator.Builder(beeline, "my-service", r -> "span-name") .setParsePropagationHook(request -> { PropagationContext context = Propagation.w3c.decode(request.getHeaders()); if (context != PropagationContext.emptyContext()) { return context; } return Propagation.honey().decode(request.getHeaders()); }) .build(); ``` ### Tracing Servlets Outside of Spring Boot If you are not using Spring Boot 2, but you are using Java Servlets, you can still use the Beeline Core's `io.honeycomb.beeline.tracing.propagation.BeelineServletFilter` to instrument your application for Honeycomb. You just need to manually configure it, depending on your framework choice. The filter is compatible with Servlet 3.1+, so for instance you could use a `ServletContextListener` to programmatically configure the filter. In the example below, we use the `io.honeycomb.beeline.DefaultBeeline` to make the setup concise. However, you could just as well use a `io.honeycomb.beeline.tracing.Beeline` with a more customized configuration. If using the **dataset-only** data model, refer to the **Honeycomb Classic** tab for instructions. Not sure? Learn more about [Honeycomb versus Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments). ```java theme={} import io.honeycomb.beeline.DefaultBeeline; import io.honeycomb.beeline.builder.BeelineBuilder; import javax.servlet.DispatcherType; import javax.servlet.ServletContextEvent; import javax.servlet.ServletContextListener; import java.util.EnumSet; import io.honeycomb.beeline.tracing.propagation.BeelineServletFilter; public class ExampleBeelineServletContextListener implements ServletContextListener { private static final String SERVICE_NAME = "myServiceName"; private final DefaultBeeline beeline = DefaultBeeline.getInstance(new BeelineBuilder().writeKey("WRITE_KEY"), SERVICE_NAME); @Override public void contextInitialized(ServletContextEvent sce) { // initialize the filter // you may wish to customize it using the builder; for instance to change the default span naming functions final BeelineServletFilter beelineServletFilter = BeelineServletFilter.builder() .setBeeline(defaultBeeline.getBeeline()) .setServiceName(SERVICE_NAME) .build(); //add the filter to the context sce.getServletContext().addFilter("beelineFilter", beelineServletFilter) .addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), true, "/*"); } @Override public void contextDestroyed(ServletContextEvent sce) { //close the beeline and its underlying resources defaultBeeline.close(); } } ``` ```java theme={} import io.honeycomb.beeline.DefaultBeeline; import javax.servlet.DispatcherType; import javax.servlet.ServletContextEvent; import javax.servlet.ServletContextListener; import java.util.EnumSet; import io.honeycomb.beeline.tracing.propagation.BeelineServletFilter; public class ExampleBeelineServletContextListener implements ServletContextListener { private static final String SERVICE_NAME = "myServiceName"; private final DefaultBeeline defaultBeeline = DefaultBeeline.getInstance("myDataset", SERVICE_NAME, "myWriteKey"); @Override public void contextInitialized(ServletContextEvent sce) { // initialize the filter // you may wish to customize it using the builder; for instance to change the default span naming functions final BeelineServletFilter beelineServletFilter = BeelineServletFilter.builder() .setBeeline(defaultBeeline.getBeeline()) .setServiceName(SERVICE_NAME) .build(); //add the filter to the context sce.getServletContext().addFilter("beelineFilter", beelineServletFilter) .addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), true, "/*"); } @Override public void contextDestroyed(ServletContextEvent sce) { //close the beeline and its underlying resources defaultBeeline.close(); } } ``` ### More on Adding Fields to Events In the above examples, we have gone over adding a calculated field to a specific span. If you are interested in adding calculated fields to all spans, use `Span.addTraceField` or `Span.AddTraceFields` instead. These will add the field(s) to the currently active span and also propagate them down the trace. Additionally, these fields will be packaged up and passed along to downstream processes if those processes are also using a beeline. These functions are good for adding information that is better scoped to the request than this specific unit of work, including: user IDs, globally relevant feature flags, errors, and other fields. ### More on Spans and Traces We encourage people to think about instrumentation in terms of "units of work". As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize as traces within the Honeycomb query builder. Spans always get a few fields: * a name * a duration - how much time elapsed between when the span was started and sent * a service\_name - generally configured during Beeline initialization * several IDs - trace, span, and parent identifiers (UUIDs) ## Sampling Events For very high throughput services, you can send only a portion of the events flowing through your service by enabling sampling. The sample rate N will send 1/N events, so a sample rate of 5 would send 20% of all events. A sample rate of 1 (or omitting sample rate entirely) will send every event. For high throughput services, a sample rate of 100 is a good start. The value of your sample rate **must** be a positive integer. ### Sampling With Spring Boot To start sampling traces with a Spring Boot app, simply set `honeycomb.beeline.sample-rate` in your `application.properties`. ```ini theme={} # (Optional) Sets the global sample rate of traces. honeycomb.beeline.sample-rate :1 ``` The Beeline will deterministically sample 1/N events based on the Span's trace id. (This means that adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.) ### Sampling Without Spring Boot Sampling of traces and Spans can be done in two ways: * The `SpanBuilderFactory` is configured with a "global" `DeterministicTraceSampler` that accepts a sample rate N and will deterministically sample 1/N events based on a target field such as the Span's `traceId`. (This means that adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent.) If a trace is not sampled, "noop" spans that carry no data are generated to minimize runtime overhead. * The `SpanBuilderFactory` can be configured with a `SpanPostProcessor` that applies sampling just before a Span is submitted to Honeycomb. This allows sampling to be performed based on the contents of the Span. By writing a custom `Sampler` you can customize your own sampling logic. The `SpanPostProcessor` will only apply to Spans that have been sampled by the "global" sampler, because the "noop" Spans carry no data. The following example creates a `SpanPostProcessor` that ignores keep alive requests, keeps all login requests and then uses a 1/10 sample rate to all other spans: ```java theme={} final SpanPostProcessor processor = new SpanPostProcessor(client, span -> { switch (span.getFields().get("request.url").toString()) { case "/x/alive": // drop health checks return 0; case "/login": // keep all login requests return 1; } return 10; // by default, sample 1/10 }) ``` When the two sampling mechanisms are used together, the effective sample rate is a product of both. However, note that you do not have to use either of the sampling mechanisms and can always configure either one or both to "always sample" with a `sampleRate` of 1 (`Sampling.alwaysSampler()` does this). Defining a Sampler overrides the deterministic sampling behavior for trace IDs. Unless you take `trace.trace_id` into account, you will get incomplete traces. ## Troubleshooting ### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. ### Troubleshooting With Spring Boot A [DebugResponseObserver](https://github.com/honeycombio/beeline-java/blob/main/beeline-spring-boot-starter/src/main/java/io/honeycomb/beeline/spring/beans/DebugResponseObserver.java) is registered by default and enables a form of debug logging of responses from Honeycomb's server, such as if the event was successfully sent, rejected by the server, or rejected client side. You can also provide your own implementation of `io.honeycomb.libhoney.ResponseObserver` either as a Spring Bean or by adding it with `io.honeycomb.libhoney.HoneyClient#addResponseObserver.` The `logging.level` setting for both the Beeline and Libhoney packages needs to be set to DEBUG for log entries to be recorded. This can be done via either the command line or by using `application.properties` entries. Example `application.properties` entries: ```ini theme={} # enable DEBUG logging in beeline & libhoney honeycomb.beeline.log-honeycomb-responses=true logging.level.io.honeycomb.beeline=DEBUG logging.level.io.honeycomb.libhoney=DEBUG ``` To explicitly disable debug logging, set `honeycomb.beeline.log-honeycomb-responses` to false in your `application.properties` file. ### Configuring The Beeline to Disable Sending Events to Honeycomb For situations in which you would like to disable sending events to Honeycomb, such as a test environment or in unit tests, use [`HoneyClient`'s constructor](https://github.com/honeycombio/libhoney-java/blob/main/libhoney/src/main/java/io/honeycomb/libhoney/HoneyClient.java) in which the `Transport` can be overridden. Create a mock (substitute) `Transport` that implements the `Transport` interface. One similar example can be seen [here](https://github.com/honeycombio/beeline-java/blob/main/beeline-spring-boot-starter/src/test/java/io/honeycomb/beeline/spring/mockmvctests/MockMvcTest.java). If setting a mock `Transport` is more than what is needed, simply changing the API key to any other non-null string will prevent sending to Honeycomb. The beeline will not function without an API key, so omitting the key will not have the desired effect. ### My Traces Are Showing Missing Root Spans There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause "missing span" gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service's beeline configuration that only Honeycomb propagation headers should be parsed. To override undesired W3C trace header propagation behavior, configure the Beeline to use a parse hook: ```java theme={} final HttpServerPropagator propagator = new HttpServerPropagator.Builder(beeline, "my-service", r -> "span-name") .setParsePropagationHook(request -> { PropagationContext context = Propagation.honey().decode(request.getHeaders()); if (context != PropagationContext.emptyContext()) { return context; } }) .build(); ``` The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See [Distributed Trace Propagation](#distributed-trace-propagation) for more details. ## Example Event With Spring Boot Here is a sample event created by the Java Beeline for Spring Boot: ```json theme={} { "Timestamp": "2018-03-20T00:47:25.339Z", "duration_ms": 107.446625, "meta.beeline_version": "1.0.3", "meta.instrumentation_count": 3, "meta.instrumentations": ["spring_mvc", "spring_rest_template", "spring_aop"], "meta.local_hostname": "cobbler.local", "meta.package": "Spring Boot", "meta.package_version": "2.1.2.RELEASE", "name": "ProcessCreationForm", "request.content_length": 99, "request.error": "", "request.error_detail": "", "request.header.accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", "request.header.content_type": "application/x-www-form-urlencoded", "request.header.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36", "request.host": "localhost", "request.http_version": "HTTP/1.1", "request.method": "POST", "request.path": "/owners/new", "request.query": "", "request.remote_addr": "0:0:0:0:0:0:0:1", "request.scheme": "http", "request.secure": false, "request.xhr": false, "response.header_content_type": "text/html;charset=UTF-8", "response.status_code": 200, "service_name": "sample app", "spring.method.name": "", "spring.request.dispatcher_type": "", "spring.request.handler_method": "OwnerController#processCreationForm", "spring.request.handler_type": "HandlerMethod", "spring.request.matched_pattern": "/owners/new", "trace.parent_id": "0676a4de-09f9-4287-baf5-a772b548362c", "trace.span_id": "9e4fe697-3ea9-48c9-b673-72d7ddf118a6", "trace.trace_id": "b64c89a9-7671-4732-bef1-9ef75ab831f6", "type": "http_server" } ``` ## Queries to Try Try these examples to get started querying your app's behavior. ### Which Endpoints Are The Slowest? * `GROUP BY`: `request.path` * `VISUALIZE`: `P99(duration_ms)` * `ORDER BY`: `P99(duration_ms) DESC` ### Which Endpoints Are The Most Frequently Hit? * `GROUP BY`: `request.path` * `VISUALIZE`: `COUNT` ## Contributions Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change [via GitHub](https://github.com/honeycombio/beeline-java). All contributions will be released under the Apache License 2.0. # Beeline for Node.js Reference Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/nodejs Reference documentation for Beeline for Node.js, Honeycomb's legacy auto-instrumentation SDK. Beelines reached end of life on August 12, 2025. Migrate to OpenTelemetry. Beelines have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. * Adding new instrumentation? We recommend that you use [OpenTelemetry for Node.js](/send-data/javascript-nodejs/) instead. * Already using a Beeline? We recommend that you [migrate to OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/). * Concerned about mixing Beelines and OpenTelemetry? Learn more about [Mixing Beelines and OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/#mixing-beelines-and-opentelemetry). The Node.js Beeline provides instant, per-request visibility for your Express application. It automatically instruments many common Node.js packages with traces and events to capture useful information from your application. The Node.js Beeline allows you to slice and dice requests by endpoint, status, or even User ID, with zero custom instrumentation required. It creates traces automatically out of the box. While this is a great way to get general insights about your app as quickly as possible, as you forge ahead on your observability journey, you may find you would like to add new events or traces to add more details specific to your app. The Node.js Beeline provides simple interfaces for adding both. If you would like to see more options in the Node.js Beeline, please [file an issue](https://github.com/honeycombio/beeline-nodejs/issues) or vote up an already filed one! If you prefer more control over your application's instrumentation, the Node.js Beeline has an API of its own for [adding to traces](#adding-spans-to-a-trace-or-starting-new-traces). ## Requirements * Node.js 10+ * A Honeycomb API key You can [find your API key](/configure/environments/manage-api-keys/#find-api-keys) in your Environment Settings. If you do not have a API key yet, sign up for a [free Honeycomb account](https://ui.honeycomb.io/signup). ## Quick Installation If using the **dataset-only** data model, refer to the **Honeycomb Classic** tab for instructions. Not sure? Learn more about [Honeycomb versus Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments). If you have a NodeJS Express or Fastify app, you can get request-level instrumentation for those frameworks and other supported packages you use, automatically. 1. Install the Node.js Beeline package using npm: ```shell theme={} npm install --save honeycomb-beeline ``` 2. Add the following code to the top of your `app.js`. **Important:** It must be **before** any `require` or `import` statements for other packages. ```javascript theme={} require("honeycomb-beeline")({ writeKey: "YOUR_API_KEY", // The name of your app/service is a good choice serviceName: "my-node-service" }); ``` If you have a NodeJS Express or Fastify app, you can get request-level instrumentation for those frameworks and other supported packages you use, automatically. 1. Install the Node.js Beeline package using npm: ```shell theme={} npm install --save honeycomb-beeline ``` 2. Add the following code to the top of your `app.js`. **Important:** It must be **before** any `require` or `import` statements for other packages. ```javascript theme={} require("honeycomb-beeline")({ writeKey: "YOUR_API_KEY", // The name of your app is a good choice to start with dataset: "my-node-distributed-app", serviceName: "my-node-service" }); ``` ## Adding Context to Events The packaged instrumentations send context to Honeycomb about requests and queries, but they can not automatically capture all context that you might want. Additional fields are an excellent way to add detail to your events. Try putting a timer around a section of code, add adding per-user information, or details about what it took to craft a response. You can add fields when and where you need to, or for some events but not others. (Error handlers are a good example of this.) Here is an example of adding a custom field to the currently-active span: ```javascript theme={} const beeline = require("honeycomb-beeline")(); const calcBigNum = num => { beeline.addContext({ oldNum: num }); // ... do thing with num beeline.addContext({ newNum: num }); }; ``` Note that `beeline.addContext` only adds fields to the current span. To add fields to all downstream spans, use `beeline.addTraceContext` instead: ```javascript theme={} const beeline = require("honeycomb-beeline")(); const handleInput = () => { beeline.addTraceContext({ userId: userId }); }; ``` Note that `beeline.addTraceContext` will prefix your field name with `app.`, so that all downstream spans will be populated with an `app.userId` field. These additional fields are your opportunity to add important and detailed context to your instrumentation. Put a timer around a section of code, add per-user information, include details about what it took to craft a response, and so on. It is expected that some fields will only be present on some requests. Error handlers are a great example of this; they will obviously only exist when an error has occurred. It is common practice to add in these fields along the way as they are processed in different levels of middleware. For example, if you have an authentication middleware, it would add a field with the authenticated user's ID and name as soon as it resolves them. Later on in the call stack, you might add additional fields describing what the user is trying to achieve with this specific HTTP request. See the [Adding Context to Spans](https://github.com/honeycombio/beeline-nodejs/blob/main/docs/API.md#adding-context-to-spans) section for more information on the full API available to you. ## Adding Spans To a Trace Or Starting New Traces We encourage people to think about instrumentation in terms of "units of work". As your program grows, what constitutes a unit of work is likely to be portions of your overall service rather than an entire run. Spans are a way of breaking up a single external action (say, an HTTP request) into several smaller units in order to gain more insight into your service. Together, many spans make a trace, which you can visualize traces within the Honeycomb query builder. Express and Fastify instrumentations automatically start and finish traces for incoming requests. If you are using a different framework, choose to disable those automatic instrumentations. If you have code that runs outside the request handlers, you will need to manage the trace lifecycle manually. `beeline.startTrace()` starts a new local trace and initializes the async context propagation machinery. You **must** have an active trace for the tracing API to work correctly. If you call `startSpan` when you are not currently in a trace, an `Error` will be thrown. The instrumentations, which must operate in both trace/non-trace environments, handle this by checking `beeline.traceActive()` and only creating spans if they're within a trace. This method also creates a root span for the trace (using `beeline.startSpan` below), and adds `metadataContext` as its initial context. This root span is installed as the current span. You must call `startTrace` outside of an `async` function. Other beeline calls will work inside `async` functions. Below is an example of starting a trace: ```javascript theme={} let trace = beeline.startTrace({ field1: value1, field2: value2 }); ``` To start a new span in the existing trace, call `beeline.startSpan();`. This returns a reference to the span which can then be used in `finishSpan` like this: ```javascript theme={} let span = beeline.startSpan({ field1: value1, field2: value2 }); let value = calculateSomeValue(); beeline.addTraceContext({ field3: value }) beeline.finishSpan(span); ``` If you are doing something synchronously (for example, looping, or using a synchronous node API) you can use `withSpan` to wrap this operation. Since it returns the return value of `fn`, it can be used in an expression context. Here is an example: ```javascript theme={} let sum = beeline.withSpan( { task: "calculating the sum" }, () => { let s = 0; for (let i of bigArray) { s += i; } return s; } ); ``` After you have added all the spans you need to your trace, call `beeline.finishTrace(trace);` to send the trace's root span, and complete the necessary teardown. Below is an example of finishing a trace: ```javascript theme={} beeline.finishTrace(trace); ``` Some traces can be expressed as a single function, for example, if you are doing something synchronously (maybe in a script). In this case you can use, `withTrace()` as seen below: ```javascript theme={} beeline.withTrace( { task: "writing a file", filePath }, () => fs.writeFileSync(filePath, fileContents) ); // Another example of withTrace, in an expression context: console.log( `the answer is ${beeline.withTrace( { task: "computing fibonacci number", n }, () => computeFib(n) )}` ); ``` ### Asynchronous Spans Each span in a trace, except the first (root) span, has a parent. Every span must finish before its parent, otherwise you will see incomplete or broken traces in Honeycomb. If you have asynchronous code, where a span might finish after its parent, use `beeline.startAsyncSpan()` to address this situation. This function takes a callback, which will be called with the new span. When using `beeline.startAsyncSpan()`, make sure to finish the new span within the callback. For example: ```javascript theme={} beeline.startAsyncSpan({ task: "writing a file", filePath, }, span => { fs.writeFile(filePath, fileContents, err => { if (err) { beeline.addTraceContext({ fileError: err.toString() }); } beeline.finishSpan(span); }); }); ``` When using `async/await` syntax, for example: ```javascript theme={} await beeline.startAsyncSpan({ task: "writing a file", filePath, }, async (span) => { try { await fs.writeFile(filePath, fileContents); } catch (e) { beeline.addTraceContext({ fileError: err.toString() }); } finally { beeline.finishSpan(span); } }); ``` ## Instrumented Packages The following is a list of packages for which we have added instrumentation. Some add context to events, while others propagate context so the Beeline can create events in downstream packages. The source for each individual instrumented package can be found in the [`lib/instrumentation` folder on GitHub](https://github.com/honeycombio/beeline-nodejs/tree/main/lib/instrumentation). ### `bluebird` Instrumented only for context propagation ### `mpromise` Instrumented only for context propagation ### `express` Adds columns with prefix `request.` **Configuration Options:** | Name | Type | | --------------------- | ---------------------------------------------- | | `express.userContext` | Array\\|Function\<(request) => Object> | **`express.userContext`** If the value of this option is an array, it is assumed to be an array of string field names of `req.user`. If a request has `req.user`, the named fields are extracted and added to events with column names of `express.user.$fieldName`. For example: If `req.user` is an object `{ id: 1, username: "toshok" }` and your config settings include `express: { userContext: ["username"] }`, the following will be included in the express event sent to honeycomb: | `request.user.username` | | :---------------------- | | `toshok` | If the value of this option is a function, it will be called on every request and passed the request as the sole argument. All key-values in the returned object will be added to the event. If the function returns a falsy value, no columns will be added. To replicate the above Array-based behavior, you could use the following configuration: `express: { userContext: (req) => req.user && { username: req.user.username } }` This function is not limited to using the request object, and can pull info from anywhere to enrich the data sent about the user. #### `http` Adds columns with prefix `http.` #### `https` Adds columns with prefix `https.` #### `mysql2` Adds columns with prefix `db.` #### `pg` Adds columns with prefix `db.` #### `sequelize` Instrumented only for context propagation #### `mongoose` Instrumented only for context propagation #### `mongodb` Adds columns with prefix `db.` **Configuration options:** | Name | Type | | -------------------------- | ------- | | `mongodb.includeDocuments` | boolean | **`mongodb.includeDocuments`** If true, documents in the api will be JSON serialized and included in the events sent to honeycomb. #### `react-dom/server` Adds columns with prefix `react.` #### `child_process` Instrumented only for context propagation ## Optional Configuration The `additional optional configuration` in the code example above is where you can add global settings (Honeycomb credentials and service name) or per-instrumentation settings: ```javascript theme={} { writeKey: "YOUR_API_KEY", serviceName: "your-service-name" $instrumentationName: { /* instrumentation specific settings */ } } ``` You may also specify `writeKey` and `serviceName` by setting `HONEYCOMB_WRITEKEY` and `SERVICE_NAME` in your environment. To add custom instrumentation settings, specify them in your configuration object as a key/value pair using the name of the instrumentation as the key. For example, to add configuration options for `express`, your configuration object might look like: ```javascript theme={} { writeKey: "YOUR_API_KEY", serviceName: "your-service-name" express: { /* express-specific settings */ } } ``` See the [Instrumented packages](#instrumented-packages) section below for available configuration options for each package. ## Customizing Instrumented Packages If you want to disable automatic instrumentation for whatever reason, for either an individual package or all packages, you can pass `enabledInstrumentations` when configuring the Beeline. It should be an array of package names to automatically instrument. For instance, if you want to enable the beeline instrumentation only for the `http` and `https` packages: ```javascript theme={} require("honeycomb-beeline")({ enabledInstrumentations: ["http", "https"] /* ... additional configuration ... */ }); ``` The Beeline also exports `getInstrumentations`, which returns a list of all enabled instrumentation. You can use this to filter out specific instrumentations you want to disable. If you want to enable all instrumentation **except** `mongodb`: ```javascript theme={} const beeline = require("honeycomb-beeline"); beeline({ enabledInstrumentations: beeline .getInstrumentations() .filter(i => i !== "mongodb") /* ... additional configuration ... */ }); ``` Finally, to disable all automatic instrumentation, pass an empty array as in: ```javascript theme={} require("honeycomb-beeline")({ enabledInstrumentations: [] /* ... additional configuration ... */ }); ``` ## Augmenting Or Scrubbing Spans If you have some transformations you would like to make on every span before it leaves the process for Honeycomb, the `presendHook` is your opportunity to make these changes. Examples are scrubbing sensitive data (for example, you may want to ensure specific fields are dropped or hashed) or augmenting data (like making out-of-band translations between an ID and a more human readable version). Similar to the `samplerHook` discussed below, you pass the `presendHook` a function that will be called on every span with the span as an argument. The function is free to mutate the span passed in and those mutations will be what finally gets sent to Honeycomb. As an example, say we have some HTTP requests that provide sensitive values which have been added to a span. This code will examine all spans just before they are sent to Honeycomb and remove the sensitive values. ```javascript theme={} const beeline = require("honeycomb-beeline"); beeline({ presendHook: function(ev) { // either: ev.data.scrubMe = undefined; // or: // delete ev.data.scrubMe } }); beeline.finishTrace(beeline.startTrace({ name: "foo", scrubMe: "sensitive data" })); ``` ## Proxy Configuration If the environment variables `HTTPS_PROXY` or `https_proxy` are set, the Beeline will pick up and configure itself to use the proxy for all event traffic to Honeycomb. ## Sampling Events We have built-in support for [sampling](/manage-data-volume/sample/guidelines/) in the Beeline. Simply set the `sampleRate` variable to your beeline configuration. This sends 1/n of all events, so a sample rate of 5 would send 20% of your events: ```javascript theme={} require("honeycomb-beeline")({ writeKey: "YOUR_API_KEY", serviceName: "your-service-name", sampleRate: 5 // This will send 1/5 (20%) of events /* ... additional optional configuration ... */ }); ``` The value of `sampleRate` **must** be a positive integer. Sampling by default will apply the same sampling decision to all spans in a trace, so adding sampling will not break your traces. Either all spans in a trace will be sent, or no spans in the trace will be sent. If sampling across multiple Beeline-instrumented services, set the same sample rate in all beeline configurations to avoid breaking traces. ### Customizing Sampling Logic The `samplerHook` configuration option is available to customize the logic used for deterministic sampling. To avoid breaking traces with the custom `samplerHook` option, ensure that sampling logic is applied to data that all spans within a trace will have (such as `trace.trace_id`). If needed, you may promote span fields to trace-level (making them available on all spans within a trace) by calling `addTraceContext()` at the beginning of a trace, for instance: `beeline.addTraceContext({ request_route: '/x/alive' })`. A custom samplerHook must return an object with this structure: ```javascript theme={} { shouldSample: boolean, // false will drop the event, true will keep it sampleRate: number // optional, will be reported to honeycomb } ``` For example, assume you have instrumented an HTTP server. You would like to keep all requests to login, skip all health checks, and sample the rest at a default rate. You could define a sampler function like so: ```javascript theme={} const beeline = require("honeycomb-beeline"); const createHash = require("crypto").createHash; beeline({ writeKey: "YOUR_API_KEY", serviceName: "your-service-name", samplerHook: samplerHook, }); // deterministicSampler function based on https://github.com/honeycombio/beeline-nodejs/blob/main/lib/deterministic_sampler.js function deterministicSampler(traceId, sampleRate) { const MAX_UINT32 = Math.pow(2, 32) - 1; const sum = createHash("sha1") .update(traceId) .digest(); const upperBound = (MAX_UINT32 / sampleRate) >>> 0; return sum.readUInt32BE(0) <= upperBound; } function samplerHook(data) { // default sample rate to 10 let sampleRate = 10; // default sampling decision to deterministic based on trace_id let shouldSample = deterministicSampler(data["trace.trace_id"], sampleRate); // drop all health checks requests if (data["request.path"] === "/x/alive") { shouldSample = false; sampleRate = 0; } return { shouldSample, sampleRate } } app.get('/x/alive', function(req, res) { res.send("I'm alive"); }); ``` Defining a sampling hook overrides the default deterministic sampling behavior for trace IDs. For head-based sampling behavior across a more complicated trace than the health check example above, you must ensure sampling decisions are based on a value all spans will have (such as trace\_id or a custom trace context as we did above). Otherwise, you will get incomplete traces. ## Distributed Trace Propagation When a service calls another service, you want to ensure that the relevant trace information is propagated from one service to the other. This allows Honeycomb to connect the two services in a trace. Distributed tracing enables you to trace and visualize interactions between multiple instrumented services. For example, your users may interact with a front-end API service, which talks to two internal APIs to fulfill their request. In order to have traces connect spans for all these services, it is necessary to propagate trace context between these services, usually by using an HTTP header. Both the sending and receiving service must use the same propagation format, and both services must be configured to send data to the same Honeycomb environment. Automatic instrumentation supports trace propagation automatically, as long as your services are using the Honeycomb beeline, and an instrumented component to send and receive requests (`express` and `https`). ### Interoperability With OpenTelemetry Trace context propagation with OpenTelemetry is done by sending and parsing headers that conform to the [W3C Trace Context specification](https://www.w3.org/TR/trace-context/). To get Beelines and OpenTelemetry instrumentation to interoperate, you will need to use W3C headers. The Beeline includes marshal and unmarshal functions that can generate and parse W3C Trace Context headers. Honeycomb Beelines default to using a Honeycomb-specific header format on outgoing requests, but can automatically detect incoming W3C headers and parse them appropriately. In mixed environments where some services are using OpenTelemetry and some are using Beeline, W3C header propagation should be used. To propagate trace context, a parser hook and propagation hook are needed. The parser hook is responsible for reading the trace propagation context out of incoming HTTP requests from upstream services. The propagation hook is responsible for returning the set of headers to add to outbound HTTP requests to propagate the trace propagation context to downstream services. Older versions of Honeycomb Beelines required HTTP parsing hooks to properly parse incoming W3C headers. Current versions of Honeycomb Beelines can automatically detect incoming W3C headers and parse them appropriately. Check the release notes for your Beeline version to confirm whether an upgraded version is needed. To specify that a service should propagate W3C Trace Context Headers with outgoing requests, you must specify a propagation hook in the beeline configuration. An `httpTraceParserHook` is a function that takes an HTTP request as an argument and returns a Honeycomb trace context object. The HTTP request is provided to the function so that the author can make decisions about whether to trust the incoming headers based on information contained in the request (for example, perhaps you do not want to accept headers from the public internet). An `httpTracePropagationHook` is a function that takes a Honeycomb trace context object as an argument and returns a map of name, value pairs representing serialized headers. This example adds these hooks to parse and propagate W3C headers: ```javascript theme={} const beeline = require('honeycomb-beeline'); beeline({ writeKey: 'YOUR_API_KEY', serviceName: 'your-service-name', httpTraceParserHook: beeline.w3c.httpTraceParserHook, httpTracePropagationHook: beeline.w3c.httpTracePropagationHook }); ``` ## Troubleshooting The Beeline ### No Traces for a Service The service name is a required configuration value. If it is unspecified, all trace data will be sent to a default dataset called `unknown_service`. ### The Events I Am Generating Do Not Contain The Content I Expect Enable debug output (sent to `STDOUT`) by setting `DEBUG=honeycomb-beeline:*` in your environment. This will print the JSON representation of events to `STDOUT` as well as other debug level details and errors. This lets you quickly see what is getting sent and allows you to modify your code accordingly. ```ini theme={} DEBUG=honeycomb-beeline:* ``` ### My Traces Are Showing Missing Root Spans There can be a number of reasons for missing root spans. One potential reason could be that there is an upstream service, load balancer, or other proxy propagating W3C trace headers as part of your distributed trace. Since beelines accept both Honeycomb and W3C headers, that service propagating a W3C header will cause "missing span" gaps in your trace if the service is not also configured to send telemetry to Honeycomb. The solution is to either instrument that service and configure it to send telemetry to Honeycomb, or to specify in the downstream service's beeline configuration that only Honeycomb propagation headers should be parsed. To override undesired W3C trace header propagation behavior, configure the Beeline to use an `httpTraceParserHook`: ```javascript theme={} const beeline = require("honeycomb-beeline"); beeline({ writeKey: "YOUR_API_KEY", serviceName: "your-service-name", httpTraceParserHook: beeline.honeycomb.httpTraceParserHook, }); ``` The above configuration will solely use the Honeycomb format when parsing incoming trace headers. See [Distributed Trace Propagation](#distributed-trace-propagation) for more details. ## Example Event Here is a sample event created by the Node.js Beeline: ```json theme={} { "Timestamp": "2018-03-20T00:47:25.339Z", "request.base_url": "", "request.fresh": false, "request.host": "localhost", "request.http_version": "HTTP/1.1", "request.remote_addr": "127.0.0.1", "request.method": "POST", "request.original_url": "/checkValid", "request.path": "/checkValid", "request.scheme": "http", "request.query": "{}", "request.secure": false, "request.url": "/checkValid", "request.xhr": true, "response.status_code": "200", "meta.instrumentation_count": 4, "meta.instrumentations": "[\"child_process\",\"express\",\"http\",\"https\"]", "meta.type": "express", "meta.version": "4.16.3", "meta.beeline_version": "1.0.2", "meta.node_version": "v9.10.0", "totals.mysql2.count": 2, "totals.mysql2.duration_ms": 13.291, "totals.mysql2.query.count": 2, "totals.mysql2.query.duration_ms": 13.291, "trace.trace_id": "11ad83a2-ca8d-4918-9db2-27524456d9f7", "trace.span_id": "4a3892ba-0936-46e1-8e17-31b887326027", "name": "request", "service_name": "express", "duration_ms": 15.229326 } ``` ## Queries To Try Try these examples to get started querying your app's behavior. ### Which Endpoints Are The Slowest? * Group By `url` * Visualize the `P99` of `duration_ms` values * Where `meta.type == express` * Order by `P99(duration_ms)` in descending (`DESC`) order ### Where Is My App Spending The Most Time? * Group By `meta.type` * Visualize the `P99` of `duration_ms` values * Order by `P99(duration_ms)` in descending (`DESC`) order ### Which Users Are Using The Endpoint I Would Like To Deprecate? * Group By `user.email` * Visualize the overall `COUNT` * Where `url ==` your endpoint url ### Which XHR Endpoints Take The Longest? * Group By `url` * Visualize the `P99` of `duration_ms` values * Where `meta.type == express` and `xhr == true` * Order by `P99(duration_ms)` in descending (`DESC`) order ## Contributions Bug fixes and other changes to Beelines are gladly accepted. Please open issues or a pull request with your change [via GitHub](https://github.com/honeycombio/beeline-nodejs). Remember to add your name to the CONTRIBUTORS file! All contributions will be released under the Apache License 2.0. # Beeline for Python Reference Source: https://docs.honeycomb.io/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/python Reference documentation for Beeline for Python, Honeycomb's legacy auto-instrumentation SDK. Beelines reached end of life on August 12, 2025. Migrate to OpenTelemetry. Beelines have reached [End of Life](/troubleshoot/product-lifecycle/release-stages/#end-of-life) and are now archived. * Adding new instrumentation? We recommend that you use [OpenTelemetry for Python](/send-data/python/) instead. * Already using a Beeline? We recommend that you [migrate to OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/). * Concerned about mixing Beelines and OpenTelemetry? Learn more about [Mixing Beelines and OpenTelemetry](/troubleshoot/product-lifecycle/recommended-migrations/migrate-from-beelines/#mixing-beelines-and-opentelemetry). The Python Beeline for Honeycomb is a quick and easy way to instrument your Python application. It has two powerful features: * Automatic instrumentation for common frameworks like Flask, Django, Lambda, and others - with a few lines of code, you can get rich events describing HTTP requests and database queries, and automatic tracing to tie these events together. With a few more lines, you can add application-specific context and tracing of your own. * A complete tracing API. Even if your application does not neatly fit into a box like Flask or Django, you can still wield the power of distributed tracing to by initiating your own traces and instrumenting components of your service with spans. Official API reference docs can be found [here](https://honeycombio.github.io/beeline-python/). To see an example of the Python Beeline in action, try one of the [Example Apps](https://github.com/honeycombio/beeline-python/tree/main/examples). If you would like to see more options or functionality in the Python Beeline, please [file an issue](https://github.com/honeycombio/beeline-python/issues) or vote up an already filed one! If you prefer to use only Structured Logs, check out our [Libhoney library](/send-data/python/libhoney/). ## Requirements * Python 3.7 or greater * A Honeycomb API key You can [find your API key](/configure/environments/manage-api-keys/#find-api-keys) in your Environment Settings. If you do not have an API key yet, sign up for a [free Honeycomb account](https://ui.honeycomb.io/signup). To use the Python Beeline for automatic instrumentation, you need to be using one of the following frameworks: * Flask * Django * AWS Lambda * Bottle * Tornado The Python Beeline does not currently support asyncio-based frameworks, and has limited support for Tornado. If you are using an asynchronous framework and would like to see support added in the Beeline, please [file an issue](https://github.com/honeycombio/beeline-python/issues). ## Installation If using the **dataset-only** data model, refer to the **Honeycomb Classic** tab for instructions. Not sure? Learn more about [Honeycomb versus Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments). To install the Python Beeline for your application: 1. Make sure your version of setuptools is up to date: ```shell theme={} pip install -U setuptools ``` 2. Install the Beeline package: ```shell theme={} pip install honeycomb-beeline ``` 3. Initialize the Beeline by including the following at the entry point to your application: ```python theme={} import beeline beeline.init( writekey='YOUR_API_KEY', # The name of your app/service is a good choice service_name='my-python-app', # debug=True, # defaults to False. if True, data doesn't get sent to Honeycomb ) ``` With `debug` set to `True` you'll see events being generated in your terminal output, but they won't be sent to Honeycomb. Set `debug=False` to start sending events to Honeycomb. To install the Python Beeline for your application: 1. Make sure your version of setuptools is up to date: ```shell theme={} pip install -U setuptools ``` 2. Install the Beeline package: ```shell theme={} pip install honeycomb-beeline ``` 3. Initialize the Beeline by including the following at the entry point to your application: ```python theme={} import beeline beeline.init( writekey='YOUR_API_KEY', # The name of your app is a good choice to start with dataset='my-python-app', service_name='my-python-app', # debug=True, # defaults to False. if True, data doesn't get sent to Honeycomb ) ``` With `debug` set to `True` you'll see events being generated in your terminal output, but they won't be sent to Honeycomb. Set `debug=False` to start sending events to Honeycomb. ### Working With Proxies Using a proxy requires overriding the default `Transmission` implementation when initializing `libhoney`: ```python theme={} import beeline from libhoney.transmission import Transmission beeline.init( writekey='YOUR_API_KEY', # The name of your app/service is a good choice service_name='my-python-app', debug=True, transmission_impl=Transmission(proxies={'https': 'https://myproxy.example.com'}), ) ``` The `proxies` map passed in is documented in the [requests](https://requests.readthedocs.io/en/latest/user/advanced/#proxies) documentation. Note that if you override `transmission_impl`, if you also have non-default values for options such as `max_concurrent_batches` and `max_batch_size` in `libhoney.init`, they will need to be specified in the new `Transmission` object. Further configuration options can be found in [the **init**.py file](https://github.com/honeycombio/libhoney-py/blob/main/libhoney/__init__.py#L45) Honeycomb API keys have the ability to create and delete data, and should be managed in the same way as your other application secrets. For example you might prefer to configure production API keys via environment variables, rather than checking them into version control. ## Using Automatic Instrumentation Implementation of automatic instrumentation varies between frameworks, but each need just a few lines of code. Each implementation instruments the entrypoint for your application (usually a request) by starting a trace, collecting some commonly used fields, ending the trace, and sending the event data to Honeycomb. You can build on this instrumentation by adding application-specific context as well as your own trace spans. The Beeline uses Django's request/response middleware (>1.10) and database query execution wrapper (>2.0). 1. In your project's `settings.py` file: ```python theme={} MIDDLEWARE = [ ... 'beeline.middleware.django.HoneyMiddleware', ... ] ``` If you would prefer to not use db instrumentation, use the `HoneyMiddlewareHttp` class instead. 2. In your app's `apps.py` file: ```python theme={} from django.apps import AppConfig import beeline class MyAppConfig(AppConfig): def ready(self): # If you use uwsgi, gunicorn, celery, or other pre-fork models, see the section below on pre-fork # models and do not initialize here. beeline.init( writekey='YOUR_API_KEY', service_name='my-app-name', debug=True, ) ``` Remember to set your app's default config in your app's `__init__.py` file: ```python theme={} default_app_config = 'myapp.apps.MyAppConfig' ``` The Beeline makes use of WSGI middleware. If you are using Flask's SQLAlchemy extension, you can also take advantage of our built-in database middleware to automatically instrument your SQL queries. Initialize the Beeline and pass your Flask app to `HoneyMiddleware`. ```python theme={} import beeline from beeline.middleware.flask import HoneyMiddleware # If you use uWSGI, Gunicorn, Celery, or other pre-fork models, do not initialize the Beeline here. # See the section below on pre-fork models. beeline.init(writekey='YOUR_API_KEY', service_name="my-app-name", debug=True) # Pass your Flask app to HoneyMiddleware app = Flask(__name__) HoneyMiddleware(app, db_events=True) # db_events defaults to True, set to False if not using our db middleware with Flask-SQLAlchemy ``` This will instrument HTTP requests. If you are using Flask's SQLAlchemy extension, it will also instrument DB events. If you are not using Flask SQLAlchemy or prefer not to instrument DB events, set `db_events` to `False`. If you use uWSGI, Gunicorn, Celery, or other pre-fork models, do not initialize the Beeline here. [See the section below on using the Python Beeline with pre-fork models.](#using-the-python-beeline-with-python-pre-fork-models) After initializing the Beeline according to those instructions, you will be able to pass your Flask app to `HoneyMiddleware` as shown here. The beeline instruments AWS Lambda using a decorator function that is applied to each Lambda handler function. ```python theme={} beeline.init(writekey='YOUR_API_KEY', service_name="my-lambda-function", debug=True) from beeline.middleware.awslambda import beeline_wrapper # wrapper starts a trace that ends when the function exits @beeline_wrapper def my_handler(event, context): # my code # add some context to this event beeline.add_context({"user_id": user_id}) ``` The Tornado integration patches the Tornado Web RequestHandlers with code to instrument HTTP requests and exceptions. It works for Tornado version 5 and above. ```python theme={} import beeline import libhoney from beeline.patch import tornado # If you use uwsgi, gunicorn, celery, or other pre-fork models, see the section below on pre-fork # models and do not initialize here. beeline.init( writekey='YOUR_API_KEY', service_name='my-app', # use a tornado coroutine rather than a threadpool to send events transmission_impl=libhoney.transmission.TornadoTransmission(), debug=True, ) ``` The beeline makes use of WSGI middleware to instrument HTTP requests. To use it, add the following code where your Bottle app is initialized: ```python theme={} import beeline from beeline.middleware.bottle import HoneyWSGIMiddleware # If you use uwsgi, gunicorn, celery, or other pre-fork models, see the section below on pre-fork # models and do not initialize here. beeline.init( writekey='YOUR_API_KEY', service_name='my-app', debug=True, ) app = bottle.app() myapp = HoneyWSGIMiddleware(app) bottle.run(app=myapp) ``` The beeline makes use of WSGI middleware to instrument HTTP requests. To use it, add the following code where your Werkzeug app is initialized: ```python theme={} import beeline from beeline.middleware.werkzeug import HoneyWSGIMiddleware # If you use uwsgi, gunicorn, celery, or other pre-fork models, see the section below on pre-fork # models and do not initialize here. beeline.init( writekey='YOUR_API_KEY', service_name='my-app', debug=True, ) app = YourApp() myapp = HoneyWSGIMiddleware(app) # ... start app ``` ### Instrumenting Outgoing Requests The Python Beeline supports auto-instrumentation of outgoing HTTP requests using the popular [requests](https://requests.readthedocs.io/en/latest/) library. When enabled, outgoing HTTP requests will generate their own span in a trace and include a Honeycomb trace context propagation header (`X-Honeycomb-Trace`) that can be used to construct distributed traces that cross process boundaries. To enable automatic instrumentation of requests, just import the appropriate patch into the current namespace: ```python theme={} import requests import beeline from beeline.patch.requests import requests beeline.init( writekey='YOUR_API_KEY', service_name='my-app', debug=True, ) result = requests.get('https://other-service.local/path/info') ``` ## Using Traces A Complete Trace

Metrics for resources running on the node are collected by communicating with the local node's kubelet directly. Logs from containers' `stdout` and `stderr` are written by the Docker daemon to the local node filesystem. The Honeycomb Kubernetes Agent reads these logs, augments them with [metadata](#metadata) from the Kubernetes API, and ships them to Honeycomb so that you can observe what is going on. Agent inside node

Screenshot of a calculated field using coalesce

CircleCI

GitHub Actions

GitLab CI

Buildkite

Jenkins X

Google Cloud Build

Bitbucket Pipelines

Travis CI

GitHub Deployment Protection

AWS Lambda

AWS Cloudwatch

Azure

Google Cloud

Claude

Cursor

Visual Studio Code

Amazon Q

GitHub Copilot

Embrace

Kubernetes

Istio

Kong

Ambassador

HashiCorp Terraform

Nginx

MySQL

PostgreSQL

JSON logs

Fastly

Fluentd

Logstash

Prometheus

HashiCorp Consul

HashiCorp Nomad

HashiCorp Vault

Anthropic

AWS PrivateLink

PagerDuty

Slack

Microsoft Teams

Webhooks