> ## Documentation Index > Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt > Use this file to discover all available pages before exploring further. # Text to JSON Quick Start > Instrument a text-to-JSON LLM workflow with OpenTelemetry and send telemetry to Honeycomb to observe model inputs, outputs, and validation steps. export const HnyIcon = ({alias, path, size = 16, iconColor}) => { const iconMap = { "home": "house.svg", "marker": "caretFilledDown.svg", "show-marker-options": "chatTextLeft.svg", "download": "arrowLineDown.svg", "trace-waterfall": "trace.svg", "show-query-details": "listDashes.svg", "table": "table.svg", "log-lines": "logLines.svg", "chart": "chartLine.svg", "show-settings": "gear.svg", "add": "plus.svg", "remove": "delete.svg", "persist": "caretDown.svg", "close": "close.svg", "copy": "copy.svg", "zoom-in": "magnifyingGlassPlus.svg", "zoom-out": "magnifyingGlassMinus.svg", "color-assignment": "drop.svg", "drag": "dots-six-vertical.svg", "drawer": "drawer.svg", "show-actions": "dotsThree.svg", "edit": "pencil.svg", "delete": "trash.svg", "move": "arrowsOutCardinal.svg", "show-legend": "circleInfo.svg", "usage-ok": "usageGood.svg", "usage-warning": "usageWarning.svg", "usage-danger": "usageDanger.svg", "open-query-builder": "query.svg", "home-menu": "house.svg", "query-menu": "query.svg", "boards-menu": "board.svg", "triggers-menu": "bell.svg", "slos-menu": "handshake.svg", "service-map-menu": "serviceMap.svg", "history-menu": "clockCounterClockwise.svg", "manage-data-menu": "cube.svg", "usage-menu": "usageGood.svg", "canvas-menu": "sparkle.svg", "anomalies-menu": "anomalies.svg", "show-details": "dotsThreeVertical.svg", "resize-handle": "board-panel-resize-handle.png", "standard-dataset": "cube.svg", "trace-dataset": "cubeChat.svg", "all-datasets": "linkedSquares.svg", "share": "arrowBentRight.svg", "run-in-query-builder": "arrowSquareUpRight.svg", "link": "link.svg", "text": "text.svg", "receive": "arrowLineDown.svg", "process": "lightning.svg", "sample": "drop.svg", "send": "arrowLineUp.svg", "submit": "arrowUp.svg", "canvas-menu": "sparkle.svg", "canvas": "sparkle.svg", "private": "lockKey.svg", "shared": "people.svg", "expand": "caretDown.svg", "previous": "caretLeft.svg", "next": "caretRight.svg" }; const iconBasePath = "/_assets/icons/"; const iconPath = path || (alias ? `${iconBasePath}${iconMap[alias]}` : undefined); return ; }; An extremely common use case for large language models (LLMs) is to convert text to JSON where the LLM: * generates a JSON object from user input * parses that object * validates its contents * uses the content elsewhere in the application Collecting observability data for this process is a great fit as clear and measurable outcomes are present for each step. This data answers questions like: * Do some user inputs result in higher latency or more errors? * Was the JSON object structurally valid as per the JSON schema? * Did the JSON object contain expected fields? * Did the new version of our prompt result in more invalid JSON objects? * Are there particular patterns in the user input that result in invalid JSON objects? Adding telemetry to your LLM and then analyzing with Honeycomb provides a flexible way to contribute towards and measure the overall user experience. Use our Quick Start to instrument your LLM application. ## Before You Begin Before instrumenting your LLM application, you'll need to do a few things: 1. **[Create a Honeycomb Account](https://ui.honeycomb.io/signup)**. Signup is free! 2. **Create a Honeycomb Team**. Complete your account creation by giving us a team name. Honeycomb uses teams to organize groups of users, grant them access to data, and create a shared work history in Honeycomb. We recommend using your company or organization name as your Honeycomb team name. 3. **Get Your Honeycomb API Key**. To send data to Honeycomb, you'll need your Honeycomb API Key. Once you create your team, you will be able to view or copy your API key. Make note of it; you will need it later! You can also [find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys) any time in your Environment Settings. ## Send Telemetry Data to Honeycomb Once you have your Honeycomb API key and your LLM application to instrument, it's time to send telemetry data to Honeycomb! To instrument your LLM, you will [add automatic instrumentation to your code](#add-automatic-instrumentation-to-your-code) for standard trace data telemetry, and then [add custom instrumentation specifically for your LLM](#add-custom-instrumentation-for-llms). ### Add Automatic Instrumentation to Your Code The quickest way to start seeing your trace data in Honeycomb is to use [OpenTelemetry](https://opentelemetry.io/), an open-source collection of tools, APIs, and SDKs, to automatically inject instrumentation code into your application without requiring explicit changes to your codebase. Automatic instrumentation works slightly differently within each language, but the general idea is that it attaches hooks into popular tools and frameworks and "watches" for certain functions to be called. When they're called, the instrumentation automatically starts and completes trace spans on behalf of your application. When you add automatic instrumentation to your code, OpenTelemetry will inject spans, which represent units of work or operations within your application that you want to capture and analyze for observability purposes. This Quick Start uses the `npm` dependency manager. For instructions with `yarn` or if using TypeScript, read our [OpenTelemetry Node.js documentation](/send-data/javascript-nodejs/opentelemetry-sdk/#add-automatic-instrumentation). #### Acquire Dependencies Open your terminal, navigate to the location of your project on your drive, and install OpenTelemetry's automatic instrumentation meta package and OpenTelemetry's Node.js SDK package: ```shell theme={} npm install --save \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-node ``` | Module | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `auto-instrumentations-node` | OpenTelemetry's meta package that provides a way to add automatic instrumentation to any Node application to capture telemetry data from a number of popular libraries and frameworks, like `express`, `dns`, `http`, and more. | | `sdk-node` | OpenTelemetry's Node.js distribution package that streamlines configuration and allows you to instrument as quickly and easily as possible. | Alternatively, install [individual instrumentation packages](https://github.com/open-telemetry/opentelemetry-js#instrumentations). If using TypeScript, install `ts-node` to run the code: ```shell theme={} npm install --save-dev ts-node ``` #### Initialize Create an initialization file, commonly known as the `tracing.js` file: ```javascript theme={} // Example filename: tracing.js 'use strict'; const { NodeSDK } = require('@opentelemetry/sdk-node'); const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); const sdk = new NodeSDK({ instrumentations: [ getNodeAutoInstrumentations(), ], }); sdk.start(); ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # (*) readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} - name: x-honeycomb-dataset value: ${HONEYCOMB_METRICS_DATASET} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Fields marked with an asterisk (\*) are required for exporting metrics to Honeycomb. Set the following environment variables before running your application: | Environment Variable | Value | | :-------------------------- | :-------------------------------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | | `HONEYCOMB_METRICS_DATASET` | The Honeycomb dataset to send metrics to (\*) | If you are using the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` in the configuration file. You can also configure the SDK using [environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/general/). If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must also specify the Dataset using the `x-honeycomb-dataset` header. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset" ``` If you are sending data directly to Honeycomb, you must configure the API key and service name. If you are using an [OpenTelemetry Collector](/send-data/opentelemetry/collector/), configure your API key at the Collector level instead. #### Run Your Application Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run the Node.js app with the initialization file: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml node -r ./tracing.js YOUR_APPLICATION_NAME.js ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. Alternatively, you can import the initialization file as the first step in your application lifecycle. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. This Quick Start uses the `pip` package manager. For instructions with `poetry`, read our [OpenTelemetry Python documentation](/send-data/python/opentelemetry-sdk/#add-automatic-instrumentation). #### Acquire Dependencies 1. Install the OpenTelemetry Python packages: ```shell theme={} python -m pip install opentelemetry-instrumentation \ opentelemetry-distro \ opentelemetry-exporter-otlp ``` 2. Install instrumentation libraries for the packages used by your application. We recommend using the `opentelemetry-bootstrap` tool that comes with the OpenTelemetry SDK to scan your application packages and print out a list of available instrumentation libraries. You should then add these libraries to your `requirements.txt` file: ```shell theme={} opentelemetry-bootstrap >> requirements.txt pip install -r requirements.txt ``` If you do not use a `requirements.txt` file, you can install the libraries directly in your current environment: ```shell theme={} opentelemetry-bootstrap --action=install ``` #### Configure the OpenTelemetry SDK Use environment variables to configure the OpenTelemetry SDK: ```shell theme={} export OTEL_SERVICE_NAME="your-service-name" export OTEL_TRACES_EXPORTER="otlp" export OTEL_LOGS_EXPORTER="otlp" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443" # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu1.honeycomb.io:443" # EU instance export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=" export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true ``` | Variable | Description | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_SERVICE_NAME` | Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The data format that the SDK uses to send telemetry to Honeycomb. For more on data format configuration options, read [Choosing between gRPC and HTTP](/send-data/python/opentelemetry-sdk/#choosing-between-grpc-and-http). | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Honeycomb endpoint to which you want to send your data. | | `OTEL_EXPORTER_OTLP_HEADERS` | Adds your Honeycomb API Key to the exported telemetry headers for authorization. [Learn how to find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys). | | `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` | Enable logs auto-instrumentation using Python root logger. | To learn more about configuration options, visit [Agent Configuration](https://opentelemetry.io/docs/languages/python/automatic/agent-config/) in the OpenTelemetry documentation. If you use [Honeycomb Classic](/troubleshoot/product-lifecycle/recommended-migrations/#migrate-from-honeycomb-classic-to-honeycomb-environments), you must also specify the Dataset using the `x-honeycomb-dataset` header. ```shell theme={} export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=your-dataset" ``` If you are sending data directly to Honeycomb, you must configure the API key and service name. If you are using an [OpenTelemetry Collector](/send-data/opentelemetry/collector/), configure your API key at the Collector level instead. #### Run Your Application Run your Python application using the OpenTelemetry Python automatic instrumentation tool `opentelemetry-instrument`, which configures the OpenTelemetry SDK: ```shell theme={} opentelemetry-instrument \ --traces_exporter console \ --metrics_exporter console \ --logs_exporter console \ python YOUR_APPLICATION_NAME.py ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. #### Acquire Dependencies The automatic instrumentation agent for OpenTelemetry Java will automatically generate trace data from your application. The agent is packaged as a JAR file and is run alongside your app. In order to use the automatic instrumentation agent, you must first download it: ```shell theme={} curl -L -O https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # (*) readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} - name: x-honeycomb-dataset value: ${HONEYCOMB_METRICS_DATASET} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Fields marked with an asterisk (\*) are required for exporting metrics to Honeycomb. Set the following environment variables before running your application: | Environment Variable | Value | | :-------------------------- | :-------------------------------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | | `HONEYCOMB_METRICS_DATASET` | The Honeycomb dataset to send metrics to (\*) | If you are using the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` in the configuration file. You can also configure the SDK using [environment variables or system properties](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure). #### Run Your Application Point the agent at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml java -javaagent:opentelemetry-javaagent.jar -jar /path/to/myapp.jar ``` In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. This Quick Start uses ASP.NET Core. #### Acquire Dependencies Install the OpenTelemetry .NET packages. For example, with the .NET CLI, use: ```shell theme={} dotnet add package OpenTelemetry dotnet add package OpenTelemetry.Extensions.Hosting dotnet add package OpenTelemetry.Instrumentation.AspNetCore dotnet add package OpenTelemetry.Instrumentation.Http ``` #### Initialize Initialize the TracerProvider during application setup. ```csharp theme={} services.AddOpenTelemetry().WithTracing(builder => builder .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddOtlpExporter()); ``` #### Configure Use environment variables to configure the OpenTelemetry SDK: ```shell theme={} export OTEL_SERVICE_NAME="your-service-name" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443" # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu1.honeycomb.io:443" # EU instance export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=" ``` | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_SERVICE_NAME` | Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | The data format that the SDK uses to send telemetry to Honeycomb. For more on data format configuration options, read [Choosing between gRPC and HTTP](/send-data/dotnet/#choosing-between-grpc-and-http). | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Honeycomb endpoint to which you want to send your data. | | `OTEL_EXPORTER_OTLP_HEADERS` | Adds your Honeycomb API Key to the exported telemetry headers for authorization. [Learn how to find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys). | #### Run Run your application. You will see the incoming requests and outgoing HTTP calls generate traces. ```shell theme={} dotnet run ``` In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. #### Acquire Dependencies Install OpenTelemetry Go packages: ```shell theme={} go get \ go.opentelemetry.io/contrib/otelconf \ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp ``` #### Initialize Prepare your application to send spans to Honeycomb. Open or create a file called `main.go`: ```go theme={} package main import ( "context" "fmt" "log" "net/http" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" "go.opentelemetry.io/contrib/otelconf" "go.opentelemetry.io/otel" ) // Implement an HTTP Handler function to be instrumented func httpHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello, World") } func main() { // use otelconf to set up OpenTelemetry SDK sdk, err := otelconf.NewSDK() if err != nil { log.Fatalf("error setting up OTel SDK - %e", err) } defer sdk.Shutdown(context.Background()) otel.SetTracerProvider(sdk.TracerProvider()) otel.SetTextMapPropagator(sdk.Propagator()) // Initialize HTTP handler instrumentation handler := http.HandlerFunc(httpHandler) wrappedHandler := otelhttp.NewHandler(handler, "hello") http.Handle("/hello", wrappedHandler) // Serve HTTP server log.Fatal(http.ListenAndServe(":3030", nil)) } ``` #### Configure the OpenTelemetry SDK Create an `otelconfig.yaml` file with the following content: ```yaml theme={} file_format: "1.0" resource: attributes: - name: service.name value: ${OTEL_SERVICE_NAME:-my-service} tracer_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} meter_provider: # (*) readers: - periodic: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} - name: x-honeycomb-dataset value: ${HONEYCOMB_METRICS_DATASET} logger_provider: processors: - batch: exporter: otlp_http: endpoint: https://api.honeycomb.io headers: - name: x-honeycomb-team value: ${HONEYCOMB_API_KEY} propagator: composite: - tracecontext: - baggage: ``` Fields marked with an asterisk (\*) are required for exporting metrics to Honeycomb. Set the following environment variables before running your application: | Environment Variable | Value | | :-------------------------- | :-------------------------------------------- | | `HONEYCOMB_API_KEY` | Your Honeycomb API key | | `OTEL_SERVICE_NAME` | The name of your service | | `HONEYCOMB_METRICS_DATASET` | The Honeycomb dataset to send metrics to (\*) | If you are using the EU instance, replace `https://api.honeycomb.io` with `https://api.eu1.honeycomb.io` in the configuration file. You can also configure the SDK using [environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/general/). #### Run Your Application Point the SDK at your configuration file using the `OTEL_CONFIG_FILE` environment variable, then run your application: ```shell theme={} OTEL_CONFIG_FILE=./otelconfig.yaml go run YOUR_APPLICATION_NAME.go ``` Be sure to replace `YOUR_APPLICATION_NAME` with the name of your application's main file. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. #### Acquire Dependencies Add these gems to your Gemfile: ```ruby theme={} gem 'opentelemetry-sdk' gem 'opentelemetry-exporter-otlp' gem 'opentelemetry-instrumentation-all' ``` | Gem | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `opentelemetry-sdk` | Required to create spans | | `opentelemetry-exporter-otlp` | An exporter to send data in the OTLP format | | `opentelemetry-instrumentation-all` | A meta package that provides instrumentation for Rails, Sinatra, several HTTP libraries, [and more](https://github.com/open-telemetry/opentelemetry-ruby#instrumentation-libraries) | Install the gems using your terminal: ```shell theme={} bundle install ``` #### Initialize Initialize OpenTelemetry early in your application lifecycle. For Rails applications, we recommend that you use a Rails initializer. For other Ruby services, initialize as early as possible in the startup process. ```ruby theme={} # config/initializers/opentelemetry.rb require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' OpenTelemetry::SDK.configure do |c| c.use_all() # enables all instrumentation! end ``` #### Configure the OpenTelemetry SDK Use environment variables to configure OpenTelemetry to send events to Honeycomb: ```ruby theme={} export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io" # US instance #export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.eu1.honeycomb.io" # EU instance export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key" export OTEL_SERVICE_NAME="your-service-name" ``` | Variable | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Base endpoint to which you want to send your telemetry data. | | `OTEL_EXPORTER_OTLP_HEADERS` | List of headers to apply to all outgoing telemetry data. Place your API Key generated in Honeycomb in the `x-honeycomb-team` header. [Learn how to find your Honeycomb API Key](/configure/environments/manage-api-keys/#find-api-keys). | | `OTEL_SERVICE_NAME` | Service name. When you send data, Honeycomb creates a dataset in which to store your data and uses this as the name. Can be any string. | #### Run Your Application Run your Ruby application. In Honeycomb's UI, you should now see your application's incoming requests and outgoing HTTP calls generate traces. If your preferred language is not covered here, you can find relevant instrumentation information in the [OpenTelemetry community documentation](https://opentelemetry.io/docs/languages/). For Rust, we recommend you use the [opentelemetry](https://crates.io/crates/opentelemetry) and [opentelemetry-otlp](https://crates.io/crates/opentelemetry-otlp) crates to send data to Honeycomb over OTLP. For any required configuration values, see [Using the Honeycomb OpenTelemetry Endpoint](/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint). ### Generate Automated Data Now that you have added automatic instrumentation to your application and have it running in your development environment, interact with your application by making a few requests. Making requests to your service will generate telemetry data and send it to Honeycomb where it will appear in the Honeycomb UI within seconds. ### Add Custom Instrumentation for LLMs With OpenTelemetry's automatic instrumentation now installed in your application, trace data is being sent to Honeycomb. The next step is to add custom instrumentation that tracks all relevant information related to your LLM feature. In OpenTelemetry, custom instrumentation is called manual instrumentation. To get the most out of your traces, you must use OpenTelemetry APIs to instrument. To add custom instrumentation, create a single span that tracks all relevant information related to your LLM feature. Specifically, the minimum information to track must include: * User ID * Prompt version * User input * Full prompt text * Full LLM response * Any error including parsing JSON and/or validating it * Error message * Token count #### Examples The following code examples show how to capture the correct information on an OpenTelemetry span: ```ts Node.js theme={} import { trace, Span, SpanStatusCode } from "@opentelemetry/api"; const tracer = trace.getTracer("llm.tracer"); function getJsonFromText( userInput: string, userId: string, promptTemplate: string, promptVersion: string ) { return tracer.startActiveSpan("app.get_json_from_text", (span: Span) => { span.setAttribute("app.user_id", userId); span.setAttribute("app.llm.prompt_version", promptVersion); span.setAttribute("app.llm.user_input", userInput); try { // Programmatically build the full prompt. // The output is the entire prompt you'd send to the LLM, // after RAG or any other context-building operations. const fullPrompt = buildFullPrompt(promptTemplate, userInput); span.setAttribute("app.llm.prompt_text", fullPrompt); // Call the LLM and get back the text of the result // and the number of tokens used. const { response, tokenCount } = callLLM(full_prompt); span.setAttribute("app.llm.response", response); span.setAttribute("app.llm.token_count", tokenCount); // Parse the JSON object and validate it, // capturing any errors you might encounter. const result = parseAndValidateResponse(response); return result; } catch (ex) { // Track any unexpected errors. span.setStatus({ code: SpanStatusCode.ERROR }); span.recordException(ex); } finally { span.end(); } }); } function parseAndValidateResponse(llmResult: string) { const currentSpan = trace.getActiveSpan(); // Extract and parse the JSON object from the LLM response. const { extracted, result, extractionError } = extractAndParseJson(llmResult); if (!extracted) { currentSpan.setAttribute("error.message", extractionError); currentSpan.setStatus({ code: SpanStatusCode.ERROR }); return null; } // Validate the structure of the result, capturing // any validation errors you might encounter. const { validated, validationError } = validateResult(result); if (!validated) { currentSpan.setAttribute("error.message", validationError); } return result; } ``` ```python Python theme={} from opentelemetry import trace from opentelemetry.trace import Status, StatusCode tracer = trace.get_tracer("llm.tracer") def get_json_from_text(user_input, user_id, prompt_template, prompt_version): with tracer.start_as_current_span("app.get_json_from_text") as span: span.set_attribute("app.user_id", user_id) span.set_attribute("app.llm.prompt_version", prompt_version) span.set_attribute("app.llm.user_input", user_input) try: # Programmatically build the full prompt. # The output is the entire prompt you'd send to the LLM, # after RAG or any other context-building operations. full_prompt = build_full_prompt(prompt_template, user_input) span.set_attribute("app.llm.prompt_text", full_prompt) # Call the LLM and get back the text of the result # and the number of tokens used. llm_result, token_count = call_llm(full_prompt) span.set_attribute("app.llm.response", response) span.set_attribute("app.llm.token_count", token_count) # Parse the JSON object and validate it, # capturing any errors you might encounter. result = parse_and_validate_response(llm_result) return result except Exception as ex: # Track any unexpected errors. current_span.set_status(Status(StatusCode.ERROR)) current_span.record_exception(ex) def parse_and_validate_response(llm_result): current_span = trace.get_current_span() # Extract and parse the JSON object from the LLM response. extracted, result, extraction_error = extract_and_parse_json(llm_result) if not extracted: current_span.set_attribute("error.message", extraction_error) current_span.set_status(Status(StatusCode.ERROR)) return None # Validate the structure of the result, capturing # any validation errors you might encounter. validated, validation_error = validate_result(result) if not validated: current_span.set_attribute("error.message", validation_error) return result ``` ## Explore Your Data With your app running and telemetry being sent to Honeycomb, it's time to explore your data. ### Create a Board For quick reference over time, you should create a [Board](/observe/boards/) to show LLM-specific queries of interest. We recommend creating a Board first before [trying queries](#create-queries), so you can save with ease later. To create a Board: 1. In the Honeycomb UI's left navigation menu, select **Boards** ()n. When the left navigation menu is compact, only the icon appears. 2. Select **New Board**. 3. In the modal that appears, name your new board, such as "LLM Dashboard." Optionally, give your Board a description to help others find and use it. Determine the board's Sharing setting - **Public to the Team** or **Limited to Collaborators**. 4. Select **Create** to finish. Your new board appears next. #### Next Steps 1. Select **Add Query** to go to the Query Builder display. 2. Use the example queries in the next section to populate your LLM Board. 3. Follow the directions to [add queries to an existing Board](/observe/boards/#adding-queries). ### Create Queries Now it's time to create your first queries for LLMs! Use the query examples below to explore the performance and behavior of your LLM application. The specific attributes should exist in your data and environment if you added [custom instrumentation for LLMs](#add-custom-instrumentation-for-llms) in the previous step. Enter each example query using the [Query Builder](/investigate/query/build/). These example queries use two to three of the `VISUALIZE`, `WHERE`, and `GROUP BY` clauses, located at the top of the Query Builder. * `VISUALIZE` - Performs a calculation and displays a corresponding graph over time. Most `VISUALIZE` queries return a line graph while the `HEATMAP` visualization shows the distribution of data over time * `WHERE` - Filters based on attribute parameter(s) * `GROUP BY` - Groups fields by attribute parameter(s) Screenshot of Visualize, Where, and Group by clauses in Query Builder #### Track Overall Latency This query tracks overall latency of all LLM-related operations and the slowest requests. | VISUALIZE | WHERE | | ---------------------------------------------- | -------------------------------- | | HEATMAP(duration\_ms)
MAX(duration\_ms) | name = app.get\_json\_from\_text | Use to identify any spikes in latency, or if latency is increasing over time. In the event of a spike, you can investigate what happened by using [BubbleUp](/investigate/analyze/identify-outliers/) to find outliers. #### Track Invalid JSON Objects This query shows each instance that a user input led to a bad JSON object, whether that was because of a parsing error or a validation error. | VISUALIZE | WHERE | GROUP BY | | --------- | ---------------------------------------------------- | ---------------------------------------------------------- | | COUNT | name = app.get\_json\_from\_text
error exists | app.llm.input
app.llm.response
error.message | Use to identify exactly which inputs lead to bad behavior, which makes it easier to identify specific bugs to solve. #### Track all User Inputs Grouped by Response and Errors This query shows groups of all inputs and LLM outputs that succeeded. | VISUALIZE | WHERE | GROUP BY | | --------- | ------------------------------------------------------------- | ------------------------------------------- | | COUNT | name = app.get\_json\_from\_text
error does-not-exists | app.llm.user\_input
app.llm.response | Use to understand general user behavior, and to identify any patterns in user input that leads to a particularly useful response. It's just as helpful to understand **what's working** as it is to understand what isn't. #### Show Token Usage Over Time This query tracks token use over time, grouped by user ID. | VISUALIZE | WHERE | GROUP BY | | ----------------------------- | -------------------------------- | ------------ | | HEATMAP(app.llm.token\_count) | name = app.get\_json\_from\_text | app.user\_id | Use to understand how many tokens being used over time, but also to identify that usage down to specific users, as often a small number of users is responsible for the majority of usage. ### Investigate Specific Traces The queries on our LLM Board act as a starting point. If curious about specific behavior, you can view a specific trace that represents one request. Select any point on a graph, and in the menu that appears, select **View trace**. The next screen displays a [trace detail view](/reference/honeycomb-ui/query/trace-waterfall/) that lets you see what happened step by step. ## Next Steps The queries on our LLM board act as a referenceable starting point. If you're curious about specific behavior(s), start with any query and: 1. Add additional fields in the **GROUP BY** clause to slice your data into revealing interesting field values. 2. Use [BubbleUp](/investigate/analyze/identify-outliers/) to find outlier behavior and identify its contributing characteristics. 3. Select a specific trace that represents one request to see what happened step-by-step.