So you want to use Honeycomb. Maybe this is because you want high cardinality or faster querying times. Maybe you are at the beginning of a project to onboard to tracing or OpenTelemetry.
But right now, you rely on unstructured logs for core analysis. Event logs are good for debugging, easy to generate, and unstructured logs are often the easiest to implement.
The downside? The lack of standardization makes debugging using unstructured logs challenging, and querying unstructured logs is highly inefficient. You can sometimes do post-processing with regular expressions, but if you have to use regular expressions to understand what is happening in production, you are wasting your analytical skills on what amounts to a data formatting problem instead of actually analyzing the information the system is sending.
The best solution is to transform your unstructured logs into a structured format before sending to Honeycomb. But regardless of whether you are ready to migrate to structured logs, if you have unstructured logs that you would like to get into Honeycomb, then you are in the right place.
Honeycomb refers to unstructured data as information that does not follow a predefined data model. Often this information takes the form of logs emitted by applications to log files, but it can also include raw text or any kind of event data that has not been formatted into a structured format like JSON or key-value pairs.
Some examples:
2024-10-15 12:00:00 ERROR Connection timeout while accessing databaseUser login failed: username=admin, reason=invalid passwordServer xyz failed to respond due to high CPU usageStacktrace: at main.py: line 23If you send unstructured logs to Honeycomb without transforming them into structured logs, then we will accept the data, but treat it as a single, opaque field, which will be harder to filter or query on. This also means you will miss out on some powerful observability features, such as tracing, high-cardinality querying, and exploring data interactively.
To avoid this, we recommend transforming unstructured data into structured data. For example, you might transform this unstructured log:
2024-10-15 12:00:00 ERROR connection timeout
Into this structured log:
{"timestamp": "2024-10-15T12:00:00Z", "status": "error", "message": "connection timeout"}
If you’re ready to transform your unstructured logs into structured logs, you can use any of the following options:
Honeycomb Telemetry Pipeline: Use the Honeycomb Telemetry Pipeline, which lets you standardize the entirety of telemetry operations on OpenTelemetry, whatever the format of existing logs, traces, and metrics.
OpenTelemetry Collector: Use the OpenTelemetry Collector as a logging agent along with the Filelog Receiver to parse unstructured logs and sending them to Honeycomb as structured logs.
HoneyTail + RegEx: Ingest unstructured logs using custom regular expressions and Honeytail, our lightweight tool that will tail your existing log files, parse the content, and send it to Honeycomb.
Once your data is in Honeycomb, you will want to enhance it. If you’ve transformed your data to structured logs, you can use the resources shared in Start Building: Structured Logs.
Otherwise, consider adding metadata like timestamps or categories to help you query later.
For structured logs, Honeycomb lets you easily query based on fields like status or timestamp. For unstructured logs, you can still use Honeycomb, but it will be harder to filter or analyze your data because the information is contained in one text blob.
To learn how to analyze your data in Honeycomb, visit Investigate Log Data in Honeycomb.