> ## Documentation Index
> Fetch the complete documentation index at: https://docs.honeycomb.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubernetes Container Logs

> Read container logs from Kubernetes nodes

<Badge className="hny-badge-enterprise-addon" stroke>Ent+</Badge>

<Note>
  This feature is available as an add-on for the [Honeycomb Enterprise plan](https://www.honeycomb.io/pricing/).
  Please contact your Honeycomb account team for details.
</Note>

## Supported Platforms

| Platform              | Metrics | Logs | Traces |
| :-------------------- | :------ | :--- | :----- |
| Kubernetes DaemonSet  |         | ✓    |        |
| OpenShift 4 DaemonSet |         | ✓    |        |

## Configuration Table

| Field             | Default                                   | Description                                                                                                                                                          |
| :---------------- | :---------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cluster Name\*    |                                           | The cluster name that will be added as the `k8s.cluster.name` resource attribute.                                                                                    |
| Log Source        | File                                      | Where to read logs from. Generally, this is `file`. `file` source supports Docker json-file and Containerd cri-o log formats. Options include `file` and `journald`. |
| File Path(s)      | `/var/log/containers/\*.log`              | When log\_source is file. File or directory paths to tail for logs. Defaults to all container logs.                                                                  |
| Journald Path\*   |                                           | When log\_source is journald. The directory containing Journald's log files.                                                                                         |
| Exclude File Path | `/var/log/containers/bindplane-*-agent-*` | File or directory paths to exclude. Generally, the collector's own log should be excluded.                                                                           |
| Start At          | end                                       | Start reading the logs from the 'beginning' or 'end'.                                                                                                                |
| Recombine Logs    |                                           | Options for configuring multi line logging. See [Multi Line Logging](#multi-line-logging).                                                                           |

## Example Configuration

The Kubernetes Container logs source has one required parameter:

* Cluster Name: The name of the cluster, which will be inserted as the `k8s.cluster.name` resource attribute

<img src="https://mintcdn.com/honeycomb/yjwgE9IWY42i02gc/_assets/images/htp/xnapper-kubernetes-container-source.png?fit=max&auto=format&n=yjwgE9IWY42i02gc&q=85&s=fc9707ab18674d927ed055866ccd9aa3" alt="Honeycomb Docs - Kubernetes Container Logs - image 1" width="2166" height="2034" data-path="_assets/images/htp/xnapper-kubernetes-container-source.png" />

Once running on an agent, logs will look like this:

<img src="https://mintcdn.com/honeycomb/yjwgE9IWY42i02gc/_assets/images/htp/xnapper-kubernetes-container-source-2.png?fit=max&auto=format&n=yjwgE9IWY42i02gc&q=85&s=0e7870deed5b0db361786d102df60c7f" alt="Honeycomb Docs - Kubernetes Container Logs - image 2" width="2942" height="1890" data-path="_assets/images/htp/xnapper-kubernetes-container-source-2.png" />

## Multi Line Logging

Multi line logging is useful for re-assembling multi line logs into a single log entry. Multi-line log
re-assembly requires that all logs emitted by the application are consistent in structure. For example, the
logs must start or end in a consistent way, in order to reliably match on the beginning or end of the log.
If your application has inconsistent logging, multi-line log re-assembly will behave in irregular ways, such
as combining two unique logs into one.

Multi line logging is supported by configuring a selector, selector match expression, and a recombine match expression.

| Field                      | Description                                                                         |
| -------------------------- | ----------------------------------------------------------------------------------- |
| Selector                   | The OTTL path to match on.                                                          |
| Selector Match Expression  | A regular expression used to match the selector.                                    |
| Recombine Type             | Whether or not to recombine logs by matching the first or last line in the log.     |
| Recombine With             | The delimiter used to recombine logs. Defaults a single space or newline character. |
| Recombine Match Expression | The regular expression used to recombine the multi-line log.                        |

### Example

In this example, there are two Deployments. One is logging XML while the other is logging JSON.

```bash theme={}
NAME                    READY   STATUS    RESTARTS      AGE
json-7d4fd9fd99-f29sl   1/1     Running   1 (38m ago)   3h49m
xml-b44858b84-hdbqp     1/1     Running   1 (38m ago)   3h49m
```

The XML logs are a combination of multi-line and single line logs. Each log has a timestamp prefix
indicating the start of the log.

```xml theme={}
2024-07-01 18:49:15 <person>
                      <name>John Doe</name>
  <age>30</age>
</person>
2024-07-01 18:49:15 <person><name>John Doe</name><age>30</age></person>
```

The JSON logs are a combination of multi-line and single line logs. Each log has a trailing `}` without
a comma, indicating the end of the log.

```json theme={}
{
  "timestamp": "2024-07-01 18:51:56",
  "name": "John Doe",
  "user": {
    "name": "John Doe",
    "age": 25
  },
  "location": "New York",
  "type": "Checking",
  "severity": "warn"
}
{"timestamp":"2024-07-01 18:51:56","name":"John Doe","user":{"name":"John Doe","age":25},"location":"New York","type":"Checking","severity":"warn"}
```

Multi-line logging can be configured by matching on the First Entry of the XML logs and the Last Entry
of the JSON logs. The `k8s.pod.name` resource attribute is used to select the XML and JSON pods.

<img src="https://mintcdn.com/honeycomb/ibXJI2unoLZRp0G4/_assets/images/htp/xnapper-kubernetes-container-multi-line.png?fit=max&auto=format&n=ibXJI2unoLZRp0G4&q=85&s=f03228de4e743a7af4be1be75bb82302" alt="Honeycomb Docs - Kubernetes Container Multi Line Logs - image" width="2014" height="1390" data-path="_assets/images/htp/xnapper-kubernetes-container-multi-line.png" />

Once configured, logs will be re-assembled into a single line.

<img src="https://mintcdn.com/honeycomb/ibXJI2unoLZRp0G4/_assets/images/htp/xnapper-kubernetes-container-multi-line-2.png?fit=max&auto=format&n=ibXJI2unoLZRp0G4&q=85&s=d99b72cd054b0a8e0cc98c12f9cd514f" alt="Honeycomb Docs - Kubernetes Container Multi Line Logs - image 2" width="3160" height="2124" data-path="_assets/images/htp/xnapper-kubernetes-container-multi-line-2.png" />
