1. Docs
2. Send Data to Honeycomb
3. Honeycomb Telemetry Pipeline
4. Pipeline Manager Sources
5. Google Cloud PubSub

Google Cloud PubSub

Table of Contents

Description 

Read messages from a Google Cloud PubSub topic and generate logs, metrics, or traces from the result.

Supported Platforms 

Configuration 

Authentication 

The service account used by the collector should have the Pub/Sub Subscriber permission to read from the subscription. Additionally, make sure the subscription channel being read from is configured with Pull delivery.

Authentication is handled via the GOOGLE_APPLICATION_CREDENTIALS environment variable. This should point to a file containing the service account key for the project.

To set this for the collector, you need to modify the service file depending on your OS.

Linux 

Use a Systemd override:

sudo systemctl edit observiq-otel-collector

Then run the following command to reload the systemd configuration:

sudo systemctl daemon-reload
sudo systemctl restart observiq-otel-collector

Windows 

Start powershell as administrator and run the following command:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\observiq-otel-collector" /v Environment /t REG_MULTI_SZ /d "GOOGLE_APPLICATION_CREDENTIALS=<path to key file>" /f

Then restart the collector service:

Restart-Service observiq-otel-collector

macOS 

Add GOOGLE_APPLICATION_CREDENTIALS to the EnvironmentVariables dict in the launchd service file /Library/LaunchDaemons/com.observiq.collector.plist (other values are shown for context):

<key>EnvironmentVariables</key>
<dict>
    <key>GOOGLE_APPLICATION_CREDENTIALS</key>
    <string><path to key file></string>
    <key>OIQ_OTEL_COLLECTOR_HOME</key>
    <string>/opt/observiq-otel-collector/</string>
    <key>OIQ_OTEL_COLLECTOR_STORAGE</key>
    <string>/opt/observiq-otel-collector/storage</string>
</dict>

Then restart the collector:

sudo launchctl unload /Library/LaunchDaemons/com.observiq.collector.plist
sudo launchctl load /Library/LaunchDaemons/com.observiq.collector.plist

Attribute Labels 

The Google Cloud PubSub receiver looks for the following attributes on the message to determine the message encoding and compression:

• content-type - How the message content should be interpreted.
  • If this value is set to application/protobuf, another attribute, ce-type, is expected to determine telemetry type. This attribute should be one of org.opentelemetry.otlp.traces.v1, org.opentelemetry.otlp.metrics.v1, or org.opentelemetry.otlp.logs.v1.
• content-encoding - Despite being called the “content-encoding” this controls the compression of the message.

If these attributes are present on the message data, the Payload Encoding and Payload Compression source parameters are ignored and should be set to “none”.

If these attributes are not present on the message data, the receiver will use the Payload Encoding and Payload Compression parameters to determine the encoding and compression.

Metrics and traces can only be OTLP Proto. Logs can be OTLP JSON, OTLP Proto, or Raw Text. For more information, see the OpenTelemetry Receiver Documentation.

Example Configuration 

In this example, we are parsing log data from the given PubSub subscription. The expected attributes are not present on the PubSub message data, so the Payload Encoding and Payload Compression parameters are used to determine the encoding and compression.

If these attributes were present, then the Payload Encoding and Payload Compression parameters should be set to “none”.

Standalone Source:

apiVersion: bindplane.observiq.com/v1
kind: Source
metadata:
  id: googlecloudpubsub
  name: googlecloudpubsub
spec:
  type: googlecloudpubsub
  parameters:
    - name: telemetry_types
      value: ['Logs']
    - name: project
      value: 'my_project_id'
    - name: subscription
      value: 'my_subscription_name'
    - name: encoding
      value: 'raw_text'
    - name: compression
      value: 'gzip'

Did you find what you were looking for?

Yes No

---

If you have questions, join our Pollinators Community Slack!
Pro/Enterprise users can visit Honeycomb Support or email support@honeycomb.io.

Submit