Refinery provides a variety of configuration options that allow operators to tune the product to handle a variety of different volumes and shapes of telemetry data.
We recommend increasing the amount of RAM and the number of cores after your initial set-up. Use our scale documentation below, and our troubleshooting documentation to learn more.
Monitor incoming_router_dropped and peer_router_dropped, and look for values above 0.
If either metric is consistently above 0, increase the corresponding receive buffer size.
Monitor libhoney_peer_queue_overflow and look for values above 0.
If it is consistently above 0, increase PeerBufferSize.
The default PeerBufferSize is 100,000.
Monitor libhoney_upstream_queue_length and look for values to stay under the UpstreamBufferSize value.
If it hits UpstreamBufferSize, then Refinery will block waiting to send upstream to the Honeycomb API.
Adjust the UpstreamBufferSize as needed.
The default UpstreamBufferSize is 10,000.
Monitor CPU usage on the host(s), and target for 80% CPU usage. Spiking to 90% is acceptable but avoid spiking to 100%. If CPU utilization is too high, add more cores or more hosts as needed.
As a starting point, set MaxMemoryPercentage to 75 to use 75% of AvailableMemory.
For example, if the system has 4GB of memory, 75% would be about 3GB.
To tune the MaxMemoryPercentage value, monitor process_uptime_seconds and look for restarts.
If Refinery restarts due to Out Of Memory exceptions or the host’s Out Of Memory Killer, decrease MaxMemoryPercentage to give Refinery more head room on the system.
