How LivePerson optimized Logstash and Kafka performance on GCP through benchmarking

LivePerson’s observability team optimized their high-volume logging pipeline by benchmarking various GCP machine types, discovering that switching to the `n4d-standard-2` instance and using LZ4 compression reduced processing costs by over 50%. These infrastructure changes significantly increased throughput and allowed for a smaller, more efficient Kafka cluster by reducing the number of required partitions. Ultimately, the study demonstrates that strategic hardware selection, rather than just software configuration, is a high-leverage method for cost optimization at scale.