Groupcache for Thanos Store

Thanos turns Prometheus into a highly available metrics system with long-term storage. Its Store Gateway serves historical data straight from object storage such as S3 or GCS. Object storage is slow and bills per request, so Store puts a caching bucket in front of it.

Before this change, caching meant running a separate service such as memcached or Redis. That's another system to deploy and operate, and it has a blind spot: when several Store instances miss on the same key at the same time, each one goes to object storage for it.

• Implemented Thanos's cache interface on top of groupcache, and added a cachekey package for the keys the caching bucket uses.
• Wired it in as a backend you can choose for the Store caching bucket.
• Moved the implementation from groupcache to galaxycache after review, which also brought TTL support.
• Added Prometheus metrics, and wrote end-to-end tests that check cache hits and loads.

It was a large change to a production codebase I was new to. It went through 26 commits and several rounds of review, and the feedback covered a lot: switch the underlying library, restructure where configuration lived, remove duplication, get the metrics right, and make the end-to-end tests reliable.

What I took from it was how to get up to speed in a large codebase quickly, and how to work feedback into a change without losing its thread.

• MergedInto thanos-io/thanos on 6 January 2022.
• Released in v0.25Listed in the changelog as “Store: Add Groupcache as a cache backend.”
• One less serviceThanos users can cache across Store peers in-process, with no memcached to run.

Further reading: the pull request, and “Distributed Systems Magic: Groupcache in Thanos” on Giedrius Statkevičius's blog.