Thanos · CNCF · LFX Mentorship, open source
Groupcache for Thanos Store
A pull request to a CNCF project: an embedded, distributed cache that fetches each object from storage once, however many queries ask for it at the same time. It was merged after three maintainers reviewed it and shipped in Thanos v0.25.
- Pull request
- thanos-io/thanos #4818
- Size
- 26 commits
- Merged
- January 2022 · released in v0.25
- Language
- Go
01Context
Thanos turns Prometheus into a highly available metrics system with long-term storage, and it's a CNCF project. Its Store Gateway serves historical data straight from object storage such as S3 or GCS. Object storage is slow and bills per request, so Store puts a caching bucket in front of it.
Before this change, caching meant running a separate service such as memcached or Redis. That's another system to deploy and operate, and it has a blind spot: when several Store instances miss on the same key at the same time, each one goes to object storage for it.
I picked this up through the Linux Foundation's LFX Mentorship program (September to November 2021): add groupcache as an embedded, distributed cache backend.
Try itInteractive demo
Three Store instances get the same query at once. Step through it with a shared external cache, then switch to groupcache and watch the bucket reads.
simplified · illustrative diagram
02What I built
- Implemented Thanos's cache interface on top of groupcache, and added a new
cachekeypackage for the keys the caching bucket uses. - Wired it in as a selectable backend for the Store caching bucket.
- Migrated the implementation from groupcache to galaxycache after maintainer review, which also brought TTL support.
- Instrumented it with Prometheus metrics, and wrote end-to-end tests that check cache hits and loads.
03Working in review
It was a large change to a production codebase I was new to. It went through 26 commits and several rounds of review with three maintainers, and their feedback covered a lot of ground: switch the underlying library, restructure where configuration lived, remove duplication, get the metrics semantics right, and make the end-to-end tests reliable.
What it taught me was ramping up quickly in a large codebase and working maintainers' feedback into a change without losing its thread.
04Outcome
Merged
Merged into thanos-io/thanos on 6 January 2022.
Released in v0.25
Listed in the release changelog: “Store: Add Groupcache as a cache backend.”
One less service
Gave Thanos users a way to cache across Store peers in-process, with no memcached to run.
Further reading: the pull request and “Distributed Systems Magic: Groupcache in Thanos” on Giedrius Statkevičius's blog.