In this post, we will talk about how to collect events from multiple Kubernetes clusters and ship it to sentry
At Clarisights, we run multiple Kubernetes clusters (GKE) for our production and staging workloads.
Events from Kubernetes objects get garbage collected to avoid putting extra load on etcd
events are stored in etcd with objects, and looks like it will stay that way for a while
When things blow up (like they do), we need those events to debug and fix the issues, so we wanted to put those events somewhere more permanent so we can come and look at it.
While searching for ways, we ran into sentry-kubernetes project, which collects and sends cluster events to sentry.
We already had self-hosted Sentry installation running, so this is just perfect, but upon looking closer we found that it sends events from a single cluster into a sentry project.
We wanted to send events from multiple clusters to a single project because we run multiple clusters and new clusters are spawned and moved around for various operational reasons.
Since code is open, we decided to add support for multiple clusters, we forked and sent a patch, yay open source 🙌
Now, let’s put it in production 🙌
Create a Sentry Project
- Create a new sentry project, select
- Copy DSN, we will need it later
To collect these events we need permissions to watch these events, we need to setup access so our collector and watch, collect and send these events to sentry
For access we will create a
ClusterRole with permissions to watch events,
and bind that role with
ServiceAccount using a
If you are new to Kubernetes RBAC, see my post on it to learn more, and see this gist for yaml file
We will use this service account in our deployment
- Build docker image
docker build -t sentry-kubernetes:latest https://github.com/getsentry/sentry-kubernetes.git
- Create and apply
sentry-kubernetesimage and service account we created, and if all goes well we should see events in sentry project.
Cluster name, namespace and other Kubernetes object attributes will show up as labels, and you should be able to filter events based on these.
See this gist for more scripts and yaml files.
I hope it was helpful :)
Stay in and Stay Safe 👋