Sending Kubernetes events to Sentry

on Kubernetes and Sentry

In this post, we will talk about how to collect events from multiple Kubernetes clusters and ship it to sentry

At Clarisights, we run multiple Kubernetes clusters (GKE) for our production and staging workloads.

Events from Kubernetes objects get garbage collected to avoid putting extra load on etcd

events are stored in etcd with objects, and looks like it will stay that way for a while

When things blow up (like they do), we need those events to debug and fix the issues, so we wanted to put those events somewhere more permanent so we can come and look at it.

While searching for ways, we ran into sentry-kubernetes project, which collects and sends cluster events to sentry.

We already had self-hosted Sentry installation running, so this is just perfect, but upon looking closer we found that it sends events from a single cluster into a sentry project.

We wanted to send events from multiple clusters to a single project because we run multiple clusters and new clusters are spawned and moved around for various operational reasons.

Since code is open, we decided to add support for multiple clusters, we forked and sent a patch, yay open source 🙌

Now, let’s put it in production 🙌

Create a Sentry Project

  • Create a new sentry project, select Other under language/framework
  • Copy DSN, we will need it later

Setup Access

To collect these events we need permissions to watch these events, we need to setup access so our collector and watch, collect and send these events to sentry

For access we will create a ServiceAccount, a ClusterRole with permissions to watch events, and bind that role with ServiceAccount using a ClusterRoleBinding.

If you are new to Kubernetes RBAC, see my post on it to learn more, and see this gist for yaml file

We will use this service account in our deployment

Deployment

  • Build docker image
    • docker build -t sentry-kubernetes:latest https://github.com/getsentry/sentry-kubernetes.git
  • Create and apply Deployment with sentry-kubernetes image and service account we created, and if all goes well we should see events in sentry project.

Closing Note

Cluster name, namespace and other Kubernetes object attributes will show up as labels, and you should be able to filter events based on these.

See this gist for more scripts and yaml files.

I hope it was helpful :)

Stay in and Stay Safe 👋


Continue Reading