Intro
The title I had in mind for this blog post was “Why you shouldn’t use Spring Boot Actuator’s /health endpoint carelessly as Kubernetes readiness probe?” but it would have been too long.
Let’s take a look at what the Kubernetes documentation says about the different types of probes:
The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.
The kubelet uses readiness probes to know when a container is ready to start accepting traffic.
A pod is considered ready when all of its containers are ready. One use of this signal is to control which pods are used as backends for a service. When a pod is not ready, it is removed from the respective load balancers.
Spring Boot Actuator’s health endpoint shows health information based on the application’s dependencies. For example, when using a relational database, it checks the database health with DataSourceHealthIndicator, if you’re sending email within your application it does the check with MailHealthIndicator, if you’re using Redis it does the check with RedisHealthIndicator. By default, all these checks are enabled once you have these dependencies and do not disable the checks in your configuration.
You get all these health checks for free with Spring Boot Actuator without writing a single line of code and it’s appealing to use them for the liveness and readiness probes.
1. Why you should not use the Actuator health endpoint as liveness probe
As stated in the documentation, when a container’s liveness probe endpoint fails, the kubelet restarts it. However, if the database or mail server is down, restarting the application will not help them to recover so it’s pointless to use their health as a liveness check. In fact, I do not think there are many cases where using a liveness probe is beneficial.
2. Be careful when using the Actuator health endpoint as readiness probe
When a container’s readiness probe fails, the container’s pod will stop receiving traffic. Do you really want to shut your application down, whenever any of its dependencies is down?
Imagine you have an application that uses a relational database to store data, a mail server to send verification & forgot password mails to your users and a key-value store like Redis to cache content. In this case, it is likely that the relational database is your single point of failure and your application does not really work without it.
What about the mail server and Redis? The application is still usable without them so it does not make sense to stop receiving traffic because they are not available.
Spring Boot 2.2 introduced health indicator groups for this use case. They allow you to separate the endpoint used by a readiness probe from the default health endpoint. For example, the following configuration causes just the database health check to be used to assess readiness:
management.endpoint.health.
group.readiness.include=db
Readiness is then reported on the endpoint /health/readiness.
If you are using an older version of Spring Boot, you have to create a custom health indicator for this in code.
In summary, you have to be careful if you configure liveness and readiness probes. Otherwise, they can cause your application to be less available than it could be. Spring Boot Actuator can help you with this, but you should not simply use its default auto-configuration.
