|
Note
|
This repository contains the guide documentation source. To view the guide in published form, view it on the Open Liberty website. |
Learn how to check the health of microservices on Kubernetes by setting up startup, liveness, and readiness probes to inspect MicroProfile Health Check endpoints.
You will learn how to create health check endpoints for your microservices. Then, you will configure Kubernetes to use these endpoints to keep your microservices running smoothly.
MicroProfile Health allows services to report their health, and it publishes the overall health status to defined endpoints. If a service reports UP, then it’s available. If the service reports DOWN, then it’s unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.
Kubernetes provides startup, liveness, and readiness probes that are used to check the health of your containers. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes startup, liveness, and readiness endpoints on your microservices. Kubernetes polls these endpoints as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.
The two microservices you will work with are called system and inventory. The system microservice returns the JVM system properties of the running container and it returns the pod’s name in the HTTP header making replicas easy to distinguish from each other. The inventory microservice adds the properties from the system microservice to the inventory. This demonstrates how communication can be established between pods inside a cluster.
Navigate to start directory to begin.
Create theInventoryStartupCheckclass.inventory/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java
InventoryStartupCheck.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java[role=include]A health check for startup allows applications to define startup probes that verify whether deployed application is fully initialized before the liveness probe takes over. This check is useful for applications that require additional startup time on their first initialization. The @Startup annotation must be applied on a HealthCheck implementation to define a startup check procedure. Otherwise, this annotation is ignored. This startup check verifies that the cpu usage is below 95%. If more than 95% of the cpu is used, a status of DOWN is returned.
Create theInventoryLivenessCheckclass.inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java
InventoryLivenessCheck.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java[role=include]A health check for liveness allows third party services to determine whether the application is running. If this procedure fails, the application can be stopped. The @Liveness annotation must be applied on a HealthCheck implementation to define a Liveness check procedure. Otherwise, this annotation is ignored. This liveness check verifies that the heap memory usage is below 90% of the maximum memory. If more than 90% of the maximum memory is used, a status of DOWN is returned.
The inventory microservice is healthy only when the system microservice is available. To add this check to the /health/ready endpoint, create a class that is annotated with the @Readiness annotation and implements the HealthCheck interface. A Health Check for readiness allows third party services to know whether the application is ready to process requests. The @Readiness annotation must be applied on a HealthCheck implementation to define a readiness check procedure. Otherwise, this annotation is ignored.
Create theInventoryReadinessCheckclass.inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java
InventoryReadinessCheck.java
link:finish/inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java[role=include]This health check verifies that the system microservice is available at http://system-service:9090/. The system-service host name is accessible only from inside the cluster; you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN, the microservice is considered to be unhealthy.
The health checks for the system microservice were already been implemented. The system microservice was set up to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.
You will configure Kubernetes startup, liveness, and readiness probes. Startup probes determine whether your application is fully initialized. Liveness probes determine whether a container needs to be restarted. Readiness probes determine whether your application is ready to accept requests. If it’s not ready, no traffic is routed to the container.
Create the kubernetes configuration file.
kubernetes.yaml
kubernetes.yaml
link:finish/kubernetes.yaml[role=include]The startup, liveness, and readiness probes are configured for the containers that are running the system and inventory microservices.
The startup probes are configured to poll the /health/started endpoint. The startup probe determines whether a container is started.
The liveness probes are configured to poll the /health/live endpoint. The liveness probes determine whether a container needs to be restarted. The initialDelaySeconds field defines the duration that the probe waits before it starts to poll so that it does not make requests before the server is started. The periodSeconds option defines how often the probe polls the given endpoint. The timeoutSeconds option defines how many seconds before the probe times out. The failureThreshold option defines how many times the probe fails before the state changes from ready to not ready.
The readiness probes are configured to poll the /health/ready endpoint. The readiness probe determines the READY status of the container, as seen in the kubectl get pods output. Similar to the liveness probes, the readiness probes also define initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold.
To build these microservices, navigate to the start directory and run the following command.
mvn packageNext, run the docker build commands to build container images for your application:
docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.The -t flag in the docker build command allows the Docker image to be labeled (tagged) in the name[:tag] format. The tag for an image describes the specific image version. If the optional [:tag] tag is not specified, the latest tag is created by default.
When the builds succeed, run the following command to deploy the necessary Kubernetes resources to serve the applications.
kubectl apply -f kubernetes.yamlUse the following command to view the status of the pods. There will be two system pods and one inventory pod, later you’ll observe their behavior as the system pods become unhealthy.
kubectl get podsNAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 59s
system-deployment-694c7b74f7-lrlf7 1/1 Running 0 59s
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 59sWait until the pods are ready. After the pods are ready, you will make requests to your services.
The default host name for Docker Desktop is localhost.
The default host name for minikube is 192.168.99.100. Otherwise it can be found using the minikube ip command.
Navigate to http://[hostname]:31000/system/properties and observe a response containing JVM system properties. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you might notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.
Similarly, navigate to http://[hostname]:32000/inventory/systems/system-service and observe that the request is successful.
An unhealthy endpoint has been provided under the system microservice to set it to an unhealthy state. The unhealthy state causes the readiness probe to fail. A request to the unhealthy endpoint puts the service in an unhealthy state as a simulation.
Navigate to http://[hostname]:31000/system/unhealthy to invoke the unhealthy endpoint by running the following curl command:
curl http://[hostname]:31000/system/unhealthyRun the following command to view the state of the pods:
kubectl get podsNAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 1m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 1m
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 1mYou will notice that one of the two system pods is no longer in the ready state. Navigate to http://[hostname]:31000/system/properties. Your request is successful because you have two replicas and one is still healthy.
Wait until the system pod is ready again. Make two requests to http://[hostname]:31000/system/unhealthy. If you see the same pod name twice, repeat the request until you see that the second pod is unhealthy. You might see the same pod twice due to a delay between when a pod becomes unhealthy and when the readiness probe notices it. Therefore, traffic might still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the inventory microservice will also fail. Observe that it’s no longer in the ready state either.
First, both system pods will no longer be ready because the readiness probe failed.
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 0/1 Running 0 5m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 5m
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 5mNext, the inventory pod is no longer ready because the readiness probe failed. The probe failed because system-service is now unavailable.
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 0/1 Running 0 6m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 6m
inventory-deployment-cf8f564c6-nctcr 0/1 Running 0 6mThen, the system pods will start to become healthy again after 60 seconds.
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 7m
system-deployment-694c7b74f7-lrlf7 0/1 Running 0 7m
inventory-deployment-cf8f564c6-nctcr 0/1 Running 0 7mNAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 7m
system-deployment-694c7b74f7-lrlf7 1/1 Running 0 7m
inventory-deployment-cf8f564c6-nctcr 0/1 Running 0 7mFinally, you will see all of the pods have recovered.
NAME READY STATUS RESTARTS AGE
system-deployment-694c7b74f7-hcf4q 1/1 Running 0 8m
system-deployment-694c7b74f7-lrlf7 1/1 Running 0 8m
inventory-deployment-cf8f564c6-nctcr 1/1 Running 0 8mRun the tests by running the following command and appropriately substituting [hostname] for the correct value.
mvn failsafe:integration-test -Dsystem.service.root=[hostname]:31000 -Dinventory.service.root=[hostname]:32000A few tests are included for you to test the basic functions of the microservices. If a test fails, then you might have introduced a bug into the code. Wait for all pods to be in the ready state before you run the tests.
When the tests succeed, you should see output similar to the following in your console.
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointIT
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.65 s - in it.io.openliberty.guides.system.SystemEndpointIT
Results:
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.inventory.InventoryEndpointIT
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 s - in it.io.openliberty.guides.inventory.InventoryEndpointIT
Results:
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.
kubectl delete -f kubernetes.yamlYou have used MicroProfile Health and Open Liberty to create endpoints that report on your microservice’s status. Then, you observed how Kubernetes uses the /health/started, /health/live, and /health/ready endpoints to keep your microservices running smoothly.