We all know that microservice architecture is the perfect fit for cloud native applications and it increases the delivery velocities greatly. Envision you have many microservices that are delivered by multiple teams, how do you observe the the overall platform and each of the service to find out exactly what is going on with each of the services? When something goes wrong, how do you know which service or which communication among the few services are causing the problem?
Istio's tracing and metrics features are designed to provide broad and granular insight into the health of all services. Istio's role as a service mesh makes it the ideal data source for observability information, particularly in a microservices environment. As requests pass through multiple services, identifying performance bottlenecks becomes increasingly difficult using traditional debugging techniques. Distributed tracing provides a holistic view of requests transiting through multiple services, allowing for immediate identification of latency issues. With Istio, distributed tracing comes by default. This will expose latency, retry, and failure information for each hop in a request.
You can read more about how Istio mixer enables telemetry reporting.
-
Verify that the Grafana, Prometheus, Kiali and Jaeger add-ons were installed successfully. All add-ons are installed into the
istio-systemnamespace.kubectl get pods -n istio-system kubectl get services -n istio-system
-
Obtain the guestbook endpoint to access the guestbook.
You can access the guestbook via the external IP for your service as guestbook is deployed as a load balancer service. Get the EXTERNAL-IP of the guestbook service via output below:
kubectl get service guestbook -n default
Go to this EXTERNAL-IP address in the browser to try out your guestbook. This service will route you to either v1 or v2, at random. If you wish to see a different version, you'll need to do a hard refresh (
cmd + shift + ron a mac, orctrl + f5on a PC).
-
Generate a small load to the app, replacing guestbook_IP with the EXTERNAL-IP.
for i in {1..40}; do sleep 0.2; curl -I http://<guestbook_IP>/; done
-
Launch the Jaeger dashboard:
istioctl dashboard jaeger
-
From the Services menu, select either the guestbook or analyzer service.
-
Scroll to the bottom and click on Find Traces button to see traces.
-
Use Ctrl-C in the terminal to exit the port-foward when you are done.
Read more about Jaeger
-
Establish port forwarding from local port 3000 to the Grafana instance:
istioctl dashboard grafana
-
Navigate to the
Istio Service Dashboardby clicking on the Home menu on the top left, then Istio, then Istio Service Dashboard. -
Select guestbook in the Service drop down.
-
In a different tab, visit the guestbook application and refresh the page multiple times to generate some load, or run the load script you used previously. Switch back to the Grafana tab.
-
Use Ctrl-C in the terminal to exit the port-foward when you are done.
This Grafana dashboard provides metrics for each workload. Explore the other dashboard provided as well.
Read more about Grafana.
-
Establish port forwarding from local port 9090 to the Prometheus instance.
istioctl dashboard prometheus
-
In the “Expression” input box, enter:
istio_request_bytes_count. Click Execute and then select Graph. -
Then try another query:
istio_requests_total{destination_service="guestbook.default.svc.cluster.local", destination_version="2.0"} -
Use Ctrl-C in the terminal to exit the port-foward when you are done.
Kiali is an open-source project that installs on top of Istio to visualize your service mesh. It provides deeper insight into how your microservices interact with one another, and provides features such as circuit breakers and request rates for your services
- Create a secret which will be used to set the login credentials for Kiali
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: kiali
namespace: istio-system
labels:
app: kiali
type: Opaque
data:
username: "YWRtaW4="
passphrase: "YWRtaW4="
EOF
-
Establish port forwarding from local port 20001 to the Kiali instance.
istioctl dashboard kiali
- Login with
adminfor both username and password. - Select Graph and then choose
defaultnamespace. You should see a visual service graph of the various services in your Istio mesh. - Use the
Edge Labelsdropdown and selectTraffic rate per secondto see the request rates as well. - Kiali has a number of views to help you visualize your services. Click through the vairous tabs to explore the service graph, and the various views for workloads, applications, and services.
Although Istio proxies are able to automatically send spans, they need some hints to tie together the entire trace. Apps need to propagate the appropriate HTTP headers so that when the proxies send span information to Zipkin or Jaeger, the spans can be correlated correctly into a single trace.
In the example, when a user visits the Guestbook app, the HTTP request is sent from the guestbook service to Watson Tone Analyzer. In order for the individual spans of guestbook service and Watson Tone Analyzer to be tied together, we have modified the guestbook service to extract the required headers (x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, x-ot-span-context) and forward them onto the analyzer service when calling the analyzer service from the guestbook service. The change is in the v2/guestbook/main.go. By using the getForwardHeaders() method, we are able to extract the required headers, and then we use the required headers further when calling the analyzer service via the getPrimaryTone() method.
-
Does a user need to modify their app to get metrics for their apps? A: 1. Yes 2. No. (2 is correct)
-
Does a user need to modify their app to get distributed tracing for their app to work properly? A: 1. Yes 2. No. (1 is correct)
-
What distributed tracing system does Istio support by default? A: 1. Zipkin 2. Kibana 3. LogStash 4. Jaeger. (1 and 4 are correct)