add kw to query Kubernetes API health endpoints

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - Kubeconfig context support [#36](https://github.com/devopsspiral/KubeLibrary/pull/36) by [@m-wcislo](https://github.com/m-wcislo)
 - Keyword for getting secrets [#31](https://github.com/devopsspiral/KubeLibrary/pull/31 )by [@Nilsty](https://github.com/Nilsty)
 - Keyword for cluster healthcheck [#40](https://github.com/devopsspiral/KubeLibrary/pull/40) by [@satish-nubolab](https://github.com/satish-nubolab)
+- Extend cluster healthcheck [#47](https://github.com/devopsspiral/KubeLibrary/pull/47) by [@mika-b](https://github.com/mika-b)
 - Keyword for list ingress [#38](https://github.com/devopsspiral/KubeLibrary/pull/38) by [@satish-nubolab](https://github.com/satish-nubolab)
 - Keyword for list daemonset [#50](https://github.com/devopsspiral/KubeLibrary/pull/50) by [@satish-nubolab](https://github.com/satish-nubolab)
 

README.md

@@ -133,7 +133,7 @@ Keywords documentation can be found in docs/.
 
 [KubeLibrary: Testing Kubernetes with RobotFramework  | Humanitec](https://humanitec.com/blog/kubelibrary-testing-kubernetes-with-robotframework)
 
-[RobotFramewrok User Guide](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html)
+[RobotFramework User Guide](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html)
 
 ## Development
 
@@ -150,14 +150,16 @@ pip install -r requirements
 
 Create keyword and test file, import KubeLibrary using below to point to library under development.
 
-| ***** Settings ***** |
+```
+*** Settings ***
 
-| Library    ../src/KubeLibrary/KubeLibrary.py |
+Library    ../src/KubeLibrary/KubeLibrary.py
+```
 
 For development cluster you can use k3s/k3d as described in [DevOps spiral article on K3d and skaffold](https://devopsspiral.com/articles/k8s/k3d-skaffold/).
 
 ### Generate docs
 
 ```
 python -m robot.libdoc src/KubeLibrary/KubeLibrary.py docs/index.html
-```
+```

src/KubeLibrary/KubeLibrary.py

@@ -61,7 +61,7 @@ def reload_config(self, kube_config=None, context=None, incluster=False, cert_va
           Path pointing to kubeconfig of target Kubernetes cluster.
         - ``context``:
           Active context. If None current_context from kubeconfig is used.
-        - ``incuster``:
+        - ``incluster``:
           Default False. Indicates if used from within k8s cluster. Overrides kubeconfig.
         - ``cert_validation``:
           Default True. Can be set to False for self-signed certificates.
@@ -531,16 +531,28 @@ def delete_service_account_in_namespace(self, name, namespace):
         ret = self.v1.delete_namespaced_service_account(name=name, namespace=namespace)
         return ret
 
-    def get_healthcheck(self):
-        """Checks cluster level healthcheck
+    def get_healthcheck(self, endpoint='/readyz', verbose=False):
+        """Performs GET on /readyz or /livez for simple health check.
+
         Can be used to verify the readiness/current status of the API server
         Returns tuple of (response data, response status and response headers)
+
+        - ``endpoint``:
+            /readyz, /livez or induvidual endpoints like '/livez/etcd'. defaults to /readyz
+        - ``verbose``:
+            More detailed output.
+
+        https://kubernetes.io/docs/reference/using-api/health-checks
+
         """
         path_params = {}
         query_params = []
         header_params = {}
         auth_settings = ['BearerToken']
-        resp = self.v1.api_client.call_api('/readyz?verbose=', 'GET',
+        if not (endpoint.startswith('/readyz') or endpoint.startswith('/livez')):
+            raise RuntimeError(f'{endpoint} does not start with "/readyz" or "/livez"')
+        endpoint = endpoint if not verbose else endpoint + '?verbose'
+        resp = self.v1.api_client.call_api(endpoint, 'GET',
                                            path_params,
                                            query_params,
                                            header_params,

testcases/healthcheck/healthcheck.robot

@@ -1,7 +1,126 @@
 *** Settings ***
+Library           Collections
 Resource          ./healthcheck_kw.robot
+Default Tags      prerelease
 
 *** Test Cases ***
 Healthcheck
-    [Tags]    other    prerelease
+    [Tags]    other
     Healthcheck
+
+Health API Reports Ok
+    [Tags]    cluster    smoke
+    ${response}=  Query Health API  verbose=False
+    Should Be equal As Strings  ${response}[0]  ok
+
+Health API Reports Checks Passed With Verbose
+    [Tags]    cluster    smoke
+    ${response}=  Query Health API
+
+    # healthz < 1.20, livez >=1.20
+    Should Match Regexp  ${response}[0]  (livez|healthz) check passed
+
+Health API Reports Ok For informer-sync
+    [Tags]    cluster    smoke
+    ${response}=  Query Health API  /readyz/informer-sync
+    Should Be equal As Strings  ${response}[0]  ok
+
+Health API Reports Ok For shutdown
+    [Tags]    cluster    smoke
+    ${response}=  Query Health API  /readyz/shutdown
+    Should Be equal As Strings  ${response}[0]  ok
+
+Health API Reports Ok For ping
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    ping  ok
+
+Health API Reports Ok For log
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    log  ok
+
+Health API Reports Ok For etcd
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    etcd  ok
+
+Health API Reports Ok For poststarthook/start-kube-apiserver-admission-initializer
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/start-kube-apiserver-admission-initializer  ok
+
+Health API Reports Ok For poststarthook/generic-apiserver-start-informers
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/generic-apiserver-start-informers  ok
+
+Health API Reports Ok For poststarthook/max-in-flight-filter
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/max-in-flight-filter  ok
+
+Health API Reports Ok For poststarthook/start-apiextensions-informers
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/start-apiextensions-informers  ok
+
+Health API Reports Ok For poststarthook/start-apiextensions-controllers
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/start-apiextensions-controllers  ok
+
+Health API Reports Ok For poststarthook/crd-informer-synced
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/crd-informer-synced  ok
+
+Health API Reports Ok For poststarthook/bootstrap-controller
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/bootstrap-controller  ok
+
+Health API Reports Ok For poststarthook/scheduling/bootstrap-system-priority-classes
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/scheduling/bootstrap-system-priority-classes  ok
+
+Health API Reports Ok For poststarthook/start-cluster-authentication-info-controller
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/start-cluster-authentication-info-controller  ok
+
+Health API Reports Ok For poststarthook/aggregator-reload-proxy-client-cert
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/aggregator-reload-proxy-client-cert  ok
+
+Health API Reports Ok For poststarthook/start-kube-aggregator-informers
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/start-kube-aggregator-informers  ok
+
+Health API Reports Ok For poststarthook/apiservice-registration-controller
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/apiservice-registration-controller  ok
+
+Health API Reports Ok For poststarthook/apiservice-status-available-controller
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/apiservice-status-available-controller  ok
+
+Health API Reports Ok For poststarthook/kube-apiserver-autoregistration
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/kube-apiserver-autoregistration  ok
+
+Health API Reports Ok For autoregister-completion
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    autoregister-completion  ok
+
+Health API Reports Ok For poststarthook/apiservice-openapi-controller
+    [Tags]    cluster    smoke
+    [Template]  Health API Template
+    poststarthook/apiservice-openapi-controller  ok

testcases/healthcheck/healthcheck_kw.robot

@@ -17,3 +17,17 @@ Healthcheck
         Should Be True      "ok" or "healthz check passed" in """${ELEMENT}""" 
     END
     Should Be Equal As Strings  ${RESPONSE}[1]  200
+
+Query Health API
+    [Arguments]  ${endpoint}=/livez  ${verbose}=True
+    ${response}=    get_healthcheck  ${endpoint}  ${verbose}
+    Should Be Equal As integers  ${response}[1]  200
+    [Return]  ${response}
+
+Health API Template
+    [Documentation]  Checks both /readyz/${endpoint} and /livez/${endpoint}
+    [Arguments]  ${endpoint}  ${expected}
+    ${readyz}=  Query Health API  /readyz/${endpoint}
+    ${livez}=  Query Health API  /livez/${endpoint}
+    Should Match Regexp  ${readyz}[0]  ${expected}
+    Should Match Regexp  ${livez}[0]  ${expected}

testcases/system_smoke.robot

@@ -63,4 +63,4 @@ Grafana is responding
     [Tags]    grafana
     Given "grafana" service ip and port in "default" is known
     When session is created
-    Then service is responding on path "/"
+    Then service is responding on path "/"