Container Orchestration with Kubernetes

1. Intro to Kubernetes

Quote from Kelsey Hightower (found in "Cloud Native DevOps with Kubernetes" book):

Kubernetes do the things that the very best system administrator would do: automation, failover, centralized logging, monitoring. It takes what we've learned in the DevOps community and makes it the default, out of the box.

2. Main Kubernetes Components

Node and Pod

Service and Ingress

notes/learning/devops-bootcamp/img/basic-k8s-components.png

ConfigMap and Secrets

Volumes

Deployment and StatefulSet

3. Kubernetes Architecture

Master Nodes

Example Cluster Set-Up

A very small cluster you're probably have

To add new Master/Worker server

  1. get new bare server
  2. install all the master/worker node processes
  3. add it to the cluster

4. Minikube and kubectl - Local Kubernetes Cluster

Minikube

Having a real cluster setup to practice would require a lot of resources, not usually available in personal computers.

Minikube is a way to test local cluster setup. You have Master and Worker Nodes processes running on ONE machine.

kubectl

kubectl is a command line tool for k8s cluster.

One of the master processes mentioned earlier is the API Server. Clients communicate with the API Server through a web UI, API calls, or a CLI. And kubectl is that CLI (and the most powerful one).

Installing & Creating a minikube cluster

# define the virtual machine driver with `--driver`
# default is 'autodetect'
minikube start --driver=hyperkit

# check minikube status
minikube status

# check the kubectl version
kubectl version

5. Main kubectl commands

Get status of components

kubectl get nodes

kubectl get pod

kubectl get services

Creating a Pod (actually a deployment)

Pod is the smallest unit of a cluster, but we're not going to create pods directly. As mentioned earlier, "Deployment" is an abstraction layer over Pods. And with kubectl we're going to create "Deployments".

# get help about ~~pod~~ deployment creation
kubectl create -h

# usage:
# kubectl create deployment NAME --image=image [--dry-run] [options]
# example creating an nginx deployment (nginx image will
# be donwloaded from Docker Hub):
$ kubectl create deployment nginx-depl --image=nginx
deployment.apps/nginx-depl created

# get deployment status
$ kubectl get deployment
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
nginx-depl   0/1     1            0           9s

# get pod status
$ kubectl get pod
NAME                          READY   STATUS              RESTARTS   AGE
nginx-depl-5c8bf76b5b-nq8dj   0/1     ContainerCreating   0          17s
# STATUS above is still 'ContainerCreating'...
# after a couple of minutes, it's 'Running'
$ kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-depl-5c8bf76b5b-nq8dj   1/1     Running   0          2m12s

# get replicaset status
# note: replicaset is managing the replicas of a pod
# note 2: the pod name is
# ${DEPLOYMENT_NAME}-${REPLICASET_HASH}-${POD_HASH}
$ kubectl get replicaset
NAME                    DESIRED   CURRENT   READY   AGE
nginx-depl-5c8bf76b5b   1         1         1       2m3s

Layers of Abstraction

notes/learning/devops-bootcamp/img/k8s-abstraction-layers.png

Everything below "Deployment" is handled by Kubernetes

Editing a Pod / Deployment

When you edit a deployment, kubernetes automatically create a new pod, and once it's up and running it kills the old pod.

# edit the deployment
# change spec.template.spec.containers.image from 'nginx' to 'nginx:1.16'
$ kubectl edit deployment nginx-depl
deployment.apps/nginx-depl edited

# checking the deployment status after edition
$ kubectl get deployment
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
nginx-depl   1/1     1            1           20m

# checking pods status
# here, the old version is running and the new one is being created
$ kubectl get pod
NAME                          READY   STATUS              RESTARTS   AGE
nginx-depl-5c8bf76b5b-nq8dj   1/1     Running             0          20m
nginx-depl-7fc44fc5d4-fbtt5   0/1     ContainerCreating   0          9s

# new pod is running, old one is terminating
$ kubectl get pod
NAME                          READY   STATUS        RESTARTS   AGE
nginx-depl-5c8bf76b5b-nq8dj   0/1     Terminating   0          21m
nginx-depl-7fc44fc5d4-fbtt5   1/1     Running       0          30s

# only the new pod is running
$ kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-depl-7fc44fc5d4-fbtt5   1/1     Running   0          37s

# the old replicaset has no pods
$ kubectl get replicaset
NAME                    DESIRED   CURRENT   READY   AGE
nginx-depl-5c8bf76b5b   0         0         0       42m
nginx-depl-7fc44fc5d4   1         1         1       21m

Debugging pods

# let's create another deployment with a mongodb image
# (which creates a more verbose log)
kubectl create deployment mongo-depl --image=mongo

# the pod is not running yet
$ kubectl logs mongo-depl-5fd6b7d4b4-rpqhg
Error from server (BadRequest): container "mongo" in pod "mongo-depl-5fd6b7d4b4-rpqhg" is waiting to start: ContainerCreating     

# checking the detailed status
$ kubectl describe pod mongo-depl-5fd6b7d4b4-rpqhg
Name:           mongo-depl-5fd6b7d4b4-rpqhg
Namespace:      default
Priority:       0
Node:           minikube/192.168.99.100
Start Time:     Thu, 10 Jun 2021 10:24:08 -0300
Labels:         app=mongo-depl
                pod-template-hash=5fd6b7d4b4
# ...
# more info
# ...
Events:
Type    Reason     Age   From               Message
----    ------     ----  ----               -------
  Normal  Scheduled  4m6s  default-scheduler  Successfully assigned default/mongo-depl-5fd6b7d4b4-rpqhg to minikube
  Normal  Pulling    4m5s  kubelet            Pulling image "mongo"
  Normal  Pulled     106s  kubelet            Successfully pulled image "mongo" in 2m18.898375616s
  Normal  Created    105s  kubelet            Created container mongo
  Normal  Started    105s  kubelet            Started container mongo

# the pod is now up and running, let's check the logs
$ kubectl logs mongo-depl-5fd6b7d4b4-rpqhg
# ... a lot of mongodb logs...

kubectl exec -it ${POD_NAME} -- /bin/bash

kubectl get deployment nginx-deployment -o yaml > nginx-deployment-result.yaml

Delete Deployment

# deleting the mongodb deployment
$ kubectl delete deployment mongo-depl
deployment.apps "mongo-depl" deleted

# after deleting a deployment, its pods are going to terminate
$ kubectl get pods
NAME                          READY   STATUS        RESTARTS   AGE
mongo-depl-5fd6b7d4b4-rpqhg   0/1     Terminating   0          18m
nginx-depl-7fc44fc5d4-fbtt5   1/1     Running       0          71m

# the replicaset is already gone
$ kubectl get replicaset
NAME                    DESIRED   CURRENT   READY   AGE
nginx-depl-5c8bf76b5b   0         0         0       92m
nginx-depl-7fc44fc5d4   1         1         1       71m

# deleting the nginx deployment too
$ kubectl delete deployment nginx-depl
deployment.apps "nginx-depl" deleted

$ kubectl get pods
NAME                          READY   STATUS        RESTARTS   AGE
nginx-depl-7fc44fc5d4-fbtt5   0/1     Terminating   0          72m

$ kubectl get replicaset
No resources found in default namespace.

Apply Configuration File

kubectl apply -f config-file.yaml

An example for nginx-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:       # deployment specs
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:   # pod specs
      containers:
      - name: nginx
        image: nginx:1.16
        ports:
        - containerPort: 80

# type the file above
$ vim nginx-deployment.yaml

# applying that config file
# note that the output says "... created"
$ kubectl apply -f nginx-deployment.yaml 
deployment.apps/nginx-deployment created

$ kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-644599b9c9-qp8lb   1/1     Running   0          7s

$ kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1/1     1            1           17s

$ kubectl get replicaset
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-644599b9c9   1         1         1       28s

# edit the file and increase the `spec.replicas` from 1 to 2
$ vim nginx-deployment.yaml 

# note that the output says "... configured"
# kubernetes know when to create/update a deployment
$ kubectl apply -f nginx-deployment.yaml 
deployment.apps/nginx-deployment configured

$ kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           65s

$ kubectl get replicaset
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-644599b9c9   2         2         2       68s

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-644599b9c9-c9nc2   1/1     Running   0          16s
nginx-deployment-644599b9c9-qp8lb   1/1     Running   0          76s

Summarizing

# CRUD commands
########################################
kubectl create deployment ${NAME}
kubectl edit deployment ${NAME}
kubectl delete deployment ${NAME}
# using files
kubectl apply -f ${YAML_FILE}
kubectl delete -f ${YAML_FILE}

# Status of different k8s components
########################################
kubectl get nodes | services | deployment | replicaset | pod

# Debugging
########################################

# get logs
kubectl logs ${POD_NAME}

# detailed info about the pod
kubectl describe pod ${POD_NAME}

# interactive shell session inside a pod
kubectl exec -it ${POD_NAME} -- /bin/bash

6. YAML Configuration File

3 Parts of a k8s Configuration File

  1. metadata
  2. specifications
    • the first two lines (apiVersion and kind)
    • and the spec part
    • attributes of spec are specific to the kind.
  3. status
    • automatically generated/added by kubernetes
    • created by comparing the desired state (from the spec part of the yaml) and the actual state
    • if the states don't match, kubernetes knows that there's something to be fixed
    • status data comes from etcd Master process
      • etcd holds the current status of any k8s component

Format of k8s Configuration File

Blueprint for Pods (Templates)

The template:

Connecting components (Labels & Selectors & Ports)

Connection is stablished using labels and selectors.

spec:
  template:
    metadata:
    labels:
        app: nginx

spec:
  selector:
    matchLabels:
      app: nginx

Ports

notes/learning/devops-bootcamp/img/k8s-ports-in-service-and-pod.png

# creating deployment and service
kubectl apply -f nginx-deployment.yaml
kubectl apply -f nginx-service.yaml

# let's check them
kubectl get pod
kubectl get service

# get more details about the service
kubectl describe service nginx-service
# check the Selector, TargetPort and Endpoints

# you can get more pod information with -o wide
kubectl get pod -o wide

# the automatically generated status
kubectl get deployment nginx-deployment -o yaml

# save it in a file and compare with the original one
# the `status` part of the file can help with debugging
kubectl get deployment nginx-deployment -o yaml > nginx-deployment-result.yaml

7. Complete Demo Project - Deploying Application in Kubernetes Cluster

Overview

notes/learning/devops-bootcamp/img/k8s-demo-project-overview.png

Request Flow

browser -> Mongo Express External Service -> Mongo Express Pod -> MongoDB Internal Service -> MongoDB Pod

1st step - MongoDB Deployment

mongo.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        env:
        - name: MONGO_INITIDB_ROOT_USERNAME
          valueFrom: # remember to create the Secret before creating the Deployment
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-username
        - name: MONGO_INITIDB_ROOT_PASSWORD
          valueFrom: # remember to create the Secret before creating the Deployment
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-password

2nd step - Create the Secret

mongo-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  mongo-root-username: # paste here the output of `echo -n username | base64`
  mongo-root-password: # paste here the output of `echo -n password | base64`

Note: the Secret file must be created before the Deployment.

# creating the Secret
kubectl apply -f mongo-secret.yaml

# check if it was actually created:
kubectl get secret

# now let's create the Deployment
kubectl apply -f mongo.yaml

# get setup info
kubectl get all

3rd step - Create a Service

Let's create a service so other pods can access the MongoDB.

Deployment and Service usually belong together, so let's put their configs in the same file. In order to achieve that you just need to separate the configs with --- in a line.

mongo.yaml

# ...
# Deployment configs
# ...
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
spec:
  selector:
    app: mongodb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017

Creating the Service:

# this will keep mongodb-deployment unchanged and create mongodb-service
kubectl apply -f mongo.yaml

# check the service status
kubectl get service

# check if the service is connected to the right pod
kubectl describe service mongodb-service
# check the 'Endpoints' IP:port address

# compare with the pod's IP
kubectl get pod -o wide

# check them all
kubectl get all
kubectl get all | grep mongodb

4th step - Create Mongo Express Deployment and ConfigMap

mongo-express.yaml

apiVersion: app/v1
kind: Deployment
metadata:
  name: mongo-express
  labels:
    app: mongo-express
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongo-express
  template:
    metadata:
      labels:
        app: mongo-express
    spec:
      containers:
      - name: mongo-express
        image: mongo-express
        ports:
        - containerPort: 8081
        env: # 3 variables needed by mongo-express to connect to mongodb
        - name: ME_CONFIG_MONGODB_ADMINUSERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-username
        - name: ME_CONFIG_MONGODB_ADMINPASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-root-password
        - name: ME_CONFIG_MONGODB_SERVER
          valueFrom:
            configMapKeyRef:
              name: mongodb-configmap
              key: database_url

Database URL goes in the ConfigMap:

mongo-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-configmap
data:
  database_url: mongodb-service

ConfigMap must already be in the k8s cluster when referencing it.

# first the configmap...
kubectl apply -f mongo-configmap.yaml

# ... and then the deployment referencing the configmap
kubectl apply -f mongo-express.yaml

# check the pod
kubectl get pod

# check the logs
kubectl logs ${POD_NAME}

5th step - Create an External Service

We need an external service to allow browsers to access Mongo Express.

Again, let's keep the Service configs in the same file as the Deployment.

mongo-express.yaml:

# ... Mongo Express Deployment configs
---
apiVersion: v1
kind: Service
metadata:
  name: mongo-express-service
spec:
  selector:
    app: mongo-express

  # The 'type: LoadBalancer' is what make it an External Service.
  # It assigns an external IP address to this Service, making it
  # accept external requests.
  type: LoadBalancer


  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081
      # with 'nodeport:' you define the external listening port
      nodePort: 30000

Create the service:

kubectl apply -f mongo-express.yaml

# check the services
kubectl get service

# note in the output above that the TYPE means
# - LoadBalancer: external service
# - ClusterIP: internal service (default)

Specific to a minikube setup:

# assign a public IP address to a service
minikube service mongo-express-service

8. Namespaces - Organizing Components

What is a Namespace?

Create a Namespace

Command line:

kubectl create namespace my-namespace

# see it in your list of namespaces
kubectl get namespaces

You can also create a namespace using a configuration file.

Why use Namespaces?

  1. Resources grouped in Namespaces
  2. Conflicts: Many teams, same application
  3. Resource sharing:
    • staging and Development
    • Blue/Green Deployment
  4. Access and Resource Limits on Namespaces

Characteristics of Namespaces

Create Components in Namespaces

Assuming you already created a namespace with kubectl create namespace my-namespace.

If no Namespace is provided, the component is created in the default Namespace. You can check it with kubectl get ${COMPONENT_KIND} -o yaml and checking the metadata.namespace.

One way is to provide the namespace in the command line:

# assuming the file doesn't have a namespace
kubectl apply -f mysql-configmap.yaml --namespace=my-namespace

Another way is just to put it in the yaml file:

metadata:
  # ...
  namespace: my-namespace

Change Active Namespace

Change the active namespaces with kubens. You have to install the tool: https://github.com/ahmetb/kubectx.

9. Services - Connecting to Applications Inside Cluster

What is a Service and why do we need it?

"The Service component is an abstraction layer that basically represents an IP address"

notes/learning/devops-bootcamp/img/k8s-3-service-type-attributes.png

ClusterIP Services

notes/learning/devops-bootcamp/img/k8s-service-selector.png

Service Endpoints

Kubernetes creates Endpoint objects. They have the same name as the Service and keep track of which Pods are the members/endpoints of the Service.

kubectl get endpoints

Service Communication: port vs targetPort

Multi-Port Services

notes/learning/devops-bootcamp/img/k8s-multi-port-services.png

Headless Services

13min

It's declared like this:

# ...
kind: Service
# ...
spec:
  clusterIP: None
# ...

And you can see it in the output of kubectl get services, in the line where the column CLUSTER-IP says None.

NodePort Services

17:35

notes/learning/devops-bootcamp/img/k8s-nodeport-services.png

LoadBalancer Services

Wrap-Up

10. Ingress - Connecting to Applications Outside Cluster

Installing Ingress Controller in minikube:

# automatically starts k8s nginx implementation of Ingress Controller
minikube addons enable ingress

# check if it's running fine:
kubectl get pod -n kube-system

kubernetes-dashboard

UPDATE: apparently the example below doesn't work anymore. The kubernetes-dashboard can be accessed via minikube dashboard (which is not really a good Ingress config exercise).

# check if it's running
kubectl get all -n kubernetes-dashboard

create dashboard-ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
spec:
  rules:
  - host: dashboard.com
    http:
      paths:
      - backend:
        serviceName: kubernetes-dashboard
        servicePort: 80

back to terminal:

kubectl apply -f dashboard-ingress.yaml

# check if it's running
kubectl get ingress -n kubernetes-dashboard

# if the ADDRESS is not visible yet, use the --watch option
kubectl get ingress -n kubernetes-dashboard --watch

# get the ADDRESS and edit your /etc/hosts
echo "${IP_ADDRESS}  dashboard.com" | sudo tee /etc/hosts

# open "http://dashboard.com/" in your browser

Multiple paths for the same host

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-fanout-example
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: myapp.com
    http:
      paths:
      - path: /analytics
        backend:
          serviceName: analytics-service
          servicePort: 3000
      - path: /shopping
        backend:
          serviceName: shopping-service
          servicePort: 8080

Multiple sub-domains or domains

Multiple hosts with 1 path. Each host represents a subdomain.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: name-virtual-host-ingress
spec:
  rules:
  - host: analytics.myapp.com
    http:
      paths:
        backend:
          serviceName: analytics-service
          servicePort: 3000
  - host: shopping.myapp.com
    http:
      paths:
        backend:
          serviceName: shopping-service
          servicePort: 8080

Configuring TLS Certificate

You only need this in your ingress yaml file:

# ...
spec:
  tls:
  - hosts:
    - myapp.com
    secretName: myapp-secret-tls
    # and, of course, have a Secret with
    # that name in the same namespace

notes/learning/devops-bootcamp/img/k8s-tls-certificate.png

NOTES:

  1. data keys need to be tls.crt and tls.key
  2. values of those keys are actual file contents, not file paths
  3. Secret component must be in the same namespace as the Ingress component

11. Volumes - Persisting Application Data

3 k8s components for storage:

The Need of Volumes

Requirements:

  1. storage must not depend on the pod lifecycle (because pods are ephemeral)
  2. storage must be available on all nodes
  3. storage needs to survive even if cluster crashes

Persistent Volume

Think in a Persistent Volume as a cluster resource just like RAM and CPU.

Local vs. Remote Volume Types

Local volume types violate requirements 2 and 3 for data persistence:

Not used for DataBase persistence. Use remote storage instead.

Who Creates Persistent Volumes

There are two roles:

Storage resource is provisioned by the admin. And the user makes their application claim the Persistent Volume.

Persistent Volume Claim is also a component kind: PersistentVolumeClaim.

Summing up:

  1. admins configure storage
  2. admins create Persistent Volumes
  3. users claim Persistent Volumes through PersistentVolumeClaim
  4. users make their pod's use the PVC in the Pod's spec (note: the Pod and the PVC must be in the same namespace)

Level of Volume Abstractions

ConfigMap and Secret

ConfigMap and Secret are local volumes, managed by kubernetes. They can be mounted into your pod/container.

Storage Class

Storage Class is used when there are hundreds of pods requesting hundreds of Persistent Volumes.

StorageClass provisions Persistent Volumes dynamically, when PersistentVolumeClaim claims it.

Usage:

  1. pod claims a volume via PVC
  2. PVC requests storage from StorageClass
  3. StorageClass creates a PersistentVolume that meets the needs of the claim (using the defined provisioner)

My words: basically Storage Class is a way to automatically create Persistent Volumes when a Pod claims one.

12. ConfigMap & Secret Volume Types

Using ConfigMap and Secret as volumes makes it easier to access customization data.

TODO: put some examples here...

Note: ConfigMap and Secret are Local Volume Types.

13. StatefulSet - Deploying Stateful Applications

https://techworld-with-nana.teachable.com/courses/devops-bootcamp/lectures/28706438

14. Managed Kubernetes Services Explained

https://techworld-with-nana.teachable.com/courses/devops-bootcamp/lectures/28706442

15. Helm - Package Manager for Kubernetes

What is Helm?

Package Manager for Kubernetes

It's useful when complex application setup needs a lot of k8s components. Then rather than creating all of them manually, you can use a "Helm chart" most of the files ready, with just some space for customization.