It used the helm to install the phpipam(helm version) on K8s.

Install phpipam command

kubectl create namespace phpipam
kubectl create secret generic mysql-password --from-literal='password=<mysql password>' -n phpipam

helm repo add phpipam https://nullconfig.github.io/phpipam/stable
helm repo update
helm install --namespace phpipam --create-namespace phpipam phpipam/phpipam

Setup Storage

if you have not setup the phpipam storge or default storage, it will show warning message on describe pods like this

$ kubectl describe pods phpipam-5f5f456d75-69hk7 -n phpipam

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  17s (x4 over 3m8s)  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolume
Claims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..

That’s means there is no PersistentVolume (pv) and PersistentVolume Claims (pvc) on your system, it needs to create one.

Here is pv.yaml contain.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: phpipam
  namespace: phpipam
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node1

k8s-node1 is your storage node, function local: only can save storage on one node, if you setup it on master node, it will see this error message

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m11s  default-scheduler  0/3 nodes are available: persistentvolumeclaim "phpipam" not found. preemption:
0/3 nodes are available: 3 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  34s    default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preem
ption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  31s    default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.i
o/control-plane: }. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for
incoming pod..

It needs to prepare pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: phpipam
  namespace: phpipam
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 100Gi

Then, it can run apply them

kubectl apply -f pv.yaml 
kubectl apply -f pvc.yaml 
kubectl get pv -A
kubectl get pvc -A

Change the phpIPAM type from ClusterIP to LoadBalancer

After finish setup the system, it needs to change the phpipam-web’s CludsterIIP to LoadBalancer to get external IP

$ kubectl get svc -A
phpipam          phpipam-db                  ClusterIP      10.101.2.59      <none>            3306/TCP                     3h22m
phpipam          phpipam-web                 ClusterIP      10.108.147.73    <none>            80/TCP                       3h22m
$ kubectl patch svc phpipam-web -n phpipam -p '{"spec": {"ports": [{"port": 443,"targetPort": 443,"name": "https"},
{"port": 80,"targetPort": 80,"name": "http"}],"type": "LoadBalancer"}}'
$ kubectl get svc -A
phpipam          phpipam-db                  ClusterIP      10.101.2.59      <none>            3306/TCP                     4h18m
phpipam          phpipam-web                 LoadBalancer   10.108.147.73    192.168.110.202   443:32226/TCP,80:30911/TCP   4h18m

Now, it can use browser to access the phpIPAM web via IP 192.168.110.202

ref.
kubectl – How to edit service spec type to LoadBalancer via command line?

The post Install phpIPAM on Kubernetes appeared first on richliu's blog.

My hardware also different, if you don’t have such powerful machine, it can skip numa part, that’s only for high end ARM64 machine.
The hardware is Ampere Mt. Collins which has 2*Ampere Altra 80 Cores CPU and 8*32G Memory, my experiment is run different VM on different CPU, to seperate them and lock it on different numa node.

There are two CPUs, so, it can create 3 nodes, so, I have plan to allocate resource like that.
1. k8s master on socket 0: 10 CPUs, 16G memory
2. k8s node1 on socket 0 : 70 CPUs, 100G memory
3. k8s node2 on socket 1: 80 CPUs, 120G memory

Kubernetes version v1.28.2
Ubuntu 22.04

Notice: Before reading this document, this document is personal experiment, not Ampere official document, take your own risk.

Prepare VM from libvirt

The server has remote control system which problem sufficent resolution to use GUI on server, so, I install ubuntu 22.04 on host system, than use virt-manager to create k8s master node, for qemu image, I create the disk image alone, seems create image when run install will allocate all disk space, seems that’s a virt-manager bug.
And the host network is configured to be a bridge interface, in my case, that’s br0.

So, when uses virt-manager to create VM, it can bridge network interface to br0.

Configure the VM with Numa

It can use numastat to check currnet numa status, this system has two numa nodes. it can use numastat and numactl to get more numa information.

# numastat -n

Per-node numastat info (in MBs):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
Numa_Hit               591239.17       791075.41      1382314.58
Numa_Miss                   0.00            0.00            0.00
Numa_Foreign                0.00            0.00            0.00
Interleave_Hit              8.63            7.13           15.77
Local_Node             519434.17       777974.99      1297409.16
Other_Node              71805.00        13100.42        84905.42

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 128377 MB
node 0 free: 80224 MB
node 1 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 1 size: 126426 MB
node 1 free: 93888 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

virt-manager doesn’t have advance command to assign CPU, probably modify xml can get that purpose, but it more converince to use the “virsh” cli command to adjust VM parameters. “virsh list” show current VMs, “virsh vcpupin list current VM CPU scheduling”, “virsh vcpupin domain-id vcpu cpulist” to modify the VCPU run on which one CPU.

For memory, it needs to use “virsh numatune” to lock the memory on different node.
“virsh numatune domain [–mode mode] [–nodeset nodeset] [[–config] [–live] | [–current]]”, –current can update current running VM machine.

$ virsh list
 Id   Name          State
-----------------------------
 9    k8s-master   running
 10   k8s-node1     running

$ virsh vcpupin k8s-master
 VCPU   CPU Affinity
----------------------
 0      0-159
 1      0-159
 2      0-159
 3      0-159
 4      0-159
 5      0-159
 6      0-159
 7      0-159
 8      0-159
 9      0-159

# this command can lock 9 k8s-master's VCPU 0 on physical CPU 0. 
$ virsh vcpupin 9 0 0 

# Show current VM's numatune, for example, following is unset status. 
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   :

$ virsh numatune 9 --nodeset 0

# the numa_nodeset was set to 0
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   : 0

The problem is, seems this script need to run everytime, alternativate solution is to modify XML, but Mt. collis has 160 cores, I won’t to modify the XML file. script is more converience on this situation, following is my sample script to adjust k8s-master, k8s-node1, and k8s-node2’s parameter, it can easy to modify it if you install other system or AltraMax CPU.

#!/bin/bash

configure_vm(){
        local VM_NAME="$1"
        local VM_VCPU="$2"
        local VM_CPU="$3"
        local VM_NUMA="$4"
        local VM_TYPE="$5" # VM_TYPE for MASTER or NODE0, NODE1 .. self define name

        id=$(virsh list | grep $VM_NAME | awk '{print $1}')
        state=$(virsh list | grep $VM_NAME | awk '{print $3}')

        if [ "$state" != "running" ]; then
          echo "$VM_TYPE is not running"
          exit 1
        else
          echo "configure $VM_TYPE"
        fi
        numa_nodeset=$(virsh numatune $id | grep numa_nodeset | awk '{print $3}')

        start_cpu=$(echo $VM_VCPU | cut -d'-' -f1)
        end_cpu=$(echo $VM_VCPU | cut -d'-' -f2)
        for ((vcpu = start_cpu; vcpu <= end_cpu; vcpu++)); do
          # echo "$id $vcpu $VM_CPU"
          virsh vcpupin $id $vcpu $VM_CPU > /dev/null
        done
        if [ -n "$numa_nodeset" ]; then
          echo "$VM_TYPE numa_nodeset is $numa_nodeset"
        else
          # echo "$id $VM_NUMA"
          virsh numatune $id --nodeset $VM_NUMA
        fi

}

configure_vm "k8s-master" "0-9" "0-9" "0" "Master"
configure_vm "k8s-node1" "0-69" "10-79" "0" "Node0"
configure_vm "k8s-node2" "0-79" "80-159" "1" "Node1"


====

# After run the script, it can run numastat <pid> to see the qemu runs numa status. ex:
$ sudo numastat 206601

Per-node process memory usage (in MBs) for PID 206601 (qemu-system-aar)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                        11.51            0.00           11.51
Stack                        0.04            0.00            0.04
Private                   5984.93            0.00         5984.94
----------------  --------------- --------------- ---------------
Total                     5996.48            0.00         5996.48

In here, on socket 0 , 0-9 and 0-69 is vcpu, so, all start from 0. After run this script, it can use “virsh vcpupin” and “virsh numatune” to get current setting. Maybe cpuset parameter in XML can more easy to get this function, but I have not try it.

Install K8s

Here is my script, just reference some later blog and my previous configurtion script, so, it can copy and paste to system and run it directly.

Install basic system and k8s

# setup timezone
sudo apt install ntpdate -y
sudo ntpdate watch.stdtime.gov.tw
sudo timedatectl set-timezone Asia/Taipei

# install and enable docker

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update
sudo apt install -y containerd.io


containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

echo "----------------"
echo "   swapoff"
echo "----------------"
sudo swapoff -a
sudo sed -i '/swap/ s/^\(.*\)$/#\1/g' /etc/fstab

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update

sudo apt-get -y install kubelet
sudo apt-get -y install kubeadm
sudo apt-get -y install kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Update kernel parametes
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# update sysctl
sudo sysctl --system

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

echo "if this is work node, it can run kubeadm join now"

echo "alias k=kubectl" >> .bashrc

Setup Master Node

Run this script on master node.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# echo "enable master to run pods"
# kubectl taint nodes --all node-role.kubernetes.io/master-

It will shows some message below, then run it on k8s-node1 and k8s-node2

kubeadm join 192.168.110.230:6443 –token e14f7k.35c3jpny81ry70re \

        –discovery-token-ca-cert-hash sha256:6636a7e4ec3542f19152f308f0789c52287b743b60efa3c13e7c23ce1147c581

If forgot command above, it can create another token to let other k8s node join, ex:

kubeadm token create --print-join-command

Flannel : Network fabric for container

K8s need a network fabric to provide internal network topology, there are many choices, but we will install matellb later, some other network like calico might have potential issue, so, we choice flannel as network fabric.

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

After run this command, the k8s node status will change to Ready, if not install network fabric, the node status will be NotReady.

$ kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   26h   v1.28.2
k8s-node1    Ready    <none>          26h   v1.28.2
k8s-node2    Ready    <none>          26h   v1.28.2

Helm: Kubbernetes Applications Manager

Install helm to further install metallb purpose, on previous article, helm2 needs another application to connect to k8s, but helm3 doesn’t need it, more easy to use on k8s.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o /tmp/get_helm.sh
chmod 700 /tmp/get_helm.sh
sudo /tmp/get_helm.sh

Metallb: Bere metal load-balancer for K8s

Install k8s on private side, it needs load-balancer to provide external IP to communication with internet or intranet. If you install Metallb, the “kubectl get svc”, the EXTERNAL-IP shows “Pending”, and k8s doesn’t response the arp on host, probably it doesn’t configure kube-proxy, it you have experience on previous metallb installtion, it might miss this part.

PATH=$PATH:/usr/local/bin/

helm repo add metallb https://metallb.github.io/metallb
helm --namespace metallb-system install --create-namespace metallb metallb/metallb -f metallb-values.yaml

# https://metallb.universe.tf/installation/
# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl apply -f - -n kube-system

metallb-values.yaml is also different like previous setting, it needs to configure L2Advertisement on metallb, without it, your metallb won’t work. Follwing is metallb-values.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: primary-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.110.200-192.168.110.219

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - primary-pool

Traefik: TBD

Linkerd

Just show the install script here

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

echo "export PATH=$PATH:/home/$USER/.linkerd2/bin" >> ~/.bashrc

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

Other

it can insert isolinux to host linux kernel parameter to isolate the Linux scheduling to k8s nodes. (TBD)

isolcpus=10-159 nohz=on nohz_full=10-159

Test with Nginx

It can deploy nginx to test this system

kubectl create deploy nginx --image=nginx
kubectl expose deploy nginx --port=80 --target-port=80 --type=LoadBalancer
kubectl get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
nginx   LoadBalancer   10.99.211.255   192.168.110.200   80:31011/TCP   25h

Now, it can try to access http://192.168.110.200.
if want to Delete nginx deploy, run following command.

kubectl delete deploy nginx
kubectl delete svc nginx

Test Numa

It can run emojivoto to test the system only runs on socket1, use it because it can general a lot of traffic between pods.

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml \
  | kubectl apply -f -

Now it can use htop to show all CPUs, and we can see most traffic on CPU 1~10 and 81~160. There are a little CPU usage on CPU 78 and 80, but it should be Linux regular scheduling(I have not add isocpu to host CPU this time)

The post ARM64 Kubernetes Qemu Metallb Linkerd Numa appeared first on richliu's blog.

Remove Packages

$ sudo yum remove docker-ce docker-ce-cli containerd.io

Install Containterd First

This is most important step, without this, docker’s install script will fail.

$ sudo dnf install https://download.docker.com/linux/centos/7/aarch64/stable/Packages/containerd.io-1.2.6-3.3.el7.aarch64.rpm

Docker Script

Get docker script and run it
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh
Add your current username to docker group if not run on root account.
$ sudo usermod -aG docker your-user

Restart Docker And Enable Firewall

$ sudo systemctl start docker
$ firewall-cmd –zone=public –add-masquerade –permanent
$ firewall-cmd –reload

ref.
[1] https://linuxconfig.org/how-to-install-docker-in-rhel-8
[2] https://docs.docker.com/engine/install/centos/
[3] https://serverfault.com/questions/987686/no-network-connectivity-to-from-docker-ce-container-on-centos-8

The post CentOS8 ARM64/AARCH64 Install Docker appeared first on richliu's blog.

helm install \
    --name nginx-ingress stable/nginx-ingress \
    --namespace ingress-nginx \
    --set rbac.create=true \
    --set controller.image.repository="quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm64" \
    --set defaultBackend.image.repository="k8s.gcr.io/defaultbackend-arm64"

ref.
https://github.com/kubernetes/ingress-nginx/issues/4876
https://github.com/kubernetes/ingress-nginx/pull/3852

The post Kubernetes install nginx-ingress on ARM64 appeared first on richliu's blog.

First, download Linkerd2 binary for ARM64 from my github website.
There are some binary files in release directory, please ignore them, just for compile source code only.

git clone https://github.com/richliu/linkerd2-arm64.git
cd linkerd2-arm64/

Install Linkerd2 for ARM64

kubectl apply -f arm64.yml

It can also install Linkerd2 demo app emojivoto, yes, it doesn’t support ARM64, so, I ported it for ARM64 platform.

kubectl apply -f emojivoto.yml

Check whether it running on your system .

Now, it can follow linkerd Getting Started to test linkerd2 on ARM64 platform.

ref.
Support for ARM based architectures?

The post Linkerd2 for ARM64 appeared first on richliu's blog.

Above is a figure to describe relationship about metallb/traefik and kubernetes.

Please reference previous article (Install Kubernetes on ARM64 Ubuntu 18.04)about how to setup environment.

Install Helm

First, install Helm, helm is a kubernetes packages manager. it can help to install traefik, for some package, like traefik, it’s not easy to install manually, I tried, but fail. Use helm to install is a good idea.

Following instruction can download and install helm, or you can visit helm webpage for latest version

Following is command to download and install helm.

wget https://get.helm.sh/helm-v2.15.1-linux-arm64.tar.gz
tar xvf helm-v2.15.1-linux-arm64.tar.gz
sudo mv linux-arm64/helm /usr/bin

Default helm(tiller seems doesn’t support ARM64 archeticture, so, it needs to use third party binary code for tiller. In here we used tiller-multiarch.

helm init --tiller-image=jessestuart/tiller 

Helm install error message

When you run helm install command, if see error message like below, please try to follow this step to solve it. (ref: TILLER AND ROLE-BASED ACCESS CONTROL and forbidden: User “system:serviceaccount:kube-system:default” cannot get namespaces in the namespace “default)

Create a rbac-config.yaml like following text
(ref. User “system:serviceaccount:kube-system:default” cannot get namespaces in the namespace “default” )

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: default

Run following command to create and upgrade helm initial

kubectl create -f rbac-config.yaml
helm init --service-account tiller --history-max 200 --tiller-image=jessestuart/tiller --upgrade

or run those commands directly.

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init --upgrade --service-account tiller 

Install metallb

Automatically Install

It can use helm to install automatically, first create a metallb-config.yaml, contain like below.

configInline:
   address-pools:
   - name: default 
     protocol: layer2
     addresses:
         - 192.168.110.80-192.168.110.100 

Save it, and run follow command to apply it.

helm install --name=metallb --namespace=metallb-system -f  metallb-config.yaml stable/metallb 

Manually Install

It also can install metallb manually, run following command to install it.
It will install v0.8.1, it can visit metallb website to check latest version.

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml 

Add a new file name metallb.yaml and put follow text

apiVersion: v1
kind: ConfigMap
metadata:
   namespace: metallb-system
   name: config
data:
   config: |
     address-pools:
     - name: my-ip-space
       protocol: layer2
       addresses:
       - 192.168.110.70/28

Apply it

kubectl apply -f  metallb.yaml 

Install Traefik

Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience.

Following is command to initial helm and used helm to initial traefik.

helm install stable/traefik --name traefik --set dashboard.enabled=true,serviceType=NodePort,dashboard.domain=dashboard-traefik.techarea.org,rbac.enabled=true  --namespace kube-system

Traefik Dashboard

When used helm to install traefik, it also install traefik dashboard on dashboard-traefik.techarea.org.

Used following command to get traefik web service ip

kubectl describe svc traefik --namespace kube-system

It can see a http hosted on 10.44.0.2 , add 10.44.0.2 daskboard-traefik.techarea.org to browser PC’s hosts file, it can use browser to access traefik dashboard .

Create a nginx web site

In here, we create three nginx website, and assigned a external ip address, it will also assign a domain name, traefik will have load balance traffic to all three pods.

First create a new file called nginx.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 4
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
  name: nginx
spec:
  rules:
  - host: nginx.techarea.org
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80

replicas means how many pods you went to create, in here, we used 3, it will create 3 pods on clients.
host means url, in here is nginx.techarea.org.

used kubectl to apply it.

kubectl apply -f nginx.yaml 

Get current service status

We can see external ip is 192.168.110.80, add 192.168.110.80 nginx.techarea.org to your browser PC’s host file, and used browser to access nginx.techarea.org, it can see welcome nginx.

Test Load Balance

Now we finished install traefik and metallb successfully, next step is to verified does that really work ?

it can use another host to run ab (apache benchmark to do this test) and used htop observe CPU loading.

Following is example to test nginx, if don’t see all CPU has loading, try to increase pods instance, like 4 .

ab -n 1000000 -c 400 -k http://nginx.techarea.org/

Below is example result for nginx load balance. 

See, that’s easy job, right ?

The post ARM64 Kubernetes Nginx Traefik Ingress Load Balance appeared first on richliu's blog.

Environment

Hardware : Ampere eMag Server with 128G RAM and 1T storage
Kubernetes master and nodes are running in QEMU image, so, we prepared three QEMU ARM64 ubuntu 18.04 images for kubernetes running. It also can use two VM to run this demo.

Prepare QEMU Image

It can download Ubuntu bionic cloud image from Taiwan mirror site

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img 

or international mirror site

https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.img

Install Necessary Package

Run following command to install necessary packages .

apt install -y qemu-system-arm bridge-utils cloud-image-utils git qemu-efi

Increase QEMU image size

qemu-img resize bionic-server-cloudimg-arm64.img +200G

Create UEFI bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Copy Image to three spare images

cp flash0.img flash0-1.img
cp flash0.img flash0-2.img 
cp flash0.img flash0-3.img 
cp flash1.img flash1-1.img
cp flash1.img flash1-2.img
cp flash1.img flash1-3.img
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-1.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-2.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-3.img 

Set up bridge and network devices

We already install bridge-utils before, now, it should configure bridge devices.

 mkdir -p /etc/qemu ; echo "allow br0" > /etc/qemu/bridge.conf 

Because there is only one network interface on machine, we should use this interface as bridge device, save text below (name nic-br.sh or what you like) and modify it to fit your environment.

ORIGNIC=enP6p1s0
ip addr flush $ORIGINIC
brctl addbr br0
brctl addif br0 $ORIGINIC
ifconfig br0 up
ifconfig br0 192.168.110.1 netmask 255.255.0.0
route add default gw 192.168.1.1
echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Configure Hugepage On Host Site (Option)

Due to previous research on dpdk runs on ARM64 server, so, configure hugepages here. But not necessary for run k8s environment.
You can ignore this part

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX="console=tty0 hugepagesz=1GB hugepages=32"

then run foloowing command to update grub.cfg

 update-grub 

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages 
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab 

Now, please reboot to make huagpage works.

Running QEMU

Following is QEMU script, save it (ex: named run-qemu.sh). It can be more simplification, but this script would be more understand what different.

IMAGE1=bionic-server-cloudimg-arm64-1.img
sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-1111-1111-1111-11111aaaaaaa \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-1.img -pflash flash1-1.img \
         -drive file=$IMAGE1,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0,mac=52:54:00:11:11:11 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest_log-1.txt 2>> guest_log-1.txt &
IMAGE2=bionic-server-cloudimg-arm64-2.img
sudo qemu-system-aarch64 -name vm2 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-2222-2222-2222-22222bbbbbbb \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-2.img -pflash flash1-2.img \
         -drive file=$IMAGE2,if=none,id=disk2 \
         -device virtio-blk-device,scsi=off,drive=disk2,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net2,ifname=tap1 \
         -device virtio-net-device,netdev=net2,mac=52:54:00:22:22:22 \
         -mem-path /mnt/hugepages \
         -serial telnet::9002,server,nowait > guest_log-2.txt 2>> guest_log-2.txt &
IMAGE3=bionic-server-cloudimg-arm64-3.img
sudo qemu-system-aarch64 -name vm3 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-3333-3333-3333-33333ccccccc \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-3.img -pflash flash1-3.img \
         -drive file=$IMAGE3,if=none,id=disk3 \
         -device virtio-blk-device,scsi=off,drive=disk3,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net3,ifname=tap2 \
         -device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \
         -mem-path /mnt/hugepages \
         -serial telnet::9003,server,nowait > guest_log-3.txt 2>> guest_log-3.txt &
sleep 5
brctl addif br0 tap0
brctl addif br0 tap1
brctl addif br0 tap2

There are some parameters need to understand in QEMU script.

-uuid d0000000-1111-1111-1111-11111aaaaaaa \

this will assign a unique uuid for each VM, k8s needs different VM run with different uuid and MAC address.
It also can use command “uuidgen” to generate different uuid for each VM, but here just for debug so assign a “easy to read” uuid.

-device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \

Assign a special mac address for VM, because for k8s internal communication, use the same mac address will cause problem, please configure different address before run qemu script. It not easy to modify it after first boot (with cloud image)

* Notice, first number for mac address cannot be odd , it would cause problem.

-mem-path /mnt/hugepages \

Tell VM to use hugepage, if were not configure host with hugepage, please remove this line.

Now, please run scripts

bash nic-br.sh
bash run-qemu.sh 

Now, it can use telnet to connect each VM’s console

telnet localhost 9001
# or 
telnet localhost 9002
# or 
telnet localhost 9003

After system booted/initialed, if you cannot login, please “killall qemu-system-aarch64” and follow this article Change Ubuntu Cloud Image Password to change all images’s password.

Somehow, for recent cloud image, it cannot modify password before run qemu script.

Install Kubernetes

There are a lot of article to talk about how to install on Ubuntu, I choice one for reference : Install and Deploy Kubernetes on Ubuntu 18.04 LTS

Below will shortly note install command here.

Install Docker and Kubeadm on every node

sudo apt update 
sudo apt -y upgrade 

# install and enable docker
sudo apt -y install docker.io 
sudo systemctl enable docker 
sudo systemctl start docker 

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add 
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update
sudo apt-get install kubeadm -y

Configure Network IP address (Optional)

Update to you network environment, here we used fix ip address.

Add new record to /etc/hosts

192.168.110.50     k8s-master
192.168.110.60     k8s-worker-node1
192.168.110.70     k8s-worker-node2

Please modify /etc/netplan/50-cloud-init.yam, remove dhcp and add fix address. remember, each node has different IP. Following is only for k8s-master.

network:
     ethernets:
         eth0:
             addresses: [192.168.110.50/16,]
             gateway4: 192.168.1.1
             nameservers:
                 addresses: [8.8.8.8,8.8.4.4]
             match:
                 macaddress: '52:54:00:11:11:11'
             set-name: eth0
     version: 2

Configure Host name

Run following command on master node

sudo hostnamectl set-hostname  k8s-master

Run following command on other nodes

sudo hostnamectl set-hostname   k8s-worker-node1 
sudo hostnamectl set-hostname   k8s-worker-node2

Disable swap (Option)

if image is not from cloud image, default Linux enabled swap function, k8s doesn’t support it, please disable it manually on all nodes and remove swap mount in /etc/fstab.

 sudo swapoff -a 

Configure Kubernetes

Now, we can initial Kubernetes. Run following command

 sudo kubeadm init --pod-network-cidr=10.244.0.0/16 

wait for few minutes, it will show some result like this

Run following command to assign current account has right to access k8s.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run this command on another node(k8s-worker-node1 and k8s-worker-node2).

sudo kubeadm join 192.168.110.50:6443 --token 46c1oj.i00vyfyn027pgew9 \
     --discovery-token-ca-cert-hash sha256:ed05198cc603ae009a862477bacef329a9e9a2bedc4a040bc4731e6d5cf4f48e

Run get nodes command, it will show current status, the master node status is NotReady, but don’t worry, it will become Ready after add pod and CNI (container networking interface) plugin.

ubuntu@k8s-master:~$ kubectl get nodes
 NAME         STATUS     ROLES    AGE     VERSION
 k8s-master   NotReady   master   3m42s   v1.16.0

This action is very import on ARM64 k8s platform, in other document, they told you use flannel or some other document told you used old flannel. all of them not work for me. Please instead flannel with weave net.
(Ref Weave Net for NetworkPolicy and CoreDNS is not starting on Ubuntu 18.04.Bionic Beaver #980)
Run following command to install weave net

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Use the following command to view the status of the network:

Get nodes again.

Great!! it works.

MicroK8s

If you think steps above are too complex and you would like to only run k8s on one machine, microk8s is a good friend for you, can run on ARM64 and very easy to use, just run command

sudo snap install microk8s

And just add a prefix “microk8s.” before k8s command, ex:

microk8s.kubectl get nodes

See, it’s very easy to use. right?

The post Install Kubernetes on ARM64 Ubuntu 18.04 (with QEMU) appeared first on richliu's blog.

一般 change image size 是用這個指令

$ sudo qemu-img resize $UBUNTUVERSION-server-cloudimg-$ARCH.img +50G0
ex:

$ sudo qemu-img resize xenial-server-cloudimg-amd64-uefi1.img +50G

將 qcow2 image format 變成 raw disk

$ sudo qemu-img convert  xenial-server-cloudimg-amd64-uefi1.img xenial-server-cloudimg-amd64-uefi1.raw

之前測 ARM64 17.04 和 17.10 都會自動增加 partition size
但是最近碰到 18.04 似乎不會. (Both X64 and ARM64)

如果碰到的是 16.04 要增加 partition size 有問題, 要用 parted 修改 (似乎 fdisk 和 gdisk 都有問題)
ex:

$ parted xenial-server-cloudimg-amd64-uefi1.raw
(parted) p
Model:  (file)
Disk /work/research/vm/1.raw: 56.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
14      1049kB  5243kB  4194kB                     bios_grub
15      5243kB  116MB   111MB   fat32              boot, esp
 1      116MB   2361MB  2245MB  ext4

(parted) resizepart
Warning: Not all of the space available to /work/research/vm/1.raw appears to be used, you can fix the GPT to use
all of the space (an extra 104857600 blocks) or continue with the current setting?
parted: invalid token: 1

Partition number? 1
End?  [2361MB]? 56G

離開就會儲存

Ubuntu 16.04 的 cloud image 不知道為什麼 cloud-init 會無法結束, 如果不能接觸到 login 那 cloud image 也沒有用.

這時可以用另一個方式移除 cloud-init 相關的套件,
1. 使用 kpartx 將 image 掛到 loop devices 如果沒有, 就先安裝 kpartx 套件

$ sudo apt install kpartx 

2. mount partition (還有 proc, dev, sys)
3. 移除 cloud 相關套件
4. umount partition (要先 umount proc, dev, sys
5. 移除 loop device
6. convert raw file 到 qcow2 格式

mount cloud image partition

$ sudo kpartx -av xenial-server-cloudimg-amd64-uefi1.raw
add map loop0p1 (253:0): 0 109147673 linear 7:0 227328
add map loop0p14 (253:1): 0 8192 linear 7:0 2048
add map loop0p15 (253:2): 0 217088 linear 7:0 10240

loop0p1 就是我們要 mount 的 disk , (註: 有可能因系統不同, 所以會有其他的 device name)

mount disk

$ sudo su
$ mkdir disk
$ mount /dev/mapper/loop0p1 disk 
$ cd disk
$ mount --bind /proc proc
$ mount --bind /dev dev
$ mount --bind /sys sys
$ cd ..
$ chroot disk 

移除 cloud-init 相關的套件

$ dpkg  -l |grep cloud
ii  cloud-guest-utils                0.27-0ubuntu25.1                           all          cloud guest utilities
ii  cloud-init                       18.3-9-g2e62cb8a-0ubuntu1~16.04.2          all          Init scripts for cloud instances
ii  cloud-initramfs-copymods         0.27ubuntu1.5                              all          copy initramfs modules into root filesystem for later use
ii  cloud-initramfs-dyn-netconf      0.27ubuntu1.5                              all          write a network interface file in /run for BOOTIF
ii  ubuntu-cloudimage-keyring        2013.11.11                                 all          GnuPG keys of the Ubuntu Cloud Image builder

$ dpkg -r cloud-guest-utils  cloud-init cloud-initramfs-copymods  ubuntu-server

umount disk

$ exit 
$ cd disk 
$ umount proc
$ umount dev
$ umount sys
$ cd ..
$ umount disk

最後將 raw disk image 轉成 qcow2

$ sudo qemu-img convert  xenial-server-cloudimg-amd64-uefi1.raw xenial-server-cloudimg-amd64-uefi1.img

這樣 16.04 開機就不會卡在 cloud init script 了.

The post Change Ubuntu Cloud Image Size appeared first on richliu's blog.

應該只有影響 ocata

Ref.
OpenStack KVM

[ocata] unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor

程式解法
aarch64: set default cpu_mode to host-passthrough

The post unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor appeared first on richliu's blog.

http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/

The post Ubuntu Cloud Image Taiwan Mirror Site appeared first on richliu's blog.

Debian/Ubuntu ARM64 kernel (4KB Pagesize)
Hugepage size is 2MB/1GB

64KB Pagesize, hugepage is 512MB.

x86_64
2MB : (has PSE cpuinfo flag)
1GB : (has PDPE1GB cpuinfo flag)

different configuration has different setting

ref.
Hugepages

The post ARM64 Hugepage(HugeTLB) Size appeared first on richliu's blog.

Add this line to QEMU command

-virtfs local,path=/path/to/share,mount_tag=host0,security_model=passthrough,id=host0

and in guest os, edit /etc/fstab and add

host0   /wherever    9p      trans=virtio,version=9p2000.L   0 0

ref.
https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu

The post QEMU Share Directory with Host Without Networking appeared first on richliu's blog.

If want to use it, it needs to add new username and password into it.
and remove some packages.Download cloud Image from ubuntu, ex: download Ubuntu Zesty Cloud Image

wget https://cloud-images.ubuntu.com/zesty/current/zesty-server-cloudimg-arm64.img 

If locate in Taiwan, there is Taiwan mirror site,http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/, it would be fast then cloud-images.ubuntu.com

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/zesty/current/zesty-server-cloudimg-arm64.img

Zesty only has UEFI image, so, there is no -disk1 or -uefi postfix keyword.

If used bionic (Ubuntu 18.04), if you found change password function doesn’t work.
Please shutdown this image (suppose run with QEMU) and run change password command again .

Suppose that something modify password when booting on Ubuntu 18.04.

Install backdoor-image package, this program can help to install new username and password into cloud image.

apt install bzr qemu-utils  
bzr branch lp:~smoser/+junk/backdoor-image  

sudo apt-get install -qy cloud-image-utils git
git clone https://gist.github.com/8c65b8771d5ab1d99c44c285323dfff6.git backdoor-image

Add new password, but first account is not work for me most time, so, I would like to add new username and password. (note: please don’t use this as default password on business purpose. )

cd backdoor-image  
sudo ./backdoor-image --user ubuntu --password ubuntu --password-auth ../zesty-server-cloudimg-arm64.img 
sudo ./backdoor-image --user test --password 123456 --password-auth ../zesty-server-cloudimg-arm64.img 

After boot into virtual machine with this image, it can remove unnecessary packages.

sudo apt remove cloud-initramfs-copymods cloud-init

The post Change Ubuntu Cloud Image Password appeared first on richliu's blog.

no valid host was found. There are not enough hosts available.
Code
500
Details
File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", 
line 526, in build_instances context, request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", 
line 597, in _schedule_instances hosts = self.scheduler_client.select_destinations(context, spec_obj) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", 
line 371, in wrapped return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", 
line 51, in select_destinations return self.queryclient.select_destinations(context, spec_obj) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", 
line 37, in __run_method return getattr(self.instance, __name)(*args, **kwargs) 
File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", 
line 32, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj) 
File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", 
line 129, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", 
line 169, in call retry=self.retry) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", 
line 97, in _send timeout=timeout, retry=retry) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 458, in send retry=retry) File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 449, in _send raise result

Possible reason

* if using cloud uefi image, it needs to install qemu-efi package in computer node, don’t know why it won’t install it automatically .
$ apt install qemu-efi
* when add new UEFI image, it needs to add new meta data parameter in Horization
hw_firmware_type=uefi

if see

2017-08-01 14:02:13.612 2409 ERROR nova.compute.manager [instance: b334f0b3-ca9a-4f9a-b488-532dce00265a] libvirtError: unsupported configuration: logfile not supported in this QEMU binary
2017-08-01 14:02:13.612 2409 ERROR nova.compute.manager [instance: b334f0b3-ca9a-4f9a-b488-532dce00265a]
# Use the "logd" backend for handling stdout/stderr from QEMU processes.

Reason
* Libvirt later version support new function, logd, it’s a new console log function, but ARM64 doesn’t support it. workaround solution is to disable it

Edit file /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py, find

MIN_LIBVIRT_VIRTLOGD = (1, 3, 3)
MIN_QEMU_VIRTLOGD = (2, 7, 0)

Replaced it with

MIN_LIBVIRT_VIRTLOGD = (9, 9, 9)
MIN_QEMU_VIRTLOGD = (9, 9, 9)

it can avoid to call libvirt logd function.

rm -f /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.pyc, make sure it will call .py
then, reboot service (or system)

Ref.
http://dozencloud.org/

The post No valid host was found. There are not enough hosts available appeared first on richliu's blog.