no-dangling-assignment-gsl -std=c++20 -Wno-trigraphs -gsimple-template-names -fno-exceptio00:01:51 [472/19443]
nc++ -isystem../../third_party/libc++/src/include -isystem../../third_party/libc++abi/src/include -fvisibility
-inlines-hidden -O2 -pipe -march=armv8.2-a -Wno-unknown-warning-option -c ../../third_party/libyuv/source/comp
are_neon64.cc -o obj/third_party/libyuv/libyuv_neon/compare_neon64.o
../../third_party/libyuv/source/compare_neon64.cc:175:8: error: instruction requires: dotprod
  175 |       "udot        v4.4s, v0.16b, v6.16b         \n"
      |        ^
<inline asm>:14:1: note: instantiated into assembly here
   14 | udot        v4.4s, v0.16b, v6.16b
      | ^
../../third_party/libyuv/source/compare_neon64.cc:176:8: error: instruction requires: dotprod
  176 |       "udot        v5.4s, v1.16b, v6.16b         \n"
      |        ^
<inline asm>:15:1: note: instantiated into assembly here
   15 | udot        v5.4s, v1.16b, v6.16b
      | ^
../../third_party/libyuv/source/compare_neon64.cc:204:8: error: instruction requires: dotprod
  204 |       "udot        v4.4s, v0.16b, v0.16b         \n"
      |        ^
<inline asm>:10:1: note: instantiated into assembly here
   10 | udot        v4.4s, v0.16b, v0.16b
      | ^
../../third_party/libyuv/source/compare_neon64.cc:205:8: error: instruction requires: dotprod
  205 |       "udot        v5.4s, v1.16b, v1.16b         \n"
      |        ^
<inline asm>:11:1: note: instantiated into assembly here
   11 | udot        v5.4s, v1.16b, v1.16b
      | ^
4 errors generated.
[10940/63891] aarch64-unknown-linux-gnu-clang++-18 -MD -MF obj/third_party/libyuv/libyuv_neon/rotate_neon.o.d

Seems because some -march/-mcpu/-mtune drop ‘dotprod’ parameters, it needs to add the extension manuall. ‘+dotprod’.

Edit the file /etc/portage/make.conf, and change the COMMON_FLAGS to

COMMON_FLAGS="-mcpu=neoverse-n1+crc+crypto+ssbs+dotprod -O2 -pipe"

It can use the following command to get current CPU flag

gcc -v -E -x c /dev/null -o /dev/null -march=native -mcpu=native 2>&1 | grep /cc1

Thanks Gentoo user zukunf to provide those information.

The post Gentoo ARM64 Altra Compile chromium Issue appeared first on richliu's blog.

Qualcomm’s new desktop SoCs might change this landscape. However, we can still leverage the Ampere Altra platform to run LLMs.

Since this is a personal blog, most articles here are my personal notes. I’ll keep them updated as my understanding evolves.

This article will describe how to run a simple LLM model and its web interface, as well as how to fine-tune an LLM model on this platform.

Before running the commands, you’ll need to install the Nvidia driver. Refer to this article, “[How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform]”, for instructions on installing the Nvidia driver and Docker driver.

Ollama and Open-Webui don’t require a GPU. However, having a GPU is beneficial. Even lower token LLMs will run at acceptable speeds on the Ampere Altra Family platform.

This article’s hardware setup is as follows:

• CPU : Ampere Altra Family
• Board: AsRock ALTRAD8UD
• GPU: Nvidia RTX 4080

Use current LLM model with GUI

At this point in the process, using Ollama and Open-Webui is a straightforward approach for running LLMs. Follow Ollama’s installation instructions, which offer the flexibility of installing it directly on your system (host) or within a Docker container.

curl -fsSL https://ollama.com/install.sh | sh

By default, the Ollama service listens on localhost (127.0.0.1). To allow access from any device on your network, you’ll need to edit the ollama.service file.

In the [Service] section, add the following line:

Environment="OLLAMA_HOST=0.0.0.0:11434"

For Open-Webui, we recommend using Docker for a simpler solution. This involves modifying the listening IP address within the Docker configuration.

Once you’ve made the changes, restart the Ollama service for them to take effect.

systemctl daemon-reload
systemctl restart ollama

Run the ollama command and try it

# ollama run llama2
>>> who are you

I'm LLaMA, an AI assistant developed by Meta AI that can understand and respond to human input in a conversational manner.
I'm here to help you with any questions or topics you'd like to discuss! Is there something specific you'd like to talk
about or ask?

>>> 

If doesn’t work, try to restart ollama and try again.

Big LLM

If choose some big LLM like LLaMA2-70b or Qwen

Open-Webui

Following command is to run the Open-Webui docker image on host, 3000 is host port for Open-Webui. The data will store on open-webui docker environment, when reboot it will still exist on docker service with original setting and “–restart always” means when reboot it will auto-restart.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

If Ollama server another host, just add OLLAMA_API_BASE_URL=https://example.com/api to the docker command,

docker run -d -p 3000:8080 -e OLLAMA_API_BASE_URL=https://example.com/api -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Upon login, you’ll need to provide an email and password. Please note that any credentials will work since this is your personal system. Once logged in, you can also edit the OLLAMA_API_BASE on the settings page.

Now, it connected from Open-Webui to ollmam backend, and we have a running sytsem on our machine.
Run sample chat here and choice llama2:latest as LLM model.

LLM Fine-Turning – LLaMA-Factory

Fine-tuning LLMs ideally requires a GPU card for better performance. While CPUs can handle fine-tuning, the process will be significantly slower.

Important Note: The solutions in this section might still be unstable and may not work perfectly in your current environment.

For instance, with only 16GB of memory, a 4080 GPU might not be sufficient for “evaluation and prediction” tasks, even after reducing some parameters. This still can lead to CUDA out-of-memory errors.

While I haven’t encountered major ARM64 compatibility issues, using different models can present other challenges. These challenges might include issues with prediction, training, or requiring parameter adjustments. Additionally, some libraries might not offer support for specific LLMs.

Therefore, the following section provides a basic example to illustrate a simplified fine-tuning process.

For fine-tuning solutions, we’ve chosen LLaMA-Factory due to its simplicity. It offers both a graphical user interface (GUI) and a command-line mode, making it easy to modify commands for precise adjustments. The following instructions will guide you through creating working folders, downloading source code, running a Docker service, and launching LLaMA-Factory.

# Create work folder
mkdir -p /nvme/model 
cd /nvme 
git clone https://github.com/hiyouga/LLaMA-Factory.git

# Run docker, it will use host GPU and map the folder into docker.
sudo docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -it --rm \
     -v /nvme:/nvme \
     -p 7860:7860 \
     nvcr.io/nvidia/pytorch:24.01-py3

# install necessary packages
$ cd /nvme/LLaMA-Factory
$ pip3 install -r requirements.txt
$ pip3 install tiktoken transformers_stream_generator

# run the LLaMA-Factory
$ CUDA_VISIBLE_DEVICES=0 python src/train_web.py

Now, it can acces the webgui via http://host IP:7860.

It can create another Docker window to run command on the LLaMA-Factory docker image.

docker exec -it <docker name> /bin/bash 

for download LLMs, you need to use git download from huggingface, ex:

# LLama 2 need username and token(not passowrd) to download it. 
git clone https://huggingface.co/meta-llama/Llama-2-7b-hf

# Qwen
git clone https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat
git clone https://huggingface.co/Qwen/Qwen1.5-1.8B

Pre-Training

Model name, choice the LLMs name , like Qwen1.5-1.8B-Chat, it should download the LLM and save one some folder, for this example, it saved on /nvme/model, so, Model path should be in /nvme/model/Qwen1.5-1.8B-Chat.

First time, I suggest to use small LLM for fine-tuning, Qweb1.5-0.8B might have issue, will suggest from 1.5-1.8B to start.

Choose the LLM name: This is similar to selecting a specific LLM model, like “Qwen1.5-1.8B-Chat”. The instructions will indicate your LLM folder. For this example, the model would be saved in /nvme/model/Qwen1.5-1.8B-Chat.

Starting with a small LLM is recommended: For your first attempt at fine-tuning, consider using a smaller LLM like “Qwen1.5-1.8B”. It’s possible that “Qweb1.5-0.8B” might cause issues, so we recommend starting with models in the 1.5-1.8B range.

Here are the fine-tuning options you can adjust in LLaMA-Factory:

Dataset: You can add datasets to this mode. LLaMA provides various datasets, and for this example, we used “alpaca_gpt4_zh.”
Gradient accumulation: This is currently set to 4.
Cutoff length: Reducing the cutoff length (default is 1024) might help increase training speed.
Compute type: The compute type depends on your environment. In this case, fp16 works, but bp16 doesn’t sometimes.
Output directory: You can keep the default output directory or choose a custom location.
Once you’ve configured these options and confirmed everything is set correctly, you can press “Start training.” Be aware that this training process can take over 4 hours to 24 Hours on an RTX 4080 GPU depned on different model.

Evaluate & Predict

Next step is evaluate and predict, Adpater path just fill previous steps’ output dir. Dataset is the same.

When everything is ready, press Start. Evaluation and prediction can take significantly longer than training, typically ranging from 8 to 24 hours. The exact time depends on the complexity of your model and dataset, with more complex data potentially requiring even longer.

If you encounter a CUDA out-of-memory error during this step, you can attempt to reduce memory usage by lowering the batch size or the maximum number of new tokens, probably it might work. [ref]

Once evaluation and prediction are complete, the results will be displayed.

Test Chat Model

You can test chat result by load module.

Export model

Remember to fill export dir, in here I used “/nvme/newmodel/Qwen1.5-1.8B-Chat”

cd /nvme/newmodel 
python3 /nvme/llama.cpp/convert-hf-to-gguf.py Qwen1.5-1.8B-Chat --outfile test.gguf

# Use llama.cpp to test this model 
/nvme/llama.cpp/build/bin/main -m test.gguf -p "who are you?"

# Command mode
/nvme/llama.cpp/build/bin/main -m test.gguf -ins

Create a file named “Modelfile”, and fill this

FROM /nvme/newmodel/test.gguf

Now, run ollama to include this LLM.

ollama create test -f Modelfile
ollama run test "who are you?"
====
 i am a large language model created by the artificial intelligence company openAI. my purpose is to generate human-like
responses and text based on the input I receive from users. can I help you with anything today? #openai

#chatbot
Sure, I'm here to assist you! How can I help you today? Is there something specific you'd like to talk about or ask me?
#openaiChatBot不断地尝试。 #AI #MachineLearning

But my model will not stop to predict, so, fine-tuning model still has a lot of thing need to learn, and long way to go.

To be continue.

The post LLM Chat WebGUI and Fine-Turning on Ampere Altra ARM64 Platform appeared first on richliu's blog.

In this article, it will walk through the process of installiung Stable Diffusion WebUI on ARM64 + Nvidia RTX platform.
For such as hardware combinarion, it needs to solve some compatiability issues, so, here will describe how to install it and fix those issues.

Platform

CPU: Ampere Altra ARM64 CPU
Board: AsRock ALTRAD8UD
OS: Ubuntu 22.04

The platform is the most powerful ARM64 platform for a homelab that we can obtain today. There are some server models available, but they come with high-speed cooling fans that are not suitable for home use.

Nvidia has provided comprehensive library support for the ARM64 platform, including drivers, CUDA, and cuDNN. However, despite these efforts, there are still gaps in support. Even Nvidia’s Jetson product line lacks adequate support for ARM64 platforms in certain applications such as PyTorch. While there exists an individual PyTorch accelerator package for direct download on x64 platforms, the same cannot be said for ARM64 platforms. Although there is technically one package available, its availability is severely limited, rendering it practically useless.

The only viable solution to overcome these limitations is to utilize Nvidia’s Docker service to install Stable Diffusion.

Installation

Install Ubuntu 22.04 without selecting the HWE (Hardware Enablement) support. It appears that the current Nvidia driver version has issues, leading to potential soft-lock bugs. A general kernel version (5.15) is preferable for stability.

Next, install the necessary Ubuntu packages.

sudo apt-get install build-essential

Download Nvidia ARM64 (aarch64) Display Driver , current version is ok, when I write this article, current version is 535.154.05

Docker

Install Docker by following the Docker guide Install Docker Engine on Ubuntu,”. Also provide the necessary installation steps here. If already installed Docker or if these steps don’t work, please refer to the original website for updated instructions.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Nvidia Container Toolkit

Follow the instructions for installing the NVIDIA Container Toolit. It can modify the Docker configuration using this toolkit. Alternatively, modifying the daemon.json file might achieve the same result, although I haven’t personally tested it. Below are the instructions:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

It can check the file /etc/docker/daemon.json file, the configuration should be like this.

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

Run Nvidia pytorch docker image

Following is example command to run the Nvidia pytorch docker image

sudo docker run --gpus all -it --rm \
        -v /host/work:/container/work \
        -p 7860:7860 \
        nvcr.io/nvidia/pytorch:XX.XX-py3

/host/work means work directory on host, /container/work means work directory in container(docker). -p 7860 mean gui default port, it needs to forward to host machine. XX.XX means container versiom please see tags to get container version, or this URL . [NGC Pytorch]. Following is example command.

sudo docker run --gpus all -it --rm \
        -v /work:/work \
        -p 7860:7860 \
        nvcr.io/nvidia/pytorch:24.01-py3

Stable Diffustion WebUI

First steps clone Stable Diffustion WebUI source code.

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

Then, it needs to modify the settings file. Edit webui.sh, and adjust the parameters to the following numbers.

use_venv=0
can_run_as_root=1

As we are operating within a container, there’s no need for a virtual environment (venv), and the script can be run as root. Modify the file webui-user.sh and add COMMANDLINE_ARGS accordingly.

export COMMANDLINE_ARGS="--no-half --precision full --listen"

or run it directly

export COMMANDLINE_ARGS="--no-half --precision full --listen" ; bash webui.sh

Currently, Stable Diffusion will download some packages, but errors may occur during the installation process. We’ll address and fix these errors in the next few steps.

CV2

Error message :

    if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 28, in __load_extra_py_code_for_module
    py_module = importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py", line 168, in <module>
    LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'

Modify file “/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py”, comment out line 169 like following.[ref]

# LayerId = cv2.dnn.DictValue

CodeFormer

Error message:

"/work/stable-diffusion-webui/repositories/CodeFormer/facelib/utils/face_restoration_helper.py", line 7, in <module>
        from facelib.detection import init_detection_model
      File "/work/stable-diffusion-webui/repositories/CodeFormer/facelib/detection/__init__.py", line 11, in <module>
        from .yolov5face.face_detector import YoloDetector
      File "/work/stable-diffusion-webui/repositories/CodeFormer/facelib/detection/yolov5face/face_detector.py", line 20, in <module>
        IS_HIGH_VERSION = tuple(map(int, torch.__version__.split('+')[0].split('.'))) >= (1, 9, 0)
    ValueError: invalid literal for int() with base 10: '0a0'

This issue caused by pytorch.version will return illegeal number like 2.2.0a0+81ea7a4, so it will fail to parse this mesage, temperory solution is to remove the version number, edit file “stable-diffusion-webui/repositories/CodeFormer/facelib/detection/yolov5face/face_detector.py”. and modify line 20, IS_HIGH_VERSION to following version to following:[ref]

version_string = torch.version.split('+')[0]
version_string = ''.join(filter(str.isdigit, version_string))
IS_HIGH_VERSION = tuple(map(int, version_string.split('.'))) >= (1, 9, 0)

Now, the Stable Diffusion GUI should be working as well as expected

The post How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform appeared first on richliu's blog.

My hardware also different, if you don’t have such powerful machine, it can skip numa part, that’s only for high end ARM64 machine.
The hardware is Ampere Mt. Collins which has 2*Ampere Altra 80 Cores CPU and 8*32G Memory, my experiment is run different VM on different CPU, to seperate them and lock it on different numa node.

There are two CPUs, so, it can create 3 nodes, so, I have plan to allocate resource like that.
1. k8s master on socket 0: 10 CPUs, 16G memory
2. k8s node1 on socket 0 : 70 CPUs, 100G memory
3. k8s node2 on socket 1: 80 CPUs, 120G memory

Kubernetes version v1.28.2
Ubuntu 22.04

Notice: Before reading this document, this document is personal experiment, not Ampere official document, take your own risk.

Prepare VM from libvirt

The server has remote control system which problem sufficent resolution to use GUI on server, so, I install ubuntu 22.04 on host system, than use virt-manager to create k8s master node, for qemu image, I create the disk image alone, seems create image when run install will allocate all disk space, seems that’s a virt-manager bug.
And the host network is configured to be a bridge interface, in my case, that’s br0.

So, when uses virt-manager to create VM, it can bridge network interface to br0.

Configure the VM with Numa

It can use numastat to check currnet numa status, this system has two numa nodes. it can use numastat and numactl to get more numa information.

# numastat -n

Per-node numastat info (in MBs):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
Numa_Hit               591239.17       791075.41      1382314.58
Numa_Miss                   0.00            0.00            0.00
Numa_Foreign                0.00            0.00            0.00
Interleave_Hit              8.63            7.13           15.77
Local_Node             519434.17       777974.99      1297409.16
Other_Node              71805.00        13100.42        84905.42

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 128377 MB
node 0 free: 80224 MB
node 1 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 1 size: 126426 MB
node 1 free: 93888 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

virt-manager doesn’t have advance command to assign CPU, probably modify xml can get that purpose, but it more converince to use the “virsh” cli command to adjust VM parameters. “virsh list” show current VMs, “virsh vcpupin list current VM CPU scheduling”, “virsh vcpupin domain-id vcpu cpulist” to modify the VCPU run on which one CPU.

For memory, it needs to use “virsh numatune” to lock the memory on different node.
“virsh numatune domain [–mode mode] [–nodeset nodeset] [[–config] [–live] | [–current]]”, –current can update current running VM machine.

$ virsh list
 Id   Name          State
-----------------------------
 9    k8s-master   running
 10   k8s-node1     running

$ virsh vcpupin k8s-master
 VCPU   CPU Affinity
----------------------
 0      0-159
 1      0-159
 2      0-159
 3      0-159
 4      0-159
 5      0-159
 6      0-159
 7      0-159
 8      0-159
 9      0-159

# this command can lock 9 k8s-master's VCPU 0 on physical CPU 0. 
$ virsh vcpupin 9 0 0 

# Show current VM's numatune, for example, following is unset status. 
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   :

$ virsh numatune 9 --nodeset 0

# the numa_nodeset was set to 0
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   : 0

The problem is, seems this script need to run everytime, alternativate solution is to modify XML, but Mt. collis has 160 cores, I won’t to modify the XML file. script is more converience on this situation, following is my sample script to adjust k8s-master, k8s-node1, and k8s-node2’s parameter, it can easy to modify it if you install other system or AltraMax CPU.

#!/bin/bash

configure_vm(){
        local VM_NAME="$1"
        local VM_VCPU="$2"
        local VM_CPU="$3"
        local VM_NUMA="$4"
        local VM_TYPE="$5" # VM_TYPE for MASTER or NODE0, NODE1 .. self define name

        id=$(virsh list | grep $VM_NAME | awk '{print $1}')
        state=$(virsh list | grep $VM_NAME | awk '{print $3}')

        if [ "$state" != "running" ]; then
          echo "$VM_TYPE is not running"
          exit 1
        else
          echo "configure $VM_TYPE"
        fi
        numa_nodeset=$(virsh numatune $id | grep numa_nodeset | awk '{print $3}')

        start_cpu=$(echo $VM_VCPU | cut -d'-' -f1)
        end_cpu=$(echo $VM_VCPU | cut -d'-' -f2)
        for ((vcpu = start_cpu; vcpu <= end_cpu; vcpu++)); do
          # echo "$id $vcpu $VM_CPU"
          virsh vcpupin $id $vcpu $VM_CPU > /dev/null
        done
        if [ -n "$numa_nodeset" ]; then
          echo "$VM_TYPE numa_nodeset is $numa_nodeset"
        else
          # echo "$id $VM_NUMA"
          virsh numatune $id --nodeset $VM_NUMA
        fi

}

configure_vm "k8s-master" "0-9" "0-9" "0" "Master"
configure_vm "k8s-node1" "0-69" "10-79" "0" "Node0"
configure_vm "k8s-node2" "0-79" "80-159" "1" "Node1"


====

# After run the script, it can run numastat <pid> to see the qemu runs numa status. ex:
$ sudo numastat 206601

Per-node process memory usage (in MBs) for PID 206601 (qemu-system-aar)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                        11.51            0.00           11.51
Stack                        0.04            0.00            0.04
Private                   5984.93            0.00         5984.94
----------------  --------------- --------------- ---------------
Total                     5996.48            0.00         5996.48

In here, on socket 0 , 0-9 and 0-69 is vcpu, so, all start from 0. After run this script, it can use “virsh vcpupin” and “virsh numatune” to get current setting. Maybe cpuset parameter in XML can more easy to get this function, but I have not try it.

Install K8s

Here is my script, just reference some later blog and my previous configurtion script, so, it can copy and paste to system and run it directly.

Install basic system and k8s

# setup timezone
sudo apt install ntpdate -y
sudo ntpdate watch.stdtime.gov.tw
sudo timedatectl set-timezone Asia/Taipei

# install and enable docker

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update
sudo apt install -y containerd.io


containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

echo "----------------"
echo "   swapoff"
echo "----------------"
sudo swapoff -a
sudo sed -i '/swap/ s/^\(.*\)$/#\1/g' /etc/fstab

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update

sudo apt-get -y install kubelet
sudo apt-get -y install kubeadm
sudo apt-get -y install kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Update kernel parametes
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# update sysctl
sudo sysctl --system

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

echo "if this is work node, it can run kubeadm join now"

echo "alias k=kubectl" >> .bashrc

Setup Master Node

Run this script on master node.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# echo "enable master to run pods"
# kubectl taint nodes --all node-role.kubernetes.io/master-

It will shows some message below, then run it on k8s-node1 and k8s-node2

kubeadm join 192.168.110.230:6443 –token e14f7k.35c3jpny81ry70re \

        –discovery-token-ca-cert-hash sha256:6636a7e4ec3542f19152f308f0789c52287b743b60efa3c13e7c23ce1147c581

If forgot command above, it can create another token to let other k8s node join, ex:

kubeadm token create --print-join-command

Flannel : Network fabric for container

K8s need a network fabric to provide internal network topology, there are many choices, but we will install matellb later, some other network like calico might have potential issue, so, we choice flannel as network fabric.

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

After run this command, the k8s node status will change to Ready, if not install network fabric, the node status will be NotReady.

$ kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   26h   v1.28.2
k8s-node1    Ready    <none>          26h   v1.28.2
k8s-node2    Ready    <none>          26h   v1.28.2

Helm: Kubbernetes Applications Manager

Install helm to further install metallb purpose, on previous article, helm2 needs another application to connect to k8s, but helm3 doesn’t need it, more easy to use on k8s.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o /tmp/get_helm.sh
chmod 700 /tmp/get_helm.sh
sudo /tmp/get_helm.sh

Metallb: Bere metal load-balancer for K8s

Install k8s on private side, it needs load-balancer to provide external IP to communication with internet or intranet. If you install Metallb, the “kubectl get svc”, the EXTERNAL-IP shows “Pending”, and k8s doesn’t response the arp on host, probably it doesn’t configure kube-proxy, it you have experience on previous metallb installtion, it might miss this part.

PATH=$PATH:/usr/local/bin/

helm repo add metallb https://metallb.github.io/metallb
helm --namespace metallb-system install --create-namespace metallb metallb/metallb -f metallb-values.yaml

# https://metallb.universe.tf/installation/
# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl apply -f - -n kube-system

metallb-values.yaml is also different like previous setting, it needs to configure L2Advertisement on metallb, without it, your metallb won’t work. Follwing is metallb-values.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: primary-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.110.200-192.168.110.219

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - primary-pool

Traefik: TBD

Linkerd

Just show the install script here

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

echo "export PATH=$PATH:/home/$USER/.linkerd2/bin" >> ~/.bashrc

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

Other

it can insert isolinux to host linux kernel parameter to isolate the Linux scheduling to k8s nodes. (TBD)

isolcpus=10-159 nohz=on nohz_full=10-159

Test with Nginx

It can deploy nginx to test this system

kubectl create deploy nginx --image=nginx
kubectl expose deploy nginx --port=80 --target-port=80 --type=LoadBalancer
kubectl get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
nginx   LoadBalancer   10.99.211.255   192.168.110.200   80:31011/TCP   25h

Now, it can try to access http://192.168.110.200.
if want to Delete nginx deploy, run following command.

kubectl delete deploy nginx
kubectl delete svc nginx

Test Numa

It can run emojivoto to test the system only runs on socket1, use it because it can general a lot of traffic between pods.

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml \
  | kubectl apply -f -

Now it can use htop to show all CPUs, and we can see most traffic on CPU 1~10 and 81~160. There are a little CPU usage on CPU 78 and 80, but it should be Linux regular scheduling(I have not add isocpu to host CPU this time)

The post ARM64 Kubernetes Qemu Metallb Linkerd Numa appeared first on richliu's blog.

[root@localhost linux-4.18.0-80.11.2.el8_0]# make install                                            [55/2101]
/bin/sh ./arch/arm64/boot/install.sh 4.18.0 \
arch/arm64/boot/Image System.map "/boot"
dkms: running auto installation service for kernel 4.18.0
Deprecated feature: REMAKE_INITRD
Deprecated feature: REMAKE_INITRD
Deprecated feature: REMAKE_INITRD
Sign command: /lib/modules/4.18.0/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub                                                               Deprecated feature: REMAKE_INITRD                                                                             
Building module:
Cleaning build area...(bad exit status: 2)
make -j160 KERNELRELEASE=4.18.0 -C /lib/modules/4.18.0/build M=/var/lib/dkms/bnxt_en/1.10.2.221.0.114.0/build
KVER=4.18.0...(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.18.0 (aarch64)
Consult /var/lib/dkms/bnxt_en/1.10.2.221.0.114.0/build/make.log for more information.
Sign command: /lib/modules/4.18.0/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Deprecated feature: REMAKE_INITRD

Building module:
Cleaning build area...(bad exit status: 2)
make -j160 KERNELRELEASE=4.18.0 -C /lib/modules/4.18.0/build M=/var/lib/dkms/bnxt_re/221.0.114.0/build KVER=4$
18.0 BNXT_EN_INC=/var/lib/dkms/bnxt_re/221.0.114.0/source BNXT_QPLIB_INC=/var/lib/dkms/bnxt_re/221.0.114.0/so$
rce...(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.18.0 (aarch64)
Consult /var/lib/dkms/bnxt_re/221.0.114.0/build/make.log for more information.
 Done.

And check the log file “/var/lib/dkms/bnxt_re/221.0.114.0/build/make.log”

It shows

DKMS make.log for bnxt_en-1.10.2.221.0.114.0 for kernel 4.18.0 (aarch64)
Tue Sep 27 15:41:04 CST 2022
Makefile:603: arch/aarch64/Makefile: No such file or directory
make[2]: *** No rule to make target 'arch/aarch64/Makefile'.  Stop.

Seems cannot find the Makefile, that because the drvier cannot locate right Makefile, because the include architecture is arm64, not aarch64 in Linux kernel .

By the way, basicly arm64 and aarch64 are the same thing, I don’t know why it shows aarch64 on kernel architecture and use arm64 as fold name. probably some traditional history.

solution is easy, go to the include folder and link arm64 to aarch, DONE! ex:

cd /lib/modules/4.18.0/build/arch
ln -sf arm64 aarch64

The post Linux Kernel Install Fail Because Broadcom NetXtreme 100G DKMS Driver Build Failed. appeared first on richliu's blog.

Install Necessary Packages

Install packages, it should have to install other packages like openssl-dev.

get package ampere-8.3.0-20191025-dynamic-nosysroot-nativetools.tar.xz from Ampere customer connect or ampere development source web.

yum groupinstall "Development Tools"

yum -y install ncurses-devel
yum -y install hmaccalc zlib-devel binutils-devel elfutils-devel libelf-devel newt-devel python-devel
yum -y install audit-libs-devel numactl-devel pciutils-devel openssl-devel
yum -y install bc bison perl-ExtUtils-Embed.noarch xmlto asciidoc
rpm –ivh kernel-4.18.0-80.11.2.el8.<version>.src.rpm


mkdir -p /opt/amp
tar -xf ampere-8.3.0-20191025-dynamic-nosysroot-nativetools.tar.xz -C /opt/amp
export PATH=/opt/amp/ampere-8.3.0-20191025-dynamic-nosysroot/bin:$PATH;
which gcc
/opt/amp/ampere-8.3.0-20191025/bin/gcc

if gcc is not in /opt/amp, please check your directory in /opt/amy

Build Kernel

command to build kernel

cd /root/rpmbuild/SOURCES/
tar xvf linux-4.18.0-80.11.2.el8_0.tar.xz
cd linux-4.18.0-80.11.2.el8_0
cp ../kernel-aarch64-emag.config .config
make olddefconfig

Try build once and make sure everything is ok

make -j `nproc`

If buiild OK, generate RPM files.

make -j `nproc` rpm-pkg

if everything is ok, the RPM files will be on /root/rpmbuild/RPMS/aarch64/

Install RPM packages

/root/rpmbuild/RPMS/aarch64
yum localinstall kernel-4.18.0-1.aarch64.rpm kernel-headers-4.18.0-1.aarch64.rpm

The post Build Ampere CentOS Kernel appeared first on richliu's blog.

NVME 4K Write Performance

Situation:
Run fio 4K write on 24 NVME drivers, but some drivers might not get higher IOPS performance

Solution:
It should be caused by enable SMMU, so, if doesn’t use VM, just disable iommu to get better performance
Add iommu.passthrough=1

General IO performance Tuning

For high bandwidth card, most time it needs to disable irq_balancing
$ systemctl stop irqbalance.service

Some Performance Tuning Parameters

ulimit -s unlimited 
ulimit -l 2097152 
echo 8 | sudo tee /proc/sys/vm/dirty_ratio 
echo 1 | sudo tee /proc/sys/vm/swappiness 
echo 1 | sudo tee /proc/sys/vm/zone_reclaim_mode 
echo 3 | sudo tee /proc/sys/vm/drop_caches 
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space 
echo 0 | sudo tee /proc/sys/kernel/numa_balancing 
echo never > /sys/kernel/mm/transparent_hugepage/enabled 
cpupower frequency-set -g performance 
export MALLOC_CONF=thp:always
 

The post ARM64 Linux Performance Tuning appeared first on richliu's blog.

Remove Packages

$ sudo yum remove docker-ce docker-ce-cli containerd.io

Install Containterd First

This is most important step, without this, docker’s install script will fail.

$ sudo dnf install https://download.docker.com/linux/centos/7/aarch64/stable/Packages/containerd.io-1.2.6-3.3.el7.aarch64.rpm

Docker Script

Get docker script and run it
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh
Add your current username to docker group if not run on root account.
$ sudo usermod -aG docker your-user

Restart Docker And Enable Firewall

$ sudo systemctl start docker
$ firewall-cmd –zone=public –add-masquerade –permanent
$ firewall-cmd –reload

ref.
[1] https://linuxconfig.org/how-to-install-docker-in-rhel-8
[2] https://docs.docker.com/engine/install/centos/
[3] https://serverfault.com/questions/987686/no-network-connectivity-to-from-docker-ce-container-on-centos-8

The post CentOS8 ARM64/AARCH64 Install Docker appeared first on richliu's blog.

Hardware topology 

[ARM64 CLIENT] <——-> [Server]  <——-> Internet 

SERVER ethernet ip is 10.0.0.1 , another interface connect to internet. 

Install necessary packages 

Install TFTP, HTTP and DHCPD Server and rpm2cpio 

$ apt install tftpd-hpa apache2 isc-dhcp-server rpm2cpio

Modify dhcpd server configuration 

$ vim /etc/dhcp/dhcpd.conf

following is example configuration,it can be changed to fit your environment 

authoritative;
default-lease-time                      259200;
max-lease-time                          518400;
option routers                          10.0.0.1;
option subnet-mask                      255.255.255.0;
option domain-name-servers              168.95.1.1,8.8.8.8;
ddns-update-style                       interim;
ignore client-updates;
allow booting;
allow bootp;
allow unknown-clients;
server-name                             pxis;

subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.100 10.0.0.200;
    option ip-forwarding off;
    option subnet-mask                  255.255.255.0;
    next-server 10.0.0.1;
    filename "shim.efi";

}

Modify tftpd setting 

$ vim /etc/default/tftpd-hpa

Add –create to TFTP_OPTIONS and change tftp directory to /tftpboot 

following is example configuration file 

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/tftpboot"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure  --create"

Restart Service 

mkdir /tftpboot 
/etc/init.d/tftpd-hpa restart
/etc/init.d/apache2 restart
/etc/init.d/isc-dhcp-server restart

Get ARM64 pxeboot file 

some times the shim and grub2-efi would be erase, just visit mirror.centos.org and find new one than download it.

cd /tftpboot 
wget http://mirror.centos.org/centos/8-stream/BaseOS/aarch64/os/Packages/shim-aa64-15-15.el8_2.aarch64.rpm
wget http://mirror.centos.org/centos/8-stream/BaseOS/aarch64/os/Packages/grub2-efi-aa64-2.02-106.el8.aarch64.rpm
rpm2cpio shim-aa64-15-15.el8_2.aarch64.rpm | cpio -dimv
rpm2cpio grub2-efi-aa64-2.02-106.el8.aarch64.rpm | cpio -dimv
cp ./boot/efi/EFI/centos/shim.efi .
cp ./boot/efi/EFI/centos/grubaa64.efi .
chmod +rx *.efi

Configure grub.conf 

$ vim /tftpboot/grub.cfg

Following is grub.conf example 

set timeout=60
menuentry 'CentOS' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}

If your system support secure boot and you want to do it, please replace linux with linuxefi and replace initrd with initrdefi. 

Get CentOS 8 stream image and copy it to web server 

cd /var/www/html 
wget http://ftp.twaren.net/Linux/CentOS/8-stream/isos/x86_64/CentOS-Stream-8-x86_64-20220215-dvd1.iso
mkdir disk 
mount -o loop CentOS-Stream-8-x86_64-20220215-dvd1.iso disk
cp -a disk centos8 
umount disk 
rmdir disk

# copy boot file into tftp directory 

cd /tftpboot
cp /var/www/html/centos8/images/ . -a

After copy all file into http file directory, it can visit via browser on server with URL http://10.0.0.1/centos8/ to check whether file exist or not. 

NAT

Server should support NAT function to forward packet. 

assume interface connect to internet is wls3
Following is a sample command to enable Linux NAT function which can make client to access internet or intranet . 

$ echo 1 > /proc/sys/net/ipv4/ip_forward ==> enable it immediately 
$ iptables -t nat -A POSTROUTING -o wls3 -j MASQUERADE

Now, you can choose pxeboot on your ARM64 machine. if everything successful, you can see CentOS on your console if everything is ok . 

Known Issue

Stdin: Invalid argument

My system cannot boot into the Ubuntu install program for an unknown reason. After checking the system, it seems that the issue may be caused by an inability to access internet resources for some reason, even though the DNS and network appear to be fine. An alternative solution is to put the ISO on a website and assign the argument to the parameter. Here is an example using Ubuntu 22.04.

menuentry 'Ubuntu22.04 Net Install' {
linux ubuntu2204/vmlinuz ip=dhcp url=http://10.0.0.1/ubuntu-22.04.1-live-server-arm64.iso only-ubiquity
initrd ubuntu2204/initrd
}

Debug 

Wireshark is your good friend, use it. 

Ubuntu PXE Environment

It can download Ubuntu Netboot images from this url.
http://cdimage.ubuntu.com/netboot/

Choice what you want, for example, url below is Ubuntu 18.04 for ARM64 image
http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/
Ubuntu 20.04 Netboot file
http://ports.ubuntu.com/ubuntu-ports/dists/focal-updates/main/installer-arm64/current/legacy-images/netboot/netboot.tar.gz

Download netboot image to tftp directory, I prefer to put different version to different directory.

$ cd /tftpboot
$ wget http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/netboot.tar.gz
$ mkdir ubuntu1804
$ cd ubuntu1804
$ tar xvf ../netboot.tar.gz

Modify grub.cfg and add Ubuntu option to grub.cfg, save it, now, this server can also provide ubuntu net install

set timeout=60
menuentry 'CentOS8 Install' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}
menuentry 'Ubuntu18.04 Install' {
  linux ubuntu1804/ubuntu-installer/arm64/linux ip=dhcp
  initrd ubuntu1804/ubuntu-installer/arm64/initrd.gz
}

Ubuntu ISO environment

It can download iso file and use the iso file as install media.

in Linux, download Ubuntu 20.04 ARM64 iso file

$ wget https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.3-live-server-arm64.iso
$ mount -o loop ubuntu-20.04.3-live-server-arm64.iso disk
$ mkdir ubuntu2004
$ cp disk/casper ubuntu2004/ -a 
$ umount disk

Modify grub.cfg as below (not test)

set timeout=60
menuentry 'CentOS8 Install' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}
menuentry 'Ubuntu20.04 Install' {
  linux ubuntu2004/casper/vmlinuz ip=dhcp
  initrd ubuntu2004/casper/initrd
}

It also can use HWE kernel

menuentry 'Ubuntu20.04 Install' {
  linux ubuntu2004/casper/hwe-vmlinux ip=dhcp
  initrd ubuntu2004/casper/hwe-initrd
}

Note for GRUB Install Command

CentOS

svr=10.0.0.1; root=boot/centos8/images/pxeboot
linux (http,$svr)/$root/vmlinuz inst.stage2=http://$svr/boot/centos8 text
initrd (http,$svr)/$root/initrd.img

Ubuntu

svr=192.168.110.254; root=boot/ubuntu2004/casper
iso=ubuntu-20.04.3-live-server-arm64.iso
linux (http,$svr)/$root/vmlinuz url=http://192.168.110.254/boot/$iso only-ubiquity ip=dhcp ---
initrd (http,$svr)/$root/initrd

linux (http,$svr)/$root/hwe-vmlinuz url=http://192.168.110.254/boot/$iso only-ubiquity ip=dhcp ---
initrd (http,$svr)/$root/hwe-initrd

Fedora 36

Fedora’s setting is more closed to CentOS, due to CentOS might be dropped recently, so, moved to Fedora environment is necessary.

Get initrd/vmlinux image

Other parts like CentOS,
Download the PXE initrd and vmlinuz image, free.nchc.org.tw is Taiwan local Fedora server, you can replace it with local server.

mkdir -p /tftpboot/f36

cd /tftpboot/f36
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/os/images/pxeboot/initrd.img
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/os/images/pxeboot/vmlinuz

cd /var/www/html
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/iso/Fedora-Server-dvd-aarch64-36-1.5.iso
mkdir disk 
mount -o loop Fedora-Server-dvd-aarch64-36-1.5.iso disk
cp -a disk f36
umount disk

Grub Fedora Setting

Add following grub entry text into /tftpboot/grub.cfg, thus, it can insert a “Fedora36” menu on pxe boot.

menuentry 'Fedora36' {
  linux f36/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/f36/
  initrd f36/initrd.img
}

Ref. 

https://docs.centos.org/en-US/centos/install-guide/pxe-server/#sect-network-boot-setup-uefi
https://docs.centos.org/en-US/8-docs/advanced-install/assembly_preparing-for-a-network-install/
https://forums.opensuse.org/showthread.php/501075-error-cannot-find-commands-linuxefi-or-initrdefi

The post ARM64 Centos8/Ubuntu PXE Boot Install on Ubuntu Server appeared first on richliu's blog.

不要用 -mfpu 指令就可以了

ref.
-mfpu=neon-fp-armv8 and unrecognized command line option

The post ARM64 mfpu=neon-fp-armv8 and unrecognized command line option appeared first on richliu's blog.

Environment

Hardware : Ampere eMag Server with 128G RAM and 1T storage
Kubernetes master and nodes are running in QEMU image, so, we prepared three QEMU ARM64 ubuntu 18.04 images for kubernetes running. It also can use two VM to run this demo.

Prepare QEMU Image

It can download Ubuntu bionic cloud image from Taiwan mirror site

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img 

or international mirror site

https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.img

Install Necessary Package

Run following command to install necessary packages .

apt install -y qemu-system-arm bridge-utils cloud-image-utils git qemu-efi

Increase QEMU image size

qemu-img resize bionic-server-cloudimg-arm64.img +200G

Create UEFI bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Copy Image to three spare images

cp flash0.img flash0-1.img
cp flash0.img flash0-2.img 
cp flash0.img flash0-3.img 
cp flash1.img flash1-1.img
cp flash1.img flash1-2.img
cp flash1.img flash1-3.img
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-1.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-2.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-3.img 

Set up bridge and network devices

We already install bridge-utils before, now, it should configure bridge devices.

 mkdir -p /etc/qemu ; echo "allow br0" > /etc/qemu/bridge.conf 

Because there is only one network interface on machine, we should use this interface as bridge device, save text below (name nic-br.sh or what you like) and modify it to fit your environment.

ORIGNIC=enP6p1s0
ip addr flush $ORIGINIC
brctl addbr br0
brctl addif br0 $ORIGINIC
ifconfig br0 up
ifconfig br0 192.168.110.1 netmask 255.255.0.0
route add default gw 192.168.1.1
echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Configure Hugepage On Host Site (Option)

Due to previous research on dpdk runs on ARM64 server, so, configure hugepages here. But not necessary for run k8s environment.
You can ignore this part

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX="console=tty0 hugepagesz=1GB hugepages=32"

then run foloowing command to update grub.cfg

 update-grub 

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages 
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab 

Now, please reboot to make huagpage works.

Running QEMU

Following is QEMU script, save it (ex: named run-qemu.sh). It can be more simplification, but this script would be more understand what different.

IMAGE1=bionic-server-cloudimg-arm64-1.img
sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-1111-1111-1111-11111aaaaaaa \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-1.img -pflash flash1-1.img \
         -drive file=$IMAGE1,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0,mac=52:54:00:11:11:11 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest_log-1.txt 2>> guest_log-1.txt &
IMAGE2=bionic-server-cloudimg-arm64-2.img
sudo qemu-system-aarch64 -name vm2 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-2222-2222-2222-22222bbbbbbb \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-2.img -pflash flash1-2.img \
         -drive file=$IMAGE2,if=none,id=disk2 \
         -device virtio-blk-device,scsi=off,drive=disk2,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net2,ifname=tap1 \
         -device virtio-net-device,netdev=net2,mac=52:54:00:22:22:22 \
         -mem-path /mnt/hugepages \
         -serial telnet::9002,server,nowait > guest_log-2.txt 2>> guest_log-2.txt &
IMAGE3=bionic-server-cloudimg-arm64-3.img
sudo qemu-system-aarch64 -name vm3 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-3333-3333-3333-33333ccccccc \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-3.img -pflash flash1-3.img \
         -drive file=$IMAGE3,if=none,id=disk3 \
         -device virtio-blk-device,scsi=off,drive=disk3,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net3,ifname=tap2 \
         -device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \
         -mem-path /mnt/hugepages \
         -serial telnet::9003,server,nowait > guest_log-3.txt 2>> guest_log-3.txt &
sleep 5
brctl addif br0 tap0
brctl addif br0 tap1
brctl addif br0 tap2

There are some parameters need to understand in QEMU script.

-uuid d0000000-1111-1111-1111-11111aaaaaaa \

this will assign a unique uuid for each VM, k8s needs different VM run with different uuid and MAC address.
It also can use command “uuidgen” to generate different uuid for each VM, but here just for debug so assign a “easy to read” uuid.

-device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \

Assign a special mac address for VM, because for k8s internal communication, use the same mac address will cause problem, please configure different address before run qemu script. It not easy to modify it after first boot (with cloud image)

* Notice, first number for mac address cannot be odd , it would cause problem.

-mem-path /mnt/hugepages \

Tell VM to use hugepage, if were not configure host with hugepage, please remove this line.

Now, please run scripts

bash nic-br.sh
bash run-qemu.sh 

Now, it can use telnet to connect each VM’s console

telnet localhost 9001
# or 
telnet localhost 9002
# or 
telnet localhost 9003

After system booted/initialed, if you cannot login, please “killall qemu-system-aarch64” and follow this article Change Ubuntu Cloud Image Password to change all images’s password.

Somehow, for recent cloud image, it cannot modify password before run qemu script.

Install Kubernetes

There are a lot of article to talk about how to install on Ubuntu, I choice one for reference : Install and Deploy Kubernetes on Ubuntu 18.04 LTS

Below will shortly note install command here.

Install Docker and Kubeadm on every node

sudo apt update 
sudo apt -y upgrade 

# install and enable docker
sudo apt -y install docker.io 
sudo systemctl enable docker 
sudo systemctl start docker 

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add 
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update
sudo apt-get install kubeadm -y

Configure Network IP address (Optional)

Update to you network environment, here we used fix ip address.

Add new record to /etc/hosts

192.168.110.50     k8s-master
192.168.110.60     k8s-worker-node1
192.168.110.70     k8s-worker-node2

Please modify /etc/netplan/50-cloud-init.yam, remove dhcp and add fix address. remember, each node has different IP. Following is only for k8s-master.

network:
     ethernets:
         eth0:
             addresses: [192.168.110.50/16,]
             gateway4: 192.168.1.1
             nameservers:
                 addresses: [8.8.8.8,8.8.4.4]
             match:
                 macaddress: '52:54:00:11:11:11'
             set-name: eth0
     version: 2

Configure Host name

Run following command on master node

sudo hostnamectl set-hostname  k8s-master

Run following command on other nodes

sudo hostnamectl set-hostname   k8s-worker-node1 
sudo hostnamectl set-hostname   k8s-worker-node2

Disable swap (Option)

if image is not from cloud image, default Linux enabled swap function, k8s doesn’t support it, please disable it manually on all nodes and remove swap mount in /etc/fstab.

 sudo swapoff -a 

Configure Kubernetes

Now, we can initial Kubernetes. Run following command

 sudo kubeadm init --pod-network-cidr=10.244.0.0/16 

wait for few minutes, it will show some result like this

Run following command to assign current account has right to access k8s.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run this command on another node(k8s-worker-node1 and k8s-worker-node2).

sudo kubeadm join 192.168.110.50:6443 --token 46c1oj.i00vyfyn027pgew9 \
     --discovery-token-ca-cert-hash sha256:ed05198cc603ae009a862477bacef329a9e9a2bedc4a040bc4731e6d5cf4f48e

Run get nodes command, it will show current status, the master node status is NotReady, but don’t worry, it will become Ready after add pod and CNI (container networking interface) plugin.

ubuntu@k8s-master:~$ kubectl get nodes
 NAME         STATUS     ROLES    AGE     VERSION
 k8s-master   NotReady   master   3m42s   v1.16.0

This action is very import on ARM64 k8s platform, in other document, they told you use flannel or some other document told you used old flannel. all of them not work for me. Please instead flannel with weave net.
(Ref Weave Net for NetworkPolicy and CoreDNS is not starting on Ubuntu 18.04.Bionic Beaver #980)
Run following command to install weave net

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Use the following command to view the status of the network:

Get nodes again.

Great!! it works.

MicroK8s

If you think steps above are too complex and you would like to only run k8s on one machine, microk8s is a good friend for you, can run on ARM64 and very easy to use, just run command

sudo snap install microk8s

And just add a prefix “microk8s.” before k8s command, ex:

microk8s.kubectl get nodes

See, it’s very easy to use. right?

The post Install Kubernetes on ARM64 Ubuntu 18.04 (with QEMU) appeared first on richliu's blog.

Hardware : Ampere eMag Server with 128GB Memory
Network Card: Intel i210 Gigabit NIC Card * 2

HOST means commands and instructions on host machines
VM means commands and instructions inside VM

Prepare Environment

Apt source

First, please add deb-src to your database, edit file /etc/apt/source.list and removed all “#” before deb-src, then, run follow commands to update apt database (suppose all commands run with sudo or root in this article)

sed -i 's/# deb-src/deb-src/g' sources.list
apt update

Install Packages

It needs to install relative packages in host

sudo apt install -y dpdk dpdk-dev dpdk-doc 

Only needs to install on HOST.

sudo apt install -y qemu-efi bridge-utils

It will install all necessary dpdk packages in host.

Preload Module

It needs to preload vfio-pci modules by default, run following command

echo "vfio-pci" >> /etc/modules

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX=”console=tty0 hugepagesz=1GB hugepages=32″

then run foloowing command to update grub.cfg

update-grub

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab

Bind Network Card for DPDK

It wants to let DPDK use current network card, it needs to bind the network card with dpdk. dpdk-devbind command is a useful command.
List current command

 $ dpdk-devbind --status
 Network devices using kernel driver
 0001:01:00.0 'I210 Gigabit Network Connection 1533' if=enP1p1s0 drv=igb unused=vfio-pci
 0002:01:00.0 'I210 Gigabit Network Connection 1533' if=enP2p1s0 drv=igb unused=vfio-pci
 0006:01:00.0 'I210 Gigabit Network Connection 1533' if=enP6p1s0 drv=igb unused=vfio-pci Active 

There are three network card in the machine, we would like to bind 1:1:0.0 and 2:1:0.0 to vfio-pci driver. please used following command to bind those network cards.

dpdk-devbind -b vfio-pci 0001:01:00.0 0002:01:00.0

and check status again

 $ dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0001:01:00.0 'I210 Gigabit Network Connection 1533' drv=vfio-pci unused=igb
0002:01:00.0 'I210 Gigabit Network Connection 1533' drv=vfio-pci unused=igb

Network devices using kernel driver
============================================
0006:01:00.0 'I210 Gigabit Network Connection 1533' if=enP6p1s0 drv=igb unused=vfio-pci Active 

Both network cards would be under Network devices using DPDK-compatible driver .

Prepare QEMU environment

Set up bridge devices

We already install bridge-utils before, now, it should configure bridge devices.

mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf

Because there is only one network interface on machine, we should use this interface as bridge device, save text below and modify it to fit your environment.

 ORIGNIC=enP6p1s0

 ip addr flush enP6p1s0


 brctl addbr br0
 brctl addif br0 enP6p1s0
 ifconfig br0 up
 ifconfig br0 192.168.110.1 netmask 255.255.0.0
 route add default gw 192.168.1.1


 echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Run script, command brctl show will show how whether it configure property or not.

Prepare QEMU image

Ubuntu 18.04 cloud image is here

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img

and please refer this article to change cloud image’s password
Change Ubuntu Cloud Image Password, before use it, don’t forget to increase cloud image size. 20G is enough for test.

qemu-img resize bionic-server-cloudimg-arm64.img +20G

Next step is to create bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Run Qemu

Here is qemu command, save text below and run it

 IMAGE=bionic-server-cloudimg-arm64.img
 sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0.img -pflash flash1.img \
         -device vfio-pci,host=0001:01:00.0 \
         -device vfio-pci,host=0002:01:00.0 \
         -drive file=$IMAGE,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest1_log.txt 2>> guest1_log.txt &
 sleep 5
 brctl addif br0 tap0 

-m 8192 : use 8G RAM for VM
–mem-path will use /mnt/hugepages for VM memory

it will create a network device tap0 for bridge device interface between VM and host. and create a telnet server on port 9001, it can use “telnet 9001” to get console, I would like to suggest to use ssh if it can access console.

Remove cloudinit service

it needs to remove cloud-initramfs-copymods, it will copy modules from ramdisk to harddriver, and cloud-init, it’s for some cloud service like aws(?)

removed them would help system runs better.

apt remove cloud-initramfs-copymods cloud-init 

Disable Services

After boot, it needs to disable some default service to improve speed, here is some services can be disabled. (needs run those command with root)

systemctl disable ufw
systemctl disable lxcfs
systemctl disable atd
systemctl disable cron
systemctl disable irqbalance
systemctl disable apparmor
systemctl disable apport
systemctl disable ebtables
systemctl disable grub-common
systemctl disable unattended-upgrades
systemctl disable iscsid

Most import service is irqbalance, without disable it, performance will be drop down.

Now, it can reboot VM now.

Prepare QEMU environment

Prepare QEMU environment and prepare host almost the same, only differences are

1. in QEMU, apt install doesn’t needs to install qemu-efi and bridge-utils
2. in QEMU, before call dpdk-devbind command, it needs to configure enable_unsafe_noiommu_mode for ARM64 platform
3. Hugepage size is different , in HOST it uses 32G, in VM we uses 4G

Enable unsafe noiommu

echo 1 > /sys/module/vfio/parameters/enable_unsafe_*

To permanently enable the no-IOMMU mode, add it to modprobe.d: (ref)

echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf

It also needs to reserve hugepages memory space for Linux kernel for VM, it reserves 4 pages and each pages 1GB, total 4GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=4 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX=”console=tty0 hugepagesz=1GB hugepages=4″

then run foloowing command to update grub.cfg

update-grub

Run DPDK l3fwd

Before run l3fwd, we need to recompile it.

. /usr/share/dpdk/dpdk-sdk-env.sh
make -C /usr/share/dpdk/examples/l3fwd

Bind network interface

dpdk-devbind -b vfio-pci  enp0s1
dpdk-devbind -b vfio-pci  enp0s2

Run l3fwd example

cd /usr/share/dpdk/examples/l3fwd/build
./l3fwd -c 1 -- -p 0x3 --config="(0,0,0),(1,0,0)"

Multi Queue example

./l3fwd -l 1,2,3,4 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2),(0,1,3),(1,1,4)"

Other

There is good tool to get hugepage information

apt install hugepages
hugeadm --pool-list

The post Ubuntu 18.04 ARM64 DPDK in VM (QEMU) appeared first on richliu's blog.

Ubuntu

It can refer to those three articles to create a new cloud image from Ubuntu server image, remember to change ARM64 image to x86_64 image .

Change Ubuntu Cloud Image Size (if you choice ubuntu 16.04 as your image)
Change Ubuntu Cloud Image Password
Ubuntu Cloud Image Taiwan Mirror Site

Install Qemu and setup bridge devices

apt install -y ovmf qemu bridge-utils
mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf
echo "allow virbr0" >> /etc/qemu/bridge.conf

Write this script to a file and run it.

SERVERFILE=xenial-server-cloudimg-amd64-uefi1.img
VERSION=`uname -r`

sudo qemu-system-x86_64 -name vm1 \
        -machine pc-i440fx-xenial,usb=off \
        -cpu qemu64 -m 8192 \
        -smp 12,sockets=1,cores=12,threads=1 \
        -nographic -nodefaults \
        -bios /usr/share/ovmf/OVMF.fd \
        -virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 \
        -drive file=$SERVERFILE  \
        -net nic,macaddr=$macaddr \
        -net tap,ifname=tap0 \
        -serial telnet::9001,server,nowait > guest1_log.txt 2>&1 &

sleep 5

brctl addif br0 tap0

it can use command
qemu-system-x86_64 –machine help
and
qemu-system-x86_64 –cpu help
to get all machine and cpu options.

it can change cpu and machine type here.

network would be bridge on tap0

below is good option to share folder between host and VM, below command will share /home/work with vm.

option 
--virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 

it can put this line to /etc/fstab to mount host’s folder to somewhere, in this case, just put the same directory like host does.

host0   /home/work   9p      trans=virtio,version=9p2000.L,user,umask=000   0 0

CentOS

CentOS document was written on 2021, but Ubuntu document on 2019, there is some difference between it. but that’s ok, I believe both ways are works.

Compile QEMU

CentOS default application doesn’t support run x86_64 code on ARM64 server, it needs to compile manually.

Please download qemu and compile it.

# Install CentOS Packages
yum config-manager --set-enabled powertools
yum install -y xorg-x11-xauth python3 ninja-build
yum install -y glib2-devel libmount-devel
yum install -y git glib2-devel libfdt-devel pixman-devel zlib-devel

# Download QEMU Source Code
wget https://download.qemu.org/qemu-5.2.0.tar.xz
# Decompress it
tar xvf qemu-5.2.0.tar.xz
# 
cd qemu-5.2.0
./configure --target-list="x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user"
make -j `nproc`
make install 

Then, qemu will be installed on /usr/local/bin

Download UEFI Image

It needs UEFI Image for x86_64 image boot up. It needs OVMF’s tianocore (Open Source UEFI BIOS) to boot up. And we can download kraxel as prebuild images.

# Install Kraxel into CentOS repos
cd /etc/yum.repos.d/
wget https://www.kraxel.org/repos/firmware.repo
yum install edk2.git-aarch64 edk2.git-ovmf-x64
cd <your vm directory>
cp /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd .
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd .

Prepare Bridge Devices

It needs bridge devices to connect to internet, default qemu bridge device name is virbr0 , due to CentOS doesn’t support brctl, follow this document(3 Ways to Create a Network Bridge in RHEL/CentOS 8) to create a CentOS bridge devices. Below is my experiment script to create bridge device, this setting will keep in system, reboot will not be reset.

nmcli conn add type bridge con-name virbr0 ifname virbr0

killall dhclient
ip addr flush enP7p2s0
nmcli conn modify virbr0 ipv4.addresses '192.168.3.191/16'
nmcli conn modify br0 ipv4.gateway '192.168.1.1'
nmcli conn modify br0 ipv4.dns '8.8.8.8'
nmcli conn modify br0 ipv4.method manual
nmcli conn add type ethernet slave-type bridge con-name bridge-br0 ifname enP7p2s0 master virbr0
nmcli conn up virbr0
nmcli conn up bridge-br0

nmcli conn show  --active

# Allow QEMU to access virbr0
mkdir -p /etc/qemu
echo "allow virbr0" >> /etc/qemu/bridge.conf

Run Qemu

It can refer above to prepare cloud image, I prefer to use ubuntu cloud image, but have not try CentOS image.

Remember Ubuntu 18.04 image won’t auto increase image size, please follow this Change Ubuntu Cloud Image Size to modify image size

qemu-system-x86_64 -m 8192 -smp 16 -nographic \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE-pure-efi.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS-pure-efi.fd \

        -hdc bionic-server-cloudimg-amd64.img \
        -netdev bridge,br=virbr0,id=net0 \
        -device virtio-net,netdev=net0,mac=00:11:22:33:44:55 \
        -serial telnet::9001,server,nowait > vmlog.txt 2>> vmlog.txt &

x86_64 application

Sometimes we don’t need to run whole VM to run some application just for x86_64, it can run user mode application directly. I am a lazy guy, so just reuse cloud image as x86_64 Linux library.

mkdir -p /home/amd64
qemu-img convert  bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.raw
losetup /dev/loop0 bionic-server-cloudimg-amd64.raw
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /home/amd64

Umount Image

umount /home/amd64
losetup -d /dev/loop0

Run x86_64 only application

qemu-x86_64 -L /home/amd64/ bootutil64e
qemu-x86_64 -L /home/amd64/ nvmupdate64e -l

Unsolved Problem

Cannot find way to update INTEL NIC’s NVM

Tried vfio and application mode, vfio cannot pass iommu to host, it will fail.

User space application will cause mmap fail.

If you know how to update Intel NIC’s NVM, welcome to leave a message for me.

QEMU –static problem

Add –static parameter will cause some problem, cannot solve it right now.

./configure –target-list=”x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user” –static
===

ERROR: sizeof(size_t) doesn’t match GLIB_SIZEOF_SIZE_T.

    You probably need to set PKG_CONFIG_LIBDIR

    to point to the right pkg-config files for your    build target

The post QEMU Run x86_64/AMD64 VM on ARM64/AARCH64 appeared first on richliu's blog.

應該只有影響 ocata

Ref.
OpenStack KVM

[ocata] unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor

程式解法
aarch64: set default cpu_mode to host-passthrough

The post unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor appeared first on richliu's blog.

no valid host was found. There are not enough hosts available.
Code
500
Details
File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", 
line 526, in build_instances context, request_spec, filter_properties) File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", 
line 597, in _schedule_instances hosts = self.scheduler_client.select_destinations(context, spec_obj) File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", 
line 371, in wrapped return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", 
line 51, in select_destinations return self.queryclient.select_destinations(context, spec_obj) File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", 
line 37, in __run_method return getattr(self.instance, __name)(*args, **kwargs) 
File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", 
line 32, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj) 
File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", 
line 129, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", 
line 169, in call retry=self.retry) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", 
line 97, in _send timeout=timeout, retry=retry) 
File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 458, in send retry=retry) File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 449, in _send raise result

Possible reason

* if using cloud uefi image, it needs to install qemu-efi package in computer node, don’t know why it won’t install it automatically .
$ apt install qemu-efi
* when add new UEFI image, it needs to add new meta data parameter in Horization
hw_firmware_type=uefi

if see

2017-08-01 14:02:13.612 2409 ERROR nova.compute.manager [instance: b334f0b3-ca9a-4f9a-b488-532dce00265a] libvirtError: unsupported configuration: logfile not supported in this QEMU binary
2017-08-01 14:02:13.612 2409 ERROR nova.compute.manager [instance: b334f0b3-ca9a-4f9a-b488-532dce00265a]
# Use the "logd" backend for handling stdout/stderr from QEMU processes.

Reason
* Libvirt later version support new function, logd, it’s a new console log function, but ARM64 doesn’t support it. workaround solution is to disable it

Edit file /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py, find

MIN_LIBVIRT_VIRTLOGD = (1, 3, 3)
MIN_QEMU_VIRTLOGD = (2, 7, 0)

Replaced it with

MIN_LIBVIRT_VIRTLOGD = (9, 9, 9)
MIN_QEMU_VIRTLOGD = (9, 9, 9)

it can avoid to call libvirt logd function.

rm -f /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.pyc, make sure it will call .py
then, reboot service (or system)

Ref.
http://dozencloud.org/

The post No valid host was found. There are not enough hosts available appeared first on richliu's blog.

安裝

$ sudo apt install virt-manager

跑一個簡單的 aarch64 image

$ sudo virt-install --name ubuntu-aarch64 --ram 2048 --arch aarch64 --boot uefi \
--disk /home/test/ubuntu.img,device=disk,bus=virtio \
--boot machine=virt,kernel=trusty-kernel,kernel_args="console=ttyAMA0 root=/dev/vda1 rw kgdboc=ttyAMA0,115200" \
--network bridge:virbr0 \
--os-type=linux \
--nographics \
--noautoconsole

ubuntu.img 是預先準備好的 ubuntu cloud image, 16.04 才可以支援比較多的 ARM64 平台.
14.04 支援的就少很多.

以上的參數比較重要的是 –noautoconsole, 這樣執行時就不會等在那邊, 會自動啟動.

–noautoconsole will install vm to libvirt without connect to console

—- 2017/03/10 —-
也可以用 virt-install + netboot install

先建立 image

$ qemu-img create -f qcow2 ./name.qcow2 30G 

再執行 virt-install , 目前測過 14.04 和 16.04 , 只有 16.04 成功, 猜想是 ports 那邊少了什麼東西

$ sudo virt-install --name ubuntu-aarch64 --ram 4096 --vcpus 4 --arch aarch64 \
        --boot uefi \
        --disk /home/richliu/disk.qcow2,device=disk,bus=virtio \
        --network bridge:virbr0 \
        --os-type=linux \
        --nographics \
        --location 'http://ports.ubuntu.com/dists/xenial/main/installer-arm64/'

這個參數可以順利安裝完, 然後再啟動系統就好

—- 2017/04/26 —-
如果是要裝 ARM64 Trusty-Update 的 Netboot kernel . 要動一下小技巧
1) 先在 host VM 上安裝 apache2
2) 到 /var/www/html 上 mirror http://ports.ubuntu.com/dists/trusty-updates/main/installer-arm64/current/images/ 下的
* MANIFEST
* MANIFEST.udebs
* xenial-generic/netboot =>

目錄要照著放, 所以會是

$ cd /var/www/html
$ mkdir -p dists/trusty-updates/main/installer-arm64/current/images/
$ cd dists/trusty-updates/main/installer-arm64/current/images/
$ lftp http://ports.ubuntu.com/dists/trusty-updates/main/installer-arm64/current/images/
[download blah blah]
$ mv xenial-generic/netboot .

然後 –location 指到 http://[ARM64 Server IP] 這樣就可以了, ex:

virt-install --name ubuntu-aarch64 --ram 8192 --vcpus 4 --arch aarch64 \
--boot uefi \
--disk /root/disk.qcow2,device=disk,bus=virtio \
--network bridge:virbr0,model=virtio \
--os-type=linux \
--nographics \
--location http://10.102.81.16/dists/trusty-updates/main/installer-arm64

對了, 這邊的 model=virtio 是強迫 network interface 不要用 vhost, 而是用 virtio

$ virsh start ubuntu-aarch64
$ virsh connect ubuntu-aarch64

ref.
KVM Guest virt-install examples with virtio drivers
Installing Virtual Machines with virt-install, plus copy pastable distro install one-liners

The post aarch64/ARM64 virt-install 啟動方法 appeared first on richliu's blog.