In this article, I will skip other steps such as installing Ollama.

System Specifications:

• OS: Ubuntu 22.04 on an ARM64 virtual machine
• CPU: Ampere Altra
• DRAM: 32GB (allocated to the VM, though smaller models do not require this much memory)

At the time of writing, Infinity does not support the ARM64 platform, but that is not an issue since we do not need it. RAGFlow uses Elasticsearch as its default AI database engine, and Elasticsearch does support the ARM64 platform.

Building RAGFlow on ARM64

Build ragflow command [ref. Build a RAGFlow Docker Image]

git clone https://github.com/infiniflow/ragflow.git
cd ragflow/
docker build --build-arg LIGHTEN=1 -f Dockerfile -t <username>/ragflow:<version> .

docker build --build-arg LIGHTEN=1 -f Dockerfile -t user/ragflow:v0.16.0 .

If everything is set up correctly, the Docker images will display an output similar to the following:

docker images
REPOSITORY                                      TAG                            IMAGE ID       CREATED         SIZE
user/ragflow                                    v0.16.0                        8a71ac9cb2fa   3 hours ago     5.73GB

Note that this image does not include pre-built models. However, models can be installed separately using other methods.

Running RAGFlow

Before running the docker-compose command, assign the RAGFlow Docker image name to the docker-compose configuration.

export RAGFLOW_IMAGE=user/ragflow:v0.16.0
docker compose -f docker/docker-compose.yml up -d

If everything is set up correctly, you should see a message similar to the following:

[+] Running 10/10
  Network docker_ragflow      Created                                                                         0.2s
  Volume "docker_minio_data"  Created                                                                         0.0s
  Volume "docker_redis_data"  Created                                                                         0.0s
  Volume "docker_esdata01"    Created                                                                         0.0s
  Volume "docker_mysql_data"  Created                                                                         0.0s
  Container ragflow-mysql     Healthy                                                                        21.7s
  Container ragflow-minio     Started                                                                         1.1s
  Container ragflow-redis     Started                                                                         1.1s
  Container ragflow-es-01     Started                                                                         1.1s
  Container ragflow-server    Started                                                                        22.3s

Downloading Models with Ollama

Ollama needs to download certain models for system functionality. Here’s an easy way to download the models—you can choose the ones you prefer.

ollama pull deepseek-r1:14b
ollama pull deepseek-r1:32b
ollama pull smartcreation/bge-large-zh-v1.5:latest

After downloading the Ollama models, you need to add them in the “USER → Model Providers → Add Ollama Model” section. Below is an example of how to add the Deepseek model. Once you understand the process, you can add more models as needed.

At a minimum, you will need one chat model and one embedding model.

After pressing OK, you can configure the System Model Settings, which should look something like this:

Using RAGFlow

Now, you are ready to use the Knowledge Base and Chat features to explore RAGFlow.

For example, you can ask the engine to generate an ARM SIMD sample code based on the ARMv8 documentation.

Conclusion

RAGFlow is an easy-to-use RAG framework. Although it does not provide a default ARM64 image, you can still build it from source to obtain a functional version of RAGFlow.

The post Ragflow on ARM64 appeared first on richliu's blog.

no-dangling-assignment-gsl -std=c++20 -Wno-trigraphs -gsimple-template-names -fno-exceptio00:01:51 [472/19443]
nc++ -isystem../../third_party/libc++/src/include -isystem../../third_party/libc++abi/src/include -fvisibility
-inlines-hidden -O2 -pipe -march=armv8.2-a -Wno-unknown-warning-option -c ../../third_party/libyuv/source/comp
are_neon64.cc -o obj/third_party/libyuv/libyuv_neon/compare_neon64.o
../../third_party/libyuv/source/compare_neon64.cc:175:8: error: instruction requires: dotprod
  175 |       "udot        v4.4s, v0.16b, v6.16b         \n"
      |        ^
<inline asm>:14:1: note: instantiated into assembly here
   14 | udot        v4.4s, v0.16b, v6.16b
      | ^
../../third_party/libyuv/source/compare_neon64.cc:176:8: error: instruction requires: dotprod
  176 |       "udot        v5.4s, v1.16b, v6.16b         \n"
      |        ^
<inline asm>:15:1: note: instantiated into assembly here
   15 | udot        v5.4s, v1.16b, v6.16b
      | ^
../../third_party/libyuv/source/compare_neon64.cc:204:8: error: instruction requires: dotprod
  204 |       "udot        v4.4s, v0.16b, v0.16b         \n"
      |        ^
<inline asm>:10:1: note: instantiated into assembly here
   10 | udot        v4.4s, v0.16b, v0.16b
      | ^
../../third_party/libyuv/source/compare_neon64.cc:205:8: error: instruction requires: dotprod
  205 |       "udot        v5.4s, v1.16b, v1.16b         \n"
      |        ^
<inline asm>:11:1: note: instantiated into assembly here
   11 | udot        v5.4s, v1.16b, v1.16b
      | ^
4 errors generated.
[10940/63891] aarch64-unknown-linux-gnu-clang++-18 -MD -MF obj/third_party/libyuv/libyuv_neon/rotate_neon.o.d

Seems because some -march/-mcpu/-mtune drop ‘dotprod’ parameters, it needs to add the extension manuall. ‘+dotprod’.

Edit the file /etc/portage/make.conf, and change the COMMON_FLAGS to

COMMON_FLAGS="-mcpu=neoverse-n1+crc+crypto+ssbs+dotprod -O2 -pipe"

It can use the following command to get current CPU flag

gcc -v -E -x c /dev/null -o /dev/null -march=native -mcpu=native 2>&1 | grep /cc1

Thanks Gentoo user zukunf to provide those information.

The post Gentoo ARM64 Altra Compile chromium Issue appeared first on richliu's blog.

Qualcomm’s new desktop SoCs might change this landscape. However, we can still leverage the Ampere Altra platform to run LLMs.

Since this is a personal blog, most articles here are my personal notes. I’ll keep them updated as my understanding evolves.

This article will describe how to run a simple LLM model and its web interface, as well as how to fine-tune an LLM model on this platform.

Before running the commands, you’ll need to install the Nvidia driver. Refer to this article, “[How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform]”, for instructions on installing the Nvidia driver and Docker driver.

Ollama and Open-Webui don’t require a GPU. However, having a GPU is beneficial. Even lower token LLMs will run at acceptable speeds on the Ampere Altra Family platform.

This article’s hardware setup is as follows:

• CPU : Ampere Altra Family
• Board: AsRock ALTRAD8UD
• GPU: Nvidia RTX 4080

Use current LLM model with GUI

At this point in the process, using Ollama and Open-Webui is a straightforward approach for running LLMs. Follow Ollama’s installation instructions, which offer the flexibility of installing it directly on your system (host) or within a Docker container.

curl -fsSL https://ollama.com/install.sh | sh

By default, the Ollama service listens on localhost (127.0.0.1). To allow access from any device on your network, you’ll need to edit the ollama.service file.

In the [Service] section, add the following line:

Environment="OLLAMA_HOST=0.0.0.0:11434"

For Open-Webui, we recommend using Docker for a simpler solution. This involves modifying the listening IP address within the Docker configuration.

Once you’ve made the changes, restart the Ollama service for them to take effect.

systemctl daemon-reload
systemctl restart ollama

Run the ollama command and try it

# ollama run llama2
>>> who are you

I'm LLaMA, an AI assistant developed by Meta AI that can understand and respond to human input in a conversational manner.
I'm here to help you with any questions or topics you'd like to discuss! Is there something specific you'd like to talk
about or ask?

>>> 

If doesn’t work, try to restart ollama and try again.

Big LLM

If choose some big LLM like LLaMA2-70b or Qwen

Open-Webui

Following command is to run the Open-Webui docker image on host, 3000 is host port for Open-Webui. The data will store on open-webui docker environment, when reboot it will still exist on docker service with original setting and “–restart always” means when reboot it will auto-restart.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

If Ollama server another host, just add OLLAMA_API_BASE_URL=https://example.com/api to the docker command,

docker run -d -p 3000:8080 -e OLLAMA_API_BASE_URL=https://example.com/api -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Upon login, you’ll need to provide an email and password. Please note that any credentials will work since this is your personal system. Once logged in, you can also edit the OLLAMA_API_BASE on the settings page.

Now, it connected from Open-Webui to ollmam backend, and we have a running sytsem on our machine.
Run sample chat here and choice llama2:latest as LLM model.

LLM Fine-Turning – LLaMA-Factory

Fine-tuning LLMs ideally requires a GPU card for better performance. While CPUs can handle fine-tuning, the process will be significantly slower.

Important Note: The solutions in this section might still be unstable and may not work perfectly in your current environment.

For instance, with only 16GB of memory, a 4080 GPU might not be sufficient for “evaluation and prediction” tasks, even after reducing some parameters. This still can lead to CUDA out-of-memory errors.

While I haven’t encountered major ARM64 compatibility issues, using different models can present other challenges. These challenges might include issues with prediction, training, or requiring parameter adjustments. Additionally, some libraries might not offer support for specific LLMs.

Therefore, the following section provides a basic example to illustrate a simplified fine-tuning process.

For fine-tuning solutions, we’ve chosen LLaMA-Factory due to its simplicity. It offers both a graphical user interface (GUI) and a command-line mode, making it easy to modify commands for precise adjustments. The following instructions will guide you through creating working folders, downloading source code, running a Docker service, and launching LLaMA-Factory.

# Create work folder
mkdir -p /nvme/model 
cd /nvme 
git clone https://github.com/hiyouga/LLaMA-Factory.git

# Run docker, it will use host GPU and map the folder into docker.
sudo docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -it --rm \
     -v /nvme:/nvme \
     -p 7860:7860 \
     nvcr.io/nvidia/pytorch:24.01-py3

# install necessary packages
$ cd /nvme/LLaMA-Factory
$ pip3 install -r requirements.txt
$ pip3 install tiktoken transformers_stream_generator

# run the LLaMA-Factory
$ CUDA_VISIBLE_DEVICES=0 python src/train_web.py

Now, it can acces the webgui via http://host IP:7860.

It can create another Docker window to run command on the LLaMA-Factory docker image.

docker exec -it <docker name> /bin/bash 

for download LLMs, you need to use git download from huggingface, ex:

# LLama 2 need username and token(not passowrd) to download it. 
git clone https://huggingface.co/meta-llama/Llama-2-7b-hf

# Qwen
git clone https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat
git clone https://huggingface.co/Qwen/Qwen1.5-1.8B

Pre-Training

Model name, choice the LLMs name , like Qwen1.5-1.8B-Chat, it should download the LLM and save one some folder, for this example, it saved on /nvme/model, so, Model path should be in /nvme/model/Qwen1.5-1.8B-Chat.

First time, I suggest to use small LLM for fine-tuning, Qweb1.5-0.8B might have issue, will suggest from 1.5-1.8B to start.

Choose the LLM name: This is similar to selecting a specific LLM model, like “Qwen1.5-1.8B-Chat”. The instructions will indicate your LLM folder. For this example, the model would be saved in /nvme/model/Qwen1.5-1.8B-Chat.

Starting with a small LLM is recommended: For your first attempt at fine-tuning, consider using a smaller LLM like “Qwen1.5-1.8B”. It’s possible that “Qweb1.5-0.8B” might cause issues, so we recommend starting with models in the 1.5-1.8B range.

Here are the fine-tuning options you can adjust in LLaMA-Factory:

Dataset: You can add datasets to this mode. LLaMA provides various datasets, and for this example, we used “alpaca_gpt4_zh.”
Gradient accumulation: This is currently set to 4.
Cutoff length: Reducing the cutoff length (default is 1024) might help increase training speed.
Compute type: The compute type depends on your environment. In this case, fp16 works, but bp16 doesn’t sometimes.
Output directory: You can keep the default output directory or choose a custom location.
Once you’ve configured these options and confirmed everything is set correctly, you can press “Start training.” Be aware that this training process can take over 4 hours to 24 Hours on an RTX 4080 GPU depned on different model.

Evaluate & Predict

Next step is evaluate and predict, Adpater path just fill previous steps’ output dir. Dataset is the same.

When everything is ready, press Start. Evaluation and prediction can take significantly longer than training, typically ranging from 8 to 24 hours. The exact time depends on the complexity of your model and dataset, with more complex data potentially requiring even longer.

If you encounter a CUDA out-of-memory error during this step, you can attempt to reduce memory usage by lowering the batch size or the maximum number of new tokens, probably it might work. [ref]

Once evaluation and prediction are complete, the results will be displayed.

Test Chat Model

You can test chat result by load module.

Export model

Remember to fill export dir, in here I used “/nvme/newmodel/Qwen1.5-1.8B-Chat”

cd /nvme/newmodel 
python3 /nvme/llama.cpp/convert-hf-to-gguf.py Qwen1.5-1.8B-Chat --outfile test.gguf

# Use llama.cpp to test this model 
/nvme/llama.cpp/build/bin/main -m test.gguf -p "who are you?"

# Command mode
/nvme/llama.cpp/build/bin/main -m test.gguf -ins

Create a file named “Modelfile”, and fill this

FROM /nvme/newmodel/test.gguf

Now, run ollama to include this LLM.

ollama create test -f Modelfile
ollama run test "who are you?"
====
 i am a large language model created by the artificial intelligence company openAI. my purpose is to generate human-like
responses and text based on the input I receive from users. can I help you with anything today? #openai

#chatbot
Sure, I'm here to assist you! How can I help you today? Is there something specific you'd like to talk about or ask me?
#openaiChatBot不断地尝试。 #AI #MachineLearning

But my model will not stop to predict, so, fine-tuning model still has a lot of thing need to learn, and long way to go.

To be continue.

The post LLM Chat WebGUI and Fine-Turning on Ampere Altra ARM64 Platform appeared first on richliu's blog.

In previous article on How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform , we installed the Nvidia driver and ran PyTorch on the platform. Now, this platform has Nvidia RTX gaming card, can we use it to play game?

The answer is yes, there are some windows x86 or x64 emulators on Linux, but the most stable solution comes from Ampere’s official release solution. Other emulators might not work as well on this combination, such as FEX, but I don’t know whether it is a driver issue or another issue.

Basically, following Ampere’s documentation makes it easy to install the driver and play games. Let’s combine both documents into one and simplify it in this article.

Hardware
CPU: Ampere Altra
Board: AsRock ALTRAD8UD
GPU: Nvidia RTX 4080

Reference Ampere’s document
NVIDIA GPU Accelerated Linux Desktop on Ampere
Steam-on-Ampere

Install Nvidia Driver

All the conditions are similar to those outlined in the article How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform, using Ubuntu 22.04 with a general kernel (not the HWE kernel, which Nvidia may address in the future).

I haven’t utilized the original document’s script; instead, I’ve copied some necessary commands from it.

sudo apt-get install build-essential
# Install Nvidia driver here
sudo rmmod nouveau
cat << EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
EOF

sudo nvidia-xconfig -a --cool-bits=31 --allow-empty-initial-configuration

# Install Desktop, personally I like kde. 
sudo apt install kde-plasma-desktop
# or it can install regular ubuntu desktop 
# sudo apt install ubuntu-desktop

Download Nvidia ARM64 (aarch64) Display Driver ; the current version, 535.154.05, is acceptable. After installing the Nvidia driver, simply reboot the system. You can then use the following command to check the driver status.

nvidia-smi

Disable aspm from kernel

I didn’t apply this option. Most of the time, it’s only necessary to do so if the system encounters the massive AER error issue. Here’s an example command from the original document:

sudo nano /etc/default/grub
# Add pcie_aspm=off to kernel parameters
# GRUB_CMDLINE_LINUX_DEFAULT="pcie_aspm=off"
sudo update-grub

Install Box86

Following commands show how to install Box86

sudo wget https://itai-nelken.github.io/weekly-box86-debs/debian/box86.list -O /etc/apt/sources.list.d/box86.list
wget -qO- https://itai-nelken.github.io/weekly-box86-debs/debian/KEY.gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/box86-debs-archive-keyring.gpg
sudo dpkg --add-architecture armhf
sudo apt update
sudo apt install box86:armhf -y

Install Box64

Following command show how to install Box64, but original instruction have update , a package name was changed.

sudo wget https://ryanfortner.github.io/box64-debs/box64.list -O /etc/apt/sources.list.d/box64.list
wget -qO- https://ryanfortner.github.io/box64-debs/KEY.gpg | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/box64-debs-archive-keyring.gpg
sudo apt update 
sudo apt install box64-arm64 -y
sudo systemctl restart systemd-binfmt

Installing Steam

Because I am using the Ubuntu 22.04 ARM64 server version with minimal, so, some packages might not be installed, Before Install Steam, install necessary packages

sudo apt install lsof

sudo apt install zenity
sudo apt install alsa

Use the following to install steam.

git clone https://github.com/ptitSeb/box86
cd box86
./install_steam.sh

Now, it can use steam now.

/usr/local/bin/steam

Okay, log in and enjoy it. However, in my experience, when installing other emulators, it might damage the box86+Steam setup, necessitating a system reinstallation to fix it.

Furthermore, the game’s performance seems not as good as expected, possibly due to my choice of Vampire Survivors and enabling high performance

Xbox JoyStick controller

I have an Xbox controller, and ALTRAD8UD doesn’t have Bluetooth capability, so I cannot use wireless conntions. However, it can connect via USB.

There are several Xbox drivers available for Linux. After testing, I found that xpadano is compatible with this platform. Simply follow their instruction guide to install the Xbox driver.

# Install DKMS package
sudo apt-get install dkms linux-headers-`uname -r`

# download source code
git clone https://github.com/atar-axis/xpadneo.git
cd xpadneo
# use DKMS to compile the driver
sudo ./install.sh

The Xbox controller driver will be installed automatically, and it works well in my tests. You can install the jstest package to perform a joystick test.

The Xbox controller functions seamlessly on Steam and Vampire Survivors. I can easily use the Xbox controller to play Vampire Survivors.

The post Play Game On Ampere Altra ARM64 Platform appeared first on richliu's blog.

In this article, it will walk through the process of installiung Stable Diffusion WebUI on ARM64 + Nvidia RTX platform.
For such as hardware combinarion, it needs to solve some compatiability issues, so, here will describe how to install it and fix those issues.

Platform

CPU: Ampere Altra ARM64 CPU
Board: AsRock ALTRAD8UD
OS: Ubuntu 22.04

The platform is the most powerful ARM64 platform for a homelab that we can obtain today. There are some server models available, but they come with high-speed cooling fans that are not suitable for home use.

Nvidia has provided comprehensive library support for the ARM64 platform, including drivers, CUDA, and cuDNN. However, despite these efforts, there are still gaps in support. Even Nvidia’s Jetson product line lacks adequate support for ARM64 platforms in certain applications such as PyTorch. While there exists an individual PyTorch accelerator package for direct download on x64 platforms, the same cannot be said for ARM64 platforms. Although there is technically one package available, its availability is severely limited, rendering it practically useless.

The only viable solution to overcome these limitations is to utilize Nvidia’s Docker service to install Stable Diffusion.

Installation

Install Ubuntu 22.04 without selecting the HWE (Hardware Enablement) support. It appears that the current Nvidia driver version has issues, leading to potential soft-lock bugs. A general kernel version (5.15) is preferable for stability.

Next, install the necessary Ubuntu packages.

sudo apt-get install build-essential

Download Nvidia ARM64 (aarch64) Display Driver , current version is ok, when I write this article, current version is 535.154.05

Docker

Install Docker by following the Docker guide Install Docker Engine on Ubuntu,”. Also provide the necessary installation steps here. If already installed Docker or if these steps don’t work, please refer to the original website for updated instructions.

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Nvidia Container Toolkit

Follow the instructions for installing the NVIDIA Container Toolit. It can modify the Docker configuration using this toolkit. Alternatively, modifying the daemon.json file might achieve the same result, although I haven’t personally tested it. Below are the instructions:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

It can check the file /etc/docker/daemon.json file, the configuration should be like this.

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

Run Nvidia pytorch docker image

Following is example command to run the Nvidia pytorch docker image

sudo docker run --gpus all -it --rm \
        -v /host/work:/container/work \
        -p 7860:7860 \
        nvcr.io/nvidia/pytorch:XX.XX-py3

/host/work means work directory on host, /container/work means work directory in container(docker). -p 7860 mean gui default port, it needs to forward to host machine. XX.XX means container versiom please see tags to get container version, or this URL . [NGC Pytorch]. Following is example command.

sudo docker run --gpus all -it --rm \
        -v /work:/work \
        -p 7860:7860 \
        nvcr.io/nvidia/pytorch:24.01-py3

Stable Diffustion WebUI

First steps clone Stable Diffustion WebUI source code.

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

Then, it needs to modify the settings file. Edit webui.sh, and adjust the parameters to the following numbers.

use_venv=0
can_run_as_root=1

As we are operating within a container, there’s no need for a virtual environment (venv), and the script can be run as root. Modify the file webui-user.sh and add COMMANDLINE_ARGS accordingly.

export COMMANDLINE_ARGS="--no-half --precision full --listen"

or run it directly

export COMMANDLINE_ARGS="--no-half --precision full --listen" ; bash webui.sh

Currently, Stable Diffusion will download some packages, but errors may occur during the installation process. We’ll address and fix these errors in the next few steps.

CV2

Error message :

    if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 28, in __load_extra_py_code_for_module
    py_module = importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py", line 168, in <module>
    LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'

Modify file “/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py”, comment out line 169 like following.[ref]

# LayerId = cv2.dnn.DictValue

CodeFormer

Error message:

"/work/stable-diffusion-webui/repositories/CodeFormer/facelib/utils/face_restoration_helper.py", line 7, in <module>
        from facelib.detection import init_detection_model
      File "/work/stable-diffusion-webui/repositories/CodeFormer/facelib/detection/__init__.py", line 11, in <module>
        from .yolov5face.face_detector import YoloDetector
      File "/work/stable-diffusion-webui/repositories/CodeFormer/facelib/detection/yolov5face/face_detector.py", line 20, in <module>
        IS_HIGH_VERSION = tuple(map(int, torch.__version__.split('+')[0].split('.'))) >= (1, 9, 0)
    ValueError: invalid literal for int() with base 10: '0a0'

This issue caused by pytorch.version will return illegeal number like 2.2.0a0+81ea7a4, so it will fail to parse this mesage, temperory solution is to remove the version number, edit file “stable-diffusion-webui/repositories/CodeFormer/facelib/detection/yolov5face/face_detector.py”. and modify line 20, IS_HIGH_VERSION to following version to following:[ref]

version_string = torch.version.split('+')[0]
version_string = ''.join(filter(str.isdigit, version_string))
IS_HIGH_VERSION = tuple(map(int, version_string.split('.'))) >= (1, 9, 0)

Now, the Stable Diffusion GUI should be working as well as expected

The post How to Install Stable Diffusion GUI on ARM64 Nvidia RTX platform appeared first on richliu's blog.

What truly impresses me is the board’s specifications. Not only does it boast a compact Micro-ATX form factor, but it also packs a powerful punch with four PCIe 4.0 x16 lanes, easily handling even demanding homelab tasks. The dual M.2 slots and dual OCULink port can enable software RAID for your NVMe disks. As a server board, it offers a BMC for remote control and system management. While the x64 platform remains popular, this board is an exceptional choice for those who prefer the non-x64 architecture, considering the limited choices with high speed fan options in server platforms.

Information

Important Spec

AsRock ALTRAD8UD-1L2T Spec web site

Where To Buy

US/Newegg

ASRock Rack ALTRAD8UD-1L2T Deep Micro-ATX Server Motherboard Single Socket (LGA 4926) Ampere Altra Max/Ampere Altra processors Dual 10G & 1G Without CPU,
Mother board only
USD$849.-

Asrock Rack Bundle ALTRAD8UD-1L2T Deep Micro-ATX Server Motherboard Single Socket (LGA 4926) with Ampere Altra Q64-22 64 cores & 2U passive cooler, Dual 10G – Integrated by Asrock Rack with CPU and heatsink
Mother board + Altra Q64-22 + 2U passive cooler(heatsink)
USD$1,500.-

UK/Server Factory Ltd

ASRock ALTRAD8UD-1L2T: Board only, Inc VAT: £870.00.

News

ASRock Rack ALTRAD8UD-1L2T Review This is the Ampere Arm Motherboard You Want

Cooling solution

Making Arm Desktops Viable Ampere Altra Noctua NH-D9 AMP-4926 4U and NH-U14S AMP-4926 (Preview tower fan system)
Ampere Altra/AltraMax AsRock ALTRAD8UD-1L2T Liquid Cooling System (my homemake Liquid/Water cooling system)

Other reference URL

ASRock Rack ALTRAD8UD Getting Started : A good note to start up to use the board.

Tricks for ALTRAD8UD

This chapter aims to guide users through using the ALTRAD8UD board, addressing potential information gaps left by the official ALTRAD8UD manual.

For memory compatibility reference, AsRock offers its own DRAM AVL (Approved Vendor List). Alternatively, users can access the Ampere Altra family device document, freely available without registration.

OpenBMC default username password

User: root
Pass: 0penBmc (first word is zero, not O).
both BMC console and GUI’s default username password are the same.

Serial Console over LAN Port

The board support Serial Port Header (9-pin COM1), it needs to use this serial port or VGA port to enter BIOS and get BMC IP.
After get the BMC IP, it can get more information from serial console over LAN port
Port 2201: ATF console
Port 2202: SCP console
Port 2203: Host console
Port 22: ssh console

Here is my script to run the sol and make it more easy to use

# It can put following two lines to bashrc
alias oit='ipmitool -U root -P 0penBmc -I lanplus -C 17'
alias ossh="sshpass -p 0penBmc ssh -o GlobalKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no"

# get host console 
ossh root@<bmc ip> -p 2203 
# get in bmc ssh console 
ossh root@<bmc ip> 

Serial Console Output To VGA

If you cannot see the serial console output to console port, probably it bacuase this option was enabled[ref], just turn it off and add console=tty0 or console=ttyAMA0 to grub.conf, ex:

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyAMA0"

Ipmitool Fan Control [ref]

Manual Full speed
ipmitool raw 0x3a 0x01 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64

Manual Half speed
ipmitool raw 0x3a 0x01 0x32 0x32 0x32 0x32 0x32 0x32 0x32 0x32

Auto fan control
ipmitool raw 0x3a 0x01 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

NC-SI Port

In the user manual, NC-SI port is typo, the NC-SI port location is 5, in the document which marked as IPMI LAN port. if use this port, both host and BMC can use the same network cable to connect to the same ethernet.

Known Issues

Intel X550 massive AER error messages[ref1][ref2]

Seems that Intel NICs have compatibility issues with non-Intel platform systems. This can cause some server safety systems, like Advanced Error Reporting (AER), to generate many alerts and experience interface problems. A temporary solution is to add aspi_aspm=off to the Linux kernel parameters, ex:

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyAMA0 pcie_aspm=off"

It will fix the AER issue.

The post AsRock ALTRAD8UD-1L2T Ampere Altra/AltraMax Note appeared first on richliu's blog.

這次升級要從 x64 平台升級到 ARM64 平台，雖然目前 Linux 下 ARM64 支援度還是不錯，但是兩個平台能用的東西還是有差異，主要是紀錄一下升級過程中有什麼需要注意的事情

ARM64 Hardware Prepareation

CPU Board: Ampere Altra CPU + AsRock ALTRAD8U : 這個組合現在已經能在 newegg 上買到了，也是目前最強大的 ARM64 平台，當 homelab 平台非常適合

DRAM: 買這種主機板用的都是 RDIMM ，有可能便宜有可能貴，可以查 Gigabyte 的 QVL List[Ref] 內的 DRAM list 購買，有朋友就是在日本買到DDR4 2933 32GB RDIMM 一隻5000羊的DDR，目前匯率 NT$1,100.- 非常便宜

Network: AsRock ALTRAD8U 有 1*Intel i210 Ethernet 和 2*10G Intel x550 Ethernet ，這部份到不是問題，我另外拿了舊的 Intel x540 4 Port ethernet 當小 switch 用

SATA: 這種板子預設都沒有 SATA interface 了，所以要買 SATA 卡，有 PCIe 轉 SATA ，這個要注意 PCIe bandwidth 和 SATA 數量，一分錢一份貨。
如果要追求速度，淘寶上也可以買到便宜 SAS 外接 SATA ，淘寶的 LSI SAS 8port internal 卡大部份 NT$2500 到 NT$6000 都有，記得買轉接線要看好規格，SAS 接頭的規格很多，一不小心就會弄錯

SSD: 板子上介面有 2* M.2 2280，如果覺得不夠，可以買 OCU 轉 U.2 轉接線，或是還有 PCIe slot ，可以買 PCIe 轉 M.2 都可以。但是 U.2 的硬碟不一定好買，而且常常價位偏高，但是相對上是比較便宜的選擇

Software Move To ARM64 Platform

QEMU/Libvirt

這部份應該是最難的部份，不過理論上也不是不可以，我原來是 VirtualBox ，但是 ARM64 沒有 VirtualBox ，只有 Proxmox 有，剛看了一下，還支援 riscv64,loongarch64 [ref]，不過我個人不太想再學一套，所以還是用 virt-manager/libvirtd 這套，一般用途 virt-manager 和 libvirtd 應該也很夠用了。

目前用到現在都可以用，Ubuntu 22.04，Windows 10 for x86，Windows 11 for ARM，安裝沒什麼問題。只有 Windows 10 for x86 非常慢，慢到我不想再開了，不知道為什麼。

Qemu bridge devices: 這個應該是問題，如果是以前用 VirtualBox ，可以直接選 NIC card interface ，但是 virt-manager 似乎沒這個選項，所以要直接設定到 bridge device，不過影響不是很大，就只是設定一個 bridge device，然後將 host internet ethernet port device 綁上去就好了

其他可能是如果 Libvirt 要用 graphic 輸出，不能用 qxl ，只能用 VGA or ramfb ，尤其是 windows 只能用 ramfb，如果 Guest 是 Linux 可以用 VGA ，顯示比較正常，如果連 GUI 都沒有，就直接看 console 輸出吧，問題更少

Qemu file sharing

QEMU 支援 virtiofs ，不過這個功能在 gentoo 內 ARM64 是不能用的，因為不能用，所以我也不知道 x64 那邊會有什麼限制
另外一個是 virtio-9p ，這個缺點是分享進 VM 之後，沒辦法使用 user 權限存取，用 root 身份限制也很多，像是存取 non-root user 檔案會有問題[ref]

有其他需求的人可以考慮改用 samba or nfs ，但是效能就會大大下降了，這點可能是有效能需求的人要考量的

Firewall

原來的 Firewall script 似乎沒辦法和 Libvirt 一起工作，我又不想修改，最後比較簡單的作法就是在我的 script 執行完之後，再重啟 libvirtd ，這樣就正常了。
正常後別忘了執行 /etc/init.d/iptables save 儲存，這樣重開機之後才會正常執行 firewall rule.

Docker

試了一下似乎沒什麼問題，但是目前沒什麼要跑的 APP ，所以應該測不出什麼
不過之前 arm64 k8s docker [ref]都跑過了，應該踩雷的機會低很多

Other Software

在 Linux 下轉移到 ARM64 平台問題不大，大部份的 software 都是安裝之後 copy 過去
就算是在系統上有資料的 mysql 和 apache 都是這樣

這次我用了一個方式轉移 configuration files
1. 將要轉移的程式設定檔 commit 到 git server
2. 從新機器將 git server 拉回來 configuration files
3. diff 之後將新的設定套上去

因為我是 gentoo 玩家，gentoo 升級 packages 常常需 merge configuration files 了，所以這個轉移的過程不難，只是很耗費時間而已。不過因此絕大部份的設定檔都沒有問題，反而是少數系統本身自帶的設定檔反而怪怪的

KDE

之前我都遠端跑 vnc + kde ，不過 Gentoo 有個 BUG 就是 plasmashell 的 CPU usage 會很高
這次換 virt-manager 之後可以用 ssh remote control VM，發現這個方式還不錯，反正我也只有用到這個功能，KDE 等有需求的時候再開 vnc 就好了，平時是不需要開的

搬移步驟

1. 備份系統碟，以備不時之需
2. 重新規劃電腦的網路卡，如果只有一張就直接設成 bridge interface
3. software raid 的部份可以參考自己之前的文章 Linux Software Raid 系統轉移
4. UEFI 的部份可以參考自己之前的文章 Gentoo ARM64 UEFI Install Guide
5. 搬移 APPs
5.1 重要的 APP 才要搬資料，ex: Web, mysql
5.2 不重要的 APP 只要搬設定檔，ex: apt-cache-ng, squid 等等
5.3 如果是切換平台，要確定 VM 沒問題才能切

Known Issues

IPv6 Default Router

不知道為什麼，在 wan 的 bridge interface 會增加一個 default route fe80::1 ，這個設定會讓 IPv6 失效，後來解決方式是在 LAN 端加上一個 IP address fe80::1 才解決

efi boot

因為我系統碟是做 software raid ，每次編完 kernel 再 update grub 時，記得 efi boot 時有時會 fail ，試著將兩顆 nvme 都 mount 上來加上再安裝一次 efi partition 似乎才會正常，找了一下沒找到原因，不管了，反正能動就好

grub-mkconfig -o /boot/grub/grub.cfg
umount /boot/efi
mount /dev/nvme0n1p1 /boot/efi
grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Gentoo
umount /boot/efi
mount /dev/nvme1n1p1 /boot/efi
grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Gentoo
umount /boot/efi

ref.
None

The post [Gentoo] Upgrade From x64 to ARM64 appeared first on richliu's blog.

Chinese article version: 在 ARM64 QEMU System 上安裝 ARM64 Windows 11

Nowadays, ARM64 systems have become relatively mature, and recently, Windows 11 for ARM64 can also be installed on these systems. Naturally, there are people who want to run it on QEMU. However, most articles on this topic are not well-organized.

Here, let’s document the things what I discoverd and need to be prepared.

Preparations Hardware and Software

1) System: Ampere Altra CPU
Small ARM64 dev boards like RPI4 or RPI5 should also work, but installing on them might not be necessary, because it only has limitation CPU power and Memory. A more cost-effective solution is to use Apple Mx series CPUs [ref], as they are also ARM64, and many people purchase them for installing Linux.

2) ARM64 ISO Image
While there are numerous options available on internet, for a relatively legal acquisition of the ISO image, it is recommended to use Windows-11-On-Ampere
This step can be a bit complex, but the key is to use UUP Dump to fetch the Windows ISO and then use Rufus to burn it onto a USB Flash drive. During the burning process, Rufus can override some of the restrictions of Windows 11, such as TPM Support, which may not be fully supported on ARM64 CPUs.

3) Windows Virtio Driver

Download the Windows Virtio driver and place it into the USB Disk that was created in the previous step.

Here has download URL can download the Windows virtio driver : Proxmox Windows VirtIO Drivers, alternative download website:  download the latest stable or  download the most recent.
After downloading, extract the ISO contents and place them onto the USB Disk.

4) Convert USB Dump to Image
Linux users are likely familiar with this step.

dd if=/dev/sda of=/storage/Win11_ARM64_INSTALL.img

Why go through these steps? The primary reason is that within QEMU, it seems Windows cannot simultaneously configure two ISO devices. I encountered a freeze when attempting to install using the straightforward method. Hence, I went through this elaborate process to make it install smoothly.

5) virt-manager
Most examples available to others involve pure QEMU commands. However, this time, I wanted to avoid typing a bunch of commands and preferred a more user-friendly approach. Therefore, the examples provided here utilize virt-manager. Nonetheless, I believe the underlying logic is the same.

New Virtual Machine on QEMU

Virt Type: KVM
Architecutre: aarch64
Machine Type: virt

Next step, choice the ISO image we just created Win11_ARM64_INSTALL.img, Choose “the operaion system you are installing: Microsoft Windows 11”

For the CPU and memory, I used 16 cores and 16GB, which should be sufficient. A smaller combination might work, but I haven’t tested it because I’m too lazy to do so.

It’s recommended to set up the Disk Image manually, virt-manager tends to allocate the entire specified disk space right from the start, which may not be space-efficient. Windows 11, with its minimum space requirement of 52GB, can be allocated around 60GB or more based on your needs. Here is example command to pre-create qemu image.

qemu-img create -f qcow2 MyImage.qcow2 60G

Before executing, choose “Customize configuration before install.” There are certain adjustments to make. Remember to enable the XML editing feature within virt-manager beforehand. Go to the main program, select Edit -> Preferences -> Enable XML editing, and ensure it is checked.

In the XML section related to CPUs, locate the segment pertaining to ‘hyperv,’ and modify it with the following settings:

    <hyperv mode="custom">
      <relaxed state="off"/>
      <vapic state="off"/>
      <spinlocks state="off"/>
    </hyperv>

In the CPU section, change the quantity from ‘sockets’ to ‘cores.’ Below is an example for 16 cores:

Add those hardware setting into current configuration:
Input -> USB Keyboard
Input -> EvTouch USB Graphics Tablet
Graphics -> Spice server
Video -> Ramfb

For the Video option, it is essential to choose Ramfb; otherwise, the graphics might not display properly. QXL or Virtio should not be used. Virtio-gpu-pci hasn’t been tested, but some suggest it might work[ref].

Lastly, in Storage, add a USB CDROM device and set it up with the Win11_ARM64_INSTALL.img file you created earlier.

In the Boot Options section, enable the option for USB CDROM 1, and then you should be ready to boot.

If the configuration is correct, you should be able to boot to this screen successfully.

If following the above steps results in a hang during the installation process, you can try adjusting the boot options by changing the boot order. Try setting “disk” first and “USB CD ROM” second.

Install Windows

Under normal circumstances, Windows might not detect the disk. In such cases, select “Load driver” -> “25982_1000_ARM64_EN-US(C:)” -> “Virtio-driver” -> “viostor” -> “win11” -> “ARM64” (Note: The labels and directories here may vary based on different configurations).

In the final step, for the NIC driver, when prompted to connect to the network upon the initial entry into Windows 11, select “Install driver” -> “25982_1000_ARM64_EN-US(D:)” -> “Virtio-driver” -> “NetKVM” -> “win11” -> “ARM64”.

With this, you should have successfully completed the installation of Windows 11.

Happy Playing Windows 11 on ARM64 Qemu

ref.
virtio-win / virtio-win-pkg-scripts

The post Install ARM64 Windows 11 on ARM64 QEMU System appeared first on richliu's blog.

English please see : Install ARM64 Windows 11 on ARM64 QEMU System

現在 ARM64 的系統相對成熟了，最近 Windows 11 for ARM64 也可以在系統上安裝，自然也會有人想弄到 QEMU 上去，不過看來看去，大部份都沒有整理完畢的文章

這邊記錄一下需要準備的東西

準備

1) 系統：Ampere Altra CPU
RPI4 or RPI5 這種開發板應該也可以，因為這些板子只有有限的 CPU Power 和記憶體，就沒有安裝 qemu 的必要性，便宜一點的解決方案就是 Apple Mx 系列的 CPU [ref]，因為也是 ARM64 ，而且很多人也是買來安裝 Linux

2) ARM64 ISO Image
網路上應該很多，不過如果要相對合法取得 ISO image ，推薦 Windows-11-On-Ampere
這步驟有點複雜，不過重點是用 UUP Dump 抓回 Windows ISO ，然後用 Rufus 燒到 USB Flash ，在燒錄的時候 Rufus 可以解除一些 Windows 11 的限制，像是 TPM Support 等等這些在 ARM64 CPU 還不完全支援的設定

3) Windows Virtio Driver

下載 Windows virtio driver，然後放進去剛剛那一步做到的 USB Disk

Proxmox Windows VirtIO Drivers 這邊有下載點， download the latest stable or  download the most recent ，都可以下載到 VirtIO ISO，下載完將 ISO 解開放到 USB Disk

4) 將 USB Dump 成 Image
Linux user 應該很熟這個步驟

dd if=/dev/sda of=/storage/Win11_ARM64_INSTALL.img

為什麼要這樣做？最大的原因就是 QEMU 內 Windows 現在似乎沒辦法同時設定二個 ISO 設備，我之前這樣裝會卡住。所以就繞了一大串讓它可以裝，還可以吧

5) virt-manager
其他人能找到的範例都是 qemu command ，不過我這次實在是不想打一堆指令，只想點點點，所以這次的範例都是用 virt-manager ，不過我相信背後的思路都是一樣的

New Virtual Machine

Virt Type: KVM
Architecutre: aarch64
Machine Type: virt

下一步，ISO image 選剛剛做好的 Win11_ARM64_INSTALL.img，Choose the operaion system you are installing: 選 Microsoft Windows 11

CPU 和記憶體我用 16 cores 和 16GB，小一點應該也可以，不過我懶得試了

如果空間不夠，Disk Image 建議自己設定，virt-manager 設定會一開始就 allocate 設定的 disk space ，不省空間，Windows 11 最小需求空間是 52G ，可以設 60G 或是更大，看需求
這邊是範例命令，可以預先建好硬碟空間

qemu-img create -f qcow2 MyImage.qcow2 60G

在執行之前選擇 Customize configuration before install，有些東西要修改，記得要先開啟 virt-manager 內的 XML 修改功能，在主程式下的 Edit -> Perferences -> Enable XML editing ，請勾選

在 CPUs 的 XML 這一段，找到 hyperv 這一段，改成如下的設定

    <hyperv mode="custom">
      <relaxed state="off"/>
      <vapic state="off"/>
      <spinlocks state="off"/>
    </hyperv>

在 CPU 數量，要從 sockets 改成 Cores，以下就是 16 Cores 的範例

Add hardware ，加入以下設定
Input -> USB Keyboard
Input -> EvTouch USB Graphics Tablet
Graphics -> Spice server
Video -> Ramfb

Video 選項一定要用 Ramfb ，否則 Graphics 會出不來，qxl or virtio 都不能用，virtio-gpu-pci 沒試過，有人說可以用[ref]
最後是 Storage 增加 USB CDROM device，並且將剛剛的 Win11_ARM64_INSTALL.img 設定上去

在 Boot Options 這邊 enable USB CDROM 1的選項，然後就可以開機了

如果設定正確，那就可以正常開機到這個畫面

如果照著以上步驟卻在安裝過程卡住了，可以試著調整 boot option 切換一下開機的順序
先 disk 後 USB CD ROM .

Install Windows

正常狀況下，Windows 是找不到 disk 的，要選 load driver -> 25982_1000_ARM64_EN-US(C:) -> Virtio-driver -> viostor -> win11 -> ARM64 (註：這部份的標籤和目錄可能因不同的設定而異）

最後一段，NIC driver ，在初次進入 Windows 11 時，要求要連線到網路時，選 Install driver -> 25982_1000_ARM64_EN-US(D:) -> Virtio-driver -> NetKVM-> win11 -> ARM64

這樣就可以完成安裝 Windows 11 了

Happy Playing Windows 11 on ARM64 Qemu

ref.
virtio-win / virtio-win-pkg-scripts

The post 在 ARM64 QEMU System 上安裝 ARM64 Windows 11 appeared first on richliu's blog.

Chinese version Article is here (Ampere Altra/AltraMax 加AsRock ALTRAD8UD-1L2T 水冷方案)
2024/01/31 Update: Ampere will have 2U passive cooler Making Arm Desktops Viable Ampere Altra Noctua NH-D9 AMP-4926 4U and NH-U14S AMP-4926

The Ampere Altra/AltraMax CPU features a maximum of 128 ARM64 cores, typically found in cloud server. However, cloud server Care not suitable for placement in offices or homes due to the high-speed fans used for heat dissipation. The noise generated by these fans is generally intolerable for regular users. Additionally, adapting server motherboards to standard consumer motherboards is inconvenient. If there is a desire to utilize such a CPU in a different setting, the ADLink Ampere Altra Dev Platform presents a viable solution, complete with the option for liquid cooling.

Another solution is the AsRock ALTRAD8UD-1L2T, currently available on Newegg. Paired with an Ampere Altra Q64-22 CPU (64 cores, 2.2GHz), it comes with a complimentary 2U passive heatsink, priced at US$1,500, which can be considered affordable. However, it’s worth noting that the included heatsink is passive. If you prefer liquid cooling over installing a fan on the passive heatsink, then this article is the right guide for you.

The author have adding a declaration at the beginning of articles that may pose potential risks:

• This article is not an official endorsement by Ampere, Newegg, or any other mentioned merchants or vendors. It is solely a personal record.
• The author has a business relationship related to the production of the mentioned motherboard.
• The author is a first-time to use liquid cooling computer system , and the content may not be entirely accurate. Please be aware of potential risks.
• This article is not an official document from Ampere or AsRock.
• The original article is in Chinese, and the English translation is done using machine translation. Please forgive any inaccuracies in wording.
• I am not responsible for any consequences resulting from this article; for example, losses incurred due to improper implementation of instructions provided in this article.

The Ampere system primarily focuses on Cloud-native CPUs, and as of now, there is no manufacturer producing general-purpose cooling systems for it. The included 2U passive heatsink in this system is designed for server use, utilizing high-speed server fans for heat dissipation. While the Q64-22 included by Newegg has lower power consumption, its only cooling method involves direct airflow from an electric fan. Alternatively, users may resort to attaching an 8-inch fan onto it. Such cooling solutions are deemed unacceptable for enthusiasts and professionals like myself.

If you intend to upgrade the cooling system, typical tower-type coolers require replacing the fixture on the back of the motherboard. Moreover, compatibility with the fastening mechanisms of this specific motherboard can be challenging to find. Considering that only front-facing screws can be used, the only viable cooling solution is a liquid cooling system. Therefore, I have been experimenting with a liquid cooling solution that works for this scenario. I’m sharing this information for those adventurous individuals who might encounter similar challenges, and perhaps this approach can inspire some creative solutions.

In the upcoming sections of this article, I will guide you on how to set up a liquid cooling system specifically tailored for the Ampere Altra on the ALTRAD8UD-1L2T motherboard. In theory, since the screw hole positions are the same, this method should be applicable to other Ampere motherboards as well. However, the focus of this content will primarily be on the ALTRAD8UD-1L2T. The article will also cover any pitfalls encountered during assembly and the items needed for the setup(Chinese website).

Preparation

ALTRAD8UD-1L2T

You can find the specifications for this motherboard online, and I won’t repeat them here. However, what might confuse some people is the term “Deep mATX.” This is a specification unique to AsRock, where they increase the depth dimension of the mATX form factor while maintaining the standard width. The purpose is to provide more space to accommodate additional components. This particular specification is commonly seen in AsRock’s server-grade motherboards.

Chassis

Generally, most computer cases should work. However, I made a mistake on my end by choosing a case that was too small. The case I purchased is the Antec P110 Silent. While it has liquid cooling mounting points, it is not suitable for adding a liquid cooling system. The internal space is too limited, lacking a designated area for securing the water pump. Additionally, the Antec P110 liquid cooling radiator cannot be placed at the top as it interferes with the motherboard. Consequently, the radiator had to be positioned at the front of the case.

When making a purchase, it’s crucial to choose a case that is suitable for water cooling. In more extreme cases, considering a caseless system might be an option (hopefully, this can be updated in the next release).

Cases designed for liquid cooling systems typically have more significant internal space to accommodate a water pump, and there is ample room for fans without obstruction. Some all-in-one water cooling designs may be more compact and might not encounter this issue.

Liquid Cooling System

Liquid cooling systems have some considerations to take into account:

• Tubing type: Choose between soft or hard tubing. Hard tubing is suitable for experienced users, offering a longer lifespan and a more aesthetic appearance. Soft tubing is easier for assembly, making it suitable for beginners.
• All-in-one (AIO) or custom loop: Decide whether to go for a pre-assembled AIO liquid cooling system or build a custom loop system.

Since the initial focus was on researching liquid cooling, the combination chosen was a custom loop with soft tubing. Building a custom loop may encounter some issues, with one of the main challenges being the uncertainty about specifications.

The first crucial aspect is determining the size of the liquid cooling tubing, as additional components will depend on this size, limiting the options for procurement. Currently, the mainstream sizes are two-eighths (2/8’), three-eighths (3/8’), and four-eighths (4/8’) liquid cooling tubing.

Another parameter is the thickness of the tubing, thick tubing and thin tubing.
* thick tubing refer to 10/16mm or 3/8”-5/8” tubing.
* thin thbing refer to 10/13mm or 3/8”-1/2″ tubing.

Most common the inner diameter specification is G 1/4, when find some liquid colling componens like pressure relief valve needs it.

Once you’ve confirmed your inner and outer diameter, you can proceed to purchase the necessary liquid cooling components.

At a minimum, you’ll need:

• CPU water block *1
• Water Pump *1
• 240mm radiator *1
• 12cm fans *2
• Liquid coolant 500cc ~ 1000cc
• Tubing 2M
• Thermal paste
• Pressure relief valve
• Fittings: Quick connectors *6, 90-degree rotary fittings *(1~2)
• Tools for filling and draining: Filling tool, draining tool (watering can, large syringe, drainage hose)

Optional components:

• Flow meter, temperature gauge (note: buying these requires purchasing two additional quick connectors).

In practical testing, if focusing solely on CPU cooling, it appears that a 240mm radiator is sufficient. The thin three-eighths (3/8”) tubing provides a large enough water flow diameter. Choosing a smaller tube diameter may not be as suitable for this purpose.

CPU water block

This turned out to be quite troublesome. My original plan was to purchase a 50mmx50mm heatsink and connect it to the heatsink on the right. Then, I intended to secure it to the CPU socket using the fixtures shown in the image. However, as expected, unexpected issues arose – the world isn’t always that kind.

1. There was no water block with a 50mmx50mm connector available. So, I ended up buying a 40mmx40mm connector, one size smaller. This resulted in poorer heat dissipation performance. Fortunately, I never had the chance to test the inferior heat dissipation, so this step failed unnoticed.
2. Here’s where I made a mistake. Custom water block connectors, like the one shown, are typically 2/8″ fittings. However, the main liquid cooling system I bought has 3/8″ fittings, making them completely incompatible. The options were either to change the fittings or change the entire system. Changing the fittings was the more cost-effective choice, so I opted for that.

After connecting , it looked like the image on the right – and just looking at it made me worry that water would leak out.

So, I bought a two-eighths(2/8″) tubing to connect to this connector. However, while it could connect, it couldn’t link up with my three-eighths(3/8″) tubing in the water cooling system. I had to reluctantly abandon the custom water block connector.

The final solution had nothing to do with this connector.

Communical CPU water block

During the installation, I discovered that the Bykski CPU-XPH-T8 CPU water block, along with the provided adapter bracket, could be securely attached to the Ampere Altra’s socket. This was fantastic news and saved me a great deal of effort.

Although it’s designed for Intel LGA1700/LGA2066 sockets, the flexibility in its hole configuration allowed for adjustments, providing me with the necessary adaptability. As for whether other CPU brackets might be compatible, it’s likely that some could work. However, I haven’t had the chance to try others yet. If you have attempted this and found success, please let me know.

The base of this water block is quite substantial, and my intuition tells me that the cooling performance should be excellent. Therefore, I didn’t explore other compliant CPU water blocks.

However, another issue arises. The water block is 56mmx56mm, whereas the width of the Ampere Altra CPU socket is 55mm. The water block does not fit. At this point, there are two possible solutions:

Upon observing the CPU’s structure, the first solution seemed unfeasible. Cutting down the thickness of this water block would compromise its structural integrity, not to mention finding the tools or a professional service for the task.

Thus, the only viable option is the second solution. The simplest and most intuitive way to implement this is by using a thermal pad made of silicone to transfer heat upwards from the CPU.

Silicone Thermal Pad

But things couldn’t be as simple as I, a fool, initially thought. In conclusion, regular silicone thermal pads are not suitable for power monsters like the Ampere Altra. This type of thermal pad may be effective for low-power devices such as routers, switches, NVMe disks, etc., but it’s not useful for high-power CPUs.

In the image, there are 2mm thick silicone thermal pads, and you need two layers because the height is just that much. Essentially, this provides only a slight improvement over having no heat dissipation at all, so it’s not worth considering.

As for the high-efficiency silicone thermal pads available in the market, I don’t know if they would work. If you have them on hand, you can give them a try. However, I believe the chances of success are quite low.

Purple Copper Thermal Pad

Nowadays, a common method for cooling is using purple copper heat pipes. This type of heat dissipation is quite effective, and the 2U passive heatsink included with Newegg utilizes these heat pipes. Therefore, if we can customize the heatsink, wouldn’t that solve the problem? Luckily, the country right next to my country is the world’s largest manufacturing factory, capable of customizing just about anything. This made the solution very affordable and instantly resolved my problem.

The dimensions of the purple copper thermal pad are: Length x Width x Height = 64x54x4mm, with a rounded R5 edge.

Alternatively, you can measure the dimensions of the piece under the 2U passive heatsink to get the specifications. Placing it on the original heatsink, I can say the dimensions match perfectly. The craftsmanship is quite impressive, and there are no issues when it’s actually placed.

Assembly

First, place the purple copper thermal pad, and remember to apply thermal paste before doing so.

Apply thermal paste before placing CPU water cooling block.

Assemble the components.

Actually, I assembled the tubing on the left side incorrectly. I’ll need to purchase materials and quick connectors to replace it when I have the time. But for now, let’s leave it as is and make do.

Overall view.

This way, it’s evident that the case isn’t ideal. It’s too cramped, affecting airflow, and it cannot accommodate the components properly.

Shopping

If you are interested in replicating my setup, all the components were purchased from Taobao. I spent extra for testing and experimenting, but in reality, you should be able to set up the entire cooling system for around US$100 or NTD$3,000. This is comparable to the cost of an efficient dual-fan tower cooler.

Since this is a Chinese website, I’ll keep the text in Chinese and provide English translations afterward.

Liquid Cooling System

watertiger定制台式机电脑分体式CPU显卡水冷软管套装冷散热器 (“WaterTiger Custom Desktop Computer Split-Type CPU GPU Liquid Cooling Soft Tube Kit Radiator”)
Enter the URL choice “單 CPU 水冷 240 冷排” (“Single CPU Liquid Cooling 240 Radiator”)

At first, I chose this brand because the photos looked appealing, but it turned out to be a fortunate coincidence that saved me a lot of exploration time. This brand integrates various components into a set for sale, and the components they use are of decent quality. For example, they use the Enermax NEOChanger as the water pump, which is also a good pump. If you are familiar enough, you might not need these kits.

This kit does not include a pressure relief valve. You’ll need to purchase a pressure relief valve separately, and depending on your needs, you might want to buy additional quick connectors

Clamping Fittings

All the components can be purchased from the same place, saving the hassle of searching everywhere. It’s a good idea to buy a few extra parts to avoid trouble later on. The time cost is much higher than the cost of the components.

电脑水冷头扣具卡具CPU显卡南北桥固定笔记本散热器压条五金夹具 (“Computer Water Cooling Head Fastening Fixture, CPU GPU North-South Bridge Fixing, Laptop Heatsink Clamp, Hardware Clamp”)

Choice 台式机CPU冷头扣具(“Desktop CPU Cooling Head Fastening Fixture”) *2

M2 M3 M4不锈钢平头盘头螺丝钉水冷头水冷排机箱风扇安装长螺丝 (“M2 M3 M4 Stainless Steel Flat Head Countersunk Screws for Water Cooling Head, Radiator, Case Fan Installation – Long Screws”)

Fastening fixtures for securing CPU coolers and water blocks. The components can be increased based on requirements. The minimum required quantity includes
M3手拧螺母 (M3 thumb nuts)*4，M3*30 (M3 screws) *4 ，M3垫片(M3 washers) *4 ，M3六角螺母(M3 hex nuts) *4

电脑水冷水流计 液冷散热器流量器 流体温度观测器透明水流指示计 (“Computer Liquid Cooling Flow Meter, Liquid Cooler Flow Indicator, Fluid Temperature Monitor Transparent Flow Meter”) (option)

Choice: 3孔水流观测器 + 温度计 (“3-Port Flow Meter + Thermometer for Water Flow Observation”)

Nice to have items, but it can make me feel better.

Test Result

This system feels relatively quiet and not noisy.

The following tests are only for verifying the liquid cooling setup and are not formal performance tests. The results are provided for liquid cooling reference and do not include testing power consumption, etc.

Room temperature: 25 degrees Celsius
CPU : Ampere AltraMax 128-30
Run command : stress-ng –cpu 128 –iomix 128 –vm 128

Gentoo Linux kernel 6.6 ，CPU temperature is approximately，Power consumption CPU 150W/ IO 19W
Ubuntu 22.04 kernel 6.2(?)，CPU temperature is approximately 72 degrees，Power consumption CPU 178W/ IO 20W

In conclusion, I hope this article has been helpful for those eager to try it out.

Afterword

Newegg has also listed another model, which is only the motherboard version, only US$849.-
ASRock Rack ALTRAD8UD-1L2T Deep Micro-ATX Server Motherboard Singel Socket (LGA 4926) Ampere Altra Max/Ampere Altra processors Dual 10G & 1G

The post Ampere Altra/AltraMax AsRock ALTRAD8UD-1L2T Liquid Cooling System appeared first on richliu's blog.

My hardware also different, if you don’t have such powerful machine, it can skip numa part, that’s only for high end ARM64 machine.
The hardware is Ampere Mt. Collins which has 2*Ampere Altra 80 Cores CPU and 8*32G Memory, my experiment is run different VM on different CPU, to seperate them and lock it on different numa node.

There are two CPUs, so, it can create 3 nodes, so, I have plan to allocate resource like that.
1. k8s master on socket 0: 10 CPUs, 16G memory
2. k8s node1 on socket 0 : 70 CPUs, 100G memory
3. k8s node2 on socket 1: 80 CPUs, 120G memory

Kubernetes version v1.28.2
Ubuntu 22.04

Notice: Before reading this document, this document is personal experiment, not Ampere official document, take your own risk.

Prepare VM from libvirt

The server has remote control system which problem sufficent resolution to use GUI on server, so, I install ubuntu 22.04 on host system, than use virt-manager to create k8s master node, for qemu image, I create the disk image alone, seems create image when run install will allocate all disk space, seems that’s a virt-manager bug.
And the host network is configured to be a bridge interface, in my case, that’s br0.

So, when uses virt-manager to create VM, it can bridge network interface to br0.

Configure the VM with Numa

It can use numastat to check currnet numa status, this system has two numa nodes. it can use numastat and numactl to get more numa information.

# numastat -n

Per-node numastat info (in MBs):
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
Numa_Hit               591239.17       791075.41      1382314.58
Numa_Miss                   0.00            0.00            0.00
Numa_Foreign                0.00            0.00            0.00
Interleave_Hit              8.63            7.13           15.77
Local_Node             519434.17       777974.99      1297409.16
Other_Node              71805.00        13100.42        84905.42

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 128377 MB
node 0 free: 80224 MB
node 1 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 1 size: 126426 MB
node 1 free: 93888 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

virt-manager doesn’t have advance command to assign CPU, probably modify xml can get that purpose, but it more converince to use the “virsh” cli command to adjust VM parameters. “virsh list” show current VMs, “virsh vcpupin list current VM CPU scheduling”, “virsh vcpupin domain-id vcpu cpulist” to modify the VCPU run on which one CPU.

For memory, it needs to use “virsh numatune” to lock the memory on different node.
“virsh numatune domain [–mode mode] [–nodeset nodeset] [[–config] [–live] | [–current]]”, –current can update current running VM machine.

$ virsh list
 Id   Name          State
-----------------------------
 9    k8s-master   running
 10   k8s-node1     running

$ virsh vcpupin k8s-master
 VCPU   CPU Affinity
----------------------
 0      0-159
 1      0-159
 2      0-159
 3      0-159
 4      0-159
 5      0-159
 6      0-159
 7      0-159
 8      0-159
 9      0-159

# this command can lock 9 k8s-master's VCPU 0 on physical CPU 0. 
$ virsh vcpupin 9 0 0 

# Show current VM's numatune, for example, following is unset status. 
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   :

$ virsh numatune 9 --nodeset 0

# the numa_nodeset was set to 0
$ virsh numatune 9
numa_mode      : strict
numa_nodeset   : 0

The problem is, seems this script need to run everytime, alternativate solution is to modify XML, but Mt. collis has 160 cores, I won’t to modify the XML file. script is more converience on this situation, following is my sample script to adjust k8s-master, k8s-node1, and k8s-node2’s parameter, it can easy to modify it if you install other system or AltraMax CPU.

#!/bin/bash

configure_vm(){
        local VM_NAME="$1"
        local VM_VCPU="$2"
        local VM_CPU="$3"
        local VM_NUMA="$4"
        local VM_TYPE="$5" # VM_TYPE for MASTER or NODE0, NODE1 .. self define name

        id=$(virsh list | grep $VM_NAME | awk '{print $1}')
        state=$(virsh list | grep $VM_NAME | awk '{print $3}')

        if [ "$state" != "running" ]; then
          echo "$VM_TYPE is not running"
          exit 1
        else
          echo "configure $VM_TYPE"
        fi
        numa_nodeset=$(virsh numatune $id | grep numa_nodeset | awk '{print $3}')

        start_cpu=$(echo $VM_VCPU | cut -d'-' -f1)
        end_cpu=$(echo $VM_VCPU | cut -d'-' -f2)
        for ((vcpu = start_cpu; vcpu <= end_cpu; vcpu++)); do
          # echo "$id $vcpu $VM_CPU"
          virsh vcpupin $id $vcpu $VM_CPU > /dev/null
        done
        if [ -n "$numa_nodeset" ]; then
          echo "$VM_TYPE numa_nodeset is $numa_nodeset"
        else
          # echo "$id $VM_NUMA"
          virsh numatune $id --nodeset $VM_NUMA
        fi

}

configure_vm "k8s-master" "0-9" "0-9" "0" "Master"
configure_vm "k8s-node1" "0-69" "10-79" "0" "Node0"
configure_vm "k8s-node2" "0-79" "80-159" "1" "Node1"


====

# After run the script, it can run numastat <pid> to see the qemu runs numa status. ex:
$ sudo numastat 206601

Per-node process memory usage (in MBs) for PID 206601 (qemu-system-aar)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                        11.51            0.00           11.51
Stack                        0.04            0.00            0.04
Private                   5984.93            0.00         5984.94
----------------  --------------- --------------- ---------------
Total                     5996.48            0.00         5996.48

In here, on socket 0 , 0-9 and 0-69 is vcpu, so, all start from 0. After run this script, it can use “virsh vcpupin” and “virsh numatune” to get current setting. Maybe cpuset parameter in XML can more easy to get this function, but I have not try it.

Install K8s

Here is my script, just reference some later blog and my previous configurtion script, so, it can copy and paste to system and run it directly.

Install basic system and k8s

# setup timezone
sudo apt install ntpdate -y
sudo ntpdate watch.stdtime.gov.tw
sudo timedatectl set-timezone Asia/Taipei

# install and enable docker

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update
sudo apt install -y containerd.io


containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd

echo "----------------"
echo "   swapoff"
echo "----------------"
sudo swapoff -a
sudo sed -i '/swap/ s/^\(.*\)$/#\1/g' /etc/fstab

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update

sudo apt-get -y install kubelet
sudo apt-get -y install kubeadm
sudo apt-get -y install kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Update kernel parametes
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# update sysctl
sudo sysctl --system

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

echo "if this is work node, it can run kubeadm join now"

echo "alias k=kubectl" >> .bashrc

Setup Master Node

Run this script on master node.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# echo "enable master to run pods"
# kubectl taint nodes --all node-role.kubernetes.io/master-

It will shows some message below, then run it on k8s-node1 and k8s-node2

kubeadm join 192.168.110.230:6443 –token e14f7k.35c3jpny81ry70re \

        –discovery-token-ca-cert-hash sha256:6636a7e4ec3542f19152f308f0789c52287b743b60efa3c13e7c23ce1147c581

If forgot command above, it can create another token to let other k8s node join, ex:

kubeadm token create --print-join-command

Flannel : Network fabric for container

K8s need a network fabric to provide internal network topology, there are many choices, but we will install matellb later, some other network like calico might have potential issue, so, we choice flannel as network fabric.

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

After run this command, the k8s node status will change to Ready, if not install network fabric, the node status will be NotReady.

$ kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   26h   v1.28.2
k8s-node1    Ready    <none>          26h   v1.28.2
k8s-node2    Ready    <none>          26h   v1.28.2

Helm: Kubbernetes Applications Manager

Install helm to further install metallb purpose, on previous article, helm2 needs another application to connect to k8s, but helm3 doesn’t need it, more easy to use on k8s.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o /tmp/get_helm.sh
chmod 700 /tmp/get_helm.sh
sudo /tmp/get_helm.sh

Metallb: Bere metal load-balancer for K8s

Install k8s on private side, it needs load-balancer to provide external IP to communication with internet or intranet. If you install Metallb, the “kubectl get svc”, the EXTERNAL-IP shows “Pending”, and k8s doesn’t response the arp on host, probably it doesn’t configure kube-proxy, it you have experience on previous metallb installtion, it might miss this part.

PATH=$PATH:/usr/local/bin/

helm repo add metallb https://metallb.github.io/metallb
helm --namespace metallb-system install --create-namespace metallb metallb/metallb -f metallb-values.yaml

# https://metallb.universe.tf/installation/
# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
        sed -e "s/strictARP: false/strictARP: true/" | \
        kubectl apply -f - -n kube-system

metallb-values.yaml is also different like previous setting, it needs to configure L2Advertisement on metallb, without it, your metallb won’t work. Follwing is metallb-values.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: primary-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.110.200-192.168.110.219

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - primary-pool

Traefik: TBD

Linkerd

Just show the install script here

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

echo "export PATH=$PATH:/home/$USER/.linkerd2/bin" >> ~/.bashrc

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

Other

it can insert isolinux to host linux kernel parameter to isolate the Linux scheduling to k8s nodes. (TBD)

isolcpus=10-159 nohz=on nohz_full=10-159

Test with Nginx

It can deploy nginx to test this system

kubectl create deploy nginx --image=nginx
kubectl expose deploy nginx --port=80 --target-port=80 --type=LoadBalancer
kubectl get svc nginx
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
nginx   LoadBalancer   10.99.211.255   192.168.110.200   80:31011/TCP   25h

Now, it can try to access http://192.168.110.200.
if want to Delete nginx deploy, run following command.

kubectl delete deploy nginx
kubectl delete svc nginx

Test Numa

It can run emojivoto to test the system only runs on socket1, use it because it can general a lot of traffic between pods.

curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml \
  | kubectl apply -f -

Now it can use htop to show all CPUs, and we can see most traffic on CPU 1~10 and 81~160. There are a little CPU usage on CPU 78 and 80, but it should be Linux regular scheduling(I have not add isocpu to host CPU this time)

The post ARM64 Kubernetes Qemu Metallb Linkerd Numa appeared first on richliu's blog.

root cause is this package needs static lib support, but regular CentOS doesn’t have it.

# make                                                                             cc -g -O3 -static -pthread  multichase.o permutation.o arena.o util.o  -lrt -lm -o multichase
/usr/bin/ld: cannot find -lrt                                                                                 /usr/bin/ld: cannot find -lm                                                                                  /usr/bin/ld: cannot find -lpthread
/usr/bin/ld: cannot find -lc                                                                                  collect2: error: ld returned 1 exit status                                                                    make: *** [<builtin>: multichase] Error 1

Install necessary package and download the source code

# Install necessary package
yum install git
yum groupinstall "Development Tools"

# Download source code
git clone https://github.com/google/multichase

# Install epel
dnf config-manager --set-enabled powertools
dnf install epel-release epel-next-release

# Install static library 
yum install glibc-static

ref.
Extra Packages for Enterprise Linux (EPEL)

The post multichase on Centos 8 Compile Issue appeared first on richliu's blog.

[root@localhost linux-4.18.0-80.11.2.el8_0]# make install                                            [55/2101]
/bin/sh ./arch/arm64/boot/install.sh 4.18.0 \
arch/arm64/boot/Image System.map "/boot"
dkms: running auto installation service for kernel 4.18.0
Deprecated feature: REMAKE_INITRD
Deprecated feature: REMAKE_INITRD
Deprecated feature: REMAKE_INITRD
Sign command: /lib/modules/4.18.0/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub                                                               Deprecated feature: REMAKE_INITRD                                                                             
Building module:
Cleaning build area...(bad exit status: 2)
make -j160 KERNELRELEASE=4.18.0 -C /lib/modules/4.18.0/build M=/var/lib/dkms/bnxt_en/1.10.2.221.0.114.0/build
KVER=4.18.0...(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.18.0 (aarch64)
Consult /var/lib/dkms/bnxt_en/1.10.2.221.0.114.0/build/make.log for more information.
Sign command: /lib/modules/4.18.0/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Deprecated feature: REMAKE_INITRD

Building module:
Cleaning build area...(bad exit status: 2)
make -j160 KERNELRELEASE=4.18.0 -C /lib/modules/4.18.0/build M=/var/lib/dkms/bnxt_re/221.0.114.0/build KVER=4$
18.0 BNXT_EN_INC=/var/lib/dkms/bnxt_re/221.0.114.0/source BNXT_QPLIB_INC=/var/lib/dkms/bnxt_re/221.0.114.0/so$
rce...(bad exit status: 2)
Error! Bad return status for module build on kernel: 4.18.0 (aarch64)
Consult /var/lib/dkms/bnxt_re/221.0.114.0/build/make.log for more information.
 Done.

And check the log file “/var/lib/dkms/bnxt_re/221.0.114.0/build/make.log”

It shows

DKMS make.log for bnxt_en-1.10.2.221.0.114.0 for kernel 4.18.0 (aarch64)
Tue Sep 27 15:41:04 CST 2022
Makefile:603: arch/aarch64/Makefile: No such file or directory
make[2]: *** No rule to make target 'arch/aarch64/Makefile'.  Stop.

Seems cannot find the Makefile, that because the drvier cannot locate right Makefile, because the include architecture is arm64, not aarch64 in Linux kernel .

By the way, basicly arm64 and aarch64 are the same thing, I don’t know why it shows aarch64 on kernel architecture and use arm64 as fold name. probably some traditional history.

solution is easy, go to the include folder and link arm64 to aarch, DONE! ex:

cd /lib/modules/4.18.0/build/arch
ln -sf arm64 aarch64

The post Linux Kernel Install Fail Because Broadcom NetXtreme 100G DKMS Driver Build Failed. appeared first on richliu's blog.

Before Build Source Code

This procedure runs on ARM64 platform, it should be also run on x86 platform, but I have not try it.

Build environment : Ampere eMAG CPU (32 cores) with 256G DRAM and uses 64G/128G RAM to build code, sometime if it cannot finsih compile tasks, probably your DRAM is not sufficient to build the code, I tried 16G and it doesn;t work.

Because host is running Ubuntu 22.04, and OpenBMC and EDKII only can be built on Ubuntu 18.04, so, it needs to use either virtual machine or contianer technologies to have Ubuntu 18.04 environment.

This note uses qemu to prepare build environment, tried docker but it doesn’t work, probably I will re-try it someday.

If use NFS or qemu overlayers filesystem to bridge folders ourside into VM, it might cause build problem, so, please increase disk size for you virtual machine virtual disk size, use nfs or share folder for copy images or source code only.

If want to build a workable EDKII code and run it on Ampere Mt. Jade platform, it needs to have Ampere fimware ATF and SCP binary, it might register Ampere customer connect to download those image files, without it, it only can build OpenBMC binary.

Prepare Build Environment

Get the Ubuntu 18.04 cloud image

$ wget https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-arm64.img
$ sudo apt install qemu-efi

Prepare hostname and userdata for VM

it needs to prepare two files, meta-file and user-data, following is example for meta-file

instance-id: Image-openbmc
local-hostname: openbmc

user-data file, this example username is ubuntu, password is ubuntu

#cloud-config
password: ubuntu
chpasswd: { expire: False }
ssh_pwauth: True
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+Y/6m4XFeEGKqR2H7wbsUQUVWvB0gC/+CeJqfyvOX++SETKapFtI5CiidkyW/B1GQM4XUlmE9zPlCAcolPfMpXB0CNQQ5l0F5Z+FW6b4RVjFV/KEcQ7R3XkKRa0XhC6hKkNc5P95fNhXr8Do1bSEmd9mPIiOrFaUEZ9EReO+ZTsGsytFIuQV8rOZfQjClktXrqk3F6jJ+TV3THcVNpKvnLfps56Hc/j9jdiadLv91yuifz4lCMbv/D/0hMtf4KhN5tV//QRgUxEUeC7aPW6sKD2GMYMxc9UsSyRRQ9IFWjJMkgzKyuxgNKe7u4WjERFoxEUrmFK01UsrX+lRw9e9n user@hostname

Than, it can create a configure file for QEMU

$ cloud-localds -v cloudinit.img user-data meta-file

Increase Ubuntu cloud image disk size, bigger is better, suggest more than 200G, minimal should be around 100G.

$ qemu-img resize ubuntu-18.04-server-cloudimg-arm64.img +200G

Prepare UEFI BIOS and storage for booting

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Running the VM script, it will also create a QEMU virtfs for file sharing, but do not use that folder to build code. How to access share folder, it can reference this article QEMU Share Directory with Host Without Networking

IMAGE1=ubuntu-18.04-server-cloudimg-arm64.img
sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 128G \
         -smp 32,sockets=1,cores=32,threads=1 \
         -nographic -nodefaults \
         -pflash flash0.img -pflash flash1.img \
         -drive file=$IMAGE1,if=none,id=disk1 \
         -device virtio-blk-device,drive=disk1 \
         -drive file=cloudinit.img,if=virtio,format=raw \
         -virtfs local,path=/nvme/obmc,mount_tag=host0,security_model=passthrough,id=host0 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0,mac=52:54:00:11:11:12 \
         -serial telnet::9001,server,nowait > guest_log-1.txt 2>> guest_log-1.txt

Build Mt. Jade EDKII

Prepare VM environment

After login into VM, it needs to install software to compile EDKII.
Command to install basic packages.

sudo apt update 
sudo apt install -y git vim tmux lftp net-tools ethtool bc
sudo apt install -y \
  build-essential uuid-dev acpica-tools nasm \
  python3 python3-distutils python3-pip \
  gawk
sudo apt install -y libssl-dev gnu-efi
sudo apt install bison flex
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1

Next step is follow this document to download Ampere EDKII source code and setup enviroment.
https://github.com/AmpereComputing/edk2.git

First, prepare build environment parameters, if VM reboot, it needs to re-run it.

export WORKSPACE=/nvme/tianocore
mkdir -p $WORKSPACE
cd $WORKSPACE
export PACKAGES_PATH=$PWD/edk2:$PWD/edk2-platforms
# PACKAGES_PATH is default include path, sometimes miss package can include more directories here. 
# if still cannot find some files, try to run submodules command on next step again. 
export PACKAGES_PATH=$PWD/edk2:$PWD/edk2-platforms:$PWD/edk2-non-osi/:$PWD/edk2-platforms/Features/

Chang into that folder, clone Ampere Computiner edk2 source code.

cd $WORKSPACE
git clone https://github.com/AmpereComputing/edk2.git
cd edk2
git submodule init && git submodule update
cd ..
git clone https://github.com/AmpereComputing/edk2-platforms.git

Get build environment

This step is to build the basic tools.

cd $WORKSPACE
. edk2/edksetup.sh

# Build BaseTool, only need to run once 
make -C edk2/BaseTools 

After compiled the EDK2 basic tool, next step is to prepare edk2-platforms and edk2-ampere-tools

edk2-ampere-tools has a very good build script,
it can help to build the EDK2 and also package ATF into together.
we will introduct it on following steps.

cd edk2-platforms
git checkout -b test remotes/origin/ampere

# make a link to library to fix some problem
ln -sf /nvme/tianocore/edk2-platforms/Features/Intel/OutOfBandManagement .
cd ..
git clone https://github.com/AmpereComputing/edk2-ampere-tools

Build and Sign the image

Because current tree Atftool migrate to OpenSSL 3.0, and Ubuntu 18.04 still use OpenSSL 1.0x,
I choice to roll back to old version to support OpenSSL 1.0x, support it can build Atftool on Ubuntu 22.04 or compile OpenSSL 3.0 manually, I might try it later.

cd /nvme/tianocore 
mkdir -p AtfTools
cd AtfTools
git init
git remote add origin -f https://github.com/ARM-software/arm-trusted-firmware.git
git config core.sparseCheckout true

echo -ne "include/tools_share\nmake_helpers\ntools/cert_create\ntools/fiptool" > .git/info/sparse-checkout

git -C . checkout --track origin/master
# Due to current AtfTool only support OpenSSL 3.0 , switch to previous version 
git checkout -b test b94bf967e62b23a376a5026de69d06a0e8d6bf78
make -C tools/cert_create CRTTOOL=cert_create
$ make -C tools/fiptool FIPTOOL=fiptool

configure PATH to fiptool and cert_create

cd /nvme/tianocore
export PATH=$PATH:$PWD/AtfTools/tools/cert_create:$PWD/AtfTools/tools/fiptool

This step is parepare Mt. Jade ATF and SCP binary for compile, it needs to get both binary file from Ampere customer connect. and also suggest to use the same version as OpenBMC/EDKII

cd $WORKSPACE
mv <path to>/altra_atf_signed_2.06.20220426.slim .
mv <path to>/altra_scp_signed_2.06.20220426.slim .
cd edk2-ampere-tools
./edk2-build.sh -b RELEASE Jade --atf-image $WORKSPACE/altra_atf_signed_2.06.20220426.slim --scp-image $WORKSPACE/altra_scp_signed_2.06.20220426.slim

If everything is perfect, the binary code would be looks like below.

Results: /nvme/tianocore/edk2-ampere-tools/BUILDS/jade_tianocore_atf_2.06.100
total 45888
-rw------- 1 root   root       3343 Jun 15 12:40 dbukey.auth
-rw------- 1 root   root       2002 Jun 15 12:40 del_dbukey.auth
-rw-r--r-- 1 root   root       2008 Jun 15 12:38 jade_board_setting.bin
-rw-rw-r-- 1 ubuntu ubuntu    14532 Jun 15 04:08 jade_board_setting.txt
-rw-r--r-- 1 root   root     670845 Jun 15 12:40 jade_scp_2.06.100.cap
-rw-r--r-- 1 root   root     262144 Jun 15 12:40 jade_scp_2.06.100.dbu.sig.img
-rw-r--r-- 1 root   root    8126464 Jun 15 12:38 jade_tianocore_2.06.100.fd
-rw-r--r-- 1 root   root   14040229 Jun 15 12:40 jade_tianocore_atf_2.06.100.cap
-rw-r--r-- 1 root   root   13631488 Jun 15 12:40 jade_tianocore_atf_2.06.100.dbu.sig.img
-rw-r--r-- 1 root   root   10225352 Jun 15 12:38 jade_tianocore_atf_2.06.100.img
------------------------------------------------------------
                          Ampere Jade (AARCH64) RELEASE pass
------------------------------------------------------------
pass    1
fail    0

It can use the jade_tianocore_atf_2.06.100.img for firmware update.

Note: When latest time I compile the edkII binary, latest version will fail to build, it need to roll back to previous version.

Manual Build EDKII

If still want to build EDKII manually, here is some command can help you.

cd edk2-platform
build -a AARCH64 -t GCC5 -b RELEASE -D SECURE_BOOT_ENABLE -p Platform/Ampere/JadePkg/Jade.dsc
or to use all CPU to compile the EDKII.
build -n `nproc` -a AARCH64 -t GCC5 -b RELEASE -D SECURE_BOOT_ENABLE -p Platform/Ampere/JadePkg/Jade.dsc

the generated image on $WORKSPACE/Build/Jade/RELEASE_GCC5/FV/BL33_JADE_UEFI.fd,
for this case, on the /nvme/tianocore/Build/Jade/RELEASE_GCC5/FV/BL33_JADE_UEFI.fd

then, it needs to use ampere special tool to generate EDKII image with Ampere ATF image.

Build Mt. Jade OpenBMC

Prepare build environment

Before build OpenBMC, it needs to install necessary packages,
it can build both OpenBMC and EDKII on the same VM, or it can build it on different VM.

Following is necessary packages for OpenBMC.

$ sudo apt-get install -yy \
     build-essential \
     chrpath \
     cpio \
     debianutils \
     diffstat \
     gawk \
     git \
     iputils-ping \
     libdata-dumper-simple-perl \
     liblz4-tool \
     libsdl1.2-dev \
     libthread-queue-any-perl \
     locales \
     python \
     python3 \
     socat \
     subversion \
     texinfo \
     wget \
     zstd

Download Open BMC source code and OpenBMC build script.

$ cd /nvme
$ git clone https://github.com/ampere-openbmc/openbmc.git
$ git clone https://github.com/openbmc/openbmc-build-scripts.git

modify openbmc-build-scripts/build-setup.sh around line #140,
replace “Clone in openbmc master to ${obmc_dir}” to https://github.com/ampere-openbmc/openbmc.git

it can download right source code.

git clone ${branch} https://github.com/ampere-openbmc/openbmc.git

If you use more than 32 cores to build the source code, it might cause compile fail because some compile tool might have problem on build code on ARM64 massive code. A easy way to check that is run dmesg to check whether Linux kernel has OOM killer on some package compile fail message like nodejs.

Limited CPU number to 32 , modify openbmc-build-scripts/build-setup.sh, on line 84.

 84 # num_cpu=${num_cpu:-$(nproc)}
 85 num_cpu=32

Build OpenBMC

This step is to prepare test build environment

$ cd /nvme
$ target=mtjade WORKSPACE=/nvme/openbmc branch="--branch ampere --single-branch" ./openbmc-build-scripts/build-setup.sh

Setup openbmc machine type to mt. Jade and compile OpenBMC.

$ export TEMPLATECONF=meta-ampere/meta-jade
$ export TEMPLATECONF=meta-ampere/meta-jade/conf/templates/default/
$ export TMPDIR=/nvme/temp
$ export LC_ALL=en_US.UTF-8

$ cd /nvme/openbmc
$ . openbmc-env
$ cd ..
$ source ./setup mtjade
$ bitbake obmc-phosphor-image

if system it mt.mitchell

$ export TEMPLATECONF=meta-ampere/meta-mitchell/conf/templates/default/
$ export TMPDIR=/nvme/temp
$ export LC_ALL=en_US.UTF-8

$ cd /nvme/openbmc
$ . openbmc-env
$ cd ..
$ source ./setup mtmitchell-dcscm
$ bitbake obmc-phosphor-image

When compile, there is an openocd compile error, it cannot access to original website probably caused by the website was down. Temperory solutoin is to remove the package.
modify file meta-ampere/meta-common/recipes-ac01/packagegroups/packagegroup-ampere-apps.bb, find openocd and delete it, down.

When finish compile, the binary will locate on tmp/deploy/images/mtjade/, absoult path is /nvme/openbmc/build/mtjade/tmp/deploy/images/mtjade/

ex: (only list partical files)

obmc-phosphor-image-mtjade-20220616040212.rootfs.manifest
obmc-phosphor-image-mtjade-20220616040212.rootfs.squashfs-xz
obmc-phosphor-image-mtjade-20220616040212.static.mtd
obmc-phosphor-image-mtjade-20220616040212.static.mtd.all.tar
obmc-phosphor-image-mtjade-20220616040212.static.mtd.tar
obmc-phosphor-image-mtjade-20220616040212.testdata.json
obmc-phosphor-image-mtjade.jffs2
obmc-phosphor-initramfs-mtjade-20220616004506.rootfs.cpio.xz
obmc-phosphor-initramfs-mtjade-20220616004506.rootfs.manifest
obmc-phosphor-initramfs-mtjade-20220616004506.testdata.json

obmc-phosphor-image-mtjade-20220616040212.static.mtd.tar is firmware file, it contain both obmc-phosphor-image-mtjade-20220616040212.static.mtd and obmc-phosphor-image-mtjade-20220616040212.rootfs.manifest files, and can be used for next step : firmware update.

Upgrade Firmware

*Notice: Update flash will take your own risk, especially physical damage your hardware..

If Mt. Jade is using AMI bios, suggest to use flash burner to update the OpenBMC firmware.
What I am using is GZU EZP XPro , it’s very cheap, only needs around US$30.- (please also purchase the SPI flash socket).

This step need to un-plug BMC flash from Mt. Jade motherboard, please remeber the flash pin 1 location, it needs to follow the original position and put it back to socket. If you miss you will broken it.

Update OpenBMC firmware

ref. OpenBMC Code Update

Update via URL

Suppose that it already has OpenBMC firmware, it can use software to upgrade firmware ,
first step is to get web token, then use this token on next step to upgrade firmware.

Following is a simple script to upgrade flash, it can be integrated into your script.

export BMC_IP=192.168.1.190
export FWFILE=obmc-phosphor-image-mtjade-20220616040212.static.mtd.tar

export token=`curl -k -H "Content-Type: application/json" -X POST https://${BMC_IP}/login -d '{"username" :  "root", "password" :  "0penBmc"}' | grep token | awk '{print $2;}' | tr -d '"'`

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    84  100    37  100    47    116    147 --:--:-- --:--:-- --:--:--   264


curl -k -H "X-Auth-Token: $token" \
       -H "Content-Type: application/octet-stream" \
       -X POST -T ${FWFILE} \
        https://${BMC_IP}/redfish/v1/UpdateService

{
  "@odata.id": "/redfish/v1/TaskService/Tasks/1",
  "@odata.type": "#Task.v1_4_3.Task",
  "Id": "1",
  "TaskState": "Running",
  "TaskStatus": "OK"
}

If conosle doesn’t not update and reboot, please use console login the OpenBMC and run “reboot” manually.
It will show message like below on OpenBMC console.

Updating kernel...
Erasing block: 69/69 (100%)
Writing kb: 4397/4397 (100%)
Verifying kb: 4397/4397 (100%)
Updating rofs...
Erasing block: 366/366 (100%)
Writing kb: 11580/23376 (49%)

Update via SSH

Just update firmware (ex: obmc-phosphor-image-mtjade-20220616040212.static.mtd.tar) to BMC’s directory /tmp/images , it will shows a hash number like /tmp/images/d6718c8f, then run the following command

busctl set-property xyz.openbmc_project.Software.BMC.Updater \
  /xyz/openbmc_project/software/d6718c8f \
  xyz.openbmc_project.Software.Activation RequestedActivation s \
  xyz.openbmc_project.Software.Activation.RequestedActivations.Active

Update UEFI firmware

Upgrade UEFI firmware
it needs to pack the UEFI image first , and it need to create MANIFEST file, looks like this

VER_purpose=xyz.openbmc_project.Software.Version.VersionPurpose.Host
version=jade_tianocore_atf_2.06.100
KeyType=OpenBMC
HashType=RSA-SHA256
MachineName=mtjade

make image file and MINIFEST file into the same file

$ tar -cvf jade_tianocore_atf_2.06.100.tar jade_tianocore_atf_2.06.100.img MANIFEST 

Run command below to upgrade UEFI firmware

export BMC_IP=192.168.1.190

export FWFILE=jade_tianocore_atf_2.06.100.tar

export token=`curl -k -H "Content-Type: application/json" -X POST https://${BMC_IP}/login -d '{"username" :  "root", "password" :  "0penBmc"}' | grep token | awk '{print $2;}' | tr -d '"'`

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    84  100    37  100    47    116    147 --:--:-- --:--:-- --:--:--   264


curl -k -H "X-Auth-Token: $token" \
       -H "Content-Type: application/octet-stream" \
       -X POST -T ${FWFILE} \
        https://${BMC_IP}/redfish/v1/UpdateService

Like OpenBMC, if it doesn’t update, just reboot to update it.
if stiall not work, try to upgrade it manually, yes, when I wrote this article, update via redfish doesn’t work, so, just copy it from github.

# on HOST
$ scp jade_tianocore_atf_2.06.100.tar root@192.168.1.190:/tmp

# on Target 
$ cd /tmp/images
$ mkdir fw
$ cd fw
$ tar xvf /tmp/jade_tianocore_atf_2.06.100.tar
$ firmware_update.sh /tmp/images/fw
--- Current Chassis State: On
--- Turning the Chassis off
--- Switch the host SPI bus to BMC.
Run update Primary Host SPI-NOR
--- Bind the ASpeed SMC driver
--- Locking power control
--- Flashing firmware to @/dev/mtd13
Erasing block: 157/157 (100%)
Writing kb: 9985/9985 (100%)
Verifying kb: 9985/9985 (100%)
--- Unlocking power control
--- Switch back to the primary Host SPI-NOR.
Turn on the Host

Update SCP firmware

It needs to use sftp to transfer SCP firmware to target.
Following is example to upgrade SCP firmware.

On host
$ scp altra_scp_signed_2.06.20220426.slim root@192.168.1.190:/tmp

On OpenBMC side
$ ampere_firmware_upgrade.sh smpmpro /tmp/altra_scp_signed_2.06.20220426.slim

After upgrade BIOS and SCP, if got any problem, suggest to reset bmc

$ ipmitool -U root -P 0penBmc -I lanplus -C 17 -H 192.168.2.18 mc reset cold

Serial Console

it can use command to get system CPU console.

# CPU console 
$ ssh root@192.168.190 -p 2200 

Reference

Ampere OpenBMC github
EDK2 Ampere Tool
Ampere EDK2
Ampere Computing Github

Error Message

Error: TEMPLATECONF value (which is /nvme/openbmc/meta-ampere/meta-mitchell) must point to meta-some-layer/conf/templates/template-name

Error message

Initializing OE build env

Error: TEMPLATECONF value (which is /nvme/openbmc/meta-ampere/meta-mitchell) must point to meta-some-layer/conf/templates/template-name

This is because followed the old configuration, it needs to point to right folder like

export TEMPLATECONF=meta-ampere/meta-jade/conf/templates/default/
. openbmc-env

.

The post Ampere Altra Mt. Jade OpenBMC and EDKII build guide on ARM64 platform appeared first on richliu's blog.

Install Necessary Packages

Install packages, it should have to install other packages like openssl-dev.

get package ampere-8.3.0-20191025-dynamic-nosysroot-nativetools.tar.xz from Ampere customer connect or ampere development source web.

yum groupinstall "Development Tools"

yum -y install ncurses-devel
yum -y install hmaccalc zlib-devel binutils-devel elfutils-devel libelf-devel newt-devel python-devel
yum -y install audit-libs-devel numactl-devel pciutils-devel openssl-devel
yum -y install bc bison perl-ExtUtils-Embed.noarch xmlto asciidoc
rpm –ivh kernel-4.18.0-80.11.2.el8.<version>.src.rpm


mkdir -p /opt/amp
tar -xf ampere-8.3.0-20191025-dynamic-nosysroot-nativetools.tar.xz -C /opt/amp
export PATH=/opt/amp/ampere-8.3.0-20191025-dynamic-nosysroot/bin:$PATH;
which gcc
/opt/amp/ampere-8.3.0-20191025/bin/gcc

if gcc is not in /opt/amp, please check your directory in /opt/amy

Build Kernel

command to build kernel

cd /root/rpmbuild/SOURCES/
tar xvf linux-4.18.0-80.11.2.el8_0.tar.xz
cd linux-4.18.0-80.11.2.el8_0
cp ../kernel-aarch64-emag.config .config
make olddefconfig

Try build once and make sure everything is ok

make -j `nproc`

If buiild OK, generate RPM files.

make -j `nproc` rpm-pkg

if everything is ok, the RPM files will be on /root/rpmbuild/RPMS/aarch64/

Install RPM packages

/root/rpmbuild/RPMS/aarch64
yum localinstall kernel-4.18.0-1.aarch64.rpm kernel-headers-4.18.0-1.aarch64.rpm

The post Build Ampere CentOS Kernel appeared first on richliu's blog.

NVME 4K Write Performance

Situation:
Run fio 4K write on 24 NVME drivers, but some drivers might not get higher IOPS performance

Solution:
It should be caused by enable SMMU, so, if doesn’t use VM, just disable iommu to get better performance
Add iommu.passthrough=1

General IO performance Tuning

For high bandwidth card, most time it needs to disable irq_balancing
$ systemctl stop irqbalance.service

Some Performance Tuning Parameters

ulimit -s unlimited 
ulimit -l 2097152 
echo 8 | sudo tee /proc/sys/vm/dirty_ratio 
echo 1 | sudo tee /proc/sys/vm/swappiness 
echo 1 | sudo tee /proc/sys/vm/zone_reclaim_mode 
echo 3 | sudo tee /proc/sys/vm/drop_caches 
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space 
echo 0 | sudo tee /proc/sys/kernel/numa_balancing 
echo never > /sys/kernel/mm/transparent_hugepage/enabled 
cpupower frequency-set -g performance 
export MALLOC_CONF=thp:always
 

The post ARM64 Linux Performance Tuning appeared first on richliu's blog.

2021/08/31 Updated : No more need Ubuntu ISO image, ARM64 EFI minimal ISO can be download from official website: https://www.gentoo.org/downloads/

Some ARM64 platform support UEFI bios system, like Ampere Computing Altea or Marvell ThunderX.
Recently Linux like CentOS or Ubuntu can support both system, but not for Gentoo ARM64 project,
if you want to install Gentoo on ARM64 bare system, it needs some trick skill.

Install System

Gentoo ARM64 install, it needs to relay on other OS’s installation software.
I were using Ubuntu/Debain net-inst to install system, below is my nvme driver partition table.

/dev/nvme0n1p1 (bootloader/efi) 500MB for EFI only
/dev/nvme0n1p2 / 800G for Gentoo system
/dev/nvme0n1p3 / 90G for Ubuntu system

Suggest to keep a ubuntu system for booting, it can be rescue system when fail to boot Gentoo system.

After Ubuntu/Debian installed system, it can go back to installation menu, there is a execute shell on menu, run it.

Decompress Gentoo Stage3 File.

Most challenge thing is, Ubuntu/Debian installation rootfs, there is no unbzip function, that’s means it needs to use installed disk and used the Ubuntu system’s bzip2. below is my command to untar it.

# use gentoo iso cdrom boot
# mount /dev/nvme0n1p3 /mnt/gentoo
# cd /mnt/gentoo
# wget https://bouncer.gentoo.org/fetch/root/all/releases/arm64/autobuilds/20210829T233646Z/stage3-arm64-systemd-20210829T233646Z.tar.xz
# tar xvf stage3-arm64-systemd-20210829T233646Z.tar.xz
# mount --bind /proc proc
# mount --bind /dev dev
# mount --bind /sys sys
# mount --bind /sys/firmware/efi/efivars/ ./sys/firmware/efi/efivars/
# cp /etc/resolv.conf /mnt/gentoo/etc/
# cd ..
# chroot gentoo 

>>>> chroot to ubuntu 
# mkdir  -p /mnt/gentoo
# mount /dev/nvme0n1p2 /mnt/gentoo 
# cd /mnt/gentoo 
# wget http://distfiles.gentoo.org/experimental/arm64/stage3-arm64-systemd-20190925.tar.bz2
# tar xvzf stage3-arm64-systemd-20190925.tar.bz2
# mount --bind /proc proc
# mount --bind /dev dev
# mount --bind /sys sys
# cp /etc/resolv.conf /mnt/gentoo/etc/
# cd ..
# chroot gentoo

Yaa.. We got Gentoo file system .

Gentoo Kernel

Suppose that reader know how to configure hostname , fstab, emerge –sync … etc.

Next step is to install Gentoo kernel, I will show how to use genkernel to generate workable kernel for ARM64 system

*Note: Some install command might need to unmask or add accept keyword, will ignore that, please try add “–autounmask-write” and run command “etc-update” to update it.

# mkdir -p /boot/efi
# mount /dev/nvme0n1p1 /boot/efi
# echo 'GRUB_PLATFORMS="efi-64"' >> /etc/portage/make.conf
# emerge --ask sys-boot/grub:2
# emerge efibootmgr
# grub-install --target=arm64-efi --efi-directory=/boot/efi --bootloader-id=Gentoo

If have problem on the stage, and it shows cannot find /dev/md0, it needs to emerge mdadm package.

Compile Linux Kernel

# emerge gentoo-sources 
# emerge genkernel
# genkernel --menuconfig all

genkernel will let you choice kernel configure before compile it. Please enable those options, Gentoo kernel didn’t enable ARM’s serial port by default.

Device Drivers  --->
	Character devices  --->
		Serial drivers  --->
			 <*> ARM AMBA PL010 serial port support
			[*]   Support for console on AMBA serial port
			 <*> ARM AMBA PL011 serial port support
			[*]   Support for console on AMBA serial port
                        [*] Early console using ARM semihosting

Save it and quit.

Next step is edit /etc/default/grub and add “console==ttyAMA0” to /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyAMA0"

Update grub.conf

grub-mkconfig -o /boot/grub/grub.cfg

Check UEFI Boot Priority

Run efibootmgr and get boot priority

# efibootmgr
BootCurrent: 0003
Timeout: 5 seconds
BootOrder: 0003,0001,0002,0004
Boot0001* ubuntu
Boot0002* UEFI: PXE IP4 Intel(R) Ethernet Server Adapter I210-T1
Boot0003* Gentoo
Boot0004* UEFI: Built-in EFI Shell

It can see current boot and boot order here , if Gentoo is not first priority, it can use option “-o” to update it.

Other

if you choice systemd as your service manager, after install openssh, please run command below to activate it and as default service.

# systemctl enable sshd.service
# systemctl start sshd.service
# systemctl status sshd.service

Conclusion

I have write down my note here, good luck to you if you have the same question then get this webpage.

The post Gentoo ARM64 UEFI Install Guide appeared first on richliu's blog.

Remove Packages

$ sudo yum remove docker-ce docker-ce-cli containerd.io

Install Containterd First

This is most important step, without this, docker’s install script will fail.

$ sudo dnf install https://download.docker.com/linux/centos/7/aarch64/stable/Packages/containerd.io-1.2.6-3.3.el7.aarch64.rpm

Docker Script

Get docker script and run it
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh
Add your current username to docker group if not run on root account.
$ sudo usermod -aG docker your-user

Restart Docker And Enable Firewall

$ sudo systemctl start docker
$ firewall-cmd –zone=public –add-masquerade –permanent
$ firewall-cmd –reload

ref.
[1] https://linuxconfig.org/how-to-install-docker-in-rhel-8
[2] https://docs.docker.com/engine/install/centos/
[3] https://serverfault.com/questions/987686/no-network-connectivity-to-from-docker-ce-container-on-centos-8

The post CentOS8 ARM64/AARCH64 Install Docker appeared first on richliu's blog.

Hardware topology 

[ARM64 CLIENT] <——-> [Server]  <——-> Internet 

SERVER ethernet ip is 10.0.0.1 , another interface connect to internet. 

Install necessary packages 

Install TFTP, HTTP and DHCPD Server and rpm2cpio 

$ apt install tftpd-hpa apache2 isc-dhcp-server rpm2cpio

Modify dhcpd server configuration 

$ vim /etc/dhcp/dhcpd.conf

following is example configuration,it can be changed to fit your environment 

authoritative;
default-lease-time                      259200;
max-lease-time                          518400;
option routers                          10.0.0.1;
option subnet-mask                      255.255.255.0;
option domain-name-servers              168.95.1.1,8.8.8.8;
ddns-update-style                       interim;
ignore client-updates;
allow booting;
allow bootp;
allow unknown-clients;
server-name                             pxis;

subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.100 10.0.0.200;
    option ip-forwarding off;
    option subnet-mask                  255.255.255.0;
    next-server 10.0.0.1;
    filename "shim.efi";

}

Modify tftpd setting 

$ vim /etc/default/tftpd-hpa

Add –create to TFTP_OPTIONS and change tftp directory to /tftpboot 

following is example configuration file 

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/tftpboot"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure  --create"

Restart Service 

mkdir /tftpboot 
/etc/init.d/tftpd-hpa restart
/etc/init.d/apache2 restart
/etc/init.d/isc-dhcp-server restart

Get ARM64 pxeboot file 

some times the shim and grub2-efi would be erase, just visit mirror.centos.org and find new one than download it.

cd /tftpboot 
wget http://mirror.centos.org/centos/8-stream/BaseOS/aarch64/os/Packages/shim-aa64-15-15.el8_2.aarch64.rpm
wget http://mirror.centos.org/centos/8-stream/BaseOS/aarch64/os/Packages/grub2-efi-aa64-2.02-106.el8.aarch64.rpm
rpm2cpio shim-aa64-15-15.el8_2.aarch64.rpm | cpio -dimv
rpm2cpio grub2-efi-aa64-2.02-106.el8.aarch64.rpm | cpio -dimv
cp ./boot/efi/EFI/centos/shim.efi .
cp ./boot/efi/EFI/centos/grubaa64.efi .
chmod +rx *.efi

Configure grub.conf 

$ vim /tftpboot/grub.cfg

Following is grub.conf example 

set timeout=60
menuentry 'CentOS' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}

If your system support secure boot and you want to do it, please replace linux with linuxefi and replace initrd with initrdefi. 

Get CentOS 8 stream image and copy it to web server 

cd /var/www/html 
wget http://ftp.twaren.net/Linux/CentOS/8-stream/isos/x86_64/CentOS-Stream-8-x86_64-20220215-dvd1.iso
mkdir disk 
mount -o loop CentOS-Stream-8-x86_64-20220215-dvd1.iso disk
cp -a disk centos8 
umount disk 
rmdir disk

# copy boot file into tftp directory 

cd /tftpboot
cp /var/www/html/centos8/images/ . -a

After copy all file into http file directory, it can visit via browser on server with URL http://10.0.0.1/centos8/ to check whether file exist or not. 

NAT

Server should support NAT function to forward packet. 

assume interface connect to internet is wls3
Following is a sample command to enable Linux NAT function which can make client to access internet or intranet . 

$ echo 1 > /proc/sys/net/ipv4/ip_forward ==> enable it immediately 
$ iptables -t nat -A POSTROUTING -o wls3 -j MASQUERADE

Now, you can choose pxeboot on your ARM64 machine. if everything successful, you can see CentOS on your console if everything is ok . 

Known Issue

Stdin: Invalid argument

My system cannot boot into the Ubuntu install program for an unknown reason. After checking the system, it seems that the issue may be caused by an inability to access internet resources for some reason, even though the DNS and network appear to be fine. An alternative solution is to put the ISO on a website and assign the argument to the parameter. Here is an example using Ubuntu 22.04.

menuentry 'Ubuntu22.04 Net Install' {
linux ubuntu2204/vmlinuz ip=dhcp url=http://10.0.0.1/ubuntu-22.04.1-live-server-arm64.iso only-ubiquity
initrd ubuntu2204/initrd
}

Debug 

Wireshark is your good friend, use it. 

Ubuntu PXE Environment

It can download Ubuntu Netboot images from this url.
http://cdimage.ubuntu.com/netboot/

Choice what you want, for example, url below is Ubuntu 18.04 for ARM64 image
http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/
Ubuntu 20.04 Netboot file
http://ports.ubuntu.com/ubuntu-ports/dists/focal-updates/main/installer-arm64/current/legacy-images/netboot/netboot.tar.gz

Download netboot image to tftp directory, I prefer to put different version to different directory.

$ cd /tftpboot
$ wget http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/netboot.tar.gz
$ mkdir ubuntu1804
$ cd ubuntu1804
$ tar xvf ../netboot.tar.gz

Modify grub.cfg and add Ubuntu option to grub.cfg, save it, now, this server can also provide ubuntu net install

set timeout=60
menuentry 'CentOS8 Install' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}
menuentry 'Ubuntu18.04 Install' {
  linux ubuntu1804/ubuntu-installer/arm64/linux ip=dhcp
  initrd ubuntu1804/ubuntu-installer/arm64/initrd.gz
}

Ubuntu ISO environment

It can download iso file and use the iso file as install media.

in Linux, download Ubuntu 20.04 ARM64 iso file

$ wget https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.3-live-server-arm64.iso
$ mount -o loop ubuntu-20.04.3-live-server-arm64.iso disk
$ mkdir ubuntu2004
$ cp disk/casper ubuntu2004/ -a 
$ umount disk

Modify grub.cfg as below (not test)

set timeout=60
menuentry 'CentOS8 Install' {
  linux images/pxeboot/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/centos8/
  initrd images/pxeboot/initrd.img
}
menuentry 'Ubuntu20.04 Install' {
  linux ubuntu2004/casper/vmlinuz ip=dhcp
  initrd ubuntu2004/casper/initrd
}

It also can use HWE kernel

menuentry 'Ubuntu20.04 Install' {
  linux ubuntu2004/casper/hwe-vmlinux ip=dhcp
  initrd ubuntu2004/casper/hwe-initrd
}

Note for GRUB Install Command

CentOS

svr=10.0.0.1; root=boot/centos8/images/pxeboot
linux (http,$svr)/$root/vmlinuz inst.stage2=http://$svr/boot/centos8 text
initrd (http,$svr)/$root/initrd.img

Ubuntu

svr=192.168.110.254; root=boot/ubuntu2004/casper
iso=ubuntu-20.04.3-live-server-arm64.iso
linux (http,$svr)/$root/vmlinuz url=http://192.168.110.254/boot/$iso only-ubiquity ip=dhcp ---
initrd (http,$svr)/$root/initrd

linux (http,$svr)/$root/hwe-vmlinuz url=http://192.168.110.254/boot/$iso only-ubiquity ip=dhcp ---
initrd (http,$svr)/$root/hwe-initrd

Fedora 36

Fedora’s setting is more closed to CentOS, due to CentOS might be dropped recently, so, moved to Fedora environment is necessary.

Get initrd/vmlinux image

Other parts like CentOS,
Download the PXE initrd and vmlinuz image, free.nchc.org.tw is Taiwan local Fedora server, you can replace it with local server.

mkdir -p /tftpboot/f36

cd /tftpboot/f36
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/os/images/pxeboot/initrd.img
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/os/images/pxeboot/vmlinuz

cd /var/www/html
wget http://free.nchc.org.tw/fedora/linux/releases/36/Server/aarch64/iso/Fedora-Server-dvd-aarch64-36-1.5.iso
mkdir disk 
mount -o loop Fedora-Server-dvd-aarch64-36-1.5.iso disk
cp -a disk f36
umount disk

Grub Fedora Setting

Add following grub entry text into /tftpboot/grub.cfg, thus, it can insert a “Fedora36” menu on pxe boot.

menuentry 'Fedora36' {
  linux f36/vmlinuz ip=dhcp inst.repo=http://10.0.0.1/f36/
  initrd f36/initrd.img
}

Ref. 

https://docs.centos.org/en-US/centos/install-guide/pxe-server/#sect-network-boot-setup-uefi
https://docs.centos.org/en-US/8-docs/advanced-install/assembly_preparing-for-a-network-install/
https://forums.opensuse.org/showthread.php/501075-error-cannot-find-commands-linuxefi-or-initrdefi

The post ARM64 Centos8/Ubuntu PXE Boot Install on Ubuntu Server appeared first on richliu's blog.

不要用 -mfpu 指令就可以了

ref.
-mfpu=neon-fp-armv8 and unrecognized command line option

The post ARM64 mfpu=neon-fp-armv8 and unrecognized command line option appeared first on richliu's blog.