Environment

Hardware : Ampere eMag Server with 128G RAM and 1T storage
Kubernetes master and nodes are running in QEMU image, so, we prepared three QEMU ARM64 ubuntu 18.04 images for kubernetes running. It also can use two VM to run this demo.

Prepare QEMU Image

It can download Ubuntu bionic cloud image from Taiwan mirror site

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img 

or international mirror site

https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.img

Install Necessary Package

Run following command to install necessary packages .

apt install -y qemu-system-arm bridge-utils cloud-image-utils git qemu-efi

Increase QEMU image size

qemu-img resize bionic-server-cloudimg-arm64.img +200G

Create UEFI bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Copy Image to three spare images

cp flash0.img flash0-1.img
cp flash0.img flash0-2.img 
cp flash0.img flash0-3.img 
cp flash1.img flash1-1.img
cp flash1.img flash1-2.img
cp flash1.img flash1-3.img
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-1.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-2.img 
cp  bionic-server-cloudimg-arm64.img   bionic-server-cloudimg-arm64-3.img 

Set up bridge and network devices

We already install bridge-utils before, now, it should configure bridge devices.

 mkdir -p /etc/qemu ; echo "allow br0" > /etc/qemu/bridge.conf 

Because there is only one network interface on machine, we should use this interface as bridge device, save text below (name nic-br.sh or what you like) and modify it to fit your environment.

ORIGNIC=enP6p1s0
ip addr flush $ORIGINIC
brctl addbr br0
brctl addif br0 $ORIGINIC
ifconfig br0 up
ifconfig br0 192.168.110.1 netmask 255.255.0.0
route add default gw 192.168.1.1
echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Configure Hugepage On Host Site (Option)

Due to previous research on dpdk runs on ARM64 server, so, configure hugepages here. But not necessary for run k8s environment.
You can ignore this part

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX="console=tty0 hugepagesz=1GB hugepages=32"

then run foloowing command to update grub.cfg

 update-grub 

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages 
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab 

Now, please reboot to make huagpage works.

Running QEMU

Following is QEMU script, save it (ex: named run-qemu.sh). It can be more simplification, but this script would be more understand what different.

IMAGE1=bionic-server-cloudimg-arm64-1.img
sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-1111-1111-1111-11111aaaaaaa \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-1.img -pflash flash1-1.img \
         -drive file=$IMAGE1,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0,mac=52:54:00:11:11:11 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest_log-1.txt 2>> guest_log-1.txt &
IMAGE2=bionic-server-cloudimg-arm64-2.img
sudo qemu-system-aarch64 -name vm2 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-2222-2222-2222-22222bbbbbbb \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-2.img -pflash flash1-2.img \
         -drive file=$IMAGE2,if=none,id=disk2 \
         -device virtio-blk-device,scsi=off,drive=disk2,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net2,ifname=tap1 \
         -device virtio-net-device,netdev=net2,mac=52:54:00:22:22:22 \
         -mem-path /mnt/hugepages \
         -serial telnet::9002,server,nowait > guest_log-2.txt 2>> guest_log-2.txt &
IMAGE3=bionic-server-cloudimg-arm64-3.img
sudo qemu-system-aarch64 -name vm3 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -uuid d0000000-3333-3333-3333-33333ccccccc \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0-3.img -pflash flash1-3.img \
         -drive file=$IMAGE3,if=none,id=disk3 \
         -device virtio-blk-device,scsi=off,drive=disk3,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net3,ifname=tap2 \
         -device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \
         -mem-path /mnt/hugepages \
         -serial telnet::9003,server,nowait > guest_log-3.txt 2>> guest_log-3.txt &
sleep 5
brctl addif br0 tap0
brctl addif br0 tap1
brctl addif br0 tap2

There are some parameters need to understand in QEMU script.

-uuid d0000000-1111-1111-1111-11111aaaaaaa \

this will assign a unique uuid for each VM, k8s needs different VM run with different uuid and MAC address.
It also can use command “uuidgen” to generate different uuid for each VM, but here just for debug so assign a “easy to read” uuid.

-device virtio-net-device,netdev=net3,mac=52:54:00:33:33:33 \

Assign a special mac address for VM, because for k8s internal communication, use the same mac address will cause problem, please configure different address before run qemu script. It not easy to modify it after first boot (with cloud image)

* Notice, first number for mac address cannot be odd , it would cause problem.

-mem-path /mnt/hugepages \

Tell VM to use hugepage, if were not configure host with hugepage, please remove this line.

Now, please run scripts

bash nic-br.sh
bash run-qemu.sh 

Now, it can use telnet to connect each VM’s console

telnet localhost 9001
# or 
telnet localhost 9002
# or 
telnet localhost 9003

After system booted/initialed, if you cannot login, please “killall qemu-system-aarch64” and follow this article Change Ubuntu Cloud Image Password to change all images’s password.

Somehow, for recent cloud image, it cannot modify password before run qemu script.

Install Kubernetes

There are a lot of article to talk about how to install on Ubuntu, I choice one for reference : Install and Deploy Kubernetes on Ubuntu 18.04 LTS

Below will shortly note install command here.

Install Docker and Kubeadm on every node

sudo apt update 
sudo apt -y upgrade 

# install and enable docker
sudo apt -y install docker.io 
sudo systemctl enable docker 
sudo systemctl start docker 

# Install k8s
sudo apt -y install apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add 
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update
sudo apt-get install kubeadm -y

Configure Network IP address (Optional)

Update to you network environment, here we used fix ip address.

Add new record to /etc/hosts

192.168.110.50     k8s-master
192.168.110.60     k8s-worker-node1
192.168.110.70     k8s-worker-node2

Please modify /etc/netplan/50-cloud-init.yam, remove dhcp and add fix address. remember, each node has different IP. Following is only for k8s-master.

network:
     ethernets:
         eth0:
             addresses: [192.168.110.50/16,]
             gateway4: 192.168.1.1
             nameservers:
                 addresses: [8.8.8.8,8.8.4.4]
             match:
                 macaddress: '52:54:00:11:11:11'
             set-name: eth0
     version: 2

Configure Host name

Run following command on master node

sudo hostnamectl set-hostname  k8s-master

Run following command on other nodes

sudo hostnamectl set-hostname   k8s-worker-node1 
sudo hostnamectl set-hostname   k8s-worker-node2

Disable swap (Option)

if image is not from cloud image, default Linux enabled swap function, k8s doesn’t support it, please disable it manually on all nodes and remove swap mount in /etc/fstab.

 sudo swapoff -a 

Configure Kubernetes

Now, we can initial Kubernetes. Run following command

 sudo kubeadm init --pod-network-cidr=10.244.0.0/16 

wait for few minutes, it will show some result like this

Run following command to assign current account has right to access k8s.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run this command on another node(k8s-worker-node1 and k8s-worker-node2).

sudo kubeadm join 192.168.110.50:6443 --token 46c1oj.i00vyfyn027pgew9 \
     --discovery-token-ca-cert-hash sha256:ed05198cc603ae009a862477bacef329a9e9a2bedc4a040bc4731e6d5cf4f48e

Run get nodes command, it will show current status, the master node status is NotReady, but don’t worry, it will become Ready after add pod and CNI (container networking interface) plugin.

ubuntu@k8s-master:~$ kubectl get nodes
 NAME         STATUS     ROLES    AGE     VERSION
 k8s-master   NotReady   master   3m42s   v1.16.0

This action is very import on ARM64 k8s platform, in other document, they told you use flannel or some other document told you used old flannel. all of them not work for me. Please instead flannel with weave net.
(Ref Weave Net for NetworkPolicy and CoreDNS is not starting on Ubuntu 18.04.Bionic Beaver #980)
Run following command to install weave net

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Use the following command to view the status of the network:

Get nodes again.

Great!! it works.

MicroK8s

If you think steps above are too complex and you would like to only run k8s on one machine, microk8s is a good friend for you, can run on ARM64 and very easy to use, just run command

sudo snap install microk8s

And just add a prefix “microk8s.” before k8s command, ex:

microk8s.kubectl get nodes

See, it’s very easy to use. right?

The post Install Kubernetes on ARM64 Ubuntu 18.04 (with QEMU) appeared first on richliu's blog.

Hardware : Ampere eMag Server with 128GB Memory
Network Card: Intel i210 Gigabit NIC Card * 2

HOST means commands and instructions on host machines
VM means commands and instructions inside VM

Prepare Environment

Apt source

First, please add deb-src to your database, edit file /etc/apt/source.list and removed all “#” before deb-src, then, run follow commands to update apt database (suppose all commands run with sudo or root in this article)

sed -i 's/# deb-src/deb-src/g' sources.list
apt update

Install Packages

It needs to install relative packages in host

sudo apt install -y dpdk dpdk-dev dpdk-doc 

Only needs to install on HOST.

sudo apt install -y qemu-efi bridge-utils

It will install all necessary dpdk packages in host.

Preload Module

It needs to preload vfio-pci modules by default, run following command

echo "vfio-pci" >> /etc/modules

Configure Line Kernel boot command

It needs to reserve hugepages memory space for Linux kernel, it reserves 32 pages and each pages 1GB, total 32GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=32 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX=”console=tty0 hugepagesz=1GB hugepages=32″

then run foloowing command to update grub.cfg

update-grub

Mount hugetlbfs by default

If wants to use hugepage, it needs to mount hugetlbfs somewhere, we can create a directory, ex: /mnt/hugepages and mount hugetblsfs there by default.

mkdir -p /mnt/hugepages
echo "nodev   /mnt/hugepages                  hugetlbfs       defaults,pagesize=1G    0 0 none" >> /etc/fstab

Bind Network Card for DPDK

It wants to let DPDK use current network card, it needs to bind the network card with dpdk. dpdk-devbind command is a useful command.
List current command

 $ dpdk-devbind --status
 Network devices using kernel driver
 0001:01:00.0 'I210 Gigabit Network Connection 1533' if=enP1p1s0 drv=igb unused=vfio-pci
 0002:01:00.0 'I210 Gigabit Network Connection 1533' if=enP2p1s0 drv=igb unused=vfio-pci
 0006:01:00.0 'I210 Gigabit Network Connection 1533' if=enP6p1s0 drv=igb unused=vfio-pci Active 

There are three network card in the machine, we would like to bind 1:1:0.0 and 2:1:0.0 to vfio-pci driver. please used following command to bind those network cards.

dpdk-devbind -b vfio-pci 0001:01:00.0 0002:01:00.0

and check status again

 $ dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0001:01:00.0 'I210 Gigabit Network Connection 1533' drv=vfio-pci unused=igb
0002:01:00.0 'I210 Gigabit Network Connection 1533' drv=vfio-pci unused=igb

Network devices using kernel driver
============================================
0006:01:00.0 'I210 Gigabit Network Connection 1533' if=enP6p1s0 drv=igb unused=vfio-pci Active 

Both network cards would be under Network devices using DPDK-compatible driver .

Prepare QEMU environment

Set up bridge devices

We already install bridge-utils before, now, it should configure bridge devices.

mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf

Because there is only one network interface on machine, we should use this interface as bridge device, save text below and modify it to fit your environment.

 ORIGNIC=enP6p1s0

 ip addr flush enP6p1s0


 brctl addbr br0
 brctl addif br0 enP6p1s0
 ifconfig br0 up
 ifconfig br0 192.168.110.1 netmask 255.255.0.0
 route add default gw 192.168.1.1


 echo nameserver 8.8.8.8 >> /etc/resolv.conf 

Run script, command brctl show will show how whether it configure property or not.

Prepare QEMU image

Ubuntu 18.04 cloud image is here

wget http://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-arm64.img

and please refer this article to change cloud image’s password
Change Ubuntu Cloud Image Password, before use it, don’t forget to increase cloud image size. 20G is enough for test.

qemu-img resize bionic-server-cloudimg-arm64.img +20G

Next step is to create bios and nvram for qemu image boot.

dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi/QEMU_EFI.fd of=flash0.img conv=notrunc
dd if=/dev/zero of=flash1.img bs=1M count=64

Run Qemu

Here is qemu command, save text below and run it

 IMAGE=bionic-server-cloudimg-arm64.img
 sudo qemu-system-aarch64 -name vm1 \
         -machine virt,gic_version=3,accel=kvm,usb=off \
         -cpu host -m 8192 \
         -smp 12,sockets=1,cores=12,threads=1 \
         -nographic -nodefaults \
         -pflash flash0.img -pflash flash1.img \
         -device vfio-pci,host=0001:01:00.0 \
         -device vfio-pci,host=0002:01:00.0 \
         -drive file=$IMAGE,if=none,id=disk1 \
         -device virtio-blk-device,scsi=off,drive=disk1,id=virtio-disk1,bootindex=1 \
         -netdev tap,id=net0,ifname=tap0 \
         -device virtio-net-device,netdev=net0 \
         -mem-path /mnt/hugepages \
         -serial telnet::9001,server,nowait > guest1_log.txt 2>> guest1_log.txt &
 sleep 5
 brctl addif br0 tap0 

-m 8192 : use 8G RAM for VM
–mem-path will use /mnt/hugepages for VM memory

it will create a network device tap0 for bridge device interface between VM and host. and create a telnet server on port 9001, it can use “telnet 9001” to get console, I would like to suggest to use ssh if it can access console.

Remove cloudinit service

it needs to remove cloud-initramfs-copymods, it will copy modules from ramdisk to harddriver, and cloud-init, it’s for some cloud service like aws(?)

removed them would help system runs better.

apt remove cloud-initramfs-copymods cloud-init 

Disable Services

After boot, it needs to disable some default service to improve speed, here is some services can be disabled. (needs run those command with root)

systemctl disable ufw
systemctl disable lxcfs
systemctl disable atd
systemctl disable cron
systemctl disable irqbalance
systemctl disable apparmor
systemctl disable apport
systemctl disable ebtables
systemctl disable grub-common
systemctl disable unattended-upgrades
systemctl disable iscsid

Most import service is irqbalance, without disable it, performance will be drop down.

Now, it can reboot VM now.

Prepare QEMU environment

Prepare QEMU environment and prepare host almost the same, only differences are

1. in QEMU, apt install doesn’t needs to install qemu-efi and bridge-utils
2. in QEMU, before call dpdk-devbind command, it needs to configure enable_unsafe_noiommu_mode for ARM64 platform
3. Hugepage size is different , in HOST it uses 32G, in VM we uses 4G

Enable unsafe noiommu

echo 1 > /sys/module/vfio/parameters/enable_unsafe_*

To permanently enable the no-IOMMU mode, add it to modprobe.d: (ref)

echo "options vfio enable_unsafe_noiommu_mode=1" > /etc/modprobe.d/vfio-noiommu.conf

It also needs to reserve hugepages memory space for Linux kernel for VM, it reserves 4 pages and each pages 1GB, total 4GB. Please modify /etc/default/grub , and add hugepagesz=1GB hugepages=4 to GRUB_CMDLINE_LINUX, ex:

GRUB_CMDLINE_LINUX=”console=tty0 hugepagesz=1GB hugepages=4″

then run foloowing command to update grub.cfg

update-grub

Run DPDK l3fwd

Before run l3fwd, we need to recompile it.

. /usr/share/dpdk/dpdk-sdk-env.sh
make -C /usr/share/dpdk/examples/l3fwd

Bind network interface

dpdk-devbind -b vfio-pci  enp0s1
dpdk-devbind -b vfio-pci  enp0s2

Run l3fwd example

cd /usr/share/dpdk/examples/l3fwd/build
./l3fwd -c 1 -- -p 0x3 --config="(0,0,0),(1,0,0)"

Multi Queue example

./l3fwd -l 1,2,3,4 -n 4 -- -p 0x3 --config="(0,0,1),(1,0,2),(0,1,3),(1,1,4)"

Other

There is good tool to get hugepage information

apt install hugepages
hugeadm --pool-list

The post Ubuntu 18.04 ARM64 DPDK in VM (QEMU) appeared first on richliu's blog.

Ubuntu

It can refer to those three articles to create a new cloud image from Ubuntu server image, remember to change ARM64 image to x86_64 image .

Change Ubuntu Cloud Image Size (if you choice ubuntu 16.04 as your image)
Change Ubuntu Cloud Image Password
Ubuntu Cloud Image Taiwan Mirror Site

Install Qemu and setup bridge devices

apt install -y ovmf qemu bridge-utils
mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf
echo "allow virbr0" >> /etc/qemu/bridge.conf

Write this script to a file and run it.

SERVERFILE=xenial-server-cloudimg-amd64-uefi1.img
VERSION=`uname -r`

sudo qemu-system-x86_64 -name vm1 \
        -machine pc-i440fx-xenial,usb=off \
        -cpu qemu64 -m 8192 \
        -smp 12,sockets=1,cores=12,threads=1 \
        -nographic -nodefaults \
        -bios /usr/share/ovmf/OVMF.fd \
        -virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 \
        -drive file=$SERVERFILE  \
        -net nic,macaddr=$macaddr \
        -net tap,ifname=tap0 \
        -serial telnet::9001,server,nowait > guest1_log.txt 2>&1 &

sleep 5

brctl addif br0 tap0

it can use command
qemu-system-x86_64 –machine help
and
qemu-system-x86_64 –cpu help
to get all machine and cpu options.

it can change cpu and machine type here.

network would be bridge on tap0

below is good option to share folder between host and VM, below command will share /home/work with vm.

option 
--virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 

it can put this line to /etc/fstab to mount host’s folder to somewhere, in this case, just put the same directory like host does.

host0   /home/work   9p      trans=virtio,version=9p2000.L,user,umask=000   0 0

CentOS

CentOS document was written on 2021, but Ubuntu document on 2019, there is some difference between it. but that’s ok, I believe both ways are works.

Compile QEMU

CentOS default application doesn’t support run x86_64 code on ARM64 server, it needs to compile manually.

Please download qemu and compile it.

# Install CentOS Packages
yum config-manager --set-enabled powertools
yum install -y xorg-x11-xauth python3 ninja-build
yum install -y glib2-devel libmount-devel
yum install -y git glib2-devel libfdt-devel pixman-devel zlib-devel

# Download QEMU Source Code
wget https://download.qemu.org/qemu-5.2.0.tar.xz
# Decompress it
tar xvf qemu-5.2.0.tar.xz
# 
cd qemu-5.2.0
./configure --target-list="x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user"
make -j `nproc`
make install 

Then, qemu will be installed on /usr/local/bin

Download UEFI Image

It needs UEFI Image for x86_64 image boot up. It needs OVMF’s tianocore (Open Source UEFI BIOS) to boot up. And we can download kraxel as prebuild images.

# Install Kraxel into CentOS repos
cd /etc/yum.repos.d/
wget https://www.kraxel.org/repos/firmware.repo
yum install edk2.git-aarch64 edk2.git-ovmf-x64
cd <your vm directory>
cp /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd .
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd .

Prepare Bridge Devices

It needs bridge devices to connect to internet, default qemu bridge device name is virbr0 , due to CentOS doesn’t support brctl, follow this document(3 Ways to Create a Network Bridge in RHEL/CentOS 8) to create a CentOS bridge devices. Below is my experiment script to create bridge device, this setting will keep in system, reboot will not be reset.

nmcli conn add type bridge con-name virbr0 ifname virbr0

killall dhclient
ip addr flush enP7p2s0
nmcli conn modify virbr0 ipv4.addresses '192.168.3.191/16'
nmcli conn modify br0 ipv4.gateway '192.168.1.1'
nmcli conn modify br0 ipv4.dns '8.8.8.8'
nmcli conn modify br0 ipv4.method manual
nmcli conn add type ethernet slave-type bridge con-name bridge-br0 ifname enP7p2s0 master virbr0
nmcli conn up virbr0
nmcli conn up bridge-br0

nmcli conn show  --active

# Allow QEMU to access virbr0
mkdir -p /etc/qemu
echo "allow virbr0" >> /etc/qemu/bridge.conf

Run Qemu

It can refer above to prepare cloud image, I prefer to use ubuntu cloud image, but have not try CentOS image.

Remember Ubuntu 18.04 image won’t auto increase image size, please follow this Change Ubuntu Cloud Image Size to modify image size

qemu-system-x86_64 -m 8192 -smp 16 -nographic \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE-pure-efi.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS-pure-efi.fd \

        -hdc bionic-server-cloudimg-amd64.img \
        -netdev bridge,br=virbr0,id=net0 \
        -device virtio-net,netdev=net0,mac=00:11:22:33:44:55 \
        -serial telnet::9001,server,nowait > vmlog.txt 2>> vmlog.txt &

x86_64 application

Sometimes we don’t need to run whole VM to run some application just for x86_64, it can run user mode application directly. I am a lazy guy, so just reuse cloud image as x86_64 Linux library.

mkdir -p /home/amd64
qemu-img convert  bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.raw
losetup /dev/loop0 bionic-server-cloudimg-amd64.raw
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /home/amd64

Umount Image

umount /home/amd64
losetup -d /dev/loop0

Run x86_64 only application

qemu-x86_64 -L /home/amd64/ bootutil64e
qemu-x86_64 -L /home/amd64/ nvmupdate64e -l

Unsolved Problem

Cannot find way to update INTEL NIC’s NVM

Tried vfio and application mode, vfio cannot pass iommu to host, it will fail.

User space application will cause mmap fail.

If you know how to update Intel NIC’s NVM, welcome to leave a message for me.

QEMU –static problem

Add –static parameter will cause some problem, cannot solve it right now.

./configure –target-list=”x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user” –static
===

ERROR: sizeof(size_t) doesn’t match GLIB_SIZEOF_SIZE_T.

    You probably need to set PKG_CONFIG_LIBDIR

    to point to the right pkg-config files for your    build target

The post QEMU Run x86_64/AMD64 VM on ARM64/AARCH64 appeared first on richliu's blog.