Ubuntu

It can refer to those three articles to create a new cloud image from Ubuntu server image, remember to change ARM64 image to x86_64 image .

Change Ubuntu Cloud Image Size (if you choice ubuntu 16.04 as your image)
Change Ubuntu Cloud Image Password
Ubuntu Cloud Image Taiwan Mirror Site

Install Qemu and setup bridge devices

apt install -y ovmf qemu bridge-utils
mkdir -p /etc/qemu
echo "allow br0" > /etc/qemu/bridge.conf
echo "allow virbr0" >> /etc/qemu/bridge.conf

Write this script to a file and run it.

SERVERFILE=xenial-server-cloudimg-amd64-uefi1.img
VERSION=`uname -r`

sudo qemu-system-x86_64 -name vm1 \
        -machine pc-i440fx-xenial,usb=off \
        -cpu qemu64 -m 8192 \
        -smp 12,sockets=1,cores=12,threads=1 \
        -nographic -nodefaults \
        -bios /usr/share/ovmf/OVMF.fd \
        -virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 \
        -drive file=$SERVERFILE  \
        -net nic,macaddr=$macaddr \
        -net tap,ifname=tap0 \
        -serial telnet::9001,server,nowait > guest1_log.txt 2>&1 &

sleep 5

brctl addif br0 tap0

it can use command
qemu-system-x86_64 –machine help
and
qemu-system-x86_64 –cpu help
to get all machine and cpu options.

it can change cpu and machine type here.

network would be bridge on tap0

below is good option to share folder between host and VM, below command will share /home/work with vm.

option 
--virtfs local,path=/home/work,mount_tag=host0,security_model=passthrough,id=host0 

it can put this line to /etc/fstab to mount host’s folder to somewhere, in this case, just put the same directory like host does.

host0   /home/work   9p      trans=virtio,version=9p2000.L,user,umask=000   0 0

CentOS

CentOS document was written on 2021, but Ubuntu document on 2019, there is some difference between it. but that’s ok, I believe both ways are works.

Compile QEMU

CentOS default application doesn’t support run x86_64 code on ARM64 server, it needs to compile manually.

Please download qemu and compile it.

# Install CentOS Packages
yum config-manager --set-enabled powertools
yum install -y xorg-x11-xauth python3 ninja-build
yum install -y glib2-devel libmount-devel
yum install -y git glib2-devel libfdt-devel pixman-devel zlib-devel

# Download QEMU Source Code
wget https://download.qemu.org/qemu-5.2.0.tar.xz
# Decompress it
tar xvf qemu-5.2.0.tar.xz
# 
cd qemu-5.2.0
./configure --target-list="x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user"
make -j `nproc`
make install 

Then, qemu will be installed on /usr/local/bin

Download UEFI Image

It needs UEFI Image for x86_64 image boot up. It needs OVMF’s tianocore (Open Source UEFI BIOS) to boot up. And we can download kraxel as prebuild images.

# Install Kraxel into CentOS repos
cd /etc/yum.repos.d/
wget https://www.kraxel.org/repos/firmware.repo
yum install edk2.git-aarch64 edk2.git-ovmf-x64
cd <your vm directory>
cp /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd .
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd .

Prepare Bridge Devices

It needs bridge devices to connect to internet, default qemu bridge device name is virbr0 , due to CentOS doesn’t support brctl, follow this document(3 Ways to Create a Network Bridge in RHEL/CentOS 8) to create a CentOS bridge devices. Below is my experiment script to create bridge device, this setting will keep in system, reboot will not be reset.

nmcli conn add type bridge con-name virbr0 ifname virbr0

killall dhclient
ip addr flush enP7p2s0
nmcli conn modify virbr0 ipv4.addresses '192.168.3.191/16'
nmcli conn modify br0 ipv4.gateway '192.168.1.1'
nmcli conn modify br0 ipv4.dns '8.8.8.8'
nmcli conn modify br0 ipv4.method manual
nmcli conn add type ethernet slave-type bridge con-name bridge-br0 ifname enP7p2s0 master virbr0
nmcli conn up virbr0
nmcli conn up bridge-br0

nmcli conn show  --active

# Allow QEMU to access virbr0
mkdir -p /etc/qemu
echo "allow virbr0" >> /etc/qemu/bridge.conf

Run Qemu

It can refer above to prepare cloud image, I prefer to use ubuntu cloud image, but have not try CentOS image.

Remember Ubuntu 18.04 image won’t auto increase image size, please follow this Change Ubuntu Cloud Image Size to modify image size

qemu-system-x86_64 -m 8192 -smp 16 -nographic \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE-pure-efi.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS-pure-efi.fd \

        -hdc bionic-server-cloudimg-amd64.img \
        -netdev bridge,br=virbr0,id=net0 \
        -device virtio-net,netdev=net0,mac=00:11:22:33:44:55 \
        -serial telnet::9001,server,nowait > vmlog.txt 2>> vmlog.txt &

x86_64 application

Sometimes we don’t need to run whole VM to run some application just for x86_64, it can run user mode application directly. I am a lazy guy, so just reuse cloud image as x86_64 Linux library.

mkdir -p /home/amd64
qemu-img convert  bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.raw
losetup /dev/loop0 bionic-server-cloudimg-amd64.raw
kpartx -a /dev/loop0
mount /dev/mapper/loop0p1 /home/amd64

Umount Image

umount /home/amd64
losetup -d /dev/loop0

Run x86_64 only application

qemu-x86_64 -L /home/amd64/ bootutil64e
qemu-x86_64 -L /home/amd64/ nvmupdate64e -l

Unsolved Problem

Cannot find way to update INTEL NIC’s NVM

Tried vfio and application mode, vfio cannot pass iommu to host, it will fail.

User space application will cause mmap fail.

If you know how to update Intel NIC’s NVM, welcome to leave a message for me.

QEMU –static problem

Add –static parameter will cause some problem, cannot solve it right now.

./configure –target-list=”x86_64-softmmu aarch64-softmmu aarch64-linux-user x86_64-linux-user” –static
===

ERROR: sizeof(size_t) doesn’t match GLIB_SIZEOF_SIZE_T.

    You probably need to set PKG_CONFIG_LIBDIR

    to point to the right pkg-config files for your    build target

The post QEMU Run x86_64/AMD64 VM on ARM64/AARCH64 appeared first on richliu's blog.