--- /dev/null
+Testing changes to the Linux Kernel CephFS driver
+=================================================
+
+This walkthrough will explain one (opinionated) way to do testing of the Linux
+kernel client against a development cluster. We will try to mimimize any
+assumptions about pre-existing knowledge of how to do kernel builds or any
+related best-practices.
+
+.. note:: There are many completely valid ways to do kernel development for
+ Ceph. This guide is a walkthrough of the author's own environment.
+ You may decide to do things very differently.
+
+Step One: build the kernel
+==========================
+
+Clone the kernel:
+
+.. code-block:: bash
+
+ git init linux && cd linux
+ git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+ git remote add ceph https://github.com/ceph/ceph-client.git
+ git fetch && git checkout torvalds/master
+
+
+Configure the kernel:
+
+.. code-block:: bash
+
+ make defconfig
+
+.. note:: You can alternatively use the `Ceph Kernel QA Config`_ for building the kernel.
+
+We now have a kernel config with reasonable defaults for the architecture you're
+building on. The next thing to do is to enable configs which will build Ceph and/or
+provide functionality we need to do testing.
+
+.. code-block:: bash
+
+ cat > ~/.ceph.config <<EOF
+ CONFIG_CEPH_FS=y
+ CONFIG_CEPH_FSCACHE=y
+ CONFIG_CEPH_FS_POSIX_ACL=y
+ CONFIG_CEPH_FS_SECURITY_LABEL=y
+ CONFIG_CEPH_LIB_PRETTYDEBUG=y
+ CONFIG_DYNAMIC_DEBUG=y
+ CONFIG_DYNAMIC_DEBUG_CORE=y
+ CONFIG_FRAME_POINTER=y
+ CONFIG_FSCACHE
+ CONFIG_FSCACHE_STATS
+ CONFIG_FS_ENCRYPTION=y
+ CONFIG_FS_ENCRYPTION_ALGS=y
+ CONFIG_KGDB=y
+ CONFIG_KGDB_SERIAL_CONSOLE=y
+ CONFIG_XFS_FS=y
+ EOF
+
+Beyond enabling Ceph-related configs, we are also enabling some useful
+debug configs and XFS (as an alternative to ext4 if needed for our root file
+system).
+
+.. note:: It is a good idea to not build anything as a kernel module. Otherwise, you would need to ``make modules_install`` on the root drive of the VM.
+
+Now, merge the configs.
+
+
+.. code-block:: bash
+
+
+ scripts/kconfig/merge_config.sh .config ~/.ceph.config
+
+
+Finally, build the kernel:
+
+.. code-block:: bash
+
+ make -j
+
+
+.. note:: This document does not discuss how to get relevant utilities for your
+ distribution to actually build the kernel, like gcc. Please use your search
+ engine of choice to learn how to do that.
+
+
+Step Two: create a VM
+=====================
+
+A virtual machine is a good choice for testing the kernel client for a few reasons:
+
+* You can more easily monitor and configure networking for the VM.
+* You can very rapidly test a change to the kernel (build -> mount in less than 10 seconds).
+* A fault in the kernel won't crash your machine.
+* You have a suite of tools available for analysis on the running kernel.
+
+The main decision for you to make is what Linux distribution you want to use.
+This document uses Arch Linux due to the author's familiarity. We also use LVM
+to create a volume. You may use partitions or whatever mechanism you like to
+create a block device. In general, this block device will be used repeatedly in
+testing. You may want to use snapshots to avoid a VM somehow corrupting your
+root disk and forcing you to start over.
+
+
+.. code-block:: bash
+
+ # create a volume
+ VOLUME_GROUP=foo
+ sudo lvcreate -L 256G "$VOLUME_GROUP" -n $(whoami)-vm-0
+ DEV="/dev/${VOLUME_GROUP}/$(whoami)-vm-0"
+ sudo mkfs.xfs "$DEV"
+ sudo mount "$DEV" /mnt
+ sudo pacstrap /mnt base base-devel vim less jq
+ sudo arch-chroot /mnt
+ # # delete root's password for ease of login
+ # passwd -d root
+ # mkdir -p /root/.ssh && echo "$YOUR_SSH_KEY_PUBKEY" >> /root/.ssh/authorized_keys
+ # exit
+ sudo umount /mnt
+
+Once that's done, we should be able to run a VM:
+
+
+.. code-block:: bash
+
+ qemu-system-x86_64 -enable-kvm -kernel $(pwd)/arch/x86/boot/bzImage -drive file="$DEV",if=virtio,format=raw -append 'root=/dev/vda rw'
+
+You should see output like:
+
+::
+
+ VNC server running on ::1:5900
+
+You could view that console using:
+
+
+.. code-block:: bash
+
+ vncviewer 127.0.0.1:5900
+
+Congratulations, you have a VM running the kernel that you just built.
+
+
+Step Three: Networking the VM
+=============================
+
+This is the "hard part" and requires the most customization depending on what
+you want to do. For this author, I currently have a development setup like:
+
+
+::
+
+ sepian netns
+ ______________
+ | |
+ | kernel VM | sepia-bounce VM vossi04.front.sepia.ceph.com
+ | ------- | | ------ -------
+ | | | | | 192.168.20.1 | | | |
+ | | |--|--|- <- wireguard -> | | <-- sepia vpn -> | |
+ | |_____| | | 192.168.20.2 |____| |_____|
+ | br0 |
+ |______________|
+
+
+The sepia-bounce VM is used as a bounce box to the sepia lab. It can proxy ssh
+connections, route any sepia-bound traffic, or serve as a DNS proxy. The use of
+a sepia-bounce VM is optional but can be useful, especially if you want to
+create numerous kernel VMs for testing.
+
+I like to use the vossi04 `developer playground`_ to build Ceph and setup a
+vstart cluster. It has sufficient resources to make building Ceph very fast
+(~5 minutes cold build) and local disk resources to run a decent vstart
+cluster.
+
+To avoid overcomplicating this document with the details of the sepia-bounce
+VM, I will note the following main configurations used for the purpose of
+testing the kernel:
+
+- setup a wireguard tunnel between the machine creating kernel VMs and the sepia-bounce VM
+- use ``systemd-resolved`` as a DNS resolver and listen on 192.168.20.2 (instead of just localhost)
+- connect to the sepia `VPN`_ and use `systemd resolved update script`_ to configure ``systemd-resolved`` to use the DNS servers acquired via DHCP from the sepia VPN
+- configure ``firewalld`` to allow wireguard traffic and to masquerade and forward traffic to the sepia vpn
+
+The next task is to connect the kernel VM to the sepia-bounce VM. A network
+namespace can be useful for this purpose to isolate traffic / routing rules for
+the VMs. For me, I orchestrate this using a custom systemd one-shot unit that
+looks like:
+
+::
+
+ # create the net namespace
+ ExecStart=/usr/bin/ip netns add sepian
+ # bring lo up
+ ExecStart=/usr/bin/ip netns exec sepian ip link set dev lo up
+ # setup wireguard to sepia-bounce
+ ExecStart=/usr/bin/ip link add wg-sepian type wireguard
+ ExecStart=/usr/bin/wg setconf wg-sepian /etc/wireguard/wg-sepian.conf
+ # move the wireguard interface to the sepian nents
+ ExecStart=/usr/bin/ip link set wg-sepian netns sepian
+ # configure the static ip and bring it up
+ ExecStart=/usr/bin/ip netns exec sepian ip addr add 192.168.20.1/24 dev wg-sepian
+ ExecStart=/usr/bin/ip netns exec sepian ip link set wg-sepian up
+ # logging info
+ ExecStart=/usr/bin/ip netns exec sepian ip addr
+ ExecStart=/usr/bin/ip netns exec sepian ip route
+ # make wireguard the default route
+ ExecStart=/usr/bin/ip netns exec sepian ip route add default via 192.168.20.2 dev wg-sepian
+ # more logging
+ ExecStart=/usr/bin/ip netns exec sepian ip route
+ # add a bridge interface for VMs
+ ExecStart=/usr/bin/ip netns exec sepian ip link add name br0 type bridge
+ # configure the addresses and bring it up
+ ExecStart=/usr/bin/ip netns exec sepian ip addr add 192.168.0.1/24 dev br0
+ ExecStart=/usr/bin/ip netns exec sepian ip link set br0 up
+ # masquerade/forward traffic to sepia-bounce
+ ExecStart=/usr/bin/ip netns exec sepian iptables -t nat -A POSTROUTING -o wg-sepian -j MASQUERADE
+
+
+When using the network namespace, we will use ``ip netns exec``. There is a
+handy feature to automatically bind mount files into the ``/etc`` namespace for
+commands run via that command:
+
+::
+
+ # cat /etc/netns/sepian/resolv.conf
+ nameserver 192.168.20.2
+
+That file will configure the libc name resolution stack to route DNS requests
+for applications to the ``systemd-resolved`` daemon running on sepia-bounce.
+Consequently, any application running in that netns will be able to resolve
+sepia hostnames:
+
+::
+
+ $ sudo ip netns exec sepian host vossi04.front.sepia.ceph.com
+ vossi04.front.sepia.ceph.com has address 172.21.10.4
+
+
+Okay, great. We have a network namespace that forwards traffic to the sepia
+VPN. The next mental step is to connect virtual machines running a kernel to
+the bridge we have configured. The straightforward way to do that is to create
+a "tap" device which connects to the bridge:
+
+.. code-block:: bash
+
+ sudo ip netns exec sepian qemu-system-x86_64 \
+ -enable-kvm \
+ -kernel $(pwd)/arch/x86/boot/bzImage \
+ -drive file="$DEV",if=virtio,format=raw \
+ -netdev tap,id=net0,ifname=tap0,script="$HOME/bin/qemu-br0",downscript=no \
+ -device virtio-net-pci,netdev=net0 \
+ -append 'root=/dev/vda rw'
+
+The new relevant bits here are (a) executing the VM in the netns we have
+constructed; (b) a ``-netdev`` command to configure a tap device; (c) a
+virtual network card for the VM. There is also a script ``$HOME/bin/qemu-br0``
+run by qemu to configure the tap device it creates for the VM:
+
+::
+
+ #!/bin/bash
+ tap=$1
+ ip link set "$tap" master br0
+ ip link set dev "$tap" up
+
+That simply plugs the new tap device into the bridge.
+
+This is all well and good but we are now missing one last crucial step. What is
+the IP address of the VM? There are two options:
+
+1. configure a static IP but the VM's root device networking stack
+ configuration must be modified
+2. use DHCP and configure the root device for VMs to always use dhcp to
+ configure their ethernet device addresses
+
+The second option is more complicated to setup, since you must run a DHCP
+server now, but provides the greatest flexibility for adding more VMs as needed
+when testing.
+
+The modified (or "hacked") standard dhcpd systemd service looks like:
+
+::
+
+ # cat sepian-dhcpd.service
+ [Unit]
+ Description=IPv4 DHCP server
+ After=network.target network-online.target sepian-netns.service
+ Wants=network-online.target
+ Requires=sepian-netns.service
+
+ [Service]
+ ExecStartPre=/usr/bin/touch /tmp/dhcpd.leases
+ ExecStartPre=/usr/bin/cat /etc/netns/sepian/dhcpd.conf
+ ExecStart=/usr/bin/dhcpd -f -4 -q -cf /etc/netns/sepian/dhcpd.conf -lf /tmp/dhcpd.leases
+ NetworkNamespacePath=/var/run/netns/sepian
+ RuntimeDirectory=dhcpd4
+ User=dhcp
+ AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW
+ ProtectSystem=full
+ ProtectHome=on
+ KillSignal=SIGINT
+ # We pull in network-online.target for a configured network connection.
+ # However this is not guaranteed to be the network connection our
+ # networks are configured for. So try to restart on failure with a delay
+ # of two seconds. Rate limiting kicks in after 12 seconds.
+ RestartSec=2s
+ Restart=on-failure
+ StartLimitInterval=12s
+
+ [Install]
+ WantedBy=multi-user.target
+
+Similarly, the referenced dhcpd.conf:
+
+::
+
+ # cat /etc/netns/sepian/dhcpd.conf
+ option domain-name-servers 192.168.20.2;
+ option subnet-mask 255.255.255.0;
+ option routers 192.168.0.1;
+ subnet 192.168.0.0 netmask 255.255.255.0 {
+ range 192.168.0.100 192.168.0.199;
+ }
+
+Importantly, this tells the VM to route traffic to 192.168.0.1 (the IP of the
+bridge in the netns) and DNS can be provided by 192.168.20.2 (via
+``systemd-resolved`` on the sepia-bounce VM).
+
+In the VM, the networking looks like:
+
+::
+
+ [root@archlinux ~]# ip link
+ 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
+ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+ 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
+ link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
+ 3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
+ link/sit 0.0.0.0 brd 0.0.0.0
+ [root@archlinux ~]# ip addr
+ 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+ inet 127.0.0.1/8 scope host lo
+ valid_lft forever preferred_lft forever
+ inet6 ::1/128 scope host noprefixroute
+ valid_lft forever preferred_lft forever
+ 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
+ link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
+ inet 192.168.0.100/24 metric 1024 brd 192.168.0.255 scope global dynamic enp0s3
+ valid_lft 28435sec preferred_lft 28435sec
+ inet6 fe80::5054:ff:fe12:3456/64 scope link proto kernel_ll
+ valid_lft forever preferred_lft forever
+ 3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
+ link/sit 0.0.0.0 brd 0.0.0.0
+ [root@archlinux ~]# systemd-resolve --status
+ Global
+ Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
+ resolv.conf mode: stub
+ Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google
+
+ Link 2 (enp0s3)
+ Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
+ Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
+ Current DNS Server: 192.168.20.2
+ DNS Servers: 192.168.20.2
+
+ Link 3 (sit0)
+ Current Scopes: none
+ Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
+
+
+Finally, some other networking configurations to consider:
+
+* Run the VM on your machine with full access to the host networking stack. If you have the sepia vpn, this will probably work without too much configuration.
+* Run the VM in a netns as above but also setup the sepia vpn in the same netns. This can help to avoid using a sepia-bounce VM. You'll still need to configure routing between the bridge and the sepia VPN.
+* Run the VM in a netns as above but only use a local vstart cluster (possibly in another VM) in the same netns.
+
+
+Step Four: mounting a CephFS file system in your VM
+---------------------------------------------------
+
+This guide uses a vstart cluster on a machine in the sepia lab. Because the mon
+addresses will change with any new vstart cluster, it will invalidate any
+static configuration we may setup for our VM mounting the CephFS via the kernel
+driver. So, we should create a script to fetch the configuration for our
+vstart cluster prior to mounting:
+
+.. code-block:: bash
+
+ #!/bin/bash
+ # kmount.sh -- mount a vstart Ceph cluster on a remote machine
+
+ # the cephx client credential, vstart creates "client.fs" by default
+ NAME=fs
+ # static fs name, vstart creates an "a" file system by default
+ FS=a
+ # where to mount on the VM
+ MOUNTPOINT=/mnt
+ # cephfs mount point (root by default)
+ CEPHFS_MOUNTPOINT=/
+
+ function run {
+ printf '%s\n' "$*" >&2
+ "$@"
+ }
+
+ function mssh {
+ run ssh vossi04.front.sepia.ceph.com "cd ceph/build && (source vstart_environment.sh; $1)"
+ }
+
+ # create the minimum config (including mon addresses) and store it in the VM's ceph.conf. This is not used for mounting; we're storing it for potential use with `ceph` commands.
+ mssh "ceph config generate-minimal-conf" > /etc/ceph/ceph.conf
+ # get the vstart cluster's fsid
+ FSID=$(mssh "ceph fsid")
+ # get the auth key associated with client.fs
+ KEY=$(mssh "ceph auth get-key client.$NAME")
+ # dump the v2 mon addresses and format for the -o mon_addr mount option
+ MONS=$(mssh "ceph mon dump --format=json" | jq -r '.mons[] | .public_addrs.addrvec[] | select(.type == "v2") | .addr' | paste -s -d/)
+
+ # turn on kernel debugging (and any other debugging you'd like)
+ echo "module ceph +p" | tee /sys/kernel/debug/dynamic_debug/control
+ # do the mount! we use the new device syntax for this mount
+ run mount -t ceph "${NAME}@${FSID}.${FS}=${CEPHFS_MOUNTPOINT}" -o "mon_addr=${MONS},ms_mode=crc,name=${NAME},secret=${KEY},norequire_active_mds,noshare" "$MOUNTPOINT"
+
+That would be run like:
+
+.. code-block:: bash
+
+ $ sudo ip netns exec sepian ssh root@192.168.0.100 ./kmount.sh
+ ...
+ mount -t ceph fs@c9653bca-110b-4f70-9f84-5a195b205e9a.a=/ -o mon_addr=172.21.10.4:40762/172.21.10.4:40764/172.21.10.4:40766,ms_mode=crc,name=fs,secret=AQD0jgln43pBCxAA7cJlZ4Px7J0UmiK4A4j3rA==,norequire_active_mds,noshare /mnt
+ $ sudo ip netns exec sepian ssh root@192.168.0.100 df -h /mnt
+ Filesystem Size Used Avail Use% Mounted on
+ fs@c9653bca-110b-4f70-9f84-5a195b205e9a.a=/ 169G 0 169G 0% /mnt
+
+
+If you run into difficulties, it may be:
+
+* The firewall on the node running the vstart cluster is blocking your connections.
+* Some misconfiguration in your networking stack.
+* An incorrect configuration for the mount.
+
+
+Step Five: testing kernel changes in teuthology
+-----------------------------------------------
+
+There 3 static branches in the `ceph kernel git repository`_ managed by the Ceph team:
+
+* `for-linus <https://github.com/ceph/ceph-client/tree/for-linus>`_: A branch managed by the primary Ceph maintainer to share changes with Linus Torvalds (upstream). Do not push to this branch.
+* `master <https://github.com/ceph/ceph-client/tree/master>`_: A staging ground for patches planned to be sent to Linus. Do not push to this branch.
+* `testing <https://github.com/ceph/ceph-client/tree/testing>`_ A staging ground for miscellaneous patches that need wider QA testing (via nightlies or regular Ceph QA testing). Push patches you believe to be nearly ready for upstream acceptance.
+
+You may also push a ``wip-$feature`` branch to the ``ceph-client.git``
+repository which will be built by Jenkins. Then view the results of the build
+in `Shaman <https://shaman.ceph.com/builds/kernel/>`_.
+
+Once a kernel branch is built, you can test it via the ``fs`` CephFS QA suite:
+
+.. code-block:: bash
+
+ $ teuthology-suite ... --suite fs --kernel wip-$feature --filter k-testing
+
+
+The ``k-testing`` filter is looking for the fragment which normally sets
+``testing`` branch of the kernel for routine QA. That is, the ``fs`` suite
+regularly runs tests against whatever is in the ``testing`` branch of the
+kernel. We are overriding that choice of kernel branch via the ``--kernel
+wip-$featuree`` switch.
+
+.. note:: Without filtering for ``k-testing``, the ``fs`` suite will also run jobs using ceph-fuse or stock kernel, libcephfs tests, and other tests that may not be of interest to you when evaluating changes to the kernel.
+
+The actual override is controlled using Lua merge scripts in the
+``k-testing.yaml`` fragment. See that file for more details.
+
+
+.. _VPN: https://wiki.sepia.ceph.com/doku.php?id=vpnaccess
+.. _systemd resolved update script: systemd-resolved: https://wiki.archlinux.org/title/Systemd-resolved
+.. _Ceph Kernel QA Config: https://github.com/ceph/ceph-build/tree/899d0848a0f487f7e4cee773556aaf9529b8db26/kernel/build
+.. _developer playground: https://wiki.sepia.ceph.com/doku.php?id=devplayground#developer_playgrounds
+.. _ceph kernel git repository: https://github.com/ceph/ceph-client
projects / ceph.git / commitdiff