-
The FreeBSD build will build most of the tools in Ceph.
-Note that the RBD dependant items will not work since FreeBSD does not have RBS
+Note that the RBD dependant items will not work since FreeBSD does not
+have RBD (yet)
+
+I started looking into Ceph, because the HAST solution with CARP and
+ggate did not really do what I was looking for. But I'm aiming for
+running a Ceph storage cluster of storage nodes that are running ZFS.
+End station would be running bhyve on RBD disk that are stored in Ceph.
+
+Progress from last report:
+==========================
+
+Most important change:
+ - All test (but one) run to completion for the current selection of
+ tools. This is only the case for "My Fork" repository. Some of the
+ commits need to be pulled into the HEAD
+
+ - Cmake is now the only way of building Ceph
+ - Reworked threading/polling code for the simple socket code.
+ Now uses a selfpipe, instead of using an odd shutdown() signaling
+ Linux feature.
+ - Modified the EventKqueue code to work around the "feature" that
+ starting threads destroys the the kqueue handles. Just finshed the
+ code so it is not yet submitted to the main repository
Build Prerequisites
===================
-Compiling and building Ceph is tested on 11-CURRENT
-It uses the CLANG toolset that is available, which needs to be atleast 3.7
-Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to compile everything
+ Compiling and building Ceph is tested on 12-CURRENT, but I guess
+ 11-RELEASE will also work. And Clang is at 3.8.0.
+ It uses the CLANG toolset that is available, 3.7 is not tested, but was
+ working when that was with 11-CURRENT.
+ Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to
+ compile everything
The following setup will get things running for FreeBSD:
- - Install bash and link it in /bin
+ - Install bash and link it in /bin
# requires root privileges
sudo pkg install bash
sudo ln -s /usr/local/bin/bash /bin/bash
+ - Need to add one compatability line to
+ /usr/include/errno.h
+ #define ENODATA 87 /* Attribute not found */
+ (Otherwise some cython compiles will fail.)
+
+Getting the FreeBSD work on Ceph:
+=================================
+
+ - cd "place to work on this"
+ git clone https://github.com/wjwithagen/ceph.git
+ cd ceph
+ git checkout wip-wjw-freebsd-cmake
Building Ceph
=============
- - Go and start building using Cmake
- ./do_freebsd-cmake.sh
- - Go and start building using Autobuild (soon depricated)
+ - Go and start building
./do_freebsd.sh
-
+
+Note that the old way of building using automake is no longer used.
+
Parts not (yet) included:
=========================
- - RBD kernel client
- Rados Block Devices is implemented in the Linux kernel
- It seems that there used to be a userspace implementation first.
- And perhaps ggated could be used as a template since it does some of the
- same, other than just between 2 disks. And it has a userspace counterpart.
- * @trociny suggests:
- Userspace RBD is available under FreeBSD (both librbd and rbd tool
- (without linux specific commands) are buildable).
- - BlueStore.
- FreeBSD and Linux have different AIO API, and that needs to be made compatible
- Next to that is there discussion in FreeBSD about aio_cancel not working for all
- devicetypes
- - CephFS
- Cython tries to access an internal field in dirent which does not compile
-
-Tests that verify the correct working of the above are also excluded from the testset
-
-Test not (yet) include:
+ - RBD
+ Rados Block Devices is implemented in the Linux kernel
+ It seems that there used to be a userspace implementation first.
+ And perhaps ggated could be used as a template since it does some of
+ the same, other than just between 2 disks. And it has a userspace
+ counterpart.
+ - BlueStore.
+ FreeBSD and Linux have different AIO API, and that needs to be made
+ compatible Next to that is there discussion in FreeBSD about
+ aio_cancel not working for all devicetypes
+ - CephFS
+ Cython tries to access an internal field in dirent which does not
+ compile
+
+Tests that verify the correct working of the above are also excluded
+from the testset
+
+Tests not (yet) include:
=======================
- - ceph-detect-init/run-tox.sh
- Because the current implementation does not know anything about FreeBSD rc-init.
- - Tests that make use on nosestests.
- Calling these doest not really work since nostests is not in /usr/bin, and calling
- thru /usr/bin/env/nosetests does not work on FreeBSD.
- test/pybind/test_ceph_argparse.py
- test/pybind/test_ceph_daemon.py
+ - run-tox-ceph-disk
+ This is the main test that tests the basic Ceph disk install tool.
+ One of the next things to complete porting for.
+ - ceph-detect-init/run-tox.sh
+ Because the current implementation does not know anything about
+ FreeBSD rc-init.
+ - Tests that make use of nosetests.
+ Calling these does not really work since nosetests is not in
+ /usr/bin, and /usr/bin/env nosetests does not work on FreeBSD.
+ test/pybind/test_ceph_argparse.py
+ test/pybind/test_ceph_daemon.py
-Things to investigate:
-======================
+Task to do:
+===========
+ - Build an automated test platform that will build ceph/master on
+ FreeBSD and report the results back to the Ceph developers. This will
+ increase the maintainability of the FreeBSD side of things.
+ Developers are signalled that they are using Linux-isms that will not
+ compile/run on FreeBSD Ceph has several projects for this: Jenkins,
+ teuthology, pulpito, ...
+ But even just a while { compile } loop and report the build data on a
+ static webpage would do for starters.
- - ceph-{osd,mon} need 2 signals before they actually terminate.
+ - Run integration tests to see if the FreeBSD daemons will work with a
+ Linux Ceph platform.
-Things to improve upon:
-=======================
+ - Get the python tests that are currently excluded to work, and test OKE
+
+ - Compile and test the user space RBD (Rados Block Device).
+
+ - Investigate and see if an in-kernel RBD device could be developed a la
+ 'ggate'
+
+ - Investigate the keystore, which could be kernel embedded on Linux an
+ currently prevents building Cephfs and some other parts.
+
+ - Scheduler information is not used atm, because the schedulers work
+ rather different. But at a certain point in time, this would need some
+ attention:
+ in: ./src/common/Thread.cc
- - Scheduler information is not used atm, because the schedulers work rather different.
- But a a certain point in time, this would need some attention:
- in: ./src/common/Thread.cc
+ - Integrate the FreeBSD /etc/rc.d init scripts in the Ceph stack. Both
+ for testing, but mainly for running Ceph on production machines.
projects / ceph.git / commitdiff