From cde38c8edf15a0d606c747e16a825b69dcaeada4 Mon Sep 17 00:00:00 2001 From: Sage Weil Date: Thu, 2 Apr 2020 20:36:06 -0500 Subject: [PATCH] doc/dev/cephadm: a few notes on developing with cephadm Signed-off-by: Sage Weil (cherry picked from commit 086acd20b956e4ec5e19f4c72bd3b2214dc46d2a) --- .github/CODEOWNERS | 3 +- doc/dev/cephadm.rst | 87 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+), 1 deletion(-) create mode 100644 doc/dev/cephadm.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e72363760fd64..86e37fa11c26b 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -19,9 +19,10 @@ /qa/tasks/mgr/test_orchestrator_cli.py @ceph/orchestrators /qa/tasks/mgr/test_cephadm_orchestrator.py @ceph/orchestrators /qa/suites/rados/cephadm @ceph/orchestrators -/doc/mgr/orchestrator_cli.rst @ceph/orchestrators +/doc/mgr/orchestrator.rst @ceph/orchestrators /doc/mgr/orchestrator_modules.rst @ceph/orchestrators /doc/cephadm @ceph/orchestrators +/doc/dev/cephadm.rst @ceph/orchestrators #ceph-volume /src/ceph-volume @ceph/ceph-volume diff --git a/doc/dev/cephadm.rst b/doc/dev/cephadm.rst new file mode 100644 index 0000000000000..2fd475f130092 --- /dev/null +++ b/doc/dev/cephadm.rst @@ -0,0 +1,87 @@ +======================= +Developing with cephadm +======================= + +There are several ways to develop with cephadm. Which you use depends +on what you're trying to accomplish. + +vstart --cephadm +================ + +- Start a cluster with vstart, with cephadm configured +- Manage any additional daemons with cephadm + +In this case, the mon and manager at a minimum are running in the usual +vstart way, not managed by cephadm. But cephadm is enabled and the local +host is added, so you can deploy additional daemons or add additional hosts. + +This works well for developing cephadm itself, because any mgr/cephadm +or cephadm/cephadm code changes can be applied by kicking ceph-mgr +with ``ceph mgr fail x``. (When the mgr (re)starts, it loads the +cephadm/cephadm script into memory.) + +:: + + MON=1 MGR=1 OSD=0 MDS=0 ../src/vstart.sh -d -n -x --cephadm + +- ``~/.ssh/id_dsa[.pub]`` is used as the cluster key. It is assumed that + this key is authorized to ssh to root@`hostname`. +- No service spec is defined for mon or mgr, which means that cephadm + does not try to manage them. +- You'll see health warnings from cephadm about stray daemons--that's because + the vstart-launched mon and mgr aren't controlled by cephadm. +- The default image is ``quay.io/ceph-ci/ceph:master``, but you can change + this by passing ``-o container_image=...`` or ``ceph config set global container_image ...``. + + +cstart and cpatch +================= + +The ``cstart.sh`` script will launch a cluster using cephadm and put the +conf and keyring in your build dir, so that the ``bin/ceph ...`` CLI works +(just like with vstart). The ``ckill.sh`` script will tear it down. + +- A unique but stable fsid is stored in ``fsid`` (in the build dir). +- The mon port is random, just like with vstart. +- The container image is ``quay.io/ceph-ci/ceph:$tag`` where $tag is + the first 8 chars of the fsid. +- If the container image doesn't exist yet when you run cstart for the + first time, it is built with cpatch. + +There are a few advantages here: + +- The cluster is a "normal" cephadm cluster that looks and behaves + just like a user's cluster would. In contract, vstart and teuthology + clusters tend to be special in subtle (and not-so-subtle) ways. + +To start a test cluster:: + + sudo ../src/cstart.sh + +The last line of this will be a line you can cut+paste to update the +container image. For instance:: + + sudo ../src/script/cpach -t quay.io/ceph-ci/ceph:8f509f4e + +By default, cpatch will patch everything it can think of from the local +build dir into the container image. If you are working on a specific +part of the system, though, can you get away with smaller changes so that +cpatch runs faster. For instance:: + + sudo ../src/script/cpach -t quay.io/ceph-ci/ceph:8f509f4e --py + +will update the mgr modules (minus the dashboard). Or:: + + sudo ../src/script/cpach -t quay.io/ceph-ci/ceph:8f509f4e --core + +will do most binaries and libraries. Pass ``-h`` to cpatch for all options. + +Once the container is updated, you can refresh/restart daemons by bouncing +them with:: + + sudo systemctl restart ceph-`cat fsid`.target + +When you're done, you can tear down the cluster with:: + + sudo ../src/ckill.sh # or, + sudo ../src/cephadm/cephadm rm-cluster --force --fsid `cat fsid` -- 2.39.5