cephadm: Check for pool existance for iscsi And NFS

Currently both iscsi and NFS require pools to be specified when they are
deployed. However, we don't actaully check these pools exist. Leading to
broken containers.

This patch uses the rados client that is part of the mgrmodule to check
that the specified pool exists. As we need to check in 2 different
daemons, a helper method:

   def _check_pool_exists(self, pool, service_name):

was added to `cephadm/module.py`.

Fixes: https://tracker.ceph.com/issues/45161
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #34430 from yison/rwl-image-writeback-cache-seq4

rbd/cache: Replicated Write Log core codes part 4

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34669 from yuvalif/remove_queue_dependency

cls/2pc_queue: remove the dependency of cls_2pc_queue with cls_queue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34664 from yuvalif/clarify_topic_creation_format

rgw/doc: clarify topic creation format

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34613 from cbodley/wip-44661

rgw: use DEFER_DROP_PRIVILEGES flag unconditionally

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #34621 from mgfritch/cephadm-test-cleanup

qa/workunits/cephadm/test_cephadm.sh: add CLEANUP var

Merge pull request #34568 from sebastian-philipp/orch-upgrade-usage

mgr/orch: upgrade: better input validation

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34565 from SUSE/wip-fix-45095

cephadm: handle adopting offline OSDs

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34442 from tspmelo/wip-fix-rgw-bucket-checkbox

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33479 from simon-rock/simon_split_stray

mds : enable fragmentation of stray dir when conditions are met

Merge pull request #34561 from kotreshhr/ceph-fs-status-output

mgr/status: Fix "ceph fs status" json format writing to stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34546 from varshar16/wip-flake8-cephfs-shell

cephfs-shell: Fix multiple flake8 errors

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34534 from ajarr/fix-43061

mon/FSCommands: Fix 'add_data_pool' command and 'fs new' command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34164 from ukernel/wip-44382

qa/cephfs: update TestClusterFull:test_barrier

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #33576 from lxbsz/fuse

unshare-ns-mount: add isolating netns for mount helper support

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #34121 from ADI10HERO/flake8_fix1

cephfs-shell: Fix flake8 errors (F841, E302, E502, E128, E305 and E222)

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #34531 from lxbsz/libfuse3

ceph-fuse: link to libfuse3 and pass `-o big_writes` to libfuse if libfuse < 3.0.0

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34274 from ukernel/wip-44801

client: reset requested_max_size if file write is not wanted

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

cls/2pc_queue: remove the dependency of cls_2pc_queue with cls_queue

both queues share the same code base, however, cls_2pc_queue should work
even if cls_queue objectclass is not loaded into the osd

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #34612 from zdover23/wip-doc-tshooting-ssh-errors-44905

doc/cephadm: Add troubleshooting guidance for ssh connection failures. [cephadm]

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34661 from wjwithagen/wjw-fix-rgw_bucket_sync_cache.h

rgw: remove noexcept from function definition

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34614 from theanalyst/rgw/reshard-num-check

rgw: reshard: skip stale bucket id entries from reshard queue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: remove noexcept from function definition

Clang complains:
src/rgw/rgw_bucket_sync_cache.h:88:3: error: exception specification of explicitly defaulted copy co
nstructor does not match the calculated one
  Handle(const Handle&) noexcept = default;
  ^
1 error generated.

And a reference that I found for this:
    https://github.com/mapnik/mapnik/issues/3274
Suggesting that the noexcept is inherited from the first definition.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw/doc: clarify topic creation format

also, fix some typos, and 'push-endpoint' parameter

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

qa/cephfs: add test case for fragmenting stray dirs

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #34605 from tchaikov/wip-45113

qa/suites/*/rados_cls_all.yaml: load all classes

Reviewed-by: Neha Ojha <nojha@redhat.com>

mds: automatically fragment stray dirs

Signed-off-by: Simon Gao <simon29rock@gmail.com>

Merge pull request #34655 from ceph/wip-yuriw-remove-wip-master

qa/tests: removed tests 'client-upgrade-octopus-pacific'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: removed tests 'client-upgrade-octopus-pacific'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34651 from ceph/wip-yuriw-crontab-master

qa/tests: fixed typo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: fixed typo

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34649 from ceph/wip-yuriw-crontab-master

qa/tests: added new clients upgrade tests `client-upgrade-octopus-pacific`

qa/tests: added new clients upgrade tests

`client-upgrade-octopus-pacific` for future 'pacific' release

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34642 from ceph/wip-yuriw-octopus-client-upgrade-octopus-pacific

qa/tests: initial checkin for clients upgrades tests for future relea…

qa/tests: initial checkin for clients upgrades tests for future release of pacific

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34608 from trociny/wip-journal-append-stripe-width

cls/journal: use EC pool stripe width for padding appends

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/rgw: allow 'ceph' user to read vault token

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/orch: upgrade: better input validation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34636 from tchaikov/wip-vstart-runner-py3

qa/tasks/vstart_runner,doc: be compatible with python3

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/fuse_mount.py: by python3 compatible

pass `StringIO()` to capture stdout whose value will be interpreted as a
string later on

Signed-off-by: Kefu Chai <kchai@redhat.com>

Add troubleshooting guidance for ssh connection failures.

For more thorough information about this commit, see:
https://tracker.ceph.com/issues/44905

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Michael Fritch <mfritch@suse.com>
ibid

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Added mgfritch's suggestion regarding checking to see if the public key
is listed in the authorized_keys file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

pybind/mgr/dashboard/run-backend-api-tests.sh: use python3 by default

since vstart_runner.py is now py3, let's use python3 by default

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner.py: be python3 compatible

differentiate `str` and `bytes` instances, and drop python2 support from
vstart_runner.py, as we've moved to python3 already

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/*/rados_cls_all.yaml: load all classes

the intention to add the whitelist was to test "sdk" class, but if we
add new classes to the list, and add tests exercising them, the tests
fail if we fail to update these `rados_cls_all.yaml` accordingly.

so in this change, the list is now '*' which allows OSD to load all
classes found in the specified directory

Fixes: https://tracker.ceph.com/issues/45113
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/developer_guide: use python3 to launch vstart_runner.py

as vstart_runner.py is compatible with python3, and so is the tests
under `qa/` directory.

and use `pip` to install teuthology, simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: make MDCache::scan_stray_dir() handle dir fragmentation

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't add null dentry to frozen stray dir

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: auth pin stray dir when purging stray inode

After stray inode gets added to purge queue, mds deletes stray dentry.
MDS needs to make sure stray dir is not being fragmented.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

ceph: allow 'dirfrag split' asok command to operate on stray dirs

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #34432 from optimistyzy/47_upstream

os/bluestore/NVMEDevice.cc: use slist for tracking mempool

Reviewed-by: Kefu Chai <kchai@redhat.com>

NVMEDevice: Remove the performance counter

Compared with other approach (KernelDevice, iouring), it does not
have such feature, so remove it to simplify the code and improve
the performance. Because, each thread will call the logger and calculate
the time, it is very time consuming. And the upper layer in bluestore has
the feature to track each I/O task.

And it is not readable if we add ifdef configure to enable this feature,
so in my mind that I suggest removing this latency track. And the
latency benefit demonstrated by SPDK is already approved, so we do not
need to tracking every I/O time when invoking SPDK driver.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

NVMEDevice.cc: Remove the data_buf_mempool

Use the linked list is enough.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

librbd: add aio_flush test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_flush

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34615 from dillaman/wip-45072

rbd-mirror: image replayer stop might race with instance replayer shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

ceph-fuse: link to libfuse3 if fuse3 is installed

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>

rbd-fuse: remove FUSE_CAP_BIG_WRITES when linking to libfuse3

The big-writes is enabled as default from 3.0.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>

rbd-fuse: switch to use utimens instead

The utime is deprecated.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34576 from jdurgin/wip-object-corpus

ceph-object-corpus: update to octopus

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34535 from changchengx/refine_build_liburing

cmake: refactor liburing support

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-fuse: enable the big writes as default

When the libfuse version < 3, without the big_writes options enabled
the write perf is very poor.

Fixes: https://tracker.ceph.com/issues/44885
Signed-off-by: Xiubo Li <xiubli@redhat.com>

test_cephadm.sh: remove unmaintained test_cephadm.sh

the script has become unmaintained and duplicates logic found in
qa/workunits/cephadm/test_cephadm.sh

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: add CLEANUP var

preserve the current test state during EXIT:
`CLEANUP=false ../qa/workunits/cephadm/test_cephadm.sh`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34584 from bk201/wip-45108

mgr/test_orchestrator: fix error when listing services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32951 from matthewoliver/cephadm_rgw_ssl

cephadm: add RGW SSL support

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34410 from lxbsz/mds_dirty_parent

tools/cephfs: add accounted_rstat/rstat when building file dentry

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #33928 from mchangir/cephfs-journal-tool-cannot-set-dry_run-arg

cephfs-journal-tool: correctly parse --dry_run argument

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

cmake: use cmake module to integrate liburing

1. WITH_LIBURING is used to set HAVE_LIBURING to decide
   use liburing in KernelDevice or not.
2. WITH_SYSTEM_LIBURING is to choose use system installed
   liburing or build the liburing from source code.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #32612 from rishabh-d-dave/fs-qa-UnicodeDecodeError

test_cephfs_shell: fix UnicodeDecodeError

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33169 from moningchao/master

client: fix blacklist removal failed

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #33711 from lxbsz/qa_umount_failed

qa/tasks/cephfs: umount the mountpoints when tearDown

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34533 from fanjunwei/fix_fuse_bug

client: fix ceph-fuse parse command line failed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rbd-mirror: track in-flight start/stop/restart in instance replayer

The shut down waits for in-flight ops to complete but the
start/stop/restart operations were previously not tracked. This
could cause a potential race and crash between an image replayer
operation and the instance replayer shutting down.

Fixes: https://tracker.ceph.com/issues/45072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: reshard: skip stale bucket id entries from reshard queue

If we encounter a reshard queue entry that has an older ID compared to the
bucket's current ID, it'd mean that some other process or a manual reshard has
already processed this entry, skip processing the entry this time. An
alternative is to verify the num_shards that we have in queue >= the current
shards, but this would mean that we may reshard a recently manual resharded
bucket again which might not be intended

Fixes: https://tracker.ceph.com/issues/45134
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: use DEFER_DROP_PRIVILEGES flag unconditionally

this flag doesn't take effect unless its set before global_init()

Fixes: https://tracker.ceph.com/issues/44661
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: combine calls to global_init/pre_init

calling global_init() without 'run_pre_init=false' will call
global_pre_init() for us

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34601 from yuriw/wip-yuriw-crontab-master

qa/tests: added octopus-p2p upgrade tests to the mix

qa/rgw: add setuser/group to test DEFER_DROP_PRIVILEGES

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rbd-mirror: switch to common C_TrackedOp context class

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: add helper C_TrackerOp context class

This wraps the functionality of starting and finishing a tracked op
into the standard context interface.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34604 from tchaikov/wip-cmake-fuse

cmake: check for FUSE version and cleanups

Reviewed-by: Xiubo Li <xiubli@redhat.com>

cls/journal: use EC pool stripe width for padding appends

to avoid ECSubRead ops to reencode a partially updated stripe.

Signed-off-by: Mykola Golub <mgolub@suse.com>

objclass: add method to get pool stripe width

Signed-off-by: Mykola Golub <mgolub@suse.com>

cmake: detect fuse3 as well

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for FUSE version

and refactor FindFUSE.cmake a little bit to conditionalize OSX support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against FUSE::FUSE

instead of specifying the include directory and linkage separately

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename Findfuse.cmake to FindFUSE.cmake

so its name is consistent with upstream name, and with the variables
exposed by FindFUSE.cmake.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: added octopus-p2p upgrade tests to the mix

Signed-off-by: Yuri Weinstein <yweinste@gmail.com>

Merge pull request #34573 from dillaman/wip-44938-2

test: tx-only peer from heartbeat can race w/ CLI

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34346 from jschmid1/osd_id_claims

mgr/cephadm: add support for osd_id_claims

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34517 from tchaikov/wip-qa-dashboard-telemetry

qa/tasks/mgr/dashboard: set last_opt_revision when setting enabled

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34210 from kotreshhr/cephfs-subvolume-info

mgr/volumes: Add interface to get subvolume metadata

mgr/test_orchestrator: fix error when listing services

- Adapting attribute change of ServiceSpec.
- Update dummy data.

Fixes: https://tracker.ceph.com/issues/45108
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard: set last_opt_revision when setting enabled

telemetry requires user to re opt-in when opt_revision is bumped up

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: Fixes some deserialization errors

Fixes errors when calling `from_json` of these classes:
- InventoryHost: parsing labels
- ServiceDescription: `last_refresh` and `created` fields should be parsed
  to datetime type.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

cephadm: bail out early during adopt if data dir not present

In case someone tries to run this again on an already adopted daemon...

Signed-off-by: Tim Serong <tserong@suse.com>

cephadm: add --force-start option to adopt

Current behaviour is to only start a newly adopted ceph daemon if it was
already running before the adopt.  Adding a --force-start option allows
the adopt command to start newly adopted daemons that weren't originally
running, to save the user having to manually invoke `systemctl start
ceph-$FSID@$DAEMMON.$ID`.

Signed-off-by: Tim Serong <tserong@suse.com>

cephadm: don't throw when disabling ceph-volume simple unit

When adopting OSDs, if a ceph-volume simple service is already disabled
(or otherwise missing) the previous implementation would raise an error,
thus killing the adopt.

Signed-off-by: Tim Serong <tserong@suse.com>

cephadm: handle adopting offline OSDSs

The current adopt behavior expects OSDs to be online, in order to read
/var/lib/ceph/osd/ceph-$ID/fsid.  To handle the case where OSDs
are offline, this change first checks to see if that file is present,
and if not, falls back to calling `ceph-volume lvm list` to see if
there's a matching OSD there, and if that doesn't work, it checks
/etc/ceph/osd/*.json to see if there's a matching old-style simple
OSD present.

For LVM OSDs, the only thing we need is the ODS's fsid; the remainer
of the adopt procedure "just works", as the various other files
in /var/lib/ceph/$FSID/osd.$ID are created by magic anyway when the
OSD is activated, so it doesn't matter if they're not present at
adoption time.

For simple (ceph-disk created) OSDs, we actually need all the files under
/var/lib/ceph/osd/ceph-$ID/ to be moved to /var/lib/ceph/$FSID/osd.$ID
so if a simple OSD is found, it's mounted first, so the existing
move_files() a bit further down around line 3200 continues to work.

Fixes: https://tracker.ceph.com/issues/45095
Signed-off-by: Tim Serong <tserong@suse.com>

doc: add object corpus to release checklist

Signed-off-by: Josh Durgin <jdurgin@redhat.com>