cephadm: bail out early during adopt if data dir not present

In case someone tries to run this again on an already adopted daemon...

Signed-off-by: Tim Serong <tserong@suse.com>

cephadm: add --force-start option to adopt

Current behaviour is to only start a newly adopted ceph daemon if it was
already running before the adopt.  Adding a --force-start option allows
the adopt command to start newly adopted daemons that weren't originally
running, to save the user having to manually invoke `systemctl start
ceph-$FSID@$DAEMMON.$ID`.

Signed-off-by: Tim Serong <tserong@suse.com>

cephadm: don't throw when disabling ceph-volume simple unit

When adopting OSDs, if a ceph-volume simple service is already disabled
(or otherwise missing) the previous implementation would raise an error,
thus killing the adopt.

Signed-off-by: Tim Serong <tserong@suse.com>

cephadm: handle adopting offline OSDSs

The current adopt behavior expects OSDs to be online, in order to read
/var/lib/ceph/osd/ceph-$ID/fsid.  To handle the case where OSDs
are offline, this change first checks to see if that file is present,
and if not, falls back to calling `ceph-volume lvm list` to see if
there's a matching OSD there, and if that doesn't work, it checks
/etc/ceph/osd/*.json to see if there's a matching old-style simple
OSD present.

For LVM OSDs, the only thing we need is the ODS's fsid; the remainer
of the adopt procedure "just works", as the various other files
in /var/lib/ceph/$FSID/osd.$ID are created by magic anyway when the
OSD is activated, so it doesn't matter if they're not present at
adoption time.

For simple (ceph-disk created) OSDs, we actually need all the files under
/var/lib/ceph/osd/ceph-$ID/ to be moved to /var/lib/ceph/$FSID/osd.$ID
so if a simple OSD is found, it's mounted first, so the existing
move_files() a bit further down around line 3200 continues to work.

Fixes: https://tracker.ceph.com/issues/45095
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #34562 from tchaikov/wip-jenkins-trigger

github: add "jenkins test classic perf" to jenkins commands list

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

github: add "jenkins test classic perf" to jenkins commands list

it's used for triggering perf test with classic OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19076 from jecluis/wip-mon-fix-osdmap-lec-trim

mon/OSDMonitor: allow trimming maps even if osds are down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34510 from aclamk/wip-rocksdb-open-close

kv/KeyValueDB: Making open/close operations complementary.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34228 from shonpaz123/add_notifications_documentation

doc/rgw: add documentation for bucket notifications filters and supported push endpoints

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #33975 from jbwyatt4/jbwyatt4-need-disk-space-info

doc: added space requirement to "Build Prerequisites"

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33901 from rs-fabrica/rados_man_page_options

doc/man/8/rados.rst: including all available options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34197 from varshar16/wip-doc-SubmittingPatches-url

doc/SubmittingPatches: Fix 'Fixes line' hyperlink

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34265 from shonpaz123/add_sc_doc

doc/rgw: add documentation for allowed storage classes for client SDKs

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34360 from gouthampacha/doc-fix

doc: fix broken link for cephfs kernel recommendations

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33806 from trociny/wip-mirror_snapshot_schedule-watcher

mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34262 from zhengchengyao/wip-deprecated-order

tools: remove all deprecated options from the help output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33365 from ifed01/wip-ifed-hybrid-alloc2

os/bluestore: introduce hybird allocator

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32704 from petrutlucian94/windows.2

common: Add dlfcn_compat.h for win32

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34336 from nSedrickm/nsedrickm-ceph-dev

mgr/dashboard: Proposed About Modal box

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34469 from votdev/issue_44991_pool_promql

monitoring: alert for pool fill up broken

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #34065 from tspmelo/wip-iscsi-advanced

mgr/dashboard: Prevent iSCSI target recreation when editing controls

Reviewed-by: Ricardo Marques <rimarques@suse.com>

common: Add dlfcn_compat.h for win32

We'll emulate the dlfcn.h interface using Windows functions, which
will allow loading Windows DLLs.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #34009 from tspmelo/wip-iscsi-auth

mgr/dashboard: Fix iSCSI's username and password validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33816 from rzarzynski/wip-msg-no-extra-alloc-in-secure

msg/async: fix unnecessary 4 kB allocation in secure mode.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34344 from jmolmo/rook_inventory

mgr/rook: Fix wrong data output when listing devices

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rbd_support: attempt to unregister watcher on mgr exit

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33989 from ifed01/wip-ifed-better-bluefs-expand

os/bluestore: open DB in read-only when expanding DB/WAL.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #33782 from aclamk/wip-alloc-aging-logging-simpler

os/bluestore: Simplified logic of sliding window.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #33434 from ifed01/wip-ifed-big-writes-deferred

os/bluestore: go deferred for 'big' writes.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #33715 from trociny/wip-pybind-rados-watch

pybind/rados: add watch object bindings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33982 from cbodley/wip-cls-cmpomap

cls/cmpomap: add cls module for CMPXATTR-like functionality in omap

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

doc/SubmittingPatches: Fix 'Fixes line' hyperlink

Signed-off-by: Varsha Rao <varao@redhat.com>

options: add cmpomap to osd_class_load_list

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: add ceph_test_cls_cmpomap

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/cmpomap: add cls module for CMPXATTR-like functionality in omap

provides a cmp_vals() op similar to librados' omap_cmp(), but with full
support for the comparison modes and operations from cmpxattr(). if any
of the requested key/value comparisons are unsuccessful, the op fails
with -ECANCELED

cmp_vals() can be composed with other ops like set_omap_vals() to make
make them conditional on the successful comparison of all existing values

also provides cmp_set_vals() and cmp_rm_keys(), which apply a mutation
to only the keys that compare successfully. these enable batch
operations where, unlike cmp_vals() + set_omap_vals(), you don't want
one failed comparison to prevent operations on unrelated keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: add cls_cxx_map_get_vals_by_keys

add an objclass api for CEPH_OSD_OP_OMAPGETVALSBYKEYS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: expose cls_cxx_map_get_val in header

this was defined in osd/objclass.cc but not declared in objclass/objclass.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: add missing != operator for bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: comparison operators take const bufferlist&

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson: add HybridAllocator to alienstore build.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: improve adjustent extents merge in hybrid allocatoc

Bitmap counterpart is inspected for adjustent free extents when
releasing extent and trying to add it to AVL tree.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: remove HybridAllocator::init_add_free method.

It's a 100% replica of AvlAllocator one which is inherited.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: configure hybrid allocator threshold to fallback.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add hybrid (avl+bitmap) allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: do not require min_length to be power of two for bmap
allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: extend AvlAllocator interface.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

tests/test_bluestore_types: show AvlAllocator's range_seg_t size.

Extend corresponding test case to be aware of the struct's sizeof.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: make AvlAllocator overridable.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: intdoduce spillover mechanics to AvlAllocator.

This to be utilized in AvlAllocator's descendant class(es) to be able
to limit amount of entries allocator tracks. Hence bringing a cap for
RAM used by the allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34455 from jtlayton/wip-44963

client: only set MClientCaps::FLAG_SYNC when flushing dirty auth caps

Merge pull request #33761 from jtlayton/wip-44441

doc: document wsync and nowsync mount options in mount.ceph(8)

Merge pull request #34231 from majianpeng/bluefs-fix-invalidate-cache-bug

os/bluestore: don't foreget invalidate cache of FileReaderBuffer.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #34272 into master

* refs/pull/34272/head:
mgr/PyModuleRegistry: ignore 'obsolete' modules
qa/suites/fs/upgrade: start with msgr2
qa/suites/fs/upgrade: s/mimic/nautilus/
qa/suites/upgrade: whitelist when orchestrator_cli can't be loaded
qa/standalone/mon/msgr-v2-transition: remove test
mon/MgrMonitor: disable orchestrator_cli if >= octopus
qa/suites/ugprade: remove mimic -> pacific upgrade tests
qa/standalone/mon/misc: update TEST_mon_features
mon/MgrMap: use always_on_modules from most recent release
mon/MonCommands: require-osd-release pacific
qa/suites/rados/thrash-old-clients: adjust octopus package list for el7
doc/dev/release-kickoff-checklist: include ceph-build change
qa/workunits/cephtool/test.sh: fix require-osd-release test
mon/MgrMonitor: define mgr modules for pacific
doc/dev/release-checklists: generalize
doc/dev/release-kickoff-checklist: quotes
qa/suites/rados/thrash-old-clients/1-install/octopus
qa/releases/pacific.yaml
mds/cephfs_features: CEPHFS_CURRENT_RELEASE
doc/dev/release-kickoff-checklist: a few other items
osd: remove redundant check
mon/OSDMonitor: make pacific changes
mon/mon_types: add mon pacific feature
include/ceph_features: define SERVER_PACIFIC
include/ceph_features: fix whitespace
mon: persistent INCOMPAT_PACIFIC item
ceph-volume: __release__ == pacific
script: update backport scripts for now-released octopus
common: pacific release constants
doc/dev/release-kickoff-checklist
src/ceph_release: pacific, 16, dev
CMakeLists: 16.0.0

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34525 from Dmole/patch-5

doc: Update index.rst

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32297 from varshar16/wip-mgr-nfs-ganesha-exports

Create and delete exports for nfs ganesha with mgr volume

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/PyModuleRegistry: ignore 'obsolete' modules

Old modules may be in the mgrmap (and always_on) but no longer exist.  Do
not try to load those or raise errors about them.

Signed-off-by: Sage Weil <sage@redhat.com>

Update index.rst

clean up "only only"

Signed-off-by: Tim <elatllat@gmail.com>

Merge pull request #34348 from ifed01/wip-ifed-fix-enforce

os/bluestore: apply bluestore_debug_enforce_settings when choosing mi…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34421 from ifed01/wip-ifed-kill-buf-in-read

os/bluestore: kill buf param in BlueFS::read

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34468 from tchaikov/wip-mgr-telegraph

mgr/telegraf: catch FileNotFoundError exception

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #34084 from sebastian-philipp/cephadm-host-ls-offline

mgr/cephadm: enhance `host ls`

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34382 from mgfritch/cephadm-nfs-grace

cephadm: add config required for ganesha grace

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #34488 from tchaikov/wip-dmclock

dmclock: pick up change to s/uint/unsigned/

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

kv/KeyValueDB: Made open/close a complementary operations.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/KeyValueDB: Test for idempotency of open/close

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34508 from tchaikov/wip-45022

selinux: add "type http_cache_port_t" in require section

Reviewed-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #34494 from idryomov/wip-krbd-unmap-spacing

tests: update unmap.t for table spacing changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

selinux: add "type http_cache_port_t" in require section

this addresses the regression introduced by
611a2a7ce1713cf92797b19d8a9a93a7dfc68fa5

Fixes: https://tracker.ceph.com/issues/45022
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34499 from ceph/wip-rtd

doc: enable Read the Docs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34487 from tchaikov/wip-test-cronjob-py3

qa/tests: pass "-t 2" to teuthology-suite when testing non-master bra…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

doc: add in-doc search from read the docs

readthedocs-sphinx-search features better user experience than the
builtin search offered by sphinx

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: use plantweb as fallback of sphinx-ditaa

RTD does not support installing system packages, the only ways to install
dependencies are setuptools and pip. while ditaa is a tool written in
Java. so we need to find a native python tool allowing us to render ditaa
images. plantweb is able to the web service for rendering the ditaa
diagram. so let's use it as a fallback if "ditaa" is not around.

also start a new line after the directive, otherwise planweb server will
return 500 at seeing the diagram.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add CACHEINODE and EXPORT_DEFAULTS sections

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add config required for ganesha grace

adds minimal config for ganesha-rados-grace

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add userid during grace add/remove

ganesha-rados-grace needs a rados userid to add/remove from the grace db

Mar 27 10:05:25 host1 bash[11684]: rados_connect: -13
Mar 27 10:05:25 host1 bash[11684]: Can't connect to cluster: -13

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34502 from ceph/kalebskeithley-patch-1-2

Update ceph.te

Merge pull request #34484 from cbodley/wip-sub-spawn

spawn: update submodule

Update ceph.te

Fixes: https://tracker.ceph.com/issues/45022
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

doc/conf.py: exclude pybindings docs from build for RTD

because it'd difficult to prepare (dummy) librados,libcephfs and librbd for
their python bindings in the building environment offered by Read the Docs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34314 from ishanrai05/html-cleanup

mgr/dashboard: Clean up HTML

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34223 from liewegas/cephadm-dashboard-grafana-ssl-verify

mgr/cephadm: disable dashboard's grafana cert ssl_verify if we generate it

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34454 from mgfritch/cephadm-osd-fsid-test

qa/workunits/cephadm/test_cephadm.sh: add fsid to ceph-volume cmds

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34249 from mgfritch/cephadm-nfs-common-config

mgr/cephadm: nfs common config

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #34485 from mgfritch/cephadm-no-func

cephadm: show error when no command is specified

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Proposed About Modal box
This commit implements proposed about modal box
Fixes : https://tracker.ceph.com/issues/35693
Signed-off-by: Ngwa Sedrick Meh <nsedrick101@gmail.com>

mgr/dashboard: Prevent iSCSI target recreation when editing controls

Also prevents updating target auth, when this was not modified.

Fixes: https://tracker.ceph.com/issues/44659
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove duplicate code in iscsi.py

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34411 from votdev/issue_44589

mgr/dashboard: lint error on plugins/debug.py

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

tests: update unmap.t for table spacing changes

Introduced in 4d42b4c5a0ed ("common/TextTable: default to 2 spaces
separating columns") and 41f003518a07 ("common/TextTable: only pad
between columns").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34452 from rhcs-dashboard/fix-44923-master

mgr/dashboard: use FQDN for failover redirection

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34471 from idryomov/wip-rbd-fio-rstrip

qa/tasks/rbd_fio: unbreak after the conversion from StringIO

Reviewed-by: Kefu Chai <kchai@redhat.com>

readthedocs: add .readthedocs.yml

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34408 from dillaman/wip-44727

rbd-mirror: improved replication statistics

Reviewed-by: Mykola Golub <mgolub@suse.com>

dmclock: pick up change to s/uint/unsigned/

to address the FTBFS with musl libc which does not define uint.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: pass "-t 2" to teuthology-suite when testing non-master branches

-t is an alias of "--teuthology-branch" which defaults to master. we
have a "py2" branch in teuthology. that branch is compatible with
python2, and the teuthology-worker is executed in a Python2 environment,
so all qa suites tested using "py2" branch of teuthology should work
fine as long as they are Python2 compatible. currently, all LTS branches
are compatible with Python2, and only master is also compatible with
Python3. but we plan to migrate master and octopus' qa suites to
python3 to ease the pain of maintainance in long term.

to achieve this goal, in this change, all non-master branches are now
tested using the "py2" teuthology branch.

once the python3 compatible changes are backported to octopus, "-t 2"
options will be removed from the command lines testing octopus.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34264 from tchaikov/wip-qa-py3

qa/tasks: be python3 compatible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34284 from liewegas/followon-34266-cleanup

mgr/DaemonServer: add missing ceph_abort

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs/upgrade: start with msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: show error when no command is specified

Traceback (most recent call last):
  File "./cephadm", line 4452, in <module>
    if args.func != command_check_host:
AttributeError: 'Namespace' object has no attribute 'func'

Signed-off-by: Michael Fritch <mfritch@suse.com>