crimson/osd: improve the const correctness

it should suffice to pass a `const OSD&` to the ctor of `OsdStatusHook`

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support "dump_pgstate_history" command

it is required by the "ceph_test_admin_socket_output" test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: make injected_args optional

this option for `injectargs` command is optional, so we should not
crash if it does not exist.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: do not check for experimental/filesystem anymore

seastar is only compatible with C++17 and C++20, so no need to check for
<experimental/filesystem> anymore. the C++ standard library should be
C++17 compatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: call config::update_legacy_vals()

so the legacy vals are populated with the changes received from conffile.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: make update_legacy_vals() public

so crimson can call it in `ConfigProxy::parse_config_files()` after calling
`get_config().parse_buffer()`. so the legacy vals are populated with the
changes received from conffile.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36842 from cyx1231st/wip-fix-crimson-heartbeat

crimson/osd: fix unexpected connection markdown in heartbeat

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: cleanup, reset event is synchronously called

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #36807 from badone/wip-tracker-47002-legacy_vals-keyring

common/config: Make sure legacy_values are updated

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36535 from hualongfeng/fixed_unused_bluefs

os/bluestore: do not update used_blocks with bluefs_extents if bluefs is not available

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix unexpected connection markdown in heartbeat

Pass reference when log Heartbeat::Connection instance, or the
destructor will be called incorrectly, and the conn be marked down
unexpectedly.

Also, the replacing conn is actually connected during replacement-reset
event.

Fixes: https://tracker.ceph.com/issues/47124
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

common/config: Make sure legacy_values are updated

Must be updated after final change to 'values'

Fixes: https://tracker.ceph.com/issues/47002
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36827 into master

* refs/pull/36827/head:
qa/suites/smoke: remove packages on test completion

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36801 from yuriw/wip-yuriw-crontab-master

qa/tests: clients' upgrades for older releases must use "-t py2"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tests: clients' upgrades for older releases must use "-t py2"

Fixes: https://tracker.ceph.com/issues/47120
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #36773 into master

* refs/pull/36773/head:
mgr/volumes: Prevent subvolume recreate if trash is not-empty
mgr/volumes: Disallow subvolume group level snapshots
mgr/volumes: Add test case to ensure subvolume is marked
mgr/volumes: handle idempotent subvolume marks
mgr/volumes: Tests amended and added to ensure subvolume trash functionality
mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume
mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash
mgr/volumes: maintain per subvolume trash directory
mgr/volumes: make subvolume_v2::_is_retained() object property

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: Prevent subvolume recreate if trash is not-empty

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Disallow subvolume group level snapshots

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Add test case to ensure subvolume is marked

with the vxattr ceph.dir.subvolume set to true.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: handle idempotent subvolume marks

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: Tests amended and added to ensure subvolume trash functionality

Amended a few test cases to ensure created subvolumes and snaps
are removed, and trash stays empty at the end of the test.

Further added one test case for create errors in a retained
v2 subvolume, to ensure metadata is sane, and created incarnation
is not present.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume

This enables snapshot scaling of subvolumes, and hence prevents
renaming, hardlinking etc. outside the subvolume.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash

For v2 subvolumes without any snapshots or pending purges
move the subvolume to trash, for the purge operation.

When removing only incarnations, leverage symlink based purge
job creation, to avoid renaming the incarnation out of the
subvolume.

For create errors, where the subvolume was retained, just remove
the created incarnation directory.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: maintain per subvolume trash directory

PR https://github.com/ceph/ceph/pull/36472 introduces changes
that disallow nested nested snapshots in a subtree (subvolume)
and renames across subvolumes. This effect asynchronous purge
in mgr/volumes as subvolume are moved to a trash directory for
asynchronous deletion by purge threads.

To workaround this, start maintaining a subvolume specific
trash directory. Use the trash directory as an index to the
subvolume specific trash directory entry.

This changes subvolume deletion logic which currently relies
on `--retain-snapshots` flag to decide if the subvolume user
directory should get purged or the subvolume base directory
itself. Deleting a subvolume moves the user facing directory
to its specific trash directory. Purge threads take care of
deleting user facing directories (in trash) and the subvolume
base directory if required (when certain conditions are met).

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: make subvolume_v2::_is_retained() object property

... and drop the leading underscore.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/suites/smoke: remove packages on test completion

A new teuthology PR [1] skips package cleanup by default. Use the
lightweight smoke suite to ensure that package cleanup continues to
work.

[1] https://github.com/ceph/teuthology/pull/1553

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36805 into master

* refs/pull/36805/head:
pybind/mgr/volumes: add global lock debug

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36774 from dillaman/wip-46669-2

librbd: drop deferred AsioEngine destruction

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36582 from rhcs-dashboard/fix-redirected-URL

mgr/dashboard: redirect to original URL after successful login

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

pybind/mgr/volumes: add global lock debug

Fixes: https://tracker.ceph.com/issues/47149
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36817 from sebastian-philipp/cephadm-pep8-tests

mgr/cephadm: PEP8tify some tests

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36315 from p-se/fix-monitoring-images-ceph-salt

cephadm: deploying of monitoring images partially broken

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #36571 from pcuzner/cephadm-tox-update

cephadm: remove py2 from tox tests

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/cephadm: PEP8tify some tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36794 from sebastian-philipp/cephadm-pep8-cephadmservice.py

mgr/cephadm: PEP8tify cephadmservice.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36761 from wjwithagen/wjw-fix-dashboard-package.json

dashboard: Fix building build shell script

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36527 from neha-ojha/wip-whitelist-blacklist

Replace 'whitelist' and 'blacklist' language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36509 from pcuzner/cephadm-add-host-facts

cephadm: add gather_facts command

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36793 from sebastian-philipp/cephadm-pep8-schedule.py

mgr/cephadm: PEP8tify schedule.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36757 from mgfritch/cephadm-raw-str-warn

mgr/cephadm: use raw string during `re.search`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36787 from tchaikov/wip-47031

crimson/osd: drop misdirected ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: PEP8tify cephadmservice.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify schedule.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36738 from ShraddhaAg/cephadm-archive-logs

qa/tasks/cephadm.py: add ceph logs directory in job's info.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: redirect to original URL after successful login

Fixes: https://tracker.ceph.com/issues/45067
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit b5359a155c6ff7e92ab6c6ed16f7ff9fc76c76ea)

crimson/osd: drop misdirected ops

see also `PrimaryLogPG::do_op()`, we should ignore the ops hitting us if
we are not supposed to serve them. this happens when the client is using
a stale osdmap.

Fixes: https://tracker.ceph.com/issues/47031
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36778 from tchaikov/wip-cmake-fio

cmake: silence "detached HEAD" warning

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

cmake: silence "detached HEAD" warning

git complains when checking out a tag in "detached HEAD", like:

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them
...

but this does not help, as, in general, we don't hack fio in Ceph,
so disable this warning. and also clone the repo in shallow mode
for the same reason -- we don't care about the whole history of
fio repo. we just use it for testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36336 from pcuzner/ceph-volume-lsm

ceph-volume: add libstoragemgmt support

cephadm: remove py2 from test_adoption

Since cephadm is py3 based, and py2 is EOL this patch
removes the py2 test iteration from test_adoption.sh

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #36684 into master

* refs/pull/36684/head:
qa/tasks/nfs: Test mounting of export created with nfs command
qa/tasks/nfs: Add helper method to check nfs cluster status
qa/tasks/nfs: Cleanup created filesystem
qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: support reverting to legacy blacklist op via neorados API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove racially insensitive terms

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

librbd: RESTORE_SOURCE_WHITELIST -> ALLOWED_RESTORE_SOURCES

Signed-off-by: Neha Ojha <nojha@redhat.com>

journal, librbd: whitelisted_laggy_clients -> ignored_laggy_clients

Signed-off-by: Neha Ojha <nojha@redhat.com>

*: convert more blacklist to blocklist

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: convert WHITELIST to ALLOWLIST

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

*: convert EBLACKLISTED to EBLOCKLISTED

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/librados_test_stub/TestMemIoCtxImpl.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/krbd/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/multimds: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librbd/managed_lock/test_mock_BreakRequest.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librados_test_stub/TestRadosClient.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon/ConfigMonitor: handle renamed config options

Signed-off-by: Sage Weil <sage@newdream.net>

librados: fall back to legacy blacklist command if blocklist gets EINVAL

I would have expected ENOSYS or EOPNOTSUPP or similar, but the mon returns
EINVAL on an unrecognized command

Signed-off-by: Sage Weil <sage@newdream.net>

misc language changes: whitelist -> ignore etc

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: whitelisted_laggy_clients -> allowlisted_laggy_clients

Signed-off-by: Sage Weil <sage@newdream.net>

osd/OSDCap: whitelisted -> allowed

Signed-off-by: Sage Weil <sage@newdream.net>

qa: log-whitelist -> log-ignorelist

Signed-off-by: Sage Weil <sage@newdream.net>

blacklist -> blocklist

Signed-off-by: Sage Weil <sage@newdream.net>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #36394 into master

* refs/pull/36394/head:
qa: add method run ceph cluster command with better interface
cephfs: allow only "good" characters

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36772 from neha-ojha/wip-47119

qa/suites/perf-basic/objectstore: remove filestore

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

librbd: drop deferred AsioEngine destruction

Commit 90bd1d7a857c0f3c57bda60975f58f9859940185 removed the need to
defer deletion of the AsioEngine but missed removing the original
code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/perf-basic/objectstore: remove filestore

There is no point testing performance on filestore anymore.
It has already been removed from objectstore backends in rados/perf.

Fixes: https://tracker.ceph.com/issues/47119
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36760 from tchaikov/wip-qa-optional-args

qa/tasks/workunit: allow passing optional args to workunit

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36677 from bk201/wip-46986

mgr/dashboard: adjust cross button color in table filter chips

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/nfs: Test mounting of export created with nfs command

Fixes: https://tracker.ceph.com/issues/46989
Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Add helper method to check nfs cluster status

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Cleanup created filesystem

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #36737 from tchaikov/wip-crimson-more-ops

crimson/osd: implement append and omap-clear ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36347 from rhcs-dashboard/wip-40767-master

mgr/dashboard: Placeholders for Documenting REST API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36765 from sebastian-philipp/man-cephadm-list-no-detail

doc/man/cephadm: Add missing parameters for `ls`

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

crimson/os/cyanstore: handle OP_RMATTR

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: avoid creating a throw-away object

for better readabilty and performnce

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement rm-xattr op

it is tested by test_rados.py:TestIoctx.test_xattrs_op, so let's
support this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: check for DNE object and return ENOENT in read ops

* omap_get_keys()
  this change addresses the failure of
  test_rados.py:TestIoctx.test_get_omap_keys
* omap_get_vals_by_keys()
  this change addresses the failure of
  test_rados.py:TestIoctx.test_get_omap_vals_by_keys
* read()
  this change addresses the failure of
  test_rados.py:TestIoctx.test_write_ops

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor PGBackend::read() to pass os and op

prepare for the table based op lookup/execution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: remove object for testing after done with it

* add vertical spacing for better readability
* remove object for testing after done with it

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/cyanstore: handle OP_ZERO

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: update oi.size after truncating an object

* update oi.size if object size changes after the object is truncated
* do not add a truncate op to trasaction of the size of object does
  not change because of truncate op.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: return rval which is negative

a less-than-zero rval indicates an error, and should not be normalized
to 0 if allows_returnvec() evaluates to false. probably we need a better
way to return a negative error code which does not fall into any known
error. but at this moment, grab the last rval and return it if it is
less than zero, can be used as a short term solution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement cmp-ext op

it is tested by test_rados.py:TestIoctx.test_cmpext, so let's
support this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/cyanstore: handle OP_OMAP_CLEAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement omap-clear op

it is tested by test_rados.py:TestIoctx.test_clear_omap, so let's support
this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement append op

it is tested by test_rados.py:TestIoctx.test_append, so let's support
this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

CODEOWNERS: Add cephadm man page to @ceph/orchestrators

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/man/cephadm: Add missing parameters for `ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36755 from dillaman/wip-47050

librbd: flush op was not properly waiting for queued IO

Reviewed-by: Mykola Golub <mgolub@suse.com>