pybind/mgr: pytest: Unconditionally mock the rados objects

Always call `mock_ceph_modules()`. Otherwise we'd need to
split the tox run between modules that need to call
`mock_ceph_modules()` and those that don't

Fixes: https://tracker.ceph.com/issues/47141
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36794 from sebastian-philipp/cephadm-pep8-cephadmservice.py

mgr/cephadm: PEP8tify cephadmservice.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36761 from wjwithagen/wjw-fix-dashboard-package.json

dashboard: Fix building build shell script

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36527 from neha-ojha/wip-whitelist-blacklist

Replace 'whitelist' and 'blacklist' language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36509 from pcuzner/cephadm-add-host-facts

cephadm: add gather_facts command

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36793 from sebastian-philipp/cephadm-pep8-schedule.py

mgr/cephadm: PEP8tify schedule.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36757 from mgfritch/cephadm-raw-str-warn

mgr/cephadm: use raw string during `re.search`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36787 from tchaikov/wip-47031

crimson/osd: drop misdirected ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: PEP8tify cephadmservice.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify schedule.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36738 from ShraddhaAg/cephadm-archive-logs

qa/tasks/cephadm.py: add ceph logs directory in job's info.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: drop misdirected ops

see also `PrimaryLogPG::do_op()`, we should ignore the ops hitting us if
we are not supposed to serve them. this happens when the client is using
a stale osdmap.

Fixes: https://tracker.ceph.com/issues/47031
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36778 from tchaikov/wip-cmake-fio

cmake: silence "detached HEAD" warning

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

cmake: silence "detached HEAD" warning

git complains when checking out a tag in "detached HEAD", like:

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them
...

but this does not help, as, in general, we don't hack fio in Ceph,
so disable this warning. and also clone the repo in shallow mode
for the same reason -- we don't care about the whole history of
fio repo. we just use it for testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36336 from pcuzner/ceph-volume-lsm

ceph-volume: add libstoragemgmt support

cephadm: remove py2 from test_adoption

Since cephadm is py3 based, and py2 is EOL this patch
removes the py2 test iteration from test_adoption.sh

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #36684 into master

* refs/pull/36684/head:
qa/tasks/nfs: Test mounting of export created with nfs command
qa/tasks/nfs: Add helper method to check nfs cluster status
qa/tasks/nfs: Cleanup created filesystem
qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: support reverting to legacy blacklist op via neorados API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove racially insensitive terms

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

librbd: RESTORE_SOURCE_WHITELIST -> ALLOWED_RESTORE_SOURCES

Signed-off-by: Neha Ojha <nojha@redhat.com>

journal, librbd: whitelisted_laggy_clients -> ignored_laggy_clients

Signed-off-by: Neha Ojha <nojha@redhat.com>

*: convert more blacklist to blocklist

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: convert WHITELIST to ALLOWLIST

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

*: convert EBLACKLISTED to EBLOCKLISTED

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/librados_test_stub/TestMemIoCtxImpl.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/krbd/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/multimds: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librbd/managed_lock/test_mock_BreakRequest.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librados_test_stub/TestRadosClient.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon/ConfigMonitor: handle renamed config options

Signed-off-by: Sage Weil <sage@newdream.net>

librados: fall back to legacy blacklist command if blocklist gets EINVAL

I would have expected ENOSYS or EOPNOTSUPP or similar, but the mon returns
EINVAL on an unrecognized command

Signed-off-by: Sage Weil <sage@newdream.net>

misc language changes: whitelist -> ignore etc

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: whitelisted_laggy_clients -> allowlisted_laggy_clients

Signed-off-by: Sage Weil <sage@newdream.net>

osd/OSDCap: whitelisted -> allowed

Signed-off-by: Sage Weil <sage@newdream.net>

qa: log-whitelist -> log-ignorelist

Signed-off-by: Sage Weil <sage@newdream.net>

blacklist -> blocklist

Signed-off-by: Sage Weil <sage@newdream.net>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #36394 into master

* refs/pull/36394/head:
qa: add method run ceph cluster command with better interface
cephfs: allow only "good" characters

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36772 from neha-ojha/wip-47119

qa/suites/perf-basic/objectstore: remove filestore

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/perf-basic/objectstore: remove filestore

There is no point testing performance on filestore anymore.
It has already been removed from objectstore backends in rados/perf.

Fixes: https://tracker.ceph.com/issues/47119
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36760 from tchaikov/wip-qa-optional-args

qa/tasks/workunit: allow passing optional args to workunit

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36677 from bk201/wip-46986

mgr/dashboard: adjust cross button color in table filter chips

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/nfs: Test mounting of export created with nfs command

Fixes: https://tracker.ceph.com/issues/46989
Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Add helper method to check nfs cluster status

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Cleanup created filesystem

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #36737 from tchaikov/wip-crimson-more-ops

crimson/osd: implement append and omap-clear ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36347 from rhcs-dashboard/wip-40767-master

mgr/dashboard: Placeholders for Documenting REST API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36765 from sebastian-philipp/man-cephadm-list-no-detail

doc/man/cephadm: Add missing parameters for `ls`

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

crimson/os/cyanstore: handle OP_RMATTR

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: avoid creating a throw-away object

for better readabilty and performnce

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement rm-xattr op

it is tested by test_rados.py:TestIoctx.test_xattrs_op, so let's
support this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: check for DNE object and return ENOENT in read ops

* omap_get_keys()
  this change addresses the failure of
  test_rados.py:TestIoctx.test_get_omap_keys
* omap_get_vals_by_keys()
  this change addresses the failure of
  test_rados.py:TestIoctx.test_get_omap_vals_by_keys
* read()
  this change addresses the failure of
  test_rados.py:TestIoctx.test_write_ops

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor PGBackend::read() to pass os and op

prepare for the table based op lookup/execution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: remove object for testing after done with it

* add vertical spacing for better readability
* remove object for testing after done with it

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/cyanstore: handle OP_ZERO

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: update oi.size after truncating an object

* update oi.size if object size changes after the object is truncated
* do not add a truncate op to trasaction of the size of object does
  not change because of truncate op.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: return rval which is negative

a less-than-zero rval indicates an error, and should not be normalized
to 0 if allows_returnvec() evaluates to false. probably we need a better
way to return a negative error code which does not fall into any known
error. but at this moment, grab the last rval and return it if it is
less than zero, can be used as a short term solution.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement cmp-ext op

it is tested by test_rados.py:TestIoctx.test_cmpext, so let's
support this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/cyanstore: handle OP_OMAP_CLEAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement omap-clear op

it is tested by test_rados.py:TestIoctx.test_clear_omap, so let's support
this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement append op

it is tested by test_rados.py:TestIoctx.test_append, so let's support
this op in crimson as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

CODEOWNERS: Add cephadm man page to @ceph/orchestrators

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/man/cephadm: Add missing parameters for `ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36755 from dillaman/wip-47050

librbd: flush op was not properly waiting for queued IO

Reviewed-by: Mykola Golub <mgolub@suse.com>

dashboard: Fix building build shell script

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #36521 from tchaikov/wip-rgw-optional

rgw: hold reloader using unique_ptr

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #36707 from dillaman/wip-47007

rbd-mirror: peer setup can still race and fail creation of peer

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36471 from dillaman/wip-41394

librbd: using migration abort can result in the loss of data

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36380 from dillaman/wip-46754

rbd: retrieve global config overrides from the MONs

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36751 from tchaikov/wip-redirect-stderr

qa/tasks/ceph: create a log file before redirecting to it

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rados: quote optional args

otherwise they will be escaped by shell

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/workunit: allow passing optional args to workunit

* add comment to _run_tests()
* use `os.path.commonpath()` instead using string matching directly
  for matching given workunit spec with executables.
* allow passing optional args to workunit

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36696 from tchaikov/wip-cmake-boost-zlib

cmake: link against ZLIB if statically linked against Boost

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36710 from tchaikov/wip-ceph-pypath

ceph: correct additional paths added to sys.path

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36722 from tchaikov/wip-test-pybind

test/pybind: label test_rados tests with attributes

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph: create a log file before redirecting to it

as it is shell who interprets ">>" and redirect the stderr to given
file, but the shell process is launched by ubuntu:ububunt without using
sudo, so the command fails with "Permission denied" failure. to address
this issue, in this change, a file with proper priviledges is created
beforehand using `install`, so shell is able to write to it.

also, instead of creating this file in `maybe_redirect_stderr()`, it
returns the command to create the log file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36560 into master

* refs/pull/36560/head:
client: choose a random replica mds to send the request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36681 into master

* refs/pull/36681/head:
mds: don't track change of config 'mds_replay_unsafe_with_closed_session'
mds: fix 'forward loop' when forward_all_requests_to_auth is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36673 into master

* refs/pull/36673/head:
mds: don't recover files after normal session close

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36632 into master

* refs/pull/36632/head:
mds: standy-replay mds remained in the "resolve" state after restart the active mds

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36131 into master

* refs/pull/36131/head:
doc: document cephfs mirroring dev work
test: add tests for `ceph fs mirror` family of commands
mds: track filesystem mirror peers in fsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36459 into master

* refs/pull/36459/head:
mds: dump granular cap info in mds_sessions
common: fix validate type CephBool cause 'invalid command json'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36598 into master

* refs/pull/36598/head:
mds: drop decoding as MDSMap in FSMap::decode()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36687 into master

* refs/pull/36687/head:
client: use ccap_string to show caps in more user friendly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36697 into master

* refs/pull/36697/head:
messages,mds: Fix decoding of enum types on big-endian systems

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: update size of object after writing to object

in the writesame op implemented in
6f7d1a435c1e80ee7ad6a9fca898d686255cc206, we failed to update the OI of
object after appending to it, in this change `oi.size` is updated
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36472 into master

* refs/pull/36472/head:
qa/workunits/fs: add test for subvolume
mds: don't move inode with nlink > 1 to global snaprealm if it's in subvolume
mds: disallow hardlink across subvolume
mds: disallow across subvolume rename
mds: disallow creating snapshot on descendent directory of subvolume
mds: add vxattr that marks/clears subvolume flag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add method run ceph cluster command with better interface

This new method should allow better control on the process launched by
the passed command. This is achieved by allowing arguments provided by
teuthology.orchestra.run.run().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/cephadm: use raw string during `re.search`

cephadmservice.py:348: DeprecationWarning: invalid escape sequence \:
    port = re.search('\:\d+\/', end_point)

Signed-off-by: Michael Fritch <mfritch@suse.com>

librbd: flush requests could race past initiation of write ops

Now that IO is being processed by multiple threads, it's possible
that a write operation that was issued prior to a flush would not
have been started prior to the processing of the flush.

Fixes: https://tracker.ceph.com/issues/47050
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reset collection state after move operation

A technical requirement of move operations requires that the collection
be reset to a known state before being used again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35789 from Reperator/fix-ceph-objectstore-tool-docs

doc/man/8/ceph-objectstore-tool.rst: remove duplicate lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36742 from hulksmaaash/patch-2

doc/rados: remove statement that ISA plugin only works on Intel chips

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36496 from Daniel-Pivonka/cephadm-44926

mgr/cephadm: rgw update period after realm, zone creation

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36716 from jmolmo/dashboard_ports_2

cephadm: Allow users to use a custom dashboard ssl port

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36721 from ceph/47036-sass-colors-typescript

mgr/dashboard: use SASS colors in TypeScript

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>

Merge pull request #36735 from rhcs-dashboard/fix-this-week-endpoint

mgr/dashboard: remove "This week/month/year" and "Today" time stamps of Grafana in dashboard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

test/pybind: remove object for testing after done with it

* add vertical spacing for better readability
* remove object for testing after done with it

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: label test_rados tests with attributes

so we can pick the tests selectively. currently, some features are not
supported by crimson. but we will cover more and more of them. so
attaching different labels to tests cases helps us to run these tests
selectively.

Signed-off-by: Kefu Chai <kchai@redhat.com>