spec: include snap_schedule in ceph spec file

Signed-off-by: Venky Shankar <vshankar@redhat.com>

snap_schedule: add initial interface and basic CRUD ops

Storage of schedules is in an in-memory sqlite DB that gets serialized
to an rados object.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/mgr_util: CephFSClient - add helper methods

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/mgr_module: allow keyword arguments

If a mgr module encounters an argument values containing a '=' parsing
switches to kwargs style parsing independent of arg position. If a
non-kw argument is encountered after the first kw argument an EINVAL
error is returned.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36816 from sebastian-philipp/cephadm-pep8-osd

mgr/cephadm: PEP8tify services/osd.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36830 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: recovery related cleanup

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36841 from matthewoliver/cephadm_iscsi_dev_volumemount

cephadm: Give better access to the /dev in the iscsi container

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36811 from sebastian-philipp/mgr-always-mock-rados

pybind/mgr: pytest: Unconditionally mock the rados objects

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36844 from BryceCao/wip-correct-log-output

rgw: correct log output when decode bucket CORS attr

Merge pull request #36736 from ceph/47052-fix-base-uri-api-tests

mgr/dashboard: fix API tests: Invalid URL: None/api/auth

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36818 from sebastian-philipp/orch-pep8

mgr/orchestrator: PEP8tify

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36590 from votdev/issue_46903_prevent_select_events

mgr/dashboard: Datatable catches select events from other datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36599 from votdev/issue_46915_login_autocomplete

mgr/dashboard: Disable autocomplete on user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36799 from mgfritch/cephadm-iscsi-check-mon-command

mgr/cephadm: use `check_mon_command` during iscsi config

Merge pull request #36797 from mgfritch/cephadm-infer-image-no-tag

cephadm: infer latest image with a valid tag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36752 from mgfritch/cephadm-nfs-redux

mgr/cephadm: clean-up nfs service

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36330 from Devp00l/wip-46551

cephadm: Update error message on host failure

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: correct log output when decode bucket CORS attr

Signed-off-by: caolei <cao.leilc@inspur.com>

Merge pull request #36842 from cyx1231st/wip-fix-crimson-heartbeat

crimson/osd: fix unexpected connection markdown in heartbeat

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: cleanup, reset event is synchronously called

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #36807 from badone/wip-tracker-47002-legacy_vals-keyring

common/config: Make sure legacy_values are updated

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36535 from hualongfeng/fixed_unused_bluefs

os/bluestore: do not update used_blocks with bluefs_extents if bluefs is not available

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix unexpected connection markdown in heartbeat

Pass reference when log Heartbeat::Connection instance, or the
destructor will be called incorrectly, and the conn be marked down
unexpectedly.

Also, the replacing conn is actually connected during replacement-reset
event.

Fixes: https://tracker.ceph.com/issues/47124
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

common/config: Make sure legacy_values are updated

Must be updated after final change to 'values'

Fixes: https://tracker.ceph.com/issues/47002
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36827 into master

* refs/pull/36827/head:
qa/suites/smoke: remove packages on test completion

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: Give better access to the /dev in the iscsi container

In testing it seems the main iscsi container's /dev related volume mount to just
/dev/log is too narrow. And in certain circumstances it needs to access
to see /dev/rbd* devices. Like if using krdb.

This patch volume mounts /dev rather then /dev/log in the main
container, but since this aligns with what we need in the tcmu-runner
container it actaully ends up simplifying the code as well.

Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #36801 from yuriw/wip-yuriw-crontab-master

qa/tests: clients' upgrades for older releases must use "-t py2"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tests: clients' upgrades for older releases must use "-t py2"

Fixes: https://tracker.ceph.com/issues/47120
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #36773 into master

* refs/pull/36773/head:
mgr/volumes: Prevent subvolume recreate if trash is not-empty
mgr/volumes: Disallow subvolume group level snapshots
mgr/volumes: Add test case to ensure subvolume is marked
mgr/volumes: handle idempotent subvolume marks
mgr/volumes: Tests amended and added to ensure subvolume trash functionality
mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume
mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash
mgr/volumes: maintain per subvolume trash directory
mgr/volumes: make subvolume_v2::_is_retained() object property

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: Prevent subvolume recreate if trash is not-empty

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Disallow subvolume group level snapshots

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Add test case to ensure subvolume is marked

with the vxattr ceph.dir.subvolume set to true.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: handle idempotent subvolume marks

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: Tests amended and added to ensure subvolume trash functionality

Amended a few test cases to ensure created subvolumes and snaps
are removed, and trash stays empty at the end of the test.

Further added one test case for create errors in a retained
v2 subvolume, to ensure metadata is sane, and created incarnation
is not present.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume

This enables snapshot scaling of subvolumes, and hence prevents
renaming, hardlinking etc. outside the subvolume.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash

For v2 subvolumes without any snapshots or pending purges
move the subvolume to trash, for the purge operation.

When removing only incarnations, leverage symlink based purge
job creation, to avoid renaming the incarnation out of the
subvolume.

For create errors, where the subvolume was retained, just remove
the created incarnation directory.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: maintain per subvolume trash directory

PR https://github.com/ceph/ceph/pull/36472 introduces changes
that disallow nested nested snapshots in a subtree (subvolume)
and renames across subvolumes. This effect asynchronous purge
in mgr/volumes as subvolume are moved to a trash directory for
asynchronous deletion by purge threads.

To workaround this, start maintaining a subvolume specific
trash directory. Use the trash directory as an index to the
subvolume specific trash directory entry.

This changes subvolume deletion logic which currently relies
on `--retain-snapshots` flag to decide if the subvolume user
directory should get purged or the subvolume base directory
itself. Deleting a subvolume moves the user facing directory
to its specific trash directory. Purge threads take care of
deleting user facing directories (in trash) and the subvolume
base directory if required (when certain conditions are met).

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: make subvolume_v2::_is_retained() object property

... and drop the leading underscore.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/osd: use map::at() for better readability

avoid using operator[] for find-or-create, to ensure that we fail if the
indexed element is missing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages,osd,crimson: MOSDPGPull::set_pulls(vector<>&&)

this allows us to pass a lvalue reference to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract load_obc_for_recovery() out

extract ReplicatedRecoveryBackend::load_obc_for_recovery()
from ReplicatedRecoveryBackend::recover_object() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract maybe_pull_missing_obj() out

extract ReplicatedRecoveryBackend::maybe_pull_missing_obj() from
ReplicatedRecoveryBackend::recover_object() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/smoke: remove packages on test completion

A new teuthology PR [1] skips package cleanup by default. Use the
lightweight smoke suite to ensure that package cleanup continues to
work.

[1] https://github.com/ceph/teuthology/pull/1553

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36805 into master

* refs/pull/36805/head:
pybind/mgr/volumes: add global lock debug

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36774 from dillaman/wip-46669-2

librbd: drop deferred AsioEngine destruction

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36582 from rhcs-dashboard/fix-redirected-URL

mgr/dashboard: redirect to original URL after successful login

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

pybind/mgr/volumes: add global lock debug

Fixes: https://tracker.ceph.com/issues/47149
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36817 from sebastian-philipp/cephadm-pep8-tests

mgr/cephadm: PEP8tify some tests

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36315 from p-se/fix-monitoring-images-ceph-salt

cephadm: deploying of monitoring images partially broken

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #36571 from pcuzner/cephadm-tox-update

cephadm: remove py2 from tox tests

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

pybind/mgr: don't load cython modules

Most of the tests already require the cython modules to
be mocked. This is the final step in making pytest
independent of those modules.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: pytest: Unconditionally mock the rados objects

Always call `mock_ceph_modules()`. Otherwise we'd need to
split the tox run between modules that need to call
`mock_ceph_modules()` and those that don't

Fixes: https://tracker.ceph.com/issues/47141
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: PEP8tify

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify some tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify services/osd.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36794 from sebastian-philipp/cephadm-pep8-cephadmservice.py

mgr/cephadm: PEP8tify cephadmservice.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36761 from wjwithagen/wjw-fix-dashboard-package.json

dashboard: Fix building build shell script

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36527 from neha-ojha/wip-whitelist-blacklist

Replace 'whitelist' and 'blacklist' language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36509 from pcuzner/cephadm-add-host-facts

cephadm: add gather_facts command

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36793 from sebastian-philipp/cephadm-pep8-schedule.py

mgr/cephadm: PEP8tify schedule.py

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: use `check_mon_command` during iscsi config

ensure the mon commands actually succeed

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use `check_mon_command` util

rather then checking the retval explicitly

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: merge NFSGanesha into NFSService class

these two classes really preform the same function

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36757 from mgfritch/cephadm-raw-str-warn

mgr/cephadm: use raw string during `re.search`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: infer latest image with a valid tag

Fixes: https://tracker.ceph.com/issues/47134
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36787 from tchaikov/wip-47031

crimson/osd: drop misdirected ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: PEP8tify cephadmservice.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify schedule.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36738 from ShraddhaAg/cephadm-archive-logs

qa/tasks/cephadm.py: add ceph logs directory in job's info.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: redirect to original URL after successful login

Fixes: https://tracker.ceph.com/issues/45067
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit b5359a155c6ff7e92ab6c6ed16f7ff9fc76c76ea)

cephadm: Update error message on host failure

Now the error message explains how to add the ssh key to the not
reachable host

Fixes: https://tracker.ceph.com/issues/46551
Signed-off-by: Stephan Müller <smueller@suse.com>

crimson/osd: drop misdirected ops

see also `PrimaryLogPG::do_op()`, we should ignore the ops hitting us if
we are not supposed to serve them. this happens when the client is using
a stale osdmap.

Fixes: https://tracker.ceph.com/issues/47031
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36778 from tchaikov/wip-cmake-fio

cmake: silence "detached HEAD" warning

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

cmake: silence "detached HEAD" warning

git complains when checking out a tag in "detached HEAD", like:

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them
...

but this does not help, as, in general, we don't hack fio in Ceph,
so disable this warning. and also clone the repo in shallow mode
for the same reason -- we don't care about the whole history of
fio repo. we just use it for testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36336 from pcuzner/ceph-volume-lsm

ceph-volume: add libstoragemgmt support

cephadm: remove py2 from test_adoption

Since cephadm is py3 based, and py2 is EOL this patch
removes the py2 test iteration from test_adoption.sh

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #36684 into master

* refs/pull/36684/head:
qa/tasks/nfs: Test mounting of export created with nfs command
qa/tasks/nfs: Add helper method to check nfs cluster status
qa/tasks/nfs: Cleanup created filesystem
qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: support reverting to legacy blacklist op via neorados API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove racially insensitive terms

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

librbd: RESTORE_SOURCE_WHITELIST -> ALLOWED_RESTORE_SOURCES

Signed-off-by: Neha Ojha <nojha@redhat.com>

journal, librbd: whitelisted_laggy_clients -> ignored_laggy_clients

Signed-off-by: Neha Ojha <nojha@redhat.com>

*: convert more blacklist to blocklist

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: convert WHITELIST to ALLOWLIST

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

*: convert EBLACKLISTED to EBLOCKLISTED

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/librados_test_stub/TestMemIoCtxImpl.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/krbd/thrash: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/multimds: log-whitelist -> log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librbd/managed_lock/test_mock_BreakRequest.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

test/librados_test_stub/TestRadosClient.cc: blacklist -> blocklist

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon/ConfigMonitor: handle renamed config options

Signed-off-by: Sage Weil <sage@newdream.net>

librados: fall back to legacy blacklist command if blocklist gets EINVAL

I would have expected ENOSYS or EOPNOTSUPP or similar, but the mon returns
EINVAL on an unrecognized command

Signed-off-by: Sage Weil <sage@newdream.net>

misc language changes: whitelist -> ignore etc

Signed-off-by: Sage Weil <sage@newdream.net>

librbd: whitelisted_laggy_clients -> allowlisted_laggy_clients

Signed-off-by: Sage Weil <sage@newdream.net>

osd/OSDCap: whitelisted -> allowed

Signed-off-by: Sage Weil <sage@newdream.net>

qa: log-whitelist -> log-ignorelist

Signed-off-by: Sage Weil <sage@newdream.net>

blacklist -> blocklist

Signed-off-by: Sage Weil <sage@newdream.net>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #36394 into master

* refs/pull/36394/head:
qa: add method run ceph cluster command with better interface
cephfs: allow only "good" characters

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36772 from neha-ojha/wip-47119

qa/suites/perf-basic/objectstore: remove filestore

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

librbd: drop deferred AsioEngine destruction

Commit 90bd1d7a857c0f3c57bda60975f58f9859940185 removed the need to
defer deletion of the AsioEngine but missed removing the original
code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>