doc: use union merge strategy for resolving conflicts in PendingReleaseNotes

Generally, the conflict resolution performed manually is always to do a
union: automate this. This avoids pointless rebases of PRs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36701 into master

* refs/pull/36701/head:
client: make Client::open() pass proper cap mask to path_walk

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36641 into master

* refs/pull/36641/head:
mds : skip show_cache if level of log is lower than 7

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36882 into master

* refs/pull/36882/head:
qa/tests: Added "--force-priority"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36729 from changchengx/zero_malloc

crush: do not malloc for empty bucket and avoid member variables after memset() with 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36820 from tchaikov/wip-test-multi_stress_watch

test/multi_stress_watch: use condition_variable for level triggering

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #36832 from neha-ojha/wip-clean-temps

tools/osdmaptool.cc: add ability to clean_temps

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

qa/tests: Added "--force-priority"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #36854 from idryomov/wip-krbd-default-pool-fix

rbd: fix default pool handling for krbd map/unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36866 from yison/fix-core-dump-issue-of-sync-read

blk/spdk/NVMEDevice.cc: fix a core dump issue of sync read

Reviewed-by: Kefu Chai <kchai@redhat.com>

blk/spdk/NVMEDevice.cc: fix a core dump issue of sync read

Signed-off-by: Yuan Lu <yuan.y.lu@intel.com>

Merge pull request #36802 from dillaman/wip-46916

librbd: expose internal plugin API to plugin libraries

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36792 from sebastian-philipp/cephadm-pep8-inventory

mgr/cephadm: PEP8tify inventory.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36848 from sebastian-philipp/cephadm-no-pep8-test_scheduling

mgr/cephadm: disable autopep8 for test_{spec,scheduling}.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #29489 from jan--f/cephfs-snap-schedule

pybind/snap_schedule: add initial interface

Merge pull request #34566 from X0Ken/master

os/bluestore: Support flock retry

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36507 from matthewoliver/msgr_intercepter_step_enum

msgr: interceptor test step enum to ease readability

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36762 from qwren/qat-doc-update

doc: the update for QAT Acceleration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35941 into master

* refs/pull/35941/head:
doc/mgr: Supplement missing keywords 'crash'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/cli/osdmaptool/help.t: add clean-temps

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: add clean-temps to osdmaptool.rst

Signed-off-by: Neha Ojha <nojha@redhat.com>

tools/osdmaptool.cc: add ability to clean_temps

This is particularly useful for debugging purposes when clean_temps()
takes abnormally high amount of time due to flaws in crush rules etc.

Fixes: https://tracker.ceph.com/issues/47159
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36572 from dmick/wip-cephadm-doc

doc/mgr: Clean up orchestrator.rst (spell/punct, awkward wording)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36750 from ifed01/wip-ifed-enable-flexible-bluefs

os/bluestore: enable more flexible bluefs space management by default.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36718 from ShraddhaAg/add-dir-to-archive-in-logs

qa/tasks/ceph.py: add ceph logs directory in job's info.yaml

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

librbd: add internal API for plugin libraries

To avoid the need to expose internal librbd symbols as public, expose
a select set of symbols via an internal plugin API for use by the
plugins.

Fixes: https://tracker.ceph.com/issues/46916
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/snap_schedule: fix imports for snap_schedule vs mgr wide tox runs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: normalize dates to utc and ignore microseconds

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: don't use as keyword in update statement

While UPDATE <table_name> AS <alias> ... should be valid SQL syntax. It seems
like only some sqlite version support it however.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: fix import locations for unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: datetime.fromisoformat fallbacks

fromisoformat was introduced in pytho3.7. This can be worked around by
either installing
https://github.com/movermeyer/backports.datetime_fromisoformat. If a
pre3.7 python is found and the backport module is not installed,
snap_schedule falls back to parsing a fully specified timestamp
format. This commit adds the necessary code.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: guard minute repeat interval

To enable the use of a minutely repeat interval in the snap schedules,
this adds a guard that checks for the SNAP_SCHED_TEST env variable.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: add type hint for mypy, fix import

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: integrate with mgr wide tox testing

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: add separate retention interface

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: add more unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: fix meta uniqueness contraint

This adds a simple test for the Schedule.store_schedule. This test
exposed a bug (couldn't add schedule on different paths with the same
repeat and start time). Bug is also fixed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: add unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_sched: improve SQL statements, logging, metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/snap_schedule: return negative error codes

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mgr/snap_schedule: refactor Schedule ctor, make more args optional

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: Add active flag to status report

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: refactor schedule into two modules

This separates the Schedule class from the SnapSchedClient class.
Motivation is cleaner code layout and simpler testing.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: add interface to de-/activate schedules

This allows to 1) activate schedules that have previously been
deactivated due to the path no existing and 2) allowing ussers to
explicitly deactivate a schedule for debugging or maintenance purposes.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: list - print ws separated results

This should make it easy to consume for cli tools.
Also simplifies some internals.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

snap_schedule: add metadata fields, refactor Schedule, multiple Schedules per path

This commit adds a bunch of interesting metadata fields to the DB and
refactors the Schedule class so that it concentrates all the DB queries
in on place. Also enables to set multiple schedules per path as long as
the (repeat interval, start time) pairs are unique.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

snap-schedule: Implement pruning according to retention specfication

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: take snapshots according to schedule.

Timers are scheduled for the next snapshot to be taken. The next Timer
is canceled in case the snapshot DB is changed. When a Timer triggers it
also schedules the next Timer.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

spec: include snap_schedule in ceph spec file

Signed-off-by: Venky Shankar <vshankar@redhat.com>

snap_schedule: add initial interface and basic CRUD ops

Storage of schedules is in an in-memory sqlite DB that gets serialized
to an rados object.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/mgr_util: CephFSClient - add helper methods

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/mgr_module: allow keyword arguments

If a mgr module encounters an argument values containing a '=' parsing
switches to kwargs style parsing independent of arg position. If a
non-kw argument is encountered after the first kw argument an EINVAL
error is returned.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

rbd: rename some MapOptions instances to unmap_options

It is no longer a global variable, so rename the ones that have to do
with options for unmapping (currently just "force").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: fix default pool handling for krbd map/unmap

The default pool name does not get passed to the kernel since commit
96f05a7956b3 ("rbd: delay determination of default pool name").  The
kernel ends up interpreting the image name as the pool name (and the
snapshot name as the image name).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36816 from sebastian-philipp/cephadm-pep8-osd

mgr/cephadm: PEP8tify services/osd.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36830 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: recovery related cleanup

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36841 from matthewoliver/cephadm_iscsi_dev_volumemount

cephadm: Give better access to the /dev in the iscsi container

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36811 from sebastian-philipp/mgr-always-mock-rados

pybind/mgr: pytest: Unconditionally mock the rados objects

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36844 from BryceCao/wip-correct-log-output

rgw: correct log output when decode bucket CORS attr

Merge pull request #36736 from ceph/47052-fix-base-uri-api-tests

mgr/dashboard: fix API tests: Invalid URL: None/api/auth

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36818 from sebastian-philipp/orch-pep8

mgr/orchestrator: PEP8tify

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36590 from votdev/issue_46903_prevent_select_events

mgr/dashboard: Datatable catches select events from other datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36599 from votdev/issue_46915_login_autocomplete

mgr/dashboard: Disable autocomplete on user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36799 from mgfritch/cephadm-iscsi-check-mon-command

mgr/cephadm: use `check_mon_command` during iscsi config

Merge pull request #36797 from mgfritch/cephadm-infer-image-no-tag

cephadm: infer latest image with a valid tag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36752 from mgfritch/cephadm-nfs-redux

mgr/cephadm: clean-up nfs service

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36330 from Devp00l/wip-46551

cephadm: Update error message on host failure

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: disable autopep8 for test_{spec,scheduling}.py

The signal to noise ratio is just too bad.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: correct log output when decode bucket CORS attr

Signed-off-by: caolei <cao.leilc@inspur.com>

Merge pull request #36842 from cyx1231st/wip-fix-crimson-heartbeat

crimson/osd: fix unexpected connection markdown in heartbeat

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: cleanup, reset event is synchronously called

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #36807 from badone/wip-tracker-47002-legacy_vals-keyring

common/config: Make sure legacy_values are updated

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36535 from hualongfeng/fixed_unused_bluefs

os/bluestore: do not update used_blocks with bluefs_extents if bluefs is not available

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix unexpected connection markdown in heartbeat

Pass reference when log Heartbeat::Connection instance, or the
destructor will be called incorrectly, and the conn be marked down
unexpectedly.

Also, the replacing conn is actually connected during replacement-reset
event.

Fixes: https://tracker.ceph.com/issues/47124
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

common/config: Make sure legacy_values are updated

Must be updated after final change to 'values'

Fixes: https://tracker.ceph.com/issues/47002
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36827 into master

* refs/pull/36827/head:
qa/suites/smoke: remove packages on test completion

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: Give better access to the /dev in the iscsi container

In testing it seems the main iscsi container's /dev related volume mount to just
/dev/log is too narrow. And in certain circumstances it needs to access
to see /dev/rbd* devices. Like if using krdb.

This patch volume mounts /dev rather then /dev/log in the main
container, but since this aligns with what we need in the tcmu-runner
container it actaully ends up simplifying the code as well.

Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #36801 from yuriw/wip-yuriw-crontab-master

qa/tests: clients' upgrades for older releases must use "-t py2"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tests: clients' upgrades for older releases must use "-t py2"

Fixes: https://tracker.ceph.com/issues/47120
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/mgr: Clean up orchestrator.rst (spell/punct, awkward wording)

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge PR #36773 into master

* refs/pull/36773/head:
mgr/volumes: Prevent subvolume recreate if trash is not-empty
mgr/volumes: Disallow subvolume group level snapshots
mgr/volumes: Add test case to ensure subvolume is marked
mgr/volumes: handle idempotent subvolume marks
mgr/volumes: Tests amended and added to ensure subvolume trash functionality
mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume
mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash
mgr/volumes: maintain per subvolume trash directory
mgr/volumes: make subvolume_v2::_is_retained() object property

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: Prevent subvolume recreate if trash is not-empty

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Disallow subvolume group level snapshots

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Add test case to ensure subvolume is marked

with the vxattr ceph.dir.subvolume set to true.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: handle idempotent subvolume marks

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: Tests amended and added to ensure subvolume trash functionality

Amended a few test cases to ensure created subvolumes and snaps
are removed, and trash stays empty at the end of the test.

Further added one test case for create errors in a retained
v2 subvolume, to ensure metadata is sane, and created incarnation
is not present.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume

This enables snapshot scaling of subvolumes, and hence prevents
renaming, hardlinking etc. outside the subvolume.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash

For v2 subvolumes without any snapshots or pending purges
move the subvolume to trash, for the purge operation.

When removing only incarnations, leverage symlink based purge
job creation, to avoid renaming the incarnation out of the
subvolume.

For create errors, where the subvolume was retained, just remove
the created incarnation directory.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: maintain per subvolume trash directory

PR https://github.com/ceph/ceph/pull/36472 introduces changes
that disallow nested nested snapshots in a subtree (subvolume)
and renames across subvolumes. This effect asynchronous purge
in mgr/volumes as subvolume are moved to a trash directory for
asynchronous deletion by purge threads.

To workaround this, start maintaining a subvolume specific
trash directory. Use the trash directory as an index to the
subvolume specific trash directory entry.

This changes subvolume deletion logic which currently relies
on `--retain-snapshots` flag to decide if the subvolume user
directory should get purged or the subvolume base directory
itself. Deleting a subvolume moves the user facing directory
to its specific trash directory. Purge threads take care of
deleting user facing directories (in trash) and the subvolume
base directory if required (when certain conditions are met).

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: make subvolume_v2::_is_retained() object property

... and drop the leading underscore.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/osd: use map::at() for better readability

avoid using operator[] for find-or-create, to ensure that we fail if the
indexed element is missing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages,osd,crimson: MOSDPGPull::set_pulls(vector<>&&)

this allows us to pass a lvalue reference to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract load_obc_for_recovery() out

extract ReplicatedRecoveryBackend::load_obc_for_recovery()
from ReplicatedRecoveryBackend::recover_object() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract maybe_pull_missing_obj() out

extract ReplicatedRecoveryBackend::maybe_pull_missing_obj() from
ReplicatedRecoveryBackend::recover_object() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/smoke: remove packages on test completion

A new teuthology PR [1] skips package cleanup by default. Use the
lightweight smoke suite to ensure that package cleanup continues to
work.

[1] https://github.com/ceph/teuthology/pull/1553

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36805 into master

* refs/pull/36805/head:
pybind/mgr/volumes: add global lock debug

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36774 from dillaman/wip-46669-2

librbd: drop deferred AsioEngine destruction

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36582 from rhcs-dashboard/fix-redirected-URL

mgr/dashboard: redirect to original URL after successful login

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

pybind/mgr/volumes: add global lock debug

Fixes: https://tracker.ceph.com/issues/47149
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36817 from sebastian-philipp/cephadm-pep8-tests

mgr/cephadm: PEP8tify some tests

Reviewed-by: Adam King <adking@redhat.com>