client: make the osdmap wait under the mount_state reader check

It will make no sense to check this if the unmouting is already
going on.

https://tracker.ceph.com/issues/47293
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #36767 from tchaikov/wip-47061

test/osd/RadosModel: check retval of read op

Reviewed-by: Samuel Just <sjust@redhat.com>

test/osd/RadosModel: assure that tid seq is contiguous

before this change:

4:  finishing write tid 1 to gen81094472-1
4:  finishing write tid 2 to gen81094472-1
4:  finishing write tid 3 to gen81094472-1
4:  finishing write tid 5 to gen81094472-1
4:  finishing write tid 6 to gen81094472-1

after this change:

1:  finishing write tid 1 to gen81117124-1
1:  finishing write tid 2 to gen81117124-1
1:  finishing write tid 3 to gen81117124-1
1:  finishing write tid 4 to gen81117124-1
1:  finishing write tid 5 to gen81117124-1

more pleasant when reading the logging messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: check retval of read op

* check racing read's return value
* check if racing read completes

Fixes: https://tracker.ceph.com/issues/47061
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: do not set done

we don't reuse TestOp, and `done` should have been reset in the
constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: use range-based for loop

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: use in-class initiailization

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd: use stack for ObjectDesc::iterator::stack

better readability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36815 from sebastian-philipp/cephadm-pep8-module

mgr/cephadm: PEP8tify module.py

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: PEP8tify module.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36858 from adk3798/cephadm-47171

mgr/cephadm: Verify non-empty list in get_active_daemon functions

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36849 from sebastian-philipp/cephadm-kick-when-monmap

mgr/cephadm: kick serve loop to trigger ceph.conf deployment

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36753 from sebastian-philipp/mgr-cephadm-schedule-redeploy

mgr/cephadm: Make daemon actions asynchronous

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35891 from changchengx/multi_stress_watch

test/multi_stress_watch: use internel API to connect cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36868 from zhoufeng2021/removeunsed

msg/async: cleanup unused function local_deliver()

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: use insert_or_assign()

for better performance, and less repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: add more comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd/RadosModel: rename local variable

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36888 from batrick/release-notes-union

doc: use union merge strategy for resolving conflicts in PendingRelea…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36886 from batrick/librados-statfs-header

librados: remove unused header

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: remove unused headers

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: use union merge strategy for resolving conflicts in PendingReleaseNotes

Generally, the conflict resolution performed manually is always to do a
union: automate this. This avoids pointless rebases of PRs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36701 into master

* refs/pull/36701/head:
client: make Client::open() pass proper cap mask to path_walk

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36641 into master

* refs/pull/36641/head:
mds : skip show_cache if level of log is lower than 7

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36882 into master

* refs/pull/36882/head:
qa/tests: Added "--force-priority"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36729 from changchengx/zero_malloc

crush: do not malloc for empty bucket and avoid member variables after memset() with 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36820 from tchaikov/wip-test-multi_stress_watch

test/multi_stress_watch: use condition_variable for level triggering

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #36832 from neha-ojha/wip-clean-temps

tools/osdmaptool.cc: add ability to clean_temps

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

qa/tests: Added "--force-priority"

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #36854 from idryomov/wip-krbd-default-pool-fix

rbd: fix default pool handling for krbd map/unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36866 from yison/fix-core-dump-issue-of-sync-read

blk/spdk/NVMEDevice.cc: fix a core dump issue of sync read

Reviewed-by: Kefu Chai <kchai@redhat.com>

blk/spdk/NVMEDevice.cc: fix a core dump issue of sync read

Signed-off-by: Yuan Lu <yuan.y.lu@intel.com>

msg/async: cleanup unused function local_deliver()

Signed-off-by: zhoufeng <zhoufeng2021@outlook.com>

Merge pull request #36802 from dillaman/wip-46916

librbd: expose internal plugin API to plugin libraries

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/cephadm: add test_daemon_check

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Make daemon actions asynchronous

Fixes broken `ceph orch redeploy <current active mgr>`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: add scheduled_daemon_actions to HostCache

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move fail_over from upgrade.py to MgrService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: _wait_for_ok_to_stop: add missing f-prefix

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: _create_daemon: remove usused parameter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: daemon redeploy: Fix missing default value

Fixes: https://tracker.ceph.com/issues/47035
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36792 from sebastian-philipp/cephadm-pep8-inventory

mgr/cephadm: PEP8tify inventory.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36848 from sebastian-philipp/cephadm-no-pep8-test_scheduling

mgr/cephadm: disable autopep8 for test_{spec,scheduling}.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #29489 from jan--f/cephfs-snap-schedule

pybind/snap_schedule: add initial interface

Merge pull request #34566 from X0Ken/master

os/bluestore: Support flock retry

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36507 from matthewoliver/msgr_intercepter_step_enum

msgr: interceptor test step enum to ease readability

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36762 from qwren/qat-doc-update

doc: the update for QAT Acceleration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35941 into master

* refs/pull/35941/head:
doc/mgr: Supplement missing keywords 'crash'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Verify non-empty list in get_active_daemon functions

The get_active_daemon functions for monitoring stack daemons
were just returning the first or last daemon in the given list
without checking the list actually contained any daemons

Fixes: https://tracker.ceph.com/issues/47171
Signed-off-by: Adam King <adking@redhat.com>

test/cli/osdmaptool/help.t: add clean-temps

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: add clean-temps to osdmaptool.rst

Signed-off-by: Neha Ojha <nojha@redhat.com>

tools/osdmaptool.cc: add ability to clean_temps

This is particularly useful for debugging purposes when clean_temps()
takes abnormally high amount of time due to flaws in crush rules etc.

Fixes: https://tracker.ceph.com/issues/47159
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36572 from dmick/wip-cephadm-doc

doc/mgr: Clean up orchestrator.rst (spell/punct, awkward wording)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36750 from ifed01/wip-ifed-enable-flexible-bluefs

os/bluestore: enable more flexible bluefs space management by default.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36718 from ShraddhaAg/add-dir-to-archive-in-logs

qa/tasks/ceph.py: add ceph logs directory in job's info.yaml

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

librbd: add internal API for plugin libraries

To avoid the need to expose internal librbd symbols as public, expose
a select set of symbols via an internal plugin API for use by the
plugins.

Fixes: https://tracker.ceph.com/issues/46916
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/snap_schedule: fix imports for snap_schedule vs mgr wide tox runs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: normalize dates to utc and ignore microseconds

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: don't use as keyword in update statement

While UPDATE <table_name> AS <alias> ... should be valid SQL syntax. It seems
like only some sqlite version support it however.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: fix import locations for unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: datetime.fromisoformat fallbacks

fromisoformat was introduced in pytho3.7. This can be worked around by
either installing
https://github.com/movermeyer/backports.datetime_fromisoformat. If a
pre3.7 python is found and the backport module is not installed,
snap_schedule falls back to parsing a fully specified timestamp
format. This commit adds the necessary code.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: guard minute repeat interval

To enable the use of a minutely repeat interval in the snap schedules,
this adds a guard that checks for the SNAP_SCHED_TEST env variable.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: add type hint for mypy, fix import

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: integrate with mgr wide tox testing

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: add separate retention interface

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: add more unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: fix meta uniqueness contraint

This adds a simple test for the Schedule.store_schedule. This test
exposed a bug (couldn't add schedule on different paths with the same
repeat and start time). Bug is also fixed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: add unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_sched: improve SQL statements, logging, metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/snap_schedule: return negative error codes

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mgr/snap_schedule: refactor Schedule ctor, make more args optional

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: Add active flag to status report

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: refactor schedule into two modules

This separates the Schedule class from the SnapSchedClient class.
Motivation is cleaner code layout and simpler testing.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: add interface to de-/activate schedules

This allows to 1) activate schedules that have previously been
deactivated due to the path no existing and 2) allowing ussers to
explicitly deactivate a schedule for debugging or maintenance purposes.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: list - print ws separated results

This should make it easy to consume for cli tools.
Also simplifies some internals.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

snap_schedule: add metadata fields, refactor Schedule, multiple Schedules per path

This commit adds a bunch of interesting metadata fields to the DB and
refactors the Schedule class so that it concentrates all the DB queries
in on place. Also enables to set multiple schedules per path as long as
the (repeat interval, start time) pairs are unique.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

snap-schedule: Implement pruning according to retention specfication

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: take snapshots according to schedule.

Timers are scheduled for the next snapshot to be taken. The next Timer
is canceled in case the snapshot DB is changed. When a Timer triggers it
also schedules the next Timer.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

spec: include snap_schedule in ceph spec file

Signed-off-by: Venky Shankar <vshankar@redhat.com>

snap_schedule: add initial interface and basic CRUD ops

Storage of schedules is in an in-memory sqlite DB that gets serialized
to an rados object.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/mgr_util: CephFSClient - add helper methods

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/mgr_module: allow keyword arguments

If a mgr module encounters an argument values containing a '=' parsing
switches to kwargs style parsing independent of arg position. If a
non-kw argument is encountered after the first kw argument an EINVAL
error is returned.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

rbd: rename some MapOptions instances to unmap_options

It is no longer a global variable, so rename the ones that have to do
with options for unmapping (currently just "force").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: fix default pool handling for krbd map/unmap

The default pool name does not get passed to the kernel since commit
96f05a7956b3 ("rbd: delay determination of default pool name").  The
kernel ends up interpreting the image name as the pool name (and the
snapshot name as the image name).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36816 from sebastian-philipp/cephadm-pep8-osd

mgr/cephadm: PEP8tify services/osd.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36830 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: recovery related cleanup

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36841 from matthewoliver/cephadm_iscsi_dev_volumemount

cephadm: Give better access to the /dev in the iscsi container

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36811 from sebastian-philipp/mgr-always-mock-rados

pybind/mgr: pytest: Unconditionally mock the rados objects

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36844 from BryceCao/wip-correct-log-output

rgw: correct log output when decode bucket CORS attr

Merge pull request #36736 from ceph/47052-fix-base-uri-api-tests

mgr/dashboard: fix API tests: Invalid URL: None/api/auth

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36818 from sebastian-philipp/orch-pep8

mgr/orchestrator: PEP8tify

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36590 from votdev/issue_46903_prevent_select_events

mgr/dashboard: Datatable catches select events from other datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36599 from votdev/issue_46915_login_autocomplete

mgr/dashboard: Disable autocomplete on user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36799 from mgfritch/cephadm-iscsi-check-mon-command

mgr/cephadm: use `check_mon_command` during iscsi config

Merge pull request #36797 from mgfritch/cephadm-infer-image-no-tag

cephadm: infer latest image with a valid tag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36752 from mgfritch/cephadm-nfs-redux

mgr/cephadm: clean-up nfs service

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36330 from Devp00l/wip-46551

cephadm: Update error message on host failure

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: kick serve loop to trigger ceph.conf deployment

Otherwise we'll wait 600s.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: disable autopep8 for test_{spec,scheduling}.py

The signal to noise ratio is just too bad.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: correct log output when decode bucket CORS attr

Signed-off-by: caolei <cao.leilc@inspur.com>