mgr/dashboard: fix style being applied globally

fixes: https://tracker.ceph.com/issues/45018

Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

Merge pull request #29489 from jan--f/cephfs-snap-schedule

pybind/snap_schedule: add initial interface

Merge pull request #34566 from X0Ken/master

os/bluestore: Support flock retry

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #36507 from matthewoliver/msgr_intercepter_step_enum

msgr: interceptor test step enum to ease readability

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36762 from qwren/qat-doc-update

doc: the update for QAT Acceleration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35941 into master

* refs/pull/35941/head:
doc/mgr: Supplement missing keywords 'crash'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36572 from dmick/wip-cephadm-doc

doc/mgr: Clean up orchestrator.rst (spell/punct, awkward wording)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36750 from ifed01/wip-ifed-enable-flexible-bluefs

os/bluestore: enable more flexible bluefs space management by default.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36718 from ShraddhaAg/add-dir-to-archive-in-logs

qa/tasks/ceph.py: add ceph logs directory in job's info.yaml

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/snap_schedule: fix imports for snap_schedule vs mgr wide tox runs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: normalize dates to utc and ignore microseconds

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: don't use as keyword in update statement

While UPDATE <table_name> AS <alias> ... should be valid SQL syntax. It seems
like only some sqlite version support it however.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: fix import locations for unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: datetime.fromisoformat fallbacks

fromisoformat was introduced in pytho3.7. This can be worked around by
either installing
https://github.com/movermeyer/backports.datetime_fromisoformat. If a
pre3.7 python is found and the backport module is not installed,
snap_schedule falls back to parsing a fully specified timestamp
format. This commit adds the necessary code.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: guard minute repeat interval

To enable the use of a minutely repeat interval in the snap schedules,
this adds a guard that checks for the SNAP_SCHED_TEST env variable.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: add type hint for mypy, fix import

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: integrate with mgr wide tox testing

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: add separate retention interface

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: add more unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: fix meta uniqueness contraint

This adds a simple test for the Schedule.store_schedule. This test
exposed a bug (couldn't add schedule on different paths with the same
repeat and start time). Bug is also fixed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: add unit tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_sched: improve SQL statements, logging, metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/snap_schedule: return negative error codes

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mgr/snap_schedule: refactor Schedule ctor, make more args optional

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/snap_schedule: Add active flag to status report

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: refactor schedule into two modules

This separates the Schedule class from the SnapSchedClient class.
Motivation is cleaner code layout and simpler testing.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: add interface to de-/activate schedules

This allows to 1) activate schedules that have previously been
deactivated due to the path no existing and 2) allowing ussers to
explicitly deactivate a schedule for debugging or maintenance purposes.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap-schedule: list - print ws separated results

This should make it easy to consume for cli tools.
Also simplifies some internals.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

snap_schedule: add metadata fields, refactor Schedule, multiple Schedules per path

This commit adds a bunch of interesting metadata fields to the DB and
refactors the Schedule class so that it concentrates all the DB queries
in on place. Also enables to set multiple schedules per path as long as
the (repeat interval, start time) pairs are unique.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

snap-schedule: Implement pruning according to retention specfication

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/snap_schedule: take snapshots according to schedule.

Timers are scheduled for the next snapshot to be taken. The next Timer
is canceled in case the snapshot DB is changed. When a Timer triggers it
also schedules the next Timer.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

spec: include snap_schedule in ceph spec file

Signed-off-by: Venky Shankar <vshankar@redhat.com>

snap_schedule: add initial interface and basic CRUD ops

Storage of schedules is in an in-memory sqlite DB that gets serialized
to an rados object.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/mgr_util: CephFSClient - add helper methods

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/mgr_module: allow keyword arguments

If a mgr module encounters an argument values containing a '=' parsing
switches to kwargs style parsing independent of arg position. If a
non-kw argument is encountered after the first kw argument an EINVAL
error is returned.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36816 from sebastian-philipp/cephadm-pep8-osd

mgr/cephadm: PEP8tify services/osd.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36830 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: recovery related cleanup

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36841 from matthewoliver/cephadm_iscsi_dev_volumemount

cephadm: Give better access to the /dev in the iscsi container

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36811 from sebastian-philipp/mgr-always-mock-rados

pybind/mgr: pytest: Unconditionally mock the rados objects

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36844 from BryceCao/wip-correct-log-output

rgw: correct log output when decode bucket CORS attr

Merge pull request #36736 from ceph/47052-fix-base-uri-api-tests

mgr/dashboard: fix API tests: Invalid URL: None/api/auth

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36818 from sebastian-philipp/orch-pep8

mgr/orchestrator: PEP8tify

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36590 from votdev/issue_46903_prevent_select_events

mgr/dashboard: Datatable catches select events from other datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36599 from votdev/issue_46915_login_autocomplete

mgr/dashboard: Disable autocomplete on user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36799 from mgfritch/cephadm-iscsi-check-mon-command

mgr/cephadm: use `check_mon_command` during iscsi config

Merge pull request #36797 from mgfritch/cephadm-infer-image-no-tag

cephadm: infer latest image with a valid tag

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36752 from mgfritch/cephadm-nfs-redux

mgr/cephadm: clean-up nfs service

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #36330 from Devp00l/wip-46551

cephadm: Update error message on host failure

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: correct log output when decode bucket CORS attr

Signed-off-by: caolei <cao.leilc@inspur.com>

Merge pull request #36842 from cyx1231st/wip-fix-crimson-heartbeat

crimson/osd: fix unexpected connection markdown in heartbeat

Reviewed-by: Xuehan Xu <xxhdx1985126@163.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: cleanup, reset event is synchronously called

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #36807 from badone/wip-tracker-47002-legacy_vals-keyring

common/config: Make sure legacy_values are updated

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36535 from hualongfeng/fixed_unused_bluefs

os/bluestore: do not update used_blocks with bluefs_extents if bluefs is not available

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix unexpected connection markdown in heartbeat

Pass reference when log Heartbeat::Connection instance, or the
destructor will be called incorrectly, and the conn be marked down
unexpectedly.

Also, the replacing conn is actually connected during replacement-reset
event.

Fixes: https://tracker.ceph.com/issues/47124
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

common/config: Make sure legacy_values are updated

Must be updated after final change to 'values'

Fixes: https://tracker.ceph.com/issues/47002
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36827 into master

* refs/pull/36827/head:
qa/suites/smoke: remove packages on test completion

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: Give better access to the /dev in the iscsi container

In testing it seems the main iscsi container's /dev related volume mount to just
/dev/log is too narrow. And in certain circumstances it needs to access
to see /dev/rbd* devices. Like if using krdb.

This patch volume mounts /dev rather then /dev/log in the main
container, but since this aligns with what we need in the tcmu-runner
container it actaully ends up simplifying the code as well.

Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #36801 from yuriw/wip-yuriw-crontab-master

qa/tests: clients' upgrades for older releases must use "-t py2"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tests: clients' upgrades for older releases must use "-t py2"

Fixes: https://tracker.ceph.com/issues/47120
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/mgr: Clean up orchestrator.rst (spell/punct, awkward wording)

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge PR #36773 into master

* refs/pull/36773/head:
mgr/volumes: Prevent subvolume recreate if trash is not-empty
mgr/volumes: Disallow subvolume group level snapshots
mgr/volumes: Add test case to ensure subvolume is marked
mgr/volumes: handle idempotent subvolume marks
mgr/volumes: Tests amended and added to ensure subvolume trash functionality
mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume
mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash
mgr/volumes: maintain per subvolume trash directory
mgr/volumes: make subvolume_v2::_is_retained() object property

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: Prevent subvolume recreate if trash is not-empty

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Disallow subvolume group level snapshots

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Add test case to ensure subvolume is marked

with the vxattr ceph.dir.subvolume set to true.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: handle idempotent subvolume marks

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: Tests amended and added to ensure subvolume trash functionality

Amended a few test cases to ensure created subvolumes and snaps
are removed, and trash stays empty at the end of the test.

Further added one test case for create errors in a retained
v2 subvolume, to ensure metadata is sane, and created incarnation
is not present.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume

This enables snapshot scaling of subvolumes, and hence prevents
renaming, hardlinking etc. outside the subvolume.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash

For v2 subvolumes without any snapshots or pending purges
move the subvolume to trash, for the purge operation.

When removing only incarnations, leverage symlink based purge
job creation, to avoid renaming the incarnation out of the
subvolume.

For create errors, where the subvolume was retained, just remove
the created incarnation directory.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

mgr/volumes: maintain per subvolume trash directory

PR https://github.com/ceph/ceph/pull/36472 introduces changes
that disallow nested nested snapshots in a subtree (subvolume)
and renames across subvolumes. This effect asynchronous purge
in mgr/volumes as subvolume are moved to a trash directory for
asynchronous deletion by purge threads.

To workaround this, start maintaining a subvolume specific
trash directory. Use the trash directory as an index to the
subvolume specific trash directory entry.

This changes subvolume deletion logic which currently relies
on `--retain-snapshots` flag to decide if the subvolume user
directory should get purged or the subvolume base directory
itself. Deleting a subvolume moves the user facing directory
to its specific trash directory. Purge threads take care of
deleting user facing directories (in trash) and the subvolume
base directory if required (when certain conditions are met).

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: make subvolume_v2::_is_retained() object property

... and drop the leading underscore.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/osd: use map::at() for better readability

avoid using operator[] for find-or-create, to ensure that we fail if the
indexed element is missing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages,osd,crimson: MOSDPGPull::set_pulls(vector<>&&)

this allows us to pass a lvalue reference to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract load_obc_for_recovery() out

extract ReplicatedRecoveryBackend::load_obc_for_recovery()
from ReplicatedRecoveryBackend::recover_object() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract maybe_pull_missing_obj() out

extract ReplicatedRecoveryBackend::maybe_pull_missing_obj() from
ReplicatedRecoveryBackend::recover_object() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/smoke: remove packages on test completion

A new teuthology PR [1] skips package cleanup by default. Use the
lightweight smoke suite to ensure that package cleanup continues to
work.

[1] https://github.com/ceph/teuthology/pull/1553

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36805 into master

* refs/pull/36805/head:
pybind/mgr/volumes: add global lock debug

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #36774 from dillaman/wip-46669-2

librbd: drop deferred AsioEngine destruction

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #36582 from rhcs-dashboard/fix-redirected-URL

mgr/dashboard: redirect to original URL after successful login

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

pybind/mgr/volumes: add global lock debug

Fixes: https://tracker.ceph.com/issues/47149
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36817 from sebastian-philipp/cephadm-pep8-tests

mgr/cephadm: PEP8tify some tests

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36315 from p-se/fix-monitoring-images-ceph-salt

cephadm: deploying of monitoring images partially broken

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #36571 from pcuzner/cephadm-tox-update

cephadm: remove py2 from tox tests

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

pybind/mgr: don't load cython modules

Most of the tests already require the cython modules to
be mocked. This is the final step in making pytest
independent of those modules.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: pytest: Unconditionally mock the rados objects

Always call `mock_ceph_modules()`. Otherwise we'd need to
split the tox run between modules that need to call
`mock_ceph_modules()` and those that don't

Fixes: https://tracker.ceph.com/issues/47141
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: PEP8tify

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify some tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify services/osd.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36794 from sebastian-philipp/cephadm-pep8-cephadmservice.py

mgr/cephadm: PEP8tify cephadmservice.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #36761 from wjwithagen/wjw-fix-dashboard-package.json

dashboard: Fix building build shell script

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36527 from neha-ojha/wip-whitelist-blacklist

Replace 'whitelist' and 'blacklist' language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36509 from pcuzner/cephadm-add-host-facts

cephadm: add gather_facts command

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #36793 from sebastian-philipp/cephadm-pep8-schedule.py

mgr/cephadm: PEP8tify schedule.py

Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: use `check_mon_command` during iscsi config

ensure the mon commands actually succeed

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use `check_mon_command` util

rather then checking the retval explicitly

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: merge NFSGanesha into NFSService class

these two classes really preform the same function

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36757 from mgfritch/cephadm-raw-str-warn

mgr/cephadm: use raw string during `re.search`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: infer latest image with a valid tag

Fixes: https://tracker.ceph.com/issues/47134
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #36787 from tchaikov/wip-47031

crimson/osd: drop misdirected ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: PEP8tify cephadmservice.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: PEP8tify schedule.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>