Merge pull request #24180 from colletj/wip-35914-mmgrconfigure-osd-query

mgr: update MMgrConfigure message to include optional OSD perf queries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: add optional OSD perf queries to MMgrConfigure message

Fixes: https://tracker.ceph.com/issues/35914
Signed-off-by: Julien Collet <julien.collet@cern.ch>

Merge pull request #24349 from ricardoasmarques/wip-ex-message-py3

mgr/dashboard: Exception.message doesn't exist on Python 3

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24266 from theanalyst/doc/releases/13.2.2

doc: release notes for 13.2.2 mimic

Reviewed-By: Nathan Cutler <nculter@suse.com>

Merge pull request #24354 from idryomov/wip-krbd-kfsx-fancy-striping

qa/suites/krbd: more fsx tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24291 from tchaikov/wip-23964

erasure-code: add clay codes

Reviewed-by: Myna Vajha <mynaramana@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24256 from dzafman/wip-36170

osd: Remove old bft= which has been superceded by backfill=

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24147 from sebastian-philipp/orchestrator-debuggability

mgr/orchestrator: Improve debuggability

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #24346 into master

* refs/pull/24346/head:
doc: misc fixes cephfs-shell

Reviewed-by: Pavani Rajula <rpavani1998@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24340 from tchaikov/wip-36260

qa/workunits/mon/test_mon_config_key.py: bump up the size limit

Merge PR #21566 into master

* refs/pull/21566/head:
test: add test for mds drop cache command
mds: command to trim mds cache and client caps
mds: implement journal flush as asynchronous context execution
mds: cleanup some asok commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24355 from alfredodeza/wip-rm36249

ceph-volume:  activate option --auto-detect-objectstore respects --no-systemd

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24044 into master

* refs/pull/24044/head:
qa: tolerate failed rank while waiting for state

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #23674 from wjwithagen/wjw-wip-ceph-volume-zfs

ceph-volume: zfs, the initial first submit.

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24275 into master

* refs/pull/24275/head:
qa: add timeout to cleaning up workunit sandbox
qa: add timeout to kclient umount
qa: do not cleanup sandbox on error
qa: use default timeout in fs workunits

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-volume lvm.activate pass no_systemd args when detecting objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.lvm verify activate works with --auto-detect-objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: zfs, the inital first submit.

Does nothing other than invoce the ceph-volume-zfs modules when
called with 'ceph-volume zfs

Like:
> ceph-volume zfs list

Use ZFS as the underlying technology for OSDs

--verbose   Increase the verbosity level

['list']
-->  TypeError: main() takes exactly 2 arguments (1 given)
Exit 1

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/suites/krbd: more fsx tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: don't use time() for seeding

When starting multiple concurrent jobs with rbd_fsx task, all jobs get
the same seed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #24073 from AntonOks/master

doc: added some Civetweb configuration options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24343 from jecluis/wip-dashboard-fix-oa

mgr/dashboard: fix reference to oA

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24237 from votdev/ignore_idea_dir

.gitignore: Ignore .idea directory

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Exception.message doesn't exist on Python 3

Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc: misc fixes cephfs-shell

Signed-off-by: Jos Collin <jcollin@redhat.com>

qa: add timeout to cleaning up workunit sandbox

If there is a bug preventing rm from completing, the workunit will get stuck.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add timeout to kclient umount

Otherwise QA sits forever waiting for the kclient to umount when there is a
problem.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not cleanup sandbox on error

Otherwise the command will hang if the mount is broken.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use default timeout in fs workunits

Six hours is unnecessarily long.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: fix reference to oA

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/workunits/mon/test_mon_config_key.py: bump up the size limit

in b38b8e980cb477ab2b0f320ab51eaa0c0fec7da6, we changed the upper
limit of size of `config key` 's value to 64k, so we need to update
the test accordingly.

Fixes: http://tracker.ceph.com/issues/36260
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24302 from hydro-b/fix-rbd-nbd-docu

doc: add "--timeout" option to rbd-nbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24289 from ukernel/wip-msg-clear-payload

msg: clear message middle when clearing encoded message buffer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24301 from tchaikov/wip-async-connect-timeout

msg/async: keep connection alive only actually sending

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #24278 from wjwithagen/wjw-fix-inline_variant

common: fix missing include boost/noncopyable.hpp

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24263 from tchaikov/wip-drop-lock-policy

auth,common: use ceph::mutex instead of LockMutex

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24336 from alfredodeza/wip-rm36251

ceph-volume add new ceph-handlers role from ceph-ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: add "--timeout" option to rbd-nbd

Fixes: https://tracker.ceph.com/issues/22333
Signed-off-by: Stefan Kooman <stefan@bit.nl>

ceph-volume tests.functional add new ceph-handlers role from ceph-ansible

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24330 from alfredodeza/wip-rm36098

ceph-volume:  lsblk can fail to find PARTLABEL, must fallback to blkid

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24315 from tchaikov/wip-jenkins-does-not-read-package-desc

install-deps: do not specify unknown options

Reviewed-by: Alfredo Deza <adeza@redhat.com>

test: add test for mds drop cache command

Fixes: http://tracker.ceph.com/issues/23362
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: command to trim mds cache and client caps

With this command, the MDS would request clients to release
caps followed by trimming its own cache and a journal flush.
The command accepts a timeout to wait for clients to respond
to session recall and flush messages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Rishabh Dave <ridave@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: implement journal flush as asynchronous context execution

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: cleanup some asok commands

Signed-off-by: Rishabh Dave <ridave@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

install-deps: do not specify unknown options

per apt.conf(5), and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550564,
"Acquire::Translation" is not documented and will be deprecated. it
would be better to use "Acquire::Languages" solely. and it's consistent
with https://github.com/ceph/ceph-build/pull/1137 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume simple.scan better detection for ceph-disk data devices

Uses the new ``CephDiskDevice`` to look into PARTLABEL from both lsblk
(the default) falling back to blkid, which in some cases has the right
value when lsblk fails.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util verify CephDiskDevice behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.conftest add blkid patching to device_info fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device add a ceph-disk device object to detect types

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24300 from trociny/wip-36236

test/rbd_mirror: race in WaitingOnLeaderReleaseLeader

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23950 from ceph/wip-rgw-dpp-metasync

rgw: log refactoring for meta sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

auth: switch from Mutex to ceph::mutex

to be aligned with our direction to use std::mutex or its wrapper. see
https://trello.com/c/aUSezBRH/365-cephmutex-etc-for-release-builds

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth,common: switch to ceph::mutex, etc

in this change, along with LockPolicy, src/common/lock_* are completely
removed.

instead of using LockPolicy based template specialization of LockMutex,
etc, it would be simpler if we can just rely on WITH_SEASTAR
preprocessor macro to tell if we are compiling code for crimson or not.
but please bear in mind, we cannot link against the plain libceph-common
in crimson anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_mutex: add shared_mutex to `ceph` namespace

for the sake of completeness, also pave the road to ditch
SharedMutex<LockPolicy>. please note, in code used by crimson, we are
not using shared_mutex or SharedMutex yet. so we are not adding
the dummy shared_mutex for WITH_SEASTAR.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: keep connection alive only actually sending

When connection stuck into odd state, we need to let connection timeout.
If send_message could update last_active, it won't play the role as
expected.
So we move last_active updated to the place actually should.

Signed-off-by: Haomai Wang <haomai@xsky.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/rbd_mirror: race in WaitingOnLeaderReleaseLeader

Fixes: http://tracker.ceph.com/issues/36236
Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/suites/rados/thrash-erasure-code: add clay 4+2

Signed-off-by: Sage Weil <sage@redhat.com>

erasure-code: add clay codes

Introducing Clay codes to ceph. These codes are optimal
in terms of network, disk traffic utilized during repair of a lost chunk.
This feature also provides gradual increase in network, disk traffic with
respect to the number of lost chunks.

Authors: Myna, Elita.
Fixes: http://tracker.ceph.com/issues/19278
Signed-off-by: Myna V <mynaramana@gmail.com>

Merge pull request #24193 from rhcs-dashboard/coverage-disabled-in-api-tests

mgr/dashboard: disable backend tests coverage

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #24250 from jecluis/wip-mon-increase-ck-default

mon/config-key: increase max key entry size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24200 from jecluis/wip-dashboard-fix-ac

mgr/dashboard: no side-effects on failed user creation

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23351 from tspmelo/wip-rbd-trash

mgr/dashboard: Add support for RBD Trash

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24293 from dillaman/wip-36220

librbd: object copy state machine might dereference a deleted object

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22659 from theanalyst/cmake-pybind-ssl

build: mgr: check for python's ssl version linkage

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24243 from smithfarm/wip-bci-rate-limit

tools: backport-create-issue: rate-limit to avoid seeming like a spammer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #23131 from smithfarm/wip-15100

qa: add test that builds example librados programs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

rgw: no DoutPrefixProvider as arg for RGWRemoteDataLog::init

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: pass DoutPrefixProvider into the entrypoints for trim

Also add a DoutPrefixProvider to all of the
cascading classes that need one as a result

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: add global prefix provider for radosgw-admin

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a5dbc81b774d0be4b51f39e37bcfa15f3b36db8)

rgw: remove unused variable RGWRados *store

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ldout with ldpp_dout in coroutine classes

The only classes in rgw_sync.cc that remain
with ldouts are:
- RGWBackoffControlCR
- RGWShardCollectCR
- RGWAsyncMetaStoreEntry
- RGWAsyncMetaRemoveEntry
- PurgePeriodLogsCR
- take_min_status
- MetaMasterTrimShardCollectCR
- MetaMasterStatusCollectCR
- MetaMasterTrimCR
- MetaPeerTrimCR
- MetaTrimPollCR

These classes do not have RGWMetaSyncEnv members.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ldouts with ldpp_dout in RGWRemoteMetaLog

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: MetaSyncStatusManager implements DoutPrefixProvider

There are no member functions of RGWMetaSyncStatusManager
with ldouts to replace with ldpp_dout.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

librbd: object copy state machine might dereference a deleted object

Fixes: http://tracker.ceph.com/issues/36220
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22043 from dillaman/wip-24139

rbd-mirror: bootstrap needs to handle local image id collision

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24253 from dillaman/wip-24516

librbd: potential object map race with copyup state machine

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23625 from XiangxiangMeng/fix-rgw-bucket-policy-not

rgw: policy: fix NotAction, NotPricipal, NotResource does not take effect

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #24062 from nrdmn/fix-quota

rgw: fix max-size in radosgw-admin and REST Admin API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24177 from cbodley/wip-24595

rgw: apply quota config to users created via external auth

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #24199 from aclamk/fio-rados-manual

test/fio: Added tips for compilation of fio with 'rados' engine.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24273 from tspmelo/wip-html-linter

mgr/dashboard: Add html-linter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23534 from cbodley/wip-26897

rgw multisite: enforce spawn_window for data full sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: keep IO blocked until after snapshot object map created

The IO was being unblocked before object map was created, allowing
a potential copyup request to fail to update a still-to-be-created
object map.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not invalidate object map if update races with copyup

The copyup state machine needs to iterate over all object maps to update
the existence for the object. If an snapshot is being removed concurrently,
it's possible to invalidate the object map for the image.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Update frontend linting scripts and documentation

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add html linter and fix lint errors

Signed-off-by: Tiago Melo <tmelo@suse.com>

msg: clear message middle when clearing encoded message buffer

Otherwise it may trigger ceph_assert(middle.length() == 0) in
Message::encode()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

common: fix missing include boost/noncopyable.hpp

Clang complains:
```
In file included from /home/jenkins/workspace/ceph-master/src/osd/ReplicatedBackend.cc:15:
In file included from /home/jenkins/workspace/ceph-master/src/osd/ReplicatedBackend.h:18:
In file included from /home/jenkins/workspace/ceph-master/src/osd/PGBackend.h:27:
In file included from /home/jenkins/workspace/ceph-master/src/osd/PGTransaction.h:25:
/home/jenkins/workspace/ceph-master/src/common/inline_variant.h:88:64: error: expected class name
struct generic_visitor : boost::static_visitor<Result>, boost::noncopyable
                                                               ^
1 error generated.
gmake[2]: *** [src/osd/CMakeFiles/osd.dir/build.make:141: src/osd/CMakeFiles/osd.dir/ReplicatedBackend.cc.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #24203 into master

* refs/pull/24203/head:
ceph-fuse: add to selinux profile

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #24221 from cbodley/wip-36034

rgw multisite: uses local DataChangesLog to track active buckets for trim

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23988 from cbodley/wip-35851

rgw: dont access rgw_http_req_data::client of canceled request

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23986 from cbodley/wip-35715

rgw: fix leak of curl handle on shutdown

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24011 from cbodley/wip-qa-rgw-multisite-speedup

qa/rgw: reduce number of multisite log shards

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23987 from cbodley/wip-35812

rgw: RGWAsyncGetBucketInstanceInfo does not access coroutine memory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rbd_mirror_thrash: let daemon gracefully shut down if possible

Otherwise, try to capture a core dump to discover what was blocking the
shutdown process.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: wait max 2 hrs for all stress images to sync

Sporadically the rbd-mirror fsx stress test would fail due to very
slow sync times due to overloaded clusters. Attempt to wait for all
images to be replicated before proceeding with the comparison.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: exclude rbd-mirror sync-point snaps from comparison

This is a temporary workaround to tracker ticket issue #36185

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests.util verify new blkid API calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24274 from ceph/wip-batch-report-fix

ceph-volume: fix journal and filestore data size in `lvm batch --report`