Merge PR #24275 into master

* refs/pull/24275/head:
qa: add timeout to cleaning up workunit sandbox
qa: add timeout to kclient umount
qa: do not cleanup sandbox on error
qa: use default timeout in fs workunits

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24073 from AntonOks/master

doc: added some Civetweb configuration options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24343 from jecluis/wip-dashboard-fix-oa

mgr/dashboard: fix reference to oA

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24237 from votdev/ignore_idea_dir

.gitignore: Ignore .idea directory

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa: add timeout to cleaning up workunit sandbox

If there is a bug preventing rm from completing, the workunit will get stuck.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add timeout to kclient umount

Otherwise QA sits forever waiting for the kclient to umount when there is a
problem.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not cleanup sandbox on error

Otherwise the command will hang if the mount is broken.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use default timeout in fs workunits

Six hours is unnecessarily long.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: fix reference to oA

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #24302 from hydro-b/fix-rbd-nbd-docu

doc: add "--timeout" option to rbd-nbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24289 from ukernel/wip-msg-clear-payload

msg: clear message middle when clearing encoded message buffer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24301 from tchaikov/wip-async-connect-timeout

msg/async: keep connection alive only actually sending

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #24278 from wjwithagen/wjw-fix-inline_variant

common: fix missing include boost/noncopyable.hpp

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24263 from tchaikov/wip-drop-lock-policy

auth,common: use ceph::mutex instead of LockMutex

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24336 from alfredodeza/wip-rm36251

ceph-volume add new ceph-handlers role from ceph-ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: add "--timeout" option to rbd-nbd

Fixes: https://tracker.ceph.com/issues/22333
Signed-off-by: Stefan Kooman <stefan@bit.nl>

ceph-volume tests.functional add new ceph-handlers role from ceph-ansible

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24330 from alfredodeza/wip-rm36098

ceph-volume:  lsblk can fail to find PARTLABEL, must fallback to blkid

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24315 from tchaikov/wip-jenkins-does-not-read-package-desc

install-deps: do not specify unknown options

Reviewed-by: Alfredo Deza <adeza@redhat.com>

install-deps: do not specify unknown options

per apt.conf(5), and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550564,
"Acquire::Translation" is not documented and will be deprecated. it
would be better to use "Acquire::Languages" solely. and it's consistent
with https://github.com/ceph/ceph-build/pull/1137 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume simple.scan better detection for ceph-disk data devices

Uses the new ``CephDiskDevice`` to look into PARTLABEL from both lsblk
(the default) falling back to blkid, which in some cases has the right
value when lsblk fails.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util verify CephDiskDevice behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.conftest add blkid patching to device_info fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device add a ceph-disk device object to detect types

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24300 from trociny/wip-36236

test/rbd_mirror: race in WaitingOnLeaderReleaseLeader

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23950 from ceph/wip-rgw-dpp-metasync

rgw: log refactoring for meta sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

auth: switch from Mutex to ceph::mutex

to be aligned with our direction to use std::mutex or its wrapper. see
https://trello.com/c/aUSezBRH/365-cephmutex-etc-for-release-builds

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth,common: switch to ceph::mutex, etc

in this change, along with LockPolicy, src/common/lock_* are completely
removed.

instead of using LockPolicy based template specialization of LockMutex,
etc, it would be simpler if we can just rely on WITH_SEASTAR
preprocessor macro to tell if we are compiling code for crimson or not.
but please bear in mind, we cannot link against the plain libceph-common
in crimson anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_mutex: add shared_mutex to `ceph` namespace

for the sake of completeness, also pave the road to ditch
SharedMutex<LockPolicy>. please note, in code used by crimson, we are
not using shared_mutex or SharedMutex yet. so we are not adding
the dummy shared_mutex for WITH_SEASTAR.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: keep connection alive only actually sending

When connection stuck into odd state, we need to let connection timeout.
If send_message could update last_active, it won't play the role as
expected.
So we move last_active updated to the place actually should.

Signed-off-by: Haomai Wang <haomai@xsky.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/rbd_mirror: race in WaitingOnLeaderReleaseLeader

Fixes: http://tracker.ceph.com/issues/36236
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24193 from rhcs-dashboard/coverage-disabled-in-api-tests

mgr/dashboard: disable backend tests coverage

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #24250 from jecluis/wip-mon-increase-ck-default

mon/config-key: increase max key entry size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24200 from jecluis/wip-dashboard-fix-ac

mgr/dashboard: no side-effects on failed user creation

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23351 from tspmelo/wip-rbd-trash

mgr/dashboard: Add support for RBD Trash

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24293 from dillaman/wip-36220

librbd: object copy state machine might dereference a deleted object

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22659 from theanalyst/cmake-pybind-ssl

build: mgr: check for python's ssl version linkage

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24243 from smithfarm/wip-bci-rate-limit

tools: backport-create-issue: rate-limit to avoid seeming like a spammer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #23131 from smithfarm/wip-15100

qa: add test that builds example librados programs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

rgw: no DoutPrefixProvider as arg for RGWRemoteDataLog::init

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: pass DoutPrefixProvider into the entrypoints for trim

Also add a DoutPrefixProvider to all of the
cascading classes that need one as a result

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: add global prefix provider for radosgw-admin

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a5dbc81b774d0be4b51f39e37bcfa15f3b36db8)

rgw: remove unused variable RGWRados *store

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ldout with ldpp_dout in coroutine classes

The only classes in rgw_sync.cc that remain
with ldouts are:
- RGWBackoffControlCR
- RGWShardCollectCR
- RGWAsyncMetaStoreEntry
- RGWAsyncMetaRemoveEntry
- PurgePeriodLogsCR
- take_min_status
- MetaMasterTrimShardCollectCR
- MetaMasterStatusCollectCR
- MetaMasterTrimCR
- MetaPeerTrimCR
- MetaTrimPollCR

These classes do not have RGWMetaSyncEnv members.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ldouts with ldpp_dout in RGWRemoteMetaLog

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: MetaSyncStatusManager implements DoutPrefixProvider

There are no member functions of RGWMetaSyncStatusManager
with ldouts to replace with ldpp_dout.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

librbd: object copy state machine might dereference a deleted object

Fixes: http://tracker.ceph.com/issues/36220
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22043 from dillaman/wip-24139

rbd-mirror: bootstrap needs to handle local image id collision

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24253 from dillaman/wip-24516

librbd: potential object map race with copyup state machine

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23625 from XiangxiangMeng/fix-rgw-bucket-policy-not

rgw: policy: fix NotAction, NotPricipal, NotResource does not take effect

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #24062 from nrdmn/fix-quota

rgw: fix max-size in radosgw-admin and REST Admin API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24177 from cbodley/wip-24595

rgw: apply quota config to users created via external auth

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #24199 from aclamk/fio-rados-manual

test/fio: Added tips for compilation of fio with 'rados' engine.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24273 from tspmelo/wip-html-linter

mgr/dashboard: Add html-linter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23534 from cbodley/wip-26897

rgw multisite: enforce spawn_window for data full sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: keep IO blocked until after snapshot object map created

The IO was being unblocked before object map was created, allowing
a potential copyup request to fail to update a still-to-be-created
object map.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not invalidate object map if update races with copyup

The copyup state machine needs to iterate over all object maps to update
the existence for the object. If an snapshot is being removed concurrently,
it's possible to invalidate the object map for the image.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Update frontend linting scripts and documentation

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add html linter and fix lint errors

Signed-off-by: Tiago Melo <tmelo@suse.com>

msg: clear message middle when clearing encoded message buffer

Otherwise it may trigger ceph_assert(middle.length() == 0) in
Message::encode()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

common: fix missing include boost/noncopyable.hpp

Clang complains:
```
In file included from /home/jenkins/workspace/ceph-master/src/osd/ReplicatedBackend.cc:15:
In file included from /home/jenkins/workspace/ceph-master/src/osd/ReplicatedBackend.h:18:
In file included from /home/jenkins/workspace/ceph-master/src/osd/PGBackend.h:27:
In file included from /home/jenkins/workspace/ceph-master/src/osd/PGTransaction.h:25:
/home/jenkins/workspace/ceph-master/src/common/inline_variant.h:88:64: error: expected class name
struct generic_visitor : boost::static_visitor<Result>, boost::noncopyable
                                                               ^
1 error generated.
gmake[2]: *** [src/osd/CMakeFiles/osd.dir/build.make:141: src/osd/CMakeFiles/osd.dir/ReplicatedBackend.cc.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #24203 into master

* refs/pull/24203/head:
ceph-fuse: add to selinux profile

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #24221 from cbodley/wip-36034

rgw multisite: uses local DataChangesLog to track active buckets for trim

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23988 from cbodley/wip-35851

rgw: dont access rgw_http_req_data::client of canceled request

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23986 from cbodley/wip-35715

rgw: fix leak of curl handle on shutdown

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24011 from cbodley/wip-qa-rgw-multisite-speedup

qa/rgw: reduce number of multisite log shards

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23987 from cbodley/wip-35812

rgw: RGWAsyncGetBucketInstanceInfo does not access coroutine memory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rbd_mirror_thrash: let daemon gracefully shut down if possible

Otherwise, try to capture a core dump to discover what was blocking the
shutdown process.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: wait max 2 hrs for all stress images to sync

Sporadically the rbd-mirror fsx stress test would fail due to very
slow sync times due to overloaded clusters. Attempt to wait for all
images to be replicated before proceeding with the comparison.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: exclude rbd-mirror sync-point snaps from comparison

This is a temporary workaround to tracker ticket issue #36185

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests.util verify new blkid API calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24274 from ceph/wip-batch-report-fix

ceph-volume: fix journal and filestore data size in `lvm batch --report`

ceph-volume util.disk expand blkid's API calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24251 from alfredodeza/wip-rm27210

ceph-volume custom cluster names fail on filestore trigger

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: only run ceph-common once during tests

Using import_role we can have just one play and avoid rerunning
ceph-common over and over again in daemon specific plays.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24020 into master

* refs/pull/24020/head:
*: set missing CLOEXEC on opened fds
msg: set O_NONBLOCK on file status flags

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24163 into master

* refs/pull/24163/head:
mds/MDCache: fix mds internal op missing events time

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24252 into master

* refs/pull/24252/head:
qa: cleanup workunit dir for each unit
qa: use sudo to cleanup workspace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24232 into master

* refs/pull/24232/head:
githubmap: update contributors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add UI for RBD Trash Purge

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash Delete

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash Restore

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash List

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash Move

Fixes: http://tracker.ceph.com/issues/24272
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add RBD Trash endpoints

Fixes: http://tracker.ceph.com/issues/24272
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace current delete icons with "fa-times"

This will allow us to better differentiate a actual delete from a move to trash.

Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/workunits/rbd: image compare should print byte offset of any deltas

This will assist in debugging any mirroring issues.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24248 from dillaman/wip-rbd-iscsi-docs

doc: ceph-iscsi-api ports should not be public facing

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

common/ceph_mutex: add dummy mutex for WITH_SEASTAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: no side-effects on failed user creation

Read operations go first, operations with side-effects go last.
Otherwise, something like

1) create user A with non-existent role R1; fails due to R1 DNE
2) create user B with existent role R2; success

will result in both A and B being created; B will have a R2 assigned, A
will have no roles.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: add test that builds example librados programs

Fixes: http://tracker.ceph.com/issues/15100
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Ignore .idea directory (created by PyCharm for example)

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #24226 into master

* refs/pull/24226/head:
osd/PG: fix not_ready_to_merge behavior for merge target

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24194 into master

* refs/pull/24194/head:
mds: fix mds damaged due to unexpected journal length

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24170 into master

* refs/pull/24170/head:
libcephfs: expose CEPH_SETATTR_MTIME_NOW and CEPH_SETATTR_ATIME_NOW

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #23187 into master

* refs/pull/23187/head:
test: make rank argument mandatory when running journal_tool
cephfs-journal-tool: make "--rank" argument mandatory
cephfs-journal-tool: pass local arg vector for Journal actions
cephfs-journal-tool: dump to per rank output file wherever necessary

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23530 into master

* refs/pull/23530/head:
qa/vstart_runner: fix daemons list
PendingReleaseNotes: note multifs support in libcephfs
test/cephfs: add pybind test for mount_root
pybind/cephfs: enable passing filesystem name to mount
libcephfs: add ceph_select_filesystem
common: add doc strings to client_mds_namespace
client: allow passing fs name to mount()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Conflicts:
    PendingReleaseNotes

ceph-volume: fail fast during tests

This also rsyncs the ceph-volume code to the testing vms before
a ceph.conf is generated because ceph-volume is needed now for this
to figure out the number of osds when you're using 'lvm batch'.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24125 into master

* refs/pull/24125/head:
mon: fix fs add_data_pool setting wrong pool metadata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24158 into master

* refs/pull/24158/head:
cephfs: remove vestiges of mds deactivate

Reviewed-by: Douglas Fuller <dfuller@redhat.com>