Merge pull request #24250 from jecluis/wip-mon-increase-ck-default

mon/config-key: increase max key entry size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24200 from jecluis/wip-dashboard-fix-ac

mgr/dashboard: no side-effects on failed user creation

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23351 from tspmelo/wip-rbd-trash

mgr/dashboard: Add support for RBD Trash

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24293 from dillaman/wip-36220

librbd: object copy state machine might dereference a deleted object

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22659 from theanalyst/cmake-pybind-ssl

build: mgr: check for python's ssl version linkage

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24243 from smithfarm/wip-bci-rate-limit

tools: backport-create-issue: rate-limit to avoid seeming like a spammer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #23131 from smithfarm/wip-15100

qa: add test that builds example librados programs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

librbd: object copy state machine might dereference a deleted object

Fixes: http://tracker.ceph.com/issues/36220
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22043 from dillaman/wip-24139

rbd-mirror: bootstrap needs to handle local image id collision

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24253 from dillaman/wip-24516

librbd: potential object map race with copyup state machine

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23625 from XiangxiangMeng/fix-rgw-bucket-policy-not

rgw: policy: fix NotAction, NotPricipal, NotResource does not take effect

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #24062 from nrdmn/fix-quota

rgw: fix max-size in radosgw-admin and REST Admin API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24177 from cbodley/wip-24595

rgw: apply quota config to users created via external auth

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #24199 from aclamk/fio-rados-manual

test/fio: Added tips for compilation of fio with 'rados' engine.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24273 from tspmelo/wip-html-linter

mgr/dashboard: Add html-linter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23534 from cbodley/wip-26897

rgw multisite: enforce spawn_window for data full sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: keep IO blocked until after snapshot object map created

The IO was being unblocked before object map was created, allowing
a potential copyup request to fail to update a still-to-be-created
object map.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not invalidate object map if update races with copyup

The copyup state machine needs to iterate over all object maps to update
the existence for the object. If an snapshot is being removed concurrently,
it's possible to invalidate the object map for the image.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Update frontend linting scripts and documentation

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add html linter and fix lint errors

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #24203 into master

* refs/pull/24203/head:
ceph-fuse: add to selinux profile

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #24221 from cbodley/wip-36034

rgw multisite: uses local DataChangesLog to track active buckets for trim

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23988 from cbodley/wip-35851

rgw: dont access rgw_http_req_data::client of canceled request

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23986 from cbodley/wip-35715

rgw: fix leak of curl handle on shutdown

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24011 from cbodley/wip-qa-rgw-multisite-speedup

qa/rgw: reduce number of multisite log shards

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23987 from cbodley/wip-35812

rgw: RGWAsyncGetBucketInstanceInfo does not access coroutine memory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rbd_mirror_thrash: let daemon gracefully shut down if possible

Otherwise, try to capture a core dump to discover what was blocking the
shutdown process.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: wait max 2 hrs for all stress images to sync

Sporadically the rbd-mirror fsx stress test would fail due to very
slow sync times due to overloaded clusters. Attempt to wait for all
images to be replicated before proceeding with the comparison.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: exclude rbd-mirror sync-point snaps from comparison

This is a temporary workaround to tracker ticket issue #36185

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24274 from ceph/wip-batch-report-fix

ceph-volume: fix journal and filestore data size in `lvm batch --report`

Merge pull request #24251 from alfredodeza/wip-rm27210

ceph-volume custom cluster names fail on filestore trigger

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: only run ceph-common once during tests

Using import_role we can have just one play and avoid rerunning
ceph-common over and over again in daemon specific plays.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24020 into master

* refs/pull/24020/head:
*: set missing CLOEXEC on opened fds
msg: set O_NONBLOCK on file status flags

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24163 into master

* refs/pull/24163/head:
mds/MDCache: fix mds internal op missing events time

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24252 into master

* refs/pull/24252/head:
qa: cleanup workunit dir for each unit
qa: use sudo to cleanup workspace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24232 into master

* refs/pull/24232/head:
githubmap: update contributors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add UI for RBD Trash Purge

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash Delete

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash Restore

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash List

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add UI for RBD Trash Move

Fixes: http://tracker.ceph.com/issues/24272
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add RBD Trash endpoints

Fixes: http://tracker.ceph.com/issues/24272
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace current delete icons with "fa-times"

This will allow us to better differentiate a actual delete from a move to trash.

Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/workunits/rbd: image compare should print byte offset of any deltas

This will assist in debugging any mirroring issues.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24248 from dillaman/wip-rbd-iscsi-docs

doc: ceph-iscsi-api ports should not be public facing

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: no side-effects on failed user creation

Read operations go first, operations with side-effects go last.
Otherwise, something like

1) create user A with non-existent role R1; fails due to R1 DNE
2) create user B with existent role R2; success

will result in both A and B being created; B will have a R2 assigned, A
will have no roles.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: add test that builds example librados programs

Fixes: http://tracker.ceph.com/issues/15100
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24226 into master

* refs/pull/24226/head:
osd/PG: fix not_ready_to_merge behavior for merge target

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24194 into master

* refs/pull/24194/head:
mds: fix mds damaged due to unexpected journal length

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24170 into master

* refs/pull/24170/head:
libcephfs: expose CEPH_SETATTR_MTIME_NOW and CEPH_SETATTR_ATIME_NOW

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #23187 into master

* refs/pull/23187/head:
test: make rank argument mandatory when running journal_tool
cephfs-journal-tool: make "--rank" argument mandatory
cephfs-journal-tool: pass local arg vector for Journal actions
cephfs-journal-tool: dump to per rank output file wherever necessary

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23530 into master

* refs/pull/23530/head:
qa/vstart_runner: fix daemons list
PendingReleaseNotes: note multifs support in libcephfs
test/cephfs: add pybind test for mount_root
pybind/cephfs: enable passing filesystem name to mount
libcephfs: add ceph_select_filesystem
common: add doc strings to client_mds_namespace
client: allow passing fs name to mount()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Conflicts:
    PendingReleaseNotes

ceph-volume: fail fast during tests

This also rsyncs the ceph-volume code to the testing vms before
a ceph.conf is generated because ceph-volume is needed now for this
to figure out the number of osds when you're using 'lvm batch'.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24125 into master

* refs/pull/24125/head:
mon: fix fs add_data_pool setting wrong pool metadata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24158 into master

* refs/pull/24158/head:
cephfs: remove vestiges of mds deactivate

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

ceph-volume: when using batch --report return journal/data size as ints

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24234 into master

* refs/pull/24234/head:
qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml: skip parents test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests/functional update dmcrypt test playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/suites/rbd: increase librbd debug level for mirror-thrash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve debug logging for create/clone state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: always send image RPC messages via cluster

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: permit better tracking between image replayer and bootstrap

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: re-generate new image id upon collision

Fixes: http://tracker.ceph.com/issues/24139
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: create image should return unique error code on id collision

The image id is composed of the librados global instance id and a random
number. For long-lived clients that create multiple images (basically
only rbd-mirror daemon), it's more likely to hit a collision.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: cleanup workunit dir for each unit

This was wrongly dropped and moved to the finalizer.

Introduced-by: de824f74dd8ac909e47335ccd53d7a085e388e41
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use sudo to cleanup workspace

Files in scratch_tmp may not be owned by ubuntu.

Fixes: http://tracker.ceph.com/issues/36165
Introduced-by: de824f74dd8ac909e47335ccd53d7a085e388e41
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests/functional/batch update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional/lvm update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon/config-key: increase max key entry size

Turns out the dashboard requires more than 4K bytes on occasion. Instead
of forcing them to split entries, we can afford them a bit more of extra
size. We could go higher, but they tell us they don't need it.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

ceph-volume main move out the load_ceph_conf_path to the configuration module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd/PG: fix not_ready_to_merge behavior for merge target

Track the *target* not being ready to merge independently from the source,
so that we do not have two PGs fighting over the state of
not_ready_ready_to_merge_source, and so that the map reflects the *source*
PGs readiness only.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24225 into master

* refs/pull/24225/head:
osd/ECBackend: suppress 'Error -2 reading object' if EC fast reads

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml: skip parents test

We can't (easily) build updated hammer packages, but all this sh script does
it run this one test binary with --gtest_filter arguments, so just do
it directly and skip the test explicitly here.  (Newer version of the .sh
understand the environemnt variable but the hammer version does not.)

Fixes: http://tracker.ceph.com/issues/36104
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests.devices patch configuration.load to avoid raising non-existing ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests reset the cluster name after every unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add custom cluster name to centos7-filestore-create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate load the config always by reading the cluster name from the LV

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration be able to set conf.ceph

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: ceph-iscsi-api ports should not be public facing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24078 from Songweibin/wip-rbd-trash-state

rbd: not allowed to restore an image when it is being deleted

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23743 from trociny/wip-rbd-config-pool

librbd: pool and image level config overrides

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23568 from rhcs-dashboard/wip-24573-landing-page

mgr/dashboard: New Landing Page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

tools: backport-create-issue: rate-limit to avoid seeming like a spammer

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23570 from jcsp/wip-rook-api

mgr/rook: update for v1beta1 API

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rgw: fix max-size-kb REST quota parameter and add max-size

Signed-off-by: Nick Erdmann <n@nirf.de>

rgw/radosgw-admin: allow negative values for --max-size

Fix regression in e664488e.

Signed-off-by: Nick Erdmann <n@nirf.de>

githubmap: update contributors

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge PR #24133 into master

* refs/pull/24133/head:
common/Finisher: convert to ceph::mutex etc
common/ceph_mutex: ceph::{mutex,condition_variable,lock_guard}
common/mutex_debug: take const char * to ctor, and require a name
common/mutex_debug: add lockdep support for recursive_mutex_debug
common/mutex_debug: fix whitespace
common/mutex_debug: refactor to remove intermediate class
common/lockdep: add recursive flag for _will_lock
do_cmake.sh: default to Debug build
.gitignore: ignore build.*/

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDCache: fix mds internal op missing events time

Fixes: http://tracker.ceph.com/issues/36114
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

osd/ECBackend: suppress 'Error -2 reading object' if EC fast reads

When fast reads are enabled, it's possible for the ordering of a shard
read to not be enforced with respect to writes that come after because
the read completes on the primary before all shards reply.  This can lead
to an ENOENT on the non-primary, and an ERR message in the cluster log,
even though everything is fine.  (The reply will go back to the primary
with the error but it will be ignored since the read has completed.)

Suppress the error message so we don't see these ERR messages in the
cluster log during the normal course of events.

Fixes: http://tracker.ceph.com/issues/26972
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24202 into master

* refs/pull/24202/head:
mon/MonClient: fix wait for monmap+config is non-cephx case

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #24217 into master

* refs/pull/24217/head:
osd/PG.cc: silence "-Wsign-compare" warnings

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24220 into master

* refs/pull/24220/head:
test/objectstore: set pool for fsck test

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24157 into master

* refs/pull/24157/head:
qa: cleanup parallel execution of fsstress

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24207 into master

* refs/pull/24207/head:
script/ptl-tool.py: fix BASE_PATH

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: cleanup parallel execution of fsstress

Two instances of fsstress clobber each other. Just build it in the local sandbox.

Fixes: http://tracker.ceph.com/issues/24177
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/Finisher: convert to ceph::mutex etc

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_mutex: ceph::{mutex,condition_variable,lock_guard}

If CEPH_DEBUG_MUTEX is defined, use the [recursive_]mutex_debug classes
that implement lockdep and a bucnh of other random debug checks.  Also
typedef ceph::condition_variable to std::condition_variable_debug, which
adds addition assertions and debug checks.

If CEPH_DEBUG_MUTEX is not defined, then use the bare-bones C++ std::mutex
primitives... or as close as we can get to them.

Since the [recursive_]mutex_debug classes take a string argument for the
lockdep piece, define factory functions ceph::make_[recursive_]mutex that
either pass arguments to the debug implementations or toss them out.

Signed-off-by: Sage Weil <sage@redhat.com>