Merge pull request #29878 from pdvian/wip-41096-nautilus

nautilus: cephfs: avoid map been inserted by mistake

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29879 from pdvian/wip-41099-nautilus

nautilus: cephfs: fix a memory leak

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29938 from pdvian/wip-41107-nautilus

nautilus: mds: delay exporting directory whose pin value exceeds max rank id

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29983 from pdvian/wip-41128-nautilus

nautilus: qa: power off still resulted in client sending session close

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30026 from smithfarm/wip-40895-nautilus

nautilus: pybind/cephfs: Modification to error message

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30030 from smithfarm/wip-40887-nautilus

nautilus: ceph_volume_client: convert string to bytes object

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30031 from smithfarm/wip-40900-nautilus

nautilus: mds: evict an unresponsive client only when another client wants its caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30032 from smithfarm/wip-41113-nautilus

nautilus: cephfs: client: more precise CEPH_CLIENT_CAPS_PENDING_CAPSNAP

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30038 from smithfarm/wip-41276-nautilus

nautilus: tests: cephfs: fix malformed qa suite config

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30039 from smithfarm/wip-41465-nautilus

nautilus: cephfs: mount.ceph: properly handle -o strictatime

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30040 from smithfarm/wip-41467-nautilus

nautilus: mds: trim cache on regular schedule

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30041 from smithfarm/wip-41477-nautilus

nautilus: mds: fix InoTable::force_consume_to()

Merge pull request #30057 from varshar16/wip-nautilus-cephfs-shell-path-conversion

nautilus: cephfs-shell: Convert paths type from string to bytes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30418 from pdvian/wip-41851-nautilus

nautilus: mds: make MDSIOContextBase delete itself when shutting down

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30442 from pdvian/wip-41855-nautilus

nautilus: client: nfs-ganesha with cephfs client, removing dir reports not empty

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30455 from smithfarm/wip-41889-nautilus

nautilus: mgr/volumes: handle exceptions in purge thread with retry

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30508 from ukernel/nautilus-41948

nautilus: mds: wake up lock waiters after forcibly changing lock state

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30325 from theanalyst/wip-41498-nautilus

nautilus: rgw: fix the bug of rgw not doing necessary checking to website configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30437 from smithfarm/wip-41846-nautilus

nautilus: rgw: increase beast parse buffer size to 64k

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30604 from vumrao/wip-vumrao-41976

nautilus: radosgw-admin: add --uid check in bucket list command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30651 from pdvian/wip-41970-nautilus

nautilus: rgw: ldap auth: S3 auth failure should return InvalidAccessKeyId

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30680 from pdvian/wip-41974-nautilus

nautilus: rgw: add minssing admin property when sync user info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30247 from pdvian/wip-41631-nautilus

nautilus: rgw: fixed "unrecognized arg" error when using "radosgw-admin zone rm".

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30248 from pdvian/wip-41627-nautilus

nautilus: rgw: RGWCoroutine::call(nullptr) sets retcode=0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30252 from smithfarm/wip-41707-nautilus

nautilus: rgw: fix cls_bucket_list_unordered() partial results

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30509 from pdvian/wip-41898-nautilus

nautilus: rgw: fix data sync start delay if remote haven't init data_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30472 from pdvian/wip-41858-nautilus

nautilus: rgw: fix memory growth while deleteing objects with

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29716 from xiexingguo/wip-build-push-segv-for-n

nautilus: core: osd/ReplicatedBackend: check against empty data_included before enabling crc

Merge pull request #29928 from smithfarm/wip-41534-nautilus

nautilus: tests: valgrind: UninitCondition in ceph::crypto::onwire::AES128GCM_OnWireRxHandler::authenticated_decrypt_update_final()

Merge pull request #29946 from smithfarm/wip-39412-nautilus

nautilus: core: pool_stat.dump() - value of num_store_stats is wrong

Merge pull request #30278 from pdvian/wip-41703-nautilus

nautilus: osd/PrimaryLogPG: update oi.size on write op implicitly truncating ob…

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30605 from vumrao/wip-vumrao-41963

nautilus: tools/rados: call pool_lookup() after rados is connected

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30607 from vumrao/wip-vumrao-41960

nautilus: tools/rados: add --pgid in help

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30708 from smithfarm/wip-epel-release-snafu-nautilus

nautilus: build/ops: install-deps: do not install if rpm already installed and ceph.spec.in: s/pkgversion/version_nodots/

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

ceph.spec.in: fix Cython package dependency for Fedora

Fedora distros do not have python3?-Cython packages, but they do have
python3-Cython ones. Fix the BuildRequires so that we only use the
python3_version_nodots based version string for RHEL.

Fixes: https://tracker.ceph.com/issues/42032
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 40c5a18f81c67c96f6a35e592ac3141b87c19701)

ceph.spec.in: s/pkgversion/version_nodots/

`python3_pkgversion` is now defined as 3, while we don't have packages
like python3-Cython yet in EPEL7. but we do have `python36-Cython`. so
let's use `python3_version_nodots` instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6d2f66846d1b84595cfdf8ac13a7ed4e1ae608ce)

install-deps: do not install if rpm already installed

there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 45a4173bcfe15a9bf56f6aeb6e6c35386a00d1c5)

Merge pull request #30579 from yuvalif/wip-yuval-backport-notif-nautilus

nautilus: rgw/pubsub: backport notifications and pubsub

Merge pull request #30601 from smithfarm/wip-42070-nautilus

nautilus: build/ops: install-deps.sh: add EPEL repo for non-x86_64 archs as well

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge branch 'nautilus' into wip-yuval-backport-notif-nautilus

Merge pull request #30686 from smithfarm/wip-40131-nautilus-follow-on

nautilus: doc: fix urls in posix.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Fixes: https://tracker.ceph.com/issues/42023
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 49d740118f9a0ee6b6bb82954630c654946f1751)

Merge pull request #30322 from theanalyst/wip-41700-nautilus

nautilus: build/ops: install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30245 from smithfarm/wip-41711-nautilus

nautilus: doc/ceph-kvstore-tool: add description for 'stats' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix urls in posix.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit c5c5f031f77f778af73f8659b01695b88f158807)

Merge pull request #30664 from smithfarm/wip-42105-nautilus

nautilus: build/ops: admin/build-doc: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30004 from smithfarm/wip-41568-nautilus

nautilus: doc: pg_num should always be a power of two

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30003 from smithfarm/wip-41529-nautilus

nautilus: doc: default values for mon_health_to_clog_* were flipped

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: add minssing admin property when sync user info.

Fixes: https://tracker.ceph.com/issues/41643
Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit 9ad975d422aba67c0bf7bc1b114079331e2f7fd5)

Merge pull request #29731 from pdvian/wip-41081-nautilus

nautilus: doc: Fix rbd namespace documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

admin/build-doc: use python3

to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ace8cb15a600daf5261c0ab349b43010a238a1d9)

Merge PR #30025 into nautilus

* refs/pull/30025/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: ldap auth: S3 auth failure should return InvalidAccessKeyId

LDAP was out of sync with the other auth modules, and the default
error code is not correct for S3.

Fixes: https://tracker.ceph.com/issues/41760
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 226227cf2d825b3414b7522a9f738180142f3956)

Merge pull request #30649 from lordcirth/wip-doc-misplaced-ratio

doc: max_misplaced option was renamed in Nautilus

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: max_misplaced was renamed in Nautilus

mgr/balancer/max_misplaced was changed to target_max_misplaced_ratio
in 14.2.0.

Signed-off-by: Nathan Fish <lordcirth@gmail.com>

Merge pull request #30554 from alfredodeza/nautilus-guits-41392

nautilus ceph-volume: do not fail when trying to remove crypt mapper

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@redhat.com>

tools/rados: add --pgid in help

This commit 825c7b9729b2410a116e123d4b21210c646704d4 brought
the support for --pgid in rados ls but was not in help.
Fixes: https://tracker.ceph.com/issues/41876
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit ae9a24eaba330fd3f6946063c2bc926f58c93772)

tools/rados: call pool_lookup() after rados is connected
This commit fixes a segmentation fault when using --pgid
option in rados ls command in combination with --pool/-p option.
The reason for the crash was that we can not use the rados object
before connecting it with the cluster using rados.init_with_context().

Fixes: https://tracker.ceph.com/issues/41875
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit 56b6dd9bd7a50371062128122e6a899f2c79c5f5)

radosgw-admin: add --uid check in bucket list command
Fixes: https://tracker.ceph.com/issues/41589
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit c53597b561837340314f0a169f6881c1814ce0b9)

 Conflicts:
src/rgw/rgw_admin.cc
        - no --new-uid

Merge pull request #30049 from smithfarm/wip-41333-nautilus

nautilus: rpm: always build ceph-test package

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

build-doc: allow building docs on fedora 30

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e17e4bfcc048abd1757c3efd12c7443169d67477)

rgw/pubsub: backporting pubsub/notifications to nautilus

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/pubsub: add notification filtering

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 543f8f7a2729ab8bd2e1a14d98a3fd2c2e19f9bd)

rgw/pubsub: support deletion markers and multipart upload

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 923799fd960f97b8ddbbe0e106139069d640588b)

rgw/pubsub: push notifications documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit c345566831c72446c8250f630337825e526847a8)

rgw/pubsub: push notifications from ops

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dc31b030ea30f3b7bfdeb9fa92e303892d17138a)

rgw/pubsub: allow pubsub REST API on master

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit a14353496d1f75a303495f538141a1a54ff28c60)

rgw/pubsub: service reordering issue

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e91489f443b5b086ef96b51051d2726728412ff5)

rgw/pubsub: fix amqp topic bug. add disabled end2end push tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit abfd6f72db3b6e1a797a660a4a601a33605e5f25)

rgw/pubsub: fix duplicates due to multiple zone synching

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 38c23833b35279cb6e1a9b575b5a463959fed348)

rgw/pubsub: run pubsub tests even if multisite fails

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 1df809144c2ed96a715c8bd3df6ed5ef48399dd3)

rgw/pubsub: make new PSZone parameters optional in test

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 0b6909c1e9832d7a6f375f38f32bed58d1e161c2)

rgw/pubsub: add conf parameter for full/incremental sync

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit af4c2d5dbb42170ade4ad78088b80b0b5a357c5a)

rgw: add tenant as parameter to User in multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 4def929395bce39e9a130d0e8740790815a6a503)

rgw/pubsub: revert the RGWSysObjectCtx change

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 10b8b57d723971604cbabf984291232c194a55ec)

rgw/pubsub: cleanup tests for multiple notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit b4005db82da1627a03d96baa45081fbb329070b1)

rgw/pubsub: fix doc on updates. attempt to fix multi-notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 3279a786ba0abc9d1955f03eaa5f94da0d12ed22)

rgw/pubsub: fix more test issues with teuthology failures

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 51fc8a40e3484bcc16244723b091ba7f3157abfb)

rgw/pubsub: fix test issue with 3 zones

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 83e5571c873a6f459aaf9a7c42a532339ed2a1b3)

rgw/pubsub: add another zone for pubsub

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit d8d817a6006e38b52f922b714870c32b9a76a169)

rgw/pubsub: make sure bucket is empty before deletion

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 588e0c0e23f66eb35a056ad110404a97fa219a15)

rgw/pubsub: fix "no module named rgw_multi.zone_ps"

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dfa501cfb77e72fe3e4cb2454aea99916705b812)

rgw/pubsub: fix "no module named rgw_multi.tests"

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit b03a56fe930be0ff7bad4e375e22a8e2ca0fed76)

rgw/pubsub: actually adding the ps tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 8362ecc7dfd8056567b74fdaaa52809082f15421)

rgw/pubsub: fix popping the wrong key

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 2d7c07bbeb9887d76b89eded7458ac7af8068026)

rgw/pubsub: fix topic arn. tenant support to multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit be64adf916e0f318b22b57cedd02c2b0b2a503f2)

rgw/pubsub: handle subscription conf errors better

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit f4e3745bcb9f75814b4052e2b651c25a39c2f17b)

rgw/pubsub: more info on notification deletion compatibility

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 2cf1872bdf63e9093884b6d3bbc20b9a42963d53)

rgw/pubsub: fix comments from PR #27493

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit cb256dd92ac39baddce252acb634b587653da4f3)

rgw/pubsub: clarify pubsub zone configuration

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit 89ddc407b48959831450638f14929f553511d240)

rgw/pubsub: wrong link in S3 doc

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 98e6a0434609d7b29d61166276419eca1d552ad0)

rgw/pubsub: fix documentation link errors

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dc904249a00aacecf9a06a26223b5b2d5c46ace2)

rgw/pubsub: fix comments from PR #27091

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit a0b43fa26c9ba1bebe6553c84fd98fad3a3d28ad)

rgw/pubsub: add more S3 compatibility documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e1e39d627ec9a044f9f7428400dd379350fc4bbb)

rgw/pubsub: test and doc bucket deletion impact on notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit cb6e894937098999e4eb7d2c13c2f180329f2432)

rgw/pubsub: avoid static creation of amqp manager

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 376a3df0d12b544a0378a1467fcf4b2c32fd8afa)

rgw/pubsub: add s3-compatible API documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 9fd0f873188532c7552f087f65c5b279eb1fc3ca)

rgw/pubsub: allow for endpoint definition via topics

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 8ac119fdd50b6306e4b338e3f0b4731818c6e6c8)

rgw/pubsub: implement S3 compatible get/delete APIs

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 0f8671b3a13af2713fed51f15c83cca8ef19dd19)

rgw: pubsub support s3 records. refactor ARN

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 2bd353233112ead32181b94a2af1b04e3fa9e1de)