krbd: avoid udev netlink socket overrun

Because the event(s) we are interested in can be deliveled while we are
still in the kernel finishing map or unmap, we start listening for udev
events before going into the kernel.  However, if (un)mapping takes its
time, udev netlink socket can be fairly easily overrun -- the filtering
is done on the listener side, so we get to process everything, not just
rbd events.  If any of the events of interest get dropped (ENOBUFS), we
hang in poll().

Go into the kernel in a separate thread and leave the main thread to
run the event loop.  The return value is communicated to the reactor
though a pipe.

Fixes: https://tracker.ceph.com/issues/41404
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5444a1111523bc100bea60958b2671674f6208ac)

krbd: reap all available events before polling again

This also exposes errors from udev_monitor_receive_device() which were
previously ignored.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 53aab34dafcca2ec022102a03905e59cfa34fc84)

krbd: separate event reaping from event processing

Move event processing into UdevMapHandler and UdevUnmapHandler
functors and replace wait_for_udev_{add,remove}() with a single
wait_for_mapping() template.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit c84f9e2f2df47361d7a928d0b25cb84ef332c055)

krbd: get rid of poll() timeout

This timeout was added as a (very poor) workaround for an issue
addressed in commit 42dd1eae630f ("krbd: fix rbd map hang due to udev
return subsystem unordered").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ffb66ff7106b9d200a7da355199ab362fb611c31)

common/thread: Fix race condition in make_named_thread

The thread may well no longer exist by the time we try to set the
name, so have the thread set its own name first thing.

Thanks to Ilya Dryomov <idryomov@gmail.com> for pointing it out.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 2bd106ec0da52e7fcf616d7b3cb20d570c1a5c50)

Merge pull request #31074 from smithfarm/wip-42417-nautilus

nautilus: doc/rbd: s/guess/xml/ for codeblock lexer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: s/guess/xml/ for codeblock lexer

this change silences the warning of

```
doc/rbd/qemu-rbd.rst:174: WARNING: Pygments lexer name 'guess' is not
known
```

see http://pygments.org/docs/lexers/, we should use "xml" for XML .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit df226da996e468d2707b08eb012d54b4e37ffdc6)

Merge pull request #31009 from SUSE/wip-doc-telemetry-default-interval-nautilus

nautilus: doc/mgr/telemetry: update default interval

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/mgr/telemetry: update default interval

Commit 712987d533 changed the default interval to 24h;
updating the docs to match (this also should go to
the Nautilus branch as that commit landed there too
in https://github.com/ceph/ceph/pull/27709)

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 3c8793c4bd077db2796b512a3b59cd6a01cb3a8c)

Merge pull request #30676 from sobelek/wip-42050-nautilus

nautilus: ceph-volume: fix warnings raised by pytest

Merge pull request #29965 from smithfarm/wip-41272-nautilus

nautilus: rgw: rgw-log issues the wrong message when decompression fails

Merge pull request #29969 from smithfarm/wip-41446-nautilus

nautilus: rgw_file: readdir: do not construct markers w/leading '/'

Merge pull request #29970 from smithfarm/wip-41459-nautilus

nautilus: rgw: Put User Policy is sensitive to whitespace

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29972 from smithfarm/wip-41482-nautilus

nautilus: rgw: potential realm watch lost

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29974 from smithfarm/wip-41493-nautilus

nautilus: rgw: multisite: radosgw-admin bucket sync status incorrectly reports "caught up" during full sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29963 from smithfarm/wip-41267-nautilus

nautilus: rgw: beast frontend throws an exception when running out of FDs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29971 from smithfarm/wip-41479-nautilus

nautilus: rgw: dns name is not case sensitive

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30037 from smithfarm/wip-41588-nautilus

nautilus: rgw: fix a bug that lifecycle expiraton generates delete marker continuously

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30068 from smithfarm/wip-41485-nautilus

nautilus: rgw: fix list bucket with delimiter wrongly skip some special keys

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30160 from pdvian/wip-41624-nautilus

nautilus: rgw/rgw_op: Remove get_val from hotpath via legacy options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30748 from smithfarm/wip-42194

nautilus: tools: ceph-backport.sh: add deprecation warning

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29591 from pdvian/wip-40894-nautilus

nautilus: mds: cleanup truncating inodes when standby replay mds trim log segments

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29878 from pdvian/wip-41096-nautilus

nautilus: cephfs: avoid map been inserted by mistake

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29879 from pdvian/wip-41099-nautilus

nautilus: cephfs: fix a memory leak

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29938 from pdvian/wip-41107-nautilus

nautilus: mds: delay exporting directory whose pin value exceeds max rank id

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29983 from pdvian/wip-41128-nautilus

nautilus: qa: power off still resulted in client sending session close

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30026 from smithfarm/wip-40895-nautilus

nautilus: pybind/cephfs: Modification to error message

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30030 from smithfarm/wip-40887-nautilus

nautilus: ceph_volume_client: convert string to bytes object

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30031 from smithfarm/wip-40900-nautilus

nautilus: mds: evict an unresponsive client only when another client wants its caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30032 from smithfarm/wip-41113-nautilus

nautilus: cephfs: client: more precise CEPH_CLIENT_CAPS_PENDING_CAPSNAP

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30038 from smithfarm/wip-41276-nautilus

nautilus: tests: cephfs: fix malformed qa suite config

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30039 from smithfarm/wip-41465-nautilus

nautilus: cephfs: mount.ceph: properly handle -o strictatime

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30040 from smithfarm/wip-41467-nautilus

nautilus: mds: trim cache on regular schedule

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30041 from smithfarm/wip-41477-nautilus

nautilus: mds: fix InoTable::force_consume_to()

Merge pull request #30057 from varshar16/wip-nautilus-cephfs-shell-path-conversion

nautilus: cephfs-shell: Convert paths type from string to bytes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30418 from pdvian/wip-41851-nautilus

nautilus: mds: make MDSIOContextBase delete itself when shutting down

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30442 from pdvian/wip-41855-nautilus

nautilus: client: nfs-ganesha with cephfs client, removing dir reports not empty

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30455 from smithfarm/wip-41889-nautilus

nautilus: mgr/volumes: handle exceptions in purge thread with retry

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #30508 from ukernel/nautilus-41948

nautilus: mds: wake up lock waiters after forcibly changing lock state

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: add deprecation warning

The script is not maintained in the stable versions of Ceph.

This commit is not a cherry-pick from master, because the script is maintained
in master. It is unmaintained only in the stable versions, like "nautilus".

Also add a --version option which, in this case, will also indicate to the
user that this is an outdated, deprecated version of the script.

Fixes: https://tracker.ceph.com/issues/42194
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30325 from theanalyst/wip-41498-nautilus

nautilus: rgw: fix the bug of rgw not doing necessary checking to website configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30437 from smithfarm/wip-41846-nautilus

nautilus: rgw: increase beast parse buffer size to 64k

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30604 from vumrao/wip-vumrao-41976

nautilus: radosgw-admin: add --uid check in bucket list command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30651 from pdvian/wip-41970-nautilus

nautilus: rgw: ldap auth: S3 auth failure should return InvalidAccessKeyId

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30680 from pdvian/wip-41974-nautilus

nautilus: rgw: add minssing admin property when sync user info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30247 from pdvian/wip-41631-nautilus

nautilus: rgw: fixed "unrecognized arg" error when using "radosgw-admin zone rm".

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30248 from pdvian/wip-41627-nautilus

nautilus: rgw: RGWCoroutine::call(nullptr) sets retcode=0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30252 from smithfarm/wip-41707-nautilus

nautilus: rgw: fix cls_bucket_list_unordered() partial results

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30509 from pdvian/wip-41898-nautilus

nautilus: rgw: fix data sync start delay if remote haven't init data_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30472 from pdvian/wip-41858-nautilus

nautilus: rgw: fix memory growth while deleteing objects with

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29716 from xiexingguo/wip-build-push-segv-for-n

nautilus: core: osd/ReplicatedBackend: check against empty data_included before enabling crc

Merge pull request #29928 from smithfarm/wip-41534-nautilus

nautilus: tests: valgrind: UninitCondition in ceph::crypto::onwire::AES128GCM_OnWireRxHandler::authenticated_decrypt_update_final()

Merge pull request #29946 from smithfarm/wip-39412-nautilus

nautilus: core: pool_stat.dump() - value of num_store_stats is wrong

Merge pull request #30278 from pdvian/wip-41703-nautilus

nautilus: osd/PrimaryLogPG: update oi.size on write op implicitly truncating ob…

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30605 from vumrao/wip-vumrao-41963

nautilus: tools/rados: call pool_lookup() after rados is connected

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30607 from vumrao/wip-vumrao-41960

nautilus: tools/rados: add --pgid in help

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30708 from smithfarm/wip-epel-release-snafu-nautilus

nautilus: build/ops: install-deps: do not install if rpm already installed and ceph.spec.in: s/pkgversion/version_nodots/

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

ceph.spec.in: fix Cython package dependency for Fedora

Fedora distros do not have python3?-Cython packages, but they do have
python3-Cython ones. Fix the BuildRequires so that we only use the
python3_version_nodots based version string for RHEL.

Fixes: https://tracker.ceph.com/issues/42032
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 40c5a18f81c67c96f6a35e592ac3141b87c19701)

ceph.spec.in: s/pkgversion/version_nodots/

`python3_pkgversion` is now defined as 3, while we don't have packages
like python3-Cython yet in EPEL7. but we do have `python36-Cython`. so
let's use `python3_version_nodots` instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6d2f66846d1b84595cfdf8ac13a7ed4e1ae608ce)

install-deps: do not install if rpm already installed

there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 45a4173bcfe15a9bf56f6aeb6e6c35386a00d1c5)

Merge pull request #30579 from yuvalif/wip-yuval-backport-notif-nautilus

nautilus: rgw/pubsub: backport notifications and pubsub

Merge pull request #30601 from smithfarm/wip-42070-nautilus

nautilus: build/ops: install-deps.sh: add EPEL repo for non-x86_64 archs as well

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge branch 'nautilus' into wip-yuval-backport-notif-nautilus

Merge pull request #30686 from smithfarm/wip-40131-nautilus-follow-on

nautilus: doc: fix urls in posix.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Fixes: https://tracker.ceph.com/issues/42023
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 49d740118f9a0ee6b6bb82954630c654946f1751)

Merge pull request #30322 from theanalyst/wip-41700-nautilus

nautilus: build/ops: install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30245 from smithfarm/wip-41711-nautilus

nautilus: doc/ceph-kvstore-tool: add description for 'stats' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix urls in posix.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit c5c5f031f77f778af73f8659b01695b88f158807)

Merge pull request #30664 from smithfarm/wip-42105-nautilus

nautilus: build/ops: admin/build-doc: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30004 from smithfarm/wip-41568-nautilus

nautilus: doc: pg_num should always be a power of two

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30003 from smithfarm/wip-41529-nautilus

nautilus: doc: default values for mon_health_to_clog_* were flipped

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: add minssing admin property when sync user info.

Fixes: https://tracker.ceph.com/issues/41643
Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit 9ad975d422aba67c0bf7bc1b114079331e2f7fd5)

ceph-volume: avoid backported version of configparser

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 88a8d5d967c9060e525e91f48b623beebee00c6d)

ceph-volume: fix warnings raised by pytest

Warnings are related to configparser. Update tests too.

Fixes: https://tracker.ceph.com/issues/41907
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 31a0c7e77bf5d0193e90e6693b424ca5305b47cc)

Merge pull request #29731 from pdvian/wip-41081-nautilus

nautilus: doc: Fix rbd namespace documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

admin/build-doc: use python3

to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ace8cb15a600daf5261c0ab349b43010a238a1d9)

Merge PR #30025 into nautilus

* refs/pull/30025/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: ldap auth: S3 auth failure should return InvalidAccessKeyId

LDAP was out of sync with the other auth modules, and the default
error code is not correct for S3.

Fixes: https://tracker.ceph.com/issues/41760
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 226227cf2d825b3414b7522a9f738180142f3956)

Merge pull request #30649 from lordcirth/wip-doc-misplaced-ratio

doc: max_misplaced option was renamed in Nautilus

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: max_misplaced was renamed in Nautilus

mgr/balancer/max_misplaced was changed to target_max_misplaced_ratio
in 14.2.0.

Signed-off-by: Nathan Fish <lordcirth@gmail.com>

Merge pull request #30554 from alfredodeza/nautilus-guits-41392

nautilus ceph-volume: do not fail when trying to remove crypt mapper

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@redhat.com>

tools/rados: add --pgid in help

This commit 825c7b9729b2410a116e123d4b21210c646704d4 brought
the support for --pgid in rados ls but was not in help.
Fixes: https://tracker.ceph.com/issues/41876
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit ae9a24eaba330fd3f6946063c2bc926f58c93772)

tools/rados: call pool_lookup() after rados is connected
This commit fixes a segmentation fault when using --pgid
option in rados ls command in combination with --pool/-p option.
The reason for the crash was that we can not use the rados object
before connecting it with the cluster using rados.init_with_context().

Fixes: https://tracker.ceph.com/issues/41875
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit 56b6dd9bd7a50371062128122e6a899f2c79c5f5)

radosgw-admin: add --uid check in bucket list command
Fixes: https://tracker.ceph.com/issues/41589
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit c53597b561837340314f0a169f6881c1814ce0b9)

 Conflicts:
src/rgw/rgw_admin.cc
        - no --new-uid

Merge pull request #30049 from smithfarm/wip-41333-nautilus

nautilus: rpm: always build ceph-test package

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

build-doc: allow building docs on fedora 30

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e17e4bfcc048abd1757c3efd12c7443169d67477)

rgw/pubsub: backporting pubsub/notifications to nautilus

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/pubsub: add notification filtering

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 543f8f7a2729ab8bd2e1a14d98a3fd2c2e19f9bd)

rgw/pubsub: support deletion markers and multipart upload

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 923799fd960f97b8ddbbe0e106139069d640588b)

rgw/pubsub: push notifications documentation

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit c345566831c72446c8250f630337825e526847a8)

rgw/pubsub: push notifications from ops

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit dc31b030ea30f3b7bfdeb9fa92e303892d17138a)

rgw/pubsub: allow pubsub REST API on master

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit a14353496d1f75a303495f538141a1a54ff28c60)

rgw/pubsub: service reordering issue

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit e91489f443b5b086ef96b51051d2726728412ff5)

rgw/pubsub: fix amqp topic bug. add disabled end2end push tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit abfd6f72db3b6e1a797a660a4a601a33605e5f25)

rgw/pubsub: fix duplicates due to multiple zone synching

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 38c23833b35279cb6e1a9b575b5a463959fed348)

rgw/pubsub: run pubsub tests even if multisite fails

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 1df809144c2ed96a715c8bd3df6ed5ef48399dd3)

rgw/pubsub: make new PSZone parameters optional in test

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 0b6909c1e9832d7a6f375f38f32bed58d1e161c2)

rgw/pubsub: add conf parameter for full/incremental sync

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit af4c2d5dbb42170ade4ad78088b80b0b5a357c5a)

rgw: add tenant as parameter to User in multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 4def929395bce39e9a130d0e8740790815a6a503)

rgw/pubsub: revert the RGWSysObjectCtx change

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 10b8b57d723971604cbabf984291232c194a55ec)