qa: add script to stress udev_enumerate_scan_devices()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b7a0e2adcbd139dae8338e23d5752d42015fa0ad)

krbd: retry on an empty list from udev_enumerate_scan_devices()

systemd 219 doesn't have the issue that is worked around in the
previous commit, but has a different one: udev_enumerate_scan_devices()
always succeeds, but sometimes returns an empty list when the device is
actually there.  This happens rarely and at random so I haven't been
able to get to the bottom of it yet, but it looks like another similar
race condition in libudev.

Since an empty list is expected if the device isn't there, retry just
twice with a small sleep in-between.  This appears to be enough: I got
7 occurrences per 600000 "rbd unmap" invocations, all of which needed
a single retry:

  rbd: udev enumerate missed a device, tries = 1

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit bd37a72e0ec783a1ba91e63e8d96f0bc06007060)

Conflicts:
src/krbd.cc [ krbd_spec not in luminous ]

krbd: retry on transient errors from udev_enumerate_scan_devices()

udev_enumerate_scan_devices() doesn't handle disappearing devices well.
If called while some devices are being removed, it sometimes propagates
ENOENT and ENODEV errors encountered operating on directory entries in
/sys that no longer exist.  Some of these errors are suppressed, but
this isn't reliable and varies across versions.  In particular, systemd
239 suppresses ENODEV from sd_device_new_from_syspath() but doesn't
suppress ENODEV from sd_device_get_devnum().  In systemd 243 the call
to sd_device_get_devnum() has been moved, but it still leaks ENOENT
from sd_device_get_is_initialized() (referring to the body of
FOREACH_DIRENT_ALL loop in enumerator_scan_dir_and_add_devices()).

Assume that all ENOENT and ENODEV errors are transient and retry the
call to udev_enumerate_scan_devices().  Don't limit the number, but log
each retry.

Fixes: https://tracker.ceph.com/issues/41036
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit e5921ef4a89f497a0bff6510fce0bb5c242d6172)

qa: add script to test udev event reaping

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 898c113f93a05a18f47f2dd6a94f7cf56c858185)

krbd: increase udev netlink socket receive buffer to 2M

Even though with the previous commit we no longer block between binding
the socket and starting handling events, we still want a larger receive
buffer to accommodate for scheduling delays.  Since the filtering is
done in the listener, an estimate focused on just rbd is not accurate,
but anyway: a pair of "rbd" and "block" events for "rbd map" take 2048
bytes in the receive buffer.  This allows for roughly a thousand of
them ("rbd map" and "rbd unmap" require root and libudev makes use of
SO_RCVBUFFORCE so rmem_max limit is ignored).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1c6cac1acaefdf59c3265d70c8d2191c59f14652)

Conflicts:
src/krbd.cc [ trivial context ]

krbd: avoid udev netlink socket overrun

Because the event(s) we are interested in can be deliveled while we are
still in the kernel finishing map or unmap, we start listening for udev
events before going into the kernel.  However, if (un)mapping takes its
time, udev netlink socket can be fairly easily overrun -- the filtering
is done on the listener side, so we get to process everything, not just
rbd events.  If any of the events of interest get dropped (ENOBUFS), we
hang in poll().

Go into the kernel in a separate thread and leave the main thread to
run the event loop.  The return value is communicated to the reactor
though a pipe.

Fixes: https://tracker.ceph.com/issues/41404
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5444a1111523bc100bea60958b2671674f6208ac)

Conflicts:
src/krbd.cc [ krbd_spec, ceph_abort_msgf(), make_named_thread()
  not in luminous; luminous is compiled with -std=c++11 ]

krbd: reap all available events before polling again

This also exposes errors from udev_monitor_receive_device() which were
previously ignored.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 53aab34dafcca2ec022102a03905e59cfa34fc84)

krbd: separate event reaping from event processing

Move event processing into UdevMapHandler and UdevUnmapHandler
functors and replace wait_for_udev_{add,remove}() with a single
wait_for_mapping() template.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit c84f9e2f2df47361d7a928d0b25cb84ef332c055)

Conflicts:
src/krbd.cc [ krbd_spec not in luminous ]

krbd: get rid of poll() timeout

This timeout was added as a (very poor) workaround for an issue
addressed in commit 42dd1eae630f ("krbd: fix rbd map hang due to udev
return subsystem unordered").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ffb66ff7106b9d200a7da355199ab362fb611c31)

Conflicts:
src/krbd.cc [ ceph_abort_msgf() not in luminous ]

krbd: fix rbd map hang due to udev return subsystem unordered

The order of subsystem returned by udev_device_get_subsystem
might not be same order as adding subsystem by
udev_monitor_filter_add_match_subsystem_devtype. So if block
event is returned first and rbd event is returned next, then
further poll will get nothing back until timed-out.

Fixes: http://tracker.ceph.com/issues/39089
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 42dd1eae630f31a628051049e4007175561f1939)

Conflicts:
src/krbd.cc [ krbd_spec not in luminous ]

Merge pull request #31257 from jan--f/wip-41289-luminous

luminous: doc: update bluestore cache settings and clarify data fraction

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc: update bluestore cache settings and clarify data fraction

Fixes: http://tracker.ceph.com/issues/39522
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 9d8336a7f418fe2bb11361dd74a214403b1e5be7)

Merge pull request #30242 from smithfarm/wip-41489-luminous

luminous: cephfs: client: return -EIO when sync file which unsafe reqs have been dropped

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #31166 from alfredodeza/wip-rm42292

luminous: qa/ceph-disk: use a Python2.7 compatible version of pytest

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #31172 from dzafman/wip-missed-backports

Luminous corrections for tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28502 from smithfarm/wip-40166-luminous

luminous: cephfs: client: ceph.dir.rctime xattr value incorrectly prefixes 09 to the nanoseconds component

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #29097 from batrick/i40807

luminous: mds: msg weren't destroyed before handle_client_reconnect returned, if the reconnect msg was from non-existent session

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29226 from smithfarm/wip-40163-luminous

luminous: cephfs: mount: key parsing fail when doing a remount

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29229 from smithfarm/wip-40218-luminous

luminous: tests: cephfs: TestMisc.test_evict_client fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29829 from batrick/i39691

luminous: mds: check dir fragment to split dir if mkdir makes it oversized.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29830 from ukernel/luminous-41000

luminous: cephfs: client: unlink dentry for inode with llref=0

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28701 from batrick/i40286

luminous: qa: remove ubuntu 14.04 testing

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30934 from smithfarm/wip-42039-luminous

luminous: client: _readdir_cache_cb() may use the readdir_cache already clear

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/ceph-disk: use a Python2.7 compatible version of pytest

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30230 from dzafman/wip-network-luminous

luminous: core: Health warnings on long network ping times

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30785 from smithfarm/wip-41919-luminous

luminous: core: osd: scrub error on big objects; make bluestore refuse to start on big objects

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30902 from smithfarm/wip-42138-luminous

luminous: osd: Remove unused osdmap flags full, nearfull from output

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #31020 from smithfarm/wip-42393-luminous

luminous: common/ceph_context: avoid unnecessary wait during service thread shutdown

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31036 from smithfarm/wip-40502-luminous

luminous: core: osd: rollforward may need to mark pglog dirty

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31049 from vshankar/wip-mds-session-ref

luminous: mds: add reference when setting Connection::priv to existing session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #31091 from smithfarm/wip-42415-luminous

luminous: doc/rbd: s/guess/xml/ for codeblock lexer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: s/guess/xml/ for codeblock lexer

this change silences the warning of

```
doc/rbd/qemu-rbd.rst:174: WARNING: Pygments lexer name 'guess' is not
known
```

see http://pygments.org/docs/lexers/, we should use "xml" for XML .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit df226da996e468d2707b08eb012d54b4e37ffdc6)

Merge pull request #30425 from smithfarm/wip-41544-luminous

luminous: test/librbd: set nbd timeout due to newer kernels defaulting it on

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30147 from smithfarm/wip-41266-luminous

luminous: rgw: don't throw when accept errors are happening on frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30323 from smithfarm/wip-41713-luminous

luminous: rgw: fix refcount tags to match and update object's idtag

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mds: fix MDSRank::send_message_client_counted()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 28f35b5dd00c773936f97af9dacc0346c3214213)

mds: add reference when setting Connection::priv to existing session

the bug was introduced by commit 72883956c26 "use intrusive_ptr for
holding Connection::priv"

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 4478e353e35d1a6b217e075bbc1c551c1c009e29)

osd: accident of rollforward may need to mark pglog dirty

refers: https://github.com/ceph/ceph/pull/27015/files#r294114392

Fixes: http://tracker.ceph.com/issues/40403
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit 35cb184becd6562edd240553dfa50f47bb120b7f)

OSD: rollforward may need to mark pglog dirty

if we rollforward at the end of PG::activate(), we may advance the *crt*,
but we did not mart the log dirty, this means we will not update the crt
within the transaction of rollforward, so it is inconsistent.

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit 10d0990dc69310864b4845ee57b32610a642464f)

Conflicts:
src/osd/PGLog.h
- dfbe5e070cc978253abcb30b86de5faa7e6a1efc is not being backported
- retain !touched_log as part of conditional in is_dirty()

Merge pull request #30981 from tchaikov/wip-luminous-42361

luminous: build/ops: python3-cephfs should provide python36-cephfs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30118 from smithfarm/wip-41439-luminous

luminous: rbd-mirror: ignore errors relating to parsing the cluster config file

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/ceph_context: avoid unnecessary wait during service thread shutdown

Fixes: https://tracker.ceph.com/issues/42332
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e8249d3b28f8789b2d4aca6fb75c75103a5cbea1)

Conflicts:
src/common/ceph_context.cc
- luminous uses Mutex::Locker instead of std::unique_lock

Merge pull request #29875 from trociny/wip-41421-luminous

luminous: qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Fixes: https://tracker.ceph.com/issues/40923
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 0866a2456b773843537681c768d93d1c9e7718a4)

Conflicts:
qa/workunits/rbd/rbd_mirror_helpers.sh:
          - trivial resolution in test_status_in_pool_dir
          - adding missing XMLSTARLET definition (added in master by 917f8a0)

Merge pull request #30298 from smithfarm/wip-40082-luminous

luminous: osd: Better error message when OSD count is less than osd_pool_default_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30132 from smithfarm/wip-40735-luminous

luminous: doc/rgw: document use of 'realm pull' instead of 'period pull'

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30071 from smithfarm/wip-40138-luminous

luminous: doc: describe metadata_heap cleanup

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30690 from smithfarm/wip-42106-luminous-follow-on

luminous: build/ops: admin/build-doc: use python3 (follow-on fix)

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: provide python2-<modname>

to be consistent with other python2 packages, and their python3
counterparts

the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit fb6290b1fab0bd8a73043f2d68210c010e2fb425)

ceph.spec.in: use python_provide macro

our python3 bindings are now named `python3-<modname>` after python3 is
now maintained by RHEL/CentOS instead EPEL. to help the users using
`python36-<modname>`, we should "Provide" `python36-<modname>`.

the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel. and we do install
`python36-devel` in install-deps.sh, and install `python3-devel` in
ceph-*build/build/setup_rpm

see also
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_provides

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 50b19e673d8200306c3e36d1abaec414a3d336b9)

Merge pull request #30608 from vumrao/wip-vumrao-luminous-rados-tools-list-objects-in-a-pg

luminous: rados tools list objects in a pg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30905 from smithfarm/wip-42153-luminous

luminous: mon/OSDMonitor: trim not-longer-exist failure reporters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30922 from vumrao/wip-vumrao-42241

luminous: osd/PG: Add PG to large omap log message

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30926 from smithfarm/wip-42127-luminous

luminous: osd/OSDMap: do not trust partially simplified pg_upmap_item

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30855 from dillaman/wip-39673-luminous

luminous: qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #30856 from dillaman/wip-40499-luminous

luminous: rbd: use the ordered throttle for the export action

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #30857 from dillaman/wip-40880-luminous

luminous: rbd: Reduce log level for cls/journal and cls/rbd expected errors

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #30858 from dillaman/wip-41621-luminous

luminous: tools/rbd-ggate: close log before running postfork

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29202 from smithfarm/wip-40266-luminous

luminous: common: data race in OutputDataSocket

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28989 from smithfarm/wip-39537-luminous

luminous: osd/ReplicatedBackend.cc: 1321: FAILED assert(get_parent()->get_log().get_log().objects.count(soid) && (get_parent()->get_log().get_log().objects.find(soid)->second->op == pg_log_entry_t::LOST_REVERT) && (get_parent()->get_log().get_log().object

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29564 from k0ste/luminous_backports4

luminous: osd/bluestore: Actually wait until completion in write_sync

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30271 from smithfarm/wip-41733-luminous

luminous: osd: clear PG_STATE_CLEAN when repair object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30722 from smithfarm/wip-epel-release-snafu-luminous

luminous: build/ops: backport miscellaneous install-deps.sh and ceph.spec.in fixes from master

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/standalone/ceph-helpers: fix test_wait_for_clean

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 30fc7f5e972bd41537c92eb135c9763b46dd09d2)

Conflicts:
qa/standalone/ceph-helpers.sh
- mimic does not have separate WAIT_FOR_CLEAN_TIMEOUT

qa/standalone: make sure an osd is running before create_rbd_pool

'rbd pool init' now does IO.  Drop the pool, or change the pool size to 1.

Fixes: http://tracker.ceph.com/issues/38585
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cba0483b090a6370390e8ed88d3f9d7767d56be5)

qa: specify filestore for misc tests

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 8a694fc2f9e75864e064fe591feda9fab943c15e)

qa/standalone/ceph-helpers: resurrect all OSD before waiting for health

address the regression introduced by e62cfceb
in e62cfceb, we wanted to test the newly introduced TOO_FEW_OSDS
warning, so we increased the number of OSD to the size of pool, so if
the number of OSD is less than pool size, monitor will send a warning
message.

but we need to bring all OSDs back if we are expecting a healthy
cluster. in this change, all OSDs are resurrect before
`wait_for_health_ok`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdba0f14201a327e824b6ec48e014233281dee2d)

qa/standalone/ceph-helpers: fix health_ok test

Stopping the osd daemon won't reliably get you HEALTH_WARN or ERR; you have
to make sure it is also marked down.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dcdca44aa45e7dc5aab56da26caec644e32024a6)

mon/PGMap: enable/disable TOO_FEW_OSDS warning with an option

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3b74fbc93d32d6cc367049d3bcbe83d3837abba5)

Conflicts
        src/mon/PGMap.cc
- cct->_conf->get_val in luminous
src/common/options.cc
- no mon_max_snap_prune_per_epoch in luminous
- in luminous, osd_pool_default_size is INT_T (int64_t). In master, it is UINT_T
  (uint64_t). So I changed the type to int64_t.

qa/standalone: remove osd_pool_default_size in test_wait_for_health_ok

Signed-off-by: zjh <jhzeng93@foxmail.com>
(cherry picked from commit e62cfceb9549e806f774ef3da549efd9f4d90171)

osd: Better error message when OSD count is less than osd_pool_default_size

Fixes: http://tracker.ceph.com/issues/38617
Signed-off-by: zjh <jhzeng93@foxmail.com>
(cherry picked from commit 94237d36932b27961afffbe65d433af26c3fe513)

Conflicts:
doc/rados/operations/health-checks.rst
- trivial
        src/mon/PGMap.cc
- cct->_conf->get_val in luminous

client: _readdir_cache_cb() may use the readdir_cache already clear

Calling function _readdir_cache_cb() means to get dir information from the cache, but in the while loop,
the contents of readdir_cache are not static.

For example, _getattr() will trigger insert_trace to update readdir_cache, such as:
1. insert_trace()->clear_dir_complete_and_ordered() or
2. insert_trace()->add_update_inode()->add_update_cap()-> check_cap_issue()->clear_dir_complete_and_ordered()
So, there is a possibility that dn is empty after the _getattr() call.

Fixes: http://tracker.ceph.com/issues/41148
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit 404c06784e46016703104db59ea73f36d09b578b)

osd/OSDMap: do not trust partially simplified pg_upmap_item

If we simplified a partially no-op pg_upmap_item, we shall still
continue to verify that the remaining part is valid.
The bug is introduced by 02e5499b350bcd7d9eac98b2072052a9a4a1f535,
before which we always validate the correctness of a pg_upmap_item
before trying to cancel or simplify it.

Fixes: https://tracker.ceph.com/issues/42052
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4196b13283144de966eeba40e6765f10b254dac6)

Conflicts:
src/osd/OSDMap.cc

mon/OSDMonitor: trim no-longer-exist failure reporters

remove the report from failure_info in OSDMonitor::check_failure(), if
the reporter does not exist in the osdmap any longer. otherwise, we will
run into assert() failure when trying to lookup the osd in osdmap using
`osdmap.get_xinfo()`.

Fixes: https://tracker.ceph.com/issues/41680
Signed-off-by: NancySu05 <su_nan@inspur.com>
(cherry picked from commit 99ccc166ede5f296b86abfe9db0a3d597100edde)

Conflicts:
src/mon/OSDMonitor.cc
- ceph_assert, g_conf()
- account for C++17ism by moving iter initialization out of if statement

osd/PG: Add PG to large omap log message

Fixes: https://tracker.ceph.com/issues/41359
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit a71d51bfa154a278a350e6dbeb842fb3585b74f5)

osd: Remove unused osdmap flags full, nearfull from output

Fixes: https://tracker.ceph.com/issues/42015
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d9c645d7d8c7355c8aaf3ff9121f2b8fc3b1290a)

osd/OSDMap: misleading message in print_oneline_summary()
The output scared people as showing so many osds were full/nearfull

Improve the output to make the message clearer

Fixes: http://tracker.ceph.com/issues/22350
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>
(cherry picked from commit 7004b35f71a657a0d9479b7cc46acc210de303d4)

Merge pull request #29713 from cbodley/wip-41322

luminous: rgw: datalog/mdlog trim commands loop until done

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

tools/rbd-ggate: close log before running postfork

Otherwise the assert in Log:open() will trigger since
all data is kept during the fork and also the `is_started` flag

Fixes: https://tracker.ceph.com/issues/41592
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 21bdd4c6ecd24b2a7dbf8691705f228ac11481ad)

Conflicts:
src/tools/rbd_ggate/main.cc : Resolved in do_map

cls/journal: reduce verbosity of debug logs for non-errors

Fixes: http://tracker.ceph.com/issues/40865
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7779860d38725636fb5438e2f52e432994dc3926)

cls/rbd: reduce verbosity of debug logs for non-errors

Fixes: http://tracker.ceph.com/issues/40865
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7a6a2aefd72faf1da6594ca6bc5f8bf14a387edd)

rbd: use the ordered throttle for the export action

This allows exports to STDOUT to use multiple concurrent operations
and also fixes a potential race condition with concurrent callbacks
and file seeking.

Fixes: http://tracker.ceph.com/issues/40435
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 751bf6957020ad02b3a29b005b9792b9c6547f96)

Conflicts:
src/tools/rbd/action/Export.cc
- mimic has g_conf->get_val and int64_t "rbd_concurrent_management_ops"

qa/workunits/rbd: wait for rbd-nbd unmap to complete

The "unmap" request is asynchronous, so wait for a short amount
of time for the "rbd-nbd" daemon process to exit.

Fixes: http://tracker.ceph.com/issues/39598
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 87278548a5a476692511b76b661278501bf72429)

Conflicts:
qa/workunits/rbd/rbd-nbd.sh : Resolved for map/unmap namespace test
(cherry picked from commit b9e4644140ce9538a5162e466615d7e6b53efa7a)

rbd-nbd: output format support for list-mapped command

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit e372e9492bb7f535c0f15a078f1eab2de3fad113)

rbd-nbd: support optionally setting device timeout

Fixes: http://tracker.ceph.com/issues/22333
Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>
(cherry picked from commit bad51ffe7fbfe8c2e33eb0db2ef7acb89a03bd1d)

rbd-nbd: allow to unmap by image or snap spec

(to match the behaviour of krbd unmap)

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 02e211c2822db7d3fb3792ce18cdb217c5de24ee)

qa/workunits/rbd: bench-write is part of exclusive option test

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 47e38ca1b9f748e2f2ce1c035cd3f982d68eb492)

Merge pull request #29269 from smithfarm/wip-40135-luminous

luminous: rgw: the Multi-Object Delete operation of S3 API wrongly handles the Code response element

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30345 from dillaman/wip-41772-luminous

luminous: common/config: hold lock while accessing mutable container

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27974 from iain-buclaw-sociomantic/luminous-rgw-cls-bi-list-log-level

luminous: rgw: cls/rgw: raise debug level of bi_log_iterate_entries output

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27992 from smithfarm/wip-39016-luminous

luminous: rgw: unable to cancel reshard operations for buckets with tenants

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29153 from davebiffuk/s3_policy_tracker_39272

luminous: rgw: policy fix for nonexistent objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29170 from cbodley/wip-qa-rgw-tempest-luminous

luminous: qa/rgw: remove incomplete rgw/tempest suite from luminous

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #29271 from smithfarm/wip-40143-luminous

luminous: rgw: multisite: 'radosgw-admin bucket sync status' should call syncs_from(source.name) instead of id

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29295 from dillaman/wip-40233-luminous

luminous: rbd/action: fix error getting positional argument

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29826 from linuxbox2/wip-luminous-apiname

luminous: rgw: fix issue for CreateBucket with BucketLocation param

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29874 from trociny/wip-41285-luminous

luminous: rbd-mirror: don't overwrite status error returned by replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30375 from smithfarm/wip-41480-luminous

luminous: rgw: make dns hostnames matching case insensitive

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30666 from smithfarm/wip-41104-luminous

luminous: rgw: mitigate bucket list with max-entries excessively high

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #30677 from sobelek/wip-42049-luminous

luminous: ceph-volume: fix warnings raised by pytest