tools/osdmaptool.cc: do not use deprecated std::random_shuffle()
the use of `std::random_shuffle()` was introduced by
b946308 .

in this change, it is replaced using `std::shuffle()`.

Fixes: https://tracker.ceph.com/issues/43084
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 30dfef208d859a67c1f1b13c314735861c536e85)

Conflicts:
src/tools/osdmaptool.cc (randomizing pools here instead of rules)

test: Add test case based on Xie script in commit comment

Other test fixes to reflect changes

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9d2e0267e1f025a634ee4925c919008cacdd3d38)

Conflict:
src/test/cli/osdmaptool/upmap-out.t (older output format)
src/test/cli/osdmaptool/upmap.t (older output format)

osd: ceph_pg_upmaps() use any overfull when there are still underfull

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f38d38398f33f64bd49f017b1c8e737452157c53)

osd: Create more_underfull with below target that aren't in underfull

Use more_underfull for finding remaps for overfull OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit fe03383e6691f6bd95189d3ba4d91fbb2ea9e9c8)

osd: calc_pg_upmaps() pick most overfull remap from try_pg_upmap()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 3f132061fd33c82dc131e18ae80cbec067b53cae)

osd: ceph_pg_upmaps() use max_deviation to determine perfect distribution

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7f4ae9312314c329b63c850e09b403a95ad18bed)

Conflicts:
src/pybind/mgr/balancer/module.py (upmap_max_deviation config option
   handled differently)

tools: osdmaptool: Perform upmap calculation as ceph-mgr does

This is the backportable commit that works with older balancer module.py

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b946308f03f1798915fafe7878f34b4a234c2ae4)

osd: For balancer crush needs the rule passed to get_parent_of_type()

Fixes: https://tracker.ceph.com/issues/42718
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 3eb69fd0cb15974e33e675babdd5d232d69c3c6d)

osdmaptool: Match default max value of 10

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 91bab14ea0e2ec7c94eb758cd76f9a620688725f)

tools: odsmaptool truncate target upmap file

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a49324ceb9c322f792ea00720f438ea9ea09555f)

mgr: Improve balancer module status

Add balancer status fields so that slow optimizations can be detected

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f04c505b9dc128bdfbcca49d82f8aac4abf8b56f)

test: Test balancer module commands

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 3a0e2c8ff1cce44bf44b139572f4d1bd9bbfb766)

mgr: Release GIL before calling OSDMap::calc_pg_upmaps()

Prevent optimize and execute commands from running with active balancer

Fixes: https://tracker.ceph.com/issues/42432
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e2a35e8c8e9f381a635ad0ca01326a2c512590a9)

Conflicts:
src/pybind/mgr/balancer/module.py (trivial)
src/test/cli-integration/balancer/misplaced.t (doesn't exist)

Merge pull request #31664 from hillpd/wip-38714-luminous

luminous: rgw: resolve bugs and clean up garbage collection code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31655 from smithfarm/wip-40496-luminous

luminous: doc: fixed --read-only argument value in multisite doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31647 from smithfarm/wip-39603-luminous

luminous: doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31696 from k0ste/wip-42849-luminous

luminous: pybind/mgr: Cancel output color control

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31587 from k0ste/wip-42784-luminous

luminous: mgr/prometheus: assign a value to osd_dev_node when obj_store is not filestore or bluestore

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #31360 from idryomov/wip-krbd-udev-fixes-luminous

luminous: krbd: avoid udev netlink socket overrun and retry on transient errors from udev_enumerate_scan_devices()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #31654 from smithfarm/wip-40350-luminous

luminous: rgw/OutputDataSocket: actually discard data on full buffer

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31653 from smithfarm/wip-40359-luminous

luminous: rgw: set null version object acl issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31652 from smithfarm/wip-40132-luminous

luminous: rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31651 from smithfarm/wip-40127-luminous

luminous: rgw: parse_copy_location defers url-decode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31649 from smithfarm/wip-39747-luminous

luminous: rgw: delete_obj_index() takes mtime for bilog

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31648 from smithfarm/wip-39733-luminous

luminous: rgw: cls/user: cls_user_set_buckets_info overwrites creation_time

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31220 from less-is-morr/wip-39732-luminous

luminous: rgw: allow radosgw-admin to list bucket w --allow-unordered

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31218 from smithfarm/wip-39696-luminous

luminous: rgw: return ERR_NO_SUCH_BUCKET early while evaluating bucket policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30946 from smithfarm/wip-40852-luminous

luminous: rgw/multisite: Don't allow certain radosgw-admin commands to run on non-master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30984 from k0ste/luminous_backports

luminous: rgw: add list user admin OP API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31598 from smithfarm/wip-42796-luminous

luminous: osd/OSDMap.cc: don't output over/underfull messages to lderr

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29122 from taodd/barckport-26564

luminous: rgw: LC: handle resharded buckets

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #31674 from smithfarm/wip-42834-luminous

luminous: os/bluestore: fix improper setting of STATE_KV_SUBMITTED.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #31449 from batrick/i41031-luminous

luminous: qa: malformed job

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #31450 from batrick/i42672

luminous: qa: cfuse_workunit_kernel_untar_build fails on Ubuntu 18.04

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #31573 from ukernel/luminous-42774

luminous: mds: add command that config individual client session

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

rgw: resolve bugs and clean up garbage collection code

Does a number of things to clean up rgw gc code:

  * adds additional logging to make future debugging easier.
  * resolves bug where the truncated flag was not always set correctly
    in gc_iterate_entries
  * resolves bug where marker in RGWGC::process was not advanced
  * resolves bug in which gc entries with a zero-length chain
    were not trimmed
  * resolves bug where same gc entry tag was added to list for
    deletion multiple times

Fixes: http://tracker.ceph.com/issues/38454
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 73d7d36)

Conflicts:
src/rgw/rgw_gc.cc dout() vs ldpp_dout()

Note: This was a clean cherry-pick from Mimic, which already resolved
      dout() vs ldpp_dout() conficts.

Signed-off-by: Dan Hill <daniel.hill@canonical.com>
(cherry picked from commit a598ccce2e)

pybind/mgr: Cancel output color control

When I do "ceph osd status" command, the units are not visible
when using a black background, So cancel outpul color control

Fixes: https://tracker.ceph.com/issues/42517
Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>
(cherry picked from commit ca2675e215acc8fbfb2d730b9c1f9f3aafc3e67d)

Conflicts:
- path: rc/pybind/mgr/mgr_util.py
  comment: mgr_util.py in luminous is status/module.py

os/bluestore: fix improper setting of STATE_KV_SUBMITTED.

Fixes: https://tracker.ceph.com/issues/42209
The issue is Nautilus and earlier releases specific as master already has
some changes making the case even worse and then fixing the whole bunch.
See https://tracker.ceph.com/issues/42189

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f8ed28a812bd068d07d4fe5013a731100038bf17)

Conflicts:
src/os/bluestore/BlueStore.cc
- luminous assert became ceph_assert later

doc: fixed --read-only argument value in multisite doc

Fixes: https://tracker.ceph.com/issues/40458
Signed-off-by: Chenjiong Deng <dengchenjiong@umcloud.com>
(cherry picked from commit c65832c1ff03e802f98cb7f4bda620e1e515264f)

rgw/OutputDataSocket: actually discard data on full buffer

A dout message in OutputDataSocket::append_output() states that
data will be dropped when appending would cause data_max_backlog
to be exceeded--but the method appends it anyway.

Log output discards at level 0, as messages will be lost.  Suppress
repeated warnings mod 100.  Switch to vector.

Fixes: http://tracker.ceph.com/issues/40178
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c806b825dae649829de8847d36cb21ffd2bbee8e)

Conflicts:
src/common/OutputDataSocket.cc
src/common/OutputDataSocket.h

rgw: set null version object acl issues

1.set null version object acl will create empty index
RGWRados::set_attrs did not clear instance, so index prepare, complete got instance=null,
which lead to empty index 1000_<obj>_i_null.
there is no harm to create empty index, but listomapkeys to find that key.

2.if object is exist with versioned key, we can set none exists null version object
order:
1) enable bucket version
2) put obj
3) disable bucket version
4) set versoned_id=null acl will succeed which should not

Fixes: http://tracker.ceph.com/issues/36763
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit ca6e5c89e8d6da498da15f0eea4bcd4d271b44e9)

rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Fixes: http://tracker.ceph.com/issues/20797
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 40e602bc3866598952eb0dd68ecec947dd7b70d6)

rgw: parse_copy_location defers url-decode

don't url-decode until after we search for the ?, or we'll truncate
object names that contain a url-encoded ?

Fixes: http://tracker.ceph.com/issues/27217
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 804bb853f1c7b5ce546c100d61c6fc526838c134)

rgw: delete_obj_index() takes mtime for bilog

writing an empty timestamp to the bilog prevents other zones from
applying the delete. this means that the --bypass-gc flag for
'radosgw-admin bucket rm' doesn't work in multisite

Fixes: http://tracker.ceph.com/issues/24991
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0316a81875ae93a06841d532bf3548c81370296e)

test/rgw: add multisite test_bucket_creation_time()

does not create any buckets itself, but checks that all creation times
match on buckets created by other tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8df991c5cc48a6709185edf506d8bdec4d28e371)

cls/user: cls_user_set_buckets_info overwrites creation_time

metadata sync of a new bucket entrypoint may call rgw_link_bucket()
(which in turn calls into cls user) without deleting/unlinking the
previous bucket entrypoint. this prevented the new bucket entrypoint
from overwriting the creation_time of the old one

Fixes: http://tracker.ceph.com/issues/39635
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c469e5150a3e95b547a7d05d1ae08fa88fad7159)

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Fixes: http://tracker.ceph.com/issues/39597
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e97668bb7a1333850a5791ae16b97496851ac854)

Added single check to avoid duplication. Included few more commands.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 14c3b4b3ebb6ef2fc44d24f8020beb2977c8e46e)

Conflicts:
        src/rgw/rgw_admin.cc
- cherry-pick was clean, but there was a build failure "error: 'class RGWRados'
has no member named 'svc'", which was fixed by making the following change:

-  bool non_master_cmd = (!store->svc.zone->is_meta_master() && !yes_i_really_mean_it);
+  bool non_master_cmd = (!store->is_meta_master() && !yes_i_really_mean_it);

- drop OPT_MFA_CREATE, OPT_MFA_REMOVE, and OPT_MFA_RESYNC which were added
  post-luminous (by 3ab463f9f538a08fd11db2b654004449994dd0fe)

rgw: Fail radosgw-admin commands on non-master zone that modify metadata
but with an option to override, allowing changes only on the local zone.

Fixes: http://tracker.ceph.com/issues/39548
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 79d884eb018cd2fea9ee9e763d0339b3e8626e6d)

rgw_admin.cc: do not compare int with boolean

Fix for:

[src/rgw/rgw_admin.cc:5862]: (warning) Comparison of a boolean
 expression with an integer.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit d5419143ac5ab0ad8d9e31bd251978b4d9355e46)

osd/OSDMap.cc: don't output over/underfull messages to lderr

There can be cases where overfull and underfull(see example in
https://tracker.ceph.com/issues/42756) will be empty, which is not
necessarily an error. These error messages can end up spamming
the ceph-mgr log.

Fixes: https://tracker.ceph.com/issues/42756
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 3cf28f09a2ebdd922fc5c1d29c65469f8331e741)

mgr/prometheus: initializing osd_dev_node = None

Signed-off-by: jiahuizeng <jhzeng93@foxmail.com>
(cherry picked from commit d7e84a0be1d6113d527a3b525647041775f643e2)

mgr/prometheus: assign a value to osd_dev_node when obj_store is not filestore or bluestore

Fixes: https://tracker.ceph.com/issues/42017
Signed-off-by: jiahuizeng <jhzeng93@foxmail.com>
(cherry picked from commit f5b51a35930bda5499531c75f0c80816afe4ab24)

qa/cephfs: test case for timeout config of individual session

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 4b28a4a7569ab516d75752c97226f598e6229ca2)

 Conflicts:
qa/tasks/cephfs/test_client_recovery.py

mds: add command that config individual client session

For now, the command only can change session's timeout config

Fixes: http://tracker.ceph.com/issues/40811
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit a9c21bdaef3473d99ac362122d441b9d869230dd)

 Conflicts:
src/mds/MDSDaemon.cc
src/mds/MDSRank.h
src/mds/mdstypes.h

mds: allow client to specify its session timeout

This is for ganesha client (it is expected to be reclaimed if it dies)
The default session timeout may be too short to reliably get ganesha
back up.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit eb4c36bf23f32cf9117d62660a4919dcfe80baa5)

 Conflicts:
src/client/Client.cc
src/client/Client.h
src/include/cephfs/libcephfs.h
src/libcephfs.cc
src/mds/Server.cc

Merge pull request #31433 from smithfarm/wip-42663-luminous

luminous: tests: qa/tasks/ceph.py: pass cluster_name to get_mons

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/ceph: keep mon addrs in ctx namespace

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 545df766bef04b1a70380ac04909b2a88521e4da)

qa/tasks/mon_seesaw: make get_mon_status use mon addr

We don't have the 'mon addr' config property any more.

This commit cannot be cherry-picked from master because qa/tasks/mon_seesaw.py
was dropped in nautilus.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/ceph_manager: make get_mon_status use mon addr

We don't have the 'mon addr' config property any more.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ac2430a43ddec469575a25be4aff75ce1628eee2)

qa/tasks/ceph.py/create_simple_monmap: use split_role

This cannot be cherry-picked from master because it fixes an issue that was
introduced into luminous by a bad backport.

Fixes: c543810cd426819538b07d2f60ebd8cdd76b9b93
Fixes: https://tracker.ceph.com/issues/42658
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/ceph.py: pass cluster_name to get_mons

This cannot be cherry-picked from master because it fixes an issue that was
introduced into luminous by a bad backport.

Fixes: c543810cd426819538b07d2f60ebd8cdd76b9b93
Fixes: https://tracker.ceph.com/issues/42658
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31286 into luminous

* refs/pull/31286/head:
mds: cleanup truncating inodes when standby replay mds trim log segments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use recent kernel to kernel build testing

Fixes: http://tracker.ceph.com/issues/24679
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit f0fe0936e64dde2be7139340489960b9635316c2)

Conflicts:
qa/cephfs/begin.yaml : Resolved for extra_system_packages
(cherry picked from commit 1cf09acdca619407c8d3438049138a42fb845d70)

qa: use cephfs standard configs for kclient

Specifically, qa/cephfs/begin.yaml.

Fixes: http://tracker.ceph.com/issues/26967
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 862b84c1e8f496be71666c35c11c665c5c259a43)
(cherry picked from commit 8e483d9bb27a193c85b233b6da4231b30b63e0f4)

qa: fix malformed suite config

Fixes: https://tracker.ceph.com/issues/41031
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 31492bb09502082868c95762f6eff46df12913a7)

Merge pull request #31234 from smithfarm/wip-42548-luminous

luminous: crush: remove invalid upmap items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30329 from smithfarm/wip-41626-luminous

luminous: rgw: RGWCoroutine::call(nullptr) sets retcode=0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30674 from smithfarm/wip-41382-luminous

luminous: rgw: housekeeping of reset stats operation in radosgw-admin and cls back-end

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30254 from smithfarm/wip-41706-luminous

luminous: rgw: fix cls_bucket_list_unordered() partial results

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30198 from taodd/backport-41111

luminous: rgw: fix drain handles error when deleting bucket with bypass-gc option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31277 from dzafman/wip-network-fix-luminous

luminous: core: osd: Fix for compatibility of encode/decode of osd_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30842 from smithfarm/wip-42264-luminous

luminous: tests: do not take ceph.conf.template from ceph/teuthology.git

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #30945 from smithfarm/wip-40318-luminous

luminous: tests: qa/workunits/rados/test_librados_build.sh: install build deps

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #31272 from badone/wip-luminous-restful-node-items

luminous: restful: Query nodes_by_id for items

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #31278 from tchaikov/wip-luminous/42580

luminous: tests: install python3-cephfs for fs suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #31293 from smithfarm/wip-42586-luminous

luminous: core: osd: fix out of order caused by letting old msg from down osd be processed

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30446 from tchaikov/wip-luminous-37866

luminous: mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30519 from smithfarm/wip-41864-luminous

luminous: mon: ensure prepare_failure() marks no_reply on op

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30896 from smithfarm/wip-42037-luminous

luminous: osd/PeeringState: recover_got - add special handler for empty log

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30917 from smithfarm/wip-41458-luminous

luminous: osd: merge replica log on primary need according to replica log's crt

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31030 from smithfarm/wip-42199-luminous

luminous: osd/PrimaryLogPG: skip obcs that don't exist during backfill scan_range

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31104 from k0ste/luminous_backports2

luminous: mgr/balancer: python3 compatibility issue

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31105 from k0ste/luminous_backports3

luminous: mgr/prometheus: replace whitespaces in metrics' names

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31107 from k0ste/luminous_backports4

luminous: mgr/prometheus: Cast collect_timeout (scrape_interval) to float

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31308 from dzafman/wip-smoke-21376-luminous

luminous: Various whitelists for tests to pass

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: add script to stress udev_enumerate_scan_devices()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit b7a0e2adcbd139dae8338e23d5752d42015fa0ad)

krbd: retry on an empty list from udev_enumerate_scan_devices()

systemd 219 doesn't have the issue that is worked around in the
previous commit, but has a different one: udev_enumerate_scan_devices()
always succeeds, but sometimes returns an empty list when the device is
actually there.  This happens rarely and at random so I haven't been
able to get to the bottom of it yet, but it looks like another similar
race condition in libudev.

Since an empty list is expected if the device isn't there, retry just
twice with a small sleep in-between.  This appears to be enough: I got
7 occurrences per 600000 "rbd unmap" invocations, all of which needed
a single retry:

  rbd: udev enumerate missed a device, tries = 1

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit bd37a72e0ec783a1ba91e63e8d96f0bc06007060)

Conflicts:
src/krbd.cc [ krbd_spec not in luminous ]

krbd: retry on transient errors from udev_enumerate_scan_devices()

udev_enumerate_scan_devices() doesn't handle disappearing devices well.
If called while some devices are being removed, it sometimes propagates
ENOENT and ENODEV errors encountered operating on directory entries in
/sys that no longer exist.  Some of these errors are suppressed, but
this isn't reliable and varies across versions.  In particular, systemd
239 suppresses ENODEV from sd_device_new_from_syspath() but doesn't
suppress ENODEV from sd_device_get_devnum().  In systemd 243 the call
to sd_device_get_devnum() has been moved, but it still leaks ENOENT
from sd_device_get_is_initialized() (referring to the body of
FOREACH_DIRENT_ALL loop in enumerator_scan_dir_and_add_devices()).

Assume that all ENOENT and ENODEV errors are transient and retry the
call to udev_enumerate_scan_devices().  Don't limit the number, but log
each retry.

Fixes: https://tracker.ceph.com/issues/41036
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit e5921ef4a89f497a0bff6510fce0bb5c242d6172)

qa: add script to test udev event reaping

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 898c113f93a05a18f47f2dd6a94f7cf56c858185)

krbd: increase udev netlink socket receive buffer to 2M

Even though with the previous commit we no longer block between binding
the socket and starting handling events, we still want a larger receive
buffer to accommodate for scheduling delays.  Since the filtering is
done in the listener, an estimate focused on just rbd is not accurate,
but anyway: a pair of "rbd" and "block" events for "rbd map" take 2048
bytes in the receive buffer.  This allows for roughly a thousand of
them ("rbd map" and "rbd unmap" require root and libudev makes use of
SO_RCVBUFFORCE so rmem_max limit is ignored).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1c6cac1acaefdf59c3265d70c8d2191c59f14652)

Conflicts:
src/krbd.cc [ trivial context ]

krbd: avoid udev netlink socket overrun

Because the event(s) we are interested in can be deliveled while we are
still in the kernel finishing map or unmap, we start listening for udev
events before going into the kernel.  However, if (un)mapping takes its
time, udev netlink socket can be fairly easily overrun -- the filtering
is done on the listener side, so we get to process everything, not just
rbd events.  If any of the events of interest get dropped (ENOBUFS), we
hang in poll().

Go into the kernel in a separate thread and leave the main thread to
run the event loop.  The return value is communicated to the reactor
though a pipe.

Fixes: https://tracker.ceph.com/issues/41404
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 5444a1111523bc100bea60958b2671674f6208ac)

Conflicts:
src/krbd.cc [ krbd_spec, ceph_abort_msgf(), make_named_thread()
  not in luminous; luminous is compiled with -std=c++11 ]

krbd: reap all available events before polling again

This also exposes errors from udev_monitor_receive_device() which were
previously ignored.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 53aab34dafcca2ec022102a03905e59cfa34fc84)

krbd: separate event reaping from event processing

Move event processing into UdevMapHandler and UdevUnmapHandler
functors and replace wait_for_udev_{add,remove}() with a single
wait_for_mapping() template.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit c84f9e2f2df47361d7a928d0b25cb84ef332c055)

Conflicts:
src/krbd.cc [ krbd_spec not in luminous ]

krbd: get rid of poll() timeout

This timeout was added as a (very poor) workaround for an issue
addressed in commit 42dd1eae630f ("krbd: fix rbd map hang due to udev
return subsystem unordered").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ffb66ff7106b9d200a7da355199ab362fb611c31)

Conflicts:
src/krbd.cc [ ceph_abort_msgf() not in luminous ]

krbd: fix rbd map hang due to udev return subsystem unordered

The order of subsystem returned by udev_device_get_subsystem
might not be same order as adding subsystem by
udev_monitor_filter_add_match_subsystem_devtype. So if block
event is returned first and rbd event is returned next, then
further poll will get nothing back until timed-out.

Fixes: http://tracker.ceph.com/issues/39089
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 42dd1eae630f31a628051049e4007175561f1939)

Conflicts:
src/krbd.cc [ krbd_spec not in luminous ]

Merge pull request #31294 from smithfarm/wip-qa-ceph-detect-init-luminous

luminous: tests/ceph-disk: drop ceph-detect-init test

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

tests/ceph-disk: drop ceph-detect-init test

This commit fixes an issue with a commit that was cherry-picked into luminous
from mimic.

17bc3dc73a14701f5f6541245955bdd343ffbee2 cherry-picked ceph-detect-init.yaml
from mimic. In mimic, this test works fine because all the supported distros use
systemd. But in luminous we support Ubuntu 14.04 which still uses Upstart
instead of systemd.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd: use next_map to determine if peer is down

or peer was once down after this msg had been sent out

Signed-off-by: Mingxin Liu <mingxin@xsky.com>
(cherry picked from commit dc279bdd8675cf26a4517ac8399a84db510eb883)

mds: cleanup truncating inodes when standby replay mds trim log segments

Standby replay mds first trims expired log segments, then replays new
log segments. It's possible a 'truncate_start' log event is in expired,
but its 'truncate_finish' counterpart is the new log segments. When mds
replays the 'truncate_finish' log event, log segment that contains the
'truncate_start' is already trimmed, so mds does nothing. This causes
leak of Inode::PIN_TRUNCATING and trigger assertion when removing
corresponding inode.

Fixes: https://tracker.ceph.com/issues/40477
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 32121878dd0eef11e481162ebf924a14a91dd341)

Conflicts:
       src/mds/MDCache.cc