interval_set: optimize intersect_of for identical spans

Optimize comparisons for identical spans of intervals.
When this patch is combined with the previous map insert
optimization, a benchmark using 400000 identical
intervals shows a 7 times performance improvement in
comparison to without the patches.

Signed-off-by: Zac Medico <zmedico@gmail.com>
(cherry picked from commit b6a035666c2765f8895ee9991348dbc025613ed7)

interval_set: optimize intersect_of insert operations

Use the std::map insert method with hint iterator to optimize
inserts. This increases performance more than 3.5 times for
large numbers of intervals. This will help performance
especially in the PGPool::update method, where profiling data
has shown that intersection operations are a hot spot. The
following benchmark data is for 400000 intervals:

    4 +-+--+----+----+----+----+----+----+----+----+--+-+
P     +    +    +    +    +    +    +    +  *************
E     |                             ********            |
R 3.5 +-+                       ****                  +-+
F     |                   ******                        |
O     |                 **                              |
R   3 +-+           ****                              +-+
M     |          ***                                    |
A     |        **                                       |
N 2.5 +-+     *                                       +-+
C     |     **                                          |
E     |     *                                           |
    2 +-+ **                                          +-+
R     |  **                                             |
A     | **                                              |
T 1.5 +**                                             +-+
I     |**                                               |
O     +*   +    +    +    +    +    +    +    +    +    +
    1 +*+--+----+----+----+----+----+----+----+----+--+-+
      0   0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9

                        SET SIZE RATIO

The above chart was generated using benchmark results
from the following program:

#include <iostream>
#include <sys/timeb.h>
#include "include/interval_set.h"

int main(int argc, char *argv[])
{
  const int interval_count = std::stoi(argv[1]);
  const int interval_distance = 4;
  const int interval_size = 2;
  const int sample_count = 8;
  const int max_offset = interval_count * interval_distance;
  interval_set<int> a, b, intersection;

  for (int i = 0; i < max_offset; i+=interval_distance) {
    a.insert(i, interval_size);
  }

  for (int m = 1; m < 100; m++) {
    float ratio = 1 / float(m);

    for (int i = 0; i < max_offset; i+=interval_distance*m) {
      b.insert(i, interval_size);
    }

    struct timeb start, end;
    int ms = 0;
    for (int i = 0; i < sample_count; i++) {
      ftime(&start);
      intersection.intersection_of(a, b);
      ftime(&end);
      ms += (int) (1000.0 * (end.time - start.time)
        + (end.millitm - start.millitm));
      intersection.clear();
    }
    b.clear();

    std::cout << ratio << "\t" << ms << std::endl << std::flush;
  }
}

Signed-off-by: Zac Medico <zmedico@gmail.com>
(cherry picked from commit 32bc0430f70b057d1bba623252e92ab9f279028d)

Merge pull request #17210 from dzafman/wip-20730-jewel

jewel: core: disable skewed utilization warning by default

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16405 from dzafman/wip-20041-jewel

jewel: osd: scrub_to specifies clone ver, but transaction include head write ver

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16169 from Vicente-Cheng/wip-20511-jewel

jewel: osd: ReplicatedPG: solve cache tier osd high memory consumption

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16167 from Vicente-Cheng/wip-20492-jewel

jewel: osd: omap threadpool heartbeat is only reset every 100 values

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16151 from Vicente-Cheng/wip-20349-jewel

jewel: cephfs: df reports negative disk "used" value when quota exceed

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16150 from Vicente-Cheng/wip-20403-jewel

jewel: mds: save  projected path into inode_t::stray_prior_path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16141 from smithfarm/wip-20270-jewel

jewel: tests: LibRadosMiscConnectFailure.ConnectFailure hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16285 from smithfarm/wip-18137-jewel

jewel: rbd-mirror: image sync should send NOCACHE advise flag

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16295 from smithfarm/wip-20265-jewel

jewel: rbd: cli: ensure positional arguments exist before casting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16276 from smithfarm/wip-19563-jewel

jewel: rgw: folders starting with _ underscore are not in bucket index

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16268 from cbodley/wip-16072-jewel

jewel: rgw: rgw_common.cc: modify the end check in RGWHTTPArgs::sys_get

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #16266 from cbodley/wip-20408

jewel: rgw: multipart copy-part remove '/' for s3 java sdk request header

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15988 from cbodley/wip-jewel-15983

jewel: rgw: fix infinite loop in rest api for log list

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15449 from smithfarm/wip-19720-jewel

jewel: rgw: rgw_file: fix size and (c|m)time unix attrs in write_finish

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #17351 from liewegas/wip-prune-past-intervals-jewel

osd: add osd_hack_prune_past_intervals

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16059 from vumrao/wip-vumrao-20353

jewel: osd: Add heartbeat message for Jumbo Frames (MTU 9000)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15726 from asheplyakov/20314-bp-jewel

jewel: mon: fail to form large quorum; msg/async busy loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

osd: add osd_hack_prune_past_intervals

Last ditch (but dangerous) method of reducing memory usage for
past_intervals, which can help very very unhappy clusters recovery.
A properly implemented version of this is in luminous.  This hacky
version was used successfully to recover multiple jewel-based clusters,
but is still only recommended for use when the OSD is otherwise unable
to recover.

This change is not cherry-picked from master because luminous implements
a more sophisticated version of this that changes the past intervals
representation entirely, but it is too invasive to backport.  This
workaround to prune just on startup should be sufficient for emergencies.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15719 from Vicente-Cheng/wip-20146-jewel

jewel: rgw: 'gc list --include-all' command infinite loop the first 1000 items

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15477 from smithfarm/wip-19767-jewel

jewel: rgw: Delete non-empty bucket in slave zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16015 from asheplyakov/19208-bp-jewel

jewel: osd: osd_internal_types: wake snaptrimmer on put_read lock, too

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15762 from smithfarm/wip-17385-jewel

jewel: ceph cli: Rados object in state configuring race

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15474 from smithfarm/wip-19559-jewel

jewel: osd: objecter full_try behavior not consistent with osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15947 from asheplyakov/20428-bp-jewel

jewel: osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid missing wakeup signal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15475 from tchaikov/wip-16239-jewel

qa: add a sleep after restarting osd before "tell"ing it

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15473 from smithfarm/wip-20080-jewel

jewel: build/ops: rpm: set subman cron attributes in spec file

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Thomas Serlin <tserlin@redhat.com>

Merge pull request #15460 from smithfarm/wip-17843-jewel

jewel: rbd: object-map: batch updates during trim operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15488 from dillaman/wip-20023-jewel

jewel: rbd: rbd-mirror: ensure missing images are re-synced when detected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15602 from tchaikov/wip-20175-jewel

jewel: tests: decouple ceph_test_librbd_api from libceph-common

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16124 from smithfarm/wip-19174-jewel

jewel: rbd: rbd_clone_copy_on_read ineffective with exclusive-lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15760 from smithfarm/wip-20325-jewel

libradosstriper silently fails to delete empty objects in jewel

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16061 from weiqiaomiao/wqm-wip-ceph-disk-jewel

jewel: ceph-disk: remove the special check to bcache devices

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge pull request #15503 from Vicente-Cheng/wip-20011-jewel

jewel: tools: ceph-disk: separate ceph-osd --check-needs-* logs

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw_file:  pre-compute unix attrs in write_finish()

New serialized Unix attrs need to reflect the change being made,
and should be reverted if the change fails.

Fixes: http://tracker.ceph.com/issues/19653
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit ed91d23ccaaac4e72a4c28a58e77485395949f04)

Conflicts:
    src/rgw/rgw_file.cc (processor->complete() takes different arguments in
        jewel than in master)

Merge pull request #14874 from ceph/wip-19805-jewel

jewel: rbd: default features should be negotiated with the OSD

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14977 from dillaman/wip-19873-jewel

jewel: rbd: rbd-mirror: failover and failback of unmodified image results in split-brain

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15461 from smithfarm/wip-19795-jewel

jewel: tests: test_notify.py: assert(not image.is_exclusive_lock_owner()) on line 147

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15463 from smithfarm/wip-20017-jewel

jewel: rbd: rbd-nbd: kernel reported invalid device size (0, expected 1073741824)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15464 from smithfarm/wip-20153-jewel

jewel: rbd: Potential IO hang if image is flattened while read request is in-flight

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15455 from smithfarm/wip-19838-jewel

jewel: rgw: reduce log level of 'storing entry at' in cls_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15456 from smithfarm/wip-20003-jewel

jewel: rgw: rgw_file:  v3 write timer does not close open handles

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15457 from smithfarm/wip-20018-jewel

jewel: rgw: multisite: memory leak on failed lease in RGWDataSyncShardCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15459 from smithfarm/wip-20144-jewel

jewel: rgw: rgw_file: cannot delete bucket w/uxattrs

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15465 from smithfarm/wip-20155-jewel

jewel: rgw: fix crash caused by shard id out of range when listing data log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: clean up object map update interface, revisited

In master, the "batch update" change [1] was merged before the "order
concurrent updates" [2], while in jewel the latter is already
backported [3]. A backport of [1] to jewel was attempted, and was
necessarily applied on top of [3] - i.e. in the reverse order compared
to how the commits went into master. This reverse ordering caused the
automated cherry-pick to miss some parts of [1] which this commit is
adding manually.

[1] https://github.com/ceph/ceph/pull/11510
[2] https://github.com/ceph/ceph/pull/12420
[3] https://github.com/ceph/ceph/pull/12909

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: batch ObjectMap updations upon trim

Shrinking a clone which has snapshots and does not share
majority of objects with its parent (i.e., there are less
objects to be copied up) involves huge number of object
map updates -- two (pre, post) per object. This results
in lots of requests to be send to OSDs especially when
trimming a gigantus image. This situation can be optimized
by sending batch ObjectMap updates for an object range
thereby significantly cutting down OSD traffic resulting
in faster trim times.

Fixes: http://tracker.ceph.com/issues/17356
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 05653b7c512334533b801013f7e426363237301b)

Merge PR #16248 into jewel

* refs/remotes/upstream/pull/16248/head:
client: don't re-send interrupted flock request
mds/flock: properly remove item from global_waiting_locks
mds/flock: properly update ceph_lock_state_t::client_waiting_lock_counts
mds/flock: don't import ceph_lock_state_t::waiting_locks

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #15454 from smithfarm/wip-19775-jewel

jewel: rgw: multisite: realm rename does not propagate to other clusters

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15453 from smithfarm/wip-19768-jewel

jewel: rgw: multisite: operating bucket's acl&cors is not restricted on slave zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15452 from smithfarm/wip-19765-jewel

jewel: rgw: when uploading objects continuously into a versioned bucket, some objects will not sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15451 from smithfarm/wip-19764-jewel

jewel: rgw: set latest object's acl failed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15450 from smithfarm/wip-19758-jewel

jewel: rgw: multisite: after CreateBucket is forwarded to master, local bucket may use different value for bucket index shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

config: disable skewed utilization warning by default

This has a few problems:

1- It does not do it's analysis over CRUSH rule roots/classes, which
means that an innocent user of classes will see skewed usage (bc hdds are
more full than ssds, say)

2- It does not take degraded clusters into account, which means the warning
will appear when a fresh OSD is added.

See http://tracker.ceph.com/issues/20730

Based on master commit 7832c53 but we've decided not to remove the code
in the older releases so it can be used if appropriate for a
particular cluster configuration.  Also, 7832c53 won't cleanly
cherry-pick so this is easier.

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/rados/test-upgrade-v11.0.0: skip ListObjectsError

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test-upgrade-v11.0.0: skip more tests

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16870 from liewegas/wip-20929-jewel

jewel: ceph-disk: dmcrypt cluster must default to ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #14659 from asheplyakov/19473-jewel

jewel: rgw: add the remove-x-delete feature to cancel swift object expiration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15322 from gregsfortytwo/jewel-19931-snaptrim-shutdown

jewel: osd: Reset() snaptrimmer on shutdown and do not default-abort on leaked pg refs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15236 from asheplyakov/20036-bp-jewel

jewel: mon: factor mon_osd_full_ratio into MAX AVAIL calc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14346 from ceph/wip-ceph-disk-fix-jewel

jewel: build/ops: Add fix subcommand to ceph-disk, fix SELinux denials, and speed up upgrade from non-SELinux enabled ceph to an SELinux enabled one

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15083 from asheplyakov/19926-bp-jewel

jewel: mon: crash on shutdown, lease_ack_timeout event

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14663 from smithfarm/wip-19228-jewel

jewel: rbd: Enabling mirroring for a pool with clones may fail

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15050 from asheplyakov/19265-bp-jewel

jewel: core: an OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15065 from asheplyakov/19915-bp-jewel

jewel: osd: PrimaryLogPG: do not call on_shutdown() if (pg.deleting)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14943 from shinobu-x/wip-18293-jewel

jewel: osd: leaked MOSDMap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14699 from smithfarm/wip-19679-jewel

jewel: mds: damage reporting by ino number is useless

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14673 from smithfarm/wip-19466-jewel

jewel: mds: log rotation doesn't work if mds has respawned

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/rados/test-upgrade-v11.0.0.sh: more tests skipped

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15442 from Vicente-Cheng/wip-20025-jewel

jewel: cephfs: osdc/Filer: truncate large file party by party

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #14691 from smithfarm/wip-19571-jewel

jewel: tests: upgrade:client-upgrade/firefly-client-x: drop CentOS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: default features should be negotiated with the OSD

Derived from f066ce8f80bce929edc209590efd47cce2196ae2

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: add get_all_features on client side

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 84f6d5c109911923c25414de639308921983e438)

Conflicts:
src/cls/rbd/cls_rbd_client.h: trivial resolution

Merge pull request #15448 from smithfarm/wip-19615-jewel

jewel: rgw: multisite: bucket zonegroup redirect not working

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: upgrade:client-upgrade/firefly-client-x: drop CentOS

The RBD suite needs ceph-cm-ansible to install qemu-kvm on CentOS, but doing
that breaks the firefly install on CentOS because:

1. the qemu-kvm that gets installed is from hammer (0.94.5)
2. qemu-kvm brings in librados2, librbd1 as dependencies

As a result, the hammer librados2 and librbd1 are installed on the test nodes
even before the teuthology install task starts. When it does start and tries
to install firefly, it fails because firefly librados2 and librbd1 cannot be
installed over their hammer versions.

Fixes: http://tracker.ceph.com/issues/19571
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15197 from smithfarm/wip-20007-jewel

jewel: rgw: remove unnecessary 'error in read_id for object name: default'

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15051 from asheplyakov/19910-bp-jewel

jewel: build/ops: extended, customizable systemd ceph-disk timeout

Reviewed-by: David Disseldorp <ddiss@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15428 from smithfarm/wip-20162-jewel

jewel: build/ops: deb: Fix logrotate packaging

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15447 from smithfarm/wip-19613-jewel

jewel: rgw: multisite: rest api fails to decode large period on "period commit"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test: remove hard-coded image name from RBD metadata test

Fixes: http://tracker.ceph.com/issues/19798
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8f72e745e157cc12d76ca6babe956c5698ee297f)

librbd: relax "is parent mirrored" check when enabling mirroring for pool

If the parent is in the same pool and has the journaling feature enabled
we can assume the mirroring will eventually be enabled for it.

Fixes: http://tracker.ceph.com/issues/19130
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit fe31bca22f90ce02f461d6421a4f66539db888d3)

Merge pull request #15196 from smithfarm/wip-20006-jewel

jewel: build/ops: rpm: fix python-Sphinx package name for SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/workunits/rados/test/test-upgrade-v11.0.0: skip Quota test

This makes a pool go full which confounds the wait for healthy check
during restarts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test-upgrade-v11.0.0: skip api_io *CmpExt*

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test-upgrade-11.0.0: skip RoundTrip3

This does the crc op, which isn't in jewel.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16343 from dillaman/wip-20630-jewel

jewel: tests: qa/tasks: rbd-mirror daemon not properly run in foreground mode

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-disk: dmcrypt cluster must default to ceph

If ceph_fsid is not found, which is the case for legacy dmcrypted OSD,
the cluster must default to ceph, as it was before support for non
standard cluster names was introduced.

Fixes: http://tracker.ceph.com/issues/20893
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit ed22159fdd7071a110e81cd7b63d7e0f2296761b)

Merge pull request #13507 from shinobu-x/wip-18468-jewel

jewel: mon: Disallow enabling 'hashpspool' option to a pool without some kind of --i-understand-this-will-remap-all-pgs flag

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc/Filer: truncate large file party by party

Fixes: http://tracker.ceph.com/issues/19755
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 5fab215e461e5ecc36c0f9d9ea867f6c45e80263)

Conflicts:
src/osdc/Filer.h
src/osdc/Filer.cc
    - add parameter to fit _modify (need onack parameter)
src/mds/MDCache.cc
    - make truncate() consistency

osd: scrub_to specifies clone ver, but transaction include head write ver

Fixes: http://tracker.ceph.com/issues/20041
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit fd598a0d23d61c645633ae774c3404a43d035e3c)

Conflicts:
src/osd/ReplicatedPG.cc (trivial)

jewel:ceph-disk:remove the special check to bcache devices

if we skip partition when coming cross the bcache device in ceph-disk,
the udev can't detect the osd in such devices when we reboot the osd's host,
so the osd who deploy in bcache device can't auto start when the host restart.

And partition is now supported by bcache devices (https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/drivers/md/bcache?id=b8c0d911ac5285e6be8967713271a51bdc5a936a),
we can remove this special check now.

This cannot be cherry-picked from master because 'accept bcache devices as data disks' [21f0216](https://github.com/ceph/ceph/commit/21f0216287485e6cce8811f324fee804ef733524)
is a jewel-only feature not present in master.

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

jewel: mon: Fix status output warning for mon_warn_osd_usage_min_max_delta

Fixes: http://tracker.ceph.com/issues/20544
Caued by: 489e810c37ed6fb9d32d1015634041a577501ee4

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 56f9808016ab289bdc0ce7cfbb0503c78b509593)

Conflicts:
apply changes to src/mon/PGMonitor.cc instead of src/mon/PGMap.cc

Merge pull request #16289 from cbodley/wip-20513

jewel: rgw: cls: ceph::timespan tag_timeout wrong units

messages/MOSDPing: optimize encode and decode of dummy payload

The dummy payload doesn't need to be processed, we can just skip over
it when decoding and we can use a single bufferptr instead of entire
bufferlist to encode it.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
(cherry picked from commit 15ce0772841f913d1eef1daebad0834e5f862383)

messages/MOSDPing: fix the inflation amount calculation

If user specifies a min_message_size small enough (or zero to disable
it altogether), OSDs will crash and burn while trying to allocate
almost 4GB of payload (both min_message_size and payload.length() are
unsigned, so it'll roll over back to 4GB and MAX(4GB, 0) will use 4GB).
If the size of dummy payload is 0, don't bother constructing bufferptr
and bufferlist, then encoding that.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
(cherry picked from commit d959735a7d684018df564573a2ff88092d870fc1)

OSD: mark two heartbeat config opts as observed

"osd heartbeat min size" and "osd heartbeat interval" can be changed
at runtime, because their values, when used, are always taken from
global Ceph configuration. Mark them as observed, so the message
the user sees once they're changed doesn't confuse them.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
(cherry picked from commit 44b7839c788b20e0b9da76bc3838ae10d3ad2f89)

messages/MOSDPing: initialize MOSDPing padding

This memory must be initialized or else valgrind will be very unhappy.

Avoid the cost of zeroing (or even allocating) the buffer for normal
padding values by (re)using a static zero buffer.

Fixes: http://tracker.ceph.com/issues/20323
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9beaf5efd74daa8c15e50b42583264d1252a85f5)

osd: heartbeat with packets large enough to require working jumbo frames.

We get periodic reports that users somehow misconfigure one of their switches
so that it drops jumbo frames, yet the servers are still passing them along. In
that case, MOSDOp messages generally don't get through because they are much
larger than the 1500-byte non-jumbo limit, but the MOSDPing messages have kept
going (as they are very small and dispatched independently, even when the
server is willing to make jumbo frames). This means peer OSDs won't mark down
the ones behind the broken switch, despite all IO hanging.
Push the MOSDPing message size over the 1500-byte limit so that anybody in
this scenario will see the OSDs stuck behind a bad switch get marked down.

Fixes: http://tracker.ceph.com/issues/20087
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 2096113d9e1589c571d96e34dd9cd841308a2567)

 Conflicts:
src/messages/MOSDPing.h
         - Changed HEAD_VERSION to 3 and kept COMPAT_VERSION to 1.
         - In class MOSDPing removed following line:
               if (header.version >= 2)
         - To keep ::decode(stamp, p) without condition because HEAD_Version
           is already 3 now and this condition is removed in the backport commit.

qa/tasks: rbd-mirror daemon not properly run in foreground mode

Fixes: http://tracker.ceph.com/issues/20630
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4fa1918717b25a5ffafbf649eedcfe7d5ab829c2)