Merge pull request #15341 from chardan/jfw-wip-rand_makes_puppies_cry

common: implement random number generator (following N3551)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #15565 from gmayyyha/ceph-osd-help-output

ceph-osd: some flags are not documented in the help output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16942 from jcsp/wip-20888

mon: rate limit on health check update logging

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17404 from rzarzynski/wip-bs-ldout-in-stupid

os/bluestore: replace dout with ldout in StupidAllocator.

Merge pull request #17406 from liewegas/wip-21206

qa/tasks/ceph_manager: revive osds before doing final rerr reset

Merge pull request #17416 from xiexingguo/wip-handle-small-block

os/bluestore: handle small main device properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17423 from liewegas/wip-21203

osd: fix build_initial_pg_history

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17304 from kungf/calc_replicated_acting

osd/PG: remove unused parameter in calc_ec_acting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17088 from zmedico/interval_set_intersection_of_optimization

interval_set: optimize intersection_of

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15816 from chardan/jfw-wip-consolidate-spinlocks

common: consolidate spinlocks

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17265 from zmedico/interval-set-intersection-insert-optimization

interval_set: optimize intersect_of insert operations

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17391 from myoungwon/wip-proxywrite-cleanup

osd: remove unused variable in do_proxy_write

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17394 from Songweibin/wip-common-cleanup

common: drop unused variables "bluestore_csum_*_block" in opts

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17398 from xiexingguo/wip-21175

os/bluestore: don't re-initialize csum-setting for existing blobs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17160 from ceph/wip-luminous-x-master

qa: add upgrade/luminous-x suite

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17405 from tchaikov/wip-remove-boost-submodule

boost: remove boost submodule

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17425 from ceph/wip-ceph-volume-docs

docs: ceph-volume CLI updates

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17426 from liewegas/wip-integration

script/build-integration-branch: print pr url list with titles

Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

script/build-integration-branch: print pr url list with titles

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17353 from scienceluo/wip-luo-fix-rgw-memory-leak-branch

rgw: fix resource leak in rgw_bucket.cc and rgw_user.cc

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/ceph-volume prepare partitions with PARTUUID

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Initial check in

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/ceph-volume prepare needs both volume group and lv names

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate re-links the journal always

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume emphasize the danger of calling prepare for an existing OSD

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: fix resource leak in rgw_bucket.cc and rgw_user.cc

Fixes: http://tracker.ceph.com/issues/21214
Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

Merge pull request #17421 from nrdmn/dashboard-colors

mgr/dashboard: change raw usage chart's color depending on usage

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17424 from theanalyst/release-eol-updates-l

doc/releases: the Kraken sleepeth, faintest sunlights flee

Reviewed-by: Nathan Cutler <ncutler@suse.com>

mon: rate limit on health check update logging

Fixes: http://tracker.ceph.com/issues/20888
Signed-off-by: John Spray <john.spray@redhat.com>

doc/releases: the Kraken sleepeth, faintest sunlights flee

updating releases to retire Kraken, Hammer. Also updated to mention
Jewel will be around until at least N is finalized

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

osd: fix build_initial_pg_history

We need to update our info about the previous interval in order to
detect interval changes properly.

Fixes: http://tracker.ceph.com/issues/21203
Signed-off-by: w11979 <wang.wenfeng@h3c.com>
Signed-off-by: Sage Weil <sage@redhat.com>

doc/ceph-volume update intro to mention dm-cache

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #17364 from theanalyst/release-updates

doc: update releases to the current state

mgr/dashboard: change raw usage chart's color depending on usage

Signed-off-by: Nick Erdmann <n@nirf.de>

mon: add helper for emitting logs by health status

We were starting to repeat our little "if(status==HEALTH_ERR"
blocks in too many places.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update releases to the current state

Adding Luminous release date, and dropping links for dev releases (since
they've been merged to 12.2.0), also rearranged the table so that newer
releases come left

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

os/bluestore: handle small main device properly

Originally reported by MarcinDulak on 'ceph-users' channel.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting nits

So we don't output anything if csum is currently off:

_do_alloc_write blob Blob(0x7fad7cc25950 blob([0x42a80000~10000] none/0x1)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't re-initialize csum-setting for existing blobs

The global checksum setting may change, e.g., from NONE to CRC32,
which can cause improper re-initialization of the csum-settings of
existing blobs(e.g., partial write/overwrite may turn out to shrink
'csum_data').

We could develop some complicated solutions but for now let's not
bother since the above scenario is rare.

Fixes: http://tracker.ceph.com/issues/21175
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17397 from ZVampirEM77/wip-doc

doc: fix doc/radosgw/admin.rst typos

Reviewed-by: Jos Collin <jcollin@redhat.com>

make-dist: download and repackage boost

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16521 from myoungwon/wip-objecter-finishers

osd: add multiple objecter finishers

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16586 from tianshan/fix_logrotate

logrotate: add systemd reload in logrotate in case of centos minimal without killall

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16643 from Aran85/wip-mon-deadlock-on-shutdown

kv/leveldb: fix deadlock when close db

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16733 from liupan1111/wip-bluefs-fio

os: add compile option to build libbluefs.so

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17244 from liewegas/wip-21121

osd: subscribe to new osdmap while waiting_for_healthy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16925 from gmayyyha/mgr-osd-metadata

mgr: implement completion of osd MetadataUpdate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17036 from majianpeng/add-pmem-submodule

submodule: make libmpem as a submodule.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17354 from liewegas/wip-bluefs-compaction

os/bluestore/BlueFS: compact log even when sync_metadata sees no work

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17357 from Liuchang0812/wip-bluefs-not-assert-eexist

os/bluestore: do not core dump when BlueRocksEnv gets EEXIST error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17360 from tchaikov/wip-crush-cleanup

crush: various CrushWrapper cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17379 from liewegas/wip-div-p

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

qa/tasks/ceph_manager: revive osds before doing final rerr reset

We assume below that rerrosd is up, but it may not be when we exit the
loop.

Fixes: http://tracker.ceph.com/issues/21206
Signed-off-by: Sage Weil <sage@redhat.com>

boost: remove boost submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: replace dout with ldout in StupidAllocator.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #17401 from liewegas/wip-disable-pidfile

test/CMakeLists: disable test-pidfile.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17390 from tchaikov/wip-test-coredump

test: Don't dump core when using EXPECT_DEATH

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: fix doc/radosgw/admin.rst typos

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

test/CMakeLists: disable test-pidfile.sh

Signed-off-by: Sage Weil <sage@redhat.com>

common: drop unused variables "bluestore_csum_*_block" in opts

Signed-off-by: songweibin <song.weibin@zte.com.cn>

test: Don't dump core when using EXPECT_DEATH

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: remove unused variable in do_proxy_write

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #17362 from C2python/update_example

doc: Revise the Example of Bucket Policy

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Adam Emerson <aemerson@redhat.com>

Merge pull request #17387 from tchaikov/wip-package-kvstoretool-man

rpm,deb: package ceph-kvstore-tool man page

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17145 from optimistyzy/822_bl

os/bluestore/NVMEDevice: change write_bl to bl

Reviewed-by: Haomai Wang <haomai@xsky.com>

NVMEDevice: change write_bl to bl

With this patch, both read and write can use this field in
task.

Signed-off-by: Ziye Yang <optimistyzy@gmail.com>
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

interval_set: optimize intersection_of

Iterate over all elements of the smaller set, and use find_inc to
locate elements from the larger set in logarithmic time. This greatly
improves performance when one set is much larger than the other:

     2 +-+--+----+----+----+----+----+----+----+----+--+-+
P      +*                                                +
E      |*                                                |
R  1.8 +*                                                +
F      | *                                               |
O      | *                                               |
R  1.6 + *                                               +
M      | *                                               |
A      | *                                               |
N  1.4 + *                                               +
C      |  *                                              |
E      |  *                                              |
   1.2 +   *                                             +
R      |    *                                            |
A      |     *                                           |
T    1 +      ***                                        +
I      |         ******                                  |
O      +               ***********************************
   0.8 +-+--+----+----+----+----+----+----+----+----+--+-+
       0   0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9   1

                          SET SIZE RATIO

The above plot compares performance of the new intersection_size_asym
function to the existing intersection_of function. The performance of
intersection_size_asym gets worse as the set size ratio approaches 1.
For set size ratios where the performance ratio is greater than 1, the
performance of intersection_size_asym is superior. Therefore, this
patch only uses intersection_size_asym when the set size ratio is less
than or equal to 0.1 (code uses the reciprocal which is 10).

The plot was generated using benchmark results produced by the
following program:

#include <iostream>
#include <sys/timeb.h>
#include "include/interval_set.h"

int main()
{
  const int interval_count = 100000;
  const int interval_distance = 4;
  const int interval_size = 2;
  const int sample_count = 8;
  const int max_offset = interval_count * interval_distance;
  interval_set<int> a, b, intersection;

  for (int i = 0; i < max_offset; i+=interval_distance) {
    a.insert(i, interval_size);
  }

  for (int m = 1; m < 100; m++) {
    float ratio = 1 / float(m);

    for (int i = 0; i < max_offset; i+=interval_distance*m) {
      b.insert(i, interval_size);
    }

    struct timeb start, end;
    int ms = 0;
    for (int i = 0; i < sample_count; i++) {
      ftime(&start);
      intersection.intersection_of(a, b);
      ftime(&end);
      ms += (int) (1000.0 * (end.time - start.time)
        + (end.millitm - start.millitm));
      intersection.clear();
    }
    b.clear();

    std::cout << ratio << "\t" << ms << std::endl << std::flush;
  }
}

Signed-off-by: Zac Medico <zmedico@gmail.com>

rpm,deb: package ceph-kvstore-tool man page

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: rm unused parameter in calc_replicated_acting

Signed-off-by: yang.wang <yang.wang@easystack.cn>

osd/PG: rm unused parameter in calc_ec_acting

Signed-off-by: yang.wang <yang.wang@easystack.cn>

Merge pull request #17382 from liewegas/wip-integration

src/script/build-integration-branch

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/script/build-integration-branch

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17333 from liewegas/wip-luminous-mon-bits

mon: final luminous compatset feature and osdmap flag

Merge pull request #17376 from dillaman/ritz303-wip-doc-20437

doc/rbd: iSCSI Gateway Documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

interval_set: optimize intersect_of for identical spans

Optimize comparisons for identical spans of intervals.
When this patch is combined with the previous map insert
optimization, a benchmark using 400000 identical
intervals shows a 7 times performance improvement in
comparison to without the patches.

Signed-off-by: Zac Medico <zmedico@gmail.com>

interval_set: optimize intersect_of insert operations

Use the std::map insert method with hint iterator to optimize
inserts. This increases performance more than 3.5 times for
large numbers of intervals. This will help performance
especially in the PGPool::update method, where profiling data
has shown that intersection operations are a hot spot. The
following benchmark data is for 400000 intervals:

    4 +-+--+----+----+----+----+----+----+----+----+--+-+
P     +    +    +    +    +    +    +    +  *************
E     |                             ********            |
R 3.5 +-+                       ****                  +-+
F     |                   ******                        |
O     |                 **                              |
R   3 +-+           ****                              +-+
M     |          ***                                    |
A     |        **                                       |
N 2.5 +-+     *                                       +-+
C     |     **                                          |
E     |     *                                           |
    2 +-+ **                                          +-+
R     |  **                                             |
A     | **                                              |
T 1.5 +**                                             +-+
I     |**                                               |
O     +*   +    +    +    +    +    +    +    +    +    +
    1 +*+--+----+----+----+----+----+----+----+----+--+-+
      0   0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9

                        SET SIZE RATIO

The above chart was generated using benchmark results
from the following program:

#include <iostream>
#include <sys/timeb.h>
#include "include/interval_set.h"

int main(int argc, char *argv[])
{
  const int interval_count = std::stoi(argv[1]);
  const int interval_distance = 4;
  const int interval_size = 2;
  const int sample_count = 8;
  const int max_offset = interval_count * interval_distance;
  interval_set<int> a, b, intersection;

  for (int i = 0; i < max_offset; i+=interval_distance) {
    a.insert(i, interval_size);
  }

  for (int m = 1; m < 100; m++) {
    float ratio = 1 / float(m);

    for (int i = 0; i < max_offset; i+=interval_distance*m) {
      b.insert(i, interval_size);
    }

    struct timeb start, end;
    int ms = 0;
    for (int i = 0; i < sample_count; i++) {
      ftime(&start);
      intersection.intersection_of(a, b);
      ftime(&end);
      ms += (int) (1000.0 * (end.time - start.time)
        + (end.millitm - start.millitm));
      intersection.clear();
    }
    b.clear();

    std::cout << ratio << "\t" << ms << std::endl << std::flush;
  }
}

Signed-off-by: Zac Medico <zmedico@gmail.com>

Merge pull request #17217 from tchaikov/wip-19605

osd/PG: discard msgs from down peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17339 from amitkumar50/cov-osd-2

osd: Deleting dead code PrimaryLogPG.cc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

mon: set purged_snapdirs OSDMap flag once snapsets have all converted

This makes it easier to test whether the upgrade + conversion has
completed.  In particular, mimic+ will be able to simply test for this
flag without waiting for complete PG stats to know whether it is safe to
upgrade beyond luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: CEPH_MON_FEATURE_INCOMPAT_LUMINOUS

- set this once we have a luminous quorum
- require luminous if we have this set locally
- include this on mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17247 from liewegas/wip-post-luminous-cleanup

osd: clean up pre-luminous compat cruft (part 1 of many)

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17071 from ivancich/wip-dup-ops-perf-regression

osd: speed up get_key_name

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17206 from rzarzynski/wip-rgw-21085

rgw: fix accessing expired memory in PrefixableSignatureHelper.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17250 from shashalu/fix-get-bucket-location

rgw: return bucket's location no matter which zonegroup it located in.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17294 from tchaikov/wip-clang-warnings

rgw,unittest_bit_alloc: silence clang analyzer warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17343 from yaozongyou/rgw-admin-check-parameters-for-friendly-prompt

rgw-admin: check input parameters for friendly prompt.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: tweaks for the RBD iSCSI configuration documentation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15994 from chuang-he/fix_decrypt

rgw: Data encryption is not follow the AWS agreement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rbd: Replaced numbered steps with a # for dynamic numbering

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the content based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the requirements based on engineering feedback

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway requirements content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway overview content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI gateway 'gwtop' monitoring content

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI targets, Ansible and CLI docs

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Added the iSCSI initiators, RHEL, Windows, and ESX docs

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>

doc/rbd: Updated the "toctree" adding the iSCSI gateway overview doc

Fixes: http://tracker.ceph.com/issues/20437
Signed-off-by: Aron Gunn <agunn@redhat.com>