Merge pull request #14363 from majianpeng/finisher-optimize

common/Finisher: batch handle perfcounter && only send signal when waiter existed.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14627 from badone/wip-peering-state-timing

osd: Implement peering state timing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14709 from majianpeng/luestore-txc-finalize-kv

os/blueStore: In osd_tp_thread, call _txc_finalize_kv.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14932 from markhpc/wip-rocksdb-compaction-readahead

common/config_opts.h: compaction readahead for bluestore/rocksdb

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14879 from wjwithagen/wip-wjw-reduce-noise-3

cls/log/cls_log.cc: reduce logging noise

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14928 from drunkard/master

doc: remove deprecated subcommand in man/8/ceph.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14723 from neha-ojha/objclass-sdk

osd: objclass sdk

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14917 from yuriw/wip-add-overrite-cls

qa/added overrides

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls/log/cls_log.cc: reduce logging noise

 - The other reference in the source as already at 20.
      ./src/cls/timeindex/cls_timeindex.cc:85:
CLS_LOG(20, "storing entry at %s", index.c_str());

   And we need not always know where in the log items are stored.
   So it looks like a leftover debug feature.

Fixes: http://tracker.ceph.com/issues/19835
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14913 from liewegas/wip-19826

mon: do not prime_pg_temp creating pgs; clean up pg create conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/Finisher: No waite_for_empty no signal.

Avoid every time to send a signal because wait_for_empty is rare event.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/Finisher: batch handle perfcounter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/perf_counter: make tinc() handle batch-processing

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/config_opts.h: compaction readahead for bluestore/rocksdb

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #14903 from dillaman/wip-19413

cls_rbd: default initialize snapshot namespace for legacy clients

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14875 from dillaman/wip-rbd-trash-parent

rbd: info command should indicate if parent is in trash

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14929 from tchaikov/wip-disable-isal-on-arm

test/compressor: disable isal tests if not available

Reviewed-by: Sage Weil <sage@redhat.com>

test/compressor: disable isal tests if not available

the isal-zlib compressor plugin is not supported on non-intel archs, and
if the user does not have a capable yasm compiler, neither will he/she
get this plugin.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: cleanup in doc/man/8/ceph.rst

Subcommand doc has been removed in 904913b37f241a73aa6722a18151d3166c8c93dd,
this is cleanup.

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

Merge pull request #14822 from jan--f/fuse-normalize-wire-o_flags

fs/ceph-fuse: normalize file open flags on the wire

Merge pull request #14909 from joscollin/wip-dmclock-error

dmclock: error: ‘function’ in namespace ‘std’ does not name a template type

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14748 from smithfarm/wip-teuthology-worker

doc: mention teuthology-worker security group

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon/OSDMonitor: skip new pgs for prime_pg_temp

If the pg didn't previously exist we don't need to (and can't) prime
pg_temp for it.  This avoids crashing on mapping.get().

Signed-off-by: Sage Weil <sage@redhat.com>

qa/added overrides

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #14412 from mikulely/cleanup-civetweb-port

rgw: drop unused port var

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13451 from xiexingguo/wip-add-up-primary-sum

mon/PGMap: add up_primary pg number field for pg-dump cmd

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14497 from tchaikov/wip-pybind-better-error-msg

pybind: better error msg

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14687 from jcsp/wip-mgr-suite

qa/suites: move mgr tests into rados suite

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14810 from liewegas/wip-status

mon: show inactive % in ceph status

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14821 from badone/wip-no-death-coredumps

tests: Don't dump core when using EXPECT_DEATH

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14857 from joscollin/wip-arch-cpuid

arch: use __get_cpuid instead of do_cpuid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12669 from hjwsm1989/unfound-objects

osd/PG: fix possible overflow on unfound objects

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12882 from wonzhq/pglog-fixes

osd: pglog trimming fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14444 from liewegas/wip-past-intervals

osd: simplify past_intervals representation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14898 from tchaikov/wip-clean-pending_created_pgs

mon/OSDMonitor: cleanup pending_created_pgs after done with it

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14904 from tchaikov/wip-19638-followon

osd: stop mgrc earlier in shutdown()

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix legacy pg create conditions

1- in reality we don't really care if pg_temp is set.
2- compare against acting_primary since that's what actually matters.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix prime_pg_temp vs creating condition

We want to *skip* creating PGs.

Fixes: http://tracker.ceph.com/issues/19826
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14858 from dmick/wip-config-dump

mon/ConfigKeyService: add 'config-key dump' to show keys and vals

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14870 from smithfarm/wip-gperftools

build/ops: rpm: gperftools-devel >= 2.4

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

dmclock: error: ‘function’ in namespace ‘std’ does not name a template type

The following error appears during make:

In file included from ceph/src/dmclock/support/src/run_every.cc:10:0:
ceph/src/dmclock/support/src/run_every.h:30:10: error: ‘function’ in namespace ‘std’ does not name a template type
     std::function<void()>     body;
          ^~~~~~~~
ceph/src/dmclock/support/src/run_every.h:46:12: error: ‘std::function’ has not been declared
       std::function<void()> _body) :
            ^~~~~~~~
ceph/src/dmclock/support/src/run_every.h:46:20: error: expected ‘,’ or ‘...’ before ‘<’ token
       std::function<void()> _body) :
                    ^
ceph/src/dmclock/support/src/run_every.h: In constructor ‘crimson::RunEvery::RunEvery(D, int)’:
ceph/src/dmclock/support/src/run_every.h:48:7: error: class ‘crimson::RunEvery’ does not have any field named ‘body’
       body(_body)
       ^~~~
ceph/src/dmclock/support/src/run_every.h:48:12: error: ‘_body’ was not declared in this scope
       body(_body)
            ^~~~~
ceph/src/dmclock/support/src/run_every.cc: In member function ‘void crimson::RunEvery::run()’:
ceph/src/dmclock/support/src/run_every.cc:70:7: error: ‘body’ was not declared in this scope
       body();
       ^~~~
ceph/src/dmclock/support/src/run_every.cc:70:7: note: suggested alternative: ‘bool’
       body();
       ^~~~
       bool

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #14869 from wjwithagen/wip-wjw-cmake-uuid

cmake: fix the linked lib reference of unittest_rgw_crypto

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14900 from jupiturliu/organization_map

mailmap: add Alibaba into organization map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14895 from liewegas/wip-19778

kv: move 'bluestore-kv' hackery out of KeyValueDB into ceph-kvstore-tool

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14853 from liewegas/wip-aio

os/fs/aio: use small_vector for aio_t; clean up header location

Reviewed-by: Kefu Chai <kchai@redhat.com>

OSD: stop mgrc earlier in shutdown()

it's better this way, as MgrClient is more like an internal consumer
of the services offered by OSD, instead of service privider. it could
prevent future problem if we can stop it earlier in the shutdown
process.

see also: http://tracker.ceph.com/issues/19638
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14897 from tchaikov/wip-19814

mon/OSDMonitor: update creating_pgs using pending_creatings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14889 from joscollin/wip-test-write-operations-warning

test: c_write_operations.cc: silence warning from -Wsign-compare

Merge pull request #14888 from joscollin/wip-test-read-operations-warning

test: c_read_operations.cc: silence warning from -Wsign-compare

Merge pull request #14799 from xiaoxichen/optimiztic_pg_active

osd/PG.cc: Optimistic estimation on PG.last_active

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14790 from wjwithagen/wip-wjw-cleanup-unused-variable-v2

common: Fix unused variable references warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14330 from ivancich/wip-bring-in-dmclock

dmclock: initial commit of dmclock QoS library

Merge pull request #14896 from liewegas/wip-19638

osd: stop MgrClient callbacks on shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14899 from liewegas/wip-19433

ceph_test_rados_api_watch_notify: make LibRadosWatchNotify.Watch3Timeout tolerate thrashing

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14901 from liewegas/wip-19552

osdc/Objecter: fix pool dne corner case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14881 from tchaikov/wip-crosscompiling-armhf

cmake: enable cross-compilation of boost

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cls_rbd: default initialize snapshot namespace for legacy clients

Creating a snapshot on >=Kraken OSDs using <=Jewel clients can result
in an improperly initialized snapshot namespace. As a result, attempting
to remove the snapshot using a >=Kraken client will result in an -EINVAL
error.

Fixes: http://tracker.ceph.com/issues/19413
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

objclass-sdk: add override to whitelist the sdk class

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #14788 from liewegas/wip-jewel-x-rgw

qa/suites/jewel-x/point-to-point: don't scane for keys on second s3tests either

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdc/Objecter: fix pool dne corner case

We have a request that is queued before a pool exists, there is one
epoch where it exists, and then the pool disappears again.  The two maps
are processed at the same time.  For the first we set needs_resend, map to
an osd, and remove from the homeless sessin.  For the second, the pool
dne, we set osd back to -1, and send a map check request.  Finally,
handle_osd_maps scans need_resend, sees the pool dne, and removes from
need_resend.  The difference from the "usual" case is that we are neither
on the need_resend list nor on the homeless session.

Fix this by concluding immediately that the pool existed (briefly) and
then no longer exists.

Fixes: http://tracker.ceph.com/issues/19552
Signed-off-by: Sage Weil <sage@redhat.com>

doc:Adding Alibaba into organization map

Signed-off-by: James Liu <james.liu@alibaba-inc.com>

ceph_test_rados_api_watch_notify: make LibRadosWatchNotify.Watch3Timeout tolerate thrashing

If the PG moves we will reconnect and fail to time out.
Wait for longer so that we mask the effects of osd thrashing.

Fixes: http://tracker.ceph.com/issues/19433
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: cleanup pending_created_pgs after done with it

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: update creating_pgs using pending_creatings

* without this change, in OSDMonitor::update_pending_pgs(), creating_pgs
  is updated directly without lock, and the change will be overwritten
  with pending changes eventually. so we should update it using
  pending_creatings
* also, should acquire the lock in OSDMonitor::send_pg_creates() when
  reading creating_pgs_by_osd_epoch and createing_pgs.

Fixes: http://tracker.ceph.com/issues/19814
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14775 from Jing-Scott/wip-con-obj-bulkupload

rgw: add support container and object levels of swift bulkupload

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14892 from tchaikov/wip-kill-warnings

test: kill warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14750 from theanalyst/rgw/fix/frontend_array

rgw: civetweb don't go past the array index while calling mg_start

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14359 from Jing-Scott/remove-zg-dup-post

rgw multisite: remove the redundant post in OPT_ZONEGROUP_MODIFY

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14390 from shashalu/drop-unused-param

rgw: drop unused param "bucket" from select_bucket_placement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14839 from chardan/jfw-wip-halflife_atomic_t-ebirah

rgw: migrate atomic_t to std::atomic<> (ebirah)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14845 from cbodley/wip-rgw-qa-s3tests

qa/rgw: add cluster name to path when s3tests scans rgw log

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

osd: stop MgrClient callbacks on shutdown

Stop the MgrClient callbacks to report PG stats at the
start of shutdown() so that we don't get a callback
during/after we are done shutting down.

Protect the cb update with the MgrClient's lock so that
we don't race with MgrClient::send_report() itself.

Fixes: http://tracker.ceph.com/issues/19638
Signed-off-by: Sage Weil <sage@redhat.com>

kv: move 'bluestore-kv' hackery out of KeyValueDB into ceph-kvstore-tool

This avoids contaminating libkv with ObjectStore/BlueStore.  It also makes
the blustore kv startup slightly less weird (no need to skip the open
step).

Fixes: http://tracker.ceph.com/issues/19778
Signed-off-by: Sage Weil <sage@redhat.com>

test/librados: silence -Woverloaded-virtual warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: silence warning from -Wsign-compare

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: info command should indicate if parent is in trash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added trash_get API method to retrieve image status

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: extended parent info API to retrieve parent id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14770 from rzarzynski/wip-rgw-19754

rgw: fix RadosGW hang during multi-chunk upload of AWSv4.

librbd: get_parent_info should return cached parent name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14887 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: misc fixes

Merge pull request #14890 from xiexingguo/wip-fix-bluestore-typo

os/bluestore: fix typo(s/trasnaction/transaction/)

os/filestore/FileJournal.cc: fix unused warning

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

msg/async/net_handler.cc: fix unused reference for option

 - If no option value is to be supplied or returned,
     optval may be NULL
 - ignore priority options for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: pass compiler and linker down to zstd

so it can respect the setting specfied by user.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: print uint64_t in a platform neutral way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: pass by const reference to kill compiling error

i have following error when compiling with gcc-6 on armhf

ceph/src/crush/CrushCompiler.cc: In member function 'int
CrushCompiler::decompile(std::ostream&)':
ceph/src/crush/CrushCompiler.cc:462:45: error: invalid initialization of
non-const reference of type 'std::pair<const long unsigned int,
crush_choo
se_arg_map>&' from an rvalue of type 'std::pair<const long unsigned int,
crush_choose_arg_map>'
       int ret = decompile_choose_args(i, out);

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rbd: print uint64_t in a platform neutral way

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix typo(s/trasnaction/transaction/)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: c_write_operations.cc: silence warning from -Wsign-compare

The following warning appears during make:

ceph/src/googletest/googletest/include/gtest/gtest.h: In instantiation of ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 = int]’:
ceph/src/googletest/googletest/include/gtest/gtest.h:1421:23:   required from ‘static testing::AssertionResult testing::internal::EqHelper<lhs_is_null_literal>::Compare(const char*, const char*, const T1&, const T2&) [with T1 = long unsigned int; T2 = int; bool lhs_is_null_literal = false]’
ceph/src/test/librados/c_write_operations.cc:246:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {
       ~~~~^~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

test: silence warning from -Wsign-compare

The following warning appears during make:
In file included from ceph/src/test/librados/TestCase.h:9:0,
                 from ceph/src/test/librados/c_read_operations.cc:10:
ceph/src/googletest/googletest/include/gtest/gtest.h: In instantiation of ‘testing::AssertionResult testing::internal::CmpHelperEQ(const char*, const char*, const T1&, const T2&) [with T1 = int; T2 = long unsigned int]’:
ceph/src/googletest/googletest/include/gtest/gtest.h:1421:23:   required from ‘static testing::AssertionResult testing::internal::EqHelper<lhs_is_null_literal>::Compare(const char*, const char*, const T1&, const T2&) [with T1 = int; T2 = long unsigned int; bool lhs_is_null_literal = false]’
ceph/src/test/librados/c_read_operations.cc:685:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {
       ~~~~^~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #14883 from tchaikov/wip-mgr-misc

mgr: Misc. bug fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14878 from tchaikov/wip-19801

msg/simple: use my addr when setting sock priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14884 from tchaikov/wip-singleton-bluestore-tests

qa/suites/rados/singleton-bluestore: concat settings

Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/suites/rados/singleton-bluestore: concat settings

otherwise the bluestore tests will fail with failures like

qa/workunits/cephtool/test.sh:1343: test_mon_osd_pool:  ceph osd pool set ec_test allow_ec_overwrites true
Error EINVAL: pool must only be stored on bluestore for scrubbing to work: osd.1 uses filestore
qa/workunits/cephtool/test.sh:1343: test_mon_osd_pool:  return 1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14876 from liewegas/wip-bluestore-alloc

os/bluestore: roundoff bluefs allocs to bluefs_alloc_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14846 from dillaman/wip-rbd-cli

qa/workunits/rbd: increased trash deferment period

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mon: tweak mgr status text

Previously at startup we saw contradictory status
that had a "no active mgr" health message and then
a line that said "active: x".

Mitigate that a bit by indicating that a !available
active daemon is starting up.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix message refcounting

Let's be consistent and push the m->put down into
the handle_(M<class>* m) functions the way
we do in the rest of the codebase.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix deadlock in dispatch

Signed-off-by: John Spray <john.spray@redhat.com>