Merge pull request #14899 from liewegas/wip-19433

ceph_test_rados_api_watch_notify: make LibRadosWatchNotify.Watch3Timeout tolerate thrashing

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14901 from liewegas/wip-19552

osdc/Objecter: fix pool dne corner case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14881 from tchaikov/wip-crosscompiling-armhf

cmake: enable cross-compilation of boost

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14788 from liewegas/wip-jewel-x-rgw

qa/suites/jewel-x/point-to-point: don't scane for keys on second s3tests either

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdc/Objecter: fix pool dne corner case

We have a request that is queued before a pool exists, there is one
epoch where it exists, and then the pool disappears again.  The two maps
are processed at the same time.  For the first we set needs_resend, map to
an osd, and remove from the homeless sessin.  For the second, the pool
dne, we set osd back to -1, and send a map check request.  Finally,
handle_osd_maps scans need_resend, sees the pool dne, and removes from
need_resend.  The difference from the "usual" case is that we are neither
on the need_resend list nor on the homeless session.

Fix this by concluding immediately that the pool existed (briefly) and
then no longer exists.

Fixes: http://tracker.ceph.com/issues/19552
Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_watch_notify: make LibRadosWatchNotify.Watch3Timeout tolerate thrashing

If the PG moves we will reconnect and fail to time out.
Wait for longer so that we mask the effects of osd thrashing.

Fixes: http://tracker.ceph.com/issues/19433
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14775 from Jing-Scott/wip-con-obj-bulkupload

rgw: add support container and object levels of swift bulkupload

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14892 from tchaikov/wip-kill-warnings

test: kill warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14750 from theanalyst/rgw/fix/frontend_array

rgw: civetweb don't go past the array index while calling mg_start

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14359 from Jing-Scott/remove-zg-dup-post

rgw multisite: remove the redundant post in OPT_ZONEGROUP_MODIFY

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14390 from shashalu/drop-unused-param

rgw: drop unused param "bucket" from select_bucket_placement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14839 from chardan/jfw-wip-halflife_atomic_t-ebirah

rgw: migrate atomic_t to std::atomic<> (ebirah)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14845 from cbodley/wip-rgw-qa-s3tests

qa/rgw: add cluster name to path when s3tests scans rgw log

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

test/librados: silence -Woverloaded-virtual warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: silence warning from -Wsign-compare

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14770 from rzarzynski/wip-rgw-19754

rgw: fix RadosGW hang during multi-chunk upload of AWSv4.

Merge pull request #14887 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: misc fixes

Merge pull request #14890 from xiexingguo/wip-fix-bluestore-typo

os/bluestore: fix typo(s/trasnaction/transaction/)

cmake: pass compiler and linker down to zstd

so it can respect the setting specfied by user.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: print uint64_t in a platform neutral way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: pass by const reference to kill compiling error

i have following error when compiling with gcc-6 on armhf

ceph/src/crush/CrushCompiler.cc: In member function 'int
CrushCompiler::decompile(std::ostream&)':
ceph/src/crush/CrushCompiler.cc:462:45: error: invalid initialization of
non-const reference of type 'std::pair<const long unsigned int,
crush_choo
se_arg_map>&' from an rvalue of type 'std::pair<const long unsigned int,
crush_choose_arg_map>'
       int ret = decompile_choose_args(i, out);

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rbd: print uint64_t in a platform neutral way

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix typo(s/trasnaction/transaction/)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14883 from tchaikov/wip-mgr-misc

mgr: Misc. bug fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14878 from tchaikov/wip-19801

msg/simple: use my addr when setting sock priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14884 from tchaikov/wip-singleton-bluestore-tests

qa/suites/rados/singleton-bluestore: concat settings

Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/suites/rados/singleton-bluestore: concat settings

otherwise the bluestore tests will fail with failures like

qa/workunits/cephtool/test.sh:1343: test_mon_osd_pool:  ceph osd pool set ec_test allow_ec_overwrites true
Error EINVAL: pool must only be stored on bluestore for scrubbing to work: osd.1 uses filestore
qa/workunits/cephtool/test.sh:1343: test_mon_osd_pool:  return 1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14876 from liewegas/wip-bluestore-alloc

os/bluestore: roundoff bluefs allocs to bluefs_alloc_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14846 from dillaman/wip-rbd-cli

qa/workunits/rbd: increased trash deferment period

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mon: tweak mgr status text

Previously at startup we saw contradictory status
that had a "no active mgr" health message and then
a line that said "active: x".

Mitigate that a bit by indicating that a !available
active daemon is starting up.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix message refcounting

Let's be consistent and push the m->put down into
the handle_(M<class>* m) functions the way
we do in the rest of the codebase.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix deadlock in dispatch

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: get fresh maps on second load of Mgr

I think we got away with this when Client::init was
re-initialising the MonClient, but now that (bogus)
stuff isn't happening any more as Client doesn't
init/teardown monc/objecter itself.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: fix mgr caps for MDS daemons

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: improve handle_command

Run the python module calls in a finisher so that
they don't block the daemonserver lock and so that
they can call back into mgr stuff if they need to.

Fix passing through commands to python modules, this
was giving EINVAL because only things with a MgrCommand
were getting let in.

Also fix get_command_descriptions, which was not
including the output of the formatter in the response.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: don't enqueue periodic digest if one is already waiting

This was causing mons to send far more digest messages than
should have been sent.  Could have been responsible for
reports of very high CPU consumption on the mgr daemon.

Fixes: http://tracker.ceph.com/issues/18994
Signed-off-by: John Spray <john.spray@redhat.com>

common/finisher: use distinct lock names

Two Finishers should not be considered equivalent
for lockdep purposes: for example in mgr we have
our in-mgr Finisher, and then any finishers that
might be in libraries called by modules.

Signed-off-by: John Spray <john.spray@redhat.com>

cmake: enable crosscompile of boost

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14837 from wjwithagen/wip-wjw-freebsd-ceph-disk-mpath

ceph-disk: ceph-disk on FreeBSD should not use mpath-code

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: ceph-disk on FreeBSD should not use mpath-code

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

msg/async: refactor NetHandler::set_priority() to sync with msg/simple

so it's sync'ed with Pipe::set_socket_options()

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple: use my addr when setting sock priority

actually we always fails to set ToS of listening
socket. 6f1037e22c2a304795895498cdc955e0ef80f8e3 reveals this issue.

Fixes: http://tracker.ceph.com/issues/19801
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14872 from tchaikov/wip-19770

qa/erasure-code: override min_size to 2

Reviewed-by: Sage Weil <sage@redhat.com>

qa/erasure-code: override min_size to 2

so isa(k=2,m=1) can survive with 1 down OSD.

Fixes: http://tracker.ceph.com/issues/19770
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/rados: add optional setting of "min_size"

this setting only affects the newly created pool

Fixes: http://tracker.ceph.com/issues/19770
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14758 from xiexingguo/fix-can_mark_out-output

mon/OSDMonitor: fix output func name in can_mark_out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14847 from smithfarm/wip-19797

tests: rados: move cephtool.yaml to new singleton/bluestore subsuite

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14849 from tchaikov/wip-19787

mon/OSDMonitor: do not alter the "created" epoch of a pg

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x: limit fs matrix

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14161 from jcsp/wip-mgr-legacy-mds

mgr: fix metadata handling from old MDS daemons

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14706 from SUSE/wip-fix-19629

mgr: fix crash on set_config from python module with insufficient caps

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14855 from tchaikov/wip-increase-lec

mon/OSDMonitor: increase last_epoch_clean's lower bound if possible

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14859 from liewegas/wip-nonnull

librados: fix rados_pool_list when buf is null

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14860 from liupan1111/wip-fix-journal-sleep-performance

os/filestore: use new sleep strategy when io_submit gets EAGAIN.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14862 from tchaikov/wip-templatize-intarith

include/intarith: templatize ctz/clz/cbits helpers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14863 from tchaikov/wip-kill-clang-warnings

osd/OSDMap: should update input param if osd dne

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14864 from renhwztetecs/renhw-wip-mgr-key

mgr: add mgr allow * to client.admin

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: roundoff bluefs allocs to bluefs_alloc_size

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #14832 from dillaman/wip-rbd-mirror-deletion

rbd-mirror: image deletions should be handled by assigned instance

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14842 from cbodley/wip-rgw-admin-suite

qa/rgw: fix assertions in radosgw_admin task

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14826 from tchaikov/wip-prime-pg-temp

mon/OSDMonitor: skip prime_pg_temp if mapping is prior to osdmap

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14818 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14817 from TsaiJin/wip-set-boolean-variable-with-boolean-value

osd: ReplicatedBackend::prep_push() remove redundant variable assignments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14835 from tchaikov/wip-19743

mgr: do shutdown using finisher so we can do it in the right order

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14838 from liewegas/wip-mgr-dirs

debian/ceph-base.dirs: create bootstrap-mgr dirs

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #14384 from hrchu/radosgw-admin-key

doc: rgw: Rewrite the key management

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/kraken-x/stress-split-erasure-code: fix

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/kraken-x/parallel: fix

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/parallel: remove stray kraken.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14848 from dillaman/wip-19798

test: remove hard-coded image name from RBD metadata test

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13315 from dang/wip-build-no-nss

cmake: Allow tests to build without NSS

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14367 from mslovy/wip-fix-infinitloops

os/filestore: fix infinit loops in fiemap()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14391 from LiumxNL/wip-170407

mon/PGMonitor: fix wrongly report "pg stuck in inactive"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14675 from ifed01/wip-bluestore-nolock

os/bluestore: eliminate some excessive stuff

Reviewed-by: Sage Weil <sage@redhat.com>

tests: rados: move cephtool.yaml to new singleton/bluestore subsuite

cephtool.yaml is bluestore-only, yet it was in singleton/ which runs against a
generalized objectstore matrix.

Fixes: http://tracker.ceph.com/issues/19797
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #14475 from liewegas/wip-message-compat-cruft

messages: remove compat cruft

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: move TransContext finalization out of osr_lock to avoid potential contention.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: get rid off Onode::flush_txn set.
It looks like flushing_count counter is enough.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

mgr: add mgr allow * to client.admin

Signed-off-by: huanwen ren ren.huanwen@zte.com.cn

rgw: migrate atomic_t to std::atomic<>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

common/hobject: use string::append() to avoid temp object

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: should update input param if osd dne

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/obj_bencher: silence divide by zero warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_async_driver: silence clang static analyzer warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: use std::atomic<uint8_t> for min_alloc_size_order

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/intarith: templatize ctz/clz/cbits helpers

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: use new sleep strategy when io_submit gets EAGAIN.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

libcephfs: handle path==null to ceph_get_osd_crush_location

Signed-off-by: Sage Weil <sage@redhat.com>

librados: fix rados_pool_list when buf is null

This was the original intent, but buf==NULL wasn't documented
and only half-way implemented.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14840 from drunkard/master

doc: style fix for doc/cephfs/client-config-ref.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: typo fixes in doc/cephfs/client-config-ref.rst

fixed:
  Reference symbol, i.e. s/`/``/
  Hyperlink typo at ``client_trace``

Signed-off-by: Drunkard Zhang <gongfan193@gmail.com>

Merge pull request #14302 from liewegas/wip-19452

osdc/Objecter: handle command target that goes down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14850 from tchaikov/wip-cmake

cmake: add "container" to required boost components

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/OSDMonitor: increase last_epoch_clean's lower bound if possible

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: drop unused param "bucket" from select_bucket_placement

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #14794 from joscollin/wip-warning-simplespin

test: fixing assert that creates warning: comparison between signed and unsigned integer expressions

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/rgw: fix assertions in radosgw_admin task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mon/OSDMonitor: use emplace() to avoid creating temporary elements

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: do not alter the "created" epoch of a pg

Fixes: http://tracker.ceph.com/issues/19787
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "container" to required boost components

it is used by our denc

Signed-off-by: Kefu Chai <kchai@redhat.com>