build: Use .S suffix for ppc64le assembly files

The assembly file name suffix was changed from .S to .s,
which caused a build break on ppc64le architecture.

To fix this issue, the suffix will be changed back to .S
for ppc64le assembly files only.  The x86 assembly files
can retain the .s suffix.

Fixes: http://tracker.ceph.com/issues/20106
Signed-off-by: Andrew Solomon <asolomon@us.ibm.com>

Merge pull request #15359 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15306 from mikulely/fix-test_multi

rgw: fix test_multi.py default config file path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15294 from liewegas/wip-osd-tree

osd: 'osd tree in|out|up|down' to filter tree results

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15225 from smithfarm/wip-s390-rgw

build/ops: conditionalize rgw Beast frontend so it isn't built on s390x architecture

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: add up|down|in|out filters to 'osd tree'

For example, 'ceph osd tree down' will show *just* down OSDs and their
ancestors.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: link radosgw with boost libraries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15355 from joscollin/wip-cleanup-testdenc-warning

test: test_denc.cc: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: test_denc.cc: silence warning from -Wsign-compare

The following warning appears during make:
ceph/src/test/test_denc.cc:669:5:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1530:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
ceph/src/googletest/googletest/include/gtest/gtest.h:1510:7:
   if (val1 op val2) {\

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15058 from yuyuyu101/wip-rocksdb-opt

kv/rocksdb: supports SliceParts interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15068 from liewegas/wip-osdmap-require

osd/OSDMap: replace require_*_osds flags with a single require_osd_release field

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15313 from Linbing/wip-pg-scrub-code-optimize

mon: it's no need to get pg action_primary osd twice in pg scrub

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: fix osd metadata update ordering

Do not do both.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: replace all require_*_osds flag checks with a require_osd_release comparison

Signed-off-by: Sage Weil <sage@redhat.com>

osd: replace require_*_osds flags with require_osd_release field

- OSDMap encode and decode translate between the flags and int
representations.
- OSDMap::Incremental only does decode; we do not expect to ever encode
an incremental osdmap for an old osd that sets any of these flags.
- the 'osd set' command still lets you set the jewel and kraken flags,
but not luminous.
- OSDMap::apply_incremental handles the conversion of legacy require flags
to the new field if the jewel or kraken flags have to be set before
starting the osd upgrade.
- clear out the legacy flags when we make the luminous transition only;
until then we keep using the old flag in the encoded and decoded version
(although the require_osd_release field will be accurate in memory in all
cases).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: new 'osd require-osd-release' command

Note that we only allow setting this to 'luminous'.  If the 'jewel' or 'kraken'
flags need to be set that can be done through the legacy interface.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add require_osd_release numeric field, CEPH_RELEASE_*

Instead of eating up a flag for each release, which is super-awkward and
annoying, just keep a numeric release version min.

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: delete excidentail reintroduced line.

- This target got reintroduced by accident after the creation
  of ceph-common in commit:
    https://github.com/ceph/ceph/commit/046b2bde6d93992c36d4943c4f7859993f034ca

  Detection was trigger by refering to libresolv, and results in a
  linking error.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

build/ops: deb: no rgw Beast frontend on s390x

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: no rgw Beast frontend on s390x

Since the Beast frontend uses boost::context which is not supported on
s390x.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: rpm: reverse s390 bcond conditional block

I'm not a fan of "if NOT x - then - else" blocks.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: build boost::context and coroutine only with rgw Beast frontend

boost::context is currently (1.63) unsupported for s390x and anyway
it makes sense to conditionalize Boost components so they are only
built with the Ceph components that need them (like is already being
done for mgr).

Fixes: http://tracker.ceph.com/issues/20048
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15324 from yuyuyu101/wip-20093

msg/async: avoid requeue racing with handle_write

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15325 from liewegas/wip-redirect

osd,librados: add manifest, redirect

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15326 from xiexingguo/fix-kvtool-tip

tools/ceph_kvstore_tool: add "bluestore-kv" to usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15337 from wjwithagen/wip-wjw-freebsd-fix-includes

common: include: Redo some includes for FreeBSD

Merge pull request #15338 from trociny/wip-get_process_name

common: get_process_name: use getprogname on bsd systems

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15345 from badone/wip-tracker-19900-wait_auth_rotating-crash-2

osd: Return early on shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Return early on shutdown

eb5c02d was missing a return statement, rectify that.

Fixes: http://tracker.ceph.com/issues/19900
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15336 from wjwithagen/wjw-wip-iso_8601

common/iso_8601.cc: Make return expression Clang compatible

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code/isa/ErasureCodePluginIsa.cc: fixup includes

 - Buffer.h is needed to prevent Clang seriously complaining
   about missing and wrong forward declarations of
ceph::buffer:ptr
   including buffer_fwd.h does not help.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

build/include: Redo som includes for FreeBSD

- During include cleanup just a too bit much was removed for FreeBSD
  to get things compiled.
  This redoes some of the includes.

Tracker: http://tracker.ceph.com/issues/19883
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15055 from mogeb/wip-with-instrument-functions

cmake: Add -finstrument-functions flag to OSD code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15319 from liewegas/wip-upmap

mon,osd/OSDMap: a couple pg-upmap fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15292 from liewegas/wip-map-cache

osd: reduce map cache size

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15296 from liewegas/wip-fix-at-end

qa/tasks/repair_test: unset flags we set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15297 from liewegas/wip-19595

mgr/MgrStandby: prevent use-after-free on just-shut-down Mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14284 from tchaikov/wip-19439

qa/tasks/rebuild_mondb: grant "mgr:allow *" to client.admin

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15284 from majianpeng/bluestore-batch-throotle

os/bluestore: batch throttle

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15302 from xiexingguo/wip-fix-reg-leak

common/ceph_context: fix leak of registered commands on exit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15320 from liewegas/wip-peon-mapping

mon/OSDMonitor: cancel mapping job from update_from_paxos

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/rebuild_mondb: update to address ceph-mgr changes

- revive ceph-mgr after updating the keyring cap
- grant "mgr:allow *" to client.admin
- minor refactors

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: get_process_name: use getprogname on bsd systems

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

common/iso_8601.cc: Make return expression Clang compatible

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools/ceph_kvstore_tool: add "bluestore-kv" to usage

We are soon going to use BlueStore as the default OS, and
hence "bluestore-kv" shall be the preferred option.

By applying this patch, the help message shall be slightly
nicer to newcomers to this tool now.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph_test_rados_api_tier: skip redirect tests if not luminous

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15111 from Liuchang0812/wip-19885-mgr-help

ceph: introduce "tell x help" subcommand

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-luminous: redirects require luminous

...so skip the at-end variant of d-luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-extensible_tier-redirect' of git://github.com/myoungwon/ceph into wip-redirect

rgw: fix test_multi.py default config file path

$ python test_multi.py --num-zonegroups=2 --num-zones 2 --gateways-per-zone 1
Traceback (most recent call last):
  File "test_multi.py", line 315, in <module>
      init(True)
        File "test_multi.py", line 162, in init
    path = tpath('test_multi.conf')
    NameError: global name 'tpath' is not defined

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

msg/async: keep _has_next_outgoing calling under write_lock

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: avoid requeue racing with handle_write

when the thread is calling AsyncConnection::handle_write, another thread may
replace it and requeue all messages. Because we remove the write_lock
protection for handle_write caller, it may happen sent racing with out_q

Fix: http://tracker.ceph.com/issues/20093

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15289 from ifed01/wip-bluestore-fix-bmap-assert

os/bluestore: fix BitMapAllocator assert on out-of-bound hint value

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15288 from yuyuyu101/wip-20062

msg/async: remove false alert "assert"

Merge pull request #15214 from gregsfortytwo/wip-19931-snaptrim-pgref

osd: don't leak pgrefs or reservations in SnapTrimmer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: do not default-abort on leaked pg refs

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: shutdown our reserver_finisher earlier in the process

This finisher thread has a lot of callbacks which can hold PGRefs. Make
sure we drain them out before checking that all the PGs have finished
and have no outstanding references.

Moving this should be safe; we've already stopped the op thread et al
and the only things still running are the OSDService's objecter_finisher,
recovery_request_timer, and snap_sleep_timer (which has definitely been emptied
by the time we get here as it's synchronously cleared out on PG shutdown).

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15316 from linuxbox2/wip-limitcheck-fixes

rgw-admin: fix bucket limit check argparse, div(0)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: fix bucket limit check argparse, div(0)

1. normalize arg parsing for "bucket limit check"
1.1 s/buckets/bucket/

2. avoid dividing by num_shards when it is 0

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mon/OSDMonitor: cancel mapping job from update_from_paxos

On the leader we cancel the mapping job in encode_pending.  On a peon,
we don't cancel it at all!  It is surprising this didn't already cause
problems, but with the PGtempMap is pretty reliably crashes with a
largish map.

Fixes: http://tracker.ceph.com/issues/20067
Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: some better pg-upmap debug output

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: allow upmap records for deleted pools to be removed

Otherwise the offline osdmaptool --upmap-cleanup can't do its thing.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15304 from Linbing/modify-note-word-error

mon: Fixed typo in function comment blocks and in other comments

os/bluestore: fix BitMapAllocator assert on out-of-bound hint value

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15256 from fangyuxiangGL/bucket-stats

rgw: display more info when using  radosgw-admin bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15221 from fangyuxiangGL/set-placement-rule

rgw: set placement rule properly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15287 from ktdreyer/doc-ceph-fs-settings

doc: describe CephFS max_file_size

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #15219 from fangyuxiangGL/extra_pool

rgw: use get_data_extra_pool() when get extra pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14341 from theanalyst/cleanup/rgw/str-find

rgw_common: use string::npos for the results of str.find

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14746 from wjwithagen/wip-wjw-reduce-noise-2

src/ceph.in: Use env(CEPH_DEV) to suppress noise from ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15247 from liewegas/wip-upmap-save

osdmaptool: require --upmap-save before modifying input osdmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15234 from trociny/wip-19832

librbd: potential read IO hang when image is flattened

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15239 from trociny/wip-19962

librbd: discard related IO should skip op if object non-existent

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15293 from linuxbox2/wip-rgwfile-buckattr

rgw_file: store bucket uxattrs on the bucket

mon: it's no need to get pg action_primary osd twice in pg scrub

Signed-off-by: linbing <linbing@t2cloud.net>

Merge pull request #15290 from idryomov/wip-krbd-unmap-straw

qa/suites/krbd: unmap subsuite needs straw buckets

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: describe differences with "mds set" vs "fs set"

Thanks to John Spray <jspray@redhat.com> on ceph-users for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: describe CephFS max_file_size

Add a description of max_file_size to the CephFS admin docs.

Thanks to John Spray <jspray@redhat.com> on ceph-users for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

os/bluestore/BlueStore: Only call once notify_all.

All txcs of deferred_running have the same osr, so only call once
notify_call.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: avoid iterating to get ios/cost in _kv_sync_thread.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: batch handle Throttle.put.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

./qa/workunits/ceph-helpers.sh: Do not trace kill_daemon

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

vstart:Suppress trivial logging info during devlopment

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/ceph.in: Use env(CEPH_DEV) to suppress noise from ceph

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Modify note error

Signed-off-by: linbing linbing@t2cloud.net

tool/ceph: support target mon in tell help subcommand

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

common/ceph_context: fix leak of registered commands on exit

By unregister them properly.
Also rearrange commands order to keep pace with register process.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: Reset() the snaptrimmer on shutdown

We were failing to exit various wait states which held PGRefs. Error!

Fixes: http://tracker.ceph.com/issues/19931
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14658 from chardan/jfw-wip-halflife_atomic_t-baragon

librados,libradosstriper,test: migrate atomic_t to std::atomic (baragon)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15227 from tchaikov/wip-20051

mon/MonClient: cancel pending commands on shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15249 from Alessandro-Barbieri/patch-2

yasm-wrapper: filter -pthread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15268 from ifed01/wip-bluestore-false-assert

os/bluestore: fix false assert in IOContext::aio_wake

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15272 from tchaikov/wip-mon-cleanups

mon: cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

osd: reduce size of osdmap cache, messages

On large clusters, these large caches can be problematic (as maps get big).
We've seen good results with extremely small caches (10s of maps).  Make
a more modest reduction.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-config.sh: fix test to isolate cases

The third test (increasing osd_map_max_advance)
was triggering a warning from the 4th case (which
it didn't before).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fadvise hint WILL_NEED when reading encoded osdmaps

This way will ensure we cache data for recent osdmaps if we need to for
the benefit of laggy clients... even if (in bluestore's case)
bluestore_default_buffered_reads = false (it's true by default).  This
should mitigate any tail latency/work even if the osdmap cache size is too
small.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: instrument osdmap bl cache hits and misses

Add perfcounters so we can see whether we are missing osdmaps in the
cache.  This will let us tell whether, given a workload or environment,
our osdmap cache might be too small.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not bother with misdirected op diagnosis by default

We enable osd_debug_misdirected_ops in QA, but this is wasted effort on
a production cluster.  In particular, it means that a idle client that
sends an op to the wrong OSD based on an old map will require that OSD to
load that old map into memory to decide whether to print a warning... all
on the off-chance that the client is buggy.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove osd_enxio_on_misdirected_op option

There is no good reason anyone would want this turned on.

Introduced 923e7f5ce5ed437af15e178299a61029ff48e4a2 (post-kraken), but
backported to kraken and jewel (10.2.6).

Signed-off-by: Sage Weil <sage@redhat.com>