tests - ceph-ansible vars additions
added symlinks for distros

Fixes http://tracker.ceph.com/issues/21822
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit f30e9a5e6bd47129f8a35c1f9067216cfe8a6a70)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #18210 from shangfufei/wip-async-fix

msg/async: fix bug of data type conversion when uint64_t -> int -> uint64_t

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #18246 from liewegas/wip-seesaw-warning

qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18270 from dillaman/wip-21391

cls/journal: possible infinite loop within tag_list class method

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

cls/journal: fixed possible infinite loop which could kill the OSD

Fixes: http://tracker.ceph.com/issues/21771
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: ceph_test_cls_journal was dropped when converting to cmake

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18258 from shinobu-x/sk-osdmonitor_move_bufferlist_before_use

mon/OSDMonitor: cleanup: move bufferlist before use

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #18235 from tchaikov/wip-qa-backfilling

qa: s/backfill/backfilling/

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18222 from adamemerson/wip-don't-fix-what-ain't-broke

config: Remove _get_val

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: Move bufferlist before use

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #17824 from shashalu/fix-lc-expiration

rgw:lc fix expiration time

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #18102 from MI-OSiRIS/mgr_influx_dbcheck

Modify mgr-influx module database check to not require admin privileges

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18104 from linuxbox2/rgw-rel-lock-earlyx

rgw: release cls lock if taken in RGWCompleteMultipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18175 from yaozongyou/fix-radosgw-admin-command-arg-bug

rgw: fix command argument error for radosgw-admin

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18193 from cbodley/wip-rgw-pool-application-jewel

qa/rgw: ignore errors from 'pool application enable'

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #18225 from adamemerson/wip-retract-assertions

rgw: Remove assertions in IAM Policy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17420 from tchaikov/wip-19705

arch/arm: set ceph_arch_aarch64_crc32 only if the build host supports crc32cx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18242 from tchaikov/wip-21758

crushtool: print error message to stderr not dout(1)

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/mon-seesaw: whitelist MON_DOWN

Mgr can get marked down when mon weirdness is happening.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18137 from cbodley/wip-21685

rgw: RGWUser::init no longer overwrites user_id

Merge pull request #18233 from gmayyyha/ceph-fs-status-21752

mgr/status: fix ceph fs status returns error

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crushtool: print error message to stderr not dout(1)

in hope to fix the mysterious test failure where cli/crushtool/build.t
prints nothing when error message is expected.

Fixes: http://tracker.ceph.com/issues/21758
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: fix bug of data type conversion in do_request()

Signed-off-by: shangfufei <shangfufei@inspur.com>

Merge pull request #18230 from ceph/liewegas-patch-1

.githubmap: Update

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: s/backfill/backfilling/

it's renamed "backfilling" in 4015343f .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18203 from majianpeng/bluestore-wake-kvfinathread

os/bluestore/BlueStore: ASAP wake up _kv_finalize_thread.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #18201 from gregsfortytwo/wip-21737-shutdown-assert

common: by default, do not assert on leaks in the shared_cache code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18218 from tchaikov/wip-21609

mon/PGMap: let pg_string_state() return boost::optional<>

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18213 from tchaikov/wip-log-on-osx

log: fix build on osx

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18216 from tchaikov/wip-cmake-cleanup

cmake: set supported language the right way

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

mgr/status: fix ceph fs status returns error

Fixes: http://tracker.ceph.com/issues/21752
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #18228 from ktdreyer/doc-mds-blacklist-interval-manual

doc: rm stray ")" character from mds config ref

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17900 from markhpc/wip-filestore-rocksdb-compaction-readahead-master

common/options.cc: Set Filestore rocksdb compaction readahead option.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Update .githubmap

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #18223 into master

* refs/pull/18223/head:
githubmap: define merge conflict resolution

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rm stray ")" character from mds config ref

I accidentally introduced this in
4fb89a63172d2dc6ce8f4702fe90904f9a8fd7bd

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rgw: Remove assertions in IAM Policy

A couple of them could be triggered by user input.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

githubmap: define merge conflict resolution

Use "union" merge to automatically resolve trivial append merge conflicts from
using ptl-tool.py.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17882 from ZVampirEM77/wip-rgw-encryption

rgw: clean up and fix some bugs for encryption

config: Remove _get_val

I added this originally to prevent a deadlock when reading a new-style
configuration variable from a called observer. I didn't realize at the
time that the config mutex was recursive, so this change was
superfluous.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17759 from liewegas/wip-21407

osd: fix waiting_for_peered vs flushing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18208 from jan--f/mgr-prometheus-fix-undef-metrics

pybind/mgr/prometheus: fix metric type undef -> untyped

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #18136 from dillaman/wip-21561

rbd-mirror: primary image should register in remote, non-primary image's journal

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18158 from dillaman/wip-21711

librbd: refresh image after applying new metadata

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: refresh image after applying new/removing old metadata

Fixes: http://tracker.ceph.com/issues/21711
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/PGMap: let pg_string_state() return boost::optional<>

better to be specific, so we don't run into the same problem even we
are using the MSB of uint64_t for a pg state bit in future. we could,
however use uint64_t(-1) to indicate the pg_string_state()'s failure to
parse the state string, because pg_string_state() only translate a
single state a time. but it's always better to be explicit than
implicit.

Fixes: http://tracker.ceph.com/issues/21609
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: use uint64_t for pg state

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: set supported language the right way

the second param of enable_launage() is not used, we should call it
multiple times to enable more than one language. switch to project()
command for simplicity.

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: fix build on osx

we can not assume that the the `rep` type is identical to `time_t` and
`susecond_t`, on osx they are `int`, not `int64_t`. so cast they as
necessary.

this fixes the error and warning of

LogClock.h:112:7: error: non-constant-expression cannot be narrowed from type 'rep' (aka 'long long') to '__darwin_suseconds_t' (aka 'int') in initializer list [-Wc++11-narrowing]
             std::chrono::duration_cast<std::chrono::microseconds>(
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/kefu/dev/ceph/src/log/LogClock.h:112:7: note: insert an explicit cast to silence this issue
             std::chrono::duration_cast<std::chrono::microseconds>(
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LogClock.h:139:46: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
                      bdt.tm_hour, bdt.tm_min, bdt.tm_sec, tv.tv_usec / 1000);
                                                           ^~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: fix metric type undef -> untyped

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #18205 from yaozongyou/fix-bluestore-cache-kv-max-type

doc: correct wrong bluestore config types

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix bluestore config type error.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18161 from Liuchang0812/wip-bluestore-ebusy

os/bluestore: do not core dump when we try to open kvstore twice

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18149 from tchaikov/wip-cleanups

common, osd: various cleanups

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18177 from tchaikov/wip-kv-kill-warning

kv: unify {create_and_,}open() methods

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: removed duplicate client registration logic

The client will now be registered before the bootstrap state machine
is invoked.

Fixes: http://tracker.ceph.com/issues/21561
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: local primary images should still register to remote images

Fixes: http://tracker.ceph.com/issues/21561
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18195 from ktdreyer/doc-mds-blacklist-interval-manual

doc: "mds blacklist interval" vs manually blacklisting

common: by default, do not assert on leaks in the shared_cache code

Update the standard qa suite cluster configs so that we continue
asserting in our nightlies, but users don't hit this.

Fixes: http://tracker.ceph.com/issues/21737
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: fix waiting_for_peered vs flushing

on_flush() requeues waiting_for_peered, but we flush twice on the
primary during peering, and we don't want to requeue the first one
(when we have the master pg log merged).

Fix by moving waiting_for_peered to waiting_for_flush if we aren't
already flush on _activate_committed.  If we get an op and are
peered but not flushed, queue ops there.  (We can simplify this
check a bit since pgbackend inactive message handling doesn't care
about flushed or not flushed.)  When flushed, we requeue
waiting_for_flush.

The waiting_for_flush, waiting_for_peered, and waiting_for_active
lists are all mutually exclusive, so this mostly serves to
clarify what we are waiting for (not to keep items separate). And
it means that on_flushed() will only requeue things that were
waiting for it specifically.

Fixes: http://tracker.ceph.com/issues/21407
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18188 from jschmid1/wip-man-osd-config-ref-syntax

doc: fixes syntax in osd-config-ref

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix syntax in rst

* Fix syntax in osd max object size section.

Fixes: http://tracker.ceph.com/issues/21733
Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc: "mds blacklist interval" vs manually blacklisting

The "mds blacklist interval" setting has no effect on the time that
the "ceph osd blacklist" command will use by default. Clarify this in
the docs.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

os/bluestore/BlueStore: ASAP wake up _kv_finalize_thread.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

qa/rgw: ignore errors from 'pool application enable'

Fixes: http://tracker.ceph.com/issues/21715
Signed-off-by: Casey Bodley <cbodley@redhat.com>

kv: unify {create_and_,}open() methods

this silences the warning of "-Woverloaded-virtual" where create() and
create_and_open() from KeyValueDB are hidden from the ones defined in
derived classes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18190 from JustL/master

msg/async/rdma: unnecessary reinitiliazation of an iterator

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18058 from Yan-waller/wip-walle-0927pgstatexpandstate

osd/PG:  extend pg state bits to fix pg ls commands error

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/rdma: unnecessary reinitiliazation of an iterator

current_buffer iterator does not need to be reinitalized as no changes
occur between the initialization and iteration steps.

Signed-off-by: Justinas Lingys jlingys@connect.ust.hk

Merge pull request #18178 from joscollin/wip-radosgw-admin-reshard-fix

rgw: update usage() with status
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18180 from oritwas/wip-rgw-21617

rgw: add reshard commands

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: Fixed typos and status in help.t

Fixed typos and status in help.t

Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: add reshard commands to the man pages

Fixes: http://tracker.ceph.com/issues/21617
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18162 from tchaikov/wip-ceph-disk-retry-on-dev-not-ready

ceph-disk: retry on OSError

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: update usage() with status

Fixed the missing 'status' in usage().

Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: fix command argument error for radosgw-admin.

Fixes: http://tracker.ceph.com/issues/21723
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

rgw:lc: using midnight time to check obj expiration

Amazon S3 calculates the expiration time by adding the number of days specified in the rule to the object creation time and rounding the resulting time to the next day midnight.
So rgw should using the rounding down midnight time to compare with the object creation time to check obj expiration.

Fixes: http://tracker.ceph.com/issues/21533
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

osd/PG: extend pg state to 64 bits

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

os/bluestore: drop unused class FreelistManager

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #18157 from batrick/ptl-tool-branch-checkout

ptl-tool: checkout branch after creation

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17708 from liewegas/wip-pg

osd: initial minimal efforts to clean up PG interface

Merge pull request #18125 from cernceph/dvanders_conf

mon: mark mon_allow_pool_delete as observed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18144 from hjwsm1989/wip-21425-part2

qa/suites/rest/basic/tasks/rest_test: whiltelist OSD_DOWN

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18159 from liewegas/wip-21712-b

os: fix 0-length zero semantics, add tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: 0-length zero should not change object size

Fixes: http://tracker.ceph.com/issues/21712
Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: make 0-length zero avoid touching object length

Fixes: http://tracker.ceph.com/issues/21712
Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: 0-length zero does not change length of object

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: s/g_ceph_context/cct/

use this->cct if possible, yet another step to remove the dependency on
global `g_ceph_context`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: return const reference instead of a copy

avoid creating temporary Option::value_t instances if possible

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: use boost::get<> instead of visitor

simpler this way with caveat of less developer friendly assertion
failure.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd,common: avoid casting useless casts

no need to cast a variable of type `Foo` to `Foo`, for example, 2.000 is
of type `double`, so (double)2.000 does not make sense.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/types.h: silence "-Wdouble-promotion" warnings

avoid implicit promotion from float to double.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: s/_get_val/_get_val_generic/

avoid distinguish two methods with their return types and non-templated
versus templated. also, this follows the convention of md_config_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18141 from adamemerson/wip-log-time

log: Use the coarse real time clock in log timestamps

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17976 from adamemerson/wip-warning-sign

mgr,osd,kv: Fix various warnings for Clang and GCC7

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: retry on OSError

we are likely to
1) create partition, for instance, sdc1
2) partprobe sdc
3) udevadm settle
4) check the device by its path: /dev/sdc1

but there is chance that the uevent sent from kernel fails to reach udev
before we call "udevadm", hence "/dev/sdc1" does not exist even after
"udevadm settle" returns. so we retry in case of OSError here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: factor out the retry logic into a decorator

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not core dump when we try to open kvstore twice

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

log: Mark timestamps for coarseness and print appropriately

Have each timestamp know whether it's coarse or fine. Print six figures
for fine stamps and three for coarse.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

log: Add log_coarse_timestamps option

And set up an observer in CephContext to coarsen/refine time stamps as
requested.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>