rocksdb: sync with upstream

since
https://github.com/facebook/rocksdb/pull/3862 has been merged by
upstream.

Fixes: http://tracker.ceph.com/issues/23653
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22232 from batrick/cephfs-release-notes-mimic

doc/releases: update CephFS mimic notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22195 from neha-ojha/wip-minor-fixes

qa/suites/rados: minor fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/releases: update CephFS mimic notes

Fixes: http://tracker.ceph.com/issues/23775
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22034 from mogeb/iostat-doc

doc: add documentation for iostat

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22112 from szeestraten/fix_24199

common: fix typo in rados bench write JSON output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22093 from liewegas/wip-24162

msg/async/Event: clear time_events on shutdown

Reviewed-by: Hoamai Wang <haomai@xsky.com>

Merge pull request #21491 from majianpeng/kerneldevice

os/bluestore/KernelDevice: misc cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22216 from votdev/fix_table_pagination_style

mgr/dashboard: Fix some datatable CSS issues

Approved-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Fix some datatable CSS issues

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #22214 from joscollin/wip-ceph-fuse-typo

doc: Fix typo in ceph-fuse(8)

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix typo in ceph-fuse

Fixes: https://github.com/ceph/ceph/pull/21616#pullrequestreview-122923127
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22205 from tchaikov/wip-librgw_admin_user

deb,rpm: package librgw_admin_user.{h,so.*}

Merge pull request #22181 from tchaikov/wip-24025

cmake/modules/BuildRocksDB.cmake: enable compressions for rocksdb

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

deb,rpm: package librgw_admin_user.{h,so.*}

* install and package librgw_admin_user.h, so developers can use it to
  create rgw user.
* package librgw_admin_user, so user can use it to create rgw user.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22198 from dzafman/wip-minor

test: wait_for_pg_stats() should do another check after last 13 secon…

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: wait_for_pg_stats() should do another check after last 13 second sleep

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #22194 from yehudasa/wip-fix-build

rgw: fix build

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/objectstore: move yaml files under a subdir

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa: whitelist SMALLER_PGP_NUM

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw: fix build

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #22190 from smithfarm/wip-google-style

doc/SubmittingPatches.rst: use Google style guide for doc patches

Reviewed-by: Sage Weil <sage@redhat.com>

doc/SubmittingPatches.rst: rv double spaces and Please

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/SubmittingPatches.rst: use Google style guide for doc patches

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22186 from cbodley/wip-cmake-rgw-request

cmake: remove deleted rgw_request.cc from CMakeLists.txt

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cmake/modules/BuildRocksDB.cmake: enable compressions for rocksdb

we should enable them if they are found.

currently, we don't have bzip2 compressor plugin, so it's not detected
in the cmake script. we can always enable it for rocksdb in future.

Fixes: http://tracker.ceph.com/issues/24025
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove deleted rgw_request.cc from CMakeLists.txt

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22116 from alfredodeza/wip-cv-ansible-deps

ceph-volume tests.functional add notario dep for ceph-ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21439 from oritwas/wip-rgw-librgwadmin

new librgw_admin_us

Merge pull request #22060 from ricardoasmarques/wip-perf-counter-breadcrumb

Add performance counters breadcrumb

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22150 from ricardoasmarques/wip-improve-no-pools-message

mgr/dashboard: Improve 'no pool' message on rbd form

Reviewed-by: Kanika Murarka <murarkakanika@gmail.com>

ceph-volume tests.functional install new ceph-ansible dependencies

Make note that ceph-ansible's requirements.txt can't be used just yet

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22158 from liewegas/wip-24222

mon/MgrMonitor: change 'unresponsive' message to info level

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22123 from liewegas/wip-24211

os/bluestore: simplify and fix SharedBlob::put()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #22127 from liewegas/wip-23386

crush: fix device_class_clone for unpopulated/empty weight-sets

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #22110 from majianpeng/bluefs-flush-dirty-devices

os/bluestore/BlueFS: only flush dirty devices when do _fsync

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22129 from rzarzynski/wip-auth-fix-openssl-on-arm

auth, test: fix building on ARMs after the NSS -> OpenSSL transition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22164 from dzafman/wip-test-fixes

test: Whitelist corrections

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22156 from rzarzynski/wip-bug-24037

common: OpTracker doesn't visit TrackedOp when nref == 0.

Reviewed-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: change 'unresponsive' message to info level

We generate a MGR_DOWN health warning at the appropriate points; having
this at WRN level just triggers failed teuthology runs but doesn't much
value for the user.

Clear out teuthology whitelisting for this message.

Fixes: http://tracker.ceph.com/issues/24222
Signed-off-by: Sage Weil <sage@redhat.com>

test: Whitelist corrections

Signed-off-by: David Zafman <dzafman@redhat.com>

common: OpTracker doesn't visit TrackedOp when nref == 0.

The patch fixes a race condition that happens between
`unregister_inflight_op` and `visit_ops_in_flight` of
`OpTracker`. When a callable passed to the former one
turns the plain reference it gets into `TrackedOpRef`,
an almost-to-terminate `TrackedOp` (with `nref == 0`)
can be resurrected (`nref++`). This will be reflected
in extra call to `unregister_inflight_op` for same op
leading to e.g. use-after-free. For more details see:
https://tracker.ceph.com/issues/24037#note-5.

The fix deals with the problem by ensuring there will
be no call to the visitor for ops with zeroized `nref`.

Fixes: http://tracker.ceph.com/issues/24037
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #21700 from cbodley/wip-rgw-dpp

rgw: use DoutPrefixProvider to add more context to log output

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #21755 from linuxbox2/wip-unordered-gc-delay

rgw: use partial-order bucket listing in RGWLC, add configurable processing delay

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #21838 from yehudasa/wip-rgw-async-watch

rgw: async watch registration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22024 from votdev/bug_24142

rgw: Do not modify email if argument is not set

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #22100 from joscollin/wip-rgw-period

rgw: update --url in usage and doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22114 from zhangsw/cleanup-rgw-cors

rgw: remove the useless is_cors_op in RGWHandler_REST_Obj_S3.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22083 from liewegas/wip-21480

os/bluestore: fix flush_commit locking

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22062 from alfredodeza/wip-rm24031

ceph-volume dmcrypt and activate --all documentation updates

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22065 from ricardoasmarques/wip-table-pagination-style

Improve table pagination style

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22149 from b-ranto/wip-fix-metadata

prometheus: Fix order of occupation values

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

os/bluestore: check input validtiy for aio_read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: using func is_valid_io to check input validity.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: fix wrong messages.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #22031 from tspmelo/wip-jest

mgr/dashboard: Add Jest Runner

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22088 from tspmelo/fix-rbd-task-metadata

mgr/dashboard: Fix RBD task metadata

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Improve 'no pool' message on rbd form

Signed-off-by: Ricardo Marques <rimarques@suse.com>

prometheus: Fix order of occupation values

When we merged the PR to unify the metadata labels, we forgot to switch
the order of hostname and disk in ceph_disk_occupation metric.

Signed-off-by: Boris Ranto <branto@redhat.com>

os/bluefs: only flush dirty devices when do _fsync.

Now _fsync call flush_bdev make data safely. But flush_bdev flush all
devices which don't care whether has data for this sync.
So add new api flush_bdev(std::array<bool, MAX_BDEV>& dirty_bdevs)
which only flush dirty devices for this sync op.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/iostat: enable module by default on new and vstart clusters

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

doc: add documentation for iostat

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

Merge pull request #22069 from tchaikov/wip-mgr-cleanup

mgr: fix typo in variable name and cleanups

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #22077 from majianpeng/config-w/o-mgr

cmake: make -DWITH_MGR=OFF work.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22089 from jcsp/wip-24175

mgr/status: output to stdout, not stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22074 from dzafman/wip-parens

test: Need to escape parens in log-whitelist for grep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22096 from yuriw/wip-yuriw-crontab

qa/tests: added filters to support distro tests for client-upgrade tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - reverted clients upgrades for luminous runs
removed ~/rhel_only_on_ovh.yaml to be able to run on ovh
forced runs on smithi only

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

test/crypto: fix wrong output buffer size in AES.DecryptNoBl.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crush: fix device_class_clone for unpopulated/empty weight-sets

If the bucket is empty or does not have weight-set weights yet, avoid
crashing when populating the parent bucket.

Fixes: http://tracker.ceph.com/issues/23386
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move txc on_commits assignment into ctor

This avoids adjusting the oncommits without a lock after the txc is
queued on the sequencer.

This is a bit defensive since the ObjectStore caller doesn't call
flush_commit() at the same time as queue_transaction(), but the could
change in the future.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22036 from linuxbox2/wip-warn-orphans

rgw: require --yes-i-really-mean-it to run radosgw-admin orphans find

test: Add CACHE_POOL_NO_HIT_SET to whitelist for mon/pool_ops.sh

Ignore
  cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Need to escape parens in log-whitelist for grep

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #22064 into master

* refs/pull/22064/head:
os/bluestore: fix signed/unsigned mismatch warning during building

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: simplify and fix SharedBlob::put()

There is a narrow race possible:

A: lookup foo
A: put on foo
A:   foo --nref == 0
B: lookup foo
B: put foo
B:   foo --nref == 0
B: try_remove() succeeds, removes
A: try_remove() tries to remove foo again, probably crashes

We could fix this by flagging the object in some way to indicate it was
removed (maybe clearing parent?), but then we need to be careful about
dereferencing foo to get parent from put().

Fix this by moving to a simpler model: make lookup fail if nref == 0.
This eliminates the races around put() entirely because once nref reaches
0 it never goes up again.

Fixes: http://tracker.ceph.com/issues/24211
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22091 into master

* refs/pull/22091/head:
crush: update choose_args on bucket removal
crush: update choose_args on bucket removal, resize, or position mismatch
crush: create weight-set on demand when doing a choose-args reweight
test/cli/crushtool: use straw2 buckets for choose-args test
crush: weight_set_size -> weight_set_positions

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix flush_commit locking

We were updating the txc state to KV_DONE and queuing the oncommits
waiters without holding any locks.  This was mostly fine, *except* that
Collection|OpSequencer::flush_commit(Context *) was looking at the state
(under qlock) and also adding items to oncommits.

The flush_commit() method is only used in 2 places: osd bench, and the
PG reset_interval_flush outgoing message blocking machinery (which is
a bit ick). The first we could get rid of, but the second is hard to
remove (despite its ick factor).

The simple fix is to take qlock while updating the state value and
working with oncommits.

Fixes: http://tracker.ceph.com/issues/21480
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21540 into wip-sage-testing-20180521.120735

* refs/pull/21540/head:
tests/crypto: print compile warning when NSS is unavailable.
tests/crypto: add tests for the no-bl encrypt/decrypt, part 2.
tests/crypto: add tests for the no-bl encrypt/decrypt.
auth: use OpenSSL for CryptoAESKeyHandler's no-bl encrypt/decrypt.
auth: extend CryptoKey with no-bl encrypt/decrypt.
auth: CryptoAESKeyHandler switches from NSS to OpenSSL.
auth: the outbuf of AES should be multiple of block size
auth: cache the PK11Context for CryptoAESKeyHandler

Merge PR #21973 into wip-sage-testing-20180521.120735

* refs/pull/21973/head:
pybind/ceph_argparse.py: show usage on fully matched cmd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #21985 into wip-sage-testing-20180521.120735

* refs/pull/21985/head:
common: partially revert 95fc248 to make get_process_name work

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21994 into wip-sage-testing-20180521.120735

* refs/pull/21994/head:
os/bluefs/BlueFS: cleanup code.

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22103 from tchaikov/wip-qa-test_envlibrados_for_rocksdb

qa/workunits/rados/test_envlibrados_for_rocksdb: install g++ not g++-4.7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: make -DWITH_MGR=OFF work.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rgw: remove is_cors_op in RGWHandler_REST_Obj_S3 which is not used.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #22084 from liewegas/wip-fix-fop

osd/PrimaryLogPG: fix try_flush_mark_clean write contention case

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: fix typo in rados bench write JSON output

Fixes typo in `max_latency` key in JSON output from rados bench write.

Fixes: http://tracker.ceph.com/issues/24199
Signed-off-by: Sandor Zeestraten <sandor@zeestrataca.com>

Merge pull request #22072 from dillaman/wip-24165

librbd: prevent the use of internal feature bits from external users

Reviewed-by: Mykola Golub <mgolub@suse.com>

buffer.h: avoid using c++14 features

so we dont't need to force librados client to move to C++14 or up.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: use cmake not make

* so we just rely on a single build system instead of two of them, the
  other place we use cmake is cmake/modules/BuildRockDB.cmake.
* disable gflags when building rocksdb, it's optional and does not help
  in the sense of testing librados support.
* disable prompts when installing on debian, to silence warnings like:
  debconf: unable to initialize frontend: Dialog
* drop --force-yes option, as it is deprecated, and is replaced with
  --allow-downgrades, --allow-remove-essential,
  --allow-change-held-packages,  but none of them apply in our case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21982 from tchaikov/mgr-brag

mgr/telemetry: Add Ceph Telemetry module to send reports back to project

Merge PR #21847 into wip-sage-testing-20180521.023125

* refs/pull/21847/head:
log: disk write coalescing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22063 from liewegas/wip-unit-spaces

include/types: space between number and units

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #22042 from liewegas/wip-rm-pool-race

ceph_test_rados_api_misc: remove obsolete LibRadosMiscPool.PoolCreationRace

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #22106 from liewegas/wip-rados-distros

qa/suites/rados/thrash-old-clients: only centos and 16.04

Merge PR #21997 into master

* refs/pull/21997/head:
mds: broadcast quota to relevant clients when quota is explicitly set

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22001 into master

* refs/pull/22001/head:
client: avoid freeing inode when it contains TX buffer heads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22023 into master

* refs/pull/22023/head:
messages: print quota info in MClientQuota msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22045 into master

* refs/pull/22045/head:
client: cleanup Cap init

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22080 into master

* refs/pull/22080/head:
client: fix issue of revoking non-auth caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>