test/librbd: expect_object_may_exist(..., false) in DiscardRemoveTruncate

This is !is_no_op_for_nonexistent_object() test -- NONEXISTENT case is
the interesting one.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: clone with parent features in DiscardRemoveTruncate

... otherwise object map expectations aren't actually set.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20659 from tchaikov/wip-msg/async/dpdk/launch-on-coreid

msg/async: execute on core specified by core_id not its index

Reviewed-by: shangfufei <shangfufei@inspur.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20519 from gregsfortytwo/wip-22882-linger-locking

osdc/Objecter: take budgets across a LingerOp instead of on child Ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osdc: create and use take_linger_budget for prefetching watch/notify budgets

We don't actually take any budget, as we don't want to hit the
total op limits (especially on eg rbd-mirror), but this prevents
our per-message budgeting from causing a deadlock.

Fixes: http://tracker.ceph.com/issues/22882
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20517 from gregsfortytwo/wip-22114-beacon-blocking-2

mon: mark OSD beacons and pg_create messages as no_reply

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #20720 from ctrlaltdel/typo

doc: Fix typo s/applicatoin/application/

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20535 from ifed01/wip-ifed-bring-fio

test/fio: enable objectstore FIO plugin building without the need to install and build FIO source code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20708 from majianpeng/bluefs-bug-fix

os/bluestore: fix wrong usage for BlueFS::_allocate.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix wrong usage for BlueFS::_allocate.

In fact, function _allocate is atomic function which don't
allocate less than want space.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueFS: cleanup code for more readable.

And remove the useless judgement.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #20699 from yaozongyou/fix-potential-memory-leak

common: fix potential memory leak in HTMLFormatter

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20701 from tchaikov/wip-23212

os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20544 from liewegas/wip-min-kv

os/bluestore: kv_max -> kv_min

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20713 from smithfarm/wip-spdk-python3

spdk: advance to upstream dc82989d

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20131 from jcsp/wip-doc-encoding

doc/dev: add a brief guide to serialization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20717 from b-ranto/wip-restful-doc

doc: Improve mgr/restful module documentation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20542 from liewegas/wip-bluefs-buffered-io

common/options: bluefs_buffered_io=true by default

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #20721 from idryomov/wip-krbd-parent-overlap-test

qa: krbd parent-overlap test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/fio: enable objectstore FIO plugin building without the need to install and build FIO source code

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20695 from mikechristie/wip-doc-mnc-update-iscsi-kernel

doc: update iSCSI upstream kernel to 4.16

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20663 from tchaikov/wip-ceph.in-py3

ceph.in: use a different variable for holding thrown exception

Reviewed-by: John Spray <john.spray@redhat.com>

Doc: Fix typo s/applicatoin/application/

Signed-off-by: Francois Deppierraz <francois@ctrlaltdel.ch>

Merge pull request #20714 from idryomov/wip-krbd-msgr-segments-test

qa: krbd msgr-segments test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Fixes: http://tracker.ceph.com/issues/23212
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Improve mgr/restful module documentation

Fixes: https://tracker.ceph.com/issues/23219
Signed-off-by: Boris Ranto <branto@redhat.com>

qa: krbd parent-overlap test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

spdk: advance to upstream dc82989dd2747db0384cafb3b310f2e8792d5c24

This fixes a py2/3-related build issue in SPDK.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa: krbd msgr-segments test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20687 from Songweibin/wip-misc-cleanup

mon,osd,crush: misc cleanup

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20485 from jcsp/wip-23017

mgr: quieten logging on missing OSD stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: fix potential memory leak in HTMLFormatter

If HTMLFormatter::set_status called twice, memory will leaked,
so fix it.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #20573 from liewegas/wip-mgr-ovh

qa/suites/rados/rest/mgr: provision openstack volumes

Merge pull request #20692 from idryomov/wip-krbd-huge-image-test

qa: krbd huge-image test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: update iSCSI upstream kernel to 4.16

Signed-off-by: Mike Christie <mchristi@redhat.com>

qa: krbd huge-image test

We've had multiple overflows in ceph_calc_file_object_mapping().
It wasn't being used by rbd, but it now is.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: rename blkroset to basic

More simple fixed-1.yaml based tests are coming.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20680 from cbodley/wip-qa-dnsmasq

qa: clean up dnsmasq task and fix EPERM error

Merge pull request #19252 from joke-lee/website_redirect

rgw: fix s3 website redirection error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20635 from mdw-at-linuxbox/wip-master-rgw-openssl

rgw: curl* reuse and for debian, use openssl not gnutls.

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20673 from ceph/wip-rm23148

docs fix ceph-volume missing sub-commands

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

crush: should break the loop if leaves overfull

Signed-off-by: songweibin <song.weibin@zte.com.cn>

osd: drop redundant assert check

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20684 from liewegas/wip-bluestore-cache-debug

os/bluestore: no trim debug noise if there is no trimming to be done

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: no trim debug noise if there is no trimming to be done

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20637 from wjwithagen/wjw-fix-MDSmonitor.h

mon/MDSMonitor: fix clang build failure

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20671 from batrick/rados-doc-flags

doc: init flags to 0 in rados example

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20653 from xiexingguo/wip-fix-upmap

mon, osd: fix potential collided *Up Set* after PG remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20655 from xiexingguo/wip-fix-calc-upmap

osd/OSDMap: skip out/crush-out osds

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20506 from jan--f/prometheus-rm-outdated-daemon-metrics

pybind/mgr/prometheus: don't export metrics for dead daemon; new metrics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20667 from tchaikov/wip-qa-standalone

qa/standalone: s/delete_erasure_pool/delete_erasure_coded_pool/

Reviewed-by: David Zafman <dzafman@redhat.com>

qa: clean up dnsmasq task and fix EPERM error

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20648 from trociny/wip-snap_set_diff-empty_snapset

librados/snap_set_diff: don't assert on empty snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20613 from Songweibin/wip-23038

rbd: import with option --export-format 2 fails to protect snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20608 from Songweibin/wip-rbd-snap-purge

rbd: allow remove all unprotected snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20578 from trociny/wip-pybind-group

librbd: small cleanup for recently merged code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20621 from cbodley/wip-rgw-data_lock

rgw: don't hold data_lock over frontend io

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20449 from cbodley/wip-rgw-beast-coroutine

rgw: switch beast frontend back to stackful coroutine

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/install add missing lvm subcommand for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados add missing lvm subcommand for ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20546 from yehudasa/wip-rgw-gc-aio

rgw: gc use aio

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #20643 into master

* refs/pull/20643/head:
ceph-volume-client: fix command argument order

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #20650 from ceph/wip-rm22841

ceph-volume be resilient to $PATH issues

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: init flags to 0 in rados example

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20656 from yaozongyou/rbd-fix-return

librbd: fix missing return in NotifyMessage::get_notify_op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: allow remove all unprotected snapshots

allow remove all unprotected snapshots when exiting
protected snapshots in the same image.
Fixes: http://tracker.ceph.com/issues/23126
Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: import with option --export-format fails to protect snapshot

Fixes: http://tracker.ceph.com/issues/23038
Signed-off-by: songweibin <song.weibin@zte.com.cn>

mon, osd: fix potential collided *Up Set* after PG remapping

The mgr balancer module are basically doing optimizations based on
the snapshots of OSDMap at certain moments, which turns out to be
the culprit of data loss since it can produce bad PG mapping results
sometimes while in upmap mode.
I.e.:
1) original cluster topology:

-5       2.00000     host host-a
 0   ssd 1.00000         osd.0       up  1.00000 1.00000
 1   ssd 1.00000         osd.1       up  1.00000 1.00000
-7       2.00000     host host-b
 2   ssd 1.00000         osd.2       up  1.00000 1.00000
 3   ssd 1.00000         osd.3       up  1.00000 1.00000
-9       2.00000     host host-c
 4   ssd 1.00000         osd.4       up  1.00000 1.00000
 5   ssd 1.00000         osd.5       up  1.00000 1.00000

2) mgr balancer applies optimization for PG 3.f:

            pg-upmap-items[3.f : 1->4]
3.f [1 3] + -------------------------> [4 3]

3) osd.3 is out/reweighted etc., original crush mapping of 3.f changed
   (while pg-upmap-items did not):

            pg-upmap-items[3.f : 1->4]
3.f [1 5] + -------------------------> [4 5]

4) we are now mapping PG 3.f to two OSDs(osd.4 & osd.5) on the same host
   (host-c).

Fix the above problem by putting a guard procedure before we can
finally encode these *unsafe* upmap remappings into OSDMap.
If any of them turns out to be inappropriate, we can simply cancel it
since balancer can still re-calculate and re-generate later if necessary.

Fixes: http://tracker.ceph.com/issues/23118
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20640 from tchaikov/wip-async-dpdk-temp-str

msg/async: avoid referencing the temporary string

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20531 from xiexingguo/wip-ec-scrub-sleep

osd/ECBackend: inject sleep during deep scrub

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone: s/delete_erasure_pool/delete_erasure_coded_pool/

it's a regression introduced by ac56a202

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: fix clang error

The specific specialisations in both .h and .cc does not really fly
with Clang

/home/jenkins/workspace/ceph-master/src/mon/MDSMonitor.cc:79:22: error: template parameter redefines default argument
template <int dblV = 7>
                     ^
/home/jenkins/workspace/ceph-master/src/mon/MDSMonitor.h:76:23: note: previous default template argument defined here
  template<int dblV = 7>
                      ^
1 error generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

ceph.in: use a differnt variable for holding thrown exception

otherwise, we will have

$ bin/ceph osd
Traceback (most recent call last):
  File "bin/ceph", line 1178, in <module>
    retval = main()
  File "bin/ceph", line 1109, in main
    verbose)
  File "bin/ceph", line 535, in new_style_command
    valid_dict = validate_command(sigdict, cmdargs, verbose)
  File "/var/ceph/ceph/src/pybind/ceph_argparse.py", line 1083, in
validate_command
    if found or e:
UnboundLocalError: local variable 'e' referenced before assignment

when using python3

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: execute on core specified by core_id not its index

Reported-by: shangfufei <shangfufei@inspur.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: do not include unused header

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/dpdk/shared_ptr.h: sync up with seastar impl

otherwise the dpdk plugin does not compile.

* remove workarounds for C++11. for example, std::void_t is available
  in C++17 now.
* drop seastar::shared_ptr<>, as it's not used in async/dpdk now.
* update the editor variable to match seastar's coding style

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: extend daemon metadata with hostname and ceph version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add fs and mds metadata metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: new mon metrics

Remove mon_quorum_count and replace it with per-MON quorum status
(mon_quorum_status). Also add mon_metadata metrics.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add Metrics class to manage Metric instances

The central change of this commit is that per-daemon metrics are now
managed by first appending the metric (using Metrics.append) to a
staging area. Then the metrics for specific paths (metric names) are
overwritten by the staged metrics (by calling Metrics.reset). This gets
rid of metrics from daemon that are no longer in the cluster. I.e. when
ceph no longer reports metrics for one OSD daemon (because it was
removed from the cluster) the prometheus module will no longer export
metrics for that daemon.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mon: drop unused variable - 'args'

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20630 from joscollin/wip-doc-monitoring

doc: Update monitoring.rst

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

doc: Update monitoring.rst

Updated monitoring.rst with
* Latest command output
* misc doc fixes

Signed-off-by: Jos Collin <jcollin@redhat.com>

rbd: fix return is missing in NotifyMessage::get_notify_op

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

osd/OSDMap: skip out/crush-out osds

We might do 'ceph osd out <osd.x>' or 'ceph osd crush reweight <osd.x> 0'
for various reasons, and hence can produce 0-weighted OSDs.
Skip those OSDs when trying to calculdate PG upmaps so we won't be able to
hit the *assert* below:

/build/ceph-13.0.1-2232-g64665c7/src/osd/OSDMap.cc: 4179: FAILED assert(target > 0)

See also:

http://pulpito.ceph.com/xxg-2018-02-28_09:02:53-rados-wip-fix-upmap-distro-basic-smithi/2235497/

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-volume process always try to expand the executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure behavior of new which utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system add a helper to find system executables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20646 from tchaikov/wip-qa/run-standalone.sh/python-path

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20634 from tchaikov/wip-delete-pool-in-standalone-test

qa/standalone: extract delete_pool()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20619 from theanalyst/doc/release-notes/12.2.4

doc: 12.2.4 release notes

Reviewed-By: Alfredo Deza <adeza@redhat.com>
Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>

librados/snap_set_diff: don't assert on empty snapset

Instead treat the diff as a full-object delta.

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

do not set PYTHONPATH=/usr/lib/python2.7/dist-packages anymore, it will
be overridden later on anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: split memory 50/50 between bluestore and rocksdb

We don't yet have a clear answer on where our memory is best spent once
we are above kv_min.  Previous testing showed that bluestore was a better
bet, but that predates the enabling of bloom filters and/or inclusion of
bloom filters in the rocksdb cache size.

A 50/50 split hedges out bets until we have more information.  At worst
we are misusing half of any addition memory given to the OSD (vs
potentially more than that for any config that steers more memory one way
or the other).

We still spend no memory on data currently.  That should probably change.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_cache_kv_max -> bluestore_cache_kv_min

Currently, the default tunables are
  cache_meta_ratio = .01
  cache_kv_ratio = .99
  cache_kv_max = 512MB

We currently have no doubts that, for low memory situations, we want to
give all of the memory to rocksdb.  The .99/.01 values are an indirect
way to communicating that.  However, we do see problems with high-memory
situations where rocksdb needs more than 512MB (due to, we believe,
bloom filters). And it is very reasonable to think that there are
situations and reasons why we might want more than 512MB for rocksdb.

So, we change kv_max -> kv_min.  For low memory situations, we give *all*
memory to rocksdb.  (If you don't want this, set kv_min = 0.)  Above that,
respect the kv_ratio and meta_ratio settings.

I don't see much reason to have an absolute kv_max on the rocksdb memory.

We adjust the kv_ratio and meta_ratio defaults in this patch so that all
high memory goes to bluestore and there is no effective change in
default behavior for this patch.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19015 from hansbogert/master

mon: Add `ceph osd get-require-min-compat-client` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #20607 from shangfufei/test_stack

msg/async:fix the incoming parameter type of EventCenter::process_events()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20509 from liewegas/wip-less-filestore-idempotent

qa/tasks/filestore_idempotent: shorter test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20498 from liewegas/wip-22616

os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20616 from tchaikov/wip-cmake

cmake/BuildBoost: fixes to ready seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>