pybind/rbd: fix python 3 str to bytes problems

Fixes: http://tracker.ceph.com/issues/23609
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21249 from dillaman/wip-18844

rbd: improve 'import-diff' corrupt input error messages

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21196 from jeffrey4l/fix_typo

ceph-volume: fix typo in ceph-volume lvm prepare help

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21270 from Devp00l/ceph-table-action-fix

mgr/dashboard: Fix the data table action selector

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21269 from b-ranto/wip-mgr-expose-rgw

mgr: Expose rgw perf counters

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21248 from dillaman/wip-23548

librbd: potential race between discard and writeback

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17531 from smithfarm/wip-calamari-tests

tests: cleanup: drop calamari tasks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

Merge pull request #21245 from ifed01/wip-ifed-shard-bug-logging

os/bluestore: be more verbose when hitting unloaded shard in extent map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20866 from jan--f/mgr-fix-metadata-return

mon/MonCommands: mgr metadata - improve parameter naming consistency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21217 from tchaikov/wip-cmp-hobject

common/hobject: preserve the order of hobject

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21211 from tchaikov/wip-clang-without-map-merge

cmake,mds: detect std::map::merge() before using it.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21161 from cbodley/wip-mon-config-obs

config: notify config observers on set_mon_vals()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21151 from xiexingguo/wip-prune-olr

osd/PGBackend: drop input "snapid_t" from objects_list_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21150 from xiexingguo/wip-large-omap-obj-counting

osd/PGBackend: fix large_omap_objects checking

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21119 from Liuchang0812/refine-optracker-history

common, osd: duplicated "start" event in OpTracker, improve OpTracker::dump_ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21095 from wido/ceph-metadata-version

common: Add metadata with only Ceph version number and release

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21055 from tchaikov/wip-pr-20994

filestore,rgw: fix types/casts making clang on 32-Bit working

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: Clang does not (yet) support std::map::merge

@tchaikov found:
    libc++ still does not support "Splicing Maps and Sets",
    see https://libcxx.llvm.org/cxx1z_status.html ,
    search for "p0083r3" .

Make this case the more explicit coded.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: detect if std::map::merge() is supported

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21274 from dzafman/wip-cot-config

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20760 into wip-sage-testing-20180407.023110

* refs/pull/20760/head:
mon/ConfigMonitor: add 'config reset' command
mon/ConfigMonitor: add 'config log' command
mon/ConfigMap: add ConfigChangeSet
mon/ConfigMonitor: avoid no-op updates
mon/ConfigMonitor: maintain history of all config changes

Merge PR #20516 into wip-sage-testing-20180407.023003

* refs/pull/20516/head:
os/filestore: advertise vdo in filestore metadata
os/filestore: implement thin provisioning utilization stats for VDO
os/bluestore/KernelDevice: advertise vdo and physical_size in metadata
os/bluestore/KernelDevice: implement thin utilization stats for VDO
os/bluestore: hook to expose utilization of thinly-provisioned device
common/blkdev: identify VDO devices beneath LVM
common/blkdev: implement VDO helper functions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #20512 into wip-sage-testing-20180407.022843

* refs/pull/20512/head:
osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21033 into wip-sage-testing-20180407.022623

* refs/pull/21033/head:
buffer: kill push_front
tools/RadosDump: avoid bufferlist::push_front
os/filestore/FileJournal: avoid push_front
erasure-code: avoid bufferlist::push_front
buffer: kill claim_prepend
unittest_compression: avoid claim_prepend
os/bluestore: avoid claim_prepend
osdc/Striper: rewrite assemble_result without push_front or claim_prepend

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19594 from smithfarm/wip-doc-make-check

doc: expand developer documentation of unit tests

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19973 from liewegas/wip-peering-fast-dispatch

osd: fast dispatch of peering events and pg_map + osd sharded wq refactor

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

The _max isn't used.  Drop the _min suffix.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21250 from liewegas/wip-recovery-slow-ops

qa/suites/rados/singleton/all/recovery_preemption: whitelist SLOW_OPS

qa/suites/rados/singleton/all/ec-lost-unfound: whitelist SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21239 from rjfd/wip-dashboard-routes

mgr/dashboard: using RoutesDispatcher as HTTP request dispatcher

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #19331 from jecluis/wip-mon-osdmap-prune

mon: osdmap prune

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix the data table action selector

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #20464 from cbodley/wip-rgw-beast-ssl

rgw: add ssl support to beast frontend

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: dev: improve headings within Testing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: Sanitize rgw perf counter names

The rgw perf counters are currently being named by the rgw names (i.e.
client.rgw.<name>.<something>). This changes their names to something
more sane like rgw.<something>. This also makes data exported by
prometheus ceph-mgr module a lot more useful as you do not need to use
the rgw name to get the metrics for an rgw.

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr: Expose rgw perf counters

We are filtering rgw perf counters in mgr_module. This commit allows us
to expose rgw perf counters to the mgr modules. This also directly
exports the rgw perf counters via prometheus module since the module
calls the get_all_perf_counters method to get all the perf counters and
then it exports them.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #21128 from ricardoasmarques/wip-usage-bar

mgr/dashboard: Add usage bar component

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21240 from rjfd/wip-dashboard-task-fixes

mgr/dashboard: TaskManager bug fixes

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

qa: enable mon osdmap pruning on 'rados/' suites

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: enable mon osdmap pruning on rados/monthrash

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: mon: osdmap pruning standalone/workunit

Keep a standalone wrapper for the workunit, so we can test it locally,
leveraging the ceph-helpers to do the setup. Keep a workunit to be
exercised by teuthology.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: change PATH to allow finding sysctl

If, by default, a user's PATH does not contain /sbin and /usr/sbin, we
may have a hard time finding sysctl to adjust the kernel core pattern.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: deduplicate cleanup code

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: discard pipe in sysctl kernel.core_pattern

At least verified in openSUSE tumbleweed and ubuntu 16.04,
`sysctl -n kernel.core_pattern` returns a pipe at the start of the
value. We reset core_pattern to its original form once the script is
about to end, but if we do not discard the pipe the new value will
contain an extra pipe (apparently, the pipe is added automatically at
some point, possibly simply on read). We are simply stripping it on
read, as to prevent this behavior.

Additionally, we are also enclosing the reset of kernel.core_pattern in
quotes, so as to make sure patterns that include spaces (e.g., ubuntu's
apport, or tumbleweed's systemd-coredump) are properly reset.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: allow triggering prunes from tick()

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: allow blocking trim if option is enabled

We need to allow blocking osdmap trimming to test osdmap pruning.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: dump osdmap manifest alongside other infos

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: prune full maps on encode_pending

If we have gone over a safety threshold for number of versions, start
pruning full osdmaps, poking holes in the sequence. To make up for the
missing maps, we will rebuild full maps based on the incremental maps
and the closest pinned full map we have available.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

doc: add dev notes for mon's full osdmap pruning

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

osd: legacy pg_create creates pg in current epoch

For the legacy create path, we pregenerate the history and instantiate
the PG at the current epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21127 from smithfarm/wip-make-dist-py3

build/ops: make-dist: fall back to python3

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #21238 into master

* refs/pull/21238/head:
doc: fix typos and other minor errors in cephfs-nfs doc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21204 into master

* refs/pull/21204/head:
mds: PendingReleaseNotes and dev doc for open file table

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: beast frontend calls shutdown before closing connections

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: verify suite tests beast with ssl

instead of adding special frontend configs like civetweb_ssl.yaml and
beast_ssl.yaml, added a new proto/ subdirectory for http.yaml and
https.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add ssl support to beast frontend

adds frontend options ssl_certificate, ssl_private_key, ssl_port, ssl_endpoint

Fixes: http://tracker.ceph.com/issues/22832
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: factor the tcp::socket out of ClientIO

remove ClientIO's dependency on a concrete socket type by moving it into
a derived StreamIO class in rgw_asio_frontend.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20992 from cbodley/wip-dmclock-remove-by-ptr

dmclock: update mClockPriorityQueue with changes in subtree

mon/OSDMonitor: fix warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: keep sending legacy create messages for now

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21261 from batrick/ceph-update-ceph-object-corpus

ceph-object-corpus: update to fix make check

ceph-object-corpus: update to fix make check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: discard should wait for in-flight cache writeback to complete

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/ObjectCacher: allow discard to complete in-flight writeback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21148 from wjwithagen/wjw-setup-virtualenv

tools: FreeBSD basic getopt does not use --options

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21237 from guzhongyan/zabbix-fix

mgr/zabbix: Fix wrong log message

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21253 from ceph/wip-yuriw-crontab

tests/qa - enabled `ceph-deploy` runs on `mira` nodes

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #21254 into master

* refs/pull/21254/head:
mds: bump mds_log_max_segments for trim buffer

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21252 from neha-ojha/wip-modify-crontab

qa/crontab: run the perf-basic suite every day

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

mds: bump mds_log_max_segments for trim buffer

Under create heavy workloads, the MDS will sometimes get behind trimming but
catch up. This avoids unnecessary warnings.

Fixes: http://tracker.ceph.com/issues/23560
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tests/qa - enabled `ceph-deploy` runs on `mira` nodes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: Apply usage-bar component on health

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Apply usage-bar component on osd

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Apply usage-bar component on cephfs

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add usage bar component

Signed-off-by: Ricardo Marques <rimarques@suse.com>

qa/crontab: run the perf-basic suite every day

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/singleton/all/recovery_preemption: whitelist SLOW_OPS

Recovery and peering can be slow enough with all the logging enabled to
trigger a slow ops warning.

Signed-off-by: Sage Weil <sage@redhat.com>

TEST Revert "mon/OSDMonitor: disallow pg_num changes until after pool is created"

This reverts commit 38319f8300ebf4a48b6910790467bad96d29941e.

I want to reproduce the race to see if there is a better way to avoid it.

This breaks tests that set pg_num, and generally makes interacting with
the cluster awkward for anything non-human.

# Conflicts:
# src/mon/CreatingPGs.h

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21101 from dzafman/wip-23267

scrub errors not cleared on replicas can cause inconsistent pg state when replica takes over primary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: improve 'import-diff' corrupt input error messages

Fixes: http://tracker.ceph.com/issues/18844
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

build/ops: make-dist: fall back to python3

If python2.7 is not available, we might be building in a Python 3-only
environment.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: move all controller infrastructure to controllers/__init__.py

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: using RoutesDispatcher as HTTP request dispatcher

Signed-off-by: Ricardo Dias <rdias@suse.com>

osd: clarify FIXME comments

These are generally things to fix aspirationally someday or things that
we'll need to clean up as part of the move to seastar.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21146 from zmc/wip-cephmetrics

qa/suites/cephmetrics: Updates for new version

osd: reimplement get_mapped_pools

Output as a proper array this time.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21244 from ifed01/wip-ifed-min-blob-descr

os/bluestore: update description for bluestore_compression_[min|max|_blob_size options

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: be more verbose when hitting unloaded shard in extent map

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: update description for
bluestore_compression_[min|max|_blob_size options

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: avoid OSDShardPGSlot allocation in fast path

If the entry already exists, we don't want to construct the PGSlot
and then throw it out.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: check for rctx contexts instead of adding nop() to transaction

This is cheaper and more explicit.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add has_contexts()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up osdmap_lock

Comment rules and take lock during init/shutdown (nice but not necessary).

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: not pg_num adjustments during pool create

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: mons before osds to avoid force-recovery issue

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop obsolete Pred

Signed-off-by: Sage Weil <sage@redhat.com>

osd: OSDShard: sdata_op_ordering_lock -> shard_lock

Signed-off-by: Sage Weil <sage@redhat.com>