osd rados command: Improve scrub handling of HashInfo (hinfo_key xattr)

Fixes: http://tracker.ceph.com/issues/23364
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't crash in get_hash_info() with a corrupt HashInfo

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Remove check that masks differences (let diff fail)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #21126 from tspmelo/wip-services

mgr/dashboard: Add Api module

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #21319 from majianpeng/doc-remove-osd-op-threads

doc: remove unused config: "osd op threads"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21112 from guzhongyan/fix-vlog

src: fix various log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21028 from rzarzynski/wip-osd-get_rw_locks-norefcnt

osd: avoid unnecessary ref-counting across PrimaryLogPG::get_rw_locks.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21235 from xiexingguo/wip-fix-obj-start

osd/osd_types: fix ideal lower bound object-id of pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21313 from ceph/wip-pg-clean-timeout

qa/tasks: Add default timeout for wait for pg clean task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21284 from tchaikov/wip-cmake-libstdcxx

cmake: define HAVE_STDLIB_MAP_SPLICING for both libstdc++ and libc++

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21290 from rjfd/wip-dashboard-fix-rest-controller

mgr/dashboard: restcontroller: fix detection of id args in element requests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21039 into master

* refs/pull/21039/head:
qa/workunits/fs: test for cephfs rstats
mds: make rstat.rctime follow inodes' ctime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21040 into master

* refs/pull/21040/head:
mds: mark damaged if sessions' preallocated inos don't match inotable

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21132 into master

* refs/pull/21132/head:
client: remove getgroups_cb
client: remove _getgrouplist
client: have init_gids just set alloced_gids to true
client: remove init_groups
fuse: handle errors appropriately when getting group list

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21144 into master

* refs/pull/21144/head:
mds: fix unhealth heartbeat during rejoin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21167 into master

* refs/pull/21167/head:
mds: make stopping mds export strays and subtree at the same time
mds: handle client requests while mds is stopping
mds: check projected linkage before migrating stray dentry

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21191 into master

* refs/pull/21191/head:
client: fix request send_to_auth was never really used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21255 into master

* refs/pull/21255/head:
mds: automatically create snaprealm for old format quota
client: check if mds create snaprealm when setting quota
mds: remove unused MDCache::snaprealm_create()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21295 from badone/wip-large-omap-scrub-big-random-loser-2

qa/suites/rados: Disable scrub backoff

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks: wait_for_clean is called after ceph task as well after osd's are up,
the default timeout is none in that case, there are cases where it can hang  forever
due to error cases, since this dumps quite a lot of info the logs grow in GB's, with
default timeout of 1200 we can avoid such huge logs and fail sooner. Any tests needing
higher timeout can pass the required value.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

doc: remove unused config: "osd op threads"

This option already removed.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #21309 from ceph/wip-yuriw-crontab

qa/tests - minor clean ups and made perf-suite run 3 times, so we can…

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #21282 from liewegas/wip-pgpool-enc

osd/osd_types: fix pg_pool_t encoding for hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Drymov <idryomov@redhat.com>

qa/tests - minor clean ups and made perf-suite run 3 times, so we can have more data points

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #21263 into master

* refs/pull/21263/head:
doc: outline upgrade procedure for mds cluster

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

mgr/dashboard: Move services provision to service module

Few services were being declared in the shared module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Api module

Moved all the api related services into the new module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20807 from Devp00l/improve-table-search

mgr/dashboard: Improve table search

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20943 from tspmelo/fix-e2e

mgr/dashboard: fix frontend e2e tests

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #20758 from ceph/wip-cv-doc-typos

ceph-volume doc updates

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21249 from dillaman/wip-18844

rbd: improve 'import-diff' corrupt input error messages

Reviewed-by: Mykola Golub <mgolub@suse.com>

osd/OSDMap: propagate POOL_SNAPS|SELFMANAGED_SNAPS to tiers

We set this when we add selfmanaged snaps to the base tier.  When those
propagate to the cache tier, propagate the flags, too!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21196 from jeffrey4l/fix_typo

ceph-volume: fix typo in ceph-volume lvm prepare help

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21270 from Devp00l/ceph-table-action-fix

mgr/dashboard: Fix the data table action selector

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: restcontroller: fix detection of id args in element requests

Fixes: https://tracker.ceph.com/issues/23593
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21269 from b-ranto/wip-mgr-expose-rgw

mgr: Expose rgw perf counters

Reviewed-by: John Spray <john.spray@redhat.com>

qa/suites/rados: Disable scrub backoff

A long run of lost coin flips can lead to a timeout in
test_large_omap_detection.py.

Fixes: http://tracker.ceph.com/issues/23578
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

misc: fix various log messages

add space between __func__ and following messages

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

Merge pull request #21248 from dillaman/wip-23548

librbd: potential race between discard and writeback

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17531 from smithfarm/wip-calamari-tests

tests: cleanup: drop calamari tasks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

Merge pull request #21245 from ifed01/wip-ifed-shard-bug-logging

os/bluestore: be more verbose when hitting unloaded shard in extent map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: define HAVE_STDLIB_MAP_SPLICING for both libstdc++ and libc++

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_types: fix pg_pool_t encoding for hammer

If we are missing the hammer feature, we will also be missing the
luminous feature, but we should still encode like hammer.

Analogous fix to e28e0c617af8825ae92cced5d87cc4f403709e48

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20866 from jan--f/mgr-fix-metadata-return

mon/MonCommands: mgr metadata - improve parameter naming consistency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21217 from tchaikov/wip-cmp-hobject

common/hobject: preserve the order of hobject

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21211 from tchaikov/wip-clang-without-map-merge

cmake,mds: detect std::map::merge() before using it.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21161 from cbodley/wip-mon-config-obs

config: notify config observers on set_mon_vals()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21151 from xiexingguo/wip-prune-olr

osd/PGBackend: drop input "snapid_t" from objects_list_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21150 from xiexingguo/wip-large-omap-obj-counting

osd/PGBackend: fix large_omap_objects checking

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21119 from Liuchang0812/refine-optracker-history

common, osd: duplicated "start" event in OpTracker, improve OpTracker::dump_ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21095 from wido/ceph-metadata-version

common: Add metadata with only Ceph version number and release

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21055 from tchaikov/wip-pr-20994

filestore,rgw: fix types/casts making clang on 32-Bit working

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: Clang does not (yet) support std::map::merge

@tchaikov found:
    libc++ still does not support "Splicing Maps and Sets",
    see https://libcxx.llvm.org/cxx1z_status.html ,
    search for "p0083r3" .

Make this case the more explicit coded.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: detect if std::map::merge() is supported

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21274 from dzafman/wip-cot-config

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20760 into wip-sage-testing-20180407.023110

* refs/pull/20760/head:
mon/ConfigMonitor: add 'config reset' command
mon/ConfigMonitor: add 'config log' command
mon/ConfigMap: add ConfigChangeSet
mon/ConfigMonitor: avoid no-op updates
mon/ConfigMonitor: maintain history of all config changes

Merge PR #20516 into wip-sage-testing-20180407.023003

* refs/pull/20516/head:
os/filestore: advertise vdo in filestore metadata
os/filestore: implement thin provisioning utilization stats for VDO
os/bluestore/KernelDevice: advertise vdo and physical_size in metadata
os/bluestore/KernelDevice: implement thin utilization stats for VDO
os/bluestore: hook to expose utilization of thinly-provisioned device
common/blkdev: identify VDO devices beneath LVM
common/blkdev: implement VDO helper functions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #20512 into wip-sage-testing-20180407.022843

* refs/pull/20512/head:
osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21033 into wip-sage-testing-20180407.022623

* refs/pull/21033/head:
buffer: kill push_front
tools/RadosDump: avoid bufferlist::push_front
os/filestore/FileJournal: avoid push_front
erasure-code: avoid bufferlist::push_front
buffer: kill claim_prepend
unittest_compression: avoid claim_prepend
os/bluestore: avoid claim_prepend
osdc/Striper: rewrite assemble_result without push_front or claim_prepend

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19594 from smithfarm/wip-doc-make-check

doc: expand developer documentation of unit tests

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19973 from liewegas/wip-peering-fast-dispatch

osd: fast dispatch of peering events and pg_map + osd sharded wq refactor

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

The _max isn't used.  Drop the _min suffix.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21250 from liewegas/wip-recovery-slow-ops

qa/suites/rados/singleton/all/recovery_preemption: whitelist SLOW_OPS

qa/suites/rados/singleton/all/ec-lost-unfound: whitelist SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Search with spaces

Removed the usage of using comma to indicated a new search term and
introduced the possibility to use quoted strings in order to search for
spaces too.

Also improved the column search handling in order to not get no search
results found while typing a column name.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Improve table search

You can now search for multiple words at a time if you separate them by
comma or space. The other improvement is that you can specify which
column should be searched by a search term. For example if you wand to
filter for a row with the ID 3 you type "id:3". The column name is case
insensitive and is separated from the search word by a colon.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #21239 from rjfd/wip-dashboard-routes

mgr/dashboard: using RoutesDispatcher as HTTP request dispatcher

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #19331 from jecluis/wip-mon-osdmap-prune

mon: osdmap prune

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix the data table action selector

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #20464 from cbodley/wip-rgw-beast-ssl

rgw: add ssl support to beast frontend

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: dev: improve headings within Testing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: Sanitize rgw perf counter names

The rgw perf counters are currently being named by the rgw names (i.e.
client.rgw.<name>.<something>). This changes their names to something
more sane like rgw.<something>. This also makes data exported by
prometheus ceph-mgr module a lot more useful as you do not need to use
the rgw name to get the metrics for an rgw.

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/dashboard: fix frontend e2e tests

Updated protractor configuration: now it's possible to run the tests inside a
docker container, added params for the username and password and added login
during the prepare phase.

Chrome is required to run the e2e tests. This can be achieved by installing
Chrome in your system or by using a docker container with Chrome installed.

Changed some timer calls, so they are runned outside of angular. This was
preventing protractor from running the tests.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr: Expose rgw perf counters

We are filtering rgw perf counters in mgr_module. This commit allows us
to expose rgw perf counters to the mgr modules. This also directly
exports the rgw perf counters via prometheus module since the module
calls the get_all_perf_counters method to get all the perf counters and
then it exports them.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #21128 from ricardoasmarques/wip-usage-bar

mgr/dashboard: Add usage bar component

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21240 from rjfd/wip-dashboard-task-fixes

mgr/dashboard: TaskManager bug fixes

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

qa: enable mon osdmap pruning on 'rados/' suites

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: enable mon osdmap pruning on rados/monthrash

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: mon: osdmap pruning standalone/workunit

Keep a standalone wrapper for the workunit, so we can test it locally,
leveraging the ceph-helpers to do the setup. Keep a workunit to be
exercised by teuthology.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: change PATH to allow finding sysctl

If, by default, a user's PATH does not contain /sbin and /usr/sbin, we
may have a hard time finding sysctl to adjust the kernel core pattern.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: deduplicate cleanup code

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: discard pipe in sysctl kernel.core_pattern

At least verified in openSUSE tumbleweed and ubuntu 16.04,
`sysctl -n kernel.core_pattern` returns a pipe at the start of the
value. We reset core_pattern to its original form once the script is
about to end, but if we do not discard the pipe the new value will
contain an extra pipe (apparently, the pipe is added automatically at
some point, possibly simply on read). We are simply stripping it on
read, as to prevent this behavior.

Additionally, we are also enclosing the reset of kernel.core_pattern in
quotes, so as to make sure patterns that include spaces (e.g., ubuntu's
apport, or tumbleweed's systemd-coredump) are properly reset.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: allow triggering prunes from tick()

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: allow blocking trim if option is enabled

We need to allow blocking osdmap trimming to test osdmap pruning.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: dump osdmap manifest alongside other infos

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: prune full maps on encode_pending

If we have gone over a safety threshold for number of versions, start
pruning full osdmaps, poking holes in the sequence. To make up for the
missing maps, we will rebuild full maps based on the incremental maps
and the closest pinned full map we have available.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

doc: add dev notes for mon's full osdmap pruning

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

osd: legacy pg_create creates pg in current epoch

For the legacy create path, we pregenerate the history and instantiate
the PG at the current epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21127 from smithfarm/wip-make-dist-py3

build/ops: make-dist: fall back to python3

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #21238 into master

* refs/pull/21238/head:
doc: fix typos and other minor errors in cephfs-nfs doc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21204 into master

* refs/pull/21204/head:
mds: PendingReleaseNotes and dev doc for open file table

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: outline upgrade procedure for mds cluster

Fixes: http://tracker.ceph.com/issues/23568
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: beast frontend calls shutdown before closing connections

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: verify suite tests beast with ssl

instead of adding special frontend configs like civetweb_ssl.yaml and
beast_ssl.yaml, added a new proto/ subdirectory for http.yaml and
https.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add ssl support to beast frontend

adds frontend options ssl_certificate, ssl_private_key, ssl_port, ssl_endpoint

Fixes: http://tracker.ceph.com/issues/22832
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: factor the tcp::socket out of ClientIO

remove ClientIO's dependency on a concrete socket type by moving it into
a derived StreamIO class in rgw_asio_frontend.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20992 from cbodley/wip-dmclock-remove-by-ptr

dmclock: update mClockPriorityQueue with changes in subtree