rgw: add distro dependencies for liboath

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: fixes following review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls/otp: support for base32 seed type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp, rgw-admin: use older liboath api, parse seed once

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: preserve otp history when modifying it

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin, cls/otp: get current time for otp resync from osd

add a new method to the otp objclass that returns the
current time.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mfa resync command

$ radosgw-admin mfa resync --uid=<uid> --totp=serial=<serial> \
                --totp-pin=<pin1> --totp-pin=<pin2>

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: integrate with liboath

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set objv_tracker for metadata mutate

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: use metadata mutate for mfa updates

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: better api for metadata mutation

Unlike the regular user and bucket metadata that are plain
raw rados objects, the otp info is controlled by object
class operations. The metadata manager mutate operation
will deal with the metadata related work that is needed
to happen (objv_tracker, update meta log), and call to
the operation that modifies the otp info.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie otp meta get/put to meta log, objv_tracker

still need to have the radosgw-admin mfa tools to go through meta
instrumentation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial implementation of otp meta handlers

Still not there yet. Need to tie into objv_tracker, and metadata log
tooling.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: client api to set multiple keys

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: update header anyway, don't use omap header

Update header even if it might have existed. If running within a compound op
we might not identify it correctly.
Also don't use omap header api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work to add otp as metadata provider

To allow transparent multisite sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix mfa setting, verify on object deletion

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: S3 get bucket versioning returns MFA status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: functionality to modify mfa entries in user metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: new mfa control commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add otp pool, create otp service functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: implement otp_info_t::dump()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_otp: treat empty header as -ENOENT

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send totp check to the otp objclass

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: req_state members init refactor

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: add client interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: implement totp check logic

Currently checking for bogus results, still need to integrate with
totp library.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mfa - initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20758 from ceph/wip-cv-doc-typos

ceph-volume doc updates

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21249 from dillaman/wip-18844

rbd: improve 'import-diff' corrupt input error messages

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21196 from jeffrey4l/fix_typo

ceph-volume: fix typo in ceph-volume lvm prepare help

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21270 from Devp00l/ceph-table-action-fix

mgr/dashboard: Fix the data table action selector

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21269 from b-ranto/wip-mgr-expose-rgw

mgr: Expose rgw perf counters

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21248 from dillaman/wip-23548

librbd: potential race between discard and writeback

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17531 from smithfarm/wip-calamari-tests

tests: cleanup: drop calamari tasks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

Merge pull request #21245 from ifed01/wip-ifed-shard-bug-logging

os/bluestore: be more verbose when hitting unloaded shard in extent map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20866 from jan--f/mgr-fix-metadata-return

mon/MonCommands: mgr metadata - improve parameter naming consistency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21217 from tchaikov/wip-cmp-hobject

common/hobject: preserve the order of hobject

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21211 from tchaikov/wip-clang-without-map-merge

cmake,mds: detect std::map::merge() before using it.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21161 from cbodley/wip-mon-config-obs

config: notify config observers on set_mon_vals()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21151 from xiexingguo/wip-prune-olr

osd/PGBackend: drop input "snapid_t" from objects_list_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21150 from xiexingguo/wip-large-omap-obj-counting

osd/PGBackend: fix large_omap_objects checking

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21119 from Liuchang0812/refine-optracker-history

common, osd: duplicated "start" event in OpTracker, improve OpTracker::dump_ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21095 from wido/ceph-metadata-version

common: Add metadata with only Ceph version number and release

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21055 from tchaikov/wip-pr-20994

filestore,rgw: fix types/casts making clang on 32-Bit working

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: Clang does not (yet) support std::map::merge

@tchaikov found:
    libc++ still does not support "Splicing Maps and Sets",
    see https://libcxx.llvm.org/cxx1z_status.html ,
    search for "p0083r3" .

Make this case the more explicit coded.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: detect if std::map::merge() is supported

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21274 from dzafman/wip-cot-config

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20760 into wip-sage-testing-20180407.023110

* refs/pull/20760/head:
mon/ConfigMonitor: add 'config reset' command
mon/ConfigMonitor: add 'config log' command
mon/ConfigMap: add ConfigChangeSet
mon/ConfigMonitor: avoid no-op updates
mon/ConfigMonitor: maintain history of all config changes

Merge PR #20516 into wip-sage-testing-20180407.023003

* refs/pull/20516/head:
os/filestore: advertise vdo in filestore metadata
os/filestore: implement thin provisioning utilization stats for VDO
os/bluestore/KernelDevice: advertise vdo and physical_size in metadata
os/bluestore/KernelDevice: implement thin utilization stats for VDO
os/bluestore: hook to expose utilization of thinly-provisioned device
common/blkdev: identify VDO devices beneath LVM
common/blkdev: implement VDO helper functions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #20512 into wip-sage-testing-20180407.022843

* refs/pull/20512/head:
osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21033 into wip-sage-testing-20180407.022623

* refs/pull/21033/head:
buffer: kill push_front
tools/RadosDump: avoid bufferlist::push_front
os/filestore/FileJournal: avoid push_front
erasure-code: avoid bufferlist::push_front
buffer: kill claim_prepend
unittest_compression: avoid claim_prepend
os/bluestore: avoid claim_prepend
osdc/Striper: rewrite assemble_result without push_front or claim_prepend

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19594 from smithfarm/wip-doc-make-check

doc: expand developer documentation of unit tests

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19973 from liewegas/wip-peering-fast-dispatch

osd: fast dispatch of peering events and pg_map + osd sharded wq refactor

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

The _max isn't used.  Drop the _min suffix.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21250 from liewegas/wip-recovery-slow-ops

qa/suites/rados/singleton/all/recovery_preemption: whitelist SLOW_OPS

qa/suites/rados/singleton/all/ec-lost-unfound: whitelist SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21239 from rjfd/wip-dashboard-routes

mgr/dashboard: using RoutesDispatcher as HTTP request dispatcher

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #19331 from jecluis/wip-mon-osdmap-prune

mon: osdmap prune

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix the data table action selector

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #20464 from cbodley/wip-rgw-beast-ssl

rgw: add ssl support to beast frontend

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: dev: improve headings within Testing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: Sanitize rgw perf counter names

The rgw perf counters are currently being named by the rgw names (i.e.
client.rgw.<name>.<something>). This changes their names to something
more sane like rgw.<something>. This also makes data exported by
prometheus ceph-mgr module a lot more useful as you do not need to use
the rgw name to get the metrics for an rgw.

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr: Expose rgw perf counters

We are filtering rgw perf counters in mgr_module. This commit allows us
to expose rgw perf counters to the mgr modules. This also directly
exports the rgw perf counters via prometheus module since the module
calls the get_all_perf_counters method to get all the perf counters and
then it exports them.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #21128 from ricardoasmarques/wip-usage-bar

mgr/dashboard: Add usage bar component

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21240 from rjfd/wip-dashboard-task-fixes

mgr/dashboard: TaskManager bug fixes

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

qa: enable mon osdmap pruning on 'rados/' suites

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: enable mon osdmap pruning on rados/monthrash

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa: mon: osdmap pruning standalone/workunit

Keep a standalone wrapper for the workunit, so we can test it locally,
leveraging the ceph-helpers to do the setup. Keep a workunit to be
exercised by teuthology.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: change PATH to allow finding sysctl

If, by default, a user's PATH does not contain /sbin and /usr/sbin, we
may have a hard time finding sysctl to adjust the kernel core pattern.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: deduplicate cleanup code

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

qa/standalone: discard pipe in sysctl kernel.core_pattern

At least verified in openSUSE tumbleweed and ubuntu 16.04,
`sysctl -n kernel.core_pattern` returns a pipe at the start of the
value. We reset core_pattern to its original form once the script is
about to end, but if we do not discard the pipe the new value will
contain an extra pipe (apparently, the pipe is added automatically at
some point, possibly simply on read). We are simply stripping it on
read, as to prevent this behavior.

Additionally, we are also enclosing the reset of kernel.core_pattern in
quotes, so as to make sure patterns that include spaces (e.g., ubuntu's
apport, or tumbleweed's systemd-coredump) are properly reset.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: allow triggering prunes from tick()

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: allow blocking trim if option is enabled

We need to allow blocking osdmap trimming to test osdmap pruning.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: dump osdmap manifest alongside other infos

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: prune full maps on encode_pending

If we have gone over a safety threshold for number of versions, start
pruning full osdmaps, poking holes in the sequence. To make up for the
missing maps, we will rebuild full maps based on the incremental maps
and the closest pinned full map we have available.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

doc: add dev notes for mon's full osdmap pruning

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

osd: legacy pg_create creates pg in current epoch

For the legacy create path, we pregenerate the history and instantiate
the PG at the current epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21127 from smithfarm/wip-make-dist-py3

build/ops: make-dist: fall back to python3

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #21238 into master

* refs/pull/21238/head:
doc: fix typos and other minor errors in cephfs-nfs doc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21204 into master

* refs/pull/21204/head:
mds: PendingReleaseNotes and dev doc for open file table

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: beast frontend calls shutdown before closing connections

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: verify suite tests beast with ssl

instead of adding special frontend configs like civetweb_ssl.yaml and
beast_ssl.yaml, added a new proto/ subdirectory for http.yaml and
https.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add ssl support to beast frontend

adds frontend options ssl_certificate, ssl_private_key, ssl_port, ssl_endpoint

Fixes: http://tracker.ceph.com/issues/22832
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: factor the tcp::socket out of ClientIO

remove ClientIO's dependency on a concrete socket type by moving it into
a derived StreamIO class in rgw_asio_frontend.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20992 from cbodley/wip-dmclock-remove-by-ptr

dmclock: update mClockPriorityQueue with changes in subtree

mon/OSDMonitor: fix warnings

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: keep sending legacy create messages for now

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21261 from batrick/ceph-update-ceph-object-corpus

ceph-object-corpus: update to fix make check

ceph-object-corpus: update to fix make check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: discard should wait for in-flight cache writeback to complete

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/ObjectCacher: allow discard to complete in-flight writeback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21148 from wjwithagen/wjw-setup-virtualenv

tools: FreeBSD basic getopt does not use --options

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21237 from guzhongyan/zabbix-fix

mgr/zabbix: Fix wrong log message

Reviewed-by: John Spray <john.spray@redhat.com>