rgw: sync plugin for aws (the humble beginnings!)

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #19283 from yehudasa/wip-rgw-mfa

rgw: mfa support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21327 from LenzGr/dashboard-fork-awesome

mgr/dashboard: Replace font-awesome with fork-awesome

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge PR #21080 into master

* refs/pull/21080/head:
src: Added const references to various function parameters

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21126 from tspmelo/wip-services

mgr/dashboard: Add Api module

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Replace font-awesome with fork-awesome

Replaces the dependency on Font Awesome with Fork Awesome
(https://forkawesome.github.io/), which is a plug-in replacement
for font-awesome 4.7.0 and adds a few more icons (see
https://forkawesome.github.io/Fork-Awesome/whats-new/ for
background information about this fork).

It is licensed under the following licenses - see
https://forkawesome.github.io/Fork-Awesome/license/ for details:

Fonts: SIL OFL 1.1
Code: MIT License
Documentation: CC BY 3.0

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21319 from majianpeng/doc-remove-osd-op-threads

doc: remove unused config: "osd op threads"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21112 from guzhongyan/fix-vlog

src: fix various log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21028 from rzarzynski/wip-osd-get_rw_locks-norefcnt

osd: avoid unnecessary ref-counting across PrimaryLogPG::get_rw_locks.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21235 from xiexingguo/wip-fix-obj-start

osd/osd_types: fix ideal lower bound object-id of pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21313 from ceph/wip-pg-clean-timeout

qa/tasks: Add default timeout for wait for pg clean task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21284 from tchaikov/wip-cmake-libstdcxx

cmake: define HAVE_STDLIB_MAP_SPLICING for both libstdc++ and libc++

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21290 from rjfd/wip-dashboard-fix-rest-controller

mgr/dashboard: restcontroller: fix detection of id args in element requests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21039 into master

* refs/pull/21039/head:
qa/workunits/fs: test for cephfs rstats
mds: make rstat.rctime follow inodes' ctime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21040 into master

* refs/pull/21040/head:
mds: mark damaged if sessions' preallocated inos don't match inotable

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21132 into master

* refs/pull/21132/head:
client: remove getgroups_cb
client: remove _getgrouplist
client: have init_gids just set alloced_gids to true
client: remove init_groups
fuse: handle errors appropriately when getting group list

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21144 into master

* refs/pull/21144/head:
mds: fix unhealth heartbeat during rejoin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21167 into master

* refs/pull/21167/head:
mds: make stopping mds export strays and subtree at the same time
mds: handle client requests while mds is stopping
mds: check projected linkage before migrating stray dentry

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21191 into master

* refs/pull/21191/head:
client: fix request send_to_auth was never really used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21255 into master

* refs/pull/21255/head:
mds: automatically create snaprealm for old format quota
client: check if mds create snaprealm when setting quota
mds: remove unused MDCache::snaprealm_create()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21295 from badone/wip-large-omap-scrub-big-random-loser-2

qa/suites/rados: Disable scrub backoff

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks: wait_for_clean is called after ceph task as well after osd's are up,
the default timeout is none in that case, there are cases where it can hang  forever
due to error cases, since this dumps quite a lot of info the logs grow in GB's, with
default timeout of 1200 we can avoid such huge logs and fail sooner. Any tests needing
higher timeout can pass the required value.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

doc: remove unused config: "osd op threads"

This option already removed.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #21309 from ceph/wip-yuriw-crontab

qa/tests - minor clean ups and made perf-suite run 3 times, so we can…

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #21282 from liewegas/wip-pgpool-enc

osd/osd_types: fix pg_pool_t encoding for hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Drymov <idryomov@redhat.com>

qa/tests - minor clean ups and made perf-suite run 3 times, so we can have more data points

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #21263 into master

* refs/pull/21263/head:
doc: outline upgrade procedure for mds cluster

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

mgr/dashboard: Move services provision to service module

Few services were being declared in the shared module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Api module

Moved all the api related services into the new module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20807 from Devp00l/improve-table-search

mgr/dashboard: Improve table search

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20943 from tspmelo/fix-e2e

mgr/dashboard: fix frontend e2e tests

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

common/options: add otp to default osd class load list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mfa documentation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/s3tests: configure mfa tests

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: require mfa only if mfa status actually changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add distro dependencies for liboath

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: fixes following review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls/otp: support for base32 seed type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp, rgw-admin: use older liboath api, parse seed once

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: preserve otp history when modifying it

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin, cls/otp: get current time for otp resync from osd

add a new method to the otp objclass that returns the
current time.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add mfa resync command

$ radosgw-admin mfa resync --uid=<uid> --totp=serial=<serial> \
                --totp-pin=<pin1> --totp-pin=<pin2>

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: integrate with liboath

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set objv_tracker for metadata mutate

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: use metadata mutate for mfa updates

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: better api for metadata mutation

Unlike the regular user and bucket metadata that are plain
raw rados objects, the otp info is controlled by object
class operations. The metadata manager mutate operation
will deal with the metadata related work that is needed
to happen (objv_tracker, update meta log), and call to
the operation that modifies the otp info.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tie otp meta get/put to meta log, objv_tracker

still need to have the radosgw-admin mfa tools to go through meta
instrumentation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial implementation of otp meta handlers

Still not there yet. Need to tie into objv_tracker, and metadata log
tooling.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: client api to set multiple keys

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: update header anyway, don't use omap header

Update header even if it might have existed. If running within a compound op
we might not identify it correctly.
Also don't use omap header api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work to add otp as metadata provider

To allow transparent multisite sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix mfa setting, verify on object deletion

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: S3 get bucket versioning returns MFA status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: functionality to modify mfa entries in user metadata

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: new mfa control commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add otp pool, create otp service functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: implement otp_info_t::dump()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_otp: treat empty header as -ENOENT

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: send totp check to the otp objclass

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: req_state members init refactor

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: add client interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: implement totp check logic

Currently checking for bogus results, still need to integrate with
totp library.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/otp: initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mfa - initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20758 from ceph/wip-cv-doc-typos

ceph-volume doc updates

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21249 from dillaman/wip-18844

rbd: improve 'import-diff' corrupt input error messages

Reviewed-by: Mykola Golub <mgolub@suse.com>

osd/OSDMap: propagate POOL_SNAPS|SELFMANAGED_SNAPS to tiers

We set this when we add selfmanaged snaps to the base tier.  When those
propagate to the cache tier, propagate the flags, too!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21196 from jeffrey4l/fix_typo

ceph-volume: fix typo in ceph-volume lvm prepare help

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21270 from Devp00l/ceph-table-action-fix

mgr/dashboard: Fix the data table action selector

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: restcontroller: fix detection of id args in element requests

Fixes: https://tracker.ceph.com/issues/23593
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21269 from b-ranto/wip-mgr-expose-rgw

mgr: Expose rgw perf counters

Reviewed-by: John Spray <john.spray@redhat.com>

qa/suites/rados: Disable scrub backoff

A long run of lost coin flips can lead to a timeout in
test_large_omap_detection.py.

Fixes: http://tracker.ceph.com/issues/23578
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

misc: fix various log messages

add space between __func__ and following messages

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

Merge pull request #21248 from dillaman/wip-23548

librbd: potential race between discard and writeback

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #17531 from smithfarm/wip-calamari-tests

tests: cleanup: drop calamari tasks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

Merge pull request #21245 from ifed01/wip-ifed-shard-bug-logging

os/bluestore: be more verbose when hitting unloaded shard in extent map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: define HAVE_STDLIB_MAP_SPLICING for both libstdc++ and libc++

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_types: fix pg_pool_t encoding for hammer

If we are missing the hammer feature, we will also be missing the
luminous feature, but we should still encode like hammer.

Analogous fix to e28e0c617af8825ae92cced5d87cc4f403709e48

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20866 from jan--f/mgr-fix-metadata-return

mon/MonCommands: mgr metadata - improve parameter naming consistency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21217 from tchaikov/wip-cmp-hobject

common/hobject: preserve the order of hobject

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21211 from tchaikov/wip-clang-without-map-merge

cmake,mds: detect std::map::merge() before using it.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21161 from cbodley/wip-mon-config-obs

config: notify config observers on set_mon_vals()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21151 from xiexingguo/wip-prune-olr

osd/PGBackend: drop input "snapid_t" from objects_list_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21150 from xiexingguo/wip-large-omap-obj-counting

osd/PGBackend: fix large_omap_objects checking

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21119 from Liuchang0812/refine-optracker-history

common, osd: duplicated "start" event in OpTracker, improve OpTracker::dump_ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21095 from wido/ceph-metadata-version

common: Add metadata with only Ceph version number and release

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21055 from tchaikov/wip-pr-20994

filestore,rgw: fix types/casts making clang on 32-Bit working

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: Clang does not (yet) support std::map::merge

@tchaikov found:
    libc++ still does not support "Splicing Maps and Sets",
    see https://libcxx.llvm.org/cxx1z_status.html ,
    search for "p0083r3" .

Make this case the more explicit coded.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: detect if std::map::merge() is supported

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21274 from dzafman/wip-cot-config

tools: Use --no-mon-config so ceph_objectstore_tool.py test doesn't hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20760 into wip-sage-testing-20180407.023110

* refs/pull/20760/head:
mon/ConfigMonitor: add 'config reset' command
mon/ConfigMonitor: add 'config log' command
mon/ConfigMap: add ConfigChangeSet
mon/ConfigMonitor: avoid no-op updates
mon/ConfigMonitor: maintain history of all config changes

Merge PR #20516 into wip-sage-testing-20180407.023003

* refs/pull/20516/head:
os/filestore: advertise vdo in filestore metadata
os/filestore: implement thin provisioning utilization stats for VDO
os/bluestore/KernelDevice: advertise vdo and physical_size in metadata
os/bluestore/KernelDevice: implement thin utilization stats for VDO
os/bluestore: hook to expose utilization of thinly-provisioned device
common/blkdev: identify VDO devices beneath LVM
common/blkdev: implement VDO helper functions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #20512 into wip-sage-testing-20180407.022843

* refs/pull/20512/head:
osd: osd_mon_report_interval_min -> osd_mon_report_interval, kill _max

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21033 into wip-sage-testing-20180407.022623

* refs/pull/21033/head:
buffer: kill push_front
tools/RadosDump: avoid bufferlist::push_front
os/filestore/FileJournal: avoid push_front
erasure-code: avoid bufferlist::push_front
buffer: kill claim_prepend
unittest_compression: avoid claim_prepend
os/bluestore: avoid claim_prepend
osdc/Striper: rewrite assemble_result without push_front or claim_prepend

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

src: Added const references to various function parameters

Added const references to various function parameters in order to avoid
copying data unnecessarily and enhancing performance

Signed-off-by: Wilson E. Alvarez <wilson.e.alvarez1@gmail.com>

Merge pull request #19594 from smithfarm/wip-doc-make-check

doc: expand developer documentation of unit tests

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>