mgr/dashboard: Takeover PR #20799 and adapt comments.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Implement RGW proxy

This implementation is basically a Rados Gateway reverse proxy.  It
additionally takes care of the authentication to the Rados Gateway, but to use
it you will have to be authenticated against the dashboards RESTful API.

The corresponding credentials can be configured using the following commands:

    dashboard set-rgw-api-secret-key <secret-key>

    dashboard set-rgw-api-access-key <access-key>

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: RGW proxy: Include `python-requests-aws`

This commit will include the `python-requests-aws` [1] library.  This library
extends the `requests` library by an authentication mechanism to S3 gateways.

[1] https://github.com/tax/python-requests-aws

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #21022 from sebastian-philipp/dashboard_get_range

mgr/dashboard: Refactor multiple duplicates of `get_rate()`

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #20920 from sebastian-philipp/dashboard_v2-erasure-code-profile

mgr/dashboard: Add ErasureCodeProfile controller

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21086 from ricardoasmarques/fix-table-without-fetch-data

mgr/dashboard: Fix table without fetchData

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge pull request #21041 from votdev/custom_validators

mgr/dashboard: Add custom validators

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21078 from tspmelo/wip-notifications

mgr/dashboard: Add notification service/component

Reviewed-by: Volker Theile vtheile@suse.com
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Loading indicator only if fechData provided

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Hide refresh button if no fetchData provided

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add custom validators.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #19220 into master

* refs/pull/19220/head:
mds: limit run time of load balancer
mds: add list to track recently used sub-directories
mds: calculate other mds' last_epoch_under locally
mds: cleanup mds_load map access/update
mds: check export pin when choosing dirfrags for exporting
mds: optimize MDBalancer::find_exports()
mds: avoid creating unnecessary subtrees during load balance
mds: mds: optimize MDBalancer::try_rebalance()
mds: don't try exporting subdir if dirfrag is already being exported
mds: don't try exporting dirfrags under mds's own mdsdir
mds: cleanup MDBalancer::try_rebalance()
mds: always pass current time to MDBalancer::{hit_inode,hit_dir}
mds: remove unused MDBalancer::export_empties()
mds: don't decay load counters when printing load
mds: adjust pop_nested when importing subtree
mds: adjust subtree popularity when import fails
mds: adjust subtree popularity after rename
mds: fix request rate calculation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21310 from tchaikov/wip-23610

osd: resend osd_pgtemp if it's not acked

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20947 from dzafman/wip-23364

Special scrub handling of hinfo_key errors

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: Add release notes for the output format changes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Be smarter about when jsonschema can be used

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Show snapset in list-inconsistent-snapset

Add SnapSet bufferlist to inconsistent_snapset_t

Partial fix for http://tracker.ceph.com/issues/23428

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: SnapSet::dump() handle snapid_t not in clone_overlap or clone_size

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add getjson variable to save output

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Change error name snapset_mismatch to snapset_error

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Clean-up test and use local values for number of objects and osds

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Without snapdir we don't have a head mismatch error anymore

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: list-inconsistent-obj attribute improvements

System attributes shown as "object_info", "snapset" and "hashinfo"
Only output user attributes as "attrs"
Drop leading undescore "_" for user attribute keys
Improve logic as to when to show user attributes or specific system attributes

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Change "oi" to "info" in scrub handling errors

data_digest_mismatch_oi -> data_digest_mismatch_info
omap_digest_mismatch_oi -> omap_digest_mismatch_info
size_mismatch_oi -> size_mismatch_info
obj_size_oi_mismatch -> obj_size_info_mismatch

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Change "oi_attr" to "info" in scrub handling errors

oi_attr_missing -> info_missing
oi_attr_corrupted -> info_corrupted

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Rename ss_attr_missing/ss_attr_corrupted to snapset_missing/snapset_corrupted

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Improve scrub handling of HashInfo (hinfo_key xattr)

Fixes: http://tracker.ceph.com/issues/23364
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't crash in get_hash_info() with a corrupt HashInfo

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Remove check that masks differences (let diff fail)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #21321 into master

* refs/pull/21321/head:
tools/cephfs: instantiate MDSUtility after global_init()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21130 from ceph/wip-rm23067

ceph-volume  Implement an 'activate all' to help with dense servers or migrating OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21304 from rjfd/wip-rbd-pybind-fixes

rbd: python bindings fixes and improvements

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21272 from cbodley/wip-rgw-beast-experiment

rgw: beast frontend no longer experimental

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #21120 from oritwas/wip-rgw-21619

rgw: reshard cancel command should clear bucket resharding flag

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: beast frontend no longer experimental

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #21302 into wip-sage-testing-20180410.153118

* refs/pull/21302/head:
qa/suites/rados/singleton/all/random-eio: whitelist eio error message

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #21315 into wip-sage-testing-20180410.153007

* refs/pull/21315/head:
qa/tasks/mgr/test_failover: fix for id -> name rename in mgr metadata

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21324 from tchaikov/wip-23495

qa/suites: whitelist SLOW_OPS

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: Replace toastr calls with notification service

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add notification component

This component is used on the navigation bar and will show the user the 10 most
recent notifications.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add notification service

This service should be used to display any notification/toast to the user.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #19283 from yehudasa/wip-rgw-mfa

rgw: mfa support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21327 from LenzGr/dashboard-fork-awesome

mgr/dashboard: Replace font-awesome with fork-awesome

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge PR #21080 into master

* refs/pull/21080/head:
src: Added const references to various function parameters

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21126 from tspmelo/wip-services

mgr/dashboard: Add Api module

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Replace font-awesome with fork-awesome

Replaces the dependency on Font Awesome with Fork Awesome
(https://forkawesome.github.io/), which is a plug-in replacement
for font-awesome 4.7.0 and adds a few more icons (see
https://forkawesome.github.io/Fork-Awesome/whats-new/ for
background information about this fork).

It is licensed under the following licenses - see
https://forkawesome.github.io/Fork-Awesome/license/ for details:

Fonts: SIL OFL 1.1
Code: MIT License
Documentation: CC BY 3.0

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

osd,mds: use map::merge() whenever appropriate

and use explicit std::move() when map::merge() is not available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: resend osd_pgtemp if it's not acked

if the osd_pgtemp message is dropped before monitor receives it, we need
to resend it. otherwise a pg could be stuck in activating state if the
pg creation was withheld by the max-pg-per-osd on the replica, and then
the replica osd removes some existing pg.

Fixes: http://tracker.ceph.com/issues/23610
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21319 from majianpeng/doc-remove-osd-op-threads

doc: remove unused config: "osd op threads"

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites: whitelist SLOW_OPS

Fixes: http://tracker.ceph.com/issues/23495
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: warn user about on going resharding when canceling it

Fixes: http://tracker.ceph.com/issues/21619
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: no need to RGWReshard::get before remove

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21112 from guzhongyan/fix-vlog

src: fix various log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21028 from rzarzynski/wip-osd-get_rw_locks-norefcnt

osd: avoid unnecessary ref-counting across PrimaryLogPG::get_rw_locks.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21235 from xiexingguo/wip-fix-obj-start

osd/osd_types: fix ideal lower bound object-id of pg

Reviewed-by: Sage Weil <sage@redhat.com>

tools/cephfs: instantiate MDSUtility after global_init()

cephfs-foo-tool from current master crashes immediately it starts.

Introduced by commit 6972273d53d "global: output usage on -h, --help,
or no args before contacting mons"

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/23624

Merge pull request #21313 from ceph/wip-pg-clean-timeout

qa/tasks: Add default timeout for wait for pg clean task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21284 from tchaikov/wip-cmake-libstdcxx

cmake: define HAVE_STDLIB_MAP_SPLICING for both libstdc++ and libc++

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21290 from rjfd/wip-dashboard-fix-rest-controller

mgr/dashboard: restcontroller: fix detection of id args in element requests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21039 into master

* refs/pull/21039/head:
qa/workunits/fs: test for cephfs rstats
mds: make rstat.rctime follow inodes' ctime

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21040 into master

* refs/pull/21040/head:
mds: mark damaged if sessions' preallocated inos don't match inotable

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21132 into master

* refs/pull/21132/head:
client: remove getgroups_cb
client: remove _getgrouplist
client: have init_gids just set alloced_gids to true
client: remove init_groups
fuse: handle errors appropriately when getting group list

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21144 into master

* refs/pull/21144/head:
mds: fix unhealth heartbeat during rejoin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21167 into master

* refs/pull/21167/head:
mds: make stopping mds export strays and subtree at the same time
mds: handle client requests while mds is stopping
mds: check projected linkage before migrating stray dentry

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21191 into master

* refs/pull/21191/head:
client: fix request send_to_auth was never really used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21255 into master

* refs/pull/21255/head:
mds: automatically create snaprealm for old format quota
client: check if mds create snaprealm when setting quota
mds: remove unused MDCache::snaprealm_create()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/test_failover: fix for id -> name rename in mgr metadata

This was changed by 428236de8b4155beffb5a4ffa8dd206230325a01

Signed-off-by: Sage Weil <sage@redhat.com>

mds: limit run time of load balancer

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: add list to track recently used sub-directories

MDBalancer::find_exports() can be more efficient with LRU list.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: calculate other mds' last_epoch_under locally

No need to get this information from MHeartbeat

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: cleanup mds_load map access/update

using map::insert to access map items looks strang

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: check export pin when choosing dirfrags for exporting

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: optimize MDBalancer::find_exports()

stop at subtree bounds; make freezing/frozen check more efficient.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: avoid creating unnecessary subtrees during load balance

When choosing dirfrags for exporting, consider subtrees imported
from other MDS first. It's because that exporting dirfrags chosen
by MDBalancer::find_export() will create new subtrees.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: mds: optimize MDBalancer::try_rebalance()

1. change import_pop_map to multimap because subtrees may have the same
   popularity.
2. avoid calculating subtrees' popularity multiple times

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't try exporting subdir if dirfrag is already being exported

It doesn't make sense to call MDBalancer::find_exports() for a dirfrag
that is already being exported.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't try exporting dirfrags under mds's own mdsdir

Migrator will refuse to export them. Besides, stray directory can
never be subtree root.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: cleanup MDBalancer::try_rebalance()

no logic changes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: always pass current time to MDBalancer::{hit_inode,hit_dir}

There were codes that pass mdr->get_mds_stamp() to {hit_inode,hit_dir}.
The stamp can be older than 'last_decay' in decay counters,  which can
cause DecayCounter::decay() to malfunction.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: remove unused MDBalancer::export_empties()

MDCache::trim() already does the same job

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't decay load counters when printing load

Changing load counters while printing load results confusing output.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: adjust pop_nested when importing subtree

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: adjust subtree popularity when import fails

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: adjust subtree popularity after rename

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix request rate calculation

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/21745

Merge pull request #21295 from badone/wip-large-omap-scrub-big-random-loser-2

qa/suites/rados: Disable scrub backoff

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks: wait_for_clean is called after ceph task as well after osd's are up,
the default timeout is none in that case, there are cases where it can hang  forever
due to error cases, since this dumps quite a lot of info the logs grow in GB's, with
default timeout of 1200 we can avoid such huge logs and fail sooner. Any tests needing
higher timeout can pass the required value.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph-volume lvm.activate error if no bluestore OSDs are found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add 'activate all' for xenial/filestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add 'activate all' for xenial/bluestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add 'activate all' for centos7/filestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add 'activate all' for centos7/bluestore/dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add 'activate all' in filestore playbook

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: remove unused config: "osd op threads"

This option already removed.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph-volume tests/functional add 'activate all' in bluestore playbook

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21309 from ceph/wip-yuriw-crontab

qa/tests - minor clean ups and made perf-suite run 3 times, so we can…

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #21282 from liewegas/wip-pgpool-enc

osd/osd_types: fix pg_pool_t encoding for hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Drymov <idryomov@redhat.com>

qa/tests - minor clean ups and made perf-suite run 3 times, so we can have more data points

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #21263 into master

* refs/pull/21263/head:
doc: outline upgrade procedure for mds cluster

Reviewed-by: Douglas Fuller <dfuller@redhat.com>