doc: update config-key->config in mgr module docs

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update mgr plugin guide

...for recent config changes.  Explain about MgrModule.OPTIONS,
and explain the new store/config distinction.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update mgr test for MgrModule.OPTIONS

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add MgrModule.OPTIONS definitions to modules

They need these to do upgrades properly.  Where there
was an existing structure describing config, merge
these together to avoid having two lists of options.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: load MgrModule.OPTIONS and use it in upgrade

Modules will now only migrate items from config-key
into config options if they appear in the module's
schema.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: introduce MgrModule.OPTIONS field

Now is a good time to start requiring
modules to explicitly list their configuration
settings, so that we can do a proper job of
migrating configuration from old config-key style,
i.e. knowing what's a config setting and what's
a KV store item.

Throw an exception if a module tries to
access a setting outside their schema, so
that we have some confidence that the schema
is complete.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix up a couple of double-lookups

Signed-off-by: John Spray <john.spray@redhat.com>

qa: test mgr live configuration updates

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: use "config set" for mgr modules

...instead of pre-mimic config-key set.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/restful: migrate to new get_store interface

Signed-off-by: John Spray <john.spray@redhat.com>

mon: warn user doing "config key set mgr/..."

This is still completely legal, but isn't going
to have the effect they wanted if they were
trying to set a configuration option for a mgr module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: replace get_config_prefix with get_store_prefix

The _prefix variant was only used for data-ish things,
so we can just move it over to operate on store instead
of config, rather than having a _prefix variant for both.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: don't migrate config values that look like data

A quickie heuristic, but sufficient to avoid picking
up e.g. SSL certs and trying to cram them into
config values.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: grant mgr profile "config" commands

...and remove redundant config-key lines (these are applied
to mgr anyway in the next block, and mgr even has a broader
config-key permission in the line above).

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: clean up ActivePyModules logging

This was running the prefix and message together.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix PyModuleConfig::set_config

Commands were slightly wrong.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: remove dead update_perf_schema fn definition

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: rework kv store load path

The locking and blocking around this was a bit
tricky.  Do the simple thing, and pull the
load_store out to Mgr so that it can be safely
done as part of the background_init process (just drop
Mgr::lock across blocking actions).

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: update python interface for store vs. config

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: separate out module_config and store_cache

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: centralized setting/getting of mgr configs

Signed-off-by: Rubab Syed<rubab.syed21@gmail.com>

Merge pull request #21588 from xiexingguo/wip-fix-ec-ar-order

osd/PG: perfer EC async_recovery_targets in reverse order of cost

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21562 from LenzGr/mgr-dashboard-documentation

doc: updated dashboard feature list (added new RGW details, Pools)

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21564 from dillaman/wip-23809

rbd: cleanup handling of IEC byte units

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21408 from hsepeng/oisizemismatch-osd-bugfix

osd/PrimaryLogPG: fix the oi size mismatch with real object size

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: perfer EC async_recovery_targets in reverse order of cost

This is a follow-up fix of https://github.com/ceph/ceph/pull/21578,
in which I forget that erasure-coded-pools share the same logic
when determining the async_recovery_targets..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21549 from rjfd/wip-dashboard-fix-qa-whitelist

qa/suites/rados/mgr/tasks/dashboard: add MDS_ALL_DOWN to whitelist

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/mgr/tasks/dashboard: add MDS_(ALL_DOWN|UP_LESS_THAN_MAX) to whitelist

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21560 from jan--f/fix_strtoll_interface

common/strtoll: remove superfluous const modifier

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21550 from wjwithagen/wjw-kevent-fix

EventKqueue: Clang want realloc return to be typed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21579 from tchaikov/wip-test.sh-with-multi-mds

qa/workunits/cephtool/test.sh: fix test_mon_mds()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21578 from xiexingguo/wip-fix-async-recovery-peer

osd/PG: perfer async_recovery_targets in reverse order of cost

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21552 from smithfarm/wip-23805

qa/standalone/ceph-helpers.sh: provide argument to dirname

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21507 from liewegas/wip-rados-test

qa/workunits/rados/test.sh: ensure tee output is valid filename

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20921 from jcsp/wip-23385

mgr: improve reporting on unloadable modules

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20343 from trociny/wip-rados-ordersnap

librados: add OPERATION_ORDERSNAP flag and yet another aio_operate method

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21521 from tchaikov/wip-stoll

common/strtol: fix strict_strtoll() so it accepts hex starting with 0x

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

qa/workunits/cephtool/test.sh: fix test_mon_mds()

"ceph fs set cephfs allow_multimds false" is deprecated, and multimds is
enabled by default, so "ceph fs set cephfs max_mds 4" won't fail with
the default settings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: perfer async_recovery_targets in reverse order of cost

Theoretically peers which have a longer list of objects to recover
shall equivalently take a longer time to recover and hence have a
bigger chance to block client ops.

Also, to minimize the risk of data loss, we want to bring those broken
(inconsistent) peers back to normal as soon as possible. Putting them
into the async_recovery_targets queue, however, did quite the oppsite.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21358 from cbodley/wip-23663

rgw: fix use of libcurl with empty header values

http://pulpito.ceph.com/cbodley-2018-04-20_19:15:33-rgw-wip-cbodley-testing-distro-basic-smithi/

Merge pull request #21380 from cbodley/wip-23680

rgw: fix radosgw linkage with WITH_RADOSGW_BEAST_FRONTEND=OFF

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #21567 from cbodley/wip-23778

random: revert change from boost::optional to std::optional

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21455 from dmick/wip-ceph-volume

ceph-volume: Nits noticed while studying code

Merge pull request #21571 from tchaikov/wip-23517

cmake: hide symbols import from other libraries in libcls_*

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21573 from tchaikov/wip-cmake

cmake: s/boost_256/boost_sha256/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21570 from alfredodeza/wip-cv-doc-updates

ceph-volume update man page for prepare/activate flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

cmake: s/boost_256/boost_sha256/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21484 from tchaikov/wip-cmake-common-cleanup

cmake, common: silence cmake and gcc warnings

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: disable gflags support in rocksdb

to silence the warnings like

CMake Warning at CMakeLists.txt:73 (find_package):
  By not providing "Findgflags.cmake" in CMAKE_MODULE_PATH this project
has
  asked CMake to find a package configuration file provided by "gflags",
but
  CMake did not find one.

  Could not find a package configuration file provided by "gflags" with
any
  of the following names:

    gflagsConfig.cmake
    gflags-config.cmake

  Add the installation prefix of "gflags" to CMAKE_PREFIX_PATH or set
  "gflags_DIR" to a directory containing one of the above files.  If
"gflags"
  provides a separate development package or SDK, be sure it has been
  installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,make-dist: use sha256 instead of md5 for checksum

the boost's download page offers the SHA256 hash, so it'd be easier to
verify the hash this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: hide symbols import from other libraries in libcls_*

so they will not be involved when resolving symbols. ld tries to
keep a shared library around even if it fails to load it if it offers
some unique symbols. in that case, the library will not be properly
unloaded, and even worse it will interfere with following dlopen()
calls, because it is marked with NODELETE by dlopen(). if it has some
unresolved symbol and does offer some "unique" symbols required by
the library to be loaded, the library will fail to load, despite the
fact that the "unique" symbol is also offered by the executable.

for more details, see
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60731 and
https://sourceware.org/bugzilla/show_bug.cgi?id=14577

Fixes: http://tracker.ceph.com/issues/23517
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/ceph-volume update flags for activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume update flags for prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>

random: revert change from boost::optional to std::optional

somehow this was breaking the seeding of thread-local engines on gcc.
we'll have to investigate this further, but for now i'm reverting this
piece to get messengers working again

Fixes: http://tracker.ceph.com/issues/23778
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: updated dashboard feature list (added new RGW details, Pools)

Updated the dashboard feature list in the documentation to mention
the possibility to list Ceph pools and the RGW users and their buckets.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21516 from rjfd/wip-dashboard-rest-improvements

mgr/dashboard: RESTController improvements

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21353 from Devp00l/pool-list

mgr/dashboard: Pool listing

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21563 from idryomov/wip-krbd-lock-timeout-fixup

doc: lock_timeout is a per mapping option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd/import_export: update unit formatting

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd/cli_generic: update unit formatting

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/cli-integration/rbd: updated unit formatting

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: correct object size formatting

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: lock_timeout is a per mapping option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21522 from idryomov/wip-krbd-lock-timeout-test

qa: krbd_exclusive_option.sh: add lock_timeout test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21513 from idryomov/wip-krbd-notrim-test

 qa: krbd_fallocate.sh: add notrim test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Add pool listing

Adds a new top level menu entry, called "Pool", which lists all pools
and contains a minimalistic detail view.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: K-v-table uses autoReload input

The "autoReload" input from the data table can now be used in the key
value table as well.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #21517 from rjfd/wip-dashboard-fix-rbd-auth

mgr/dashboard: rbd: add @AuthRequired to snapshots controller

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20869 from votdev/dashboard_rgw_ro

mgr/dashboard: Add RGW user and bucket lists (read-only)

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

common/strtoll: remove superfluous const modifier

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21386 from votdev/api_interceptor

mgr/dashboard: Rename and refactor ApiInterceptorService class

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21531 from LenzGr/mgr-dashboard-documentation

doc: updated the dashboard feature list

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

qa/standalone/ceph-helpers.sh: provide argument to dirname

Fixes: http://tracker.ceph.com/issues/23805
Signed-off-by: Nathan Cutler <ncutler@suse.com>

EventKqueue: Clang want realloc return to be typed

Otherwise Clang start complaining:
src/msg/async/EventKqueue.cc:202:18: error: assigning to 'struct SaveEvent *' from incompatible type 'void *'
    sav_events = realloc(sav_events, sizeof(struct SaveEvent)*newsize);
                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.`

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc: updated the dashboard feature list

Updated and improved the Mgr Dashboard feature list: added RBD management
features.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #21499 into master

* refs/pull/21499/head:
doc/cephfs: update kernel client quotas support info

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21472 into master

* refs/pull/21472/head:
mds: optimize purge queue expire pos update

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21434 into master

* refs/pull/21434/head:
client: add client option descriptions

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21498 into master

* refs/pull/21498/head:
client: flush the mdlog in _fsync before waiting on unstable reqs

Reviewed-by: dongdong tao <tdd21151186@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21530 from jan--f/mgr-prometheus-fix-osd-metadata

pybind/mgr/prometheus: fix creation of osd_metadata metric

Reviewed-by: Boris Ranto <branto@redhat.com>

common/strtol: fix strict_strtoll() so it accepts hex starting with 0x

- simplify the error handling of strtoll().
- only instantiate ostringstream when handling errors.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: fix creation of osd_metadata metric

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21446 from smithfarm/wip-ctest-py3-fallback

cmake/mgr: use Python 3 virtualenv if mgr subinterpreter is Python 3

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21477 from liewegas/wip-kvstore-tool

ceph-kvstore-tool: make it a bit more friendly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21488 from tchaikov/wip-i-choose-to-die-instead-of-leak

msg/async/EventKqueue: assert on OOM

Reviewed-by: Haomai Wang <haomai@xsky.com>

mgr/dashboard: Add ability to cancel notifications.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Do not show notifications for HTTP 409 errors.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Rename auth-interceptor and refactor it to display notifications for more errors than 401 and 500.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #21496 from ceph/wip-rm23678

ceph-volume: allow skipping systemd interactions on activate/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com

mgr/dashboard: Modify RGW proxy response of errors to display a detailed message in the error notification.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Force datatable to hide the loading indicator in case of an error.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add RGW user and bucket lists (read-only)

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: rbd: add @AuthRequired to snapshots controller

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21466 from jcsp/wip-dashboard-cmake-all

cmake: include frontend build in 'all' target

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: restcontroller: removed @args_from_json

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21497 from votdev/forceidentifier_datatable

mgr/dashboard: Add 'forceIdentifier' attribute to datatable

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21469 from liewegas/wip-log-nomsgs

mon/LogMonitor: do not crash on log sub w/ no messages

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21467 from tchaikov/wip-perf-stats-ns-incarnation

include/ceph_features: fix OS_PERF_STAT_NS's incarnation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #21459 from runsisi/wip-mgrc-register

mgr/MgrClient: service registration filtered by service name instead of daemon name

Reviewed-by: John Spray <john.spray@redhat.com>