mgr/DaemonServer: 'ceph config show-with-defaults'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: get config defaults on session open

Leave them encoded until they are used, since usually we won't use them.
This is about 50k currently.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: fix daemon registration

We want to take in the initial config for non-service daemons, too!
Restructure this so it makes more sense.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: get_defaults_bl()

Signed-off-by: Sage Weil <sage@redhat.com>

osd,mds: 'config unset' tell command

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: implement 'config unset <var>'

Clear an override value (if any).  This works for 'ceph daemon ...'
asok commands.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: show fabricated options as level 'unknown'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: include LEVEL column in config dump|get

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMap: const * to Option, so we don't have to copy it

These are biggish structs (lots of strings), so it's expensive to make
copies of Option when it is an immutable struct.  However, we need to
handle arbitrary strings that we don't recognize, too.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: have MgrStandby send mgr reports to active mgr (even self!)

This allows the mgr daemons to register state with the active mgr just
like the rest of the cluster, including perfcounters and current config.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: 'config help'

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: fix dump()

By convention dump() does not include the containing object section.

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: operator<< for value_t

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: move debug values over to mon config

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-osd: do not set/override osd_objectstore config option

We use the one on disk, and nobody consumes the option except ceph-volume/
ceph-disk during mkfs.  Do not pollute the config space with a value that
isn't consumed.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: "conffile" -> "file"

Easier on the eyes

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: exclude host, fsid from config reproted to mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: load_config() a bit less often

We could get very clever and avoid reparsing the full config every time
too, but that is a bit more work.  Leave that as a TODO.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: implement 'config {rm,get,dump}'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: remove old 'config set' command

This is a cli command to set a config option on the mgr.  It doesn't
really belong because it only sets it on the mgr that happens to be
active.  It was probably mixed up with the asok or tell command.

Added in 057b73d641decb9403aba50caae9d139f3a34dd4

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: one-shot mon connection on started to fetch config

This is not particularly efficient, but it works:

 - connect to the monitor to get the monmap and config
 - tear it all down
 - proceed with normal startup (which presumably involves reconnecting
   to the mon all over again).

This allows us to set config options that may affect the mon communication
itself, like ms_type.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: always send initial config, even if empty

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: count crypto/shutdown cycles

We need to allow crypto to be set up and torn down before a fork.  Count
the init/shutdown cycles to make that safe.

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

mon: handle MGetConfig

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: use mon config for many things

We also trim out some obsolete options while we're at it.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'config set <who> <name> <value>'

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: refresh mon configs too

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMap: some improvements

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMap: move option mask parsing into helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: 'config show <who>'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: report running config to mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: add 'prefix' arg to 'config-key dump ...'

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: less config noise

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: remove obsolete 'crushtool' option

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: subscribe to config, handle updates

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: add monitor service for distributing configs

Add 'config' subscribe option to get MConfigs on auth and when config is
updated.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMap: parsed cluster-wide config

Breaks down options by global, type, daemon, and can filter options by
crush location.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MConfig: send config map

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: set_mon_vals()

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: some cleanup; rm_val

Signed-off-by: Sage Weil <sage@redhat.com>

cleanup: src/.libs -> build/lib

Clean up remaining traces of automake's .libs.  This makes unittest_*
tests easier to run by hand (mostly).

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: rewrite (╯°□°)╯︵ ┻━┻

This is too complete a rewrite to reasonably break down into small steps,
and even if I could, it would be harder to review that way than to simply
review the new implementation.  The semantics of the old one were so weird
that it's harder to reason about the change in behavior than to simply
review the new behavior.

That's my story, at least, and I'm sticking to it!

So, here are the highlights:

 - $foo meta expansions are evaluated at get_val() time.  This means the
weird bool arguments to set_val specifying whether things were expanded
are removed (they didn't make any sense unless you were thinking about the
old implementation).
 - for every option, we track values from any inputs (config, mon,
   override).  At get_val() time, we pick the highest priority one.
 - diff() is rewritten to be simple and to show you all of the above.
 - internal interfaces are simplified, and in terms of Option::value_t
   whenever possible.
 - unit tests simplified somewhat based on the above.

Known issues:

 - legacy values get pushed out in select cases.  Notably if foo=$bar
   and bar is updated, we do not update $foo (there is no dependency
   tracking to do this efficiently).

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: streamline get_val_generic

Lookup value first; avoid a schema lookup entirely.

Signed-off-by: Sage Weil <sage@redhat.com>

common: refactor default config value handling

- pass an optional map<string,string> instead of an arg list
- remember default values in md_config_t
- some nicer helpers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass rocksdb options via map, not global config

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: pass kv_options map to ctor

This will let us avoid passing options via global config options.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: make debug_* regular options

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: get_full_location by name

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: move parse_value() into Option

It is more reuable there.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: new get_val() variant

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: record is_daemon

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: find_option

Signed-off-by: Sage Weil <sage@redhat.com>

common: don't pass alt_def_args if we don't use them

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: convert bool safe to a flags field

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigKeyService: make STORE_PREFIX public

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20681 from liewegas/wip-fork-vs-msgr

common,rbd-nbd: fix up prefork behavior vs AsyncMessenger singletons

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17530 from smithfarm/wip-21264

tools: cleanup: rip out ceph-rest-api

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #20688 into wip-sage-testing-20180306.131906

* refs/pull/20688/head:
doc: Drop the output of pg query
doc: Update Monitoring OSDs and PGs

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20718 from ceph/wip-rm23163

ceph-volume tests alleviate libvirt timeouts when reloading

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20726 from b-ranto/wip-doc-upgrade

build-doc: Upgrade ceph python libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20103 from openattic/wip-mgr-dashboard_v2

mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard)

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20659 from tchaikov/wip-msg/async/dpdk/launch-on-coreid

msg/async: execute on core specified by core_id not its index

Reviewed-by: shangfufei <shangfufei@inspur.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

tests: workunits: drop rest/test.py workunit

This workunit is only used by the two rest-api test cases that are also being removed.

Fixes: http://tracker.ceph.com/issues/21264
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: rados suite: drop rest-api test cases

Fixes: http://tracker.ceph.com/issues/21264
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20519 from gregsfortytwo/wip-22882-linger-locking

osdc/Objecter: take budgets across a LingerOp instead of on child Ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osdc: create and use take_linger_budget for prefetching watch/notify budgets

We don't actually take any budget, as we don't want to hit the
total op limits (especially on eg rbd-mirror), but this prevents
our per-message budgeting from causing a deadlock.

Fixes: http://tracker.ceph.com/issues/22882
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20517 from gregsfortytwo/wip-22114-beacon-blocking-2

mon: mark OSD beacons and pg_create messages as no_reply

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #20720 from ctrlaltdel/typo

doc: Fix typo s/applicatoin/application/

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20535 from ifed01/wip-ifed-bring-fio

test/fio: enable objectstore FIO plugin building without the need to install and build FIO source code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20708 from majianpeng/bluefs-bug-fix

os/bluestore: fix wrong usage for BlueFS::_allocate.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix wrong usage for BlueFS::_allocate.

In fact, function _allocate is atomic function which don't
allocate less than want space.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueFS: cleanup code for more readable.

And remove the useless judgement.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #20699 from yaozongyou/fix-potential-memory-leak

common: fix potential memory leak in HTMLFormatter

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20701 from tchaikov/wip-23212

os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20544 from liewegas/wip-min-kv

os/bluestore: kv_max -> kv_min

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20713 from smithfarm/wip-spdk-python3

spdk: advance to upstream dc82989d

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20131 from jcsp/wip-doc-encoding

doc/dev: add a brief guide to serialization

Reviewed-by: Kefu Chai <kchai@redhat.com>

build-doc: Upgrade ceph python libraries

If you have python ceph bindings installed on your system, the doc-build
script will fail since pip detects them and fails to install the updated
bindings in the virtualenv. This fixes the issue by adding --upgrade
flag to pip so that it overrides the system-installed libraries.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #20717 from b-ranto/wip-restful-doc

doc: Improve mgr/restful module documentation

Reviewed-by: John Spray <john.spray@redhat.com>

build/ops: deb: move python-jinja2 dependency to mgr

87399bea8321a7a9ab554085be3a3d5e0563f27f introduced an explicit dependency on
python-jinja2, but mistakenly as an overall build dependency instead of as a
runtime dependency of ceph-mgr as intended.

Fixes: http://tracker.ceph.com/issues/22457
Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: PendingReleaseNotes: deprecate/drop ceph-rest-api

References: http://tracker.ceph.com/issues/21264
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: cleanup: rip out ceph-rest-api

Obsoleted by the mgr REST API.

Fixes: http://tracker.ceph.com/issues/21264
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20542 from liewegas/wip-bluefs-buffered-io

common/options: bluefs_buffered_io=true by default

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #20721 from idryomov/wip-krbd-parent-overlap-test

qa: krbd parent-overlap test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/fio: enable objectstore FIO plugin building without the need to install and build FIO source code

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20695 from mikechristie/wip-doc-mnc-update-iscsi-kernel

doc: update iSCSI upstream kernel to 4.16

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20663 from tchaikov/wip-ceph.in-py3

ceph.in: use a different variable for holding thrown exception

Reviewed-by: John Spray <john.spray@redhat.com>

Doc: Fix typo s/applicatoin/application/

Signed-off-by: Francois Deppierraz <francois@ctrlaltdel.ch>

Merge pull request #20714 from idryomov/wip-krbd-msgr-segments-test

qa: krbd msgr-segments test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Fixes: http://tracker.ceph.com/issues/23212
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Improve mgr/restful module documentation

Fixes: https://tracker.ceph.com/issues/23219
Signed-off-by: Boris Ranto <branto@redhat.com>

qa: krbd parent-overlap test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard_v2: added script to run backend API tests locally

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/suites/mgr/dashboard_v2: added suite yaml for testing dashboard_v2

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/tasks/mgr/dashboard_v2: implementation of backend API tests

We moved the dashboard_v2 test cases that required a real ceph cluster
to run into qa/tasks/mgr with the necessary adaptations.
These tests can now be run in teuthology using the cephfs_test_runner
task.

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/tasks/mgr: move test initialization to setUpClass method

With this change, we avoid the disabling/enabling of the ceph-mgr module
being tested for each test function declared in each test case. Now
the ceph-mgr module being tested is disabled/enabled only once for each
test case.

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard_v2: Added smoke test.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>