mgr/dashboard_v2: add mocked service provider for TcmuIscsiService

This removes the warnings shown when running the tests with PhantomJS.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20768 from tchaikov/wip-cmake-dashboard

cmake,test/mgr: restructure dashboard tests and cmake related fixes

Reviewed-by: Ricardo Dias <rdias@suse.com>

test/mgr: restructure dashboard tests

* restructure dashboard tests so they follow the model of other tests
  based on ceph-helpers.sh. i.e. extract the TEST_* out of run()
  function.
* expose mon-host using CEPH_ARGS env variable.
* mgr-dashboard_v2-smoke.sh: do not export CEPH_ARGS in a subshell, so
  the run_mgr() and ceph cli can also have access to the updated
  CEPH_ARGS, and hence they will be able to locate the mon. and because
  auth-supported=none is populated using CEPH_ARGS, this test won't fail
  because of missing keyring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: run 'npm' command in a directory specified by absolute path

in case developer wants to run the test in "./build" or some random
directory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for "npm" before using it

and allows use to specify the path in which npm is installed. in my
case, if $HOME/.npmrc points "prefix" to ${HOME}/.npm-packages, i will
be able to run

NPM_ROOT=${HOME}/.npm-packages cmake ..

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20742 from votdev/perf_counter_units

mgr/dashboard v2: Add units to performance counters

Merge pull request #20744 from votdev/fix_loading_indicator

mgr/dashboard v2: Display loading indicator in datatables during first load

Merge pull request #20746 from votdev/datatable_selection

mgr/dashboard v2: Introduce CdTableSelection model

Merge PR #20472 into master

* refs/pull/20472/head:
mds: add DUMP_DIRFRAGS to CInode and DUMP_ITEMS to CDir
mds: Enhance dump_tree performance by traversing the tree directly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20502 into master

* refs/pull/20502/head:
test: cap waits in delegation tests at 60s
tests: add a test for doing a getattr while caps are recalled but before returned
client: allow getattr, lookup, fstatx, and readdir to use implemented caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20541 into master

* refs/pull/20541/head:
ceph-mds: exit if invalid id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20575 into master

* refs/pull/20575/head:
mds: set journaler iohint correctly when mds daemon going to active

Reviewed-by: dongdong tao <tdd21151186@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20583 into master

* refs/pull/20583/head:
cephfs-journal-tool: add event time output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20615 into master

* refs/pull/20615/head:
mds: modify help info of dump_historic_ops and dump_historic_ops_by_duration
mds:fix typo in MDSRank

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20172 from liewegas/wip-config

mon: centralized config

Reviewed-by: John Spray <john.spray@redhat.com>

common/config: normalize key name for get_val (external) variants

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: -c to ceph cli

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: process subs from update_from_paxos

This is what OSDMonitor and MDSMonitor do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: fix auth timeout vs error race

It's possible that we successfully set active_con *and* time out the
cond WaitUntil.  Only set the error if we don't have a connection; if we
set it *and* time out then let's call it success.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: fix dump when by_type is missing an item

Signed-off-by: Sage Weil <sage@redhat.com>

common/common_init: use unique admin_socket path for unprivileged daemons

These qualify as 'daemon', but their path is usually not unique.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: intercept "keyfile", translate into "key"

The keyfile arg might be - (stdin), which we can only read once.  Ensure
that we consume it once by intercepting the CLI value early and inserting
the value into the 'key' option.

This robs future code of the knowledge that the key came from --keyfile
and not --key, but avoids the issue of multiple users (notably, KeyRing.cc
and the OSD mkfs code).

Remove the - special case from OSD at the same time, since it can no
longer be reached (unless something other than the CLI specified '-', but
neither ceph.conf nor the mon config make sense here).

Signed-off-by: Sage Weil <sage@redhat.com>

librados: fix common_init_finish timing

Common_init_finish does start_service_thread and does
set_safe_to_start_threads() on the cct, which switches us to 'runtime'
mode where we can't accept many config options. Do that *after* we fetch
our config from the mon so that we can accept+set runtime options (and
not complain to stderr about it).

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: check against raw value (no meta) to detect unchanged option

If we are looking at a new value from the mon and comparing it to what we
already have active, compare the non-meta-substituted form.  This way a
value from the mon that can't update at runtime but we have already set to
the same value will not be falsely flagged as ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: add missing #include, tweak types

Seems to resolve a build error on some compilers!  Meh.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/ceph-conf: fix test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rebuild_mondb.py: --no-mon-config to ceph-objectstore-tool

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: add --mon-config option (opposite of --no-mon-config)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: apply timeout while fetching config

The normal timeouts automatically apply during the authenticate() stage,
but not to the explicit wait for a config.  If we don't get that quickly
we shoudl retry another monitor because it is possible we will connect to
an out-of-quorum (or otherwise unresponsive) mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: tolerate pre-mimic mons

Return success if the mons are pre-mimic (and thus have no config).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: --no-mon-config

Some of these cases make sense to fetch mon configs, but we are deprecating
ceph-disk anyway, and the tests currently make use of ceph-disk in places
that do not have a mon_host defined via a ceph.conf or other environment.
This avoids breaking those test cases without any real impact on users
(which will either use ceph-volume or presumably remain in a legacy config
environment).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: fix mon_caps

Pass --no-mon-config so we don't get

 failed to fetch mon config (--no-mon-config to skip)

instead.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: --no-mon-config for ceph-objectstore-tool --op mkfs ..

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: infer osd id to authenticate

In order to get mon configs we need to authenticate as somebody.
Infer it from the data directory.

Note that this won't work for --op mkfs.  Those users need to pass
--no-mon-config or -n.

Signed-off-by: Sage Weil <sage@redhat.com>

global/global_init: send startup errors to cerr, not log

(Flush the log, too, as something that has useful error messages.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: --no-mon-config during ceph-osd mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: use assimilate-conf to ingest config options quickly

Separate 'config set' commands is slow.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: only send config_bl if it has changed

The mgr already treats the config_bl as optional in the MMgrReport
messages.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix osd bootstrap

Install the key before trying to use it for mkfs.  Among other things
this means the osd can fetch it's config from the mon before doing the
mkfs step.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: require authentication but no caps for monmap and config

Signed-off-by: Sage Weil <sage@redhat.com>

mon: clean up dealth_with bool weirdness

Just return!

Signed-off-by: Sage Weil <sage@redhat.com>

mon: require authentication for MGetConfig and mon metadata

More importantly, the set of non-authentication operations that are allowed
is more restricted: auth messages, getting monmap, ping.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add authenticated flag in Session

Authentication is orthogonal to the caps; the client might have no
capabilities but still be authenticated such that we know *who* it is.

Signed-off-by: Sage Weil <sage@redhat.com>

test/mgr/mgr-dashboard-smoke: don't clobber CEPH_ARGS for other daemons

Signed-off-by: Sage Weil <sage@redhat.com>

global/global_init: apply_changes() for config fetch session

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: exclude unchanged defaults from diff

If we have a CONF_DEFAULT in the values map we do not need to include it
in the diff!

Signed-off-by: Sage Weil <sage@redhat.com>

test/run-cli-tests: CEPH_ARGS="--no-mon-config"

These are all offline tool tests.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: parse --no-mon-config into md_config_t, not CephInitParameters

We want to accept this in $CEPH_ARGS, which makes md_config_t a better
place to keep it.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: tweak order of env parsing

CEPH_ARGS is half-way between env and cmdline; parse it at the end of
env.

Signed-off-by: Sage Weil <sage@redhat.com>

test/clie/ceph-conf: fix loop detection output test

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: use cerr for --show-config-value

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: normalize the default value

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rebuild_mondb: --no-mon-config for ceph-monstore-tool

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore/test_fuse.sh: no mon config for ceph-objectstore-tool

There isn't a running cluster for this test.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: streamline test_mon_injectargs

Be less sensitive to formatting of output string on set commands.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: implement tell/daemon 'config get'

Signed-off-by: Sage Weil <sage@redhat.com>

rados: only suggest changing mon_allow_pool_delete on EPERM

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: only common_init_finish() once per cct

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: 'developer' -> 'dev'

I like shorter.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/configuration: big update

- remove a bunch of outdated info
- update to reflect new monitor-based configuration
- move a bunch of common settings to another document/page

Signed-off-by: Sage Weil <sage@redhat.com>

qa: test config CLI interface

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: fix stringification

There is a generic operator<< for boost::variant--we can't overload it
for Option::value_t.  Make a to_str() method instead.

Add a parse_option() output arg for the normalized value for convenience.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: indent 'config dump'

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/config-key: document how we structure things in config-key

Signed-off-by: Sage Weil <sage@redhat.com>

common/config,mgr: fix config for new encode/decode syntax

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMgrOpen: fix decode

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: validate and normalize option on 'config set'

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: flag a bunch of pool settings for RUNTIME

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: 'config (get|show)' RW -> RO

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: eliminiate global osd_pg[p]_bits option

osdmaptool is the only user.

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: some osd startup-only options

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: a few more [CLUSTER_]CREATE options

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: replace 'mkfs' tag with FLAG_[CLUSTER_]CREATE

Signed-off-by: Sage Weil <sage@redhat.com>

common/options,ceph_test_msgr: unmark misc ms options as RUNTIME

These were marked RUNTIME for the benefit of the test.  Replace with a
no-op observer.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config_obs: NoopConfigObserver

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: replace FLAG_SAFE with FLAG_{RUNTIME,STARTUP,CLUSTER_CREATE,DAEMON_CREATE}

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: is_safe() -> can_update_at_runtime()

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: show changeable for 'config get|show'

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: 'can update at runtime' in 'config help'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: add IGNORES column to 'config show'

If the daemon is ignoring a mon config, show that.  Note that this
column only ever says 'mon', which might be confusing to users.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: report failed mon config values to mgr

If we can't set a mon config value, report that back to the mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: move unsafe runtime change into _set_val

We want this to apply to other callers, set_mon_vals() in particular.

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: drop 'safe' flag on plugin dir options

These were only marked safe to accomodate the unit test CEPH_LIB hackery,
which is now removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: 'config show <who> <key>'

Signed-off-by: Sage Weil <sage@redhat.com>

mon: 'ceph config get <who> <key>'

Simple command to get a config value for a given entity (either the set value or the
default), similar to ceph-conf or --show-config-value.

Signed-off-by: Sage Weil <sage@redhat.com>

test: CINIT_FLAG_NO_MON_CONFIG or NO_DEFAULT_CONFIG_FILE

Usually we use NO_DEFAULT_CONFIG_FILE, but for those tests that need the
plugin directory overrides in order to work properly when run manually
(e.g., "bin/unittest_compression"), read the config and only skip the
mon config.

The NO_DFEAULT_CONFIG_FILE (which also skips the mon) would work for
everything when run via ctest, but as a dev one often runs the unittest
manually and having to pass CEPH_LIB=lib is annoying.  Note that
do_cmake.sh already writes out a trivial ceph.conf with these options
set for us.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_daemon_config: add needed rm_val()

Otherwise we polluate later tests.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk/tests: simplify CEPH_LIB hack

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone: drop CEPH_LIB hacks

Signed-off-by: Sage Weil <sage@redhat.com>

test: drop hackery for CEPH_LIB

Generic config code parses this now as part of global_init.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: interpret CEPH_LIB in parse_env()

Apply to the three plugin directory options.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/man/8/ceph-conf: make not about ceph-conf's lack of mon config support

Steer people toward the mon explicitly or the --show-config-value option.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: comment config levels

Signed-off-by: Sage Weil <sage@redhat.com>

drop unnecessary env_to_vec calls

global[_pre]_init does this for us now.

Signed-off-by: Sage Weil <sage@redhat.com>

librados,libcephfs,...: use parse_env()

Signed-off-by: Sage Weil <sage@redhat.com>

common/global_init: parse environment

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: take level to parse_argv

A few callers are explicitly parsing args from the environment.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: distinguish between cmdline and override (injectargs)

This makes for a more informative 'ceph config show ...'.

Signed-off-by: Sage Weil <sage@redhat.com>

global/global_init: make --show-config and --show-config-val reflect mon configs

Pull down all configs, including both argv and mon, before we do the
--show-config[-val] step.

Signed-off-by: Sage Weil <sage@redhat.com>