mgr/dashboard: HTTP request logging

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20745 from Liuchang0812/jsonify-pool-get

mon: invalid JSON returned when querying pool parameters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20775 from tspmelo/fix-iscsi-unit-test

mgr/dashboard_v2: add mocked service provider for TcmuIscsiService

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20778 from p-na/wip-mgr-fix-dashboard-backend-tests

mgr/dashboard: Fix backend tests for newer CherryPy versions

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20782 from sebastian-philipp/dashboard_v2_fix_config_options_endpoint

mgr/dashboard_v2: Fix test_cluster_configuration test

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20786 from dzafman/wip-zafman-log-trim

tools/ceph-objectstore-tool: command to trim the pg log

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tools: Add pg log trim command to ceph-objectstore-tool

Add test script that verifies the command in qa/standalone/osd

Fixes: http://tracker.ceph.com/issues/23242
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20769 from oritwas/wip-rgw-20448

rgw: fix crash with rgw_run_sync_thread false

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20590 from hitoshikamei/rbd-bench

rbd: fix thread_offsets calculation of rbd bench

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20689 from Songweibin/wip-23038

rbd/test: add snap protection test for ex/import

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20697 from Songweibin/wip-trash-rm

rbd: allow trash rm/purge when pool quota is full used

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20795 from Songweibin/wip-fix-trash

rbd: pool_percent_used should not divided by 100

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20759 from dzafman/wip-cleanup

test: Make clearer by moving code out of loop

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20801 from theanalyst/dashboard-build-typo

cmake: typo fix when npm is not found

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

dashboard: cmake: typo fix when npm is not found

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #20390 from theanalyst/rgw/openssl-init

rgw: setup locks for libopenssl

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #20649 from tianshan/rgw_datalog_list

rgw: datalog list support --shard-id and --marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20686 from fangyuxiangGL/mcc

rgw: fix 'copy part' without 'x-amz-copy-source-range' when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20707 from Liuchang0812/rgw-es-plugin-urlencode

rgw: url_encode key name and instance in es sync module

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20756 from cbodley/wip-qa-rgw-verify-combine

qa/rgw: combine swift, s3tests, ragweed into single verify task

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #20767 from xiexingguo/wip-pg-delete

osd/OSD: batch-list objects to reduce memory consumption

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20764 from chenyupeng360/master

msg/async: size of EventCenter::file_events should be greater than fd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20740 from Songweibin/wip-skip-logical-devices

osd: skip dumping logical devices

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20743 from rjfd/wip-dashboard-settings

mgr/dashboard_v2: Configuration settings support

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #20727 from linuxbox2/wip-clang-5

bluestore: define default value of LoglevelV only once (3 templates)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20711 from guzhongyan/mgr-nodels

mon: 'node ls' mgr support

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20706 from xiexingguo/wip-balancer-03

pybind/mgr/balancer: fix pool-deletion vs auto-optimization race

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20704 from xiexingguo/wip-balancer-02

mon/OSDMonitor: kill pending upmap changes too if pool is gone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20703 from xiexingguo/wip-balancer-01

osd/OSDMap: ignore PGs from pools of failure-domain OSD

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20644 from jcsp/wip-23166

mgr/prometheus: skip OSD output if missing from CRUSH devices

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20773 from idryomov/wip-krbd-include-sysmacros

krbd: include sys/sysmacros.h for major, minor and makedev

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20750 from idryomov/wip-krbd-whole-object-discard-test

qa: krbd whole-object-discard test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20755 from LenzGr/mgr-dashboard-documentation

doc: Updated dashboard v2 feature list

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20792 from yangdongsheng/krbd_osd_request_timeout

rbd: support osd_request_timeout in rbd map command

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd: support osd_request_timeout in rbd map command

Default of osd_request_timeout in krbd is 0 means no timeout,
but we sometimes need to set this value by rbd command. then
pass osd_request_timeout to kernel rbd in rbd mapping.

Fixes: http://tracker.ceph.com/issues/23073
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: pool_percent_used should not divided by 100

The value of 'percent_used' in `ceph df --format=json`
has already been divided by 100.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20729 from dillaman/wip-22738

qa/suites/rbd: segregated v1 image format tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #20705 into master

* refs/pull/20705/head:
doc: Fix -d description in ceph-fuse

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard_v2: removed unused code that broke teuthology

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Added instructions on how to add config settings to HACKING

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: added settings module

Signed-off-by: Ricardo Dias <rdias@suse.com>

common/config: Fix dumping of `md_config_t::config_options`.

Caused by 49a5e775aa92a8218bb659909de20637d5aa660a

This fixes the dashboard_v2 `cluster_conf` endpoint.

Fixes: http://tracker.ceph.com/issues/23265
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: combine swift, s3tests, ragweed into single verify task

these tests don't need to be split across the job matrix the same way
that we split features like frontend, ssl, objectstore, etc. by
combining them, we can still test the whole matrix of features, but with
only 1/3 of the total jobs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20693 from cbodley/wip-rgw-s3website-redirect

rgw: s3website error handler uses original object name

Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

mgr/dashboard: Fix backend tests for newer CherryPy versions

The backend test run by `run-backend-api-tests.sh` fails for CherryPy
versions 13.x and 14.x.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard_v2: add mocked service provider for TcmuIscsiService

This removes the warnings shown when running the tests with PhantomJS.

Signed-off-by: Tiago Melo <tmelo@suse.com>

qa: krbd whole-object-discard test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: include sys/sysmacros.h for major, minor and makedev

Previously we got these through sys/types.h, but that's now deprecated:

  warning: In the GNU C Library, "major" is defined
   by <sys/sysmacros.h>. For historical compatibility, it is
   currently defined by <sys/types.h> as well, but we plan to
   remove this soon. To use "major", include <sys/sysmacros.h>
   directly. If you did not intend to use a system-defined macro
   "major", you should undefine it after including <sys/types.h>.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20768 from tchaikov/wip-cmake-dashboard

cmake,test/mgr: restructure dashboard tests and cmake related fixes

Reviewed-by: Ricardo Dias <rdias@suse.com>

rgw: fix crash with rgw_run_sync_thread false

Fixes: http://tracker.ceph.com/issues/20448
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

test/mgr: restructure dashboard tests

* restructure dashboard tests so they follow the model of other tests
  based on ceph-helpers.sh. i.e. extract the TEST_* out of run()
  function.
* expose mon-host using CEPH_ARGS env variable.
* mgr-dashboard_v2-smoke.sh: do not export CEPH_ARGS in a subshell, so
  the run_mgr() and ceph cli can also have access to the updated
  CEPH_ARGS, and hence they will be able to locate the mon. and because
  auth-supported=none is populated using CEPH_ARGS, this test won't fail
  because of missing keyring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: run 'npm' command in a directory specified by absolute path

in case developer wants to run the test in "./build" or some random
directory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for "npm" before using it

and allows use to specify the path in which npm is installed. in my
case, if $HOME/.npmrc points "prefix" to ${HOME}/.npm-packages, i will
be able to run

NPM_ROOT=${HOME}/.npm-packages cmake ..

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD: batch-list objects to reduce memory consumption

For PG with a huge amount of objects, it wouldn't be an ideal
way to list all of them at a time. Split them into small batches
which we can handle individually efficiently should instead be
the preferred option.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20742 from votdev/perf_counter_units

mgr/dashboard v2: Add units to performance counters

Merge pull request #20744 from votdev/fix_loading_indicator

mgr/dashboard v2: Display loading indicator in datatables during first load

Merge pull request #20746 from votdev/datatable_selection

mgr/dashboard v2: Introduce CdTableSelection model

msg/async: size of EventCenter::file_events should be greater than fd.

When trying to generate a file event from an FD greater than or equal to the size of file_events vector,
make sure that the resized vector can hold the FD safely.

Fixes: http://tracker.ceph.com/issues/23253
Signed-off-by: Yupeng Chen <chenyupeng-it@360.cn>

doc: Fix -d description in ceph-fuse

Fixes: http://tracker.ceph.com/issues/23214
Signed-off-by: Jos Collin <jcollin@redhat.com>

mon/OSDMonitor: kill pending upmap changes too if pool is gone

There are quite a lot mgr/balancer related tests that I can observe
the following logged errors:
```
2018-03-03 08:14:00.359946 7fdda18cc700 -1 maybe_remove_pg_upmaps unable to load crush-rule of pg 5.b
```
which turns out to be a *pool-deletion* vs *balancer-auto-injected-upmap-changes*
race issue.
The root cause is that we don't clean up those *pending*
pg_upmap/pg_upmap_items injected by the mgr/balancer properly simultaneously
when the corresponding pool is gone, and hence the above problem can be fixed by:
1. clean up any pending upmap changes too if the corresponding pool is gone
2. re-check pending pool removal queue before we can safely apply any new upmap changes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: skip dumping logical devices

The code meant to do so but actually it does not ...

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge PR #20472 into master

* refs/pull/20472/head:
mds: add DUMP_DIRFRAGS to CInode and DUMP_ITEMS to CDir
mds: Enhance dump_tree performance by traversing the tree directly

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20502 into master

* refs/pull/20502/head:
test: cap waits in delegation tests at 60s
tests: add a test for doing a getattr while caps are recalled but before returned
client: allow getattr, lookup, fstatx, and readdir to use implemented caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20541 into master

* refs/pull/20541/head:
ceph-mds: exit if invalid id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20575 into master

* refs/pull/20575/head:
mds: set journaler iohint correctly when mds daemon going to active

Reviewed-by: dongdong tao <tdd21151186@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20583 into master

* refs/pull/20583/head:
cephfs-journal-tool: add event time output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20615 into master

* refs/pull/20615/head:
mds: modify help info of dump_historic_ops and dump_historic_ops_by_duration
mds:fix typo in MDSRank

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20172 from liewegas/wip-config

mon: centralized config

Reviewed-by: John Spray <john.spray@redhat.com>

common/config: normalize key name for get_val (external) variants

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: -c to ceph cli

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: process subs from update_from_paxos

This is what OSDMonitor and MDSMonitor do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: fix auth timeout vs error race

It's possible that we successfully set active_con *and* time out the
cond WaitUntil.  Only set the error if we don't have a connection; if we
set it *and* time out then let's call it success.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: fix dump when by_type is missing an item

Signed-off-by: Sage Weil <sage@redhat.com>

common/common_init: use unique admin_socket path for unprivileged daemons

These qualify as 'daemon', but their path is usually not unique.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: intercept "keyfile", translate into "key"

The keyfile arg might be - (stdin), which we can only read once.  Ensure
that we consume it once by intercepting the CLI value early and inserting
the value into the 'key' option.

This robs future code of the knowledge that the key came from --keyfile
and not --key, but avoids the issue of multiple users (notably, KeyRing.cc
and the OSD mkfs code).

Remove the - special case from OSD at the same time, since it can no
longer be reached (unless something other than the CLI specified '-', but
neither ceph.conf nor the mon config make sense here).

Signed-off-by: Sage Weil <sage@redhat.com>

librados: fix common_init_finish timing

Common_init_finish does start_service_thread and does
set_safe_to_start_threads() on the cct, which switches us to 'runtime'
mode where we can't accept many config options. Do that *after* we fetch
our config from the mon so that we can accept+set runtime options (and
not complain to stderr about it).

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: check against raw value (no meta) to detect unchanged option

If we are looking at a new value from the mon and comparing it to what we
already have active, compare the non-meta-substituted form.  This way a
value from the mon that can't update at runtime but we have already set to
the same value will not be falsely flagged as ignored.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: add missing #include, tweak types

Seems to resolve a build error on some compilers!  Meh.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/ceph-conf: fix test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rebuild_mondb.py: --no-mon-config to ceph-objectstore-tool

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: add --mon-config option (opposite of --no-mon-config)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: apply timeout while fetching config

The normal timeouts automatically apply during the authenticate() stage,
but not to the explicit wait for a config.  If we don't get that quickly
we shoudl retry another monitor because it is possible we will connect to
an out-of-quorum (or otherwise unresponsive) mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: tolerate pre-mimic mons

Return success if the mons are pre-mimic (and thus have no config).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: --no-mon-config

Some of these cases make sense to fetch mon configs, but we are deprecating
ceph-disk anyway, and the tests currently make use of ceph-disk in places
that do not have a mon_host defined via a ceph.conf or other environment.
This avoids breaking those test cases without any real impact on users
(which will either use ceph-volume or presumably remain in a legacy config
environment).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: fix mon_caps

Pass --no-mon-config so we don't get

 failed to fetch mon config (--no-mon-config to skip)

instead.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: --no-mon-config for ceph-objectstore-tool --op mkfs ..

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: infer osd id to authenticate

In order to get mon configs we need to authenticate as somebody.
Infer it from the data directory.

Note that this won't work for --op mkfs.  Those users need to pass
--no-mon-config or -n.

Signed-off-by: Sage Weil <sage@redhat.com>

global/global_init: send startup errors to cerr, not log

(Flush the log, too, as something that has useful error messages.)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph.py: --no-mon-config during ceph-osd mkfs

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: use assimilate-conf to ingest config options quickly

Separate 'config set' commands is slow.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: only send config_bl if it has changed

The mgr already treats the config_bl as optional in the MMgrReport
messages.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix osd bootstrap

Install the key before trying to use it for mkfs.  Among other things
this means the osd can fetch it's config from the mon before doing the
mkfs step.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: require authentication but no caps for monmap and config

Signed-off-by: Sage Weil <sage@redhat.com>

mon: clean up dealth_with bool weirdness

Just return!

Signed-off-by: Sage Weil <sage@redhat.com>

mon: require authentication for MGetConfig and mon metadata

More importantly, the set of non-authentication operations that are allowed
is more restricted: auth messages, getting monmap, ping.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add authenticated flag in Session

Authentication is orthogonal to the caps; the client might have no
capabilities but still be authenticated such that we know *who* it is.

Signed-off-by: Sage Weil <sage@redhat.com>

test/mgr/mgr-dashboard-smoke: don't clobber CEPH_ARGS for other daemons

Signed-off-by: Sage Weil <sage@redhat.com>

global/global_init: apply_changes() for config fetch session

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: exclude unchanged defaults from diff

If we have a CONF_DEFAULT in the values map we do not need to include it
in the diff!

Signed-off-by: Sage Weil <sage@redhat.com>