Merge pull request #21636 from rjfd/wip-dashboard-fix-pool-py3

mgr/dashboard: pool: fix python3 dict_keys error

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21618 from votdev/loading_panel

mgr/dashboard: Add 'cd-loading-panel' component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #21633 from ukernel/wip-mds-stat-rstat

mds: fix CEPH_STAT_RSTAT definition

mgr/dashboard: pool: fix python3 dict_keys error

Signed-off-by: Ricardo Dias <rdias@suse.com>

mds: fix CEPH_STAT_RSTAT definition

(CEPH_CAP_GWREXTEND << 16) is (1<<22), which causes gaps in caps bits.
CEPH_CAP_FILE_WREXTEND is an unused bit, reuse it for CEPH_STAT_RSTAT.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #21273 from jdurgin/wip-23195

osd/ECBackend: only check required shards when finishing recovery reads

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21620 from jcsp/wip-mgr-balancer-options

mgr/balancer: fix OPTIONS definition

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21606 from ceph/fix-build-CEPH_CLS_API

rados/objclass.h: fix build define CEPH_CLS_API in all cases

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21621 from theanalyst/doc/releases/12.2.5

doc: v12.2.5 luminous release notes

Reviewed-By: Sage Weil <sage@redhat.com>

mgr/dashboard: Introduce 'cd-loading-panel' component

Signed-off-by: Volker Theile <vtheile@suse.com>

doc: v12.2.5 luminous release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #21595 from votdev/load_on_init

mgr/dashboard: Load the datatable content on component initialization

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #20899 from Devp00l/deletion-dialog

mgr/dashboard: Adds reusable deletion dialog

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/balancer: fix OPTIONS definition

I typo'd a couple of these, which was causing test failures
when the access to the correct names got an exception.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: Replaces delete-confirmation-modal

This replaces usage of "delete-confirmation-modal" with the use of
"delete-modal".

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Change deletion link to modal only

Due to CSS problems the link solution wasn't the best way, now it will
represent only the modal content. The downside of this solution is that
it put's the burden on the developer to use it the right way and import
a view things to get it working. But on the upside CSS styles will work
as expected.

The unit test example was updated accordingly this way it should be easy
to understand how it can be implemented the right way.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Changes deletion button to link

This change was made because a link can be placed anywhere instead of a
button element.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Adds reusable deletion dialog

You can now simply use a deletion dialog without having to import a lot
of different things from ngx-bootstrap. Its easy to extend the dialog
by a detail description.

Signed-off-by: Tiago Melo <tmelo@suse.com>
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add modal component

This component should be used each time you define a new modal.
This will allows us to keep all modals with the same visual aspect.

Signed-off-by: Tiago Melo <tmelo@suse.com>
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #21604 from liewegas/wip-ec-write-to-read

osd/ECBackend: wait for apply for luminous peers

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20875 from jan--f/mgr-server-list-hostname

mgr: add mgr daemon to DaemonStateIndex with metadata (hostname)

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21555 from tchaikov/wip-23763

test/librados: increase pgp_num along with pg_num

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21139 from guzhongyan/cmd-wildcard

mon,mgr: remove single wildcard '*' from ceph comand line description

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21424 from MI-OSiRIS/influx

mgr/influx: Added Additional Stats

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21607 from jcsp/wip-mgr-selftest-fixup

mgr: reconcile can_run checks and selftest

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21559 from votdev/autofocus

mgr/dashboard: Add 'autofocus' directive

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21602 from votdev/fix_rbd_form

mgr/dashboard: Adapt RBD form to new application_metadata type

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #21166 into master

* refs/pull/21166/head:
mds: fix occasional dir rstat inconsistency between multi-MDSes

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21510 into master

* refs/pull/21510/head:
mon: respect standby_for_fscid when choosing standby replay mds
mon: fix standby replay in multimds setup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21535 into master

* refs/pull/21535/head:
MDSMonitor: do not resize cluster when MDS is starting

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #21536 into master

* refs/pull/21536/head:
qa: check old_max_mds is not wiped by marked down twice
MDSMonitor: do not wipe old_max_mds when marked down twice

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

qa/suite: whitelist PG_AVAILABILITY in rados_api_tests.yaml

pg will be created when increasing pgp-num and pg-num. so at that
moment, PG_AVAILABILITY is reported. so whitelist it in all tests which
run rados/test.sh. that script exercises ceph_test_rados_api_list.

Fixes: http://tracker.ceph.com/issues/23763
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados: increase pgp_num along with pg_num

Fixes: http://tracker.ceph.com/issues/23763
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21590 from sebastian-philipp/dashboard-fix-taskmanager-race

mgr/dashboard: Fix data race and use-before-assignment

Reviewed-by: Ricardo Dias <rdias@suse.com>

MDSMonitor: do not resize cluster when MDS is starting

Fixes: http://tracker.ceph.com/issues/23799
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: handle commands sent to unavailable modules

There are some up-front checks in DaemonServer
but it shouldn't assume that its checks are
necessarily going to match the choices about
how ActivePyModules composes its ::modules member,
so let's have some extra checks to avoid
risk of crashing mgr on commands sent to
unhealthy/unloaded modules.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20959 from LiumxNL/fix-pg-fake-down

pg: handle MNotifyRec event in down state

Reviewed-by: Sage Weil <sage@redhat.com>

mgr: execute modules even if can_run=false

We execute modules even if can_run=false, so that it is possible
to load them for running their selftest hooks.  However,
we already raise health messages about the fact that they're
enabled but can't run, so we don't want to also raise
health messages about whatever exceptions they raise
from serve()

Signed-off-by: John Spray <john.spray@redhat.com>

qa/tasks/mgr: pass "--force" to "mgr module enable"

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: let ceph-mgr suggests python-influxdb

the influx plugin requires influxdb python module to function, but
influxdb mgr plugin is optional for users who don't use influxdb.
so it's marked "Suggests" at this moment before we use a more flexible
packaging scheme.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21000 from ceph/evelu-23320

common: Improving message sent to user when getting signals

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20761 from liewegas/wip-service-unregister

mgr: allow service daemons to unregister from ServiceMap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rados/objclass.h: fix build define CEPH_CLS_API in all cases

Fix for bug introduced with 096668ec5df93725a0726b53bbd30b4d4ab50f49:

In file included from src/objclass/objclass.h:13:0,
                 from src/cls/hello/cls_hello.cc:37:
src/include/rados/objclass.h:31:1: error: ‘CEPH_CLS_API’ does not name a
type; did you mean ‘CEPH_CAP_ANY’?
 CEPH_CLS_API void name##_cls_init()
 ^
/develop/ceph/source/ceph/src/include/rados/objclass.h:31:1: note: in
definition of macro ‘CLS_INIT’
 CEPH_CLS_API void name##_cls_init()
 ^~~~~~~~~~~~

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ECBackend: wait for apply for luminous peers

Avoid completing a write until luminous peers also apply the change.
This is overkill, but works around a problem where completing our write
here allows the next read to start too early.  This is because the
PrimaryLogPG is handling the write-to-read ordering, but completing the
write releases the write lock in the PG.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21442 from jcsp/wip-config

mgr: centralized setting/getting of mgr configs

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Fix bug in RBD form.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #21580 from xiexingguo/wip-add-assert

osd/PGLog: assert out on performing overflowed log trimming

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21572 from tchaikov/wip-boost-1.67

cmake,make-dist: bump up boost version to 1.67

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19985 from Liuchang0812/osd-pool-stats-in-mgr

mon, mgr: move "osd pool stats" command to mgr and mgr python module

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21058 from trociny/wip-osd-empty-snapset

osd: don't crash on empty snapset

Reviewed-by: Sage Weil <sage@redhat.com>

doc: update config-key->config in mgr module docs

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update mgr plugin guide

...for recent config changes.  Explain about MgrModule.OPTIONS,
and explain the new store/config distinction.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update mgr test for MgrModule.OPTIONS

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add MgrModule.OPTIONS definitions to modules

They need these to do upgrades properly.  Where there
was an existing structure describing config, merge
these together to avoid having two lists of options.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #21188 from guzhongyan/mgr-disable-fix

mon,mgr: improve 'mgr module disable' cmd

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21597 from LenzGr/dashboard-releasenotes

PendingReleaseNotes: Added note about Dashboard v2, fixed typo

Address comments.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add 'autofocus' directive

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #21581 from tchaikov/wip-objcls-sdk

objclass-sdk: expose __cls_init() to the world

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Load the datatable content after the component is initialized.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr: load MgrModule.OPTIONS and use it in upgrade

Modules will now only migrate items from config-key
into config options if they appear in the module's
schema.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: introduce MgrModule.OPTIONS field

Now is a good time to start requiring
modules to explicitly list their configuration
settings, so that we can do a proper job of
migrating configuration from old config-key style,
i.e. knowing what's a config setting and what's
a KV store item.

Throw an exception if a module tries to
access a setting outside their schema, so
that we have some confidence that the schema
is complete.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix up a couple of double-lookups

Signed-off-by: John Spray <john.spray@redhat.com>

qa: test mgr live configuration updates

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: use "config set" for mgr modules

...instead of pre-mimic config-key set.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/restful: migrate to new get_store interface

Signed-off-by: John Spray <john.spray@redhat.com>

mon: warn user doing "config key set mgr/..."

This is still completely legal, but isn't going
to have the effect they wanted if they were
trying to set a configuration option for a mgr module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: replace get_config_prefix with get_store_prefix

The _prefix variant was only used for data-ish things,
so we can just move it over to operate on store instead
of config, rather than having a _prefix variant for both.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: don't migrate config values that look like data

A quickie heuristic, but sufficient to avoid picking
up e.g. SSL certs and trying to cram them into
config values.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: grant mgr profile "config" commands

...and remove redundant config-key lines (these are applied
to mgr anyway in the next block, and mgr even has a broader
config-key permission in the line above).

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: clean up ActivePyModules logging

This was running the prefix and message together.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix PyModuleConfig::set_config

Commands were slightly wrong.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: remove dead update_perf_schema fn definition

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: rework kv store load path

The locking and blocking around this was a bit
tricky.  Do the simple thing, and pull the
load_store out to Mgr so that it can be safely
done as part of the background_init process (just drop
Mgr::lock across blocking actions).

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: update python interface for store vs. config

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: separate out module_config and store_cache

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: centralized setting/getting of mgr configs

Signed-off-by: Rubab Syed<rubab.syed21@gmail.com>

PendingReleaseNotes: Added note about Dashboard v2, fixed typo

Added a bullet point about the new Ceph mgr dashboard, fixed typo in the
LZ4 compression bullet point.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21588 from xiexingguo/wip-fix-ec-ar-order

osd/PG: perfer EC async_recovery_targets in reverse order of cost

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21562 from LenzGr/mgr-dashboard-documentation

doc: updated dashboard feature list (added new RGW details, Pools)

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix data race and use-before-assignment

The race happes, if a task raises an exception very early.
Then, `self.getter_thread.event` fails as `getter_thread` is
already `None`

Also fix use-before-assignment, as `t0` and `t1` are only defined,
if no exception was raised.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #21564 from dillaman/wip-23809

rbd: cleanup handling of IEC byte units

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21408 from hsepeng/oisizemismatch-osd-bugfix

osd/PrimaryLogPG: fix the oi size mismatch with real object size

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer: use helper to get DaemonKey

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: close mgr session on daemon MgrClient shutdown

If we are a service daemon, send a MMgrClose on shutdown to clear ourselves
out of the ServiceMap.

Note that this is a best-effort attempt; we don't block shutdown if the
mgr isn't currently available.

Signed-off-by: Sage Weil <sage@redhat.com>

mon,mgr: improve 'mgr module disable/enable' cmds

when running those cmds, check if the module do exist
or is disabled already.
also check if the module is enabled already.

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

osd/PG: perfer EC async_recovery_targets in reverse order of cost

This is a follow-up fix of https://github.com/ceph/ceph/pull/21578,
in which I forget that erasure-coded-pools share the same logic
when determining the async_recovery_targets..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21549 from rjfd/wip-dashboard-fix-qa-whitelist

qa/suites/rados/mgr/tasks/dashboard: add MDS_ALL_DOWN to whitelist

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/mgr/tasks/dashboard: add MDS_(ALL_DOWN|UP_LESS_THAN_MAX) to whitelist

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21560 from jan--f/fix_strtoll_interface

common/strtoll: remove superfluous const modifier

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21550 from wjwithagen/wjw-kevent-fix

EventKqueue: Clang want realloc return to be typed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21579 from tchaikov/wip-test.sh-with-multi-mds

qa/workunits/cephtool/test.sh: fix test_mon_mds()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21578 from xiexingguo/wip-fix-async-recovery-peer

osd/PG: perfer async_recovery_targets in reverse order of cost

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21552 from smithfarm/wip-23805

qa/standalone/ceph-helpers.sh: provide argument to dirname

Reviewed-by: David Zafman <dzafman@redhat.com>

objclass-sdk: expose __cls_init() to the world

the __cls_init() function should be visible. otherwise we cannot call it
to register the cls methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGLog: assert out on performing overflowed log trimming

Performing overflowed log-trim can be a sign of big trouble, e.g.,
the **complete_to** iterator will now point to an invalid position
of the original pg-log list when the trimming is done, and hence
randomly trigger **Segmentation fault**s as below:
```
2018-03-07 17:38:46.109018 7f274a4ed700 -1 *** Caught signal (Segmentation fault) **
1: (()+0xa51f31) [0x7f278290bf31]
2: (()+0xf370) [0x7f277fb4f370]
3: (PrimaryLogPG::recover_got(hobject_t, eversion_t)+0x266) [0x7f2782512786]
4: (PrimaryLogPG::on_local_recover(hobject_t const&, ObjectRecoveryInfo const&, std::shared_ptr<ObjectContext>, bool, ObjectStore::Tran
saction*)+0x2a4) [0x7f278251f3b4]
5: (ReplicatedBackend::handle_push(pg_shard_t, PushOp const&, PushReplyOp*, ObjectStore::Transaction*)+0x2e2) [0x7f2782690f82]
6: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x194) [0x7f2782691224]
7: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2f1) [0x7f278269fd41]
8: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x7f27825c2470]
```

The root cause of why PGs are starting to trim more log entries than
we expect is still lost to me, but setting the trap here should generally
do no harm and hopefully expose the above problem a little bit more offen.
We'll see.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21507 from liewegas/wip-rados-test

qa/workunits/rados/test.sh: ensure tee output is valid filename

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20921 from jcsp/wip-23385

mgr: improve reporting on unloadable modules

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20343 from trociny/wip-rados-ordersnap

librados: add OPERATION_ORDERSNAP flag and yet another aio_operate method

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21521 from tchaikov/wip-stoll

common/strtol: fix strict_strtoll() so it accepts hex starting with 0x

Reviewed-by: Jan Fajerski <jfajerski@suse.com>