os/bluestore: add main device expand capability.

One can do that via ceph-bluestore-tool's bluefs-bdev-expand command

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #25205 from smithfarm/wip-37330

build/ops: do_cmake.sh: automate py3 build options for certain distros

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25211 from badone/wip-selinux-cr-repo-issue

install-deps.sh: Remove CR repo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24821 from k0ste/prometheus_metadata

mgr: prometheus: added bluestore db and wal/journal devices to ceph_disk_occupation metric.

Merge pull request #25200 from tchaikov/wip-silence-Wunused

erasure-code,test: silence -Wunused-variable warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25199 from tchaikov/wip-mgr-silence-py2-warning

mgr: silence GCC warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25204 from cbodley/wip-cmake-without-rbd-cephfs

cmake: vstart target can build WITH_CEPHFS/RBD/MGR=OFF

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25114 from trociny/wip-osd-perf-query-generic

 mgr: generalize osd perf query and make counters accessible from modules

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25209 from runsisi/wip-fix-missing-prefix

test/librbd: metadata key for config should be prefixed with conf_

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24936 from mikechristie/igw-selinux

selinux: add support for ceph iscsi

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24965 from yangdongsheng/token_fix

common: complete all throttle blockers when we set average or max to 0

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25055 from runsisi/wip-fix-resize

librbd: fix missing unblock_writes if shrink is not allowed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25152 from Songweibin/wip-fix-get-key

 rbd: fix error parse arg when getting key

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: Remove CR repo

Remove the continuous release repos for CentOS and Virtuozzo 7 as they
should no longer be needed since http://tracker.ceph.com/issues/13997 is
no longer relevant and the newer versions of selinux packages pulled in
by the build system are causing problems for systems without CR repos
enabled.

Fixes: https://tracker.ceph.com/issues/37335
Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

test/librbd: metadata key for config should be prefixed with conf_

Signed-off-by: runsisi <luo.runbing@zte.com.cn>

Merge pull request #24452 from noahdesu/doc-edit-report

doc: edit on github

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

build/ops: do_cmake.sh: automate py3 build options for certain distros

Before this patch, run-make-check.sh was setting the cmake options for
py3 build based only on the absence of a python2 binary in the system.
This was not sufficiently deterministic, and it didn't help folks who
run do_cmake.sh directly.

With this patch, we explicitly force the py3 build in do_cmake.sh, for
those OS types/versions we know need it.

Fixes: https://tracker.ceph.com/issues/37330
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25157 from tspmelo/wip-e2e-stop

mgr/dashboard: Fix run-frontend-e2e-tests.sh

Reviewed-by: Laura Paduano <lpaduano@suse.com>

cmake: vstart-base only depends on ceph-mgr if WITH_MGR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: dont depend on cython_rbd if WITH_RBD=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: dont build ceph-mds if WITH_CEPHFS=OFF

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25155 from rhcs-dashboard/37283-mgr-info-card-improvements

mgr/dashboard: Status info cards' improvements

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

erasure-code,test: silence -Wunused-variable warnings

GCC 7.3 does not support [[maybe_unused]] very well, so it emits
-Wunused-variable warnings even if the variables are marked with
maybe_unused. moreover, the C++17 standard does not forbid these
warnings:

> For an entity marked maybe_unused, implementations are encouraged not
to emit a warning that the entity is unused, or that the entity is used
despite the presence of the attribute.

see also n4659, dcl.attr.unused

so, in this change, the warnings are silenced manually.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Status info cards' improvements
- MGR info card:
  Show amount of daemons active instead of active mgr name.
  Daemons active text line: show mgr active name when 'on mouse over'.
- OSD/MDS info cards (for consistency):
  The same text appearance as MGR card.
- Hosts/Object Gateway/iSCSI cards: added 'total' after amount for clarification.
- Some CSS refactor.

Fixes: https://tracker.ceph.com/issues/37283
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #24610 from SUSE/wip-orchestrator-deepsea-redux

mgr/deepsea: DeepSea orchestrator module

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr: prometheus: added bluestore db and wal devices to ceph_disk_occupation metric.

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

mgr/deepsea: DeepSea orchestrator module

This module provides integration between Ceph's orchestrator framework
and DeepSea, a Ceph deployment and management system built upon Salt.

Included so far are implementations of get_inventory() and
describe_service().  This allows `ceph orchestrator device ls`, `ceph
orchestrator service ls` and `ceph orchestrator service status` to
operate correctly.

To test, try:

  # ceph mgr module enable orchestrator_cli
  # ceph mgr module enable deepsea
  # ceph orchestrator set backend deepsea
  # ceph deepsea config-set salt_api_url <salt api url>
  # ceph deepsea config-set salt_api_username <salt api username>
  # ceph deepsea config-set salt_api_password <salt api shared secret>
  # ceph orchestrator device ls
  # ceph orchestrator service ls

Signed-off-by: Tim Serong <tserong@suse.com>

mgr: silence GCC warning

the signature of PyObject_CallMethod() is different in python2 and
python3:

in python2: it is

PyObject* PyObject_CallMethod(PyObject *o, char *method, char *format,
...)

while in python3, it is

PyObject* PyObject_CallMethod(PyObject *obj, const char *name, const
char *format, ...)

so, if we compile mgr with python2, we will have following warning:

warning: ISO C++ forbids converting a string constant to ‘char*’
[-Wwrite-strings]
        (char*)NULL);
                   ^

it'd be simpler if we just const_cast<> the method name string, to
silence the warning just like other places we call
PyObject_CallMethod().

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #25167 into master

* refs/pull/25167/head:
mgr: Diskprediction local activate when config changes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24795 into master

* refs/pull/24795/head:
qa/tasks/ceph: gather crash dumps

Merge pull request #14369 from rzarzynski/wip-rgw-19514

rgw: fix wrong debug related to user ACLs in rgw_build_bucket_policies()

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16658 from rzarzynski/wip-rgw-20795

rgw: Swift's TempURL can handle temp_url_expires written in ISO8601.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #19684 from taoCH/wip-librgw-symbolic-link

librgw: support symbolic link

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

mgr/osd_perf_query: improvements for testing module

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22599 from anthonyeleven/patch-2

doc: Tidy up description wording and spelling

Merge PR #24737 into master

* refs/pull/24737/head:
doc/ceph-bluestore-tool: add help for migrate and new DB/WAL commands.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25010 from nathan-weinberg/breadcrumb-tests

mgr/dashboard: Added additional breadcrumb tests to Cluster

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25141 from rhcs-dashboard/wip-37293-master

mgr/dashboard: Fix Forbidden Error with some roles

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25176 from cyx1231st/wip-crimson-msgr-extract-states

crimson/net: encapsulate protocol implementations with states

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: remove recursive call to handle_connect_reply()

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: fix minor issue and add TODO for error-handling

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: extract state transition out of repeat_connect()

and extract state transition out of repeat_handle_connect()

in this change, the connect/handle-connect loop is restructured, to
avoid ad-hoc state changes in helper functions. this pave the road to
explicit state transtion using named states.

also, exception is thrown instead in handle_connect_reply(), we should
not proceed in case of failures. and we need do error handling in the
named state in future.

currentl, `state` is set to `state_t::open` in `start_connect()` and
`start_accept()`, the next step is to set it in a named state.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25149 from p-na/pna-fix-safe-to-destroy

mgr/dashboard: Fix failing QA test: test_safe_to_destroy

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/net: define an alias of seastar::stop_iteration

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: check short reads in `read_exactly()`

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge PR #24844 into master

* refs/pull/24844/head:
osd: drop PGBackend::Listener::get_epoch().
osd: massively switch to get_osdmap_epoch().
osd: switch the return type of PG::get_osdmap().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24867 into master

* refs/pull/24867/head:
mon: compensate for not having STDLIB_MAP_SPLICING

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #24929 into master

* refs/pull/24929/head:
osd: set lower bound+docs on osd_max_write_size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25058 into master

* refs/pull/25058/head:
mgr: Fixed devicehealth plugin resource leak

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25125 into master

* refs/pull/25125/head:
mon: require keyring even if cephx is optional

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: gather crash dumps

Gather crash dumps as part of the teuthology run.

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/net: connection closed during connecting

It is possible during `seastar::connect()`, the connection is marked
closed (when SocketConnection::socket is unavailable), then
`seastar::connect()` successfully returns a connected socket.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #25121 from zmc/wip-dashboard-test-runner

mgr/dashboard: Small improvements for running teuthology tests

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr: parse OSDPerfMetricQuery spec that comes from python module

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: make dynamic osd perf counters accessible from modules

Signed-off-by: Mykola Golub <mgolub@suse.com>

osd: fixup to recently added dynamic perf stats

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: fix PyFormatter's dump_unsigned

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: store osd perf counters received in osd reports

Make the counters accessible for the query owners.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: generalize OSDPerfMetricQuery to support different queries

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24851 from rhcs-dashboard/grafana-roles

mgr/dashboard: Show/Hide Grafana tabs according to user role

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24762 from tspmelo/wip-credits

credits.sh: Ignore package-lock.json and .xlf files

Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25169 from tchaikov/wip-install-libboost-bionic

install-deps.sh: fix gcc detection and install pre-built libboost on bionic

Tested-by: Changcheng Liu <changcheng.liu@intel.com>
Reviewed-by: Changcheng Liu <changcheng.liu@intel.com>

install-deps.sh: install prebuilt libboost dependencies for bionic also

* extract boost install function into a helper, so we don't need to
  repeat it when adding support for new distros.
* install ceph-libboost-* for bionic build also. we check for the
  existence in run-make-check.sh, so there is no need to update
  the build script elsewhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: correct gcc version info as major.minor.patch

On ubuntu 18.04, "gcc -dumpversion" output "7", it result in
that below check will return false:
    dpkg --compare-versions 7 ge 7.0
Then, this script will install other gcc.

Actully, the full gcc version "gcc -dumpfullversion" output is
"7.3.0", than below check will turn true:
    dpkg --compare-version 7.3.0 ge 7.0
So, there's no need to install other gcc.

In case of hitting error on ubuntu16.04, use below parameter:
   gcc -dumpfullversion -dumpversion

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr: Diskprediction local activate when config changes

Trigger diskprediction local action when receive notify configuration changed.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

Merge PR #24874 into master

* refs/pull/24874/head:
os/filestore: collect partition/device metadata from journal
os/filestore: include journal in get_devices result

Reviewed-by: Rick Chen <rick.chen@prophetstor.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #25140 into master

* refs/pull/25140/head:
mon: by default, do NOT warn about misplaced objects
mon: add mon_warn_on_misplaced option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #25147 into master

* refs/pull/25147/head:
mgr/devicehealth: implement config_notify hook
mgr: call config_notify method when mgr's config has changed
mon/MonClient: add callback for (any) config change
common/config: fix debug output for config callbacks changes

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #25158 into master

* refs/pull/25158/head:
[doc] Fix typo in luminious release notes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fix run-frontend-e2e-tests.sh

e2e tests were always returning positive result, even when they failed.

Fixed problem with jq, where it was failing when it was not receiving a string.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #25111 into master

* refs/pull/25111/head:
test: Add test for requested scrub priority
osd: Prioritize user specified scrubs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23819 from dongbula/nf-add-bucket-option-in-usage

rgw: add bucket as option when show/trim usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23318 from theanalyst/wip-curl-errstring

rgw: http client: print curl error messages during curl failures

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25069 from markhpc/wip-memory-autotune-doc

doc/rados: Add bluestore memory autotuning docs.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24492 from theanalyst/wip-rgw-es-attr-fixes

rgw: ES sync: be more restrictive on object system attrs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24730 from pritha-srivastava/wip-rgw-role-fixes

rgw: Minor fixes to radosgw-admin commands for a role.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24846 from taoCH/wip-fix-librgw-quota-check

rgw_file: not check max_objects when creating file

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

[doc] Fix typo in luminious release notes

Signed-off-by: Maxime <maxime@root314.com>

Merge pull request #24935 from leseb/rbd-mirror-on-reboot

systemd: enable ceph-rbd-mirror.target

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24863 from jtlayton/wip-orchestrator

mgr/orchestrator: Add support for "ceph orchestrator service ls"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/dashboard: Fix failing QA test: test_safe_to_destroy

The behavior of `safe-to-destroy` has changed in
432f19435523d455ecd4f386e58ee1f033cf97e2 (PR#24799) and the backend
needs to be adapted accordingly.

Fixes: http://tracker.ceph.com/issues/37290
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24932 from tspmelo/wip-nodejs-update

mgr/dashboard: Update Node.js to current LTS

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24779 from smithfarm/wip-luminous-release-notes

doc: Luminous v12.2.9 release notes

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

rbd: fix error parse arg when getting image-meta key

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: fix error parse arg when getting config key

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #25136 from Devp00l/wip-easy-e2e-dev

mgr/dashboard: npm run e2e:dev

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25139 from tchaikov/wip-xmlsec

test/dashboard: fix segfault when importing dm.xmlsec.binding

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Show/Hide Grafana tabs according to user role

Fixes: http://tracker.ceph.com/issues/36655
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #25096 from dillaman/wip-rbd-mirror-peers

librbd: move mirror peer attribute handling from CLI to API

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #20665 into master

* refs/pull/20665/head:
mgr/balancer: add crush_compat_metrics param

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: implement config_notify hook

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: call config_notify method when mgr's config has changed

This provides a hook for the module to refresh any config values it has
cached in module-local variables.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: add callback for (any) config change

This is a coarse "there was some config update".  It's fired whether or
not it values are observed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25135 from cbodley/wip-test-log-garble

test: fix libc++ crash in Log.GarbleRecovery

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

mgr/dashboard: Fix Forbidden Error with some roles

Fixes: http://tracker.ceph.com/issues/37293
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Remove any dashboard .pyc files before testing

Things like 'git rm' and 'git mv' don't clean up any .pyc files that
might have been generated previously, so if those aren't removed, we
might not be running the exact code that we intend to.

Signed-off-by: Zack Cerza <zack@redhat.com>

When cloning teuthology, pass --depth 1

We don't need history, so we may as well save some time.

Signed-off-by: Zack Cerza <zack@redhat.com>

test/dashboard: fix segfault when importing dm.xmlsec.binding

python-saml depends on dm.xmlsec.binding which is a python binding of
xmlsec C library. but without -DXMLSEC_NO_SIZE_T the compiled
dm.xmlsec.binding segfaults when `import dm.xmlsec.binding`. see
https://github.com/onelogin/python-saml/issues/30 and
https://github.com/4teamwork/ftw.saml2auth/issues/3 .

in long term, we might want to switch to python-saml, see
http://tracker.ceph.com/issues/37081

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: by default, do NOT warn about misplaced objects

Misplaced objects are not something that puts the health or safety of
data in jeopardy.  Don't warn about it by default.

Since this is a change in behavior, add a release note.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add mon_warn_on_misplaced option

Control whether we issue HEALTH_WARN due to misplaced objects.

Signed-off-by: Sage Weil <sage@redhat.com>