rbd: cleanup code and output formatting in status action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37414 from votdev/issue_47647_pylint_isort

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #37221 into master

* refs/pull/37221/head:
ceph-volume: add no-systemd argument to zap

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37532 from tchaikov/wip-crimson-more-privacy

crimson: mark more methods private

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37516 from anthonyeleven/anthonyeleven/misc-doc-tweaks

doc/rbd: readability and spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37085 from sunnyku/wip-fix-46743

mgr/devicehealth: device_health_metrics pool gets created even without any OSDs in the cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #37487 from tchaikov/wip-backtrace

common/BackTrace: print path without symbol

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37489 from Rethan/remove_address_helper

common/address_helper: remove address_helper*

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37494 from idryomov/wip-mon-session-timeout-fix

mon: set session_timeout when adding to session_map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37526 from ifed01/wip-ifed-verify-range

os/bluestore: kill BitmapFreeListManager::_verify_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37423 from ronen-fr/wip-rf-scrub6

mon: fixing extra copying in some loops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #37524 from jschmid1/dont_skip_unavailable_devices

python-common: do not skip unavailable devices

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37525 from ifed01/wip-ifed-no-recursion-clist

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37528 from batrick/i47689

qa: ignore evicted client warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33963 from rosinL/wip-40668

mon/OSDMonitor: Add a safety builtin to mon_osd_report_timeout

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #37289 from pcuzner/prometheus-add-compr

mgr/prometheus: add pool compression stats

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36234 from vshankar/wip-rados-pybind-aio-notify

pybind/rados: add aio_notify()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36732 from rishabh-d-dave/vr-rotate-logs

vstart_runner: rotate logs between testcases

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37304 from tchaikov/wip-osd-message-ref

osd,crimson/osd: pass message using intrusive_ptr<>

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37076 from changchengx/env_to_args

common: refine env_to_vec implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34906 from ErwanAliasr1/patch-1

doc/config: osd_scrub_during_recovery is disabled by default

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark Operation::dump_detail() private

Operation::dump_detail() is called by Operation::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Operation::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark md_config_obs_t a private parent class

it registers itself as an observer in its ctor, so there is no need to
mark md_config_obs_t a public parent class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark dump_detail() private

Blocker::dump_detail() is called by Blocker::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Blocker::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make OrderedPipelinePhase::mutex private

as inner class has access to all outers class's members, even if they
are private.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37527 from tchaikov/wip-admin-use-feature

admin: be compatible with pip shipped by distro older than v20.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37502 from neha-ojha/wip-symlink-cephadm

qa/suites/rados: move thrash-old-clients under cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: ignore evicted client warnings for nautilus upgrades

Nautilus monitors do not note the client sessions in the mgrmap. So when
we upgrade the warnings are unavoidable.

Fixes: https://tracker.ceph.com/issues/47689
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: log reason for respawn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

admin: be compatible with pip shipped by distro older than v20.2

pip v20.2 introduced a new option named --use-feature, which is not
available with <20.2. so we need to check for it before using it. see
also
https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Fixes: https://tracker.ceph.com/issues/47636
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill BitmapFreeListManager::_verify_range()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

python-common: do not skip unavailable devices

Fixes: https://tracker.ceph.com/issues/47726
We should assemble the device set based on the drivegroups in
mgr/cephadm (regardless of their availability) and pass it to c-v.

This patch is the missing piece to actually do this.

related changes: https://github.com/ceph/ceph/pull/34740/commits/7d168ad7bdbb6d6d5231a4ae540ab03040b49a38

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34770 from Exotelis/wip-42963

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc/rbd: readability and spelling

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37448 from anthonyeleven/patch-1

doc/rados: erasure-code-lrc corrections

Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/suites/rados: move thrash-old-clients under cephadm

Since it uses cephadm, at the moment it makes sense to run it as a part of
rados/cephadm. This gives better test coverage for developers and has exposed
bugs such as https://tracker.ceph.com/issues/45421 and
https://tracker.ceph.com/issues/47709. We can always restructure this later.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37501 from linuxbox2/wip-file-hang

rgw_file: avoid long-ish delay on shutdown

Merge PR #36473 into master

* refs/pull/36473/head:
mds: remove snap past parents compat code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37392 from tchaikov/wip-doc-pip

admin: pass --use-feature=2020-resolver to pip

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37406 from MahatiC/wip-refactor-rwl

librbd/cache: Refactor common code for RWL and SSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37238 from cbodley/wip-rgw-sync-etag-verify

rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37316 from trociny/wip-46803-2

librbd: track complete async operation return code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37499 from tspmelo/wip-ng-profiler

mgr/dashboard: Enable Angular's debug tools when using dev mode

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix tox warning about PYTHONPATH

Get rid of the warning "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." by declaring "passenv=PYTHONPATH" according to the tox documentation.

> PYTHONPATH will be passed down if explicitly defined. If PYTHONPATH exists in the host environment but is not declared in passenv a warning will be emitted.

See https://tox.readthedocs.io/en/latest/config.html?highlight=passenv#conf-passenv.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add isort to rearrange imports

* Add 'isort' tox environment.
* Apply 'isort' fixes.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: update pylint to 2.6.0

* Update pylint to 2.6.0.
* Fix pylint issues.

Fixes: https://tracker.ceph.com/issues/47647
Signed-off-by: Volker Theile <vtheile@suse.com>
s

librbd/cache: Refactor common code for RWL and SSD

... from AbstractWriteLog class

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr/dashboard: Enable Angular's debug tools when using dev mode

Enabling this will allows us to run `ng.profiler.timeChangeDetection();`
in the browser.

Fixes: https://tracker.ceph.com/issues/47699
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: improve logs

Changes the text that is being displayed when no log entry was found.
Also changes the log level to debug.
User can no longer select a date in the future to filter log entries.

Fixes: https://tracker.ceph.com/issues/42963
Signed-off-by: Sebastian Krah <skrah@suse.com>

rgw_file: avoid long-ish delay on shutdown

Normal shutdown was likely to be delayed a substantial fraction
of rgw_nfs_namespace_expire_secs due to an ill-advised use of
this_thread::sleep_for.

Fixes: https://tracker.ceph.com/issues/47710
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: reset complete async request expiration time

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: mark quiesce notification complete after quiesce complete

Previoulsy it was marked complete only after unqiesce was
complete, which did not allow us to use the error code stored in
m_async_complete for a quiesce dup coming between the quiesce was
complete and unqiesce was complete.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37443 from ktdreyer/node-12.18.2

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36771 from rhcs-dashboard/wip-47118-master

mgr/dashboard: add JUnit reports to Dashboard e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34740 from jan--f/c-v-refactor-batch-use-create

ceph-volume: major batch refactor

Merge pull request #37014 from lixiaoy1/rwl_plugin

rbd/cache: update RWL as ImageDispatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: set session_timeout when adding to session_map

With msgr2, the session is added in Monitor::ms_handle_accept()
which is queued by ProtocolV2 at the end of handling CLIENT_IDENT
frame, before responding with SERVER_IDENT frame.  session_timeout
is 0 and gets set only in Monitor::ms_dispatch(), so if the session
trimming code in Monitor::tick() gets to the session before the peer
receives our SERVER_IDENT, handles it, sends the first message and
we receive it, the session is wrongly closed.

This doesn't happen with msgr1, because there the session is added in
Monitor::ms_dispatch(), upon receive of the first message (MSG_AUTH).

Fixes: https://tracker.ceph.com/issues/47697
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37030 from Amrita42/wip-doc-Amy-radosgw

doc: Bucket notifications in the radosgateway-the usage of 'if not'

Merge pull request #37312 from votdev/issue_47578_clipboard_firefox

mgr/dashboard: Copy to clipboard does not work in Firefox

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: add JUnit reports to tests

Fixes: https://tracker.ceph.com/issues/47118
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37411 from p-se/fix-cephadm-custom-template-escaping

mgr/cephadm: do not escape HTML for custom templates

Merge pull request #36622 from dmick/wip-46665

mgr/cephadm: continue trying all hosts for service deployment

rbd/cache: update RWL as ImageDispatch

This is a preparation work to make ReplicatedWriteLog as a plugin.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/io: split IO check

Move IO check to interfaces in Utils.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37112 from sebastian-philipp/cephadm-services-types-mandatory

mgr/cephadm: Make type annotations for cephadm.services mandatory

Merge pull request #37397 from adk3798/mirror

mgr/cephadm: get rbd-mirror daemon-id when checking for strays

Merge pull request #37432 from sebastian-philipp/cephadm-upgrade-state-none

mgr/cephadm: upgrade_state can be literally "null"

common/BackTrace: print path without symbol

before ea6abcf34c11c80dec507f7dee8babd26440196c, we have

2: (()+0x14140) [0x7f4462af6140]

after ea6abcf34c11c80dec507f7dee8babd26440196c, we have

2: ()

after this change, we have

2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f7137e2f140]

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/common/address_helper: remove address_helper* because it will never be called

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

ceph-volume batch: fix very_fast_allocation plan and add tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch: call the right prepare method

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

python-common drive-groups: allow strings for wal and db size

This enables user to pass sizes like "10G", which batch now understands.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume inventory: add option to filter unwanted devices

Some device we never want to pass to the batch subcommand. For now this
includes devices that have a partition or are mounted on the machine.
One goal is to filter the root device, so it is not included on a batch
command and thus would contribute to its implicit sizing calculation.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: address review comments

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36392 from Devp00l/wip-46375

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: upgrade_state can be literally "null"

Fixes: https://tracker.ceph.com/issues/47580
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

vstart_runner: rotate logs between testcases optionally

Fixes: https://tracker.ceph.com/issues/46059
Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart.sh: write logrotate conf to build dir

This should allow vstart_runner.py to rotate logs and also should allow
vstart cluster users to rotate logs manually.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: rearrange exec_test()

Rearrange the code that triggers the test runner so that in the later
commits the way tests are triggered can be changed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: Speed improvements of pool details

Now all variables that are shown in a listing that trigger a
more complex render cycle will only be updated if pool properties have
changed and not only the time series data, which isn't shown anymore,
as it can be seen graphically enhanced in the pool listing.

The pool detail component now uses the `onPush` change detection
strategy, to only call `ngOnChanges` if one of the input variables
have changed.

The function that only updates variables if they have changed is
now available through a helper class in order to provide this
useful functionality through out the dashboard.

Fixes: https://tracker.ceph.com/issues/46375
Signed-off-by: Stephan Müller <smueller@suse.com>

doc/rados: erasure-code-lrc corrections

doc/rados: erasure-code-lrc corrections

- Clarified / corrected jerasure recovery characterization cf. https://library.eecs.utk.edu/storage/125phpw0xFqAut-cs-07-603.pdf
- Capitalization
- Word choice:  OSDs are discrete / cardinal, so "fewer" instead of  "less"
- Reworked a few phrases to be less stilted

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37286 from dillaman/wip-46822

osdc/ObjectCacher: overwrite might cause stray read request callbacks

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37074 from dillaman/wip-librbd-list-snaps-api

librbd: generic internal API for computing snapshot deltas

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37281 from tchaikov/wip-fmt

rpm: add fmt-devel as build deps

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37417 from Songweibin/wip-rbd-metadata-update

librbd: should not permit update image metadata via snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37429 from trociny/wip-47665

rbd-nbd: don't ignore namespace when unmapping by image spec

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: build with NodeJS 12.18.2

Update the version of NodeJS that we use to build the dashboard.

RHEL 8 and CentOS 8 ship NodeJS 12.18.2, so I've picked this version to
make it easier to build outside of nodeenv.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #37399 from smithfarm/wip-relax-backporting-rule

doc/SubmittingPatches-backports.rst: clarify where to explain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37359 into master

* refs/pull/37359/head:
mon: test before adding/removing required_client_features

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36912 into master

* refs/pull/36912/head:
mds: defer encoding and storing the inode backtrace
mds: defer encoding and storing the CDir dentries
mds: add error handler with lock support

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36730 into master

* refs/pull/36730/head:
client: switch to unique_lock to avoid use the client_lock directly
client: unlock client_lock when copying data to bufferlist
client: do more check for the client_lock
client: simplify the iterating related code to make it more readable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37424 from ronen-fr/rf_neo_comp

neorados: removing an unused captured variable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36016 from rhcs-dashboard/wip-45863-master

mgr/dashboard/api: generate static API documentation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Karen Norteman <knortema@redhat.com>

Merge pull request #37387 from rhcs-dashboard/expose-always-on

mgr/dashboard: cluster > manager modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard/api: generate offline API docs

Generate Sphinx documentation from OpenAPI YAML spec:
- Fixed Docs controller doc generation
- Now dashboard Python doesn't fails if dashboard/frontend/dist doesn't exist
- OpenAPI added to @ceph/api CODEOWNERS
- Renamed Ceph-Dashboard API to Ceph REST or RESTful API.
- OpenAPI Docs: fixed decorators and docstrings.
- Sphinx Docs: updated dashboard and dev ones.

Co-authored-by: Ernesto Puerta <epuertat@redhat.com>
Fixes: https://tracker.ceph.com/issues/45863
Signed-off-by: Courtney Caldwell <ccaldwel@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37328 from badone/wip-npm-cache-config-changes

cmake: Don't try to access users' home directory

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: add no-systemd argument to zap

In some container deployment scenarios, systemd services are not
used, so the no-systemd parameter is useful.

Fixes: https://tracker.ceph.com/issues/47541
Signed-off-by: wanghongxu <wanghongxu@t2cloud.net>

cmake: Don't try to access users' home directory

Don't store .npmrc or cache directory in a user's home directory.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rbd-nbd: don't ignore namespace when unmapping by image spec

Fixes: https://tracker.ceph.com/issues/47665
Signed-off-by: Mykola Golub <mgolub@suse.com>