crimson/osd: mark Operation::dump_detail() private

Operation::dump_detail() is called by Operation::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Operation::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark md_config_obs_t a private parent class

it registers itself as an observer in its ctor, so there is no need to
mark md_config_obs_t a public parent class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark dump_detail() private

Blocker::dump_detail() is called by Blocker::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Blocker::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make OrderedPipelinePhase::mutex private

as inner class has access to all outers class's members, even if they
are private.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37527 from tchaikov/wip-admin-use-feature

admin: be compatible with pip shipped by distro older than v20.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37502 from neha-ojha/wip-symlink-cephadm

qa/suites/rados: move thrash-old-clients under cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>

admin: be compatible with pip shipped by distro older than v20.2

pip v20.2 introduced a new option named --use-feature, which is not
available with <20.2. so we need to check for it before using it. see
also
https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Fixes: https://tracker.ceph.com/issues/47636
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34770 from Exotelis/wip-42963

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37448 from anthonyeleven/patch-1

doc/rados: erasure-code-lrc corrections

Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/suites/rados: move thrash-old-clients under cephadm

Since it uses cephadm, at the moment it makes sense to run it as a part of
rados/cephadm. This gives better test coverage for developers and has exposed
bugs such as https://tracker.ceph.com/issues/45421 and
https://tracker.ceph.com/issues/47709. We can always restructure this later.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37501 from linuxbox2/wip-file-hang

rgw_file: avoid long-ish delay on shutdown

Merge PR #36473 into master

* refs/pull/36473/head:
mds: remove snap past parents compat code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37392 from tchaikov/wip-doc-pip

admin: pass --use-feature=2020-resolver to pip

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37406 from MahatiC/wip-refactor-rwl

librbd/cache: Refactor common code for RWL and SSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37238 from cbodley/wip-rgw-sync-etag-verify

rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37316 from trociny/wip-46803-2

librbd: track complete async operation return code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37499 from tspmelo/wip-ng-profiler

mgr/dashboard: Enable Angular's debug tools when using dev mode

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd/cache: Refactor common code for RWL and SSD

... from AbstractWriteLog class

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr/dashboard: Enable Angular's debug tools when using dev mode

Enabling this will allows us to run `ng.profiler.timeChangeDetection();`
in the browser.

Fixes: https://tracker.ceph.com/issues/47699
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: improve logs

Changes the text that is being displayed when no log entry was found.
Also changes the log level to debug.
User can no longer select a date in the future to filter log entries.

Fixes: https://tracker.ceph.com/issues/42963
Signed-off-by: Sebastian Krah <skrah@suse.com>

rgw_file: avoid long-ish delay on shutdown

Normal shutdown was likely to be delayed a substantial fraction
of rgw_nfs_namespace_expire_secs due to an ill-advised use of
this_thread::sleep_for.

Fixes: https://tracker.ceph.com/issues/47710
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: reset complete async request expiration time

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: mark quiesce notification complete after quiesce complete

Previoulsy it was marked complete only after unqiesce was
complete, which did not allow us to use the error code stored in
m_async_complete for a quiesce dup coming between the quiesce was
complete and unqiesce was complete.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37443 from ktdreyer/node-12.18.2

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36771 from rhcs-dashboard/wip-47118-master

mgr/dashboard: add JUnit reports to Dashboard e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34740 from jan--f/c-v-refactor-batch-use-create

ceph-volume: major batch refactor

Merge pull request #37014 from lixiaoy1/rwl_plugin

rbd/cache: update RWL as ImageDispatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37030 from Amrita42/wip-doc-Amy-radosgw

doc: Bucket notifications in the radosgateway-the usage of 'if not'

Merge pull request #37312 from votdev/issue_47578_clipboard_firefox

mgr/dashboard: Copy to clipboard does not work in Firefox

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: add JUnit reports to tests

Fixes: https://tracker.ceph.com/issues/47118
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37411 from p-se/fix-cephadm-custom-template-escaping

mgr/cephadm: do not escape HTML for custom templates

Merge pull request #36622 from dmick/wip-46665

mgr/cephadm: continue trying all hosts for service deployment

rbd/cache: update RWL as ImageDispatch

This is a preparation work to make ReplicatedWriteLog as a plugin.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/io: split IO check

Move IO check to interfaces in Utils.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37112 from sebastian-philipp/cephadm-services-types-mandatory

mgr/cephadm: Make type annotations for cephadm.services mandatory

Merge pull request #37397 from adk3798/mirror

mgr/cephadm: get rbd-mirror daemon-id when checking for strays

Merge pull request #37432 from sebastian-philipp/cephadm-upgrade-state-none

mgr/cephadm: upgrade_state can be literally "null"

ceph-volume batch: fix very_fast_allocation plan and add tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch: call the right prepare method

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

python-common drive-groups: allow strings for wal and db size

This enables user to pass sizes like "10G", which batch now understands.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume inventory: add option to filter unwanted devices

Some device we never want to pass to the batch subcommand. For now this
includes devices that have a partition or are mounted on the machine.
One goal is to filter the root device, so it is not included on a batch
command and thus would contribute to its implicit sizing calculation.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: address review comments

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36392 from Devp00l/wip-46375

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: upgrade_state can be literally "null"

Fixes: https://tracker.ceph.com/issues/47580
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Speed improvements of pool details

Now all variables that are shown in a listing that trigger a
more complex render cycle will only be updated if pool properties have
changed and not only the time series data, which isn't shown anymore,
as it can be seen graphically enhanced in the pool listing.

The pool detail component now uses the `onPush` change detection
strategy, to only call `ngOnChanges` if one of the input variables
have changed.

The function that only updates variables if they have changed is
now available through a helper class in order to provide this
useful functionality through out the dashboard.

Fixes: https://tracker.ceph.com/issues/46375
Signed-off-by: Stephan Müller <smueller@suse.com>

doc/rados: erasure-code-lrc corrections

doc/rados: erasure-code-lrc corrections

- Clarified / corrected jerasure recovery characterization cf. https://library.eecs.utk.edu/storage/125phpw0xFqAut-cs-07-603.pdf
- Capitalization
- Word choice:  OSDs are discrete / cardinal, so "fewer" instead of  "less"
- Reworked a few phrases to be less stilted

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37286 from dillaman/wip-46822

osdc/ObjectCacher: overwrite might cause stray read request callbacks

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37074 from dillaman/wip-librbd-list-snaps-api

librbd: generic internal API for computing snapshot deltas

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37281 from tchaikov/wip-fmt

rpm: add fmt-devel as build deps

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37417 from Songweibin/wip-rbd-metadata-update

librbd: should not permit update image metadata via snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37429 from trociny/wip-47665

rbd-nbd: don't ignore namespace when unmapping by image spec

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: build with NodeJS 12.18.2

Update the version of NodeJS that we use to build the dashboard.

RHEL 8 and CentOS 8 ship NodeJS 12.18.2, so I've picked this version to
make it easier to build outside of nodeenv.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #37399 from smithfarm/wip-relax-backporting-rule

doc/SubmittingPatches-backports.rst: clarify where to explain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37359 into master

* refs/pull/37359/head:
mon: test before adding/removing required_client_features

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36912 into master

* refs/pull/36912/head:
mds: defer encoding and storing the inode backtrace
mds: defer encoding and storing the CDir dentries
mds: add error handler with lock support

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36730 into master

* refs/pull/36730/head:
client: switch to unique_lock to avoid use the client_lock directly
client: unlock client_lock when copying data to bufferlist
client: do more check for the client_lock
client: simplify the iterating related code to make it more readable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37424 from ronen-fr/rf_neo_comp

neorados: removing an unused captured variable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36016 from rhcs-dashboard/wip-45863-master

mgr/dashboard/api: generate static API documentation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Karen Norteman <knortema@redhat.com>

Merge pull request #37387 from rhcs-dashboard/expose-always-on

mgr/dashboard: cluster > manager modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard/api: generate offline API docs

Generate Sphinx documentation from OpenAPI YAML spec:
- Fixed Docs controller doc generation
- Now dashboard Python doesn't fails if dashboard/frontend/dist doesn't exist
- OpenAPI added to @ceph/api CODEOWNERS
- Renamed Ceph-Dashboard API to Ceph REST or RESTful API.
- OpenAPI Docs: fixed decorators and docstrings.
- Sphinx Docs: updated dashboard and dev ones.

Co-authored-by: Ernesto Puerta <epuertat@redhat.com>
Fixes: https://tracker.ceph.com/issues/45863
Signed-off-by: Courtney Caldwell <ccaldwel@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37328 from badone/wip-npm-cache-config-changes

cmake: Don't try to access users' home directory

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Don't try to access users' home directory

Don't store .npmrc or cache directory in a user's home directory.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rbd-nbd: don't ignore namespace when unmapping by image spec

Fixes: https://tracker.ceph.com/issues/47665
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37282 from neha-ojha/wip-46323

qa/*/thrash_cache_writeback_proxy_none.yaml: remove cache-mode proxy

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37077 from wjwithagen/wjw-fix-cls_fifo.cc-__lea64

cls: replace __le{32,64} types by their Ceph counterparts

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37149 from ifed01/wip-ifed-fix-alloc-cmd-multi-reg

os/bluestore: provide a different name for fallback allocator

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37150 from pdvian/wip-crush-failure-domain

mon: validate crush-failure-domain

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37263 from ifed01/wip-ifed-more-structs-to-mempool

os/bluestore: add missing structs to bluestore_cache_other mempool

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37153 from tchaikov/wip-46124

blk/kernel: use open file description lock if available

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Niklas Hambüchen <mail@nh2.me>

Merge pull request #37329 from changchengx/add_conn

mon: set MonClient::_add_conn return type to void

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37321 from changchengx/mon_make

cmake: remove undefined source file lists

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: remove undefined source file lists

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

librbd: should not permit update image metadata via snapshot

Fixes: https://github.com/ceph/ceph/pull/37121
Signed-off-by: songweibin <song.weibin@zte.com.cn>

neorados: removing an unused captured variable

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37388 from johscheuer/add-missing-python-dependency

debian: Add missing Python dependency for ceph-mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37380 from changchengx/ceph_port

msgr: remove the macro not referenced anywhere

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36779 from athanatos/sjust/wip-dirty-writeout

crimson/seastore: add dirty extent writeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore/test_transaction_manager: add random overwrite test

Also improves replay() by recycling all structures except for the
segment_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add segment_cleaner

Adds SegmentCleaner component to manage rewriting
dirty segments.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add rewrite_extent

Adds support for writing out an extent to a new location and updating
lba mappings.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add more debugging for replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: initialize current_journal_segment in find_replay_segments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: return next write location from open_for_write

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: replay_delta handle obsolete extents properly

If we see a delta at verison v (v != 0) and we haven't seen version v-1,
we know that the extent is obsolete and can be ignored (presuming that
journal trimming is behaving).

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: pass journal_seq with each replay delta

We'll need this to track where extents became dirty during replay.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: maintain dirty_from value on dirty CachedExtents

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: list-snaps needs to writeout fancy-striped extents

If the delta is being generated for a fancy-stiped extent such that
the provided object extents are not within the actual snapshot deltas,
we still need to indicate that the object exists (as a whiteout) to
prevent the possibility of an incorrect read request to the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: get rbd-mirror daemon-id when checking for strays

Currently, list_servers() gets the rbd-mirror service-id instead
of the daemon-id so the daemon is marked as stray. This PR uses
that service-id to find the daemon-id and uses that to check if
the daemon is stray.

Fixes: https://tracker.ceph.com/issues/47639
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #37400 from cbodley/wip-rgw-DatalogTrimImplCR-logging

rgw: raise debug level of DatalogTrimImplCR message

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #36482 from cbodley/wip-rgw-bucket-sync-checkpoint

rgw: add 'bucket sync checkpoint' command to radosgw-admin

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/cephadm: do not escape HTML for custom templates

Fixes: https://tracker.ceph.com/issues/47648
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

ceph-volume: batch: fix size retrieval for lvs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: include encryption in batch report

Fixes: https://tracker.ceph.com/issues/44783
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume lvm batch: use namedtuple instead of tuple

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: address review comments, mostly tidying, clarification

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

idempotency must result in the same outcome

...not should

Co-authored-by: Joshua Schmid <jschmid@suse.de>
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch test should pass --journal-devices with filestore

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: make --journal optional, add --journal-slots

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

doc: update ceph-volume lvm batch docs

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

py-common/deployment: drop workarounds for old batch

This commit drops two workarounds in drive groups for the old batch
behaviour. OSDs on LVs no longer need to be created individually by
create (related to functional testing) and ceph-volume batch now filters
unavailable drives itself.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>