Merge PR #36457 into master

* refs/pull/36457/head:
qa/tasks/cephfs: fix fs authorize cmd args
mds: add root_squash mode in MDS auth caps
vstart_runner: add write_file and sudo_write_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36556 into master

* refs/pull/36556/head:
doc: cephfs mirroring design document

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37483 from dzafman/wip-46405

osd/osd-rep-recov-eio.sh: TEST_rados_repair_warning:  return 1

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37500 from ricardoasmarques/fix-bootstrap-container-init

cephadm: Fix error setting 'mgr/cephadm/container_init' config

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37582 from ivancich/wip-fix-bucket-list-namespace

rgw: fix setting of namespace in ordered and unordered bucket listing

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37572 from ivancich/wip-orphan-list-namespace-locator

rgw: allow rgw-orphan-list to note when rados objects are in namespace

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37511 from ivancich/wip-paginate-rgw-bucket-list

rgw: radosgw-admin should paginate internally when listing bucket

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #37584 from rhcs-dashboard/report-preview-helper

mgr/dashboard: Add short descriptions to the telemetry report preview

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #37591 into master

* refs/pull/37591/head:
ceph-volume: don't exit before empty report can be printed

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: cephfs mirroring design document

Signed-off-by: Venky Shankar <vshankar@redhat.com>

ceph-volume: don't exit before empty report can be printed

get_plan() called exit in case of an empty plan. This prevented a report
being printed under these circumstances. Avoid exit in this case. Also
adds tests to ensure an empty report is printed.

Fixes: https://tracker.ceph.com/issues/47760
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37561 from tchaikov/wip-cmake-node-mirror

cmake: support Node.js and npm registry mirror

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37570 from ceph/wip-install-ceph-deploy

doc: add a link to ceph-deploy document

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37446 into master

* refs/pull/37446/head:
mds: remove dead mdsmap broadcast code

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #37497 into master

* refs/pull/37497/head:
client,mds: Drop execute permission bits

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37571 into master

* refs/pull/37571/head:
qa/tasks/cephfs/nfs: Check ganesha daemon status after restarting mgr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37579 into master

* refs/pull/37579/head:
SubmittingPatches: use "doc" prefix in title of doc-only commits

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Add short descriptions to the telemetry report preview

A small info icon with a popover on the telemetry report preview form step2 for
Report ID and Report Preview.

Report ID: "A randomized UUID to identify a particular cluster over the course of several telemetry reports."
Report preview: "The actual telemetry data that will be submitted."

Fixes: https://tracker.ceph.com/issues/47610
Signed-off-by: Nizamudeen A <nia@redhat.com>

rgw: fix setting of namespace in ordered and unordered bucket listing

The namespace is not always set correctly during bucket listing. This
can, for example, cause the listing of incomplete multipart uploads,
which are in the _multipart_ namespace, to not paginate correctly, and
cause entries to be re-listed.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #37577 into master

* refs/pull/37577/head:
PendingReleaseNotes: add note about batch refactor

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: allow rgw-orphan-list to note when rados objects are in namespace

Currently namespaces and locators are ignored when `rados ls` is run
by rgw-orphan-list to record RADOS's known objects.

However there have been cases where RADOS objects have a locator, and
when one is included in the listing, the script does not handle it
correctly. Now when objects have locators, we will prevent their
output from entering the .intermediate file.

Additionally we do not expect RGW data objects to be in RADOS
namespaces, so when a namespaced object is detected, we'll error out
with a message.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

PendingReleaseNotes: add note about batch refactor

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

SubmittingPatches: use "doc" prefix in title of doc-only commits

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37498 from jan--f/cadm-new-batch-report

pybind/mgr/orchestrator: fix preview for new c-v batch output

Merge pull request #37556 from dillaman/wip-rbd-status

rbd: cleanup code and output formatting in status action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37521 from jschmid1/remove_orch_osd_spec_mention

docs: <orch osd spec> is no longer present

Merge pull request #37574 from tchaikov/wip-crimson-doc

doc/crimson: cleanups

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/crimson/poseidonstore: reformat

to render it as expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: move dev/crimson.rst into dev/crimson/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36343 from myoungwon/wip-doc-poseidon-store

doc: introduce PoseidonStore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37567 from liewegas/wip-stable-mod

include: ceph_stable_mod in public domain

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/cephfs/nfs: Check ganesha daemon status after restarting mgr

Fixes: https://tracker.ceph.com/issues/47591
Signed-off-by: Varsha Rao <varao@redhat.com>

doc: add a link to ceph-deploy document

Merge pull request #37565 from tchaikov/wip-crimson-cleanup

crimson: do not capture unused variables

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: mark class with "final" dtor "final"

to silence warning like

scheduler.cc:145:30: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
  ~ClassedOpQueueScheduler() final {};
                             ^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-pr/src/crimson/osd/scheduler/scheduler.cc:52:7: note: mark 'ClassedOpQueueScheduler<T>' as 'final' to silence this warning
class ClassedOpQueueScheduler : public Scheduler {
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark dtor of interface class "virtual"

we use a smart_ptr in Seastore to manage the life cycle of
`TransactionManager` which is in turn a subclass
`ExtentCallbackInterface`, so should declare the dtor of latter as
virtual.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: cleanup code and output formatting in status action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

include: ceph_stable_mod in public domain

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37395 from tchaikov/wip-doc-crimson

doc/dev/crimson: add instruction on running install-deps.sh for build…

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: do not capture unused variables

also silence warnings like:

Signe-off-by: Kefu Chai <kchai@redhat.cop_btree_node.h:103:36: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
                                  [this, ec, len, &extents] (auto i) {
                                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up document fixes

pick up changes to address the split script failure with HTML generated
with the latest pandoc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add links to seastar documents

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37523 from Exotelis/wip-42404

mgr/dashboard: Show warning when replicated size is 1

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/dev/crimson: add instruction on running install-deps.sh for building crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: introduce Poseidon store

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Jiwoong Park <jw12.park@samsung.com>
Signed-off-by: Kyuho Son <kyuho.son@samsung.com>
Signed-off-by: Sungkyu Park <sk921.park@samsung.com>

cmake: read $NPM_REGISTRY for npm registry mirror

in case the offical npm registry is slow or unreachable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract add_npm_options()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: read $NODE_MIRROR for Node.js mirror

in case the offical Node.js https://nodejs.org/dist/ is slow or
unreachable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37134 from myoungwon/wip-tier-evict

osd: add tier_evict

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37189 from orozery/striper-assemble-multiple-extents

librbd: support assembling results of multiple object extents

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37509 from lixiaoy1/fix_rwl

rbd/tools: fix rwl related names

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd/tools: fix rwl related names

These attributes have been renamed.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37507 from ofriedma/wip-ofriedma-leaked-logs

rgw: print to log leaked obj for any debug level

Merge PR #37531 into master

* refs/pull/37531/head:
ceph-volume batch: return valid empty json reports

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #37545 into master

* refs/pull/37545/head:
ceph-volume: pass filter_for_batch as keyword argument

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

librbd: support assembling sparse results of multiple object extents

Currently, Striper supports assembling results representing a single object extent.
Recently, the object dispatch API was extended allowing to read multiple object extents per rados operation.
This commit enables the Striper to correctly un-sparsify the results of the new read extents API.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

ceph-volume: pass filter_for_batch as keyword argument

This PR also removes an unused ctor argument in the Devices class.

Fixes: 7d168ad7bdbb6d6d5231a4ae540ab03040b49a38
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37414 from votdev/issue_47647_pylint_isort

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #37221 into master

* refs/pull/37221/head:
ceph-volume: add no-systemd argument to zap

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37532 from tchaikov/wip-crimson-more-privacy

crimson: mark more methods private

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37516 from anthonyeleven/anthonyeleven/misc-doc-tweaks

doc/rbd: readability and spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37085 from sunnyku/wip-fix-46743

mgr/devicehealth: device_health_metrics pool gets created even without any OSDs in the cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #37487 from tchaikov/wip-backtrace

common/BackTrace: print path without symbol

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37489 from Rethan/remove_address_helper

common/address_helper: remove address_helper*

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37494 from idryomov/wip-mon-session-timeout-fix

mon: set session_timeout when adding to session_map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37526 from ifed01/wip-ifed-verify-range

os/bluestore: kill BitmapFreeListManager::_verify_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37423 from ronen-fr/wip-rf-scrub6

mon: fixing extra copying in some loops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #37524 from jschmid1/dont_skip_unavailable_devices

python-common: do not skip unavailable devices

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37525 from ifed01/wip-ifed-no-recursion-clist

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37528 from batrick/i47689

qa: ignore evicted client warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33963 from rosinL/wip-40668

mon/OSDMonitor: Add a safety builtin to mon_osd_report_timeout

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #37289 from pcuzner/prometheus-add-compr

mgr/prometheus: add pool compression stats

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36234 from vshankar/wip-rados-pybind-aio-notify

pybind/rados: add aio_notify()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36732 from rishabh-d-dave/vr-rotate-logs

vstart_runner: rotate logs between testcases

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37304 from tchaikov/wip-osd-message-ref

osd,crimson/osd: pass message using intrusive_ptr<>

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37076 from changchengx/env_to_args

common: refine env_to_vec implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34906 from ErwanAliasr1/patch-1

doc/config: osd_scrub_during_recovery is disabled by default

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume batch: return valid empty json reports

Fixes: https://tracker.ceph.com/issues/47729
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

crimson/osd: mark Operation::dump_detail() private

Operation::dump_detail() is called by Operation::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Operation::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark md_config_obs_t a private parent class

it registers itself as an observer in its ctor, so there is no need to
mark md_config_obs_t a public parent class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark dump_detail() private

Blocker::dump_detail() is called by Blocker::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Blocker::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make OrderedPipelinePhase::mutex private

as inner class has access to all outers class's members, even if they
are private.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37527 from tchaikov/wip-admin-use-feature

admin: be compatible with pip shipped by distro older than v20.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37502 from neha-ojha/wip-symlink-cephadm

qa/suites/rados: move thrash-old-clients under cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: ignore evicted client warnings for nautilus upgrades

Nautilus monitors do not note the client sessions in the mgrmap. So when
we upgrade the warnings are unavoidable.

Fixes: https://tracker.ceph.com/issues/47689
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: log reason for respawn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

admin: be compatible with pip shipped by distro older than v20.2

pip v20.2 introduced a new option named --use-feature, which is not
available with <20.2. so we need to check for it before using it. see
also
https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Fixes: https://tracker.ceph.com/issues/47636
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill BitmapFreeListManager::_verify_range()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Show warning when replicated size is 1

Issues a warning when creating a replicated pool and the replicated size
is 1. It won't stop the user from creating the pool, but will give the
user a hint that no replication will be created.

Fixes: https://tracker.ceph.com/issues/42404
Signed-off-by: Sebastian Krah <skrah@suse.com>

pybind/mgr/orchestrator: fix preview for new c-v batch output

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

python-common: do not skip unavailable devices

Fixes: https://tracker.ceph.com/issues/47726
We should assemble the device set based on the drivegroups in
mgr/cephadm (regardless of their availability) and pass it to c-v.

This patch is the missing piece to actually do this.

related changes: https://github.com/ceph/ceph/pull/34740/commits/7d168ad7bdbb6d6d5231a4ae540ab03040b49a38

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34770 from Exotelis/wip-42963

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

docs: <orch osd spec> is no longer present

Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc/rbd: readability and spelling

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37448 from anthonyeleven/patch-1

doc/rados: erasure-code-lrc corrections

Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/suites/rados: move thrash-old-clients under cephadm

Since it uses cephadm, at the moment it makes sense to run it as a part of
rados/cephadm. This gives better test coverage for developers and has exposed
bugs such as https://tracker.ceph.com/issues/45421 and
https://tracker.ceph.com/issues/47709. We can always restructure this later.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37501 from linuxbox2/wip-file-hang

rgw_file: avoid long-ish delay on shutdown

Merge PR #36473 into master

* refs/pull/36473/head:
mds: remove snap past parents compat code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test: Avoid races by waiting for PGs go clean before query

Fixes: https://tracker.ceph.com/issues/46405
Signed-off-by: David Zafman <dzafman@redhat.com>