os/bluestore: fix memory accounting in TwoQBufferCacheShard

Variable buffer_bytes has been redefined in TwoQBufferCacheShard,
causing PriCache to see always 0 usage when 2q cache was selected,
as it looks at it through BufferCacheShard::_get_bytes().

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #37498 from jan--f/cadm-new-batch-report

pybind/mgr/orchestrator: fix preview for new c-v batch output

Merge pull request #37556 from dillaman/wip-rbd-status

rbd: cleanup code and output formatting in status action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37521 from jschmid1/remove_orch_osd_spec_mention

docs: <orch osd spec> is no longer present

Merge pull request #37574 from tchaikov/wip-crimson-doc

doc/crimson: cleanups

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc/crimson/poseidonstore: reformat

to render it as expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: move dev/crimson.rst into dev/crimson/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36343 from myoungwon/wip-doc-poseidon-store

doc: introduce PoseidonStore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37567 from liewegas/wip-stable-mod

include: ceph_stable_mod in public domain

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37565 from tchaikov/wip-crimson-cleanup

crimson: do not capture unused variables

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: mark class with "final" dtor "final"

to silence warning like

scheduler.cc:145:30: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
  ~ClassedOpQueueScheduler() final {};
                             ^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-pr/src/crimson/osd/scheduler/scheduler.cc:52:7: note: mark 'ClassedOpQueueScheduler<T>' as 'final' to silence this warning
class ClassedOpQueueScheduler : public Scheduler {
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark dtor of interface class "virtual"

we use a smart_ptr in Seastore to manage the life cycle of
`TransactionManager` which is in turn a subclass
`ExtentCallbackInterface`, so should declare the dtor of latter as
virtual.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: cleanup code and output formatting in status action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

include: ceph_stable_mod in public domain

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37395 from tchaikov/wip-doc-crimson

doc/dev/crimson: add instruction on running install-deps.sh for build…

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: do not capture unused variables

also silence warnings like:

Signe-off-by: Kefu Chai <kchai@redhat.cop_btree_node.h:103:36: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
                                  [this, ec, len, &extents] (auto i) {
                                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up document fixes

pick up changes to address the split script failure with HTML generated
with the latest pandoc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add links to seastar documents

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37523 from Exotelis/wip-42404

mgr/dashboard: Show warning when replicated size is 1

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/dev/crimson: add instruction on running install-deps.sh for building crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: introduce Poseidon store

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Jiwoong Park <jw12.park@samsung.com>
Signed-off-by: Kyuho Son <kyuho.son@samsung.com>
Signed-off-by: Sungkyu Park <sk921.park@samsung.com>

Merge pull request #37134 from myoungwon/wip-tier-evict

osd: add tier_evict

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37189 from orozery/striper-assemble-multiple-extents

librbd: support assembling results of multiple object extents

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37509 from lixiaoy1/fix_rwl

rbd/tools: fix rwl related names

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd/tools: fix rwl related names

These attributes have been renamed.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37507 from ofriedma/wip-ofriedma-leaked-logs

rgw: print to log leaked obj for any debug level

Merge PR #37531 into master

* refs/pull/37531/head:
ceph-volume batch: return valid empty json reports

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #37545 into master

* refs/pull/37545/head:
ceph-volume: pass filter_for_batch as keyword argument

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

librbd: support assembling sparse results of multiple object extents

Currently, Striper supports assembling results representing a single object extent.
Recently, the object dispatch API was extended allowing to read multiple object extents per rados operation.
This commit enables the Striper to correctly un-sparsify the results of the new read extents API.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

ceph-volume: pass filter_for_batch as keyword argument

This PR also removes an unused ctor argument in the Devices class.

Fixes: 7d168ad7bdbb6d6d5231a4ae540ab03040b49a38
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37414 from votdev/issue_47647_pylint_isort

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #37221 into master

* refs/pull/37221/head:
ceph-volume: add no-systemd argument to zap

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37532 from tchaikov/wip-crimson-more-privacy

crimson: mark more methods private

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37516 from anthonyeleven/anthonyeleven/misc-doc-tweaks

doc/rbd: readability and spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37085 from sunnyku/wip-fix-46743

mgr/devicehealth: device_health_metrics pool gets created even without any OSDs in the cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #37487 from tchaikov/wip-backtrace

common/BackTrace: print path without symbol

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37489 from Rethan/remove_address_helper

common/address_helper: remove address_helper*

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37494 from idryomov/wip-mon-session-timeout-fix

mon: set session_timeout when adding to session_map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37526 from ifed01/wip-ifed-verify-range

os/bluestore: kill BitmapFreeListManager::_verify_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37423 from ronen-fr/wip-rf-scrub6

mon: fixing extra copying in some loops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #37524 from jschmid1/dont_skip_unavailable_devices

python-common: do not skip unavailable devices

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37525 from ifed01/wip-ifed-no-recursion-clist

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37528 from batrick/i47689

qa: ignore evicted client warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33963 from rosinL/wip-40668

mon/OSDMonitor: Add a safety builtin to mon_osd_report_timeout

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #37289 from pcuzner/prometheus-add-compr

mgr/prometheus: add pool compression stats

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36234 from vshankar/wip-rados-pybind-aio-notify

pybind/rados: add aio_notify()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36732 from rishabh-d-dave/vr-rotate-logs

vstart_runner: rotate logs between testcases

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37304 from tchaikov/wip-osd-message-ref

osd,crimson/osd: pass message using intrusive_ptr<>

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37076 from changchengx/env_to_args

common: refine env_to_vec implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34906 from ErwanAliasr1/patch-1

doc/config: osd_scrub_during_recovery is disabled by default

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume batch: return valid empty json reports

Fixes: https://tracker.ceph.com/issues/47729
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

crimson/osd: mark Operation::dump_detail() private

Operation::dump_detail() is called by Operation::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Operation::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark md_config_obs_t a private parent class

it registers itself as an observer in its ctor, so there is no need to
mark md_config_obs_t a public parent class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark dump_detail() private

Blocker::dump_detail() is called by Blocker::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Blocker::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make OrderedPipelinePhase::mutex private

as inner class has access to all outers class's members, even if they
are private.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37527 from tchaikov/wip-admin-use-feature

admin: be compatible with pip shipped by distro older than v20.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37502 from neha-ojha/wip-symlink-cephadm

qa/suites/rados: move thrash-old-clients under cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: ignore evicted client warnings for nautilus upgrades

Nautilus monitors do not note the client sessions in the mgrmap. So when
we upgrade the warnings are unavoidable.

Fixes: https://tracker.ceph.com/issues/47689
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: log reason for respawn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

admin: be compatible with pip shipped by distro older than v20.2

pip v20.2 introduced a new option named --use-feature, which is not
available with <20.2. so we need to check for it before using it. see
also
https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Fixes: https://tracker.ceph.com/issues/47636
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill BitmapFreeListManager::_verify_range()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Show warning when replicated size is 1

Issues a warning when creating a replicated pool and the replicated size
is 1. It won't stop the user from creating the pool, but will give the
user a hint that no replication will be created.

Fixes: https://tracker.ceph.com/issues/42404
Signed-off-by: Sebastian Krah <skrah@suse.com>

pybind/mgr/orchestrator: fix preview for new c-v batch output

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

python-common: do not skip unavailable devices

Fixes: https://tracker.ceph.com/issues/47726
We should assemble the device set based on the drivegroups in
mgr/cephadm (regardless of their availability) and pass it to c-v.

This patch is the missing piece to actually do this.

related changes: https://github.com/ceph/ceph/pull/34740/commits/7d168ad7bdbb6d6d5231a4ae540ab03040b49a38

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34770 from Exotelis/wip-42963

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

docs: <orch osd spec> is no longer present

Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc/rbd: readability and spelling

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37448 from anthonyeleven/patch-1

doc/rados: erasure-code-lrc corrections

Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/suites/rados: move thrash-old-clients under cephadm

Since it uses cephadm, at the moment it makes sense to run it as a part of
rados/cephadm. This gives better test coverage for developers and has exposed
bugs such as https://tracker.ceph.com/issues/45421 and
https://tracker.ceph.com/issues/47709. We can always restructure this later.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37501 from linuxbox2/wip-file-hang

rgw_file: avoid long-ish delay on shutdown

Merge PR #36473 into master

* refs/pull/36473/head:
mds: remove snap past parents compat code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37392 from tchaikov/wip-doc-pip

admin: pass --use-feature=2020-resolver to pip

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37406 from MahatiC/wip-refactor-rwl

librbd/cache: Refactor common code for RWL and SSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37238 from cbodley/wip-rgw-sync-etag-verify

rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37316 from trociny/wip-46803-2

librbd: track complete async operation return code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37499 from tspmelo/wip-ng-profiler

mgr/dashboard: Enable Angular's debug tools when using dev mode

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix tox warning about PYTHONPATH

Get rid of the warning "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." by declaring "passenv=PYTHONPATH" according to the tox documentation.

> PYTHONPATH will be passed down if explicitly defined. If PYTHONPATH exists in the host environment but is not declared in passenv a warning will be emitted.

See https://tox.readthedocs.io/en/latest/config.html?highlight=passenv#conf-passenv.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add isort to rearrange imports

* Add 'isort' tox environment.
* Apply 'isort' fixes.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: update pylint to 2.6.0

* Update pylint to 2.6.0.
* Fix pylint issues.

Fixes: https://tracker.ceph.com/issues/47647
Signed-off-by: Volker Theile <vtheile@suse.com>
s

rgw: print to log leaked obj for any debug level
print to log leaked obj for any debug level

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

librbd/cache: Refactor common code for RWL and SSD

... from AbstractWriteLog class

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr/dashboard: Enable Angular's debug tools when using dev mode

Enabling this will allows us to run `ng.profiler.timeChangeDetection();`
in the browser.

Fixes: https://tracker.ceph.com/issues/47699
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: improve logs

Changes the text that is being displayed when no log entry was found.
Also changes the log level to debug.
User can no longer select a date in the future to filter log entries.

Fixes: https://tracker.ceph.com/issues/42963
Signed-off-by: Sebastian Krah <skrah@suse.com>

rgw_file: avoid long-ish delay on shutdown

Normal shutdown was likely to be delayed a substantial fraction
of rgw_nfs_namespace_expire_secs due to an ill-advised use of
this_thread::sleep_for.

Fixes: https://tracker.ceph.com/issues/47710
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: reset complete async request expiration time

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: mark quiesce notification complete after quiesce complete

Previoulsy it was marked complete only after unqiesce was
complete, which did not allow us to use the error code stored in
m_async_complete for a quiesce dup coming between the quiesce was
complete and unqiesce was complete.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37443 from ktdreyer/node-12.18.2

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36771 from rhcs-dashboard/wip-47118-master

mgr/dashboard: add JUnit reports to Dashboard e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34740 from jan--f/c-v-refactor-batch-use-create

ceph-volume: major batch refactor

Merge pull request #37014 from lixiaoy1/rwl_plugin

rbd/cache: update RWL as ImageDispatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: set session_timeout when adding to session_map

With msgr2, the session is added in Monitor::ms_handle_accept()
which is queued by ProtocolV2 at the end of handling CLIENT_IDENT
frame, before responding with SERVER_IDENT frame.  session_timeout
is 0 and gets set only in Monitor::ms_dispatch(), so if the session
trimming code in Monitor::tick() gets to the session before the peer
receives our SERVER_IDENT, handles it, sends the first message and
we receive it, the session is wrongly closed.

This doesn't happen with msgr1, because there the session is added in
Monitor::ms_dispatch(), upon receive of the first message (MSG_AUTH).

Fixes: https://tracker.ceph.com/issues/47697
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37030 from Amrita42/wip-doc-Amy-radosgw

doc: Bucket notifications in the radosgateway-the usage of 'if not'

Merge pull request #37312 from votdev/issue_47578_clipboard_firefox

mgr/dashboard: Copy to clipboard does not work in Firefox

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: add JUnit reports to tests

Fixes: https://tracker.ceph.com/issues/47118
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37411 from p-se/fix-cephadm-custom-template-escaping

mgr/cephadm: do not escape HTML for custom templates

Merge pull request #36622 from dmick/wip-46665

mgr/cephadm: continue trying all hosts for service deployment

rbd/cache: update RWL as ImageDispatch

This is a preparation work to make ReplicatedWriteLog as a plugin.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/io: split IO check

Move IO check to interfaces in Utils.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37112 from sebastian-philipp/cephadm-services-types-mandatory

mgr/cephadm: Make type annotations for cephadm.services mandatory