rgw: allow rgw-orphan-list to note when rados objects are in namespace

Currently namespaces and locators are ignored when `rados ls` is run
by rgw-orphan-list to record RADOS's known objects.

However there have been cases where RADOS objects have a locator, and
when one is included in the listing, the script does not handle it
correctly. Now when objects have locators, we will prevent their
output from entering the .intermediate file.

Additionally we do not expect RGW data objects to be in RADOS
namespaces, so when a namespaced object is detected, we'll error out
with a message.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37567 from liewegas/wip-stable-mod

include: ceph_stable_mod in public domain

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37565 from tchaikov/wip-crimson-cleanup

crimson: do not capture unused variables

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: mark class with "final" dtor "final"

to silence warning like

scheduler.cc:145:30: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
  ~ClassedOpQueueScheduler() final {};
                             ^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-pr/src/crimson/osd/scheduler/scheduler.cc:52:7: note: mark 'ClassedOpQueueScheduler<T>' as 'final' to silence this warning
class ClassedOpQueueScheduler : public Scheduler {
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark dtor of interface class "virtual"

we use a smart_ptr in Seastore to manage the life cycle of
`TransactionManager` which is in turn a subclass
`ExtentCallbackInterface`, so should declare the dtor of latter as
virtual.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include: ceph_stable_mod in public domain

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #37395 from tchaikov/wip-doc-crimson

doc/dev/crimson: add instruction on running install-deps.sh for build…

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: do not capture unused variables

also silence warnings like:

Signe-off-by: Kefu Chai <kchai@redhat.cop_btree_node.h:103:36: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
                                  [this, ec, len, &extents] (auto i) {
                                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up document fixes

pick up changes to address the split script failure with HTML generated
with the latest pandoc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add links to seastar documents

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37523 from Exotelis/wip-42404

mgr/dashboard: Show warning when replicated size is 1

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/dev/crimson: add instruction on running install-deps.sh for building crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37134 from myoungwon/wip-tier-evict

osd: add tier_evict

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37189 from orozery/striper-assemble-multiple-extents

librbd: support assembling results of multiple object extents

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37509 from lixiaoy1/fix_rwl

rbd/tools: fix rwl related names

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd/tools: fix rwl related names

These attributes have been renamed.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37507 from ofriedma/wip-ofriedma-leaked-logs

rgw: print to log leaked obj for any debug level

Merge PR #37531 into master

* refs/pull/37531/head:
ceph-volume batch: return valid empty json reports

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #37545 into master

* refs/pull/37545/head:
ceph-volume: pass filter_for_batch as keyword argument

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

librbd: support assembling sparse results of multiple object extents

Currently, Striper supports assembling results representing a single object extent.
Recently, the object dispatch API was extended allowing to read multiple object extents per rados operation.
This commit enables the Striper to correctly un-sparsify the results of the new read extents API.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

ceph-volume: pass filter_for_batch as keyword argument

This PR also removes an unused ctor argument in the Devices class.

Fixes: 7d168ad7bdbb6d6d5231a4ae540ab03040b49a38
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37414 from votdev/issue_47647_pylint_isort

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #37221 into master

* refs/pull/37221/head:
ceph-volume: add no-systemd argument to zap

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37532 from tchaikov/wip-crimson-more-privacy

crimson: mark more methods private

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37516 from anthonyeleven/anthonyeleven/misc-doc-tweaks

doc/rbd: readability and spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37085 from sunnyku/wip-fix-46743

mgr/devicehealth: device_health_metrics pool gets created even without any OSDs in the cluster

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #37487 from tchaikov/wip-backtrace

common/BackTrace: print path without symbol

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37489 from Rethan/remove_address_helper

common/address_helper: remove address_helper*

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37494 from idryomov/wip-mon-session-timeout-fix

mon: set session_timeout when adding to session_map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37526 from ifed01/wip-ifed-verify-range

os/bluestore: kill BitmapFreeListManager::_verify_range()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37423 from ronen-fr/wip-rf-scrub6

mon: fixing extra copying in some loops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #37524 from jschmid1/dont_skip_unavailable_devices

python-common: do not skip unavailable devices

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37525 from ifed01/wip-ifed-no-recursion-clist

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37528 from batrick/i47689

qa: ignore evicted client warnings

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33963 from rosinL/wip-40668

mon/OSDMonitor: Add a safety builtin to mon_osd_report_timeout

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #37289 from pcuzner/prometheus-add-compr

mgr/prometheus: add pool compression stats

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36234 from vshankar/wip-rados-pybind-aio-notify

pybind/rados: add aio_notify()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #36732 from rishabh-d-dave/vr-rotate-logs

vstart_runner: rotate logs between testcases

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37304 from tchaikov/wip-osd-message-ref

osd,crimson/osd: pass message using intrusive_ptr<>

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37076 from changchengx/env_to_args

common: refine env_to_vec implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34906 from ErwanAliasr1/patch-1

doc/config: osd_scrub_during_recovery is disabled by default

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume batch: return valid empty json reports

Fixes: https://tracker.ceph.com/issues/47729
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

crimson/osd: mark Operation::dump_detail() private

Operation::dump_detail() is called by Operation::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Operation::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark md_config_obs_t a private parent class

it registers itself as an observer in its ctor, so there is no need to
mark md_config_obs_t a public parent class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark dump_detail() private

Blocker::dump_detail() is called by Blocker::dump(), the latter is a
member of Blocker. so there is no need to mark dump_detail() protected
or public to grant the access Blocker::dump() the access to this method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make OrderedPipelinePhase::mutex private

as inner class has access to all outers class's members, even if they
are private.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37527 from tchaikov/wip-admin-use-feature

admin: be compatible with pip shipped by distro older than v20.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37502 from neha-ojha/wip-symlink-cephadm

qa/suites/rados: move thrash-old-clients under cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: ignore evicted client warnings for nautilus upgrades

Nautilus monitors do not note the client sessions in the mgrmap. So when
we upgrade the warnings are unavoidable.

Fixes: https://tracker.ceph.com/issues/47689
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: log reason for respawn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

admin: be compatible with pip shipped by distro older than v20.2

pip v20.2 introduced a new option named --use-feature, which is not
available with <20.2. so we need to check for it before using it. see
also
https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Fixes: https://tracker.ceph.com/issues/47636
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: kill BitmapFreeListManager::_verify_range()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: avoid recursive calls in SimpleCollectionListIterator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Show warning when replicated size is 1

Issues a warning when creating a replicated pool and the replicated size
is 1. It won't stop the user from creating the pool, but will give the
user a hint that no replication will be created.

Fixes: https://tracker.ceph.com/issues/42404
Signed-off-by: Sebastian Krah <skrah@suse.com>

python-common: do not skip unavailable devices

Fixes: https://tracker.ceph.com/issues/47726
We should assemble the device set based on the drivegroups in
mgr/cephadm (regardless of their availability) and pass it to c-v.

This patch is the missing piece to actually do this.

related changes: https://github.com/ceph/ceph/pull/34740/commits/7d168ad7bdbb6d6d5231a4ae540ab03040b49a38

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34770 from Exotelis/wip-42963

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc/rbd: readability and spelling

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37448 from anthonyeleven/patch-1

doc/rados: erasure-code-lrc corrections

Reviewed-by: Zac Dover <zac.dover@gmail.com>

qa/suites/rados: move thrash-old-clients under cephadm

Since it uses cephadm, at the moment it makes sense to run it as a part of
rados/cephadm. This gives better test coverage for developers and has exposed
bugs such as https://tracker.ceph.com/issues/45421 and
https://tracker.ceph.com/issues/47709. We can always restructure this later.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37501 from linuxbox2/wip-file-hang

rgw_file: avoid long-ish delay on shutdown

Merge PR #36473 into master

* refs/pull/36473/head:
mds: remove snap past parents compat code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37392 from tchaikov/wip-doc-pip

admin: pass --use-feature=2020-resolver to pip

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37406 from MahatiC/wip-refactor-rwl

librbd/cache: Refactor common code for RWL and SSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37238 from cbodley/wip-rgw-sync-etag-verify

rgw: rgw_sync_obj_etag_verify accounts for compressed multipart uploads

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37316 from trociny/wip-46803-2

librbd: track complete async operation return code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37499 from tspmelo/wip-ng-profiler

mgr/dashboard: Enable Angular's debug tools when using dev mode

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix tox warning about PYTHONPATH

Get rid of the warning "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." by declaring "passenv=PYTHONPATH" according to the tox documentation.

> PYTHONPATH will be passed down if explicitly defined. If PYTHONPATH exists in the host environment but is not declared in passenv a warning will be emitted.

See https://tox.readthedocs.io/en/latest/config.html?highlight=passenv#conf-passenv.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add isort to rearrange imports

* Add 'isort' tox environment.
* Apply 'isort' fixes.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: update pylint to 2.6.0

* Update pylint to 2.6.0.
* Fix pylint issues.

Fixes: https://tracker.ceph.com/issues/47647
Signed-off-by: Volker Theile <vtheile@suse.com>
s

rgw: print to log leaked obj for any debug level
print to log leaked obj for any debug level

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

librbd/cache: Refactor common code for RWL and SSD

... from AbstractWriteLog class

Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

mgr/dashboard: Enable Angular's debug tools when using dev mode

Enabling this will allows us to run `ng.profiler.timeChangeDetection();`
in the browser.

Fixes: https://tracker.ceph.com/issues/47699
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: improve logs

Changes the text that is being displayed when no log entry was found.
Also changes the log level to debug.
User can no longer select a date in the future to filter log entries.

Fixes: https://tracker.ceph.com/issues/42963
Signed-off-by: Sebastian Krah <skrah@suse.com>

rgw_file: avoid long-ish delay on shutdown

Normal shutdown was likely to be delayed a substantial fraction
of rgw_nfs_namespace_expire_secs due to an ill-advised use of
this_thread::sleep_for.

Fixes: https://tracker.ceph.com/issues/47710
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: reset complete async request expiration time

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: mark quiesce notification complete after quiesce complete

Previoulsy it was marked complete only after unqiesce was
complete, which did not allow us to use the error code stored in
m_async_complete for a quiesce dup coming between the quiesce was
complete and unqiesce was complete.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37443 from ktdreyer/node-12.18.2

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36771 from rhcs-dashboard/wip-47118-master

mgr/dashboard: add JUnit reports to Dashboard e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34740 from jan--f/c-v-refactor-batch-use-create

ceph-volume: major batch refactor

Merge pull request #37014 from lixiaoy1/rwl_plugin

rbd/cache: update RWL as ImageDispatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: set session_timeout when adding to session_map

With msgr2, the session is added in Monitor::ms_handle_accept()
which is queued by ProtocolV2 at the end of handling CLIENT_IDENT
frame, before responding with SERVER_IDENT frame.  session_timeout
is 0 and gets set only in Monitor::ms_dispatch(), so if the session
trimming code in Monitor::tick() gets to the session before the peer
receives our SERVER_IDENT, handles it, sends the first message and
we receive it, the session is wrongly closed.

This doesn't happen with msgr1, because there the session is added in
Monitor::ms_dispatch(), upon receive of the first message (MSG_AUTH).

Fixes: https://tracker.ceph.com/issues/47697
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37030 from Amrita42/wip-doc-Amy-radosgw

doc: Bucket notifications in the radosgateway-the usage of 'if not'

Merge pull request #37312 from votdev/issue_47578_clipboard_firefox

mgr/dashboard: Copy to clipboard does not work in Firefox

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: add JUnit reports to tests

Fixes: https://tracker.ceph.com/issues/47118
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37411 from p-se/fix-cephadm-custom-template-escaping

mgr/cephadm: do not escape HTML for custom templates

Merge pull request #36622 from dmick/wip-46665

mgr/cephadm: continue trying all hosts for service deployment

rbd/cache: update RWL as ImageDispatch

This is a preparation work to make ReplicatedWriteLog as a plugin.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/io: split IO check

Move IO check to interfaces in Utils.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37112 from sebastian-philipp/cephadm-services-types-mandatory

mgr/cephadm: Make type annotations for cephadm.services mandatory

Merge pull request #37397 from adk3798/mirror

mgr/cephadm: get rbd-mirror daemon-id when checking for strays

Merge pull request #37432 from sebastian-philipp/cephadm-upgrade-state-none

mgr/cephadm: upgrade_state can be literally "null"

common/BackTrace: print path without symbol

before ea6abcf34c11c80dec507f7dee8babd26440196c, we have

2: (()+0x14140) [0x7f4462af6140]

after ea6abcf34c11c80dec507f7dee8babd26440196c, we have

2: ()

after this change, we have

2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f7137e2f140]

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/common/address_helper: remove address_helper* because it will never be called

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

ceph-volume batch: fix very_fast_allocation plan and add tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: batch: call the right prepare method

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

python-common drive-groups: allow strings for wal and db size

This enables user to pass sizes like "10G", which batch now understands.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume inventory: add option to filter unwanted devices

Some device we never want to pass to the batch subcommand. For now this
includes devices that have a partition or are mounted on the machine.
One goal is to filter the root device, so it is not included on a batch
command and thus would contribute to its implicit sizing calculation.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: address review comments

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #36392 from Devp00l/wip-46375

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: upgrade_state can be literally "null"

Fixes: https://tracker.ceph.com/issues/47580
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>