test/mon: fix compiler errors in MonMap unittest

The code atrophied. Clean this up.

The tests are disabled because they SIGSEGV during SetUp.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/mon: build MonMap unittest

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37678 from ideepika/wip-drop-hammer-from-qa

qa: drop hammer branch qa tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37193 from rhcs-dashboard/log_download

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36449 from s0nea/wip-dashboard-osd-flags-indiv

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37440 from Devp00l/wip-44433

mgr/dashboard: Add clay plugin support

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37512 from tspmelo/wip-ng-validation

mgr/dashboard: Extract ng-bootstrap-form-validation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37748 from Rethan/msg_remove_dispatch_strategy

msg: move DispatchStrategy into test/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37404 from badone/wip-docker-test-allow-persistent-containers

test/docker-test: Allow persistent containers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36754 from rzarzynski/wip-bs-filewriter-appender

common/bl, os/bluestore: eradicate bufferlist rebuilds caused by FileWriter of BlueFS

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/docker-test: Allow persistent containers

It can be useful to allow the container to persist for debugging
purposes. Also show the command used to create the container.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

msg: move DispatchStrategy into test/

Since commit cc9a914, xio related code were all removed, which made
direct_messenger become the only user of DispatchStraegy. So move
these code there.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

Merge pull request #37760 from athanatos/sjust/wip-fix-lba-leaf

crimson/seastore: remove invalid mutate_mapping assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36204 into master

* refs/pull/36204/head:
client: add command_lock support
client: defer swapping the newmap to mdsmap
client: try to grab the client_lock when needed
client: clean up the client code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: remove invalid mutate_mapping assert

A leaf node can have fewer than min_capacity items when it is the root.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: test mutate_mapping on more split/merge states

incref_mapping has a slightly different relationship with merge than
decref_mapping, update split/merge test to ensure that we call into
mutate_mapping at each step as the nodes fill, split, empty, and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37715 from anthonyeleven/rados-doc-improvements

doc/rados: operations.rst: clarity, detail, modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #37702 into master

* refs/pull/37702/head:
mon/MDSMonitor: do not ignore mds's down:dne request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37663 into master

* refs/pull/37663/head:
msg/MFSMap: remove unused variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37313 into master

* refs/pull/37313/head:
cephfs-mirror: restart mirroring on blocklist
cephfs-mirror: public method to fetch peer list
cephfs-mirror: dump rados address in "fs mirror status" asok command
cephfs-mirror: handle blocklist errors during rewatch
cephfs-mirror: periodically drive mirror enable, disable and peer updates
cephfs-mirror: identify a ceph filesystem using (id, name)
cephfs-mirror: introduce Filesystem type
cephfs-mirror: record failed FSMirror instances
cephfs-mirror: protect FSMirror::is_stopping() under lock
cephfs-mirror: unregister asok commands outside lock
cephfs-mirror: spawn peer replayers on peer changes
cephfs-mirror: s/blacklist/blocklist/i
cephfs-mirror: s/dir_name/dir_path/

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36004 into master

* refs/pull/36004/head:
mds: detect mds-mds messages not of type MMDSOp
messages: inherit MMDSPing and MMDSMetrics from MMDSOp

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36544 into master

* refs/pull/36544/head:
vstart_runner: omit result line only when testcases are run one by one
vstart_runner: don't quit running testsuite on a test failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Extract ng-bootstrap-form-validation

Remove ng-bootstrap-form-validation and
extract part of its code into the dashboard.

Fixes: https://tracker.ceph.com/issues/47721
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35355 from ofriedma/wip-ofriedma-beast-timeout

rgw: Add request timeout to beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37666 from pritha-srivastava/wip-rgw-gc-list-loop

rgw/gc: fixing the condition when marker for a queue is

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37409 from pritha-srivastava/wip-rgw-gc-perf-counter

rgw/gc: fix for incrementing the perf counter 'gc_retire_object'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #37752 into master

* refs/pull/37752/head:
vstart.sh: fix fs set max_mds bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36199 from ofriedma/wip-ofriedma-lc-fixes-tags-and

rgw: fix expiration header returned even if there is only one tag in …

mgr/dashboard: Created a Download and Copy-to-Clipboard option for the logs

Created Download Button Component.
Added a Download and Copy-to-Clipboard button for the Cluster and Audit Logs in the Log section. Also added a flag in the current Copy2ClipboardButton directive to indicate the incoming string is actually a log.

Fixes: https://tracker.ceph.com/issues/47498
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37745 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: add more privacy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart.sh: fix fs set max_mds bug

Fix a bug where the name used when creating a volume and the name used when setting max_mds were different.
Fixes: https://tracker.ceph.com/issues/47946
Signed-off-by: Jinmyeong Lee <jinmyeong.lee@linecorp.com>

Merge pull request #37557 from wjwithagen/wjw-fix-warnings-clang-iterators

mds,osd: fix clang warnings to prevent unneeded copies

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: detect mds-mds messages not of type MMDSOp

* Detect mds-mds messages not of type MMDSOp.
* Return true from MetricAggregator::ms_dispatch2 only when the peer is an MDS and the metrics is handled.

Fixes: https://tracker.ceph.com/issues/46426
Signed-off-by: Jos Collin <jcollin@redhat.com>

messages: inherit MMDSPing and MMDSMetrics from MMDSOp

The inter-MDS messages MMDSPing and MMDSMetrics should inherit from
MMDSOp before the MDS dispatch the message.

Fixes: https://tracker.ceph.com/issues/46426
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #37709 from tchaikov/wip-common-pick-address

common/pick_address: fail if cannot bind with specified network family

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: add more privacy

* reorder the member variable and member function definitions so the
  private members are listed at the end of class. as requested by
  the CodingStyle
* mark some member variables and member functions private, as long as
  they are not used by other classes or non-friend classes.
* document UrgentRecovery
* refactor BackgroundRecovery::get_scheduler_params() to dedup
  the comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37739 from ivancich/wip-rados-ls-flush

tools/rados: flush formatter periodically during json output of `rados ls`

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge PR #37717 into master

* refs/pull/37717/head:
doc/cephfs: add sample cephfs-shell.conf

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37622 from ivancich/wip-rgw-orphan-list-comma

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

tools/rados: flush formatter periodically during json output of `rados ls`

While `rados ls` is emitting object info through a json formatter,
flush the formatter after there are at least 4096 bytes are buffered
for output.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/rados/operations: clarity, detail, modernization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37615 from rhcs-dashboard/remove-last-change-column

mgr/dashboard: Remove the "Last change" column in Pools table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: assign flags to single OSDs

Add the possibility to assign the flags ['noup',
'nodown', 'noin', 'noout'] to single OSDs.

Fixes: https://tracker.ceph.com/issues/40739
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37088 from Devp00l/wip-37408

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37416 from tspmelo/wip-rbd-clone

mgr/dashboard: Disable RBD clone action when conditions are not met

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37396 from SMIL-IT/fix-keycount

rgw: fix: S3 API KeyCount incorrect return.

mgr/dashboard: remove custom class `oa-hr-small`

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: remove unnecessary blank

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37727 from dillaman/wip-librbd-readresult-clip

librbd: remove io::ReadResult::set_clip_length

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37726 from dillaman/wip-librbd-remove-image-cache

librbd: remove remainder of ImageCache API hooks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires

Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/cephfs: add sample cephfs-shell.conf

And elaborate the description about it.

Fixes: https://tracker.ceph.com/issues/43028
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37403 from athanatos/sjust/wip-seastore-gc

crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37731 from adk3798/cephadm-logs

doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: document additional journal args in cephadm logs command

Signed-off-by: Adam King <adking@redhat.com>

test/crimson/seastore/test_btree_lba_manager: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs

insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals

Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add support for node dependent values

node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions

Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: extract common test logic for transaction_maanager initialization

This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: add online gc

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs

We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add space accounting to segment_cleaner and wire in

Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

The previous version that used "json-pretty" output for `rados ls`
added complications due to json's escaping of special characters. So
this version returns to the "plain" output for `rados ls` but deals
with entries (oids) that might have namespaces and/or locators as
well.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

librbd: remove io::ReadResult::set_clip_length

Combined the functionality with the newer 'set_image_extents' call
to simplify the call-site logic for IO reads.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #37707 into master

* refs/pull/37707/head:
doc: document the mds_autoscaler mgr plugin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: remove remainder of ImageCache API hooks

With the incorporation of the ImageDispatcher, there is no longer a
need for the ImageCache (and related) classes which were added as
a temporary workaround to incorporate the PWL cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

client: add command_lock support

The Client::command_table member could get rid of the client_lock.
And this could make the client have a higher concurrency.

Fixes: https://tracker.ceph.com/issues/46620
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: defer swapping the newmap to mdsmap

This is preparing for adding the command lock support.

For the new mds map we could access it without the client_lock, so
before swaping it to the `mdsmap` member, it will be safe to access.

With this we can defer to hold the client_lock and it will be safe
for the new command_lock to avoid possible dead lock in future or
something else.

Fixes: https://tracker.ceph.com/issues/46620
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: try to grab the client_lock when needed

Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: clean up the client code

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37692 from tchaikov/wip-bump-up-pytest

pybind/mgr/dashboard: s/pytest<4/pytest/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

pybind/mgr/dashboard: disable too-many-branches lint warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use setUpClass for initializeing class

instead of relying on __init__(), use setUpClass() to initialize class
for testing. it turns out in pytest > 4, __init__() is called for the
test class but the attributes of the instantiated class is in turn overriden.

so we have to use setUpClass to do this job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: refactor overlong statement

to silence lint warning like:

services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent':'./services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent'}
2: 1     E126 continuation line over-indented for hanging indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: reindent to appease lint

silences warning like:

************* Module dashboard.services.tcmu_service
services/tcmu_service.py:64:0: C0330: Wrong continued indentation (add 18 spaces).
                    or [[0, 0]])[-1][1] / 1000000000
                    ^                 | (bad-continuation)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: s/pytest<4/pytest/

to address following failure:

    The user requested pytest<4
    pytest-cov 2.10.1 depends on pytest>=4.6

when building the target of "mgr-dashboard-virtualenv"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: bump flake8 to 3.8.4

to address the failure of

The conflict is caused by:
    flake8 3.7.8 depends on pycodestyle<2.6.0 and >=2.5.0
    autopep8 1.5.4 depends on pycodestyle>=2.6.0

when building the "mgr-dashboard-venv" target

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: pass --use-feature=2020-resolver to pip

as long as pip supports this option, pass it to `pip install`

to silence warnings and errors like:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

autopep8 1.5.4 requires pycodestyle>=2.6.0, but you'll have pycodestyle 2.5.0 which is incompatible.
pytest-cov 2.10.1 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36948 from bk201/wip-46492

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rgw/gc: fixing the condition when marker for a queue is
always reset to empty which causes RGWGC::list to get stuck in
a loop, which ultimately is broken out of when the queue's truncated
flag is false.

1. Check for entries size also while evaluating whether objects cache for
a gc object should be marked as 'transitioned' in case of cls_rgw_gc_list.
When there are no entries, we get back a return value of 0, and the
object cache is not marked as 'transitioned'.

2. Also for the last gc object, we need to check whether the queue is still
under process and set the correct flag.

Missing the two conditions above causes the GC::list to loop continously
over the same gc object.

Fixes: https://tracker.ceph.com/issues/47909
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/gc: fix for incrementing the perf counter 'gc_retire_object'
in the new gc queue code for omap offload, when gc objects from queue
are deleted. This was missed out initially.

Fixes: https://tracker.ceph.com/issues/47908
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>