mon/MonMap: fix unconditional failure for init_with_hosts

This bug prevents setting mon_host to a DNS name.

Fixes: https://tracker.ceph.com/issues/47951
Fixes: 7a1f02acfe6b5d8a760efd16bb594a0656b39eac
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/mon: add tests for mon_host build by hostname

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/mon: fix compiler errors in MonMap unittest

The code atrophied. Clean this up.

The tests are disabled because they SIGSEGV during SetUp.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/mon: build MonMap unittest

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37678 from ideepika/wip-drop-hammer-from-qa

qa: drop hammer branch qa tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37193 from rhcs-dashboard/log_download

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36449 from s0nea/wip-dashboard-osd-flags-indiv

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37440 from Devp00l/wip-44433

mgr/dashboard: Add clay plugin support

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37512 from tspmelo/wip-ng-validation

mgr/dashboard: Extract ng-bootstrap-form-validation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37748 from Rethan/msg_remove_dispatch_strategy

msg: move DispatchStrategy into test/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37404 from badone/wip-docker-test-allow-persistent-containers

test/docker-test: Allow persistent containers

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36754 from rzarzynski/wip-bs-filewriter-appender

common/bl, os/bluestore: eradicate bufferlist rebuilds caused by FileWriter of BlueFS

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/docker-test: Allow persistent containers

It can be useful to allow the container to persist for debugging
purposes. Also show the command used to create the container.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

msg: move DispatchStrategy into test/

Since commit cc9a914, xio related code were all removed, which made
direct_messenger become the only user of DispatchStraegy. So move
these code there.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

Merge pull request #37760 from athanatos/sjust/wip-fix-lba-leaf

crimson/seastore: remove invalid mutate_mapping assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36204 into master

* refs/pull/36204/head:
client: add command_lock support
client: defer swapping the newmap to mdsmap
client: try to grab the client_lock when needed
client: clean up the client code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: remove invalid mutate_mapping assert

A leaf node can have fewer than min_capacity items when it is the root.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: test mutate_mapping on more split/merge states

incref_mapping has a slightly different relationship with merge than
decref_mapping, update split/merge test to ensure that we call into
mutate_mapping at each step as the nodes fill, split, empty, and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37715 from anthonyeleven/rados-doc-improvements

doc/rados: operations.rst: clarity, detail, modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #37702 into master

* refs/pull/37702/head:
mon/MDSMonitor: do not ignore mds's down:dne request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37663 into master

* refs/pull/37663/head:
msg/MFSMap: remove unused variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37313 into master

* refs/pull/37313/head:
cephfs-mirror: restart mirroring on blocklist
cephfs-mirror: public method to fetch peer list
cephfs-mirror: dump rados address in "fs mirror status" asok command
cephfs-mirror: handle blocklist errors during rewatch
cephfs-mirror: periodically drive mirror enable, disable and peer updates
cephfs-mirror: identify a ceph filesystem using (id, name)
cephfs-mirror: introduce Filesystem type
cephfs-mirror: record failed FSMirror instances
cephfs-mirror: protect FSMirror::is_stopping() under lock
cephfs-mirror: unregister asok commands outside lock
cephfs-mirror: spawn peer replayers on peer changes
cephfs-mirror: s/blacklist/blocklist/i
cephfs-mirror: s/dir_name/dir_path/

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36004 into master

* refs/pull/36004/head:
mds: detect mds-mds messages not of type MMDSOp
messages: inherit MMDSPing and MMDSMetrics from MMDSOp

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36544 into master

* refs/pull/36544/head:
vstart_runner: omit result line only when testcases are run one by one
vstart_runner: don't quit running testsuite on a test failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Extract ng-bootstrap-form-validation

Remove ng-bootstrap-form-validation and
extract part of its code into the dashboard.

Fixes: https://tracker.ceph.com/issues/47721
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35355 from ofriedma/wip-ofriedma-beast-timeout

rgw: Add request timeout to beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37666 from pritha-srivastava/wip-rgw-gc-list-loop

rgw/gc: fixing the condition when marker for a queue is

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37409 from pritha-srivastava/wip-rgw-gc-perf-counter

rgw/gc: fix for incrementing the perf counter 'gc_retire_object'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #37752 into master

* refs/pull/37752/head:
vstart.sh: fix fs set max_mds bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36199 from ofriedma/wip-ofriedma-lc-fixes-tags-and

rgw: fix expiration header returned even if there is only one tag in …

mgr/dashboard: Created a Download and Copy-to-Clipboard option for the logs

Created Download Button Component.
Added a Download and Copy-to-Clipboard button for the Cluster and Audit Logs in the Log section. Also added a flag in the current Copy2ClipboardButton directive to indicate the incoming string is actually a log.

Fixes: https://tracker.ceph.com/issues/47498
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37745 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: add more privacy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart.sh: fix fs set max_mds bug

Fix a bug where the name used when creating a volume and the name used when setting max_mds were different.
Fixes: https://tracker.ceph.com/issues/47946
Signed-off-by: Jinmyeong Lee <jinmyeong.lee@linecorp.com>

Merge pull request #37557 from wjwithagen/wjw-fix-warnings-clang-iterators

mds,osd: fix clang warnings to prevent unneeded copies

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: detect mds-mds messages not of type MMDSOp

* Detect mds-mds messages not of type MMDSOp.
* Return true from MetricAggregator::ms_dispatch2 only when the peer is an MDS and the metrics is handled.

Fixes: https://tracker.ceph.com/issues/46426
Signed-off-by: Jos Collin <jcollin@redhat.com>

messages: inherit MMDSPing and MMDSMetrics from MMDSOp

The inter-MDS messages MMDSPing and MMDSMetrics should inherit from
MMDSOp before the MDS dispatch the message.

Fixes: https://tracker.ceph.com/issues/46426
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #37709 from tchaikov/wip-common-pick-address

common/pick_address: fail if cannot bind with specified network family

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: add more privacy

* reorder the member variable and member function definitions so the
  private members are listed at the end of class. as requested by
  the CodingStyle
* mark some member variables and member functions private, as long as
  they are not used by other classes or non-friend classes.
* document UrgentRecovery
* refactor BackgroundRecovery::get_scheduler_params() to dedup
  the comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37739 from ivancich/wip-rados-ls-flush

tools/rados: flush formatter periodically during json output of `rados ls`

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge PR #37717 into master

* refs/pull/37717/head:
doc/cephfs: add sample cephfs-shell.conf

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37622 from ivancich/wip-rgw-orphan-list-comma

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

tools/rados: flush formatter periodically during json output of `rados ls`

While `rados ls` is emitting object info through a json formatter,
flush the formatter after there are at least 4096 bytes are buffered
for output.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/rados/operations: clarity, detail, modernization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37615 from rhcs-dashboard/remove-last-change-column

mgr/dashboard: Remove the "Last change" column in Pools table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: assign flags to single OSDs

Add the possibility to assign the flags ['noup',
'nodown', 'noin', 'noout'] to single OSDs.

Fixes: https://tracker.ceph.com/issues/40739
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37088 from Devp00l/wip-37408

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37416 from tspmelo/wip-rbd-clone

mgr/dashboard: Disable RBD clone action when conditions are not met

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37396 from SMIL-IT/fix-keycount

rgw: fix: S3 API KeyCount incorrect return.

mgr/dashboard: remove custom class `oa-hr-small`

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: remove unnecessary blank

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37727 from dillaman/wip-librbd-readresult-clip

librbd: remove io::ReadResult::set_clip_length

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37726 from dillaman/wip-librbd-remove-image-cache

librbd: remove remainder of ImageCache API hooks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires

Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/cephfs: add sample cephfs-shell.conf

And elaborate the description about it.

Fixes: https://tracker.ceph.com/issues/43028
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37403 from athanatos/sjust/wip-seastore-gc

crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37731 from adk3798/cephadm-logs

doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: document additional journal args in cephadm logs command

Signed-off-by: Adam King <adking@redhat.com>

test/crimson/seastore/test_btree_lba_manager: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs

insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals

Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add support for node dependent values

node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions

Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: extract common test logic for transaction_maanager initialization

This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: add online gc

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs

We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add space accounting to segment_cleaner and wire in

Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

The previous version that used "json-pretty" output for `rados ls`
added complications due to json's escaping of special characters. So
this version returns to the "plain" output for `rados ls` but deals
with entries (oids) that might have namespaces and/or locators as
well.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

librbd: remove io::ReadResult::set_clip_length

Combined the functionality with the newer 'set_image_extents' call
to simplify the call-site logic for IO reads.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #37707 into master

* refs/pull/37707/head:
doc: document the mds_autoscaler mgr plugin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: remove remainder of ImageCache API hooks

With the incorporation of the ImageDispatcher, there is no longer a
need for the ImageCache (and related) classes which were added as
a temporary workaround to incorporate the PWL cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

client: add command_lock support

The Client::command_table member could get rid of the client_lock.
And this could make the client have a higher concurrency.

Fixes: https://tracker.ceph.com/issues/46620
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: defer swapping the newmap to mdsmap

This is preparing for adding the command lock support.

For the new mds map we could access it without the client_lock, so
before swaping it to the `mdsmap` member, it will be safe to access.

With this we can defer to hold the client_lock and it will be safe
for the new command_lock to avoid possible dead lock in future or
something else.

Fixes: https://tracker.ceph.com/issues/46620
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: try to grab the client_lock when needed

Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: clean up the client code

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37692 from tchaikov/wip-bump-up-pytest

pybind/mgr/dashboard: s/pytest<4/pytest/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

pybind/mgr/dashboard: disable too-many-branches lint warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use setUpClass for initializeing class

instead of relying on __init__(), use setUpClass() to initialize class
for testing. it turns out in pytest > 4, __init__() is called for the
test class but the attributes of the instantiated class is in turn overriden.

so we have to use setUpClass to do this job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: refactor overlong statement

to silence lint warning like:

services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent':'./services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent'}
2: 1     E126 continuation line over-indented for hanging indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: reindent to appease lint

silences warning like:

************* Module dashboard.services.tcmu_service
services/tcmu_service.py:64:0: C0330: Wrong continued indentation (add 18 spaces).
                    or [[0, 0]])[-1][1] / 1000000000
                    ^                 | (bad-continuation)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: s/pytest<4/pytest/

to address following failure:

    The user requested pytest<4
    pytest-cov 2.10.1 depends on pytest>=4.6

when building the target of "mgr-dashboard-virtualenv"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: bump flake8 to 3.8.4

to address the failure of

The conflict is caused by:
    flake8 3.7.8 depends on pycodestyle<2.6.0 and >=2.5.0
    autopep8 1.5.4 depends on pycodestyle>=2.6.0

when building the "mgr-dashboard-venv" target

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: pass --use-feature=2020-resolver to pip

as long as pip supports this option, pass it to `pip install`

to silence warnings and errors like:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

autopep8 1.5.4 requires pycodestyle>=2.6.0, but you'll have pycodestyle 2.5.0 which is incompatible.
pytest-cov 2.10.1 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36948 from bk201/wip-46492

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>