doc/releases/nautilus: fix typo

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37745 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: add more privacy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37557 from wjwithagen/wjw-fix-warnings-clang-iterators

mds,osd: fix clang warnings to prevent unneeded copies

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37709 from tchaikov/wip-common-pick-address

common/pick_address: fail if cannot bind with specified network family

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/osd: add more privacy

* reorder the member variable and member function definitions so the
  private members are listed at the end of class. as requested by
  the CodingStyle
* mark some member variables and member functions private, as long as
  they are not used by other classes or non-friend classes.
* document UrgentRecovery
* refactor BackgroundRecovery::get_scheduler_params() to dedup
  the comments

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37739 from ivancich/wip-rados-ls-flush

tools/rados: flush formatter periodically during json output of `rados ls`

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge PR #37717 into master

* refs/pull/37717/head:
doc/cephfs: add sample cephfs-shell.conf

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37622 from ivancich/wip-rgw-orphan-list-comma

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

tools/rados: flush formatter periodically during json output of `rados ls`

While `rados ls` is emitting object info through a json formatter,
flush the formatter after there are at least 4096 bytes are buffered
for output.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37615 from rhcs-dashboard/remove-last-change-column

mgr/dashboard: Remove the "Last change" column in Pools table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37088 from Devp00l/wip-37408

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37416 from tspmelo/wip-rbd-clone

mgr/dashboard: Disable RBD clone action when conditions are not met

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37396 from SMIL-IT/fix-keycount

rgw: fix: S3 API KeyCount incorrect return.

Merge pull request #37727 from dillaman/wip-librbd-readresult-clip

librbd: remove io::ReadResult::set_clip_length

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37726 from dillaman/wip-librbd-remove-image-cache

librbd: remove remainder of ImageCache API hooks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires

Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/cephfs: add sample cephfs-shell.conf

And elaborate the description about it.

Fixes: https://tracker.ceph.com/issues/43028
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37403 from athanatos/sjust/wip-seastore-gc

crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37731 from adk3798/cephadm-logs

doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: document additional journal args in cephadm logs command

Signed-off-by: Adam King <adking@redhat.com>

test/crimson/seastore/test_btree_lba_manager: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs

insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals

Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add support for node dependent values

node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions

Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: extract common test logic for transaction_maanager initialization

This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: add online gc

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs

We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add space accounting to segment_cleaner and wire in

Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: rgw-orphan-list should use "plain" formatted `rados ls` output

The previous version that used "json-pretty" output for `rados ls`
added complications due to json's escaping of special characters. So
this version returns to the "plain" output for `rados ls` but deals
with entries (oids) that might have namespaces and/or locators as
well.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

librbd: remove io::ReadResult::set_clip_length

Combined the functionality with the newer 'set_image_extents' call
to simplify the call-site logic for IO reads.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #37707 into master

* refs/pull/37707/head:
doc: document the mds_autoscaler mgr plugin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: remove remainder of ImageCache API hooks

With the incorporation of the ImageDispatcher, there is no longer a
need for the ImageCache (and related) classes which were added as
a temporary workaround to incorporate the PWL cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37692 from tchaikov/wip-bump-up-pytest

pybind/mgr/dashboard: s/pytest<4/pytest/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

pybind/mgr/dashboard: disable too-many-branches lint warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use setUpClass for initializeing class

instead of relying on __init__(), use setUpClass() to initialize class
for testing. it turns out in pytest > 4, __init__() is called for the
test class but the attributes of the instantiated class is in turn overriden.

so we have to use setUpClass to do this job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: refactor overlong statement

to silence lint warning like:

services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent':'./services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent'}
2: 1     E126 continuation line over-indented for hanging indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: reindent to appease lint

silences warning like:

************* Module dashboard.services.tcmu_service
services/tcmu_service.py:64:0: C0330: Wrong continued indentation (add 18 spaces).
                    or [[0, 0]])[-1][1] / 1000000000
                    ^                 | (bad-continuation)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: s/pytest<4/pytest/

to address following failure:

    The user requested pytest<4
    pytest-cov 2.10.1 depends on pytest>=4.6

when building the target of "mgr-dashboard-virtualenv"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: bump flake8 to 3.8.4

to address the failure of

The conflict is caused by:
    flake8 3.7.8 depends on pycodestyle<2.6.0 and >=2.5.0
    autopep8 1.5.4 depends on pycodestyle>=2.6.0

when building the "mgr-dashboard-venv" target

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: pass --use-feature=2020-resolver to pip

as long as pip supports this option, pass it to `pip install`

to silence warnings and errors like:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

autopep8 1.5.4 requires pycodestyle>=2.6.0, but you'll have pycodestyle 2.5.0 which is incompatible.
pytest-cov 2.10.1 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36948 from bk201/wip-46492

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/test_ipaddr: do not leak CephContext

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: fail if cannot bind with specified network family

this change partially reverts 9f75dfbf364f5140b3f291e0a2c6769bc3d8cbac

we should not proceed against user's will if dual stack is specified but
only one network for a network family can be found. the right fix is
have better error message and documentation, not to tolerate the
failure.

Fixes: https://tracker.ceph.com/issues/46845
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_ipaddr: avoid using global

the CephContext created using `new` would suffice

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link unittest_ipaddr against GTest::Main

otherwise its main() is not defined, and fails to link.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: document the mds_autoscaler mgr plugin

The MDS Autoscaler provides automation to maintain a required count of
MDSs in the system.
The MDS count in the system can be afected by:
* max_mds config option
* standby_count_wanted config option
* death of an active MDS Rank

Fixes: https://tracker.ceph.com/issues/46884
Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge PR #37646 into master

* refs/pull/37646/head:
mds: account for closing sessions in hit_session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37608 into master

* refs/pull/37608/head:
mon/MDSMonitor: divide mds identifier and mds real name with dot

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37529 into master

* refs/pull/37529/head:
qa: set rados op timeouts for mds/ceph-fuse
qa: print debug info on mount cleanup
qa: remove redundant rmr
qa: use null mode to prevent undesired changes to mountpoint
qa: unmount all clients before deleting the file system
osdc: add timeout configs for mons/osds
common: accept timespan for SaferCond.wait_for

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #37294 into master

* refs/pull/37294/head:
mds: assert when inode has no parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/crimson/.../test_transaction_manager: reset ephemeral segment_manager state on replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: introduce weak transactions

This way, we can do a bulk scan of the store without building up
an unbounded amount of state in Transaction::read_set.  Note that
such transactions will not be snapshot isolated.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager/btree: hold a reference to parent until added to cache

Currently, we need to rely on the Transaction::read_set to ensure cache
residence of a leaf node until TransactionManager adds the logical
extent to the cache.  The next patch, however, will introduce a lazy
flag for Transaction's to enable doing snapshot inconsistent scans
without populating the read_set, so we'll want this to work without
it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: rearrange to put private members at end

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add scan_mappings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: expose segment_state_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/ephemeral: fix segment bounds checks

Also adds some debugging.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37644 from mgfritch/orch-device-lsm_data

mgr/orchestrator: set `lsm_data` to default of Dict type

mgr/cephadm: do not configure Dashboard Ganesha settings

The Dashboard can get cluster information from the Orchestrator.
For settings that are set by previous revisions, the Dashboard will
check them and ask user to remove them.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc/dashboard: add information for Orchestrator NFS-Ganesha clusters

Fixes: https://tracker.ceph.com/issues/46492
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: support Orchestrator and user-defined Ganesha clusters

This change make the Dashboard support two types of Ganesha clusters:

- Orchestrator clusters (Since Octopus)
  - Deployed by the Orchestrator.
  - The Dashboard gets the pool/namespace that stores Ganesha
    configuration objects from the Orchestrator.
  - The Dashboard gets the daemons in a cluster from the Orchestrator.

- User-defined clusters (Since Nautilus)
  - Clusters defined by using `ceph dashboard
    set-ganesha-clusters-rados-pool-namespace` command is treated as
    user-defined clusters.
  - Each daemon has its own RADOS configuration objects. The
    Dashboard uses these objects to deduce daemons.

Fixes: https://tracker.ceph.com/issues/46492
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: refator orchestrator service and daemon APIs

- Allow listing services by service_type.
- Allow listing daemons by daemon_type.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37175 from changchengx/log_start

common: remove log_early configuration option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37617 from rhcs-dashboard/46869-sass-modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37700 from tchaikov/wip-build-integration-branch

build-integration-branch: send token in the header

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37564 from s0nea/wip-dashboard-nfs-daemons-param

mgr/dashboard/qa: consider `daemons` parameter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37658 from neha-ojha/wip-crimson-suite

qa/crontab/teuthology-cronjobs: add crimson-rados

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common: remove log_early configuration option

After deciding to always enable tracking log in early phase, there's no
need to keep "log_early" option here and remove it directly.

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: remove useless log_early check condition here

The code logic shows that Log thread needs to be started even
log_early is false.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37315 from changchengx/paxos_extend

mon: define paxos services indexes using enum and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #36536 from matthewoliver/pick_address_warning_doc_update

common/pick_address: support IPv4 and IPv6 dual stack mode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37670 from changchengx/crypto_plugin

crypto: remove useless code and unify file name variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37635 from jecluis/wip-46816

mon: have 'mon stat' output json as well

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33392 from alram/master

osdmaptool: add --mark-{up,in} and --adjust-crush-weight options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36603 into master

* refs/pull/36603/head:
test: add test for setting ceph mirror virtual xattr
tests/pybind/cephfs: cleanup xattrs before starting tests
client: filter ceph.* xattrs from listing
client: force an attr fetch for ceph xattrs
client: changes for ceph.mirror.info xattr
mds: restrict setting/removing certain xattrs in ceph namespace
mds: introduce ceph.mirror.info virtual xattr
mds: customize xattr handling using dispatch handlers
mds: introduce is_ceph_vxattr() helper

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37699 from dillaman/wip-47880

journal: possible race condition between flush and append callback

Reviewed-by: Mykola Golub <mgolub@suse.com>

build-integration-branch: change to be pep8 compliant

Signed-off-by: Kefu Chai <kchai@redhat.com>

build-integration-branch: send token in the header

see
https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37698 from tchaikov/wip-crimson-do-until

crimson/common: schedule action only if the future is not available

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37695 from tchaikov/wip-dashboard-pep8

mgr/dashboard: do not use "l" for variable name

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37645 from dillaman/wip-librbd-aio-completion-race

librbd: update AioCompletion return value before evaluating pending count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37643 from dillaman/wip-47840

librbd: ignore -ENOENT error when disabling object-map

Reviewed-by: Mykola Golub <mgolub@suse.com>