mgr/dashboard: assign flags to single OSDs

Add the possibility to assign the flags ['noup',
'nodown', 'noin', 'noout'] to single OSDs.

Fixes: https://tracker.ceph.com/issues/40739
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: remove custom class `oa-hr-small`

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: remove unnecessary blank

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires

Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37403 from athanatos/sjust/wip-seastore-gc

crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37731 from adk3798/cephadm-logs

doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: document additional journal args in cephadm logs command

Signed-off-by: Adam King <adking@redhat.com>

test/crimson/seastore/test_btree_lba_manager: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs

insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals

Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add support for node dependent values

node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions

Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: extract common test logic for transaction_maanager initialization

This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: add online gc

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs

We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add space accounting to segment_cleaner and wire in

Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #37707 into master

* refs/pull/37707/head:
doc: document the mds_autoscaler mgr plugin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37692 from tchaikov/wip-bump-up-pytest

pybind/mgr/dashboard: s/pytest<4/pytest/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

pybind/mgr/dashboard: disable too-many-branches lint warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use setUpClass for initializeing class

instead of relying on __init__(), use setUpClass() to initialize class
for testing. it turns out in pytest > 4, __init__() is called for the
test class but the attributes of the instantiated class is in turn overriden.

so we have to use setUpClass to do this job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: refactor overlong statement

to silence lint warning like:

services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent':'./services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent'}
2: 1     E126 continuation line over-indented for hanging indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: reindent to appease lint

silences warning like:

************* Module dashboard.services.tcmu_service
services/tcmu_service.py:64:0: C0330: Wrong continued indentation (add 18 spaces).
                    or [[0, 0]])[-1][1] / 1000000000
                    ^                 | (bad-continuation)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: s/pytest<4/pytest/

to address following failure:

    The user requested pytest<4
    pytest-cov 2.10.1 depends on pytest>=4.6

when building the target of "mgr-dashboard-virtualenv"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: bump flake8 to 3.8.4

to address the failure of

The conflict is caused by:
    flake8 3.7.8 depends on pycodestyle<2.6.0 and >=2.5.0
    autopep8 1.5.4 depends on pycodestyle>=2.6.0

when building the "mgr-dashboard-venv" target

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: pass --use-feature=2020-resolver to pip

as long as pip supports this option, pass it to `pip install`

to silence warnings and errors like:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

autopep8 1.5.4 requires pycodestyle>=2.6.0, but you'll have pycodestyle 2.5.0 which is incompatible.
pytest-cov 2.10.1 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36948 from bk201/wip-46492

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: document the mds_autoscaler mgr plugin

The MDS Autoscaler provides automation to maintain a required count of
MDSs in the system.
The MDS count in the system can be afected by:
* max_mds config option
* standby_count_wanted config option
* death of an active MDS Rank

Fixes: https://tracker.ceph.com/issues/46884
Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge PR #37646 into master

* refs/pull/37646/head:
mds: account for closing sessions in hit_session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37608 into master

* refs/pull/37608/head:
mon/MDSMonitor: divide mds identifier and mds real name with dot

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37529 into master

* refs/pull/37529/head:
qa: set rados op timeouts for mds/ceph-fuse
qa: print debug info on mount cleanup
qa: remove redundant rmr
qa: use null mode to prevent undesired changes to mountpoint
qa: unmount all clients before deleting the file system
osdc: add timeout configs for mons/osds
common: accept timespan for SaferCond.wait_for

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #37294 into master

* refs/pull/37294/head:
mds: assert when inode has no parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/crimson/.../test_transaction_manager: reset ephemeral segment_manager state on replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: introduce weak transactions

This way, we can do a bulk scan of the store without building up
an unbounded amount of state in Transaction::read_set.  Note that
such transactions will not be snapshot isolated.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager/btree: hold a reference to parent until added to cache

Currently, we need to rely on the Transaction::read_set to ensure cache
residence of a leaf node until TransactionManager adds the logical
extent to the cache.  The next patch, however, will introduce a lazy
flag for Transaction's to enable doing snapshot inconsistent scans
without populating the read_set, so we'll want this to work without
it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: rearrange to put private members at end

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add scan_mappings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: expose segment_state_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/ephemeral: fix segment bounds checks

Also adds some debugging.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37644 from mgfritch/orch-device-lsm_data

mgr/orchestrator: set `lsm_data` to default of Dict type

mgr/cephadm: do not configure Dashboard Ganesha settings

The Dashboard can get cluster information from the Orchestrator.
For settings that are set by previous revisions, the Dashboard will
check them and ask user to remove them.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc/dashboard: add information for Orchestrator NFS-Ganesha clusters

Fixes: https://tracker.ceph.com/issues/46492
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: support Orchestrator and user-defined Ganesha clusters

This change make the Dashboard support two types of Ganesha clusters:

- Orchestrator clusters (Since Octopus)
  - Deployed by the Orchestrator.
  - The Dashboard gets the pool/namespace that stores Ganesha
    configuration objects from the Orchestrator.
  - The Dashboard gets the daemons in a cluster from the Orchestrator.

- User-defined clusters (Since Nautilus)
  - Clusters defined by using `ceph dashboard
    set-ganesha-clusters-rados-pool-namespace` command is treated as
    user-defined clusters.
  - Each daemon has its own RADOS configuration objects. The
    Dashboard uses these objects to deduce daemons.

Fixes: https://tracker.ceph.com/issues/46492
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: refator orchestrator service and daemon APIs

- Allow listing services by service_type.
- Allow listing daemons by daemon_type.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37175 from changchengx/log_start

common: remove log_early configuration option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37617 from rhcs-dashboard/46869-sass-modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37700 from tchaikov/wip-build-integration-branch

build-integration-branch: send token in the header

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37564 from s0nea/wip-dashboard-nfs-daemons-param

mgr/dashboard/qa: consider `daemons` parameter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37658 from neha-ojha/wip-crimson-suite

qa/crontab/teuthology-cronjobs: add crimson-rados

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common: remove log_early configuration option

After deciding to always enable tracking log in early phase, there's no
need to keep "log_early" option here and remove it directly.

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: remove useless log_early check condition here

The code logic shows that Log thread needs to be started even
log_early is false.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37315 from changchengx/paxos_extend

mon: define paxos services indexes using enum and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #36536 from matthewoliver/pick_address_warning_doc_update

common/pick_address: support IPv4 and IPv6 dual stack mode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37670 from changchengx/crypto_plugin

crypto: remove useless code and unify file name variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37635 from jecluis/wip-46816

mon: have 'mon stat' output json as well

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33392 from alram/master

osdmaptool: add --mark-{up,in} and --adjust-crush-weight options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36603 into master

* refs/pull/36603/head:
test: add test for setting ceph mirror virtual xattr
tests/pybind/cephfs: cleanup xattrs before starting tests
client: filter ceph.* xattrs from listing
client: force an attr fetch for ceph xattrs
client: changes for ceph.mirror.info xattr
mds: restrict setting/removing certain xattrs in ceph namespace
mds: introduce ceph.mirror.info virtual xattr
mds: customize xattr handling using dispatch handlers
mds: introduce is_ceph_vxattr() helper

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37699 from dillaman/wip-47880

journal: possible race condition between flush and append callback

Reviewed-by: Mykola Golub <mgolub@suse.com>

build-integration-branch: change to be pep8 compliant

Signed-off-by: Kefu Chai <kchai@redhat.com>

build-integration-branch: send token in the header

see
https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37698 from tchaikov/wip-crimson-do-until

crimson/common: schedule action only if the future is not available

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37695 from tchaikov/wip-dashboard-pep8

mgr/dashboard: do not use "l" for variable name

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37645 from dillaman/wip-librbd-aio-completion-race

librbd: update AioCompletion return value before evaluating pending count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37643 from dillaman/wip-47840

librbd: ignore -ENOENT error when disabling object-map

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37444 from dillaman/wip-librbd-copyup-api

librbd: support the ability to process parent data prior to copyup

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Or Ozeri <oro@il.ibm.com>

journal: possible race condition between flush and append callback

When notifying the journal recorder of an overflow or if the object
close request has completed due to no more in-flight IO, it was
possible for a race between a flush request and the processing of
an append completion to attempt to kick off duplicate notifications.
Since the overflowed and closed callbacks are properly protected from
duplicates, use a counter instead of a boolean to track possible
in-flight handler callbacks.

Fixes: https://tracker.ceph.com/issues/47880
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crimson/common: schedule action only if the future is not available

otherwise we could call do_until() recursively if we have other tasks
which need to prempt the reactor and current future's state is actually
always available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: do not use "l" for variable name

see also https://www.flake8rules.com/rules/E741.html

also silences flake8 warnings like:

2: {tty:'./controllers/home.py:90:26: E741 ambiguous variable name 'l'':'./controllers/home.py:90:26: E741 ambiguous variable name 'l''}
2: {tty:'./controllers/logs.py:52:13: E741 ambiguous variable name 'l'':'./controllers/logs.py:52:13: E741 ambiguous variable name 'l''}
2: {tty:'./services/ganesha.py:36:40: E741 ambiguous variable name 'l'':'./services/ganesha.py:36:40: E741 ambiguous variable name 'l''}
2: 3     E741 ambiguous variable name 'l'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37687 from tchaikov/wip-crimson-errorator

crimson/common: do not take from a future twice

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37602 from tspmelo/wip-flake8-ignore

mgr/dashboard: Remove some Flake8 ignore rules

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/common: do not take from a future twice

before this change, in our specialization of seastar::do_until(),
we access `f` after calling `f.get()`, this is not correct. as `f.get()`
actually moves `f._state` away and detaches the associated promise if any.
so we cannot call `f._then()` anymore after calling `f.get()`. as
`f._then()` schedules `f` by detaching the future from promise and
attaching the scheduled task to the promise. but `future_base::detach_promise()`
does not check `_promise` before accessing it, hence the segfault.

after this change, the order of the checks is rearranged so that
`f.get()` is called at the end. and also use `f.get0()` to be more
explicit, as we are accessing the only element of the returned
value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37684 from ivancich/rgw-yummy-orphan-testing

rgw: use yum rather than dnf for teuthology testing of rgw-orphan-list

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: use yum rather than dnf for testing rgw-orphan-list

The teuthology testing for rgw-orphan-list needs to install
`s3cmd`. Switch from using dnf to yum to work on a wider variety of
platforms.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37660 from adamemerson/wip-datalog-fix

cls/fifo: Switch use CLS_ERR for errors
rgw/fifo: Fix a few missed return value assignments
rgw/fifo: Add some error logging
rgw/fifo: Catch two instances journaling a new part
rgw/fifo: Use unique_ptr and explicit release for callbacks

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37575 from bk201/wip-47742

qa/suites/rados/cephadm/dashboard: fix creating OSD failure

Merge pull request #37669 from lixiaoy1/fix_cache_state_ut_1015

rbd/cache: fix UTs related to image cache state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rbd/cache: fix UTs related to image cache state

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37633 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: add more type hintings and cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37624 from lxbsz/cram

qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37621 from dsavineau/cephadm_ntpsec

cephadm: allow ntpsec service

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result

qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Revert "spec: Podman (temporarily) requires apparmor-abstractions on suse"

This reverts commit 070e5c3e35ea476815ff9fa4f71aed147fe1ea79.

This commit was working around a bug that was present in old (pre-1.8.0)
versions of podman. Since we are no longer running cephadm with versions
that old, we can safely drop this workaround.

Fixes: https://tracker.ceph.com/issues/47862
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "rpm: on SUSE, podman is required for cephadm to work"

This reverts commit 009ade0c3ea624ecf47c8a3f9d1673b342e967cc.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

crypto/isa-l: remove redundant declare shared library

isal_crypto_plugin_objs is same as ceph_crypto_isal

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>