Merge pull request #43490 from Daniel-Pivonka/cephadm-doc-adopt-sshuser

doc/cephadm: document setting ssh-user during adopt

Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm: document setting ssh-user during adopt

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #43377 from rhcs-dashboard/fix-clients-connection-query

 mgr/dashboard: replace "Ceph-cluster" Client connections with active-standby MGRs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43500 from rzarzynski/wip-crimson-interruptible-condalias

crimson: osd operations respect interruptor's InterruptCondition.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43501 from rzarzynski/wip-crimson-parse-env

crimson: respect the CEPH_ARGS environment variable.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43274 from pcuzner/add-mib

monitoring:Adding the Ceph MIB

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43509 from ceph/revert-43462-wip-test-erasure-code-for-aarch64

Revert "qa: support isal ec test for aarch64"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "qa: support isal ec test for aarch64"

This commit has been causing scheduled jobs to request e.g. aarch64
smithi machines, which don't exist. The dispatcher then tries to find them forever, requiring the dispatcher to be killed and restarted. The queue
will sit idle until someone notices the problem.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #43492 from xxhdx1985126/wip-52894

crimson/os/seastore: set ExtentPlacementManager::allocated_to before rolling segments

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: respect the CEPH_ARGS environment variable.

Rook uses `CEPH_ARGS` to convey the `mon_host` configurable. Lack of
support for it was the root cause of inability to reach out to monitors.

See: https://gist.github.com/rzarzynski/95746aa73a48e811749a3b0aaeb31680#gistcomment-3924233.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: rectify a coding style issue in PglogBasedRecovery::do_recovery().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: osd operations respect interruptor's InterruptCondition.

For the sake of DRY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42052 from TRYTOBE8TME/wip-rgw-dpp-3

rgw: DPP addition 4

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #43181 from xxhdx1985126/wip-extent-placement-manager-multi-device-3

crimson/os/seastore: add multi-device support

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: set ExtentPlacementManager::allocated_to before rolling segments

There are circumstances in which a transaction that are supposed to roll the current segment
is invalidated after it finished writing and before it rolls the segment. If we don't set
ExtentPlacementManager::allocated_to in this situation, another transaction can try to write
to the old "allocated_to" position, which would cause an invalid write error

Fixes: https://tracker.ceph.com/issues/52894
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43449 from rzarzynski/wip-crimson-interruptible-peeringevent

crimson/osd: implement interruptions in PeeringEvent.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43483 from cyx1231st/wip-seastore-record-header-fullness

crimson/os/seastore: measure inline/ool record header fullness

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43471 from liewegas/fix-rook-1-node

qa/suites/orch/rook: replicate across osds for 1-node clusters

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/dashboard: replace Client connections with active-stdby mgrs

Fixes: https://tracker.ceph.com/issues/52121
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #43376 from Daniel-Pivonka/cephadm-apply-health

mgr/cephadm: set HEALTH warnings during apply phase in serve

Reviewed-by: Melissa Li <mingkli@redhat.com>

Merge pull request #43455 from liewegas/qa-nvme-loop

qa: use nvme_loop devices for (some) cephadm tests

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43450 from guits/fix_selinux_apply_spec

cephadm: shell --mount shouldnt enforce ':z' option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43458 from mgfritch/cephadm-scp

mgr/cephadm: use `asyncssh.scp` to write remote files

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/rook: specify failureDomain on CephObjectStore CR

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43473 from rhcs-dashboard/wip-pr_deps-master

.github: check if PR deps are merged/closed

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43485 from rhcs-dashboard/fix-pr_triage-master

.github/pr-triage: rename GH token

Reviewed-by: Nizamudeen A <nia@redhat.com>

.github/pr-triage: rename GH token

Repo projects use GITHUB_TOKEN instead of MY_GITHUB_TOKEN:
https://github.com/srggrs/assign-one-project-github-action/blob/master/entrypoint.sh#L19

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

src/rgw: New commit to fix make check

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_kms.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_keystone.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_sts.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #43472 from rhcs-dashboard/wip-gh_action_project_dashboard-master

.github: add dashboard PRs to Dashboard project

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: measure inline/ool record header fullness

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup record_size_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_write_stats_t::overhead_bytes

To header_bytes to be more explicit.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43182 from CongMinYin/fix-writesame-assert

librbd/cache/pwl: initialize number_log_entries

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os/seastore/segment_cleaner: update segment_info_set_t to use segment_map_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: rework SpaceTracker* to use a simpler segment map type

segment_info_set_t does extra things not relevant to these structures.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: max_device -> DEVICE_ID_MAX, clarify users

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore_types.h: clarify segment_id_t visibility, constants

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: check for null in SpaceTrackerSimple constructor

Since we're passing a sparse pointer vector, we need to check for null
in every user.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner.cc: adjust whitespace in init_segments()

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: remove unnecessary template declarations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: adjust segment cleaner's space counter calculation

Until now, segment cleaner calculate available/used spaces assuming that only the journal
does all the writes, which is not true any more. This commit make segment cleaner track
all segment managers' empty/open segments and further calculate the various space usage
based on that information

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: multi-device support

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: enable SegmentCleaner to hold multiple segmented devices

For now, all segmented devices are treated the same as SEGMENTED. In the future,
there may be different kinds of segmeted devices, like SEGMENTED_NVME, SEGMENTED_SSD,
and even SEGMENTED_SATA. We plan to use a dedicated segment cleaner for each kind of
those devices, and if there are multiple devices of the same kind, they share the same
segment cleaner.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: change Scanner to ExtentReader

This commit makes Scanner an extent reader that route read requests to the corresponding
backing devices according to the device ids encapsulated in the segment ids.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: make segment managers device-id aware

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: introduce device id into segment_id_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43166 from rzarzynski/wip-crimson-no-subproc

 crush, crimson: don't support crush_location_hook as crimson lacks SubProcess

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43346 from hjwsm1989/fix-52765

tools/ceph-kvstore-tool: fix segfaults when repair the rocksdb

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43452 from adk3798/agent-perf1

cephadm: agent: subtract average time of previous iterations off wait time

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43401 from romoh/master

src/cephadm: Add support for mariner

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge pull request #43443 from tchaikov/wip-elapsed-in-double

msg/async/ProtocolV2: cast usec to double when converting from nsec

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: dongdong tao <dongdong.tao@canonical.com>

Merge pull request #43343 from trociny/wip-crushdiff-compat

tools/crushdiff: support old format json dump

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43324 from neha-ojha/wip-52640

common/options/global.yaml.in: add runtime flag for mon_max_pg_per_osd

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #43232 from Rethan/feat-bluestore-fail-eio

os/bluestore: add option for bluestore fail eio

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

qa/tasks/nvme_loop: loop until 'nvme list' shows new devs

Sometimes this doesn't happen immediately.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/roch/cephadm/smoke: use nvme_loop

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/smoke-roleless: no osd roles, use nvme loop

Signed-off-by: Sage Weil <sage@newdream.net>

qa/overrides/nvme_loop: reusable fragment to use nvme_loop devs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: wait for osds to start explicitly

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: if no osd roles, --all-available-devices

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/nvme_loop: set up nvme_loop on scratch_devs

Using an nvme loop device makes the LVs look like "real" disks,
which means we can exercise all of the normal code paths for
provisioning, deprovisioning, and zapping.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #43163 into master

* refs/pull/43163/head:
qa: fsync dir for asynchronous creat on stray tests
qa: refactor and generalize create_n_files
qa: only set frag confs for workloads
mds: improve debugging for fragment size check

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

.github: check if PR deps are merged/closed

If a PR description includes `blocked by <pr>` or `depends on <pr>`,
this check will fail until the referred PRs are either merged or closed.

If this check is made required, it'll also block the merge button.

The details of the supported syntax can be checked at:
https://github.com/marketplace/actions/pr-dependency-check

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

.github: add dashboard PRs to Dashboard project

This action automatically adds PRs with 'dashboard' label to the
'Dashboard' project (https://github.com/ceph/ceph/projects/6).

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa/suites/orch/rook: replicate across osds for 1-node clusters

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43462 from Zhiwei-Dai/wip-test-erasure-code-for-aarch64

qa: support isal ec test for aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43444 from idryomov/wip-read-from-replica-comment

librados,librbd: make it clear that replica reads are safe for general use

Reviewed-by: Samuel Just <sjust@redhat.com>

librbd/cache/pwl: initialize number_log_entries

Using uninitialized number_log_entries cause writesame req space
calculation error. sometimes fail in TestMockCacheSSDWriteLog.writesame.

Fixes: https://tracker.ceph.com/issues/52852
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #43137 from rhcs-dashboard/fix-52480-master

mgr/dashboard: clean-up controllers

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43459 from cyx1231st/wip-seastore-optimize-record-overhead

crimson/os/seastore: reduce write amplification from record overhead and implement placement hint

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa: support isal ec test for aarch64

modified:   qa/standalone/erasure-code/test-erasure-code-plugins.sh
new file:   qa/suites/rados/thrash-erasure-code-isa/arch/aarch64.yaml

Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>

crimson/os/seastore: implement placement_hint HOT/COLD/REWRITE

The current strategy is to inline allocate non-REWRITE extents in order
to reduce the record overhead. The observation shows the transactions
from user is usually very small and doesn't deserve to be written in ool
records, which requires at least a block to store the record metadata.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_placement_hint_t

To placement_hint_t as the hint is not limited to ool extents.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/cephadm: use str type for `stdin`

avoid encode/decode confusion by using a str data type to both
send (stdin) and receive (stdout) data

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use `asyncssh.scp` to write remote files

`tee` via stdin happens to work when the file is a utf-8 byte encoded
string, but won't work if the file happens to be binary data

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43248 from inspur-wyq/wip-doc

doc/rbd/vault.rst: fix typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43373 from gregsfortytwo/wip-mon-stretch-new-tiebreaker-52126

mon: Allow specifying new tiebreaker monitors

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43331 from adk3798/agent-drain

mgr/cephadm: fix host drain with agent

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/cephadm: set health check warning for apply spec failures and daemon place failures in serve

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: set health check warning in `_apply_service_config` for invalid config options and failures to set options

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_check_for_strays` to use `remove_health_warning` helper

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_refresh_hosts_and_daemons` to use `set_health_warning` and `remove_health_warning` helpers

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_update_paused_health` to use `set_health_warning` and `remove_health_warning` helpers

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_set_maintenance_health_check` to use `set_health_warning` helper

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

qa: fsync dir for asynchronous creat on stray tests

Use the enhanced create_n_files to dedup code. Also split the large test
into three.

Fixes: https://tracker.ceph.com/issues/52606
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: refactor and generalize create_n_files

Few things:

- Allow calling fsync on directory (to support async create kernel).
- Allow immediately unlinking the created file (for stray testing).
- Close any file descriptors created.
- Write unique content (the i variable) to each file.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: only set frag confs for workloads

Otherwise these local conf overrides prevent functional testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: improve debugging for fragment size check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43442 from linuxbox2/wip-rgwadmin-logfix

qa/rgw: fix ops log tests to handle non-bucket ops (which are now valid)

cephadm: bootstrap --apply-spec shouldn't enforce :z

There's no need to mount this file with the `:z` option.
read-only access is enough for this file.

Fixes: https://tracker.ceph.com/issues/52855
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43448 from rzarzynski/wip-crimson-monmap-moredbgs

crimson, monmap: inform about errors when interacting with DNS

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cephadm: agent: subtract average time of previous iterations off wait time

We want the agent to actually report metadata at the rate we set
it for. Before this, that rate was just being used as the wait time
between iterations so the actual time between iterations was the
given interval plus the time to gather metadata. Now the time between
reports should actually be roughly the given interval.

Signed-off-by: Adam King <adking@redhat.com>

msg/async/ProtocolV2: cast usec to double when converting from nsec

to be consistent with the same logic in V1, and more importantly, it is
more correct.

the change was suggested by Ilya Dryomov <idryomov@redhat.com>.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #43231 into master

* refs/pull/43231/head:
qa: fix promotion test

Reviewed-by: Ramana Raja <rraja@redhat.com>

cephadm: shell --mount shouldnt enforce ':z' option

cephadm shouldn't enforce this option.
For instance, it can be an issue when you try to bindmount a file in /usr

Fixes: https://tracker.ceph.com/issues/52853
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

crimson/osd: more interruptions in PeeringEvent.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>