Merge pull request #43471 from liewegas/fix-rook-1-node

qa/suites/orch/rook: replicate across osds for 1-node clusters

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43376 from Daniel-Pivonka/cephadm-apply-health

mgr/cephadm: set HEALTH warnings during apply phase in serve

Reviewed-by: Melissa Li <mingkli@redhat.com>

Merge pull request #43455 from liewegas/qa-nvme-loop

qa: use nvme_loop devices for (some) cephadm tests

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43450 from guits/fix_selinux_apply_spec

cephadm: shell --mount shouldnt enforce ':z' option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43458 from mgfritch/cephadm-scp

mgr/cephadm: use `asyncssh.scp` to write remote files

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/rook: specify failureDomain on CephObjectStore CR

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43473 from rhcs-dashboard/wip-pr_deps-master

.github: check if PR deps are merged/closed

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43485 from rhcs-dashboard/fix-pr_triage-master

.github/pr-triage: rename GH token

Reviewed-by: Nizamudeen A <nia@redhat.com>

.github/pr-triage: rename GH token

Repo projects use GITHUB_TOKEN instead of MY_GITHUB_TOKEN:
https://github.com/srggrs/assign-one-project-github-action/blob/master/entrypoint.sh#L19

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #43472 from rhcs-dashboard/wip-gh_action_project_dashboard-master

.github: add dashboard PRs to Dashboard project

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43182 from CongMinYin/fix-writesame-assert

librbd/cache/pwl: initialize number_log_entries

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43166 from rzarzynski/wip-crimson-no-subproc

 crush, crimson: don't support crush_location_hook as crimson lacks SubProcess

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43346 from hjwsm1989/fix-52765

tools/ceph-kvstore-tool: fix segfaults when repair the rocksdb

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43452 from adk3798/agent-perf1

cephadm: agent: subtract average time of previous iterations off wait time

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43401 from romoh/master

src/cephadm: Add support for mariner

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge pull request #43443 from tchaikov/wip-elapsed-in-double

msg/async/ProtocolV2: cast usec to double when converting from nsec

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: dongdong tao <dongdong.tao@canonical.com>

Merge pull request #43343 from trociny/wip-crushdiff-compat

tools/crushdiff: support old format json dump

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43324 from neha-ojha/wip-52640

common/options/global.yaml.in: add runtime flag for mon_max_pg_per_osd

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #43232 from Rethan/feat-bluestore-fail-eio

os/bluestore: add option for bluestore fail eio

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

qa/tasks/nvme_loop: loop until 'nvme list' shows new devs

Sometimes this doesn't happen immediately.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/roch/cephadm/smoke: use nvme_loop

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/smoke-roleless: no osd roles, use nvme loop

Signed-off-by: Sage Weil <sage@newdream.net>

qa/overrides/nvme_loop: reusable fragment to use nvme_loop devs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: wait for osds to start explicitly

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: if no osd roles, --all-available-devices

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/nvme_loop: set up nvme_loop on scratch_devs

Using an nvme loop device makes the LVs look like "real" disks,
which means we can exercise all of the normal code paths for
provisioning, deprovisioning, and zapping.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #43163 into master

* refs/pull/43163/head:
qa: fsync dir for asynchronous creat on stray tests
qa: refactor and generalize create_n_files
qa: only set frag confs for workloads
mds: improve debugging for fragment size check

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

.github: check if PR deps are merged/closed

If a PR description includes `blocked by <pr>` or `depends on <pr>`,
this check will fail until the referred PRs are either merged or closed.

If this check is made required, it'll also block the merge button.

The details of the supported syntax can be checked at:
https://github.com/marketplace/actions/pr-dependency-check

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

.github: add dashboard PRs to Dashboard project

This action automatically adds PRs with 'dashboard' label to the
'Dashboard' project (https://github.com/ceph/ceph/projects/6).

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa/suites/orch/rook: replicate across osds for 1-node clusters

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43462 from Zhiwei-Dai/wip-test-erasure-code-for-aarch64

qa: support isal ec test for aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43444 from idryomov/wip-read-from-replica-comment

librados,librbd: make it clear that replica reads are safe for general use

Reviewed-by: Samuel Just <sjust@redhat.com>

librbd/cache/pwl: initialize number_log_entries

Using uninitialized number_log_entries cause writesame req space
calculation error. sometimes fail in TestMockCacheSSDWriteLog.writesame.

Fixes: https://tracker.ceph.com/issues/52852
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #43137 from rhcs-dashboard/fix-52480-master

mgr/dashboard: clean-up controllers

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43459 from cyx1231st/wip-seastore-optimize-record-overhead

crimson/os/seastore: reduce write amplification from record overhead and implement placement hint

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa: support isal ec test for aarch64

modified:   qa/standalone/erasure-code/test-erasure-code-plugins.sh
new file:   qa/suites/rados/thrash-erasure-code-isa/arch/aarch64.yaml

Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>

crimson/os/seastore: implement placement_hint HOT/COLD/REWRITE

The current strategy is to inline allocate non-REWRITE extents in order
to reduce the record overhead. The observation shows the transactions
from user is usually very small and doesn't deserve to be written in ool
records, which requires at least a block to store the record metadata.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_placement_hint_t

To placement_hint_t as the hint is not limited to ool extents.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/cephadm: use str type for `stdin`

avoid encode/decode confusion by using a str data type to both
send (stdin) and receive (stdout) data

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use `asyncssh.scp` to write remote files

`tee` via stdin happens to work when the file is a utf-8 byte encoded
string, but won't work if the file happens to be binary data

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43248 from inspur-wyq/wip-doc

doc/rbd/vault.rst: fix typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43373 from gregsfortytwo/wip-mon-stretch-new-tiebreaker-52126

mon: Allow specifying new tiebreaker monitors

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43331 from adk3798/agent-drain

mgr/cephadm: fix host drain with agent

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/cephadm: set health check warning for apply spec failures and daemon place failures in serve

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: set health check warning in `_apply_service_config` for invalid config options and failures to set options

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_check_for_strays` to use `remove_health_warning` helper

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_refresh_hosts_and_daemons` to use `set_health_warning` and `remove_health_warning` helpers

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_update_paused_health` to use `set_health_warning` and `remove_health_warning` helpers

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

mgr/cephadm: refactor `_set_maintenance_health_check` to use `set_health_warning` helper

Fixes: https://tracker.ceph.com/issues/44414
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

qa: fsync dir for asynchronous creat on stray tests

Use the enhanced create_n_files to dedup code. Also split the large test
into three.

Fixes: https://tracker.ceph.com/issues/52606
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: refactor and generalize create_n_files

Few things:

- Allow calling fsync on directory (to support async create kernel).
- Allow immediately unlinking the created file (for stray testing).
- Close any file descriptors created.
- Write unique content (the i variable) to each file.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: only set frag confs for workloads

Otherwise these local conf overrides prevent functional testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: improve debugging for fragment size check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43442 from linuxbox2/wip-rgwadmin-logfix

qa/rgw: fix ops log tests to handle non-bucket ops (which are now valid)

cephadm: bootstrap --apply-spec shouldn't enforce :z

There's no need to mount this file with the `:z` option.
read-only access is enough for this file.

Fixes: https://tracker.ceph.com/issues/52855
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43448 from rzarzynski/wip-crimson-monmap-moredbgs

crimson, monmap: inform about errors when interacting with DNS

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cephadm: agent: subtract average time of previous iterations off wait time

We want the agent to actually report metadata at the rate we set
it for. Before this, that rate was just being used as the wait time
between iterations so the actual time between iterations was the
given interval plus the time to gather metadata. Now the time between
reports should actually be roughly the given interval.

Signed-off-by: Adam King <adking@redhat.com>

msg/async/ProtocolV2: cast usec to double when converting from nsec

to be consistent with the same logic in V1, and more importantly, it is
more correct.

the change was suggested by Ilya Dryomov <idryomov@redhat.com>.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #43231 into master

* refs/pull/43231/head:
qa: fix promotion test

Reviewed-by: Ramana Raja <rraja@redhat.com>

cephadm: shell --mount shouldnt enforce ':z' option

cephadm shouldn't enforce this option.
For instance, it can be an issue when you try to bindmount a file in /usr

Fixes: https://tracker.ceph.com/issues/52853
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

crimson, monmap: inform about errors in MonMap::init_with_dns_srv().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, monmap: limit the scope of the seastar namespace inclusion.

More consistency that way. This also enables us to define the usual
`logger()` function without having a clash with `seastar::logger`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43384 from bosc0/master

mgr: Add check to prevent mgr from crashing

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41445 from guits/cephadm_quick_patch

cephadm: add daemon_name in daemon description

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43335 from liewegas/debug-51815

mon,auth: fix proposal (and mon db rebuild) of rotating secrets

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43379 from myoungwon/wip-evict-range

osd: trim corresponding chunks on tier_evict

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43378 from ivancich/wip-build-without-mgr

rados: build minimally when "WITH_MGR" is off

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43360 from jianwei1216/fix-shard-threads-cannot-wakeup-bug

osd: fix shard-threads cannot wakeup bug

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43307 from taodd/recv_stamp_fix

msg/async/ProtocolV2: Set the recv_stamp at the beginning of receiving a message

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Dan Hill <daniel.hill@canonical.com>

Merge pull request #43380 from myoungwon/wip-dont-promote-dirty

osd: do not promote objects if op is is_dirty

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43435 from rzarzynski/wip-crimson-pickv2only

crimson/osd: pick only v2 addresses.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

librbd: clarify rbd_read_from_replica_policy vs older options

If rbd_read_from_replica_policy is set appropriately, setting
rbd_balance/localize_snap_reads or rbd_balance/localize_parent_reads
is redundant.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librados: OPERATION_BALANCE/LOCALIZE_READS comment is outdated

With the introduction of min_last_complete_ondisk on the OSD side,
it is safe to use these for non-immutable data.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/rgw: fix ops log tests to handle non-bucket ops (which are now valid)

After 3863eb89512f1698b8e56f1f1ffc78a6ca8d5826--rgw: permit logging of
list-bucket (and any other no-bucket op-- the radosgw ops-log
contains entries for ops with no associated buckets--e.g., list_buckets.
When examining such a log object in the radosgw_admin task, don't assert
that it has any bucket name.

Fixes: https://tracker.ceph.com/issues/52647
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #43256 from rhcs-dashboard/fix-48845-master

qa/mgr/dashboard/test_pool: don't check HEALTH_OK

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: clean-up controllers

Fixes: https://tracker.ceph.com/issues/52589
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42964 from trociny/wip-52448

osd: re-cache peer_bytes on every peering state activate

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/osd: pick only v2 addresses.

Otherwise we'll run into an assertion failure when binding:

```
INFO  2021-10-05 15:14:51,075 [shard 0] osd - picked address v2:0.0.0.0:0/0
INFO  2021-10-05 15:14:51,075 [shard 0] osd - picked address v1:0.0.0.0:0/0
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-8025-geb9c0f59/rpm/el8/BUILD/ceph-17.0.0-8025-geb9c0f59/src/crimson/net/SocketMessenger.cc:89: crimson::errorator<crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<std::errc::address_in_use>))>, crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<std::errc::address_not_available>))> >::future<> crimson::net::SocketMessenger::try_bind(const entity_addrvec_t&, uint32_t, uint32_t): Assertion `addrs.size() == 1' failed.
Aborting on shard 0.
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #43399 into master

* refs/pull/43399/head:
mgr/volumes/fs/operations: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43398 into master

* refs/pull/43398/head:
mgr/volumes/fs: fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43397 into master

* refs/pull/43397/head:
mgr/volumes/fs: fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43395 into master

* refs/pull/43395/head:
mgr/volumes/fs/operations/versions: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43392 into master

* refs/pull/43392/head:
mgr/volumes/fs/operations: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43391 into master

* refs/pull/43391/head:
mgr/volumes/fs/vol_spec.py: fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43390 into master

* refs/pull/43390/head:
mgr/volumes/fs/operations : Fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43387 into master

* refs/pull/43387/head:
mgr/volumes: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43375 into master

* refs/pull/43375/head:
mgr/volumes/fs: add extra blank line

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43357 into master

* refs/pull/43357/head:
mgr/nfs: stick to lazy evaluation of logger messages
mgr/nfs: change _cmd_rgw_export_create_rgw() name
mgr/nfs: don't log fsal keys
mgr/nfs: Add more debug log messages

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

qa: fix promotion test

The test is not needing to check that the new MDS becomes active, only
that a replacement occurs.

Fixes: https://tracker.ceph.com/issues/52677
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42977 from amathuria/wip-amathuri-scrub-duration

osd/scrub: Add scrub duration to pg dump stats

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

osd: wake up more shard workers when new osdmap is consumed

Reproduce:
(1) ceph cluster not running any client IO
(2) only ceph osd in osd.14 operation

Reason:
(1) one shard-queue has three shard-threads
(2) one or some PeeringOp's epoch > osdmap's epoch held by current osd,
    and these PeeringOp _add_slot_waiter()
(3) shard-queue become empty and three shard-threads cond.wait()
(4) new osdmap consume and it _wake_pg_slot()
    Problem in here
1> OSDShard::consume() exec loop all pg's slot wait
     and requeue more than one PeeringOp to shard-queue
        2> but it only notify one shard-thread to wakeup,
           the other two shard-threads continue cond.wait()
3> OSD::ShardedOpWQ::_enqueue() found the shard-queue not empty
   and not notify all shard-thread to wakeup

     In a period of time, only one shard-thread of 3 shard-threads is running.

Fixes: https://tracker.ceph.com/issues/52781
Signed-off-by: Jianwei Zhang <jianwei1216@qq.com>
Change-Id: I4617db2fd95082007e6d9fa2b60f17f2a6296b5b

Merge pull request #43175 from ideepika/wip-supress-jaeger-cmake-logs

cmake,rgw: use jaeger-base target, as a dependency for building dbstore

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: update stretch mode documentation for new commands

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: dump tiebreaker_mon in MonMap

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: udpate MonMap::last_changed on stretch commands which didn't

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: remove deleted monitors from the disallowed_leaders set

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: don't let users remove the tiebreaker monitor from the map

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: add "mon set_new_tiebreaker" command, so stretch clusters can replace it

Fixes: https://tracker.ceph.com/issues/52126
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/cephadm: Add mariner as a supported distro.

Using cephadm install commands on Mariner will result in an error
"Distro not supported" though the packages are supported and can be
installed manually. This patch adds mariner as a supported distro.
Fix validated on mariner.

Signed-off-by: Roaa Sakr <romoh@microsoft.com>