mgr/rook: implement `orch device zap` in rook orchestrator

This commit implements orch device zap by creating a pod on the target
host that mounts the /dev directory and runs either overwrites the first
few blocks of the device with zeros if it's a raw device or if it's not
a raw device it will use `ceph-volume lvm zap`.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #43538 from rzarzynski/wip-crimson-net-actual-bind-failure

crimson/osd: print the actual reason on bind failure.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42557 from ceph/feature-50336-cluster-creation-wizard

mgr/dashboard: Cluster Creation/Expansion Wizard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #43453 from rhcs-dashboard/e2e-host-fix-teuth

mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: print the actual reason on bind failure.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43308 from TRYTOBE8TME/wip-rgw-post-object-upload

src/rgw: HTTP POST object upload

Merge pull request #43526 from adk3798/disable-agent-default

mgr/cephadm: disable agent by default

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #43049 into master

* refs/pull/43049/head:
mgr/rook: apply mds using placement spec and osd_pool_default_size
mgr/rook: factor out replica/failureDomain calc

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

mgr/cephadm: disable agent by default

to avoid issues while it is stabilized

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #43490 from Daniel-Pivonka/cephadm-doc-adopt-sshuser

doc/cephadm: document setting ssh-user during adopt

Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm: document setting ssh-user during adopt

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #43377 from rhcs-dashboard/fix-clients-connection-query

 mgr/dashboard: replace "Ceph-cluster" Client connections with active-standby MGRs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Cluster expansion e2e cleanups

Fixes: https://tracker.ceph.com/issues/52645
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: introduce gather facts in host list

Fixes: https://tracker.ceph.com/issues/52017
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Cluster Creation Add Services Section

Add Services section in cluster creation wizard
Create Cluster OSD Section Followups

1. The device preview disappearing when going to next step and coming back to the previous step
2. Even when clearing the device preview, the Storage Capacity count and the drive group spec doesn't get cleared.
3. Expanding the cluster without selecting any devices gives a 400
   error.
4. Renamed "Delete Host" to "Remove Host"
5. Generalizing most of the sub component code

Fixes: https://tracker.ceph.com/issues/52499
Fixes: https://tracker.ceph.com/issues/51991
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Cluster Creation Create OSDs Section

Create OSDs section in cluster creation wizard

Fixes: https://tracker.ceph.com/issues/51991
Fixes: https://tracker.ceph.com/issues/52298
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Review Section for the Create Cluster Workflow

Fixes: https://tracker.ceph.com/issues/50566
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Cluster Creation Add Host Section and e2es

Add host section of the cluster creation workflow.

1. Fix bug in the modal where going forward one step on the wizard and coming back opens up the add host modal.
2. Rename Create Cluster to Expand Cluster as per the discussions
3. A skip confirmation modal to warn the user when he tries to skip the
   cluster creation
4. Adapted all the tests
5. Did some UI improvements like fixing and aligning the styles,
   colors..
- Used routed modal for host Additon form
- Renamed the Create to Add in Host Form

Fixes: https://tracker.ceph.com/issues/51517
Fixes: https://tracker.ceph.com/issues/51640
Fixes: https://tracker.ceph.com/issues/50336
Fixes: https://tracker.ceph.com/issues/50565
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Create Cluster Workflow welcome screen and e2e tests

A module option called CLUSTER_STATUS has two option. INSTALLED
AND POST_INSTALLED. When CLUSTER_STATUS is INSTALLED it will allow to show the
create-cluster-wizard after login the initial time.  After the cluster
creation is succesfull this option is set to POST_INSTALLED
Also has the e2e codes for the Review Section

Fixes: https://tracker.ceph.com/issues/50336
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>

src/test: HTTP POST object upload

This solves the tracker: https://tracker.ceph.com/issues/51724
Basically it is using 'generate_presigned_post()' boto3 API.
This is verified under AMQP endpoint.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #43500 from rzarzynski/wip-crimson-interruptible-condalias

crimson: osd operations respect interruptor's InterruptCondition.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43501 from rzarzynski/wip-crimson-parse-env

crimson: respect the CEPH_ARGS environment variable.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43274 from pcuzner/add-mib

monitoring:Adding the Ceph MIB

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43509 from ceph/revert-43462-wip-test-erasure-code-for-aarch64

Revert "qa: support isal ec test for aarch64"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "qa: support isal ec test for aarch64"

This commit has been causing scheduled jobs to request e.g. aarch64
smithi machines, which don't exist. The dispatcher then tries to find them forever, requiring the dispatcher to be killed and restarted. The queue
will sit idle until someone notices the problem.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #43492 from xxhdx1985126/wip-52894

crimson/os/seastore: set ExtentPlacementManager::allocated_to before rolling segments

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

The test is failing on deleting a host because the agent daemon is
present in that host. Its not possible to simply delete a host. We need
to drain it first and then delete it.

Fixes: https://tracker.ceph.com/issues/52764
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson: respect the CEPH_ARGS environment variable.

Rook uses `CEPH_ARGS` to convey the `mon_host` configurable. Lack of
support for it was the root cause of inability to reach out to monitors.

See: https://gist.github.com/rzarzynski/95746aa73a48e811749a3b0aaeb31680#gistcomment-3924233.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: rectify a coding style issue in PglogBasedRecovery::do_recovery().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: osd operations respect interruptor's InterruptCondition.

For the sake of DRY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cephadm: expose gather-facts api method

Fixes: https://tracker.ceph.com/issues/51209
This PR intends to expose host metadata(gather-facts) api method

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #42052 from TRYTOBE8TME/wip-rgw-dpp-3

rgw: DPP addition 4

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #43181 from xxhdx1985126/wip-extent-placement-manager-multi-device-3

crimson/os/seastore: add multi-device support

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: set ExtentPlacementManager::allocated_to before rolling segments

There are circumstances in which a transaction that are supposed to roll the current segment
is invalidated after it finished writing and before it rolls the segment. If we don't set
ExtentPlacementManager::allocated_to in this situation, another transaction can try to write
to the old "allocated_to" position, which would cause an invalid write error

Fixes: https://tracker.ceph.com/issues/52894
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43449 from rzarzynski/wip-crimson-interruptible-peeringevent

crimson/osd: implement interruptions in PeeringEvent.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43483 from cyx1231st/wip-seastore-record-header-fullness

crimson/os/seastore: measure inline/ool record header fullness

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/rook: apply mds using placement spec and osd_pool_default_size

This commit changes the apply_mds command in the rook orchestrator
to support some placement specs and also sets the replica size according
to the osd_pool_default_size ceph option.

This commit also adds `orch apply mds` to the QA to test if the command
runs.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: factor out replica/failureDomain calc

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43471 from liewegas/fix-rook-1-node

qa/suites/orch/rook: replicate across osds for 1-node clusters

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/dashboard: replace Client connections with active-stdby mgrs

Fixes: https://tracker.ceph.com/issues/52121
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #43376 from Daniel-Pivonka/cephadm-apply-health

mgr/cephadm: set HEALTH warnings during apply phase in serve

Reviewed-by: Melissa Li <mingkli@redhat.com>

Merge pull request #43455 from liewegas/qa-nvme-loop

qa: use nvme_loop devices for (some) cephadm tests

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43450 from guits/fix_selinux_apply_spec

cephadm: shell --mount shouldnt enforce ':z' option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43458 from mgfritch/cephadm-scp

mgr/cephadm: use `asyncssh.scp` to write remote files

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/rook: specify failureDomain on CephObjectStore CR

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43473 from rhcs-dashboard/wip-pr_deps-master

.github: check if PR deps are merged/closed

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43485 from rhcs-dashboard/fix-pr_triage-master

.github/pr-triage: rename GH token

Reviewed-by: Nizamudeen A <nia@redhat.com>

.github/pr-triage: rename GH token

Repo projects use GITHUB_TOKEN instead of MY_GITHUB_TOKEN:
https://github.com/srggrs/assign-one-project-github-action/blob/master/entrypoint.sh#L19

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

src/rgw: New commit to fix make check

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_kms.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_keystone.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_sts.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #43472 from rhcs-dashboard/wip-gh_action_project_dashboard-master

.github: add dashboard PRs to Dashboard project

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: measure inline/ool record header fullness

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup record_size_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_write_stats_t::overhead_bytes

To header_bytes to be more explicit.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43182 from CongMinYin/fix-writesame-assert

librbd/cache/pwl: initialize number_log_entries

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os/seastore/segment_cleaner: update segment_info_set_t to use segment_map_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: rework SpaceTracker* to use a simpler segment map type

segment_info_set_t does extra things not relevant to these structures.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: max_device -> DEVICE_ID_MAX, clarify users

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore_types.h: clarify segment_id_t visibility, constants

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: check for null in SpaceTrackerSimple constructor

Since we're passing a sparse pointer vector, we need to check for null
in every user.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner.cc: adjust whitespace in init_segments()

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: remove unnecessary template declarations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: adjust segment cleaner's space counter calculation

Until now, segment cleaner calculate available/used spaces assuming that only the journal
does all the writes, which is not true any more. This commit make segment cleaner track
all segment managers' empty/open segments and further calculate the various space usage
based on that information

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: multi-device support

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: enable SegmentCleaner to hold multiple segmented devices

For now, all segmented devices are treated the same as SEGMENTED. In the future,
there may be different kinds of segmeted devices, like SEGMENTED_NVME, SEGMENTED_SSD,
and even SEGMENTED_SATA. We plan to use a dedicated segment cleaner for each kind of
those devices, and if there are multiple devices of the same kind, they share the same
segment cleaner.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: change Scanner to ExtentReader

This commit makes Scanner an extent reader that route read requests to the corresponding
backing devices according to the device ids encapsulated in the segment ids.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: make segment managers device-id aware

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: introduce device id into segment_id_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43166 from rzarzynski/wip-crimson-no-subproc

 crush, crimson: don't support crush_location_hook as crimson lacks SubProcess

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43346 from hjwsm1989/fix-52765

tools/ceph-kvstore-tool: fix segfaults when repair the rocksdb

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43452 from adk3798/agent-perf1

cephadm: agent: subtract average time of previous iterations off wait time

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43401 from romoh/master

src/cephadm: Add support for mariner

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge pull request #43443 from tchaikov/wip-elapsed-in-double

msg/async/ProtocolV2: cast usec to double when converting from nsec

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: dongdong tao <dongdong.tao@canonical.com>

Merge pull request #43343 from trociny/wip-crushdiff-compat

tools/crushdiff: support old format json dump

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43324 from neha-ojha/wip-52640

common/options/global.yaml.in: add runtime flag for mon_max_pg_per_osd

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #43232 from Rethan/feat-bluestore-fail-eio

os/bluestore: add option for bluestore fail eio

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

qa/tasks/nvme_loop: loop until 'nvme list' shows new devs

Sometimes this doesn't happen immediately.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/roch/cephadm/smoke: use nvme_loop

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/cephadm/smoke-roleless: no osd roles, use nvme loop

Signed-off-by: Sage Weil <sage@newdream.net>

qa/overrides/nvme_loop: reusable fragment to use nvme_loop devs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: wait for osds to start explicitly

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: if no osd roles, --all-available-devices

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/nvme_loop: set up nvme_loop on scratch_devs

Using an nvme loop device makes the LVs look like "real" disks,
which means we can exercise all of the normal code paths for
provisioning, deprovisioning, and zapping.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #43163 into master

* refs/pull/43163/head:
qa: fsync dir for asynchronous creat on stray tests
qa: refactor and generalize create_n_files
qa: only set frag confs for workloads
mds: improve debugging for fragment size check

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

.github: check if PR deps are merged/closed

If a PR description includes `blocked by <pr>` or `depends on <pr>`,
this check will fail until the referred PRs are either merged or closed.

If this check is made required, it'll also block the merge button.

The details of the supported syntax can be checked at:
https://github.com/marketplace/actions/pr-dependency-check

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

.github: add dashboard PRs to Dashboard project

This action automatically adds PRs with 'dashboard' label to the
'Dashboard' project (https://github.com/ceph/ceph/projects/6).

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

qa/suites/orch/rook: replicate across osds for 1-node clusters

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43462 from Zhiwei-Dai/wip-test-erasure-code-for-aarch64

qa: support isal ec test for aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43444 from idryomov/wip-read-from-replica-comment

librados,librbd: make it clear that replica reads are safe for general use

Reviewed-by: Samuel Just <sjust@redhat.com>

librbd/cache/pwl: initialize number_log_entries

Using uninitialized number_log_entries cause writesame req space
calculation error. sometimes fail in TestMockCacheSSDWriteLog.writesame.

Fixes: https://tracker.ceph.com/issues/52852
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #43137 from rhcs-dashboard/fix-52480-master

mgr/dashboard: clean-up controllers

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43459 from cyx1231st/wip-seastore-optimize-record-overhead

crimson/os/seastore: reduce write amplification from record overhead and implement placement hint

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa: support isal ec test for aarch64

modified:   qa/standalone/erasure-code/test-erasure-code-plugins.sh
new file:   qa/suites/rados/thrash-erasure-code-isa/arch/aarch64.yaml

Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>

crimson/os/seastore: implement placement_hint HOT/COLD/REWRITE

The current strategy is to inline allocate non-REWRITE extents in order
to reduce the record overhead. The observation shows the transactions
from user is usually very small and doesn't deserve to be written in ool
records, which requires at least a block to store the record metadata.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_placement_hint_t

To placement_hint_t as the hint is not limited to ool extents.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/cephadm: use str type for `stdin`

avoid encode/decode confusion by using a str data type to both
send (stdin) and receive (stdout) data

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use `asyncssh.scp` to write remote files

`tee` via stdin happens to work when the file is a utf-8 byte encoded
string, but won't work if the file happens to be binary data

Signed-off-by: Michael Fritch <mfritch@suse.com>