qa/suites/orch/rook/smoke: use nvme-loop

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42997 from josephsawaya/wip-mgr-rook-orch-ls

mgr/rook: orch ls displays info about OSDs

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #43332 from sebastian-philipp/sphinx-rgw-spec

doc/cephadm: use sphinx autoclass to document RGWSpec

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43239 from trociny/wip-48959

osd: handle inconsistent hash info during backfill and deep scrub gracefully

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43498 from dheart-joe/bluestore_fix

os/bluestore: _do_write_small fix head_pad

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42099 from aclamk/wip-bluefs-fine-grain-locking-2

os/bluestore: BlueFS fine grain locking

Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #43374 from ljflores/wip-remove-pool-name

mgr/telemetry: remove pool name from telemetry perf report

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge pull request #43463 from Zhiwei-Dai/wip-enhance-qa-python3-compatibility

qa/tasks: replace iterkeys() with keys() for Python 3

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43405 from ljflores/wip-perfcounter-priorities

os/bluestore: update perf counter priorities

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43422 from ronen-fr/wip-rf-exact-time-sec

common/ceph_time: fix wrong seconds output in exact_timespan_str()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43515 from fengchunsong/dpdk-dev

msgs/async/dpdk: fix dpdk build error

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43531 from tchaikov/wip-pick-addr

common/pick_address: refactor pick_addresses()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43073 from strenuous-life/wip-cephadm-osd-rm-status

mgr/cephadm: the display of 'orch osd rm status' is incorrect

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43214 from batrick/i52654

pybind/mgr/cephadm: set allow_standby_replay during CephFS upgrade

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43253 from guits/fix_orch_ls_osd

cephadm: fix osd size reported in 'orch ls'

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43339 from adk3798/volume-chunk

mgr/cephadm: use json-pretty for ceph-volume calls

Reviewed-by: Sage Weil <sage@newdream.net>

Merge PR #43517 into master

* refs/pull/43517/head:
doc / cephfs: health message codes should be permalinks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43538 from rzarzynski/wip-crimson-net-actual-bind-failure

crimson/osd: print the actual reason on bind failure.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42557 from ceph/feature-50336-cluster-creation-wizard

mgr/dashboard: Cluster Creation/Expansion Wizard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #43453 from rhcs-dashboard/e2e-host-fix-teuth

mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

src/msg/dpdk: reserve funcs capacity to avoid reallocation

When a new vector is added larger than then current vector capacity,
it reallocates space. lamda function accesses the previous adress will
cause a segment fault. Therefore, reserve sufficient funcs space to
avoid reallocation.

Fixes: https://tracker.ceph.com/issues/52907
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

msgs/async/dpdk: fix dpdk build error

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

crimson/osd: print the actual reason on bind failure.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43308 from TRYTOBE8TME/wip-rgw-post-object-upload

src/rgw: HTTP POST object upload

common/pick_address: refactor pick_addresses()

* consolidate the logic handling CEPH_PICK_ADDRESS_PREFER_IPV4 using
  std::sort(). this might be overkill. but it helps to explain
  what CEPH_PICK_ADDRESS_PREFER_IPV4 is for, and helps to dedup
  the code to order the addresses.
* let fill_in_one_address() return an optional<entity_addrvec_t>.
  more readable this way
* early return if the required address is not found, instead of
  checking variables like ipv4_r
* rename fill_in_one_address() to get_one_address() to reflect
  the change of the function's return value's type

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

doc / cephfs: health message codes should be permalinks

... so that such links can be included in alert warnings.

Additionally, document some other health warnings. Credit to @pcuzner
to point out that not all health warnings have been documented.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

common/pick_address: use string_view when appropriate

we don't need a string when string_view is good enough.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43526 from adk3798/disable-agent-default

mgr/cephadm: disable agent by default

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #43049 into master

* refs/pull/43049/head:
mgr/rook: apply mds using placement spec and osd_pool_default_size
mgr/rook: factor out replica/failureDomain calc

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

mgr/cephadm: disable agent by default

to avoid issues while it is stabilized

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #43490 from Daniel-Pivonka/cephadm-doc-adopt-sshuser

doc/cephadm: document setting ssh-user during adopt

Reviewed-by: Adam King <adking@redhat.com>

doc/cephadm: document setting ssh-user during adopt

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

os/bluestore: update priorities and nicks of bluestore perf counters

These perf counters do not show up in telemetry unless they are set to a "useful" priority or higher. Fetching these counters in telemetry may help to diagnose problems with RocksDB / BlueFS prefetching / insufficient cache sizes.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #43377 from rhcs-dashboard/fix-clients-connection-query

 mgr/dashboard: replace "Ceph-cluster" Client connections with active-standby MGRs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Cluster expansion e2e cleanups

Fixes: https://tracker.ceph.com/issues/52645
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: introduce gather facts in host list

Fixes: https://tracker.ceph.com/issues/52017
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Cluster Creation Add Services Section

Add Services section in cluster creation wizard
Create Cluster OSD Section Followups

1. The device preview disappearing when going to next step and coming back to the previous step
2. Even when clearing the device preview, the Storage Capacity count and the drive group spec doesn't get cleared.
3. Expanding the cluster without selecting any devices gives a 400
   error.
4. Renamed "Delete Host" to "Remove Host"
5. Generalizing most of the sub component code

Fixes: https://tracker.ceph.com/issues/52499
Fixes: https://tracker.ceph.com/issues/51991
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Cluster Creation Create OSDs Section

Create OSDs section in cluster creation wizard

Fixes: https://tracker.ceph.com/issues/51991
Fixes: https://tracker.ceph.com/issues/52298
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Review Section for the Create Cluster Workflow

Fixes: https://tracker.ceph.com/issues/50566
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/dashboard: Cluster Creation Add Host Section and e2es

Add host section of the cluster creation workflow.

1. Fix bug in the modal where going forward one step on the wizard and coming back opens up the add host modal.
2. Rename Create Cluster to Expand Cluster as per the discussions
3. A skip confirmation modal to warn the user when he tries to skip the
   cluster creation
4. Adapted all the tests
5. Did some UI improvements like fixing and aligning the styles,
   colors..
- Used routed modal for host Additon form
- Renamed the Create to Add in Host Form

Fixes: https://tracker.ceph.com/issues/51517
Fixes: https://tracker.ceph.com/issues/51640
Fixes: https://tracker.ceph.com/issues/50336
Fixes: https://tracker.ceph.com/issues/50565
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Create Cluster Workflow welcome screen and e2e tests

A module option called CLUSTER_STATUS has two option. INSTALLED
AND POST_INSTALLED. When CLUSTER_STATUS is INSTALLED it will allow to show the
create-cluster-wizard after login the initial time.  After the cluster
creation is succesfull this option is set to POST_INSTALLED
Also has the e2e codes for the Review Section

Fixes: https://tracker.ceph.com/issues/50336
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>

src/test: HTTP POST object upload

This solves the tracker: https://tracker.ceph.com/issues/51724
Basically it is using 'generate_presigned_post()' boto3 API.
This is verified under AMQP endpoint.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #43500 from rzarzynski/wip-crimson-interruptible-condalias

crimson: osd operations respect interruptor's InterruptCondition.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43501 from rzarzynski/wip-crimson-parse-env

crimson: respect the CEPH_ARGS environment variable.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43274 from pcuzner/add-mib

monitoring:Adding the Ceph MIB

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43509 from ceph/revert-43462-wip-test-erasure-code-for-aarch64

Revert "qa: support isal ec test for aarch64"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Revert "qa: support isal ec test for aarch64"

This commit has been causing scheduled jobs to request e.g. aarch64
smithi machines, which don't exist. The dispatcher then tries to find them forever, requiring the dispatcher to be killed and restarted. The queue
will sit idle until someone notices the problem.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #43492 from xxhdx1985126/wip-52894

crimson/os/seastore: set ExtentPlacementManager::allocated_to before rolling segments

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

The test is failing on deleting a host because the agent daemon is
present in that host. Its not possible to simply delete a host. We need
to drain it first and then delete it.

Fixes: https://tracker.ceph.com/issues/52764
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson: respect the CEPH_ARGS environment variable.

Rook uses `CEPH_ARGS` to convey the `mon_host` configurable. Lack of
support for it was the root cause of inability to reach out to monitors.

See: https://gist.github.com/rzarzynski/95746aa73a48e811749a3b0aaeb31680#gistcomment-3924233.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: rectify a coding style issue in PglogBasedRecovery::do_recovery().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: osd operations respect interruptor's InterruptCondition.

For the sake of DRY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: _do_write_small fix head_pad

Signed-off-by: dheart <dheart_joe@163.com>

cephadm: expose gather-facts api method

Fixes: https://tracker.ceph.com/issues/51209
This PR intends to expose host metadata(gather-facts) api method

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #42052 from TRYTOBE8TME/wip-rgw-dpp-3

rgw: DPP addition 4

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #43181 from xxhdx1985126/wip-extent-placement-manager-multi-device-3

crimson/os/seastore: add multi-device support

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: set ExtentPlacementManager::allocated_to before rolling segments

There are circumstances in which a transaction that are supposed to roll the current segment
is invalidated after it finished writing and before it rolls the segment. If we don't set
ExtentPlacementManager::allocated_to in this situation, another transaction can try to write
to the old "allocated_to" position, which would cause an invalid write error

Fixes: https://tracker.ceph.com/issues/52894
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43449 from rzarzynski/wip-crimson-interruptible-peeringevent

crimson/osd: implement interruptions in PeeringEvent.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43483 from cyx1231st/wip-seastore-record-header-fullness

crimson/os/seastore: measure inline/ool record header fullness

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/rook: apply mds using placement spec and osd_pool_default_size

This commit changes the apply_mds command in the rook orchestrator
to support some placement specs and also sets the replica size according
to the osd_pool_default_size ceph option.

This commit also adds `orch apply mds` to the QA to test if the command
runs.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: factor out replica/failureDomain calc

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43471 from liewegas/fix-rook-1-node

qa/suites/orch/rook: replicate across osds for 1-node clusters

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/dashboard: replace Client connections with active-stdby mgrs

Fixes: https://tracker.ceph.com/issues/52121
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #43376 from Daniel-Pivonka/cephadm-apply-health

mgr/cephadm: set HEALTH warnings during apply phase in serve

Reviewed-by: Melissa Li <mingkli@redhat.com>

Merge pull request #43455 from liewegas/qa-nvme-loop

qa: use nvme_loop devices for (some) cephadm tests

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43450 from guits/fix_selinux_apply_spec

cephadm: shell --mount shouldnt enforce ':z' option

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43458 from mgfritch/cephadm-scp

mgr/cephadm: use `asyncssh.scp` to write remote files

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/rook: specify failureDomain on CephObjectStore CR

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #43473 from rhcs-dashboard/wip-pr_deps-master

.github: check if PR deps are merged/closed

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43485 from rhcs-dashboard/fix-pr_triage-master

.github/pr-triage: rename GH token

Reviewed-by: Nizamudeen A <nia@redhat.com>

.github/pr-triage: rename GH token

Repo projects use GITHUB_TOKEN instead of MY_GITHUB_TOKEN:
https://github.com/srggrs/assign-one-project-github-action/blob/master/entrypoint.sh#L19

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

src/rgw: New commit to fix make check

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_kms.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_keystone.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: DPP addition to rgw_sts.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #43472 from rhcs-dashboard/wip-gh_action_project_dashboard-master

.github: add dashboard PRs to Dashboard project

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: measure inline/ool record header fullness

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup record_size_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_write_stats_t::overhead_bytes

To header_bytes to be more explicit.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43182 from CongMinYin/fix-writesame-assert

librbd/cache/pwl: initialize number_log_entries

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os/seastore/segment_cleaner: update segment_info_set_t to use segment_map_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: rework SpaceTracker* to use a simpler segment map type

segment_info_set_t does extra things not relevant to these structures.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: max_device -> DEVICE_ID_MAX, clarify users

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore_types.h: clarify segment_id_t visibility, constants

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: check for null in SpaceTrackerSimple constructor

Since we're passing a sparse pointer vector, we need to check for null
in every user.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner.cc: adjust whitespace in init_segments()

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: remove unnecessary template declarations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: adjust segment cleaner's space counter calculation

Until now, segment cleaner calculate available/used spaces assuming that only the journal
does all the writes, which is not true any more. This commit make segment cleaner track
all segment managers' empty/open segments and further calculate the various space usage
based on that information

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: multi-device support

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: enable SegmentCleaner to hold multiple segmented devices

For now, all segmented devices are treated the same as SEGMENTED. In the future,
there may be different kinds of segmeted devices, like SEGMENTED_NVME, SEGMENTED_SSD,
and even SEGMENTED_SATA. We plan to use a dedicated segment cleaner for each kind of
those devices, and if there are multiple devices of the same kind, they share the same
segment cleaner.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: change Scanner to ExtentReader

This commit makes Scanner an extent reader that route read requests to the corresponding
backing devices according to the device ids encapsulated in the segment ids.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: make segment managers device-id aware

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: introduce device id into segment_id_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43166 from rzarzynski/wip-crimson-no-subproc

 crush, crimson: don't support crush_location_hook as crimson lacks SubProcess

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43346 from hjwsm1989/fix-52765

tools/ceph-kvstore-tool: fix segfaults when repair the rocksdb

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43452 from adk3798/agent-perf1

cephadm: agent: subtract average time of previous iterations off wait time

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43401 from romoh/master

src/cephadm: Add support for mariner

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge pull request #43443 from tchaikov/wip-elapsed-in-double

msg/async/ProtocolV2: cast usec to double when converting from nsec

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: dongdong tao <dongdong.tao@canonical.com>

qa/tasks: replace iterkeys() with keys() in Python 3

Python 2.7 reached the end of its lifetime, the pr fixes teuthology task
error in Python 3.x

Fixes: https://tracker.ceph.com/issues/52878
Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>

Merge pull request #43343 from trociny/wip-crushdiff-compat

tools/crushdiff: support old format json dump

Reviewed-by: Neha Ojha <nojha@redhat.com>