osd: correct unsigned/signed compiler wrn

    /home/pdonnell/ceph/src/osd/OSD.cc: In member function ‘void OSD::ShardedOpWQ::stop_for_fast_shutdown()’:
    /home/pdonnell/ceph/src/osd/OSD.cc:11143:41: warning: comparison of integer expressions of different signedness: ‘int’ and ‘uint32_t’ {aka ‘unsigned int’} [-Wsign-compare]
    11143 |   for (int shard_index = 0; shard_index < osd->num_shards; shard_index++) {

Fixes: https://tracker.ceph.com/issues/62851
Fixes: 210dbd4ff19ea66fd2f0109cc15aad53349be52f
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53735 from idryomov/wip-63010

qa/suites/krbd: stress test for recovering from watch errors

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #50350 from BBoozmen/oozmen_rgw_beast_log_bucket_name

rgw/logging: add bucket and object names to the log line for ops-log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52144 from linuxbox2/wip-nochown-bucket

rgw/file: make setattr(...) a no-op on buckets

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52960 from cbodley/wip-62411

rgw/lc: remove_bucket_config() doesn't update xattrs on bucket delete

Reviewed-by Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #53266 from VVoidV/lbr-chunk-upload-buffer-fix

rgw: improve buffer list utilization in the chunkupload scenario

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53525 from RickyMaRui/fix-SignatureDoesNotMatch

rgw: fix SignatureDoesNotMatch when extra headers start with 'x-amz'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53602 from cbodley/wip-62938

rgw: s3website doesn't prefetch for web_dir() check

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53614 from xrmeng8756/master

rgw: fix rgw rate limiting RGWRateLimitInfo class decode_json max_rea…

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #53676 from smanjara/data-sync-fairness-crash

rgw/multisite: call drain before flushing markers in incremental sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53680 from cbodley/wip-62989

rgw/keystone: EC2Engine uses reject() for ERR_SIGNATURE_NO_MATCH

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53684 from cbodley/wip-rgw-keystone-yield

rgw/async: use optional_yield for keystone and kms requests

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #53691 from max-licc/radosgw_start_watcher_crash

rgw: fix unwatch crash at radosgw startup

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #53714 from cbodley/wip-63004

[CVE-2023-43040] rgw: Fix bucket validation against POST policies

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53724 from adk3798/rm-get_unit_name_by_instance

cephadm: remove get_unit_name_by_instance func

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53715 from adk3798/other-haproxy-container-version

cephadm: fix haproxy version with certain containers

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53744 from zdover23/wip-doc-2023-10-02-architecture-16-of-x

doc/architecture: edit "Calculating PG IDs"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/suites/krbd: stress test for recovering from watch errors

Fixes: https://tracker.ceph.com/issues/63010
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #53606 from Patty8122/control-reaches-end-fix

test: corrected control reaches end by adding a return

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

doc/architecture: edit "Calculating PG IDs"

Edit the section "Calcluating PG IDs" in doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test: corrected control reaches end by adding a return

Signed-off-by: Patty8122 <divyapattisapu@uchicago.edu>

Merge pull request #53580 from AliMasarweh/wip-alimasa-persistent-q-migration

RGW | Bucket Notification: migrating old entries to support persistency control

Reviewed-by: yuvalif <ylifshit@redhat.com>

Merge pull request #46692 from pdvian/wip-unreachable-osd

osd: Report health error if OSD public address is not within subnet

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52509 from ericqzhao/main

mgr: add throttle policy for DaemonServer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #53740 from zdover23/wip-doc-2023-09-30-architecture-15-of-x

doc/architecture: edit several sections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #53618 from adk3798/cephadm-refactor-more-net-utils

cephadm: move more funcs into net_utils.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/architecture: edit several sections

Edit the following sections in doc/architecture.rst:

 1. Dynamic Cluster Management
 2. About Pools
 3. Mapping PGs to OSDs

The tone of "Dynamic Cluster Management" remains a bit too close to the
tone of marketing material, in my opinion, but I will return to firm it
up when I have finished a once-over of architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #53702 from adk3798/cephadm-refactor-registry-login

cephadm: move registry_login to container_engines.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53617 from adk3798/cephadm-refactor-more-data-utils

cephadm: move more funcs into data_utils.py

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #53612 from adk3798/fix-refreshed-column

mgr/cephadm: fix REFRESHED column of orch ps being unpopulated

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: re-format black data_utils.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move more funcs into data_utils.py

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53682 from phlogistonjohn/jjm-cephadm-move-pathify

cephadm: move pathify & get_file_timestamp to file_utils

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53683 from phlogistonjohn/jjm-cephadm-move-getters

cephadm: move context getters

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #53726 from zdover23/wip-doc-2023-09-29-architecture-14-of-x

doc/architecture: edit "Replication"

Reviewed-by: Cole Mitchell <cole.mitchell.ceph.@gmail.com>

Merge pull request #53605 from sayantani11/osd-1

script/backport-resolve-issue: Update script with latest versions

rgw: fix rgw rate limiting RGWRateLimitInfo class decode_json max_read_bytes and max_write_bytes field mismatch

Fixes: https://tracker.ceph.com/issues/62955
Signed-off-by: xiangrui meng <mengxr@chinatelecom.cn>

Merge pull request #53723 from zdover23/wip-doc-2023-09-29-architecture-13-of-x

doc/architecture: edit "Data Scrubbing"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "Replication"

Edit "Replication" in the "Smart Daemons Enable Hyperscale" section of
doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/architecture: edit "Data Scrubbing"

Edit the "Data Scrubbing" listitem in the list of benefits conferred by
the use by OSDs of the aggregate power of the cluster, in the section
"Smart Daemons Enable Hyperscale" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53594 into main

* refs/pull/53594/head:
qa: fix "no orch backend set" in nfs suite

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53707 from zdover23/wip-doc-2023-09-28-architecture-12-of-x

doc/architecture: edit "OSD Membership and Status"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

cephadm: remove get_unit_name_by_instance func

As it is one line, quite simple, and only
had a single caller, it was decided we'd remove
this function as part of the cephadm refactor.

Signed-off-by: Adam King <adking@redhat.com>

doc/architecture: edit "OSD Membership and Status"

Edit "OSD Membership and Status" in the "Smart Daemons Enable
Hyperscale" section of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

cephadm: fix haproxy version with certain containers

Some builds of haproxy containers' output
from "haproxy -v" start with

HAProxy version

rather than

HA-Proxy version

no reason on our end not to accept both

Signed-off-by: Adam King <adking@redhat.com>

rgw: Fix bucket validation against POST policies

It's possible that user could provide a form part as a part of a POST
object upload that uses 'bucket' as a key; in this case, it was
overriding what was being set in the validation env (which is the real
bucket being modified). The result of this is that a user could actually
upload to any bucket accessible by the specified access key by matching
the bucket in the POST policy in said POST form part.

Fix this simply by setting the bucket to the correct value after the
POST form parts are processed, ignoring the form part above if
specified.

Fixes: https://tracker.ceph.com/issues/63004
Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>

Merge pull request #53608 from cbodley/wip-62536

qa/rgw/sts: keycloak task installs java manually

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw/file: make setattr(...) a no-op on buckets

Shallow fix for apparent unstable behavior after nfs "chown" on
an RGW bucket via RGW NFS.  While we allow buckets to be created
(and subject to ordinary rules, deleted), chown against a bucket
hasn't been tested and potentially is not valid.  Prevent it
altogether for now--if permissions would allow it, chown will
succeed but won't have any effect.

Fixes: https://tracker.ceph.com/issues/61689
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

RGW | Bucket Notification: migrating old entries to support persistency control

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

qa/suites/krbd: rename singleton to singleton-msgr-failures

A "singleton without msgr-failures" is wanted in the next commit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #53432 from liu-chunmei/seastore_device_size

crimson/vstart: default seastore_device_size will be out of space f…

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: re-format black container_engines.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move registry_login to container_engines.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move logging from registry_login to command_registry_login

So that registry_login can be moved to container_engines.py
without creating a dependency on logging there

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #53607 from cfsnyder/wip-cfsnyder-bucket-check-bug

rgw: fix radosgw-admin bucket check stat calculation bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52393 from yuvalif/wip-yuval-enable-trace-lua

common/tracer: remove is_enabled check in add_span methods

reviewed-by: cbodley, ljflores

rgw: fix unwatch crash at radosgw startup

During radosgw initialization, if there is an exception in init_watch that causes the watcher registration to fail,
When finalize_watch is executed, a crash occurs due to unregister an unregistered watch.

Fixes: https://tracker.ceph.com/issues/60094
Signed-off-by: lichaochao <lichaochao2_yewu@cmss.chinamobile.com>

Merge pull request #53523 from rhcs-dashboard/add-ceph-daemon-label-labeled-counters

exporter: add ceph_daemon labels to labeled counters as well

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52828 from rhcs-dashboard/angular-15

mgr/dashboard: upgrade to angular v15

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #53596 from rhcs-dashboard/fix-validations-in-cephfs-forms

mgr/dashboard: fix cephfs forms validations

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge PR #53529 into main

* refs/pull/53529/head:
doc: update colorama, packaging
cephfs-shell: drop LooseVersion for version.parse

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53685 from zdover23/wip-doc-2023-09-27-architecture-11-of-x

doc/architecture: edit "OSDs service clients directly"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "OSDs service clients directly"

Edit "OSDs service clients directly" in the list in
"Smart Daemons Enable Hyperscale" in doc/architecure.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/async: use optional_yield for keystone and kms requests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: remove (doc)string

Remove a, now irrelevant (IMO), docstring that might have been
associated with the recently moved `cached_stdin` global. It's not
really clear how helpful it is in light of the new "compiled"
cephadm, so I am opting to remove it rather than move it.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: black format context_getters.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: rename fetch_tcp_ports to fetch_endpoints

Rename fetch_tcp_ports to fetch_endpoints to more closely match what
the function is doing.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move context based getters to context_getters.py

Move functions that exist mainly to pull information out of the
CephadmContext in various ways to a new context_getters.py module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move a logging line closer to where the data is used

Move a logging line closer to where the data being logging is
used. This avoids having a dependency on logging in a fairly
simple function and should make moving the function in a future
commit easier.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: move pathify & get_file_timestamp to file_utils

Signed-off-by: John Mulligan <jmulligan@redhat.com>

rgw/keystone: EC2Engine uses reject() for ERR_SIGNATURE_NO_MATCH

ERR_SIGNATURE_NO_MATCH means that we found the given access key in
keystone, so we should use reject() instead of deny() to prevent
other engines like LocalEngine from looking up the access key again

this change causes us to return the SignatureDoesNotMatch error expected
by s3test case test_list_buckets_bad_auth()

Fixes: https://tracker.ceph.com/issues/62989
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/multisite: call drain before flushing markers in incremental sync

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #53615 from ronen-fr/wip-rf-great-move

tools, rgw/test: add std:: qualifiers to 'move'

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #53048 from YiteGu/add-slow-count-committed-kv

os/bluestore: add some slow count for bluestore

Merge pull request #52930 from leonid-s-usov/msg_wait

msg/AsyncMessenger: re-evaluate the stop condition when woken up in 'wait()'

Merge pull request #52906 from sseshasa/wip-fix-config-dump-pretty-print

mon/ConfigMonitor: Show localized name in "config dump --format json" output

Merge pull request #52200 from jinmyeonglee/61819

client: append to buffer list to save all result from wildcard command

Merge pull request #50585 from ssdohammer-sl/fix-metadata-obj-promote

osd: chunked type manifest object not to be promoted

mgr/dashboard: upgrade to cypress 12

Looks like chrome 117 will need cypress >=12.15.0
https://github.com/cypress-io/cypress-documentation/issues/5479

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: adapt and refactor jest test files

Use the `configureTestBed` as the placeholder for adding the
declarations, imports... that is required for the unit tests to run

Fixes: https://tracker.ceph.com/issues/62844
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51923 from kamoltat/wip-ksirivad-fix-61570

pybind/mgr/pg_autoscaler: fix warn when not too few pgs
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53324 from aclamk/wip-aclamk-osd-op-scraper-fixes

Set of scraper improvements

exporter: add ceph_daemon labels to labeled counters as well

Exporter missed adding the `ceph_daemon` or `instance_id`
labels(in case if rgw metrics) to the new labeled performance counters.

Fixes: https://tracker.ceph.com/issues/62874
Signed-off-by: avanthakkar <avanjohn@gmail.com>

Merge pull request #53284 from xxhdx1985126/wip-seastore-remap-intermediate_key

crimson/os/seastore/transaction_manager: move intermediate_key by "remap_offset" when remapping the "back" half of the original pin

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #53455 from xxhdx1985126/wip-seastore-no-touch-pending

crimson/os/seastore/cache: don't add EXIST_CLEAN extents to lru

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #53657 from zdover23/wip-doc-2023-09-26-architecture-10-of-x

doc/architecture: edit "SDEH"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/architecture: edit "SDEH"

Edit the front matter of the "Smart Daemons Enable Hyperscale" section
of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/cache: replace is_clean by is_stable_clean wherever
possible

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: move intermediate_key by
"remap_offset" when remapping the "back" half of the original pin

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #52442 from kamoltat/wip-ksirivad-fix-tracker-61922

pybind/mgr/pg_autoscaler: noautoscale flag retains individual pool configs
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

mgr/cephadm: add unit test for _process_ls_output

This is a weird function to make a unit test for
since it's essentially just moving data from a
list of dicts into a list of DaemonDescriptions,
but wanted to have some coverage to lower the
chance of breaking something again.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: fix REFRESHED column of orch ps being unpopulated

The way the daemon ls data was processed was changed in
https://github.com/ceph/ceph/commit/1fd4132c7c03602719f29230732b12c8afa04779
and it seems that commit removed a line that set the
last_refresh field. This commit just adds it back
in the new location after the change.

Without this in "ceph orch ps" the REFRESHED column
for every daemon just reports "-"

Fixes: https://tracker.ceph.com/issues/62954
Signed-off-by: Adam King <adking@redhat.com>

cephadm: re-format black net_utils.py

There was a conflict here between what black
and flake8 were okay with. After running
format-black flake8 would report

cephadmlib/net_utils.py:211:29: E203 whitespace before ':'
cephadmlib/net_utils.py:259:25: E203 whitespace before ':'
cephadmlib/net_utils.py:272:27: E203 whitespace before ':'

but removing the whitespace before the ":" would
cause black to complain. For parse_mon_ip and
parse_mon_addrv, it was doing array slicing with
a start of "0" so I believe we can just remove the
start point without affecting anything (since "0" is
just the beginning of the string anyway). For
get_ipv6_address it had to actually be altered in
a way that had the potential to be done incorrectly,
so I added a unit test for it in a previous commit
in order to make sure we maintain the behavior.

Signed-off-by: Adam King <adking@redhat.com>

cephadm: add unit test for get_ipv6_address

I wanted to modify this function slightly
to try to make both black and flake8 happy
with it, so adding a unit test to make sure
I don't break it.

Signed-off-by: Adam King <adking@redhat.com>

cephadm: move more funcs into net_utils.py

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #51926 from adk3798/host-drain-zap

mgr/cephadm: add ability to zap OSDs' devices while draining host

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53589 from adk3798/cephadm-refactor-host-facts

cephadm: create host_facts.py in cephadmlib

Reviewed-by: John Mulligan <jmulligan@redhat.com>

cephadm: format black host_facts.py

Signed-off-by: Adam King <adking@redhat.com>

cephadm: create host_facts.py in cephadmlib

For storing classes/functions related to gathering
information about the hosts such as disk enclosures
and networks

Signed-off-by: Adam King <adking@redhat.com>

qa/workunits: modified tests for noautoscale flag change

modified:

`qa/workunits/mon/test_noautoscale_flag.sh`
`qa/workunits/cephtool/test.sh`

adding test coverage to files mentioned above

Fixes: https://tracker.ceph.com/issues/61922
Signed-off-by: Kamoltat <ksirivad@redhat.com>