Merge pull request #55476 from zdover23/wip-doc-2024-02-07-cephadm-tshooting

doc/cephadm - edit t11ing.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54596 from cbodley/wip-cmake-dencoder-modules

cmake: ceph-dencoder depends on ceph-dencoder-modules

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55376 from guits/guits-cv-64248

ceph-volume: fix zap_partitions() in devices.lvm.zap

doc/cephadm - edit t11ing.

Edit "Running Repeated Debugging Sessions" in
doc/cephadm/troubleshooting.rst. Improve grammar. Fix a misspelling.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55444 from Matan-B/wip-crimson-seastar-upgrade

Crimson: bump up seastar version

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54280 from xxhdx1985126/wip-rbm-scattered-allocation

crimson/os/seastore/random_block_manager: allow allocating multiple extents for a single disk space allocation request

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

ceph.spec.in: crimson add protobuf

Following https://github.com/scylladb/seastar/commit/eea4102091bb4d689e0b5a769512966b716ee8a2

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/crush/CrushLocation: fix compilation errors

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

CMakeLists.txt: fix Seastar expected dirs

```
  # create the directory so cmake won't complain when looking at the imported
  # target: Seastar exports this directory created at build-time
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

seastar: bump up seastar submodule version

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55054 from pereman2/zns-remove

os/bluestore: remove zoned namespace support
It has never been finished and now its in the way of future improvements.

Merge pull request #55275 from qiuxinyidian/rgw-noti-dev

rgw: add topic owner user check when creating

reviewed-by: cbodley, kchheda3

Merge pull request #55447 from zdover23/wip-doc-2024-02-05-rados-operations-pools

doc/rados: update PG guidance

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: modify topic owner check when creating

add tests to cover topic policies
as well as behavior when no policies are defined

Fixes: https://tracker.ceph.com/issues/64124
Signed-off-by: Zhipeng Li <qiuxinyidian@gmail.com>

Merge pull request #54311 from gabriel-samfira/do-not-escape-slash

RGW: fix cloud-sync not being able to sync folders

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55252 from jzhu116-bloomberg/wip-21800

rgw/multisite: avoid writing multipart parts to the bucket index log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55293 from jzhu116-bloomberg/wip-63178

rgw/lc: pass in flag as an argument when calling function complete(..)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54661 from lightmelodies/rgw-gc

rgw: optimize gc chain size calculation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55452 from nbalacha/wip-typo-fix

rbd-mirror: fix a typo in NotifyAckPayload::dump()

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55406 from cbodley/wip-64255

rgw: SiteConfig::load() falls back to local zonegroup

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #55448 from guits/subclass-thread

node-proxy: refactor entrypoint

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55280 from ronen-fr/wip-rf-extra-lg

osd: remove superfluous '>' in spg_t formatter

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55414 from yuvalif/wip_yuval_url_decode_sns

rgw/rest: fix url decode of post params passed as attributes

reviewed-by: cbodley

rbd-mirror: fixed a typo in instance_watcher

Fixes a typo in NotifyAckPayload::dump().

Signed-off-by: N Balachandran <nibalach@redhat.com>

node-proxy: refactor entrypoint

This commit introduces a major refactor of the main
entrypoint.

- subclass threading.Thread:
  - Introduce a new class `BaseThread()` that is a
    `threading.Thread()` abstraction class in order
    to monitor the different threads.
  - `BaseSystem()` inherits from `BaseThread()`.
  - Handle `SIGTERM` signal in order to gracefully shutdown
    node-proxy (make threads exit gracefully, log out from RedFish API, etc.)

Additionally, this:
  - drops the class `Logger()` from util.py which
was not adding value. It is now replaced with a simple `get_logger()`
function.
  - changes the node-proxy API port from 8080 to 9456
    (8080 being widely used for frontend apps...)
  - changes the container entrypoint in order to use the
    `ceph-node-proxy` binary from the packaging

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

doc/rados: update PG guidance

Update the "Creating a Pool" section of doc/rados/operations/pools.rst
so that the documentation no longer insists that the user change the
values of "osd_pool_default_pg_num" and "osd_pool_default_pgp_num".

See also: https://github.com/ceph/ceph/pull/55419

Tracker: https://tracker.ceph.com/issues/64259

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: allow allocating multiple
extents when rewriting data extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: allow alloc multiple extents
for a single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: fix errorator mismatch

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: add unittest cases for scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: test avlallocator's scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: TransactionManager::alloc_extent(s) may return enospc

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add new interfaces to alloc multiple extents for a
single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55089 from cbodley/wip-boost-mirrors

make-dist: update boost mirrors

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55434 from anthonyeleven/rgwyamlin

src/common/options: Correct typo in rgw.yaml.in

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55276 from Matan-B/wip-matanb-crimson-alienstore-cores

crimson: rework CPU pinning

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55419 from zdover23/wip-doc-2024-02-02-rados-config-pool-pg-config-ref

doc/rados: update config for autoscaler

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

src/common/options: Correct typo in rgw.yaml.in

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc/rados: update config for autoscaler

Update doc/rados/configuration/pool-pg-config-ref.rst to account for the
behavior of autoscaler.

Previously, this file was last meaningfully altered in 2013, prior to
the invention of autoscaler. A recent confusion was brought to my
attention on the Ceph Slack whereby a user attempted to alter the
default values of a Quincy cluster, as suggested in this documentation.
That alteration caused Ceph to throw the error "Error ERANGE: 'pgp_num'
must be greater than 0 and lower or equal than 'pg_num', which in this
case is one" and a related "rgw_init_ioctx ERROR" reading in part
"Numerical result out of range". The user removed the
"osd_pool_default_pgp_num" configuration line from ceph.conf and the
cluster worked as expected. I presume that this is because the removal
of this configuration line allowed autoscaler to work as intended.

Fixes: https://tracker.ceph.com/issues/64259
Co-authored-by: David Orman <ormandj@corenode.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: SiteConfig::load() falls back to local zonegroup

allow radosgw-admin commands like 'user create' to operate on a new zone
that hasn't been committed to the period yet. this follows similar logic
in RGWSI_Zone::do_start()

Fixes: https://tracker.ceph.com/issues/64255
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54312 from dvanders/dvanders_mcltf_true

qa/tasks/cephadm: enable mon_cluster_log_to_file

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54837 from imtzw/urlencode

rgw: fix list obj v1 nextmarker

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54867 from BBoozmen/oozmen_lc_subsystem

rgw/common: add rgw lifecycle specific debug log subsystem

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #55162 from cfsnyder/wip-64014-cfsnyder-pacific

rgw: fix issue with concurrent versioned deletes leaving behind olh entries

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #55229 from tobias-urdin/http-error-keystone-issue-admin-token-request

rgw: fix http error check in keystone issue_admin_token_request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55330 from baum/centos9-install-build-deps

🧹build dependencies: centos9

Merge PR #55233 into main

* refs/pull/55233/head:
qa: use centos 9.stream for cephfs stock kernel testing

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53901 from cbodley/wip-qa-container-distros-s

qa/distros: bump container host distros from centos 8->9

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55413 from petrutlucian94/flipped_tid

msg: update MOSDOp() to use ceph_tid_t instead of long

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson: introduce crimson_seastar_cpu_cores

seastar::smp::count is deduced from the cpuset provided.

from seastar/src/core/reactor:
```
   if (smp_opts.smp) {
        smp::count = smp_opts.smp.get_value();
    } else {
        smp::count = cpu_set.size();
    }
```

Co-authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

rgw/rest: fix url decode of post params passed as attributes

this is fixing a regression with the SNS policies
cause by: 4bdc5d18dd68b95c6ccd4c0e77a1bd04ad86dbb8

the changes to the test code is to accomodate different
boto3 versions

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

msg: update MOSDOp() to use ceph_tid_t instead of long

The MOSDOp constructor receives the the transaction ID as a long
instead of ceph_tid_t.

The issue is that "long" uses 32b on Windows instead of 64 bits,
so it flips after about 2 billion requests. At that point, the OSD
replies are dropped because of transaction ID mismatches.

We'll solve the issue by using the correct type for the transaction
id, specifically ceph_tid_t.

Fixes: https://tracker.ceph.com/issues/64291
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #55373 from ivoalmeida/snapshot-schedule-form-directory-typeahead

Snapshot schedule form directory typeahead

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55393 from rkachach/fix_issue_64266

mgr/rook: adding deployment to ceph image built for rook e2e testing

vstarh.sh: Crimson - set crimson_alien_thread_cpu_cores

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/os/alienstore: cleanup N_CORES_FOR_SEASTAR constant

No CPU pinning if crimson_alien_thread_cpu_cores is not set.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/rook: adding deployment to ceph image built for rook e2e testing
Fixes: https://tracker.ceph.com/issues/64266
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

crimson/os/seastore/avlallocator: add the interface that allocate
multiple regions

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #47723 from mdw-at-linuxbox/wip-master-tempurl

rgw: swift: tempurl fixes for ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55403 from cbodley/wip-doc-rgw-topic-attrs

doc/rgw: fix Attributes index in CreateTopic example

Merge pull request #54742 from guits/node-proxy

orch: implement hardware monitoring

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #55397 from jagombar/fixInvalidSnapId2

librbd: return ENOENT from Snapshot::get_timestamp for nonexistent snap_id

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rgw: fix Attributes index in CreateTopic example

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: added dir search to snap schdl form

Fixes: https://tracker.ceph.com/issues/64246
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #54781 from rhcs-dashboard/nvmeof-integration

mgr/dashboard: implement APIs for nvmeof management

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #54900 from ivoalmeida/snapshot-schedule-create

added snap schedule form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55340 from ronen-fr/wip-rf-reserver2

osd/scrub: use an AsyncReserver to handle scrub reservations on the replica side

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55198 from JoshSalomon/balancer-perf

osd: Read balancer for devices with different sizes (performance)

doc: add node-proxy documentation

This commit adds some documentation about the
'hardware inventory / monitoring' feature (node-proxy agent).

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55396 from zdover23/wip-doc-2024-01-31-arch-monitors

doc/architecture: improve some paragraphs

Merge pull request #55383 from ceph/cpp-redis-cxx-standard

cpp_redis: update to a version with the CMAKE_CXX_STANDARD fixed

Reviewed-by: Ali Maredia <amaredia@redhat.com>

ceph-volume: fix zap_partitions() in devices.lvm.zap

The current logic is unable to find the holders for the
partition being zapped.

This commit fixes this issue.

Fixes: https://tracker.ceph.com/issues/64248
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

librbd: return ENOENT from Snapshot::get_timestamp for nonexistent snap_id

Fixes: https://tracker.ceph.com/issues/47287
Signed-off-by: John Agombar <agombar@uk.ibm.com>

doc/architecture: improve some paragraphs

Improve paragraphs under the heading "The Ceph Storage Cluster". Remove
a sentence that was pleonastic in its context in the paragraph.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: added snap schedule form

Fixes: https://tracker.ceph.com/issues/63827
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #55305 from Matan-B/wip-matanb-crimson-do_delete_work

crimson/osd/pg: do_delete_work use claim_and_reset

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

cpp_redis: update to a version with the CMAKE_CXX_STANDARD fixed

This updates the cpp_redis submodule to a version
which contains https://github.com/ceph/cpp_redis/pull/5

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

test: fix test_scrub_sched following recent scrub changes

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: restore 'dump_scrub_reservations' functionality

As ScrubResources is no longer involved in remote reservations, some
of the data listed by 'dump_scrub_reservations' is now collected by
OsdScrub itself (prior to this change, OsdScrub just forwarded the
request to ScrubResources).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: control reservation queueing using a config option

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: directly manage remote reservations in the FSM

The FSM now interacts with the scrub_reserver directly.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

vstart: add nvmeof_gw to the vstart script

so that it can be later used by the dashboard to configure the nvmeof
through UI

and create rbd pool in UI

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55379 from ljflores/wip-revert-labler-changes

tools: add 'osd-size-aware' mode to osdmaptool

Sample command:
`osdmaptool om --read out.txt --read-pool cephfs.a.data --osd-size-aware`

Signed-off-by: Laura Flores <lflores@ibm.com>

mgr/balancer: add new keys to balancer mgr module

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #55372 from zdover23/wip-doc-2024-01-30-architecture-typo

doc/architecture.rst - fix typo

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

osd: unit tests for read balance osd size aware score

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Add score for read balance osd size aware policy

This score works for pools in which the read_ratio
value is set.
Current limitations:
- This mechanism ignores osd read affinty
  - There is a plan adding support for read affinity 0
    in the next version.
- This mechanism works only when all PGs are full
- If read_ration is not set - the existing mechanism (named
  fair score) is used.

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Unit tests for read balancer on OSDs with different sizes

Test cases for the read balancer which takes osd sizes into account.
Some balancing code refactoring and reorg for code that is used in
multiple tests

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Read balancer for OSDs with different sizes

This commit adds calculation for desired primary distribution which
takes into account the osd size. This way smaller OSDs can take more
read operations (by adding more primaries) and the larger OSDs take less
primaries and the load of the cluater can increase. (This feature offset
a bit the weakest link in the chain effect under some conditions). In
order to calculate the loads correctly there is a need to know the
read/write ratio for the pool, and this commit assumes the read_ratio
parameter is available for the pool.

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

osd: Add 'read_ratio' pool parameterr

This parameter is used for better read balancing with non identical
devices.
- This parameter is controlled using the commands 'ceph osd pool set/get'
- This parameter is applicable only for replicated pools
- Valid values are integers in the range [0..100] and represent the
  percentage of read IOs out of all IOs in the pool
  - Value of 0 unsets this parameter and the value will be the default
    value (this is the generic behavior of the command 'ceph osd pool
    set'
  - default value can be set by config parameter
    `osd_pool_default_read_ratio`

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

.github/workflows: revert version bump to labler action

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #55364 from ljflores/wip-pr-triage

.github/workflows: disable sync-labels in pr-triage

agent/node-proxy: fix wrong host name used in data endpoint

data['cephx']['name'] will return something like:

node-proxy.hostname123

the prefix "node-proxy." has the be removed otherwise there will be
a mismatch between what is actually expected.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/cephadm: update node-proxy unit tests

The recent migration to a separate daemon implied
some changes which have broken these tests.
This commit fixes them.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/cephadm: add a new config option 'oob_default_addr'

So there's a default value (169.254.1.1) which is the default
address for the 'OS to iDrac pass-through' interface.
Given that node-proxy will reach the RedFish API through this interface,
we can make users avoid to pass that addr when providing the host spec
at bootstrap time.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

qa/distros: add supported-container-hosts for cephadm and upgrade suites

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53610 from Patty8122/documentation-update-fsck-flagdeep

doc/man/8: Add valid options in ceph-bluestore-tool.rst

Merge PR #54515 into main

* refs/pull/54515/head:
qa: get_global_id() returns a string (global_id)
qa: correct usage of DEBUGFS_META_DIR in dedent

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #55313 from rkachach/fix_issue_64183

mgr/rook: adding metrics monitoring e2e testing