doc: add the reject the clone when threads are not available feature in the document

Fixes: https://tracker.ceph.com/issues/59714
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

qa: add test cases for the support to reject clones feature

Fixes: https://tracker.ceph.com/issues/59714
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

mgr/volumes: support to reject CephFS clones if cloner threads are not available

CephFS clone creation have a limit of 4 parallel clones by default at a time and rest
of the clone create requests are queued. This makes CephFS cloning very slow when
there is large amount of clones being created.After this patch clone requests won't be accepeted
when the requests exceed the `max_concurrent_clones` config value.

Fixes: https://tracker.ceph.com/issues/59714
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #55465 from neesingh-rh/wip-63991

doc: add description of metric fields for cephfs-top

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #55479 from ronen-fr/wip-rf-fifolegacy

rgw: mark maybe_unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55475 from zdover23/wip-doc-2024-02-07-radosgw-admin-confval

doc/radosgw: add confval directives

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55149 from rkachach/fix_issue_63992

mgr/prometheus: fix orch check to prevent Prometheus from crashing

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #54623 from alimaredia/wip-rgw-op-metrics-section-rename

rgw: change section name of rgw_op counters

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: fix orch check to prevent Prometheus from crashing
Fixes: https://tracker.ceph.com/issues/63992
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55476 from zdover23/wip-doc-2024-02-07-cephadm-tshooting

doc/cephadm - edit t11ing.

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add confval directives

Add confval directives to the documentation of "quota cache" options.
This addresses a request made by Antony D'Atri in https://github.com/ceph/ceph/pull/55075/files#r1444006246.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: mark maybe_unused variables

Specifically, variables that are only used in debug builds.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54596 from cbodley/wip-cmake-dencoder-modules

cmake: ceph-dencoder depends on ceph-dencoder-modules

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55376 from guits/guits-cv-64248

ceph-volume: fix zap_partitions() in devices.lvm.zap

doc/cephadm - edit t11ing.

Edit "Running Repeated Debugging Sessions" in
doc/cephadm/troubleshooting.rst. Improve grammar. Fix a misspelling.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc: add description of metric fields for cephfs-top

Fixes: https://tracker.ceph.com/issues/63991
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #55444 from Matan-B/wip-crimson-seastar-upgrade

Crimson: bump up seastar version

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54280 from xxhdx1985126/wip-rbm-scattered-allocation

crimson/os/seastore/random_block_manager: allow allocating multiple extents for a single disk space allocation request

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

ceph.spec.in: crimson add protobuf

Following https://github.com/scylladb/seastar/commit/eea4102091bb4d689e0b5a769512966b716ee8a2

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/crush/CrushLocation: fix compilation errors

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

CMakeLists.txt: fix Seastar expected dirs

```
  # create the directory so cmake won't complain when looking at the imported
  # target: Seastar exports this directory created at build-time
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

seastar: bump up seastar submodule version

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55054 from pereman2/zns-remove

os/bluestore: remove zoned namespace support
It has never been finished and now its in the way of future improvements.

Merge pull request #55275 from qiuxinyidian/rgw-noti-dev

rgw: add topic owner user check when creating

reviewed-by: cbodley, kchheda3

Merge pull request #55447 from zdover23/wip-doc-2024-02-05-rados-operations-pools

doc/rados: update PG guidance

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: modify topic owner check when creating

add tests to cover topic policies
as well as behavior when no policies are defined

Fixes: https://tracker.ceph.com/issues/64124
Signed-off-by: Zhipeng Li <qiuxinyidian@gmail.com>

Merge pull request #54311 from gabriel-samfira/do-not-escape-slash

RGW: fix cloud-sync not being able to sync folders

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55252 from jzhu116-bloomberg/wip-21800

rgw/multisite: avoid writing multipart parts to the bucket index log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55293 from jzhu116-bloomberg/wip-63178

rgw/lc: pass in flag as an argument when calling function complete(..)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54661 from lightmelodies/rgw-gc

rgw: optimize gc chain size calculation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55452 from nbalacha/wip-typo-fix

rbd-mirror: fix a typo in NotifyAckPayload::dump()

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55406 from cbodley/wip-64255

rgw: SiteConfig::load() falls back to local zonegroup

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #55448 from guits/subclass-thread

node-proxy: refactor entrypoint

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #55280 from ronen-fr/wip-rf-extra-lg

osd: remove superfluous '>' in spg_t formatter

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55414 from yuvalif/wip_yuval_url_decode_sns

rgw/rest: fix url decode of post params passed as attributes

reviewed-by: cbodley

rbd-mirror: fixed a typo in instance_watcher

Fixes a typo in NotifyAckPayload::dump().

Signed-off-by: N Balachandran <nibalach@redhat.com>

node-proxy: refactor entrypoint

This commit introduces a major refactor of the main
entrypoint.

- subclass threading.Thread:
  - Introduce a new class `BaseThread()` that is a
    `threading.Thread()` abstraction class in order
    to monitor the different threads.
  - `BaseSystem()` inherits from `BaseThread()`.
  - Handle `SIGTERM` signal in order to gracefully shutdown
    node-proxy (make threads exit gracefully, log out from RedFish API, etc.)

Additionally, this:
  - drops the class `Logger()` from util.py which
was not adding value. It is now replaced with a simple `get_logger()`
function.
  - changes the node-proxy API port from 8080 to 9456
    (8080 being widely used for frontend apps...)
  - changes the container entrypoint in order to use the
    `ceph-node-proxy` binary from the packaging

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

doc/rados: update PG guidance

Update the "Creating a Pool" section of doc/rados/operations/pools.rst
so that the documentation no longer insists that the user change the
values of "osd_pool_default_pg_num" and "osd_pool_default_pgp_num".

See also: https://github.com/ceph/ceph/pull/55419

Tracker: https://tracker.ceph.com/issues/64259

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/transaction_manager: allow allocating multiple
extents when rewriting data extents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/object_data_handler: allow alloc multiple extents
for a single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: fix errorator mismatch

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: add unittest cases for scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

test/crimson/seastore: test avlallocator's scattered allocation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: TransactionManager::alloc_extent(s) may return enospc

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: add new interfaces to alloc multiple extents for a
single request

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55089 from cbodley/wip-boost-mirrors

make-dist: update boost mirrors

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #55434 from anthonyeleven/rgwyamlin

src/common/options: Correct typo in rgw.yaml.in

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55276 from Matan-B/wip-matanb-crimson-alienstore-cores

crimson: rework CPU pinning

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55419 from zdover23/wip-doc-2024-02-02-rados-config-pool-pg-config-ref

doc/rados: update config for autoscaler

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

src/common/options: Correct typo in rgw.yaml.in

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc/rados: update config for autoscaler

Update doc/rados/configuration/pool-pg-config-ref.rst to account for the
behavior of autoscaler.

Previously, this file was last meaningfully altered in 2013, prior to
the invention of autoscaler. A recent confusion was brought to my
attention on the Ceph Slack whereby a user attempted to alter the
default values of a Quincy cluster, as suggested in this documentation.
That alteration caused Ceph to throw the error "Error ERANGE: 'pgp_num'
must be greater than 0 and lower or equal than 'pg_num', which in this
case is one" and a related "rgw_init_ioctx ERROR" reading in part
"Numerical result out of range". The user removed the
"osd_pool_default_pgp_num" configuration line from ceph.conf and the
cluster worked as expected. I presume that this is because the removal
of this configuration line allowed autoscaler to work as intended.

Fixes: https://tracker.ceph.com/issues/64259
Co-authored-by: David Orman <ormandj@corenode.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: SiteConfig::load() falls back to local zonegroup

allow radosgw-admin commands like 'user create' to operate on a new zone
that hasn't been committed to the period yet. this follows similar logic
in RGWSI_Zone::do_start()

Fixes: https://tracker.ceph.com/issues/64255
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54312 from dvanders/dvanders_mcltf_true

qa/tasks/cephadm: enable mon_cluster_log_to_file

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #54837 from imtzw/urlencode

rgw: fix list obj v1 nextmarker

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54867 from BBoozmen/oozmen_lc_subsystem

rgw/common: add rgw lifecycle specific debug log subsystem

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #55162 from cfsnyder/wip-64014-cfsnyder-pacific

rgw: fix issue with concurrent versioned deletes leaving behind olh entries

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #55229 from tobias-urdin/http-error-keystone-issue-admin-token-request

rgw: fix http error check in keystone issue_admin_token_request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55330 from baum/centos9-install-build-deps

🧹build dependencies: centos9

Merge PR #55233 into main

* refs/pull/55233/head:
qa: use centos 9.stream for cephfs stock kernel testing

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53901 from cbodley/wip-qa-container-distros-s

qa/distros: bump container host distros from centos 8->9

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55413 from petrutlucian94/flipped_tid

msg: update MOSDOp() to use ceph_tid_t instead of long

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson: introduce crimson_seastar_cpu_cores

seastar::smp::count is deduced from the cpuset provided.

from seastar/src/core/reactor:
```
   if (smp_opts.smp) {
        smp::count = smp_opts.smp.get_value();
    } else {
        smp::count = cpu_set.size();
    }
```

Co-authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

rgw/rest: fix url decode of post params passed as attributes

this is fixing a regression with the SNS policies
cause by: 4bdc5d18dd68b95c6ccd4c0e77a1bd04ad86dbb8

the changes to the test code is to accomodate different
boto3 versions

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

msg: update MOSDOp() to use ceph_tid_t instead of long

The MOSDOp constructor receives the the transaction ID as a long
instead of ceph_tid_t.

The issue is that "long" uses 32b on Windows instead of 64 bits,
so it flips after about 2 billion requests. At that point, the OSD
replies are dropped because of transaction ID mismatches.

We'll solve the issue by using the correct type for the transaction
id, specifically ceph_tid_t.

Fixes: https://tracker.ceph.com/issues/64291
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #55373 from ivoalmeida/snapshot-schedule-form-directory-typeahead

Snapshot schedule form directory typeahead

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55393 from rkachach/fix_issue_64266

mgr/rook: adding deployment to ceph image built for rook e2e testing

vstarh.sh: Crimson - set crimson_alien_thread_cpu_cores

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/os/alienstore: cleanup N_CORES_FOR_SEASTAR constant

No CPU pinning if crimson_alien_thread_cpu_cores is not set.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/rook: adding deployment to ceph image built for rook e2e testing
Fixes: https://tracker.ceph.com/issues/64266
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

crimson/os/seastore/avlallocator: add the interface that allocate
multiple regions

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

rgw: change topic label to lowercase

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: change section name of rgw_op counters

The rgw_op section of `counter dump/schema` becomes:
- rgw_op_global for the global op counters
- rgw_op_per_user for the user labeled counters
- rgw_op_per_bucket for the bucket labeled counters

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #47723 from mdw-at-linuxbox/wip-master-tempurl

rgw: swift: tempurl fixes for ceph

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55403 from cbodley/wip-doc-rgw-topic-attrs

doc/rgw: fix Attributes index in CreateTopic example

Merge pull request #54742 from guits/node-proxy

orch: implement hardware monitoring

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #55397 from jagombar/fixInvalidSnapId2

librbd: return ENOENT from Snapshot::get_timestamp for nonexistent snap_id

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rgw: fix Attributes index in CreateTopic example

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: added dir search to snap schdl form

Fixes: https://tracker.ceph.com/issues/64246
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #54781 from rhcs-dashboard/nvmeof-integration

mgr/dashboard: implement APIs for nvmeof management

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #54900 from ivoalmeida/snapshot-schedule-create

added snap schedule form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55340 from ronen-fr/wip-rf-reserver2

osd/scrub: use an AsyncReserver to handle scrub reservations on the replica side

Reviewed-by: Samuel Just <sjust@redhat.com>-

Merge pull request #55198 from JoshSalomon/balancer-perf

osd: Read balancer for devices with different sizes (performance)

doc: add node-proxy documentation

This commit adds some documentation about the
'hardware inventory / monitoring' feature (node-proxy agent).

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55396 from zdover23/wip-doc-2024-01-31-arch-monitors

doc/architecture: improve some paragraphs

Merge pull request #55383 from ceph/cpp-redis-cxx-standard

cpp_redis: update to a version with the CMAKE_CXX_STANDARD fixed

Reviewed-by: Ali Maredia <amaredia@redhat.com>

ceph-volume: fix zap_partitions() in devices.lvm.zap

The current logic is unable to find the holders for the
partition being zapped.

This commit fixes this issue.

Fixes: https://tracker.ceph.com/issues/64248
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

librbd: return ENOENT from Snapshot::get_timestamp for nonexistent snap_id

Fixes: https://tracker.ceph.com/issues/47287
Signed-off-by: John Agombar <agombar@uk.ibm.com>

doc/architecture: improve some paragraphs

Improve paragraphs under the heading "The Ceph Storage Cluster". Remove
a sentence that was pleonastic in its context in the paragraph.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: added snap schedule form

Fixes: https://tracker.ceph.com/issues/63827
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #55305 from Matan-B/wip-matanb-crimson-do_delete_work

crimson/osd/pg: do_delete_work use claim_and_reset

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

cpp_redis: update to a version with the CMAKE_CXX_STANDARD fixed

This updates the cpp_redis submodule to a version
which contains https://github.com/ceph/cpp_redis/pull/5

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

test: fix test_scrub_sched following recent scrub changes

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: restore 'dump_scrub_reservations' functionality

As ScrubResources is no longer involved in remote reservations, some
of the data listed by 'dump_scrub_reservations' is now collected by
OsdScrub itself (prior to this change, OsdScrub just forwarded the
request to ScrubResources).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: control reservation queueing using a config option

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: directly manage remote reservations in the FSM

The FSM now interacts with the scrub_reserver directly.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

vstart: add nvmeof_gw to the vstart script

so that it can be later used by the dashboard to configure the nvmeof
through UI

and create rbd pool in UI

Fixes: https://tracker.ceph.com/issues/64201
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55379 from ljflores/wip-revert-labler-changes

tools: add 'osd-size-aware' mode to osdmaptool

Sample command:
`osdmaptool om --read out.txt --read-pool cephfs.a.data --osd-size-aware`

Signed-off-by: Laura Flores <lflores@ibm.com>

mgr/balancer: add new keys to balancer mgr module

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #55372 from zdover23/wip-doc-2024-01-30-architecture-typo

doc/architecture.rst - fix typo

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>