mgr/rook: increase minikube mem to 6GB to avoid stability issues

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/rook: adding some basic rook e2e testing
Fixes: https://tracker.ceph.com/issues/64029
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge PR #54869 into main

* refs/pull/54869/head:
mds: dump sr objects in debug output

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #54915 into main

* refs/pull/54915/head:
doc/cephfs/client-auth.rst: correct `This so because` to  `This is because`
doc/cephfs/client-auth.rst: correct ``fs authorize cephfs1 /dir1 client.x rw``

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54972 into main

* refs/pull/54972/head:
src: kernel mount command returning misleading error message

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55170 from zdover23/wip-doc-2024-01-14-radosgw-admin-set-bucket-rate-limit

doc/radosgw: edit admin.rst "set bucket rate limit"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54990 from ifed01/wip-ifed-fix-expand-with-ncb

os/bluestore: fix free space update after bdev-expand in NCB mode.

Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #55188 from cyx1231st/wip-crimson-increase-obc-lru

common/options/crimson: increase crimson_osd_obc_lru_size to 512

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54759 from jzhu116-bloomberg/wip-63178

rgw/lc: do not add datalog/bilog for some lc actions

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #54973 from Matan-B/wip-matanb-should-whiteout-helper

osd/osd/PrimaryLogPG: use should_whiteout helper in _delete_oid()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54780 from jianwei1216/log_to_syslog_main

common: LogClient do not output meaningless logs by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #54116 from EdwardVitor/fix-warning-mon

mon: resolve warning about inconsistent variable types

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55161 from tobydarling/patch-2

doc/rados/operations: Fix off-by-one erors in control.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55172 from Matan-B/wip-matanb-crimson-neorados

rados_api_tests: Crimson Skip unsupported

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53281 from xxhdx1985126/wip-crimson-recover-snapset

crimson/osd/pg: set obc's ssc when handling pull responses

Merge pull request #55202 from zhscn/fix-ssc

crimson/osd: get SnapSetContext from head_obc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

src: kernel mount command returning misleading error message

Fixes: https://tracker.ceph.com/issues/63866
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #53814 from rhcs-dashboard/fix-61720-main

mgr/dashboard: upgrade from old 'graph' type panels to the new 'timeseries' panel

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #55181 from rkachach/fix_issue_64027

mgr/rook: always recreate kvm default network + fix groups refresh

Merge PR #53080 into main

* refs/pull/53080/head:
mds/journal: make used_preallocated_ino crash more graceful

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #53238 into main

* refs/pull/53238/head:
mds: just wait the client flushes the snap and dirty buffer

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54511 into main

* refs/pull/54511/head:
test/libcephfs: move the snapdiff to a dedicated binary

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #54958 into main

* refs/pull/54958/head:
mds: func name fixup, plus, and additional log message

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #54682 into main

* refs/pull/54682/head:
qa: drop mon_host check from test_cephfs_mirror_peer_bootstrap
pybind/mgr/mirroring: drop mon_host from peer_list

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/rook: always recreate kvm default network + fix groups refresh
Fixes: https://tracker.ceph.com/issues/64027
This change also includes:
- adding ~/.local/bin to path so behave binary can be found
- adding requirements.txt file for testing dependencies
- increasing timeout used to wait for tools deployment to 90s
- increasing timeout used to wait for kvm network to 20s

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55168 from anthonyeleven/ec-table

doc/rados/operations: add EC overhead table to erasure-code.rst

doc/rados/operations: add EC overhead table to erasure-code.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge PR #54041 into main

* refs/pull/54041/head:
qa: fix error reporting string in assert_cluster_log

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55105 from ronen-fr/wip-rf-nonorecovery

osd/scrub: allow replicas to scrub even if their OSDs have ongoing backfills

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

qa/config/crimson_qa_overrides.yaml:  Decrease lru_size for testing

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/neorados/misc: Crimson - Skip NeoRadosMisc.WriteSame

See: https://tracker.ceph.com/issues/64040

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/librados/aio_cxx: Crimson - Skip LibRadosAio.MultiReads

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/neorados/list: Seperate NeoRadosECTest from NeoRadosTest

otherwise SKIP_IF_CRIMSON won't work in this case

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/neorados/aio_cxx: Seperate NeoRadosECTest from NeoRadosTest

SKIP_IF_CRIMSON won't work here since we try to create EC pools
prior to the test being run.
Skip if the entire test instead by seperating EC tests.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/neorados/io: Crimson - Skip NeoRadosIo.Limits

See: https://tracker.ceph.com/issues/64040

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/neorados/cls: Crimson - Skip NeoRadosCls.RemoteReads

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: fix dangling reference bug

Cherry-picked from: 3fad54e1547c2e5d8b5c54cfcdc48f2a520c3292

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #54070 from rhcs-dashboard/add-multisite-sync-status-unit-tests

mgr/dashboard: add frontend unit tests for rgw multisite sync status card

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #54765 from Matan-B/wip-matanb-crimson-submit-error-cleanup

crimson/osd: submit_error_log cleanup

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55171 from Matan-B/wip-matanb-crimson-return-no-supp

crimson/osd/osd_operations/client_request: return OpInfo errors

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55175 from ronen-fr/wip-rf-max-test-options

qa/suites/rados/thrash: modify selection of max-scrubs configuration values

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55189 from zhscn/fix-clone-indiect-mapping

crimson/os/seastore: correct the parameter passed to LBAManager::clone_mapping

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: get SnapSetContext from head_obc

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #55115 from ljflores/wip-tracker-63967

qa/tasks: fix syntax for deep-scrub command

Merge pull request #54966 from yuvalif/wip-yuval-63855

rgw/notifications: dont send notifications if op has failed

reviewed-by: cbodley

Merge pull request #55169 from zdover23/wip-doc-2024-01-14-radosgw-admin-enable-disable-user-rate-limit

docs/radosgw: edit admin.rst "enable/disable user rate limit"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55148 from cbodley/wip-63994

rgw: fix use of creds in forward_iam_request()

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54803 from rhcs-dashboard/increase-plottable-graphs-dashboard

mgr/dashboard: increase the number of plottable graphs in charts

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/osd/osd_operations/client_request: return OpInfo errors

See NeoRadosCls.DNE or LibRadosCls.DNE

Fixes: https://tracker.ceph.com/issues/64000
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: move maybe_submit_error_log

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: on_removal to assert empty log_entry map

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: submit_error_log cleanup

* error log completion logic is moved into maybe_submit_error_log
* renamed it and it2
* maybe_submit_error_log is moved outside of failure_func
* failure_func no longer gets rep_tid and record_error params
* log_entry_version is removed, submit_error_log returns the version instead

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: do_osd_ops_execute drop error_func_ptr

std::move to the only user instead

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54484 from batrick/mds-const-mdreqref

mds: use const qualifier for MDRequestRef

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #54390 from shminjs/feat-purgequeue-executed-ops

mds/purgequeue: add l_pq_executed_ops counter

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #53670 from mchangir/mds-enqueue-all-child-frags-for-scrub-for-a-given-fragset

mds/scrub: enqueue all child frags for a given fragset

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54208 from neesingh-rh/json_asok_fix

qa: fix rank_asok() to handle errors from asok commands

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54381 from lxbsz/wip-cleanup-auth

mds: misc fixes for MDSAuthCaps code

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54406 from vshankar/wip-63482

qa: enable debug mds/client for fs/nfs suite

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54519 from lxbsz/wip-63552

mds: use explicitly sized types for network and disk encoding

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54590 from vshankar/wip-57087

qa: set mds config with `config set` for a particular test

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #55185 from rhcs-dashboard/docs-build-fix

admin/doc-requirements: bump Sphinx to 5.0.2

Reviewed-by: Matan-B <NOT@FOUND>

mgr/dashboard: add frontend unit tests for rgw multisite sync status
card

Fixes: https://tracker.ceph.com/issues/64039
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

admin/doc-requirements: bump Sphinx to 5.0.2

```
Running Sphinx v4.5.0

Sphinx version error:
The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.
```

Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore: correct the parameter passed to LBAManager::clone_mapping

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #55143 from afreen23/main

mgr/dashboard: Fixes multisite topology page breadcrumb

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54645 from rhcs-dashboard/revert-tls-1.2-main

Revert "mgr/dashboard: allow tls 1.2 with a config option"

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

common/options/crimson: increase crimson_osd_obc_lru_size to 512

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #55120 from rhcs-dashboard/snapshot-delete

mgr/dashboard: delete cephfs snapshot

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>

mgr/dashboard: increase the number of plottable graphs in charts

Fixes: https://tracker.ceph.com/issues/64024
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

test/scrub: modify selection of max-scrubs configuration values

As the osd-max-scrubs default was increased from 1 to (currently) 3, the
original set of optional values under rados/thrash/3-scrub-overrides are
no longer useful.  This commits changes the set of optional values to
reflect the current default.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54974 from fweimer-rh/c99

GCC 14 compatibility fixes

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/radosgw: edit admin.rst "set bucket rate limit"

Edit "Set Bucket Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55039 from cyx1231st/wip-crimson-alienstore-submit-concurrently

crimson/os/alienstore: submit from the alien world concurrently

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

docs/radosgw: edit admin.rst "enable/disable user rate limit"

Edit "Enable/Disable User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados/operations: Fix off-by-one errors in control.rst

Description of examples was off by one

Signed-off-by: tobydarling <anothercoffee@gmail.com>

rgw/common: rgw_bl_str(bl) avoids bl.c_str()

where bl contains multiple buffer segments, c_str() has to
rellocate and copy those segments into a single buffer. use c_str()
instead, which just copies each segment into the resulting string

this allows the function to take the bufferlist argument by const ref

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: forward_iam_request_to_master() trims trailing null from response

fix a regression from commit d3ad0efaee6a19c84b2af9b3dfa1b6c902674774
which changed how we parse the response bufferlist:

-  std::string r = response.c_str();
+  std::string r = response.to_str();

when the response contains a trailing null character, this now ends up in
`r` and breaks json parsing in `parser.parse(r.c_str(), r.length(), 1)`

replace `response.to_str()` with `rgw_bl_str(response)` which trims
trailing nulls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: metadata sync checkpoint waits on global 'sync' status

in testing, i was seeing meta sync checkpoints finish even though sync
hadn't started yet:
```
rgw_multi.tests: DEBUG: current meta sync status={
    "sync_status": {
        "info": {
            "status": "building-full-sync-maps",
```

wait for the global status to reach "sync" before starting to compare
period epochs or sync markers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: delete cephfs snapshot

Fixes: https://tracker.ceph.com/issues/63990
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os/alienstore: submit from the alien world concurrently

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #53678 from cbodley/wip-qa-rgw-upgrade-reef

qa/rgw/upgrade: add reef as a starting release

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: fix use of creds in forward_iam_request()

variable `creds` was moved into the `RGWRESTConn` constructor before being
passed into `forward_iam_request()`. change `forward_iam_request()` so
it uses the member variable from the constructor instead of taking it as
an argument

Fixes: https://tracker.ceph.com/issues/63994
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54882 from Svelar/isa_xor_dev

erasure-code/isa/xor_op: add neon-based region_xor implementation

Merge pull request #55142 from zdover23/wip-doc-2024-01-11-radosgw-user-rate-limit-ops

doc/radosgw - edit admin.rst "set user rate limit"

Merge pull request #55146 from zdover23/wip-doc-2024-01-11-radosgw-get-user-rate-limit

doc/radosgw: edit admin.rst "get user rate limit"

doc/radosgw: edit admin.rst "get user rate limit"

Edit "Get User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: Fixes multisite topology page breadcrumb

The multi-site topology page just says object in breadcrumb.
The fix adds the missing "multi-site" breadcrumb.

Fixes https://tracker.ceph.com/issues/63635

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

doc/radosgw - edit admin.rst "set user rate limit"

Edit "Set User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53999 into main

* refs/pull/53999/head:
PendingReleaseNotes: support for subvolumes and subvolume groups in snap_schedule
snap_schedule/tests: fix db upgrade issue
qa: add yaml for on demand subvol version testing
qa: add test cases for testing --subvol and --group arguments
mgr/volumes: conditionalize subvolume upgrade
mgr/volumes: ensure correct init of v1 subvol
mgr/snap_schedule: add subvol and subvol group arguments to cli
mds/snap_schedule: add subvolume group column management
mgr/volumes: add remote helper methods to fetch subvolume info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw/lc: use flag and named constants instead of booleans as argument in interface

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

rgw/lc: remove default arguments in virtual functions

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

rgw/lc: do not add datalog/bilog for some lc actions

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #55101 from Matan-B/wip-matanb-crimson-snaptrim_event-cleanup

crimson/osd/osd_operations: snaptrim_event cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52815 from adk3798/upgrade-nfs-migration-check

qa/cephadm: mgr-nfs-upgrade, match any migration > 2

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #51963 from adk3798/async-timeout-fixup

mgr/cephadm: fixups for asyncio based timeout

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54721 from adk3798/pull-stable-compiled-cephadm

qa/cephadm: add support to pull compiled cephadm for stable branches

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55106 from zdover23/wip-doc-2024-01-09-radosgw-admin-rate-limit

doc/rgw: edit admin.rst - rate limit management

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: edit admin.rst - rate limit management

Edit the "Rate Limit Management" section in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/osd/osd_operations: let pg reference destruct last

Prevent from SnapTrimEvent::handle destruct before SnapTrimEvent::pg.
The pg reference should be destructed last.

May releate to: https://tracker.ceph.com/issues/63647

Signed-off-by: Matan Breizman <mbreizma@redhat.com>