rgw/notification: add rgw notification specific debug log subsystem.

decorate the events with event details while logging.

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

Merge pull request #55401 from kamoltat/wip-ksirivad-osdmap-encode-bug

src/osd/OSDMap.cc: Fix encoder to produce same bytestream

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #55686 from rkachach/fix_issue_64516

mgr/rook: adding empty calls to upgrade_ls and upgrade_status

Merge pull request #55350 from rkachach/fix_issue_64211

mgr/rook: removing all the code related to OSDs creation/removal

Merge pull request #55642 from ivoalmeida/subvolume-group-edit-fix

mgr/dashboard: fix subvolume group edit

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/rook: adding empty calls to upgrade_ls and upgrade_status

added empty calls to upgrade_ls and upgrade_status to avoid
dashboard errors when entering the view Cluster > Upgrade. Empty
calls are used because we don't support the upgrade functionality
in rook as we do for normal Ceph deployments. In case of rook user
has to follow a different process to upgrade Ceph.

Fixes: https://tracker.ceph.com/issues/64516
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55535 from rhcs-dashboard/nvmeof-api-1.0.0

mgr/dashboard: update nvmeof protoc to 1.0.0

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: baum <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #55608 from athanatos/wip-debug-64457

unittest-seastar-socket: tolerate connection_reset in test_unexpected_down

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge PR #55660 into main

* refs/pull/55660/head:
qa: disable snapshot_clone_no_wait for some tests

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #55677 from stackhpc/typo-osd-config-ref

doc/rados/configuration/osd-config-ref: fix typo

doc/rados/configuration/osd-config-ref: fix typo

Signed-off-by: Pierre Riteau <pierre@stackhpc.com>

Merge pull request #55670 from zdover23/wip-doc-2024-02-21-rados-ops-add-or-rm-mons

doc/rados: fix broken links

doc/rados: fix broken links

Fix broken links in doc/rados/operations/add-or-rm-mons.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

qa: disable snapshot_clone_no_wait for some tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #55087 into main

* refs/pull/55087/head:
client: Add exception to POSIX ACL xattrs in removexattr
client: Fix parsing of xattr namespace in removexattr

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #55239 from jianwei1216/fix_pg_created_lost_main3

Solution 1: osd/OSD: reply pg_created when pg is peered

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55218 from tchaikov/debian/postinst

debian/*.postinst: add adduser as a dependency and specify --home when adduser

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Adam King adking@redhat.com

Merge pull request #52114 from yuvalif/wip-end2end-tracing

tracer/osd/librados/build/rgw: rgw and osd end2end tracing using opentelemetry

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/rook: raise an exception when user calls OSD related methods

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55667 from zdover23/wip-doc-2024-02-20-radosgw/multisite

doc/radosgw: edit multisite.rst

Merge PR #55601 into main

* refs/pull/55601/head:
qa: test fuse/kclient for mds upgrade seq
qa: ignore OSD_DOWN during cephadm upgrades
qa: ignore warning "Replacing daemon"
qa: ignore MDS_INSUFFICIENT_STANDBY
qa: remove ignorelist error parenthesis
mds: update comment on kclient decoding of MDSMap
qa: do upgrades from quincy and older reef minor releases
mds: reverse MDSMap encoding of max_xattr_size/bal_rank_mask

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/radosgw: edit multisite.rst

Edit "Varieties of Multi-Site Configuration" in
doc/radosgw/multisite.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55632 from rosinL/wip-64420

debian/control: add new dependencies protobuf for crismon

Merge PR #52670 into main

* refs/pull/52670/head:
doc: add the reject the clone when threads are not available feature in the document
qa: add test cases for the support to reject clones feature
mgr/volumes: support to reject CephFS clones if cloner threads are not available

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: test fuse/kclient for mds upgrade seq

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore OSD_DOWN during cephadm upgrades

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore warning "Replacing daemon"

This is expected for cephadm deployments where join_fs is configured, causing
affinity replacements.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore MDS_INSUFFICIENT_STANDBY

This is expected when bringing a volume and its mds up initially.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove ignorelist error parenthesis

Some messages are duplicated to the cluster log lookign like:

    2024-02-15T22:54:31.244 INFO:teuthology.orchestra.run.smithi033.stdout:2024-02-15T22:50:00.000263+0000 mon.smithi033 (mon.0) 558 : cluster 4 [ERR] MDS_ALL_DOWN: 1 filesystem is offline

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: update comment on kclient decoding of MDSMap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do upgrades from quincy and older reef minor releases

Fixes: https://tracker.ceph.com/issues/64441
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: reverse MDSMap encoding of max_xattr_size/bal_rank_mask

Commit e134c890 adds the bal_rank_mask with encoded (ev) version 17.  This was
merged into main Oct 2022 and made it into the reef release normally.

Commit 7b8def5c adds the max_xattr_size also with encoded (ev) version 17 but
places it before bal_rank_mask. This is problematic as there were no plans to
backport e134c890 to quincy or pacific so piggybacking on the ev 17 bump would
not work and otherwise would require the backports to be done as a set to
ensure consistency (including with the kernel client).

However, the real issue is that 7b8def5c was not merged until after reef was
already cut. This required 7b8def5c to be backported separately in [1] which
was not merged until after v18.2.1 (current reef HEAD as of this commit).
Ultimately, this means that there are reef versions (v18.2.[01]) in the wild
which expect bal_rank_mask to be encoded at ev17 and not (max_xattr_size,
bal_rank_mask). Adding to the complications, the kernel client has already
merged code [2] expecting max_xattr_size for ev17.

It was decided in a github discussion [3] to move bal_rank_mask to ev18 to
avoid updating the kernel client which was done in the main branch via 36ee8e7e
and update the reef max_xattr_size backport with the same change (d8cebd67).

Unfortunately, this breaks upgrades v18.2.[01] to newer reef versions or to
main.  The reason is that monitors will encode v17 with bal_rank_mask
(max_xattr_size is not merged yet) and send that to upgraded mgrs (which are
upgraded first). The mgr will attempt to decode bal_rank_mask as a uint64_t
(max_xattr_size) but fail because an empty (by default) bal_rank_mask is simply
encoded as a signed 32-bit integer. Consequently, the mgr will fail decoding
with:

    failed to decode message of type 45 v1: End of buffer [buffer:2]

Of course the problem does not stop there, even if the mgr were able to handle
this, the monitors/mds/clients would fail in similar fashion.

So the only choice left is to fix max_xattr_size to be encoded at ev18.
Fortunately, v18.2.2 has not been released nor has any max_xattr_size backport
to quincy/pacific been merged. The main downside will be that kernels will
wrongly decode ev17 (which is already true for ceph clusters running
v18.2.[01]). A follow-up kernel fix will be required.

[1] https://tracker.ceph.com/issues/59405
[2] linux.git d93231a6bc8a452323d5fef16cca7107ce483a27
[3] https://github.com/ceph/ceph/pull/53340#discussion_r1399255031

Fixes: https://tracker.ceph.com/issues/64440
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #55522 from cbodley/wip-63373

rgw/datalog: RGWDataChangesLog::add_entry() uses null_yield

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #54554 from clwluvw/s3select-usage

rgw: add s3select usage to log usage

Reviewed-by: Gal Salomon <gsalomon@redhat.com>

Merge pull request #55236 from tobias-urdin/keystone-invalidate-admin-token

rgw: invalidate and retry keystone admin token

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55286 from BBoozmen/oozmen_decorate_lc_events

rgw/lc: decorating log events with more details

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55451 from pritha-srivastava/wip-rgw-admin-ops-user-info

rgw: code to display the complete user id that includes tenant, names…

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55508 from cbodley/wip-59186

rgw/user: add 'active' flag to RGWAccessKey

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #55509 from ivancich/wip-display-manifest

rgw: add new `object manifest` sub-command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55637 from zdover23/wip-doc-2024-02-19-cephfs-add-remove-mds-warning-notes

doc/cephfs: edit add-remove-mds

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit add-remove-mds

Disambiguate a note in doc/cephfs/add-remove-mds.rst to help readers
distinguish between cases in which they might want to use an automated
tool such as cephadm to deploy MDSes and cases in which they might want
to manually deploy MDSes.

See: https://github.com/ceph/ceph/pull/45639

Tracker: https://tracker.ceph.com/issues/54551

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: fix subvolume group edit

Fixes: https://tracker.ceph.com/issues/64487
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

debian/control: add new dependencies protobuf for crismon

As PR https://github.com/ceph/ceph/pull/55444 update seastar version,
the new seastar imports new dependencies protobuf.

Fixes: https://tracker.ceph.com/issues/64420
Signed-off-by: luo rixin <luorixin@huawei.com>

Merge PR #55471 into main

* refs/pull/55471/head:
qa: verify labelled replication perf metrics
qa: test per-client labelled perf counters
mds: export per-client metrics as labelled perf counters
cephfs_mirror: add labeled replication performance metrics
cephfs-mirror: typo ending bracket

Reviewed-by: Robin H. Johnson <robbat2@orbis-terrarum.net>

Merge pull request #55633 from zdover23/wip-doc-2024-02-18-man-ceph-objectstore-tool

doc/man: edit "manipulating the omap key"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/man: edit "manipulating the omap key"

Edit the section "Manipulating the Object Map Key" in
doc/man/8/ceph-objectstore-tool.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54586 from xxhdx1985126/wip-pglog-concatenate

tools/ceph_objectstore_tool: add op "expand-log"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55626 from zdover23/wip-doc-2024-02-17-rados-operations-placement-groups-basic-definition

doc/rados: add PG definition

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: add PG definition

Add a definition of Placement Groups to
doc/rados/operations/placement-groups.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

common/tracer: fix decoding when jaeger tracing is disabled

We aren't currently using jaeger tracing on Windows. The issue is
that Windows hosts (or any other host that doesn't use jaeger)
are experiencing message decoding failures after a recent change [1].

This change updates the tracer encoding so that messages from
non-jaeger hosts may be decoded by services that use jaeger.

[1] https://github.com/ceph/ceph/pull/47457

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
This commit rebrings 3701ffa6733b001d4278a0b68395c5efe2382f25 which
got reverted due to an implicit dependency with other revert. Please
see https://github.com/ceph/ceph/pull/52114#issuecomment-1950288188.

Conflicts:
src/common/tracer.h
  formatting conflict with 7179ac0037320480f089e1a9e291cdc37282d250

Merge pull request #54963 from DimStar77/cmake328

cmake: Ensure git exists before executing it

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa: verify labelled replication perf metrics

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa: test per-client labelled perf counters

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: export per-client metrics as labelled perf counters

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs_mirror: add labeled replication performance metrics

Fixes: http://tracker.ceph.com/issues/63945
Signed-off-by: Jos Collin <jcollin@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: typo ending bracket

Signed-off-by: Venky Shankar <vshankar@redhat.com>

unittest-seastar-socket: tolerate connection_reset in test_unexpected_down

Fixes: https://tracker.ceph.com/issues/64457
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #55605 from ronen-fr/wip-rf-warns0224

osd: clean compiler warnings

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54922 from pponnuvel/disable_network_stats

mon, mgr: do not output network ping stats

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #53565 from pereman2/shared-blob-to-blob

os/bluestore: optional SharedBlob on Blob structure

Merge pull request #55570 from rzarzynski/wip-bug-64192

osd: always send returnvec-on-errors for client's retry

Merge pull request #55602 from ceph/wip-yuriw-add-squid-main

qa/tests: added squid option

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #55582 from cbodley/wip-63642

rgw/putobj: RadosWriter uses part head object for multipart parts

Reviewed-by: Mark Kogan <mkogan@ibm.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #55453 from ronen-fr/wip-rf-0741-logs

osd/scrub: improve scheduling decision logs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54856 from linuxbox2/wip-accept-new-awssigv4

rgw: cumulatively fix 6 AWS SigV4 request failure cases

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon, mgr: do not output network ping stats

When doing PG dump using 'ceph pg dump --format json-pretty'
the output is extremely big that the command hangs and also
the ceph-mgr hangs and eventuall fails over.

The exact size depends on the number of OSDs in the cluster
and the number of peers for each OSD.

In tests, it's been identified that the network ping times
is the largest component in terms of size which is removed
from the output now so as to limit the overall size.

Fixes https://tracker.ceph.com/issues/57460

Signed-off-by: Ponnuvel Palaniyappan <pponnuvel@gmail.com>

Merge pull request #55579 from idryomov/wip-64423

librbd: fix split() for SparseExtent and SparseBufferlistExtent

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #55530 from trociny/wip-64376

tools/rbd: make 'children' command support --image-id

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

test/osd: fix test_scrub_sched following scrubber changes

Replacing PgScrubber::determine_scrub_time() with a local copy,
as a stop-gap measure to keep the test running.
The scrub scheduling refactoring will remove the need for
this function, and the test will be updated accordingly.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: log parameter entering scrub_requested()

... and a minor formatting fix in the same function.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #53289 from rzarzynski/wip-mon-set-allow-crimson

mon/OSDMonitor: fix the hint for set-allow-crimson

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55404 from JoshSalomon/fix-warn

osd: Fix compilation warnings for PR https://github.com/ceph/ceph/pull/55198

unittest-seastar-socket: debug to error on unexpected return from dispatch_rw_bounded

Related: https://tracker.ceph.com/issues/64457
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #54930 from rzarzynski/wip-osd-ec-rework

osd: rework EC for the sake of integration with crimson

osd: avoid copying chunks in a loop

to clear a static analyzer warning

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd: clean compiler warnings

... for unused variables, for code that has no effect, and
for trying to std::move out of a const lvalue.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/tests: added squid option

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #55566 from zdover23/wip-doc-2024-02-14-cephadm-services-nfs

doc/cephadm: correct nfs config pool name

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55539 from zdover23/wip-doc-2024-02-12-glossary-osd-terms

doc/glossary: improve OSD definitions

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55537 from guits/node-proxy-handle-fetch-oob-error

node-proxy: improve http error handling in fetch_oob_details

Merge pull request #55297 from VallariAg/fix-gtest-xml-path

qa/workunits/rados/test.sh: fix GTEST_OUTPUT path

doc/glossary: improve OSD definitions

Remove expletive construction from three OSD-id-related terms.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55585 from zdover23/wip-doc-2024-02-15-rados-operations-placement-groups-sage-blog-post

doc/rados: add link to pg blog post

Merge pull request #55584 from batrick/backport-create-squid

script: add squid release to backport-create-issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa: move cert import into rgw task

Signed-off-by: Ali Maredia <amaredia@redhat.com>

qa: enable test_awssdkv4_sig workunit to run in teuthology

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw_sigv4: catch exceptions on client builder

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55533 from ronen-fr/wip-rf-sdocs-1

doc/osd: documentation changes related to scrub

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

script: add squid to backports

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: add link to pg blog post

Add a link in the "Placement Groups" section that links to Sage Weil's
April 5, 2019 blog post "New in Nautilus: PG merging and autotuning".

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/putobj: RadosWriter uses part head object for multipart parts

the cleanup logic in the RadosWrite destructor was using the wrong
`head_obj` to avoid races between cleanup and part re-uploads. it
pointed at the final location of the multipart upload, rather than the
head object of the current part

Fixes: https://tracker.ceph.com/issues/63642
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55551 from ivoalmeida/cephfs-mount-details

mgr/dashboard: added cephfs mount details

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

librbd: refactor merge() for SparseBufferlistExtent

- pass left.length + right.length instead of bl.length()
  for consistency and to avoid circumventing the assert in
  SparseBufferlistExtent constructor
- claim_append() takes an lvalue reference, no need to move
- follow the pattern used in split()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: fix split() for SparseExtent and SparseBufferlistExtent

SparseExtents and SparseBufferlist are typedefs for interval_map.  In
both cases, split() handler is broken: for the former the extent isn't
actually split and for the latter incorrect bufferlist is attached to
the split extent.

Fortunately, both SnapshotDelta as produced by ObjectListSnapsRequest
and SparseBufferlist used in a couple of places seem to be collections
where only disjoint intervals are inserted and splitting doesn't occur
(at least in the common case).  But still, this is a landmine waiting
for someone to step on it.

Fixes: https://tracker.ceph.com/issues/64423
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55577 from cityofships/typo

doc/start: fix wording & syntax

doc/osd: fix 'scrub a PG' commands in the documentation

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/start: fix wording & syntax

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

mgr/dashboard: added cephfs mount details

Fixes: https://tracker.ceph.com/issues/64405
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

mgr/cephadm: add fullreport in ceph orch CLI (node-proxy)

This adds the `fullreport` category to the `ceph orch hardware status` CLI.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/cephadm: conditionally deploy node-proxy

Only attempt to deploy the node-proxy agent when oob details
are provided for a given host.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>