Merge pull request #51832 from rkachach/fix_issue_61499

cephadm: adding support for ipv6 when deploying ceph-exporter

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51817 from rkachach/fix_issue_61494

mgr/cephadm: validate host label before removing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51070 from adk3798/reconfig-firewall-ports

cephadm: allow ports to be opened in firewall during adoption, reconfig, redeploy

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #52044 from vedanshbhartia/coverity_fix_tags

rgw: user correct tags for coverity annotations and add the SUPPRESS keyword

rgw: user correct tags for coverity annotations and add the SUPPRESS
keyword

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #52012 from cbodley/wip-61428

qa: suppress SyscallParam error during startup on jammy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52014 from cbodley/wip-test-cls-refcount-rename

test/cls: rename cls_refcount tests

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52009 from zdover23/wip-doc-2023-06-12-rados-operations-placement-groups-math-markup

doc/rados: add math markup to placement-groups.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52011 from samarahu/d4n-warning-cleanup

RGW: Cleanup D4N unit test warning

Merge pull request #52028 from zdover23/wip-doc-2023-06-13-dev-deploying-dev-cluster-link

doc/dev: fix "deploying dev cluster" link

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51989 from samarahu/remove-trace-comment

RGW: Replace trace with d4n in yaml long desc

Merge pull request #52018 from zdover23/wip-doc-2023-06-13-operations-crush-map-1-of-x

doc/rados: edit crush-map.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: fix "deploying dev cluster" link

Fix a link to "Deploying a Development Cluster" that had been broken
since the transition from master to main.

This is the first part of an initiative to improve the vstart
documentation.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit crush-map.rst (1 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51966 from rhcs-dashboard/grafana-url-deprecated

mgr/dashboard: change deprecated grafana URL in daemon logs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #51953 from rhcs-dashboard/dashboard-status-popover

mgr/dashboard: add popover to cluster status card

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51945 from rzarzynski/wip-bug-61504

crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51973 from Matan-B/wip-matan-lsan-supp

qa/lsan.supp: update heap_profiler suppression and ASAN_OPTIONS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51703 from lxbsz/wip-59683

xfstests_dev: install extra packages from powertools repo for xfsprogs

Merge pull request #50728 from lxbsz/wip-59195

qa: switch to use the merge fragment for fscrypt

qa: suppress SyscallParam error during startup on jammy

Fixes: https://tracker.ceph.com/issues/61428
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51972 from vedanshbhartia/coverity_resource_leak

rgw: Add coverity annotations for resource leak false positives

test/cls: rename cls_refcount tests

test cases were copy/pasted from test_cls_rgw.cc without renaming

Signed-off-by: Casey Bodley <cbodley@redhat.com>

RGW: Cleanup D4N unit test warning

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

doc/rados: add math markup to placement-groups.rst

Add some math markup to some of the mathematical expressions in
doc/rados/operations/placement-groups.rst.

This change has advantages as well as disadvantages. On the one hand,
the mathematical expressions look nicer and are easier to read.  On the
other hand, the rendered mathematical expressions are not searchable as
a part of the text, which makes CTRL+F searches useless in finding them.

This PR is offered as an RFC on the matter of whether mathematical
expressions should be rendered with the :math: extension.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51864 from yuvalif/wip-yuval-fix-61540

rgw/kafka: make sure that destroy is called after connection is removed

xfstests_dev: install extra packages from powertools repo for xfsprogs

Centos Stream 8 has removed the 'device-mapper-devel', 'libedit-devel'
and 'userspace-rcu-devel' packages from the mirrors and we need to
install it from powertools repo.

Fixes: https://tracker.ceph.com/issues/59683
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #51994 from zdover23/wip-doc-2023-06-11-rados-operations-placement-groups-3-of-x

doc/rados: edit placement-groups.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit placement-groups.rst (3 of x)

Edit doc/rados/operations/placement-groups.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51366 from SMIL-Infra/fix-test-build

rgw: fix tests build w/o AMQP

Merge pull request #51468 from vedanshbhartia/coverity_time

rgw: Add coverity annotations to ignore warnings about 32 bit time

Merge pull request #51992 from zdover23/wip-doc-2023-06-10-rados-operations-change-mon-elections

doc/rados: edit change-mon-elections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51991 from zdover23/wip-doc-2023-06-09-rados-operations-placement-groups-2-of-x

doc/rados: edit placement-groups.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit placement-groups.rst (2 of x)

Edit doc/rados/operations/placement-groups.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit change-mon-elections

Edit doc/rados/operations/change-mon-elections.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51988 from ljflores/wip-release-checklist

doc/dev: update release checklist

crimson/osd: suppress the assertion in Watch::notify_ack

Replaced the assert with an error log entry but ultimately
this commit should be reverted.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Fixes: https://tracker.ceph.com/issues/61504
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49316 from BryceCao/wip-add-initialization-for-RGWGC-process-call

rgw : fix add initialization for RGWGC::process()

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51715 from awojno-bloomberg/replication-race-condition-fix

RGW: Pass and check destination zone trace when fetching object

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

RGW: Replace trace with d4n in yaml long desc

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

doc/dev: update release checklist

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #51975 from zdover23/wip-doc-2023-06-09-rados-operations-placement-groups-1-of-x

doc/rados: edit placement-groups.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit placement-groups.rst (1 of x)

Edit doc/rados/operations/placement-groups.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #48879 from samarahu/wip-d4n

RGW: WIP D4N

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

qa/lsan.supp: update heap_profiler suppression and ASAN_OPTIONS

In continuation to: 8c099a534044bf7182e04f250e342aab76bc3e54

Regsiter singleton leak seems to be replaced by:
```
-----------------------------------------------------
Suppressions used:
  count      bytes template
      1          8 ^InitModule
-----------------------------------------------------
```

In similarity to Crimson's suppression: 6ed8d839b421442a64410444ca8f88f157ae28b3

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51967 from Dazul/add-config-missing-custom-container-images

doc: Add missing custom container_image

Merge pull request #51958 from zdover23/wip-doc-2023-06-07-rados-operations-pools-4-of-x

doc/rados: edit pools.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

mgr/dashboard: change deprecated grafana URL in daemon logs

See https://github.com/grafana/grafana/issues/51276

I am also increasting the height of the panel

Fixes: https://tracker.ceph.com/issues/61618
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51955 from rhcs-dashboard/teuth-e2e-fix

qa/workunits: fix test_dashboard_e2e.sh: no spec files found

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51911 from ronen-fr/wip-rf-recovery-updates

qa/standalone: osd-recovery-scrub: fix slow updates and recovery concurrency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

doc: Add missing custom container_image

While testing cephadm in order to migrate to it in the near
futur, I needed to use some custom images for haproxy
and keepalived. I needed to see into the code to see what
existed. So I put what I found on the code into documentation.

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>

Merge pull request #51948 from xxhdx1985126/wip-61608

crimson/os/seastore/cached_extent: add prepare_commit interface

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #51917 from sseshasa/wip-mclock-cost-fix-relnotes

PendingReleaseNotes: Document mClock scheduler fixes and enhancements

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: Add coverity annotations for resource leak false positives

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #51892 from rzarzynski/wip-bug-61547

mon: fix timestamp formatting in cluster log

Reviewed-by: Laura Flores <lflores@redhat.com>

mgr/dashboard: add popover to cluster status card

Previous dashboard has a popover for the status when there are some
warnings in the Cluster Health. Introducing the same behaviour on the
new landing page

Fixes: https://tracker.ceph.com/issues/61611
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/rados: edit pools.rst (4 of x)

Edit doc/operations/rados/pools.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51930 from zdover23/wip-doc-2023-06-06-rados-operations-pools-3-of-x

doc/rados: edit pools.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46062 from markhpc/wip-crimson-tcmalloc

crimson: Enable tcmalloc when using seastar

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/workunits: fix test_dashboard_e2e.sh: no spec files found

Fixes: https://tracker.ceph.com/issues/61578
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50054 from rhcs-dashboard/cache-invalidation

mgr/dashboard: RBD cache invalidation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

PendingReleaseNotes: Document mClock scheduler fixes and enhancements

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/os/seastore/cached_extent: add prepare_commit interface

move new fixedkv-btree nodes' inter-node pointer adjustments into
the prepare_commit interface

Fixes: https://tracker.ceph.com/issues/61608
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #51927 from ljflores/wip-rook-tests

qa/suites/rados: remove rook coverage from the rados suite

QA: Add D4N teuthology suite

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

Merge pull request #51943 from dvanders/update-contact-dan

Update affiliation for Dan van der Ster

Merge pull request #51925 from ljflores/wip-tracker-59192

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED for cls tests

Update affiliation for Dan van der Ster

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>

crimson: Enable tcmalloc when using seastar

classic-osds have always caused significant memory fragmentation
when using the libc memory allocator due to the way that Ceph
tends to utilize memory.  In recent testing, crimson-osd was found
to use 25-27GB of RAM with the stock 3GB bluestore cache settings
(osd_memory_target is only used when tcmalloc is available).  Upon
further testing, it was found that the classic OSD is even worse,
using between 32-33GB of RAM after a 5 minute 4K sequential
write test when using libc malloc.

The good news is that it appears that crimson-osd is able to use
tcmalloc for alienstore without significant modification. Better
still, it drastically reduces memory usage.  In the same test that
resulted in 25GB RSS memory usage for crimson-osd with libc malloc,
a tcmalloc linked version took around 9GB (with an 8GB
osd_memory_target).  Since we do not yet (afaik) expose classic OSD
debugging in crimson it is tough to tell why we are still a little
over, but it's clear that for alienstore we are going to need to
use tcmalloc as we do in classic.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Revert "[CHERRY-PICKED FOR TESTING ONLY] crimson: Enable tcmalloc when using seastar"

This reverts commit 380bc6d2e4ec62e724612dbac57ffdbd82511f9d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51875 from Matan-B/wip-matanb-crimson-supress-tcmalloc

crimson/osd/lsan_suppressions.cc: Add MallocExtension::Initialize()

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados: edit pools.rst (3 of x)

Edit doc/operations/rados/pools.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51914 from zdover23/wip-doc-2023-06-05-rados-operations-pools-2-of-x

doc/rados: edit pools.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51932 from stackhpc/typo-fix

doc/rados/operations: fix typo in balancer.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit pools.rst (2 of x)

Edit doc/operations/rados/pools.rst.

There remains confusion in this part of the document regarding pg_num
and pgp_num. pg_num and pgp_num are not explained with sufficient
clarity. A future commit will clear up this confusion. There is also
some potential confusion between on the one hand the strings "pg-num"
and "pgp-num" and on the other hand "pg_num" and "pgp_num". The strings
with the hyphens are used in dummy commands, and the strings with the
underscores are used as key names. I think it possible that this could
confuse a reader, but I am open to discussion on the matter.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados/operations: fix typo in balancer.rst

Signed-off-by: Pierre Riteau <pierre@stackhpc.com>

Merge pull request #50341 from yangdongsheng/ubbd

rbd: add support for new device type of ubbd

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED for rados cls tests

Fixes: https://tracker.ceph.com/issues/59192
Signed-off-by: Laura Flores <lflores@redhat.com>

qa/suites/rados: remove rook coverage from the rados suite

The rook team relies on a daily CI system to validate
rook changes. It doesn't seem that the teuthology tests
are maintained, so it makes sense to remove them from the
rados suite.

By removing this symlink, rook test coverage will remain
in the orch suite, and coverage will only be removed from the
rados suite.

Workaround for: https://tracker.ceph.com/issues/58585
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #51294 from cfsnyder/wip-59588

mgr/dashboard: add 'omit_usage' query param to dashboard api 'get rbd' endpoint

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

cmake/rgw: include cpp_redis as system headers

this hides a lot of c++20-related warnings

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: replace WITH_CPP_REDIS with WITH_RADOSGW_D4N_FILTER

Signed-off-by: Casey Bodley <cbodley@redhat.com>

point cpp_redis submodule to ceph/cpp_redis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

RGW: Add D4N classes and unit testing; update cpp_redis submodule

Signed-off-by: samarah <suriarte@redhat.com>

rgw: Add cpp-redis submodule to ceph

Signed-off-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Arjun Sharma <arjsharm@redhat.com>

Merge pull request #51729 from NitzanMordhai/wip-nitzan-striper-api-aio-futex-wait

test: futex fail if more notification sent after destroy

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #51706 from Wangwenjuan1105/wwj_clist_lat

Bluestore: fix bluestore collection_list latency perf counter

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #51668 from YiteGu/fix-TrackedOp-event-order

osd/TrackedOp: Fix TrackedOp event order

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #51424 from NitzanMordhai/wip-nitzan-monclient-reconnect-finish-auth-with-error

mon/MonClient: before complete auth with error, reopen session

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #50651 from rosinL/cleanup

Cleanup the LevelDB residue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

[CHERRY-PICKED FOR TESTING ONLY] crimson: Enable tcmalloc when using seastar

classic-osds have always caused significant memory fragmentation
when using the libc memory allocator due to the way that Ceph
tends to utilize memory.  In recent testing, crimson-osd was found
to use 25-27GB of RAM with the stock 3GB bluestore cache settings
(osd_memory_target is only used when tcmalloc is available).  Upon
further testing, it was found that the classic OSD is even worse,
using between 32-33GB of RAM after a 5 minute 4K sequential
write test when using libc malloc.

The good news is that it appears that crimson-osd is able to use
tcmalloc for alienstore without significant modification. Better
still, it drastically reduces memory usage.  In the same test that
resulted in 25GB RSS memory usage for crimson-osd with libc malloc,
a tcmalloc linked version took around 9GB (with an 8GB
osd_memory_target).  Since we do not yet (afaik) expose classic OSD
debugging in crimson it is tough to tell why we are still a little
over, but it's clear that for alienstore we are going to need to
use tcmalloc as we do in classic.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 9f31ad7b7388996307808e26722260aed29d4ea1)

test/crimson: embed lsan_suppressions into tests executables

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51915 from zdover23/wip-doc-2023-06-05-rados-operations-pools-typo

doc/rados: pools.rst: "decreaesed"

mgr/dashboard: invalidate rbd image cache on CRUD ops

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Fixes: https://tracker.ceph.com/issues/58710

Merge pull request #51746 from rosinL/fix-61227

crimson/osd/ops_executer: fix double apply_stats() when writing

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

doc/rados: pools.rst: "decreaesed"

s/decreased/decreased/

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51908 from zdover23/wip-doc-2023-06-04-rados-operations-pools-1-of-x

doc/rados: edit pools.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit pools.rst (1 of x)

Edit doc/rados/operations/pools.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/standalone: osd-recovery-scrub: fix slow updates and recovery concurrency

1. Setting frequent scrub status updates, to compensate for the removal
of some 'send updates' in PR#50283.

2. Switching back to using the wpq scheduler, as otherwise the number of
concurrent recovery operations is below what the test expects.

Fixes: https://tracker.ceph.com/issues/61386
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>