doc:dev update how to use deduplication

Add prerequisite for deduplication, example outputs, and chunk-repair operation.

Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>

Merge pull request #52064 from ajarr/wip-61688

mgr/rbd_support: add user-friendly stderr message when module is not ready

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51962 from vedanshbhartia/coverity_1511072

rgw: pass boost intrusive_ptr to DataLogBackends flat_map

Merge pull request #51676 from vedanshbhartia/coverity_move

rgw: Do not forward functor objects in rgw_iam_policy.h

Merge pull request #51965 from vedanshbhartia/coverity_1510687

rgw: Fix use after move in cls_fifo_legacy.cc

Merge pull request #50330 from mohamedawnallah/fix-RGWEnv.get()-nullptr-default-value-to-the-default-returned-value

src/rgw/rgw_env.cc: Fix potential null pointer dereference in RGWEnv::get()

mgr/rbd_support: log number of images

... that have one snapshot request pending when the
mirror_snapshot_schedule handler is shutting down.

Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #52058 from guits/tracker_61673

ceph-volume: set lvm membership for mpath type devices

Merge PR #52029 into main

* refs/pull/52029/head:
debian: install cephfs-mirror systemd unit files and man page

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #52039 from zdover23/wip-doc-2023-06-14-operations-crush-map-2-of-x

doc/rados: edit crush-map.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

debian: install cephfs-mirror systemd unit files and man page

Fixes: https://tracker.ceph.com/issues/59682
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #50862 into main

* refs/pull/50862/head:
mon/MDSMonitor: batch last_metadata update with pending
mon/MDSMonitor: add debugging for update_metadata

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/rados: edit crush-map.rst (2 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52040 from zdover23/wip-doc-2023-06-14-operations-crush-map-ordered-list-fix

doc/rados: fix list in crush-map.rst

mgr/rbd_support: add user-friendly stderr message

... when the rbd_support module is not ready.

Fixes: https://tracker.ceph.com/issues/61688
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #51946 from phlogistonjohn/jjm-cephadm-write-new

cephadm: add write_new function for robust file writes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51924 from adk3798/limit-policy-log-level

python-common/drive_selection: lower log level of limit policy message

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51861 from adk3798/spec-no-spec-validate

python-common/drive_group: handle fields outside of 'spec' even when 'spec' is provided

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #51267 from phlogistonjohn/jjm-cephadm-versioning

Support cephadm "binary" versioning

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51832 from rkachach/fix_issue_61499

cephadm: adding support for ipv6 when deploying ceph-exporter

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51817 from rkachach/fix_issue_61494

mgr/cephadm: validate host label before removing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51070 from adk3798/reconfig-firewall-ports

cephadm: allow ports to be opened in firewall during adoption, reconfig, redeploy

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

ceph-volume: set lvm membership for mpath type devices

If any lvs is present, self.lvs should be set accordingly for mpath
type devices as well.
Otherwise, when re-running a command like following:

```
ceph-volume lvm batch --bluestore --no-auto  /dev/mapper/mpatha /dev/mapper/mpathb
```

fails with error : "Device /dev/mapper/mpatha has a filesystem."

where the expected behavior is:

```
--> passed data devices: 2 physical, 0 LVM
--> relative data size: 1.0
--> All data devices are unavailable

Total OSDs: 0

  Type            Path                                                    LV Size         % of device
```

Fixes: https://tracker.ceph.com/issues/61673
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #52044 from vedanshbhartia/coverity_fix_tags

rgw: user correct tags for coverity annotations and add the SUPPRESS keyword

rgw: user correct tags for coverity annotations and add the SUPPRESS
keyword

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #52012 from cbodley/wip-61428

qa: suppress SyscallParam error during startup on jammy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc/rados: fix list in crush-map.rst

Correct the numbering and formatting of an ordered list in
doc/rados/operations/crush-map.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52014 from cbodley/wip-test-cls-refcount-rename

test/cls: rename cls_refcount tests

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52009 from zdover23/wip-doc-2023-06-12-rados-operations-placement-groups-math-markup

doc/rados: add math markup to placement-groups.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52011 from samarahu/d4n-warning-cleanup

RGW: Cleanup D4N unit test warning

Merge pull request #52028 from zdover23/wip-doc-2023-06-13-dev-deploying-dev-cluster-link

doc/dev: fix "deploying dev cluster" link

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51989 from samarahu/remove-trace-comment

RGW: Replace trace with d4n in yaml long desc

Merge pull request #52018 from zdover23/wip-doc-2023-06-13-operations-crush-map-1-of-x

doc/rados: edit crush-map.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: fix "deploying dev cluster" link

Fix a link to "Deploying a Development Cluster" that had been broken
since the transition from master to main.

This is the first part of an initiative to improve the vstart
documentation.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit crush-map.rst (1 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51966 from rhcs-dashboard/grafana-url-deprecated

mgr/dashboard: change deprecated grafana URL in daemon logs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #51953 from rhcs-dashboard/dashboard-status-popover

mgr/dashboard: add popover to cluster status card

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51945 from rzarzynski/wip-bug-61504

crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51973 from Matan-B/wip-matan-lsan-supp

qa/lsan.supp: update heap_profiler suppression and ASAN_OPTIONS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51703 from lxbsz/wip-59683

xfstests_dev: install extra packages from powertools repo for xfsprogs

Merge pull request #50728 from lxbsz/wip-59195

qa: switch to use the merge fragment for fscrypt

qa: suppress SyscallParam error during startup on jammy

Fixes: https://tracker.ceph.com/issues/61428
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51972 from vedanshbhartia/coverity_resource_leak

rgw: Add coverity annotations for resource leak false positives

test/cls: rename cls_refcount tests

test cases were copy/pasted from test_cls_rgw.cc without renaming

Signed-off-by: Casey Bodley <cbodley@redhat.com>

RGW: Cleanup D4N unit test warning

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

doc/rados: add math markup to placement-groups.rst

Add some math markup to some of the mathematical expressions in
doc/rados/operations/placement-groups.rst.

This change has advantages as well as disadvantages. On the one hand,
the mathematical expressions look nicer and are easier to read.  On the
other hand, the rendered mathematical expressions are not searchable as
a part of the text, which makes CTRL+F searches useless in finding them.

This PR is offered as an RFC on the matter of whether mathematical
expressions should be rendered with the :math: extension.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51864 from yuvalif/wip-yuval-fix-61540

rgw/kafka: make sure that destroy is called after connection is removed

xfstests_dev: install extra packages from powertools repo for xfsprogs

Centos Stream 8 has removed the 'device-mapper-devel', 'libedit-devel'
and 'userspace-rcu-devel' packages from the mirrors and we need to
install it from powertools repo.

Fixes: https://tracker.ceph.com/issues/59683
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #51994 from zdover23/wip-doc-2023-06-11-rados-operations-placement-groups-3-of-x

doc/rados: edit placement-groups.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit placement-groups.rst (3 of x)

Edit doc/rados/operations/placement-groups.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51366 from SMIL-Infra/fix-test-build

rgw: fix tests build w/o AMQP

Merge pull request #51468 from vedanshbhartia/coverity_time

rgw: Add coverity annotations to ignore warnings about 32 bit time

Merge pull request #51992 from zdover23/wip-doc-2023-06-10-rados-operations-change-mon-elections

doc/rados: edit change-mon-elections

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51991 from zdover23/wip-doc-2023-06-09-rados-operations-placement-groups-2-of-x

doc/rados: edit placement-groups.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit placement-groups.rst (2 of x)

Edit doc/rados/operations/placement-groups.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit change-mon-elections

Edit doc/rados/operations/change-mon-elections.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51988 from ljflores/wip-release-checklist

doc/dev: update release checklist

crimson/osd: suppress the assertion in Watch::notify_ack

Replaced the assert with an error log entry but ultimately
this commit should be reverted.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Fixes: https://tracker.ceph.com/issues/61504
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49316 from BryceCao/wip-add-initialization-for-RGWGC-process-call

rgw : fix add initialization for RGWGC::process()

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #51715 from awojno-bloomberg/replication-race-condition-fix

RGW: Pass and check destination zone trace when fetching object

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

RGW: Replace trace with d4n in yaml long desc

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

doc/dev: update release checklist

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #51975 from zdover23/wip-doc-2023-06-09-rados-operations-placement-groups-1-of-x

doc/rados: edit placement-groups.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit placement-groups.rst (1 of x)

Edit doc/rados/operations/placement-groups.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #48879 from samarahu/wip-d4n

RGW: WIP D4N

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

qa/lsan.supp: update heap_profiler suppression and ASAN_OPTIONS

In continuation to: 8c099a534044bf7182e04f250e342aab76bc3e54

Regsiter singleton leak seems to be replaced by:
```
-----------------------------------------------------
Suppressions used:
  count      bytes template
      1          8 ^InitModule
-----------------------------------------------------
```

In similarity to Crimson's suppression: 6ed8d839b421442a64410444ca8f88f157ae28b3

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51967 from Dazul/add-config-missing-custom-container-images

doc: Add missing custom container_image

Merge pull request #51958 from zdover23/wip-doc-2023-06-07-rados-operations-pools-4-of-x

doc/rados: edit pools.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

mgr/dashboard: change deprecated grafana URL in daemon logs

See https://github.com/grafana/grafana/issues/51276

I am also increasting the height of the panel

Fixes: https://tracker.ceph.com/issues/61618
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51955 from rhcs-dashboard/teuth-e2e-fix

qa/workunits: fix test_dashboard_e2e.sh: no spec files found

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51911 from ronen-fr/wip-rf-recovery-updates

qa/standalone: osd-recovery-scrub: fix slow updates and recovery concurrency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

doc: Add missing custom container_image

While testing cephadm in order to migrate to it in the near
futur, I needed to use some custom images for haproxy
and keepalived. I needed to see into the code to see what
existed. So I put what I found on the code into documentation.

Signed-off-by: Luis Domingues <domingues.luis@protonmail.ch>

rgw: Fix use after move in cls_fifo_legacy.cc

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #51948 from xxhdx1985126/wip-61608

crimson/os/seastore/cached_extent: add prepare_commit interface

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #51917 from sseshasa/wip-mclock-cost-fix-relnotes

PendingReleaseNotes: Document mClock scheduler fixes and enhancements

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: Add coverity annotations for resource leak false positives

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

rgw: pass boost intrusive_ptr to DataLogBackends flat_map
When emplacing a new backend, use an explicitly constructed
intrusive_ptr so that it gets destroyed if the emplace fails.

Though the destructor is already called, this behaviour cannot be relied
upon. std::map does not exhibit the same behaviour, and would have
leaked memory had it been used instead.

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #51892 from rzarzynski/wip-bug-61547

mon: fix timestamp formatting in cluster log

Reviewed-by: Laura Flores <lflores@redhat.com>

mgr/dashboard: add popover to cluster status card

Previous dashboard has a popover for the status when there are some
warnings in the Cluster Health. Introducing the same behaviour on the
new landing page

Fixes: https://tracker.ceph.com/issues/61611
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/rados: edit pools.rst (4 of x)

Edit doc/operations/rados/pools.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51930 from zdover23/wip-doc-2023-06-06-rados-operations-pools-3-of-x

doc/rados: edit pools.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46062 from markhpc/wip-crimson-tcmalloc

crimson: Enable tcmalloc when using seastar

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/workunits: fix test_dashboard_e2e.sh: no spec files found

Fixes: https://tracker.ceph.com/issues/61578
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50054 from rhcs-dashboard/cache-invalidation

mgr/dashboard: RBD cache invalidation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

PendingReleaseNotes: Document mClock scheduler fixes and enhancements

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

crimson/os/seastore/cached_extent: add prepare_commit interface

move new fixedkv-btree nodes' inter-node pointer adjustments into
the prepare_commit interface

Fixes: https://tracker.ceph.com/issues/61608
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #51927 from ljflores/wip-rook-tests

qa/suites/rados: remove rook coverage from the rados suite

cephadm: use DEFAULT_MODE constant at additional locations

Now that we have a DEFAULT_MODE constant we can replace other locations
where 0o600 is used with the constant.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: use 0o600 as the default mode for write_new

Add a constant DEFAULT_MODE of `0o600`, and make it the default of
the perms argument to write_new. This reduces a lot of code since
0o600 is the majority of the permissions used. Other cases can continue
to pass None to indicate no particular permissions are desired.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: convert SNMPGateway create_daemon_conf to use write_new

While it is not entirely clear why this pattern of using os.open and
posix open flags instead of `open` directly was used I determined (using
strace) that the only major difference between these open flags and
those used by `open` was the lack of O_TRUNC. Unlike some other cases
this function does not use an intermediate temporary file.  This means
that if the file being written already exists and the data being written
is smaller then the remaining data will not be over-written.

Example:
```
$ cat existing
AAAAAAAA
$ cat existing
bbbAAAAA
```

I looked over the context that this function is used in and decided that
this behavior must not be intentional. Thus it should be safe
to convert this function to `write_new`.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: convert more temporary file writes to use write_new

Some functions are using the pattern:
```
with open(os.open(name + '.new, os.O_CREAT | os.O_WRONLY, 0o600), 'w') as f:
    f.write(...)
    os.rename(name + '.new', name)
```
While it is not entirely clear why this pattern was first used,
it accomplishes the same goal as `write_new` only directly calling
the posix open call. I analyzed the open flags for `write_new` and
these calls using `strace` and noted that the only significant
difference was the lack of O_TRUNC in these cases. Since the ".new"
files should not exist the lack of O_TRUC ought not make any difference.
With this decided we can convert these instances to `write_new`.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: convert _write_custom_conf_files to use write_new

We double checked the meaning of "w+" and it will open the file
read-write. Since the file is never read there's no real reason
to keep it that way so its OK to convert to `write_new`.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: convert some functions to use write_new

Convert a lot of the basic uses of the pattern:
with open(...) as f:
  f.write(...)
  os.fchown(f, ...)  # sometimes
  os.fchmod(f, ...)  # sometimes
  os.rename(...)   # sometimes

These are the most obvious cases to convert to `write_new`
and should largely be uncontroversial.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: create functional mock for fchown

The pyfakefs library apparently doesn't have its own mock for os.fchown.
This means that code using fchown currently calls into a mock with
no affect on the fake fs. For some reason I don't fully understand,
existing test cases work because they don't always follow the pattern
of open-write-rename. Switching to `write_new`, which always does a
rename, breaks some of the assertions performed in the tests on the fake
fs. Add a mock fchown that updates the state of the fake fs so
that converting call sites to use `write_new` will continue to work.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add unit tests for write_new

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: add write_new function for robust file writes

The cephadm code has a very common pattern made of at least one of
the three following steps:
* call fchown on the open file to set ownership
* call fchmod on the open file to set permissions
* rename the file from a temp name to final name

Add the write_new function to encapsulate these common actions.
If owner is not None then fchown will be called.
If perms is not None then fchmod will be called.
An optional encoding value may be passed.
It always uses a temporary file as a temporary file ensures that
there can never be a partially written file even in the event of
a power outage or system crash.
Encapsulating this all into a function also allows us to make
changes to this approach in the future without touching every
call site using `open(..., "w")` etc.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

QA: Add D4N teuthology suite

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

Merge pull request #51943 from dvanders/update-contact-dan

Update affiliation for Dan van der Ster

Merge pull request #51925 from ljflores/wip-tracker-59192

qa/suites/rados: whitelist POOL_APP_NOT_ENABLED for cls tests