install-deps.sh: do not install libpmem from chacra

this change reverts 17d2bc3707bb0078e2fa1b4eef31b39804e45135, before
we recreate a chacra repo hosting libpmem packages, we are not able
to query the repo from shaman or pull the dependencies from chacra.

in future, we should be able to get the libpmem dependencies from
offical ubuntu package repo and fedora, CentOS Stream and RHEL repos.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit fbfb369722fe9f92ad333f8368095637c36c10ec)

Merge pull request #46807 from aaSharma14/wip-56156-quincy

quincy: mgr/dashboard: Add daemon logs tab to Logs component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #46718 from rhcs-dashboard/wip-56076-quincy

quincy: mgr/dashboard: bump moment from 2.29.1 to 2.29.3 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #46528 from rhcs-dashboard/wip-54624-quincy

quincy: mgr/dashboard: BDD approach for the dashboard cephadm e2e

Reviewed-by: Sarthak Gupta <sarthak.dev.0702@gmail.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #46672 from rhcs-dashboard/wip-56037-quincy

quincy: mgr/dashboard: stop polling when page is not visible

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #46789 from adk3798/wip-55949-quincy

quincy: mgr/cephadm: check if a service exists before trying to restart it

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46793 from adk3798/wip-56044-quincy

quincy: mgr/cephadm: try to get FQDN for active instance

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #46728 from guits/wip-56085-quincy

quincy: ceph-volume: do not log sensitive details

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46730 from guits/wip-56039-quincy

quincy: ceph-volume: make is_valid() optional

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46794 from adk3798/wip-56069-quincy

quincy: mgr/nfs: validate virtual_ip parameter

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46809 from rhcs-dashboard/wip-56163-quincy

quincy: mgr/dashboard: dashboard help command showing wrong syntax for login-banner

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #46818 from zdover23/wip-doc-2022-06-23-backport-46769-quincy-dev-guide-command-context

quincy: doc/dev: add context note to dev guide config

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46070 from guits/wip-55480-quincy

quincy: ceph-volume: only warn when config file isn't found

ceph-volume: make is_valid() optional

There are cases where `ceph-volume` doesn't have to require
a `ceph.conf` file that has the `fsid` parameter.

See: https://github.com/rook/rook/pull/10333#pullrequestreview-994958873

Fixes: https://tracker.ceph.com/issues/55970
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7ffea899921dc9996f20ef88531bd1daf82d44aa)

ceph-volume: do not print the secret of osd keyring

during osd preparation, ceph-volume logs the secret of the osd keyring to file
```
[2022-06-15 12:31:17,466][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAM0aliR5JvDRAAQBu0stWl9ZhZrcjijg2BIQ==
[2022-06-15 12:31:17,481][ceph_volume.process][INFO  ] stdout creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQAM0aliR5JvDRAAQBu0stWl9ZhZrcjijg2BIQ==)
```

This shouldn't be logged nor printed on terminal.

Fixes: https://tracker.ceph.com/issues/56071
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 4b9cc6b303588e0c44443debe4f04c6160adf5a2)

ceph-volume: do not print luks key encryption

During osd activation, ceph-volume logs the luks key to its log file.

```
[2022-06-15 12:50:35,180][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.osd-lockbox.51d0770d-403d-4f81-93e6-e99f627f246c --keyring /var/lib/ceph/osd/ceph-0/lockbox.keyring config-key get dm-crypt/osd/51d0770d-403d-4f81-93e6-e99f627f246c/luks
[2022-06-15 12:50:35,522][ceph_volume.process][INFO  ] stdout ut9NjMK6YtMh1BLMJZ/mE2A7zTNyrp9pW1kHV8F2ipfz1BIX9MkEWhdYB2Azm1JPZ1d7ahIjBMUbrC/Iqqr2jQhP3MIsDzUYj1enw+sw7LeVvGPf0qNUdKmEGu5tUmvtQ+5pbk4T/9PF36kT6vCHKfNML/3fL6nnY8FDySrI4LY=
[2022-06-15 12:50:35,522][ceph_volume.process][INFO  ] Running command: /usr/sbin/cryptsetup --key-size 512 --key-file - --allow-discards luksOpen /dev/ceph-83c307d3-710b-4197-8ecd-0484e17395e3/osd-block-51d0770d-403d-4f81-93e6-e99f627f246c a9HhDO-MiYD-DtYm-SKJf-nO1d-5O3u-FmcCrd
```

Fixes: https://tracker.ceph.com/issues/56066
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 0d97a93faae431f1197d72ee3c4347387f6b1c73)

mgr/dashboard: save mgr logs inside a folder on kcli e2e

Save the mgr container logs of cephadm inside a folder and later on
archive it and get it as an artifact on the cephadm dashboard e2e jobs

Fixes: https://tracker.ceph.com/issues/55247
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit d74defbbb3282f056e26619ecd65fc63216f2f59)

Merge pull request #46551 from guits/bkp-quincy-46481

quincy: backport of cephadm: fix osd adoption with custom cluster name

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46569 from guits/wip-55943-quincy

quincy: cephadm: add 'is_paused' field in orch status output

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #46714 from guits/wip-56041-quincy

quincy: cephadm: change default keepalived/haproxy container images

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46783 from adk3798/wip-55947-quincy

quincy: mgr/cephadm: capture exception when not able to list upgrade tags

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46791 from adk3798/wip-55988-quincy

quincy: mgr/cephadm: store device info separately from rest of host cache

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

doc/dev: add context note to dev guide config

This PR adds a note directing first-time cloners of
their Ceph git forks to make sure to cd into the ceph/
directory before trying to run the "git config" commands.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 3e6bcd7f0be2d0f896146c563a3381c9b7ea0e83)

mgr/dashboard: dashboard help command showing wrong syntax for login-banner

Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>
(cherry picked from commit 826741f192e754ed32b1636b06b9939efa5c5226)

Merge pull request #46686 from rhcs-dashboard/wip-55942-quincy

quincy: mgr/dashboard: Feature 54330 osd creation workflow

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Add daemon logs tab to Logs component

Add Daemon logs to the logs component

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit a581660f47d85da0e985c5bcba6e9a6cd6b47dae)

mgr/nfs: validate virtual_ip parameter
Fixes: https://tracker.ceph.com/issues/54581
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 75aafdaf367cd3624cb33ac5a99a73a27a9e3f52)

mgr/cephadm: try to get FQDN for active instance

Fixes: https://tracker.ceph.com/issues/55674
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit d0385e030b391f588b4ec0dc707d5d46778a2aaa)

mgr/cephadm: store device info separately from rest of host cache

device info tends to take up the most space out of
everything, so the hope is by giving it its own
location in the config key store we can avoid hitting
issues where the host cache value we attempt to
place in the config key store exceeds the size limit

Fixes: https://tracker.ceph.com/issues/54251
Fixes: https://tracker.ceph.com/issues/53624
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit e35d4144d380cef190a04517b4d7b30d520d5b4f)

mgr/cephadm: check if a service exists before trying to restart it
Fixes: https://tracker.ceph.com/issues/55800
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 6b76753c3cabf9663fa1daa47c7bcb7df110a94c)

Merge pull request #46361 from adk3798/build-wip-quincy

quincy: script/build-integration-branch: add quincy to the list of releases

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/cephadm: capture exception when not able to list upgrade tags
Fixes: https://tracker.ceph.com/issues/55801
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 0e7a4366c0c1edd74d52acad5ed4dc3df0ef7679)

Merge pull request #46763 from idryomov/wip-wnbd-main-branch-quincy

quincy: win32_deps_build.sh: master -> main for wnbd

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

win32_deps_build.sh: master -> main for wnbd

wnbd is switching, see https://github.com/cloudbase/wnbd/pull/67.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 27048bf2b29a7060958a2b6dda8247cfad8a0334)

Merge pull request #46706 from rzarzynski/wip-pglog-trim-dups-quincy

quincy: tools: ceph-objectstore-tool is able to trim pg log dups' entries.

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46719 from dang/wip-dang-56029

RGW - Swift retarget needs bucket set on object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46721 from zdover23/wip-doc-2022-06-15-backport-46712-quincy-master-to-main-dev-guide-basic-workflow

quincy: doc/dev: s/master/main/ in title

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: s/master/main/ in title

This changes "master" to "main" in a title. If we lived in an
ideal world, this would have been a part of PR#46678.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit f5fd158bea2a54b83c7f75cd806885de44045ad6)

RGW - Swift retarget needs bucket set on object

When a bucket list is retargeted to an object get, then the object may
not yet have it's bucket set.  Ensure the bucket is set on the object.

Fixes: https://tracker.ceph.com/issues/56029
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

mgr/dashboard: bump moment in /src/pybind/mgr/dashboard/frontend

Bumps [moment](https://github.com/moment/moment) from 2.29.1 to 2.29.3.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/2.29.3/CHANGELOG.md)
- [Commits](https://github.com/moment/moment/compare/2.29.1...2.29.3)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit 39b3f4dc6da1dd58edb40b9c20795e88baac19d1)

cephadm/tests: decrease number of docker.io occurances

with haproxy/keepalived default images change, this check needs to be updated.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 409105d7f06e86e5d220857b861ffdaa0ded44dc)

cephadm: change default keepalived/haproxy container img

given that we now host haproxy and keepalived container images at
`quay.io/ceph` (keepalived and haproxy) let's use them as
defaults in cephadm.

Fixes: https://tracker.ceph.com/issues/54287
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit ae22c330ae4b701b4742a73a49db217823f50250)

containers: add keepalived container image

we need to maintain a fork of keepalived container image in order
to push it to quay.io so we can get around the docker rate limitation

Fixes: https://tracker.ceph.com/issues/54287
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 2e11c6dfc3a010abc4a3d595b347ed1bfa2a3518)

Merge pull request #46709 from zdover23/wip-doc-2022-06-15-backport-quincy-46705-master-to-main-dev-guide-merging

quincy: doc/dev_guide: s/master/main in merging.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev_guide: s/master/main in merging.rst

This changes the branch name "master" to the branch name
"main" in merging.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 52da71f0ab0a4606a73be749b3d983969bd34972)

Merge pull request #46703 from zdover23/wip-doc-2022-06-15-backport-quincy-46678-master-to-main-dev-guide-basic-workflow

quincy: doc/dev: s/master/main/ in basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

tools: ceph-objectstore-tool is able to trim pg log dups' entries.

The main assumption is trimming just dups doesn't need any update
to the corresponding pg_info_t.

Testing:

1. cluster without the autoscaler
```
rzarz@ubulap:~/dev/ceph/build$ MON=1 MGR=1 OSD=3 MGR=1 MDS=0 ../src/vstart.sh -l -b -n -o "osd_pg_log_dups_tracked=3000000" -o "osd_pool_default_pg_autoscale_mode=off"
```

2. 8 PGs in the testing pool.
```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph osd pool create test-pool 8 8
```

3. Provisioning dups with rados bench
```
bin/rados bench -p test-pool 300 write -b 4096  --no-cleanup
...
Total time run:         300.034
Total writes made:      103413
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.34637
Stddev Bandwidth:       0.589071
Max bandwidth (MB/sec): 2.4375
Min bandwidth (MB/sec): 0.902344
Average IOPS:           344
Stddev IOPS:            150.802
Max IOPS:               624
Min IOPS:               231
Average Latency(s):     0.0464151
Stddev Latency(s):      0.0183627
Max latency(s):         0.0928424
Min latency(s):         0.0131932
```

4. Killing osd.0
```
rzarz@ubulap:~/dev/ceph/build$ kill 2572129 # pid of osd.0
```

5. Listing PGs on osd.0 and calculating number of pg log's entries and
dups:

```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph-objectstore-tool --data-path dev/osd0 --op list-pgs --pgid 2.c > osd0_pgs.txt
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
3100
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

6. Trimming dups
```
rzarz@ubulap:~/dev/ceph/build$ CEPH_ARGS="--osd_pg_log_dups_tracked 2500 --osd_pg_log_trim_max=100" bin/ceph-objectstore-tool --data-path dev/osd0 --op trim-pg-log-dups --pgid 2.7
max_dup_entries=2500 max_chunk_size=100
Removing keys dup_0000000020.00000000000000000001 - dup_0000000020.00000000000000000100
Removing keys dup_0000000020.00000000000000000101 - dup_0000000020.00000000000000000200
Removing keys dup_0000000020.00000000000000000201 - dup_0000000020.00000000000000000300
Removing keys dup_0000000020.00000000000000000301 - dup_0000000020.00000000000000000400
Removing keys dup_0000000020.00000000000000000401 - dup_0000000020.00000000000000000500
Removing keys dup_0000000020.00000000000000000501 - dup_0000000020.00000000000000000600
Finished trimming, now compacting...
Finished trimming pg log dups
```

7. Checking number of pg log's entries and dups
```
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
2500
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit a2190f901abf2fed20c65e59f53b38c10545cb5a)

Merge pull request #46661 from zdover23/wip-doc-2022-06-13-backport-quincy-46651-master-to-main

quincy: doc/dev: s/master/main/ essentials.rst dev guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46665 from zdover23/wip-doc-2022-06-13-backport-quincy-46659-hardware-recs-polish

quincy: doc/start: Polish network section of hardware-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: s/master/main/ in basic workflow

This PR changes "master" to "main" in the
basic_workflow.rst file. I have even changed
"master" to "main" in some terminal output from
several years ago. This isn't historically ac-
curate, of course, but my hope is that this change
will prevent someone in the future from being con-
fused about why an antiquated branch name is ref-
erred to.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit f5cfc22445a3e454e572873cdcfbc1852a38aa17)

mgr/dashboard: test througput deployment option

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit d58b9499208cd611897518009495f9e80515349d)

mgr/dashboard: throughput optimized option enabled

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit f2474bcb767893dc750b8f1231b4583925f9bfb1)

mgr/dashboard: OSD Creation Workflow initial works

Introducing the Cost/Capacity Optimized deployment option
Used bootstrap accordion
Adapted the e2e but not written new tests for the deployment option

Fixes: https://tracker.ceph.com/issues/54340
Fixes: https://tracker.ceph.com/issues/54563
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>
(cherry picked from commit 6c2dcb740efb793a3f6ef593793151a34c19ca01)

mgr/dashboard: retrieve disk status

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit a1d1c853a5e4ff9a317591b99b75e005ccc862c9)

mgr/dashboard: stop polling when page is not visible

Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>
(cherry picked from commit fa8a62fb8f8da3a854764e3e24d9b7abc00204e2)

mgr/dashboard: fix drain e2e failure

Cypress sometimes fail to register the click and that causes the
deselect/select to not happen properly. Deselecting the row immediately
after performing the action makes it pass from cypress.

Fixes: https://tracker.ceph.com/issues/55741
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit dcf0445153694b87ac685f4cfc4094d7d1a708da)

doc/start: Polish network section of hardware-recommendations.rst

Harmonize network throughput notation, minor tweaks to wording.
Followup to #46637

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 2eb173fef9dc9ff34078b0ae13e9fcc398cdbaa3)

doc/dev: s/master/main/ essentials.rst dev guide

This PR changes all reference to the "master" branch
to references to the "main" branch (because we renamed
"master" to main", and the docs now need to reflect that).

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 728b8f26746c733988302d52cce4c46ebc607f63)

Merge pull request #46652 from zdover23/wip-doc-2022-06-13-backport-46637-quincy-hardware-recs-network

quincy: backport doc/start: rewrite hardware-recs networks section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46655 from zdover23/wip-doc-2022-06-13-backport-46583-quincy-hardware-recs-osd-and-mds-acros

quincy: doc/start: make OSD and MDS structures parallel

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46658 from zdover23/wip-doc-2022-06-13-backport-46633-quincy-start-intro-crush-para-rewrite

quincy: doc/start: rewrite CRUSH para

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: rewrite CRUSH para

This PR supersedes https://github.com/ceph/ceph/pull/46584
and makes changes suggested by Anthony D'Atri that improve
the coherence and consistency of the paragraph that explains
the basics of the CRUSH algorithm.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit ba1a85b292fcd30020897a486cb452381c998690)

doc/start: make OSD and MDS structures parallel

This PR makes the "Ceph OSDs" and "MDSs" bullet points
parallel by naming "object storage daemon" before referring
to the (admittedly more common and colloquial, but surely
unknown to people who genuinely require a document called
'Intro') acronym "OSD".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 4f6edb92b9c4b7c47354d8ad66fdee6c06389541)

doc/start: rewrite hardware-recs networks section

This rewrites the first two-thirds of the "Networks"
section of the Hardware Recommendations page in the
Intro to Ceph document. I have tried to divide the
techincal content in this section into subsections
that foreground the various subjects covered.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 778d3c0b59148e2a3b6bb662437778692b1988e3)

Merge pull request #46577 from cbodley/wip-quincy-qa-smoke-s3tests

qa/smoke: use ceph-quincy branch of s3tests

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #46607 from rzarzynski/wip-55982-quincy

quincy: osd: log the number of 'dups' entries in a PG Log

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46605 from rzarzynski/wip-55981-quincy

quincy: revert backport of #45529

Reviewed-by: Laura Flores <lflores@redhat.com>

osd: log the number of 'dups' entries in a PG Log

We really want to have the ability to know how many
entries `PGLog::IndexedLog::dups` has inside.
The current ways are either invasive (stopping an OSD)
or indirect (examination of `dump_mempools`).

The code comes from Nitzan Mordechai (part of
ede37edd79a9d5560dfb417ec176327edfc0e4a3).

Fixes: https://tracker.ceph.com/issues/55982
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 8f1c8a7309976098644bb978d2c1095089522846)

Revert "tools/ceph_objectstore_took: Add duplicate entry trimming"

This reverts commit 5245fb33dd02e53cf0ef5d2f7be5904b6fbe63ce.

Although the chunking in off-line `dups` trimming (via COT) seems
fine, the `ceph-objectstore-tool` is a client of `trim()` of
`PGLog::IndexedLog` which means than a partial revert is not
possible without extensive changes.

The backport ticket is: https://tracker.ceph.com/issues/55981

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Revert "osd/PGLog.cc: Trim duplicates by number of entries"

This reverts commit 3ff0df6a28a1d9e197bdba40be7126fed8a14ae9
which is the in-OSD part of the fix for accumulation of `dup`
entries in a PG Log. Brainstorming it has brought questions
on the OSD's behaviour during an upgrade if there are tons of
dups in the log. What must be double-checked before bringing
it back is ensuring we chunk the deletions properly to not
impose OOMs / stalls in, to exemplify, RocksDB.

The backport ticket is: https://tracker.ceph.com/issues/55981

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/smoke: use ceph-quincy branch of s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46544 from yaarith/rook-telemetry-release-notes-quincy

quincy: PendingReleaseNotes: add a note about Rook telemetry

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46508 from nmshelke/wip-55797-quincy

quincy: mgr/volumes: set, get, list and remove metadata of snapshot

Reviewed-by: Venky Shankar <vshankar@redhat.com>

cephadm: add 'is_paused' field in orch status output

it can be convenient for automation tool if `orch status` output reports
whether the upgrade is currently paused.

Fixes: https://tracker.ceph.com/issues/55843
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit de1c428dfe9744b0585a42f1862b4f9ee46c22b2)

Merge pull request #46449 from zdover23/wip-doc-2022-05-31-backport-46430-quincy

quincy: doc/start: update "memory" in hardware-recs.rst

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #46492 from kalebskeithley/wip-55810-quincy

quincy: rocksdb: build with rocksdb-7.y.z

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46497 from lxbsz/wip-55658

quincy: mds: trigger to flush the mdlog in handle_find_ino()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46496 from lxbsz/wip-55661

quincy: qa: add filesystem/file sync stuck test support

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46476 from lxbsz/wip-55447

quincy: client: add option to disable collecting and sending metrics

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46175 from cfsnyder/wip-55441-quincy

quincy: os/bluestore: set upper and lower bounds on rocksdb omap iterators

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: fix osd adoption with custom cluster name

When adopting Ceph OSD containers from a Ceph cluster with a custom name, it fails
because the name isn't propagated in unit.run.
The idea here is to change the lvm metadata and enforce 'ceph.cluster_name=ceph'
given that cephadm doesn't support custom names anyway.

Fixes: https://tracker.ceph.com/issues/55654
Signed-off-by: Adam King <adking@redhat.com>
Co-authored-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e720a658d6a1582c0497bdf709ef4bd26bb5bb73)

Merge pull request #46542 from idryomov/wip-rbd-codeowners-quincy

quincy: CODEOWNERS: add RBD team

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

PendingReleaseNotes: add a note about Rook telemetry

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
(cherry picked from commit 1742d1b36555684aa7a75cbaa9f128131af9b6a7)

Conflicts:
Needed to reorder notes to appear in the 17.2.1 section.

CODEOWNERS: add RBD team

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 00a44f1c6b3c5270f1c9d75cf6dcac3f0d470fa9)

Merge pull request #46519 from neha-ojha/wip-46415-quincy

quincy: .github/CODEOWNERS: tag core devs on core PRs

Reviewed-by: Laura Flores lflores@redhat.com

Merge pull request #46469 from gregsfortytwo/wip-55746-quincy

quincy: Implement CIDR blocklisting

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46468 from ljflores/wip-55811-quincy

quincy: os/bluestore: turn `bluestore zero block detection` off by default

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #46418 from ronen-fr/wip-rf-46320-quincy

quincy: osd/scrub: restart snap trimming after a failed scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: fix 01-hosts.e2e test error

Its a regression introduced by BDD testing
https://github.com/ceph/ceph/pull/44835/commits/79f8545deba48a2bac6816caa7281e2198f8e5b8

Fixes: https://tracker.ceph.com/issues/54990
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 4a34762330e0ae23c6cc86a8b04c10c6cd4db7f1)

mgr/dashboard: BDD approach for the dashboard cephadm e2e

Files under the directory cypress/integration/common/* will contain
common specs which can be used on all the .feature files. We can change
the common directory to cypress/integration/* from the package.json, but
if we do that now then we'll need to take care of all the absolute
import in that path. So for now at least that's not a good choice.

The bug in the cypress-browserify-preprocessor which doesn't allow to
take our tsconfig.json file forced me to go with relative imports rather
than the absolute import. We'll need to wait for this to be fixed before
 changing all our tests to BDD.

Fixes: https://tracker.ceph.com/issues/54224
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 79f8545deba48a2bac6816caa7281e2198f8e5b8)

 Conflicts:
src/pybind/mgr/dashboard/frontend/package-lock.json
  - Regenerated new package-lock.json
src/pybind/mgr/dashboard/frontend/package.json
  - Only added the cypress-browserify dependency

.github/CODEOWNERS: tag core devs on core PRs

Start with everything that is present under core in .github/labeler.yml.

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 8303c6b911154ee936adb46e7c3491b174d22df8)

Merge pull request #46486 from yaarith/wip-55816-quincy

quincy: mgr/telemetry: add Rook data

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46453 from rhcs-dashboard/wip-55115-quincy

quincy: mgr/dashboard:  don't log 3xx as errors

Reviewed-by: nSedrickm <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

qa: set, get, list and remove custom metadata for snapshot

Following test are added:
1. Set custom metadata for subvolume snapshot.
2. Set custom metadata for subvolume snapshot(Idempotency).
3. Get custom metadata for specified key.
4. Get custom metadata if specified key not exist (Expecting error ENOENT).
5. Get custom metadata if no any key-value is added means section not exist (Expecting error ENOENT).
6. Update value for existing key in custom metadata.
7. List custom metadata of subvolume snapshot.
8. List custom metadata of subvolume snapshot if no any key-value is added (Expect empty json/dictionary)
9. Remove custom metadata for specified key.
10. Remove custom metadata if specified key not exist (Expecting error ENOENT).
11. Remove custom metadata if no any key-value is added means section not exist (Expecting error ENOENT).
12. Remove custom metadata with --force option.
13. Remove custom metadata with --force option if specified key not exist (Expecting command to succeed because of '--force' option)
14. Remove subvolume snapshot and verify whether metadata for snapshot is removed or not

Fixes: https://tracker.ceph.com/issues/55401
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
(cherry picked from commit 6fd28cc9d67b96ba87f0dffbf41d626229e904e3)

docs: set, get, list and remove custom metadata for snapshot

Set custom metadata on the snapshot as a key-value pair using
    $ ceph fs subvolume snapshot metadata set <vol_name> <subvol_name> <snap_name> <key_name> <value> [--group_name <subvol_group_name>]
    note: If the key_name already exists then the old value will get replaced by the new value.
    note: The key_name and value should be a string of ASCII characters (as specified in python's string.printable). The key_name is case-insensitive and always stored in lower case.
    note: Custom metadata on a snapshots is not preserved when snapshotting the subvolume, and hence, is also not preserved when cloning the subvolume snapshot.

Get custom metadata set on the snapshot using the metadata key::
    $ ceph fs subvolume snapshot metadata get <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>]

List custom metadata (key-value pairs) set on the snapshot using::
    $ ceph fs subvolume snapshot metadata ls <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]

Remove custom metadata set on the snapshot using the metadata key::
    $ ceph fs subvolume snapshot metadata rm <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>] [--force]
    Using the '--force' flag allows the command to succeed that would otherwise fail if the metadata key did not exist.

Fixes: https://tracker.ceph.com/issues/55401
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
(cherry picked from commit 59a0cbc14bf2832080e983729de5c462ddc70bb3)

mgr/volumes: set, get, list and remove custom metadata for snapshot

If CephFS in ODF configured in external mode, user like to use
subvolume snapshot metadata to store some Openshift specific
information, as the PVC/PV/namespace the subvolumes/snapshot
are coming from. For RBD volumes, it's possible to add metadata
information to the images using the 'rbd image-meta' command.
However, this feature is not available for CephFS volumes.
We'd like to request this capability.

Adding following commands:
    ceph fs subvolume snapshot metadata set <vol_name> <sub_name> <snap_name> <key_name> <value> [<group_name>]
    ceph fs subvolume snapshot metadata get <vol_name> <sub_name> <snap_name> <key_name> [<group_name>]
    ceph fs subvolume snapshot metadata ls <vol_name> <sub_name> <snap_name> [<group_name>]
    ceph fs subvolume snapshot metadata rm <vol_name> <sub_name> <snap_name> <key_name> [<group_name>] [--force]

Fixes: https://tracker.ceph.com/issues/55401
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
(cherry picked from commit 559222cfe8d552cd2d7aef7361de4140820ae74a)

Merge pull request #46503 from rhcs-dashboard/wip-55831-quincy

quincy: qa: fix teuthology master branch ref

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>

qa: fix teuthology master branch ref

Fixes: https://tracker.ceph.com/issues/55826
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit e91773df68c286266a2855e69bf542b4c73379d9)

mds: trigger to flush the mdlog in handle_find_ino()

If the the CInode was just created by using openc in current
auth MDS, but the client just sends a getattr request to another
replica MDS. Then here it will make a path of '#INODE-NUMBER'
only because the CInode hasn't been linked yet, and the replica
MDS will keep retrying until the auth MDS flushes the mdlog and
the C_MDS_openc_finish and link_primary_inode are called at most
5 seconds later.

Fixes: https://tracker.ceph.com/issues/55240
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 5d6dd5d1acf94eade4a2c0f48777c95473d454ae)

qa: add file/filesystem sync crash test case

This is one test case for the possible kernel crash bug when doing
the file sync or filesystem sync.

Fixes: https://tracker.ceph.com/issues/55329
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit cd3e903b0c525c7946794b5fbf002da482b3b4bc)