Merge pull request #46631 from rzarzynski/wip-pglog-trim-dups-pacific

pacific: tools: ceph-objectstore-tool is able to trim solely pg log dups' entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #46596 from kotreshhr/wip-55927-pacific

pacific: client/fuse: Fix directory DACs overriding for root

Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #46567 from vshankar/tr-55932

pacific: mds: do not assert early on when issuing client leases

Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #45293 from joscollin/wip-54479-pacific

pacific: mgr/stats: be resilient to offline MDS rank-0

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #46608 from rzarzynski/wip-55982-pacific

pacific: osd: log the number of 'dups' entries in a PG Log

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #46690 from rhcs-dashboard/wip-55941-pacific

pacific: mgr/dashboard: Feature 54330 osd creation workflow

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46691 from sseshasa/wip-56059-pacific

pacific: mon/OSDMonitor: Ensure kvmon() is writeable before handling "osd new" cmd

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #46722 from zdover23/wip-doc-2022-06-15-backport-46712-pacific-master-to-main-dev-guide-basic-workflow

pacific: doc/dev: s/master/main/ in title

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: s/master/main/ in title

This changes "master" to "main" in a title. If we lived in an
ideal world, this would have been a part of PR#46678.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit f5fd158bea2a54b83c7f75cd806885de44045ad6)

Merge pull request #45928 from kotreshhr/wip-55349-pacific

pacific: mgr/volumes: Show clone failure reason in clone status command

Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #45961 from vshankar/tr-55375

pacific: mgr/volumes: set, get, list and remove metadata of subvolume

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #46710 from zdover23/wip-doc-2022-06-15-backport-pacific-46705-master-to-main-dev-guide-merging

pacific: doc/dev_guide: s/master/main in merging.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev_guide: s/master/main in merging.rst

This changes the branch name "master" to the branch name
"main" in merging.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 52da71f0ab0a4606a73be749b3d983969bd34972)

Merge pull request #46704 from zdover23/wip-doc-2022-06-15-backport-pacific-46678-master-to-main-dev-guide-basic-workflow

pacific: doc/dev: s/master/main/ in basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46662 from zdover23/wip-doc-2022-06-13-backport-pacific-46651-master-to-main

pacific: doc/dev: s/master/main/ essentials.rst dev guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46663 from zdover23/wip-doc-2022-06-13-backport-pacific-46659-hardware-recs-polish

pacific: doc/start: Polish network section of hardware-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: s/master/main/ in basic workflow

This PR changes "master" to "main" in the
basic_workflow.rst file. I have even changed
"master" to "main" in some terminal output from
several years ago. This isn't historically ac-
curate, of course, but my hope is that this change
will prevent someone in the future from being con-
fused about why an antiquated branch name is ref-
erred to.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit f5cfc22445a3e454e572873cdcfbc1852a38aa17)

Merge pull request #46039 from cbodley/wip-55456

pacific: rgw: OpsLogFile::stop() signals under mutex

Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #45410 from cfsnyder/wip-54537-pacific

pacific: rgw: fix segfault in OpsLogRados::log when realm is reloaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46112 from Matan-B/wip-55514-pacific

pacific: mount/conf: Fix IPv6 parsing

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45893 from nkshirsagar/wip-55296-pacific

pacific: Catch exception if thrown by __generate_command_map()

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45847 from dvanders/wip-55250-pacific

pacific: rgw_reshard: drop olh entries with empty name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45486 from cfsnyder/wip-53868-pacific

pacific: rgw: return OK on consecutive complete-multipart reqs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46139 from kotreshhr/wip-55335-pacific

pacific: mgr/volumes: Fix idempotent subvolume rm

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #46138 from nmshelke/wip-55412-pacific

pacific: mds: add a perf counter to record slow replies

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #45968 from vshankar/tr-55384

pacific: mgr/snap_schedule: add time zone suffix to snapshot dir name

Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #45792 from lxbsz/wip-55192

pacific: client: buffer the truncate if we have the Fx caps

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45900 from ivancich/wip-55044-pacific

pacific: cls/rgw: rgw_dir_suggest_changes detects race with completion

Reviewed-by: casey Bodley <cbodley@redhat.com>

mon/OSDMonitor: Ensure kvmon() is writeable before handling "osd new" cmd

Before proceeding to handle "osd new" mon command as part of
OSDMonitor::prepare_command_impl(), a check is made to verify if the
authmon is writeable. Later on, prepare_command_osd_new() invokes
KVMonitor::do_osd_new() to create pending dmcrypt keys and calls
propose_pending(). The propose could fail (with an assertion failure)
if there was a prior mon command that resulted in the kvmon invoking
propose_pending().

In order to avoid such a situation, introduce a check to verify that
kvmon is also writeable in OSDMonitor::prepare_command_impl(). If it
is not writeable, the op is pushed into the wait_for_active context
queue to be retried later.

Fixes: https://tracker.ceph.com/issues/55773
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 9a0d42c9fbcf64711a791edcf9cafdba57d901d7)

mgr/dashboard: test througput deployment option

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit d58b9499208cd611897518009495f9e80515349d)

Conflicts:
src/pybind/mgr/dashboard/tests/test_osd.py

Besides test deployment options, there were extra changes that came
from main which had to be removed in favor of specific deployment
options tests.

mgr/dashboard: throughput optimized option enabled

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit f2474bcb767893dc750b8f1231b4583925f9bfb1)

mgr/dashboard: OSD Creation Workflow initial works

Introducing the Cost/Capacity Optimized deployment option
Used bootstrap accordion
Adapted the e2e but not written new tests for the deployment option

Fixes: https://tracker.ceph.com/issues/54340
Fixes: https://tracker.ceph.com/issues/54563
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>
(cherry picked from commit 6c2dcb740efb793a3f6ef593793151a34c19ca01)

mgr/dashboard: retrieve disk status

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit a1d1c853a5e4ff9a317591b99b75e005ccc862c9)

Conflicts:
src/pybind/mgr/dashboard/tests/test_osd.py

There were some extra tests that came with this commit. This was
resolved by adding test_deployment_options alone instead of all the
incoming changes.

Merge pull request #46456 from rhcs-dashboard/wip-55590-pacific

pacific: mgr/dashboard:  WDC multipath bug fixes

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: nSedrickm <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

rgw: fix segfault in OpsLogRados::log when realm is reloaded

We weren't previously handling the deallocation of the store when
a realm was reloaded. Now passing a const reference to the pointer.

Fixes: https://tracker.ceph.com/issues/54130
Signed-off-by: Cory Snyder <csnyder@iland.com>
(cherry picked from commit 0713f65355586b2f6ceeb6bbce8763158847e5ed)

Conflicts:
src/rgw/rgw_log.cc
src/rgw/rgw_log.h

Cherry-pick notes:
- conflicts due to rename of RGWRados to rgw::sal::Store after Pacific

(cherry picked from commit 44a37dc0023a69da33620dc7b6e112221f7a6a0b)

doc/start: Polish network section of hardware-recommendations.rst

Harmonize network throughput notation, minor tweaks to wording.
Followup to #46637

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 2eb173fef9dc9ff34078b0ae13e9fcc398cdbaa3)

doc/dev: s/master/main/ essentials.rst dev guide

This PR changes all reference to the "master" branch
to references to the "main" branch (because we renamed
"master" to main", and the docs now need to reflect that).

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 728b8f26746c733988302d52cce4c46ebc607f63)

Merge pull request #46653 from zdover23/wip-doc-2022-06-13-backport-46637-pacific-hardware-recs-network

pacific: doc/start: rewrite hardware-recs networks section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46656 from zdover23/wip-doc-2022-06-13-backport-46583-pacific-hardware-recs-osd-and-mds-acros

pacific: doc/start: make OSD and MDS structures parallel

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46657 from zdover23/wip-doc-2022-06-13-backport-46633-pacific-start-intro-crush-para-rewrite

pacific: doc/start: rewrite CRUSH para

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46368 from tchaikov/pacific-pr-46355

pacific: ceph.spec.in: disable annobin plugin if compile with gcc-toolset

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #46365 from tchaikov/pacific-pr-44007

pacific: cmake: check for python(\d)\.(\d+) when building boost

Reviewed-by: Dirk Mueller <dmueller@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #46312 from b-ranto/wip-alerts-pacific

[pacific] mgr/alerts: Add Message-Id and Date header to sent emails

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #46255 from tserong/wip-55616-pacific

pacific: test: No direct use of nose

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start: rewrite CRUSH para

This PR supersedes https://github.com/ceph/ceph/pull/46584
and makes changes suggested by Anthony D'Atri that improve
the coherence and consistency of the paragraph that explains
the basics of the CRUSH algorithm.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit ba1a85b292fcd30020897a486cb452381c998690)

doc/start: make OSD and MDS structures parallel

This PR makes the "Ceph OSDs" and "MDSs" bullet points
parallel by naming "object storage daemon" before referring
to the (admittedly more common and colloquial, but surely
unknown to people who genuinely require a document called
'Intro') acronym "OSD".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 4f6edb92b9c4b7c47354d8ad66fdee6c06389541)

doc/start: rewrite hardware-recs networks section

This rewrites the first two-thirds of the "Networks"
section of the Hardware Recommendations page in the
Intro to Ceph document. I have tried to divide the
techincal content in this section into subsections
that foreground the various subjects covered.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 778d3c0b59148e2a3b6bb662437778692b1988e3)

tools: ceph-objectstore-tool is able to trim pg log dups' entries.

The main assumption is trimming just dups doesn't need any update
to the corresponding pg_info_t.

Testing:

1. cluster without the autoscaler
```
rzarz@ubulap:~/dev/ceph/build$ MON=1 MGR=1 OSD=3 MGR=1 MDS=0 ../src/vstart.sh -l -b -n -o "osd_pg_log_dups_tracked=3000000" -o "osd_pool_default_pg_autoscale_mode=off"
```

2. 8 PGs in the testing pool.
```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph osd pool create test-pool 8 8
```

3. Provisioning dups with rados bench
```
bin/rados bench -p test-pool 300 write -b 4096  --no-cleanup
...
Total time run:         300.034
Total writes made:      103413
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.34637
Stddev Bandwidth:       0.589071
Max bandwidth (MB/sec): 2.4375
Min bandwidth (MB/sec): 0.902344
Average IOPS:           344
Stddev IOPS:            150.802
Max IOPS:               624
Min IOPS:               231
Average Latency(s):     0.0464151
Stddev Latency(s):      0.0183627
Max latency(s):         0.0928424
Min latency(s):         0.0131932
```

4. Killing osd.0
```
rzarz@ubulap:~/dev/ceph/build$ kill 2572129 # pid of osd.0
```

5. Listing PGs on osd.0 and calculating number of pg log's entries and
dups:

```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph-objectstore-tool --data-path dev/osd0 --op list-pgs --pgid 2.c > osd0_pgs.txt
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
3100
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

6. Trimming dups
```
rzarz@ubulap:~/dev/ceph/build$ CEPH_ARGS="--osd_pg_log_dups_tracked 2500 --osd_pg_log_trim_max=100" bin/ceph-objectstore-tool --data-path dev/osd0 --op trim-pg-log-dups --pgid 2.7
max_dup_entries=2500 max_chunk_size=100
Removing keys dup_0000000020.00000000000000000001 - dup_0000000020.00000000000000000100
Removing keys dup_0000000020.00000000000000000101 - dup_0000000020.00000000000000000200
Removing keys dup_0000000020.00000000000000000201 - dup_0000000020.00000000000000000300
Removing keys dup_0000000020.00000000000000000301 - dup_0000000020.00000000000000000400
Removing keys dup_0000000020.00000000000000000401 - dup_0000000020.00000000000000000500
Removing keys dup_0000000020.00000000000000000501 - dup_0000000020.00000000000000000600
Finished trimming, now compacting...
Finished trimming pg log dups
```

7. Checking number of pg log's entries and dups
```
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
2500
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

Conflicts:
        src/tools/ceph_objectstore_tool.cc -- undetected conflict
        with d5445b8f113797718a0dbb05e884a6bffbfed76a. Fixed by
        adopting the patch no not require the `unique_ptr<T>::get()`.

Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit a2190f901abf2fed20c65e59f53b38c10545cb5a)

Revert "tools/ceph_objectstore_took: Add duplicate entry trimming"

This reverts commit 1f3fede173cf8224f4c9f1d24b30d6fa8dfda216.

Although the chunking in off-line `dups` trimming (via COT) seems
fine, the `ceph-objectstore-tool` is a client of `trim()` of
`PGLog::IndexedLog` which means than a partial revert is not
possible without extensive changes. Moreover, trimming pg log
is not enough without modifying pg_info_t accordingly which
the reverted patch lacks.

The backport ticket is: https://tracker.ceph.com/issues/55989

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #46211 from idryomov/wip-use-system-libpmem-on-c9s-pacific

pacific: rpm: use system libpmem on Centos 9 Stream

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #45991 from idryomov/wip-make-check-enable-rbd-caches-pacific

pacific: run-make-check.sh: enable RBD persistent caches

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: pass -Wno-error when building PMDK

It's hitting pacific with a nuisance -Werror=array-parameter= const
char * vs const char[37] mismatch.  Follow commit 91a616b26e83 ("cmake:
pass RTE_DEVEL_BUILD=n when building dpdk") and just disable -Werror.

Fixes: https://tracker.ceph.com/issues/55977
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 03d27945a646124e8d8a915771ac94ed7b684366)

qa: make test_perf_stats_stale_metrics check only the clients created for the tests

Uses the client's global id to get the metrics, instead of using the index.
This ensures that test_perf_stats_stale_metrics checks only the clients mounted for
the tests.

Fixes: https://tracker.ceph.com/issues/54971
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 1621308214cd18750c8be803fc014bdf73e2218a)

qa: test `ceph fs perf stats` doesn't output stale metrics

That `ceph fs perf stats` doesn't output stale metrics
after the rank0 MDS failover.

Fixes: https://tracker.ceph.com/issues/50033
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 116e89a2f2849ed7cb711d1ae465c6f510b2810d)

Conflicts:
        qa/tasks/cephfs/test_mds_metrics.py

mgr, pybind/mgr, mgr/stats: be resilient to offline rank0 MDS

Reregister the user queries during the rank0 MDS failover event
by calling listener.handle_query_updated(). This enables
`ceph fs perf stats` to receive the updated metrics after the
failover.

Fixes: https://tracker.ceph.com/issues/50033
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit c2470f271cce4d512f2cf00552c9b753e4c69f71)

Conflicts:
       src/pybind/mgr/stats/fs/perf_stats.py

Merge pull request #46297 from dparmar18/pacific_put_get_backport

pacific: cephfs-shell: fix put and get cmd

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46183 from vshankar/tr-55427

pacific: mds: include encoded stray inode when sending dentry unlink message to replicas

Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #46180 from lxbsz/wip-55343

Pacific: mds: reset heartbeat when fetching or committing entries

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46179 from lxbsz/wip-55346

Pacific: client: stop forwarding the request when exceeding 256 times

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #46146 from neesingh-rh/wip-55539-pacific

pacific: cephfs-top: Multiple filesystem support

Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Revert "osd/PGLog.cc: Trim duplicates by number of entries"

This reverts commit d49ff13c80bbed5fb6c064f9689bb2b6a0a251b8.
which is the in-OSD part of the fix for accumulation of `dup`
entries in a PG Log. Brainstorming it has brought questions
on the OSD's behaviour during an upgrade if there are tons of
dups in the log. What must be double-checked before bringing
it back is ensuring we chunk the deletions properly to not
impose OOMs / stalls in, to exemplify, RocksDB.

The backport ticket is: https://tracker.ceph.com/issues/55989

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: log the number of 'dups' entries in a PG Log

We really want to have the ability to know how many
entries `PGLog::IndexedLog::dups` has inside.
The current ways are either invasive (stopping an OSD)
or indirect (examination of `dump_mempools`).

The code comes from Nitzan Mordechai (part of
ede37edd79a9d5560dfb417ec176327edfc0e4a3).

Fixes: https://tracker.ceph.com/issues/55982
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 8f1c8a7309976098644bb978d2c1095089522846)

Merge pull request #46470 from gregsfortytwo/wip-55747-pacific

pacific: Implement CIDR blocklisting

Reviewed-by: Venky Shankar <vshankar@redhat.com>

client/fuse: Fix directory DACs overriding for root

DACs are overridable for directories. For files,
Read/write DACs are always overridable but executable
DACs are overridable when there is at least one exec bit
set.

The files and directory DACS overriding were handled the
same way for root which is incorrect. This patch fixes
DACs overriding as described above for the root.

Fixes: https://tracker.ceph.com/issues/55313
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 2e1f43c99b1818c2ffde64f5b01083c1907a9f87)

run-make-check.sh: enable RBD persistent caches

This was attempted in commit 69a7ed4eab36 ("run-make-check: enable
WITH_RBD_RWL when WITH_PMEM is true") but never completed.  We soon
bumped the requirement on libpmem, so WITH_SYSTEM_PMDK=ON wouldn't
have worked anyway.

Enable the RWL mode conditionally based on WITH_RBD_RWL variable.
Enable the SSD mode unconditionally as it has no special dependencies
and can be built on any architecture.

Fixes: https://tracker.ceph.com/issues/55285
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0f1634a21f5da2250915d8ac05a6f179d4e76d03)

Conflicts:
run-make-check.sh [ commit 69a7ed4eab36 ("run-make-check:
  enable WITH_RBD_RWL when WITH_PMEM is true") not in pacific ]

test/encoding/check-generated.sh: show diff if binary reencode check fails

Take bf0b161115aa ("test/encoding/check-generated.sh: show diff if cmp
fails") a bit further.  Suggesting "cmp $tmp1 $tmp2" isn't very helpful
since cmp would report just the mismatch offset.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 59d928a06c028bf381307001d2b68fa8545d8fc4)

librbd/cache/pwl: WriteLogCacheEntry constructor must initialize flags

Initializing the individual bit field members leaves the remaining two
bits uninitialized and that garbage state gets persisted.

In general, using bit fields in a structure where the layout actually
matters is not desirable.  Even with a few single bits, such as here,
their order, strictly speaking, is not guaranteed:

    An implementation may allocate any addressable storage unit large
    enough to hold a bit-field. If enough space remains, a bit-field
    that immediately follows another bit-field in a structure shall be
    packed into adjacent bits of the same unit. If insufficient space
    remains, whether a bit-field that does not fit is put into the next
    unit or overlaps adjacent units is implementation-defined. The
    order of allocation of bit-fields within a unit (high-order to
    low-order or low-order to high-order) is implementation-defined.
    The alignment of the addressable storage unit is unspecified.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 91d270b210a908ea2f3578dd7db3263383da95a8)

librbd/cache/pwl: initialize generate_test_instances() objects

... to prevent check-generated.sh failures such as:

**** librbd::cache::pwl::WriteLogPoolRoot test 1 dump_json check failed ****
   ceph-dencoder type librbd::cache::pwl::WriteLogPoolRoot select_test 1 dump_json > /tmp/typ-cAoWrqlHC
   ceph-dencoder type librbd::cache::pwl::WriteLogPoolRoot select_test 1 encode decode dump_json > /tmp/typ-ES5yHpfGL
5c5
<     "flushed_sync_gen": 0,
---
>     "flushed_sync_gen": 255,

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 2c131f57d63454de39210375ce75a282df6fe365)

Conflicts:
src/librbd/cache/pwl/Types.cc [ commit 6eb14774fec0 ("librbd:
  build without "using namespace std"") not in pacific ]
src/librbd/cache/pwl/ssd/Types.h [ ditto ]

librbd/cache/pwl: fix -Wunused-lambda-capture warnings

Reported by clang on "make check" and "make check arm64" builds.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 753aa038fdbc26a2ee0978f54d3f7dcfa052e833)

Merge pull request #46556 from adk3798/pacific-fqdn-autotune

pacific: mgr/cephadm: use host shortname for osd memory autotuning

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #45698 from tserong/wip-55097-pacific

pacific: ceph.spec.in: remove build directory at end of %install

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46545 from adk3798/pacific-raw-osd-fixup

pacific: mgr/cephadm: don't redeploy osds seen in raw list if cephadm knows them

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #46552 from guits/bkp-pacific-46481

pacific: backport of cephadm: fix osd adoption with custom cluster name

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46555 from adk3798/pacific-allow-crush-class

pacific: python-common: allow crush device class to be set from osd service spec

Reviewed-by: Cory Snyder <csnyder@iland.com>

rgw: OpsLogFile drops lock before flush()

flush() acquires the same lock, so lockdep complains about recursive
locking here

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5473ae75276b3f552ec70ae308a6fe0db1fe22a5)

rgw: remove unnecessary OpsLogFile::flush_mutex

this mutex was only held by one function, OpsLogFile::flush(). this
private member function is only ever called from the background thread,
so doesn't need to be protected by a mutex

as a further cleanup, i renamed 'cond' and 'mutex' now that we don't
need to differentiate between different locks

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 279d664f8b25d2eeafac0b5d60e532dbcf56218f)

rgw: OpsLogFile::stop() signals under mutex

this shuts up ceph::debug_condition_variable's assertion that the
associated mutex is held during notify_one(). this is not strictly
required for correct use, but is a common source of bugs

Fixes: https://tracker.ceph.com/issues/55432
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cf6f99e67a6dbbc4e39d9e1fda32356c18051f5a)

Merge pull request #46429 from pdvian/wip-55309-pacific

pacific: mgr, mgr/prometheus: Fix regression with prometheus metrics

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46427 from pdvian/wip-55308-pacific

pacific: mgr, mon: Keep upto date metadata with mgr for MONs

Reviewed-by: Laura Flores <lflores@redhat.com>

mds: do not assert early on when issuing client leases

Move the relevant ceph_assert()'s when the MDS is issuing leases
to clients since clients may not be capable of reading projected
linkages.

Fixes: http://tracker.ceph.com/issues/54701
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 907b1f41e5e08afea6ee1a91ecd6d05646727b09)

mgr/cephadm: use hostname from crush map for osd memory autotuning

Fixes: https://tracker.ceph.com/issues/55841
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 50f28aa56edd348c3816335bef3bbfaf5133ae54)

client: force a request to update the perms

If the caller perms is different with the current dirtier's, we
can still cache the attributes locally if possible, but will
mark the ctime needs to be updated forcely and the request will
implicitly flush the dirty caps for us.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 569770afe52726e1df970c2219b8aa2c080e8a31)

client: improve the code of setattr for atime/mtime

When the Fw caps is issued and increasing the atime/mtime, we can
buffer the atime/mtime. And when the Fs caps is issued and they
are not changing we can just ignore them.

In case when recovering the atime/mtime maybe decreased, no matter
the Fsw caps are issued, always send the request immediatelly.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit d3d1be4c62a8358569f0fea3a9ac17475c1e173b)

client: try to buffer the truncate if we have the Fx caps

If the Fx caps is issued and new size is larger we can try to
buffer the truncate, and just ignore in case the new size equals
to current size.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit b557d642aeb21acf8b456a4addf6da63b17e1f68)

client: try to skip changing the gid/uid/mode/btime if As is issued

If the As caps is issued and the gid/uid/mode/btime don't change,
then it's safe to skip sending out a setattr request to MDS.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit ec32f0915b637b4bc6bbc6c9b2f675d058a5c02b)

python-common: allow crush device class to be set from osd service spec

Adds crush_device_class parameter to DriveGroupSpec so that device class
can be set via service specs with cephadm.

Fixes: https://tracker.ceph.com/issues/55813
Signed-off-by: Cory Snyder <csnyder@iland.com>
(cherry picked from commit c2f314ab8c187b54f12c04ec26034d451bd82273)

Conflicts:
src/python-common/ceph/deployment/drive_group.py

Merge pull request #46215 from rzarzynski/wip-tests-bl-fix-rebuild-pacific

pacific: test/bufferlist: ensure rebuild_aligned_size_and_memory() always rebuilds.

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #46120 from myoungwon/backport-50806

pacific: osd: fix wrong input when calling recover_object()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45870 from idryomov/wip-stretch-last-force-resend-pacific

pacific: mon/OSDMonitor: properly set last_force_op_resend in stretch mode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #45171 from pponnuvel/wip-53391-pacific

pacific: Fix data corruption in bluefs truncate()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #46252 from rzarzynski/wip-45529-pacific

pacific: osd/PGLog.cc: Trim duplicates by number of entries

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #46549 from MrFreezeex/wip-55652-pacific

pacific: ceph-mixin: backport of recent cleanups

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: fix osd adoption with custom cluster name

When adopting Ceph OSD containers from a Ceph cluster with a custom name, it fails
because the name isn't propagated in unit.run.
The idea here is to change the lvm metadata and enforce 'ceph.cluster_name=ceph'
given that cephadm doesn't support custom names anyway.

Fixes: https://tracker.ceph.com/issues/55654
Signed-off-by: Adam King <adking@redhat.com>
Co-authored-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e720a658d6a1582c0497bdf709ef4bd26bb5bb73)

Merge pull request #46541 from idryomov/wip-rbd-codeowners-pacific

pacific: CODEOWNERS: add RBD team

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-mixin: remove timepicker override in every dashboards

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 37add644d1e65b4d6375a63dcbef742420a5a4c3)

ceph-mixin: rationalize local helper functions to utils

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 5db37300fde5c6cc2ec9f3ead34ea1b93126f5bf)

ceph-mixin: fix typos

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 0b7cc6bc998fec47a29abfa94224cacee2dd598d)

ceph-mixin: fix test with rate and label changes

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit c8f086c182b87f1a813cb37fd58ad1e753a6b0bf)

ceph-mixin: don't add cluster matcher if showcluster is disabled

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 3b6356c8724ee2b299743d20ff5df0401181228b)

ceph-mixin: refactor the structure of _config and utils

Before this refactor we couln't override the config externally. Now the
_config is correctly propagated and not only taken from the
config.libsonnet file.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit fd4f484d220d98ba684878c87488cd74c502b4ff)