ceph-fuse: link to libfuse3 if fuse3 is installed

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>

rbd-fuse: remove FUSE_CAP_BIG_WRITES when linking to libfuse3

The big-writes is enabled as default from 3.0.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>

rbd-fuse: switch to use utimens instead

The utime is deprecated.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph-fuse: enable the big writes as default

When the libfuse version < 3, without the big_writes options enabled
the write perf is very poor.

Fixes: https://tracker.ceph.com/issues/44885
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34584 from bk201/wip-45108

mgr/test_orchestrator: fix error when listing services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32951 from matthewoliver/cephadm_rgw_ssl

cephadm: add RGW SSL support

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34410 from lxbsz/mds_dirty_parent

tools/cephfs: add accounted_rstat/rstat when building file dentry

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #33928 from mchangir/cephfs-journal-tool-cannot-set-dry_run-arg

cephfs-journal-tool: correctly parse --dry_run argument

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32612 from rishabh-d-dave/fs-qa-UnicodeDecodeError

test_cephfs_shell: fix UnicodeDecodeError

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33169 from moningchao/master

client: fix blacklist removal failed

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #33711 from lxbsz/qa_umount_failed

qa/tasks/cephfs: umount the mountpoints when tearDown

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34533 from fanjunwei/fix_fuse_bug

client: fix ceph-fuse parse command line failed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34601 from yuriw/wip-yuriw-crontab-master

qa/tests: added octopus-p2p upgrade tests to the mix

Merge pull request #34604 from tchaikov/wip-cmake-fuse

cmake: check for FUSE version and cleanups

Reviewed-by: Xiubo Li <xiubli@redhat.com>

cmake: detect fuse3 as well

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for FUSE version

and refactor FindFUSE.cmake a little bit to conditionalize OSX support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against FUSE::FUSE

instead of specifying the include directory and linkage separately

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename Findfuse.cmake to FindFUSE.cmake

so its name is consistent with upstream name, and with the variables
exposed by FindFUSE.cmake.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: added octopus-p2p upgrade tests to the mix

Signed-off-by: Yuri Weinstein <yweinste@gmail.com>

Merge pull request #34573 from dillaman/wip-44938-2

test: tx-only peer from heartbeat can race w/ CLI

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34346 from jschmid1/osd_id_claims

mgr/cephadm: add support for osd_id_claims

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34517 from tchaikov/wip-qa-dashboard-telemetry

qa/tasks/mgr/dashboard: set last_opt_revision when setting enabled

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34210 from kotreshhr/cephfs-subvolume-info

mgr/volumes: Add interface to get subvolume metadata

mgr/test_orchestrator: fix error when listing services

- Adapting attribute change of ServiceSpec.
- Update dummy data.

Fixes: https://tracker.ceph.com/issues/45108
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard: set last_opt_revision when setting enabled

telemetry requires user to re opt-in when opt_revision is bumped up

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: Fixes some deserialization errors

Fixes errors when calling `from_json` of these classes:
- InventoryHost: parsing labels
- ServiceDescription: `last_refresh` and `created` fields should be parsed
  to datetime type.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #34577 from tchaikov/wip-doc-dev-corpus

doc/dev/corpus.rst: tweak formatting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev/corpus.rst: tweak formatting

* use `shell` lexer, otherwise the Python one is used, and the rendered
  result does not look right
* be consistent when indenting -- either use tab or spaces, otherwise
  the indent in code block would be wrong.
* double quote the variables in text

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: add RGW SSL support

This patch adds SSL support to RGW when using cephadm.
If an SSL certificate is provided, inside the json supplied with:

  cpeh orchestrator rgw create -i rgw.json

Then the SSL cert and/or key will be added to pushed into the mon config-key database
using the key `rgw/cert/<rgw_realm>/<rgw_zone>.[crt|key]`.
Which will then be referenced in the config:

  rgw_frontends = beast port=80 ssl_port=443 ssl_certificate=config://rgw/cert/<rgw_realm>/<rgw_zone>.crt

And if an ssl key is also supplied this becomes something like:

  rgw_frontends = beast port=80 ssl_port=443 ssl_certificate=config://rgw/cert/<rgw_realm>/<rgw_zone>.crt ssl_key=config://rgw/cert/<rgw_realm>/<rgw_zone>.key

Of course you could also just upload the cert and key yourself to
config-key location, and ssl will be enabled as well. But this patch
let's you either supply them via `-i` or as a manual upload step.

Co-Authored-By: Michael Fritch <mfritch@suse.com>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #34541 from neha-ojha/wip-balancer-on

mgr: turn on balancer in upmap mode by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: retry the addition of a mirror pool peer

We might race with the remote rbd-mirror daemon creating a
tx-only peer when adding a new peer. Therefore, delete the
tx-only peer and attempt to re-create it.

Fixes: https://tracker.ceph.com/issues/44938
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: mirror peer add error message should include site name

It previously included the pointer to string holding the generated
uuid (neither of which would mean much to an end user).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: improved error message when adding a duplicate mirror pool peer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32747 from Exotelis/wip-40702-collapse

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34094 from cbodley/wip-rgw-data-sync-cache

rgw multisite: data sync optimizations

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/cephadm: add replacement flag to _create_osd

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34549 from dillaman/wip-qa-rbd-read-balance

qa/suites/rbd: tweak 'require_osd_release' to 'pacific'

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34496 from dillaman/wip-44105

rbd-mirror: clean-up unnecessary non-primary snapshots

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Add expand/collapse datatable feature

Adds expand/collapse feature to every datatable with details.

Fixes: https://tracker.ceph.com/issues/40702
Signed-off-by: Sebastian Krah <skrah@suse.com>

mgr/cephadm: use --osd-ids instead of --osd-id

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: add support for osd_id_claims

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34529 from maran/master

cephadm: warn when no valid version arguments are supplied to add-repo

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34490 from sebastian-philipp/cephadm-command_add_repo-no-podman

cephadm: add-repo: Don't test for podman

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34443 from sebastian-philipp/cephadm-revert-33974

cephadm: Update images used

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34556 from ahaase-c4u/patch-1

mgr/cephadm: corrected documentation for starting an upgrade

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34555 from ahaase-c4u/45065

mgr/cephadm: Fixed warn_on_stray_hosts setting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34415 from sebastian-philipp/orch-fix-rgw-daemon-description-id

mgr/orch: Fix DaemonDescription().daemon_id() for RGWs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34216 from jschmid1/preview_drivegroups

mgr/cephadm: preview osd creation for osdspecs

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34558 from mgfritch/cephadm-enter-container

cephadm: use CephContainer during command `enter`

Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: enable osd previews

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: corrected documentation for starting an upgrade

Signed-off-by: Andreas Haase <andreas.haase@community4you.de>

doc/rados/operations: update docs to reflect balancer being on by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34562 from tchaikov/wip-jenkins-trigger

github: add "jenkins test classic perf" to jenkins commands list

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

github: add "jenkins test classic perf" to jenkins commands list

it's used for triggering perf test with classic OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

PendingReleaseNotes: add note about balancer being on by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #19076 from jecluis/wip-mon-fix-osdmap-lec-trim

mon/OSDMonitor: allow trimming maps even if osds are down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34510 from aclamk/wip-rocksdb-open-close

kv/KeyValueDB: Making open/close operations complementary.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34228 from shonpaz123/add_notifications_documentation

doc/rgw: add documentation for bucket notifications filters and supported push endpoints

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #33975 from jbwyatt4/jbwyatt4-need-disk-space-info

doc: added space requirement to "Build Prerequisites"

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33901 from rs-fabrica/rados_man_page_options

doc/man/8/rados.rst: including all available options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34197 from varshar16/wip-doc-SubmittingPatches-url

doc/SubmittingPatches: Fix 'Fixes line' hyperlink

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34265 from shonpaz123/add_sc_doc

doc/rgw: add documentation for allowed storage classes for client SDKs

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34360 from gouthampacha/doc-fix

doc: fix broken link for cephfs kernel recommendations

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: use CephContainer during command `enter`

instead of the `get_container` helper func

Signed-off-by: Michael Fritch <mfritch@suse.com>

rbd-mirror: utilize structured binding in 'for' loop

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/mon/osd-pool-df.sh: flush_pg_stats explicitly

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33806 from trociny/wip-mirror_snapshot_schedule-watcher

mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34262 from zhengchengyao/wip-deprecated-order

tools: remove all deprecated options from the help output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: Fixed warn_on_stray_hosts setting

Fixes: https://tracker.ceph.com/issues/45065
Signed-off-by: Andreas Haase <andreas.haase@community4you.de>

Merge pull request #33365 from ifed01/wip-ifed-hybrid-alloc2

os/bluestore: introduce hybird allocator

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32704 from petrutlucian94/windows.2

common: Add dlfcn_compat.h for win32

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: Update images used

Print warning when using cephadm from master

See also "use quay octopus tip until 15.2 tag is available"
* a9b15c7e1a0c14376cd66f166370694294398494.

See also  "update default container images"
* 1f05f7578794380f969a7e93db07345626b3e4df.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34336 from nSedrickm/nsedrickm-ceph-dev

mgr/dashboard: Proposed About Modal box

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34469 from votdev/issue_44991_pool_promql

monitoring: alert for pool fill up broken

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

qa/suites/rbd: tweak 'require_osd_release' to 'pacific'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34065 from tspmelo/wip-iscsi-advanced

mgr/dashboard: Prevent iSCSI target recreation when editing controls

Reviewed-by: Ricardo Marques <rimarques@suse.com>

common: Add dlfcn_compat.h for win32

We'll emulate the dlfcn.h interface using Windows functions, which
will allow loading Windows DLLs.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #34009 from tspmelo/wip-iscsi-auth

mgr/dashboard: Fix iSCSI's username and password validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33816 from rzarzynski/wip-msg-no-extra-alloc-in-secure

msg/async: fix unnecessary 4 kB allocation in secure mode.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/mon/osd-pool-df.sh: fix test to check for the right values

Though the test passed, we weren't checking for the correct values:

.../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  jq .stats.total_avail_bytes
../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  local global_avail=0
.../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  jq '.pools | map(select(.name == "$rep_poolname"))[0].stats.max_avail'
../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  local rep_avail=null
.../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  jq '.pools | map(select(.name == "$ec_poolname"))[0].stats.max_avail'
../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  local ec_avail=null
../qa/standalone/mon/osd-pool-df.sh:66: TEST_ceph_df:  echo '0 >= null*3'
../qa/standalone/mon/osd-pool-df.sh:66: TEST_ceph_df:  bc
1
../qa/standalone/mon/osd-pool-df.sh:67: TEST_ceph_df:  echo '0 >= null*1.5'
../qa/standalone/mon/osd-pool-df.sh:67: TEST_ceph_df:  bc
1

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/standalone/mgr/balancer.sh: adapt test

Now that the balancer is on by default the test needs these changes.

Signed-off-by: Neha Ojha <nojha@redhat.com>

pybind/mgr/balancer/module.py: turn on balancer in upmap mode by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/workunits/cephtool/test.sh: don't set-require-min-compat-client luminous

This should not be required since mon_osd_initial_require_min_compat_client
is now luminous.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34344 from jmolmo/rook_inventory

mgr/rook: Fix wrong data output when listing devices

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: dont use RGWOmapAppend for error_repo

the error_repo writes need to be synchronous

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove per-bucket-shard sync leases

bucket sync now gets a const pointer to the DataSyncShard's lease to
check whether the lease has expired

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ContinuousLeaseCR::is_locked() is const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ContinuousLeaseCR doesn't need locks/atomics

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use objv_tracker to read/write bucket sync status

use cls_version on bucket sync status to detect racing writes - whether
from other gateways, or from radosgw-admin commands like 'bucket sync'
or 'bucket sync init'

classes that require a non-null version tracker take it by reference

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RadosRemoveCR takes objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SimpleRadosReadAttrsCR takes objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SysObj service applies objv on set_attrs

like write(), we need to apply the writev back to readv

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: data sync no longer needs index_key_to_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry loops based on progress

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry does not duplicate bucket sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry takes cached state

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add unittest_rgw_bucket_sync_cache

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data sync cache for bucket shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: track last timestamp written for bucket sync

bucket sync remembers the latest timestamp that it successfully wrote to
the bucket sync status. data sync can use this to make future decisions
without having to reread its sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/rbd_support: attempt to unregister watcher on mgr exit

Signed-off-by: Mykola Golub <mgolub@suse.com>