doc: add object corpus to release checklist

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph-object-corpus: pick up octopus 15.2.1-40-ga838bb1aae

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

include/denc: replace bufferlist::copy with iterator version

This version was only compiled as part of ceph-object-corpus
generation, when ENCODE_DUMP_PATH is defined, so it was missed
when bufferlist::copy() was removed.

Fixes: https://tracker.ceph.com/issues/45023
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34541 from neha-ojha/wip-balancer-on

mgr: turn on balancer in upmap mode by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32747 from Exotelis/wip-40702-collapse

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34094 from cbodley/wip-rgw-data-sync-cache

rgw multisite: data sync optimizations

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #34549 from dillaman/wip-qa-rbd-read-balance

qa/suites/rbd: tweak 'require_osd_release' to 'pacific'

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34496 from dillaman/wip-44105

rbd-mirror: clean-up unnecessary non-primary snapshots

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Add expand/collapse datatable feature

Adds expand/collapse feature to every datatable with details.

Fixes: https://tracker.ceph.com/issues/40702
Signed-off-by: Sebastian Krah <skrah@suse.com>

Merge pull request #34529 from maran/master

cephadm: warn when no valid version arguments are supplied to add-repo

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34490 from sebastian-philipp/cephadm-command_add_repo-no-podman

cephadm: add-repo: Don't test for podman

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34443 from sebastian-philipp/cephadm-revert-33974

cephadm: Update images used

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34556 from ahaase-c4u/patch-1

mgr/cephadm: corrected documentation for starting an upgrade

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34555 from ahaase-c4u/45065

mgr/cephadm: Fixed warn_on_stray_hosts setting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34415 from sebastian-philipp/orch-fix-rgw-daemon-description-id

mgr/orch: Fix DaemonDescription().daemon_id() for RGWs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34216 from jschmid1/preview_drivegroups

mgr/cephadm: preview osd creation for osdspecs

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34558 from mgfritch/cephadm-enter-container

cephadm: use CephContainer during command `enter`

Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: enable osd previews

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: corrected documentation for starting an upgrade

Signed-off-by: Andreas Haase <andreas.haase@community4you.de>

doc/rados/operations: update docs to reflect balancer being on by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34562 from tchaikov/wip-jenkins-trigger

github: add "jenkins test classic perf" to jenkins commands list

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

github: add "jenkins test classic perf" to jenkins commands list

it's used for triggering perf test with classic OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

PendingReleaseNotes: add note about balancer being on by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #19076 from jecluis/wip-mon-fix-osdmap-lec-trim

mon/OSDMonitor: allow trimming maps even if osds are down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34510 from aclamk/wip-rocksdb-open-close

kv/KeyValueDB: Making open/close operations complementary.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34228 from shonpaz123/add_notifications_documentation

doc/rgw: add documentation for bucket notifications filters and supported push endpoints

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #33975 from jbwyatt4/jbwyatt4-need-disk-space-info

doc: added space requirement to "Build Prerequisites"

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33901 from rs-fabrica/rados_man_page_options

doc/man/8/rados.rst: including all available options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34197 from varshar16/wip-doc-SubmittingPatches-url

doc/SubmittingPatches: Fix 'Fixes line' hyperlink

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34265 from shonpaz123/add_sc_doc

doc/rgw: add documentation for allowed storage classes for client SDKs

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34360 from gouthampacha/doc-fix

doc: fix broken link for cephfs kernel recommendations

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: use CephContainer during command `enter`

instead of the `get_container` helper func

Signed-off-by: Michael Fritch <mfritch@suse.com>

rbd-mirror: utilize structured binding in 'for' loop

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/mon/osd-pool-df.sh: flush_pg_stats explicitly

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33806 from trociny/wip-mirror_snapshot_schedule-watcher

mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34262 from zhengchengyao/wip-deprecated-order

tools: remove all deprecated options from the help output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: Fixed warn_on_stray_hosts setting

Fixes: https://tracker.ceph.com/issues/45065
Signed-off-by: Andreas Haase <andreas.haase@community4you.de>

Merge pull request #33365 from ifed01/wip-ifed-hybrid-alloc2

os/bluestore: introduce hybird allocator

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32704 from petrutlucian94/windows.2

common: Add dlfcn_compat.h for win32

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: Update images used

Print warning when using cephadm from master

See also "use quay octopus tip until 15.2 tag is available"
* a9b15c7e1a0c14376cd66f166370694294398494.

See also  "update default container images"
* 1f05f7578794380f969a7e93db07345626b3e4df.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34336 from nSedrickm/nsedrickm-ceph-dev

mgr/dashboard: Proposed About Modal box

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34469 from votdev/issue_44991_pool_promql

monitoring: alert for pool fill up broken

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

qa/suites/rbd: tweak 'require_osd_release' to 'pacific'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34065 from tspmelo/wip-iscsi-advanced

mgr/dashboard: Prevent iSCSI target recreation when editing controls

Reviewed-by: Ricardo Marques <rimarques@suse.com>

common: Add dlfcn_compat.h for win32

We'll emulate the dlfcn.h interface using Windows functions, which
will allow loading Windows DLLs.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #34009 from tspmelo/wip-iscsi-auth

mgr/dashboard: Fix iSCSI's username and password validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33816 from rzarzynski/wip-msg-no-extra-alloc-in-secure

msg/async: fix unnecessary 4 kB allocation in secure mode.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/mon/osd-pool-df.sh: fix test to check for the right values

Though the test passed, we weren't checking for the correct values:

.../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  jq .stats.total_avail_bytes
../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  local global_avail=0
.../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  jq '.pools | map(select(.name == "$rep_poolname"))[0].stats.max_avail'
../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  local rep_avail=null
.../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  jq '.pools | map(select(.name == "$ec_poolname"))[0].stats.max_avail'
../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  local ec_avail=null
../qa/standalone/mon/osd-pool-df.sh:66: TEST_ceph_df:  echo '0 >= null*3'
../qa/standalone/mon/osd-pool-df.sh:66: TEST_ceph_df:  bc
1
../qa/standalone/mon/osd-pool-df.sh:67: TEST_ceph_df:  echo '0 >= null*1.5'
../qa/standalone/mon/osd-pool-df.sh:67: TEST_ceph_df:  bc
1

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/standalone/mgr/balancer.sh: adapt test

Now that the balancer is on by default the test needs these changes.

Signed-off-by: Neha Ojha <nojha@redhat.com>

pybind/mgr/balancer/module.py: turn on balancer in upmap mode by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/workunits/cephtool/test.sh: don't set-require-min-compat-client luminous

This should not be required since mon_osd_initial_require_min_compat_client
is now luminous.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34344 from jmolmo/rook_inventory

mgr/rook: Fix wrong data output when listing devices

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: dont use RGWOmapAppend for error_repo

the error_repo writes need to be synchronous

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove per-bucket-shard sync leases

bucket sync now gets a const pointer to the DataSyncShard's lease to
check whether the lease has expired

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ContinuousLeaseCR::is_locked() is const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ContinuousLeaseCR doesn't need locks/atomics

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use objv_tracker to read/write bucket sync status

use cls_version on bucket sync status to detect racing writes - whether
from other gateways, or from radosgw-admin commands like 'bucket sync'
or 'bucket sync init'

classes that require a non-null version tracker take it by reference

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RadosRemoveCR takes objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SimpleRadosReadAttrsCR takes objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SysObj service applies objv on set_attrs

like write(), we need to apply the writev back to readv

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: data sync no longer needs index_key_to_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry loops based on progress

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry does not duplicate bucket sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry takes cached state

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add unittest_rgw_bucket_sync_cache

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data sync cache for bucket shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: track last timestamp written for bucket sync

bucket sync remembers the latest timestamp that it successfully wrote to
the bucket sync status. data sync can use this to make future decisions
without having to reread its sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/rbd_support: attempt to unregister watcher on mgr exit

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33989 from ifed01/wip-ifed-better-bluefs-expand

os/bluestore: open DB in read-only when expanding DB/WAL.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #33782 from aclamk/wip-alloc-aging-logging-simpler

os/bluestore: Simplified logic of sliding window.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #33434 from ifed01/wip-ifed-big-writes-deferred

os/bluestore: go deferred for 'big' writes.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #33715 from trociny/wip-pybind-rados-watch

pybind/rados: add watch object bindings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: store timestamp in InitBucketShardSyncStatus

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add struct data_sync_obligation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_single_entry() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use optional for DataSyncShardMarkerTrack

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: parse bucket-shard before DataSyncSingleEntry

it's easier for DataSyncShard to handle parsing failures before calling
MarkerTrack::start() and DataSyncSingleEntry

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: data sync tracks timestamps for error repo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add error repo primitives with timestamp comparisons

the sync error repo stores omap keys for each datalog entry that needs a
retry. this adds a new primitive for that, that also stores a timestamp
with each omap key, and will only allow overwrites/removals when a newer
timestamp is provided

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncShardCR uses GetOmapVals

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add CR for omap_get_vals()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: incremental bucket sync status stores timestamp

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add [coarse_]real_time support to ceph_json.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: don't write to error repo for async notification failures

async notifications are just hints, and don't imply an obligation to
sync the bucket shard. if we fail to sync, don't write it to the error
repo for retry. we'll see the change later when processing the datalog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWWaitCR

coroutines that want to sleep should just call RGWCoroutine::wait()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33982 from cbodley/wip-cls-cmpomap

cls/cmpomap: add cls module for CMPXATTR-like functionality in omap

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

doc/SubmittingPatches: Fix 'Fixes line' hyperlink

Signed-off-by: Varsha Rao <varao@redhat.com>

options: add cmpomap to osd_class_load_list

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: add ceph_test_cls_cmpomap

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/cmpomap: add cls module for CMPXATTR-like functionality in omap

provides a cmp_vals() op similar to librados' omap_cmp(), but with full
support for the comparison modes and operations from cmpxattr(). if any
of the requested key/value comparisons are unsuccessful, the op fails
with -ECANCELED

cmp_vals() can be composed with other ops like set_omap_vals() to make
make them conditional on the successful comparison of all existing values

also provides cmp_set_vals() and cmp_rm_keys(), which apply a mutation
to only the keys that compare successfully. these enable batch
operations where, unlike cmp_vals() + set_omap_vals(), you don't want
one failed comparison to prevent operations on unrelated keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: add cls_cxx_map_get_vals_by_keys

add an objclass api for CEPH_OSD_OP_OMAPGETVALSBYKEYS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: expose cls_cxx_map_get_val in header

this was defined in osd/objclass.cc but not declared in objclass/objclass.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: add missing != operator for bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: comparison operators take const bufferlist&

Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson: add HybridAllocator to alienstore build.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: improve adjustent extents merge in hybrid allocatoc

Bitmap counterpart is inspected for adjustent free extents when
releasing extent and trying to add it to AVL tree.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: remove HybridAllocator::init_add_free method.

It's a 100% replica of AvlAllocator one which is inherited.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: configure hybrid allocator threshold to fallback.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add hybrid (avl+bitmap) allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>