]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
23 months agoos/bluestore: Fix split_cache 51442/head
Adam Kupczyk [Mon, 17 Jul 2023 07:31:05 +0000 (07:31 +0000)]
os/bluestore: Fix split_cache

Fix cache was broken as it moved blobs and extents between cache shards
without changing atomic num_blobs, num_extent counters.

Now tracking counters correctly.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
23 months agoos/bluestore: Add UT for ExtentMap::dup_esb()
Adam Kupczyk [Thu, 4 May 2023 08:50:10 +0000 (08:50 +0000)]
os/bluestore: Add UT for ExtentMap::dup_esb()

Add unit tests for ExtentMap::dup_esb() to test_bluestore_types.
Extra hooks added to BlueStore:
1) bluestore_extent_ref_map_t::debug_peek inspects how many times specific AU is used
2) ExtentMap::debug_list_disk_layout to extract how onode maps to AUs
3) BlueStore::debug_punch_hole to use logic from _wctx_finish internal function

Unit tests are limited to operations aligned to allocation unit.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
23 months agotest/bluestore_types: introduce basic UT for new ExtentMap::dup
Igor Fedotov [Wed, 19 Apr 2023 15:00:10 +0000 (18:00 +0300)]
test/bluestore_types: introduce basic UT for new ExtentMap::dup

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
23 months agoos/store_test: Change test limits
Adam Kupczyk [Mon, 20 Feb 2023 08:52:48 +0000 (08:52 +0000)]
os/store_test: Change test limits

Test ReproNoBlobMultiTest required very narrow value of l_bluestore_gc_merged.
After ExtentMap::dup changes it can no longer reach it.
Generally that perf counter is useless; as we have now less allocated space
912798515 vs 9160228864 before.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
23 months agoos/store_test: New SyntheticShardingLimited tests operate on limited objects
Adam Kupczyk [Mon, 13 Feb 2023 14:04:21 +0000 (14:04 +0000)]
os/store_test: New SyntheticShardingLimited tests operate on limited objects

Iterate over few objects with limited onode shard size.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
23 months agoos/store_test: New SyntheticLimited tests operate on limited objects
Adam Kupczyk [Thu, 9 Feb 2023 09:25:48 +0000 (09:25 +0000)]
os/store_test: New SyntheticLimited tests operate on limited objects

New SyntheticLimited tests operate on restiricted amount of objects.
This helps to more easily replicate problems that occur when we
manipulate objects that have in-flight "writing" buffers.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
23 months agotests/ceph_test_objectstore: Fix clone_range
Adam Kupczyk [Mon, 16 Jan 2023 13:29:15 +0000 (13:29 +0000)]
tests/ceph_test_objectstore: Fix clone_range

It was possible to get srcoff and dstoff different.
It is untested and error-prone.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
23 months agoos/bluestore: Simplify 51441/head
Adam Kupczyk [Wed, 9 Aug 2023 07:30:03 +0000 (07:30 +0000)]
os/bluestore: Simplify

Remove unnecessary arguments from functions.
Reduce unneeded indirections for accessing extent_map.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Enable selection of old/new ExtentMap::dup
Adam Kupczyk [Fri, 31 Mar 2023 09:41:45 +0000 (09:41 +0000)]
os/bluestore: Enable selection of old/new ExtentMap::dup

Modify _do_clone_range to select variant of ExtentMap::dup depending on elastic_shared_blob.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add conf option elastic_shared_blobs
Adam Kupczyk [Fri, 31 Mar 2023 09:04:40 +0000 (09:04 +0000)]
os/bluestore: Add conf option elastic_shared_blobs

Having new ExtentMap::dup that introduces heavy changes to blob processing
seems risky. We will enable it only on demand. In future, once the feature
is tested in production, the choice should be removed (and feature always on).

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Resurrect ExtentMap::dup
Adam Kupczyk [Thu, 11 May 2023 10:53:24 +0000 (10:53 +0000)]
os/bluestore: Resurrect ExtentMap::dup

1. Rename ExtentMap::dup to ExtentMap::dup_esb
2. Ressurect ExtentMap::dup

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Fix copying Buffers
Adam Kupczyk [Mon, 16 Jan 2023 12:46:39 +0000 (12:46 +0000)]
os/bluestore: Fix copying Buffers

Only copy Buffers that were not copied before.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Modify ExtentMap::dup to reuse blobs
Adam Kupczyk [Wed, 10 May 2023 12:20:10 +0000 (12:20 +0000)]
os/bluestore: Modify ExtentMap::dup to reuse blobs

Make ExtentMap::dup able to reuse some already existing shared blobs,
when a regular blob has to be transformed to shared blob.

Make ExtentMap::dup() use make_range_shared_maybe_merge a primary tool for cloning.

Modify ExtentMap::dup to make a diligent merge of blobs.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add make_range_shared_maybe_merge
Adam Kupczyk [Wed, 10 May 2023 12:40:38 +0000 (12:40 +0000)]
os/bluestore: Add make_range_shared_maybe_merge

Adds function that converts a specified range in object to shared blobs,
possibly merging them with other shared blobs.

Modify make_range_shared_maybe_merge to allow for merging blobs.
It now uses can_merge_blobs and merge_blobs.

Add discard_unused_buffers() to make_range_shared.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Introduce reblob_extents
Adam Kupczyk [Wed, 10 May 2023 11:22:16 +0000 (11:22 +0000)]
os/bluestore: Introduce reblob_extents

Func reblob_extents is used to modify extents.
It is final step of melding 2 blobs together.
It removes reference to blob that is to be phased out,
and replaces it with reference to the blob that is the sum of them.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add scan_shared_blobs and find_best_companion
Adam Kupczyk [Wed, 10 May 2023 12:18:38 +0000 (12:18 +0000)]
os/bluestore: Add scan_shared_blobs and find_best_companion

Introduce function that will scan through relevant range that is to be cloned,
and function that will find best matching blob to attach to.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add discard_unused_buffers function
Adam Kupczyk [Wed, 10 May 2023 09:30:23 +0000 (09:30 +0000)]
os/bluestore: Add discard_unused_buffers function

When we punch_hole in blobs we leave Buffers unchanged.
Normally it is not a problem, but when we merge blobs there is a collision.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add can_merge_blob and merge_blob
Adam Kupczyk [Wed, 10 May 2023 09:53:06 +0000 (09:53 +0000)]
os/bluestore: Add can_merge_blob and merge_blob

Two new functions:
1) can_marge_blob checks if 2 blobs are compatible to be merged together
2) merge_blob merges 2 blobs into 1, emptying source and putting all to destination

Modify merge_blob() to return logical length of produced blob.

Clear "unused" bitmap

When make blob shared or merge it with other blob, clear unused.
It drops some potential optimizations for writing into large blobs,
but it is unlikely to be useful.
Such info can be useful only after it reverts from Shared to regular blob.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add functions for partial blob copy
Adam Kupczyk [Mon, 16 Jan 2023 11:16:43 +0000 (11:16 +0000)]
os/bluestore: Add functions for partial blob copy

Add functions that can copy parts of blobs.
It is necessary for merging blobs together,
which happens on cloning (ExtentMap::dup).

Fixed:
Blob::copy_extents_over_empty was faulty when insertion was targetting
last extent and that extent was invalid(empty).

Add dup() for bluestore_blob_t and bluestore_blob_use_tracker.

Changed:
Modified Blob::copy_from for better readability.
Added bluestore_blob_t::adjust_to initization that conforms to
other blob specifics.

Move assert for is_mutable() out of bluestore_blob_t::add_tail,
so it can be used in blobs that are shared.

Modify bluestore_blob_use_tracker_t::get to automatically expand
when accessing more AUs then originally declared.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Remove inject_21040.
Adam Kupczyk [Mon, 21 Nov 2022 11:25:40 +0000 (11:25 +0000)]
os/bluestore: Remove inject_21040.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Add wrappers to access Blob::bc (BufferSpace)
Adam Kupczyk [Thu, 25 May 2023 19:36:02 +0000 (19:36 +0000)]
os/bluestore: Add wrappers to access Blob::bc (BufferSpace)

The reason was historical - it was to give one access method to bc regardles where it was defined SharedBlob or Blob.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Fix tracking of num_blobs
Adam Kupczyk [Thu, 25 May 2023 19:26:28 +0000 (19:26 +0000)]
os/bluestore: Fix tracking of num_blobs

By moving BufferSpace from SharedBlob to Blob tracking of num_blobs get broken.
Fixed that and reinforced by adding asserts to BufferCacheShard destructor.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add lock in ExtentMap::dup
Adam Kupczyk [Thu, 25 May 2023 19:20:51 +0000 (19:20 +0000)]
os/bluestore: Add lock in ExtentMap::dup

_dup_writing requires locking of BufferCacheShard

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Refactor, add _dup_writing when cloning
Adam Kupczyk [Sun, 27 Nov 2022 11:44:10 +0000 (11:44 +0000)]
os/bluestore: Refactor, add _dup_writing when cloning

With BufferSpace now attached to Blob (was SharedBlob inside it),
on-the-fly 'writing' buffers must be copied to clones.
Otherwise those objects will read data from disk before it is written there.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Fix BlueStore::_do_remove
Adam Kupczyk [Fri, 24 Mar 2023 20:49:30 +0000 (20:49 +0000)]
os/bluestore: Fix BlueStore::_do_remove

Fix blobs having the same empty shared blob.

Each blob on creation gets its own unique (empty) SharedBlob object.
ExtentMap::dup() sometimes merges blobs together, so 2 different blobs
get the same SharedBlob object.

Function _do_remove() tries to convert shared blobs into regular ones.
If it succeeds we could get 2 blobs having the same EMPTY SharedBlob object.

The solution is to create detached SharedBlob if necessary.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Adapt split_cache
Adam Kupczyk [Sun, 27 Nov 2022 11:26:27 +0000 (11:26 +0000)]
os/bluestore: Adapt split_cache

Adapt split_cache to new situation.

Now buffers are attached to Blob, and we need always move them,
regardless that we already moved relevant SharedBlob.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Move BufferSpace from SharedBlob to Blob
Adam Kupczyk [Wed, 23 Nov 2022 15:07:23 +0000 (15:07 +0000)]
os/bluestore: Move BufferSpace from SharedBlob to Blob

This is necessary to enable adding more Buffers to Blobs that are shared.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Refactor finish_write
Adam Kupczyk [Wed, 23 Nov 2022 11:49:47 +0000 (11:49 +0000)]
os/bluestore: Refactor finish_write

Move finish_write from SharedBlob to Blob.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Refactor TransContext
Adam Kupczyk [Wed, 23 Nov 2022 11:20:25 +0000 (11:20 +0000)]
os/bluestore: Refactor TransContext

Modified TransContext.
Changed
std::set<SharedBlobRef> shared_blobs_written
to
std::set<BlobRef> blobs_written

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Better dout for bluestore_blob_t
Adam Kupczyk [Mon, 16 Jan 2023 13:32:59 +0000 (13:32 +0000)]
os/bluestore: Better dout for bluestore_blob_t

Add printing of len to operator<< for const bluestore_blob_t.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add operator<< for BufferSpace
Adam Kupczyk [Mon, 16 Jan 2023 09:28:47 +0000 (09:28 +0000)]
os/bluestore: Add operator<< for BufferSpace

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Added printing of csum size
Adam Kupczyk [Thu, 19 Jan 2023 13:22:10 +0000 (13:22 +0000)]
os/bluestore: Added printing of csum size

Modifed bluestore_blob_t to include current size of csum_data.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Debugs before ceph_assert in resharding.
Adam Kupczyk [Wed, 10 May 2023 11:37:33 +0000 (11:37 +0000)]
os/bluestore: Debugs before ceph_assert in resharding.

When encode_some fails twice ceph aborts.
Now we print object details just before.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: add ExtentMap::maybe_reshard
Adam Kupczyk [Thu, 16 Feb 2023 12:46:33 +0000 (12:46 +0000)]
os/bluestore: add ExtentMap::maybe_reshard

maybe_reshard is created to filter out unnecessary calls to request_reshard.
The intended use is to let just request maybe_reshard, and delegate check
if the action is really necessary to the implementation detail level.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Fix ExtentMap::reshard
Adam Kupczyk [Thu, 16 Feb 2023 12:21:53 +0000 (12:21 +0000)]
os/bluestore: Fix ExtentMap::reshard

When reshard is applied for the first time, expand reshard range to encompas whole object.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Improve ExtentMap::fault_range
Adam Kupczyk [Thu, 16 Feb 2023 12:40:25 +0000 (12:40 +0000)]
os/bluestore: Improve ExtentMap::fault_range

Make faster exit when sharding not enabled.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Add printing of broken shared blobs
Adam Kupczyk [Tue, 10 Jan 2023 17:40:57 +0000 (17:40 +0000)]
os/bluestore: Add printing of broken shared blobs

When we do fsck with non-repair mode, we do not get any info about shared blobs
that actually were corrupted. Now we print them.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Refactor foreach_shared_blob in fsck
Adam Kupczyk [Tue, 10 Jan 2023 10:50:54 +0000 (10:50 +0000)]
os/bluestore: Refactor foreach_shared_blob in fsck

Upgrade local foreach_shared_blob into _fsck_foreach_shared_blob
that can be used on entire BlueStore scope.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2 years agoos/bluestore: Expand TwoQBufferCacheShard::_audit
Adam Kupczyk [Fri, 27 Jan 2023 15:13:10 +0000 (15:13 +0000)]
os/bluestore: Expand TwoQBufferCacheShard::_audit

Add more checks on consistency.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/bluestore: Fix DEBUG_CACHE
Adam Kupczyk [Tue, 24 Jan 2023 13:05:52 +0000 (13:05 +0000)]
os/bluestore: Fix DEBUG_CACHE

Fix it, so it can be enabled and work.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoos/store_test: Retune tests to current code
Adam Kupczyk [Fri, 17 Feb 2023 14:51:55 +0000 (14:51 +0000)]
os/store_test: Retune tests to current code

After introduction of lazy statfs updates and mechanism to store them at exit,
some tests required tune-up.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
2 years agoMerge pull request #50660 from zdover23/wip-doc-2023-03-25-rados-operations-bluestore...
Anthony D'Atri [Fri, 24 Mar 2023 17:54:06 +0000 (13:54 -0400)]
Merge pull request #50660 from zdover23/wip-doc-2023-03-25-rados-operations-bluestore-migration-prompt-fix

doc/operations: fix prompt in bluestore-migration

2 years agodoc/operations: fix prompt in bluestore-migration 50660/head
Zac Dover [Fri, 24 Mar 2023 17:47:10 +0000 (03:47 +1000)]
doc/operations: fix prompt in bluestore-migration

Fix a single prompt in bluestore-migration.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2 years agoMerge pull request #50302 from weirdwiz/rbd-perf-counters
Ilya Dryomov [Fri, 24 Mar 2023 12:00:10 +0000 (13:00 +0100)]
Merge pull request #50302 from weirdwiz/rbd-perf-counters

rbd-mirror: switch to labeled perf counters

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agoMerge pull request #50497 from rishabh-d-dave/fs-qa-caps-helper
Rishabh Dave [Fri, 24 Mar 2023 10:58:38 +0000 (16:28 +0530)]
Merge pull request #50497 from rishabh-d-dave/fs-qa-caps-helper

qa/cephfs: add more helper methods to caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2 years agoMerge pull request #49302 from petrutlucian94/adapter_resets
Ilya Dryomov [Fri, 24 Mar 2023 10:41:04 +0000 (11:41 +0100)]
Merge pull request #49302 from petrutlucian94/adapter_resets

rbd-wnbd: optionally handle wnbd adapter restart events

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agoMerge pull request #50369 from rhcs-dashboard/exporter-labeled-counters
Avan [Fri, 24 Mar 2023 09:26:28 +0000 (14:56 +0530)]
Merge pull request #50369 from rhcs-dashboard/exporter-labeled-counters

exporter: add support for exposing labeled perf counters

2 years agorbd-wnbd: consistently use negative error codes in rbd-wnbd 49302/head
Lucian Petrut [Thu, 23 Mar 2023 08:24:01 +0000 (08:24 +0000)]
rbd-wnbd: consistently use negative error codes in rbd-wnbd

The rbd-wnbd iterators return positive errors, which is why
in certain cases we may end up with both positive and negative
error codes.

This change ensures that we'll consistently use negative
error codes.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2 years agocommon, rbd-wnbd: bump Windows log level
Lucian Petrut [Wed, 7 Dec 2022 13:18:48 +0000 (15:18 +0200)]
common, rbd-wnbd: bump Windows log level

We're increasing the log level for certain Windows operational log
messages.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2 years agorbd-wnbd: optionally handle wnbd adapter restart events
Lucian Petrut [Wed, 16 Nov 2022 11:24:34 +0000 (13:24 +0200)]
rbd-wnbd: optionally handle wnbd adapter restart events

The WNBD adapter may be reset in certain situations (e.g. driver
upgrade, MS WHQL tests, etc).

We're going to monitor the WNBD adapter using WMI[1] events, restarting
the rbd-wnbd disk mappings whenever necessary. Adapter monitoring can be
enabled by passing the --adapter-monitoring-enabled flag to the service.

This feature is optional for the following reasons:

* it's mainly used during development / driver certification
* we had to use a relatively small polling interval, which might imply
  additional resource usage. WMI quotas also have to be considered.

While at it, we're updating two lambdas that are submitted to thread pools,
avoiding default reference capturing and explicitly specifying the variables
that get copied.

[1] https://learn.microsoft.com/en-us/windows/win32/wmisdk/wmi-start-page

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2 years agoqa/cephfs: add more helper methods to caps_helper.py 50497/head
Rishabh Dave [Mon, 13 Mar 2023 10:46:13 +0000 (16:16 +0530)]
qa/cephfs: add more helper methods to caps_helper.py

Add methods that will accept read/write permissions, CephFS names and
CephFS mount point and in return will generate string form of MON, OSD
and MDS caps exactly as it is reported in Ceph keyrings.

Replace similar code in test_multifs_auth.py with calls to these helper
methods.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2 years agoMerge pull request #50649 from Thingee/foundation-mem-update-20230323
Anthony D'Atri [Fri, 24 Mar 2023 00:11:04 +0000 (20:11 -0400)]
Merge pull request #50649 from Thingee/foundation-mem-update-20230323

doc/foundation: Update Foundation members

2 years agodoc/foundation: Update Foundation members 50649/head
Mike Perez [Thu, 23 Mar 2023 22:51:17 +0000 (15:51 -0700)]
doc/foundation: Update Foundation members

Removing EasyStack, Vexxhost and adding 42on

Signed-off-by: Mike Perez <thingee@gmail.com>
2 years agoMerge pull request #50545 from ivancich/wip-fix-bi-restore-script-installation
J. Eric Ivancich [Thu, 23 Mar 2023 19:29:19 +0000 (15:29 -0400)]
Merge pull request #50545 from ivancich/wip-fix-bi-restore-script-installation

rgw: install rgw scripts with common files rather than radosgw files

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoMerge pull request #50617 from ivancich/wip-add-unordered-list-restore-index
J. Eric Ivancich [Thu, 23 Mar 2023 19:17:34 +0000 (15:17 -0400)]
Merge pull request #50617 from ivancich/wip-add-unordered-list-restore-index

rgw: add unordered listing to reindex to force stats update

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoexporter: export labeled perf-counters as prometheus metrics 50369/head
avanthakkar [Fri, 3 Mar 2023 13:20:00 +0000 (18:50 +0530)]
exporter: export labeled perf-counters as prometheus metrics

Fixes: https://tracker.ceph.com/issues/59063
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2 years agoMerge pull request #50634 from zdover23/wip-doc-2023-03-23-rados-operations-user...
zdover23 [Thu, 23 Mar 2023 13:05:28 +0000 (23:05 +1000)]
Merge pull request #50634 from zdover23/wip-doc-2023-03-23-rados-operations-user-management-1-of-x

doc/rados: edit user-management.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/rados: edit user-management.rst (1 of x) 50634/head
Zac Dover [Thu, 23 Mar 2023 02:21:06 +0000 (12:21 +1000)]
doc/rados: edit user-management.rst (1 of x)

Edit doc/rados/operations/user-management.rst. (1 of x)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2 years agoMerge pull request #50555 from myoungwon/wip-rbm-get-stat
Yingxin [Thu, 23 Mar 2023 09:23:01 +0000 (17:23 +0800)]
Merge pull request #50555 from myoungwon/wip-rbm-get-stat

crimson/os/seastore: implement get_stat() for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agocrimson/os/seastore/rbm: implement get_stat() 50555/head
myoungwon oh [Thu, 16 Mar 2023 06:35:59 +0000 (15:35 +0900)]
crimson/os/seastore/rbm: implement get_stat()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2 years agoMerge pull request #50334 from kamoltat/wip-ksirivad-fix-autoscale-warn
Yuri Weinstein [Wed, 22 Mar 2023 23:24:51 +0000 (16:24 -0700)]
Merge pull request #50334 from kamoltat/wip-ksirivad-fix-autoscale-warn

pybind/mgr/pg_autoscaler: Reorderd if statement for the func: _maybe_adjust

Reviewed-by: Laura Flores <lflores@redhat.com>
2 years agoMerge pull request #49748 from ifed01/wip-ifed-bound-rm-range-keys
Yuri Weinstein [Wed, 22 Mar 2023 23:23:56 +0000 (16:23 -0700)]
Merge pull request #49748 from ifed01/wip-ifed-bound-rm-range-keys

kv/RocksDBStore: use bounded iterators in rm_range_keys

Reviewed-by: Cory Snyder <csnyder@iland.com>
2 years agoMerge pull request #49380 from Matan-B/wip-matanb-lrod-logs
Yuri Weinstein [Wed, 22 Mar 2023 23:23:10 +0000 (16:23 -0700)]
Merge pull request #49380 from Matan-B/wip-matanb-lrod-logs

osd/PeeringState: Add logs around can_serve_replica_read() / last_complete_ondisk()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2 years agoMerge pull request #50510 from cbodley/wip-rgw-server-name
Casey Bodley [Wed, 22 Mar 2023 17:24:44 +0000 (13:24 -0400)]
Merge pull request #50510 from cbodley/wip-rgw-server-name

rgw: add default Server response header

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2 years agoMerge pull request #50504 from cbodley/wip-59048
Casey Bodley [Wed, 22 Mar 2023 17:22:33 +0000 (13:22 -0400)]
Merge pull request #50504 from cbodley/wip-59048

rgw/s3: DeleteObjects response uses correct delete_marker flag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2 years agocommon: add win32/wstring.h
Lucian Petrut [Wed, 16 Nov 2022 08:50:14 +0000 (10:50 +0200)]
common: add win32/wstring.h

Windows APIs heavily use wchar. ceph-dokan and rbd-wnbd
have some duplicated helpers that convert wstrings to/from
utf8 strings.

To avoid duplication and allow reusing those helpers, we're moving
them to common/win32/wstring.h.

We're using the "win32" subfolder because it's unlikely that this
will ever be used on other platforms.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2 years agoMerge pull request #50492 from Matan-B/wip-matanb-c-logging-levels
Matan [Wed, 22 Mar 2023 14:26:53 +0000 (16:26 +0200)]
Merge pull request #50492 from Matan-B/wip-matanb-c-logging-levels

crimson/common/log: Fix debug logging macros

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agoMerge pull request #50491 from Matan-B/wip-matanb-c-snaps-tests
Matan [Wed, 22 Mar 2023 14:25:18 +0000 (16:25 +0200)]
Merge pull request #50491 from Matan-B/wip-matanb-c-snaps-tests

test/librados/snapshots*: Enable Crimson supported tests

Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agoMerge pull request #50546 from Matan-B/wip-matanb-crimson-only-order-snap
Matan [Wed, 22 Mar 2023 14:24:02 +0000 (16:24 +0200)]
Merge pull request #50546 from Matan-B/wip-matanb-crimson-only-order-snap

crimson/osd: Support librados::OPERATION_ORDERSNAP

Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agoMerge pull request #49328 from BryceCao/wip-fix-python-script-using-s3cmd-with-error...
Casey Bodley [Wed, 22 Mar 2023 12:59:19 +0000 (08:59 -0400)]
Merge pull request #49328 from BryceCao/wip-fix-python-script-using-s3cmd-with-error-code-403

rgw : fix python script using s3cmd with error code 403

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoMerge pull request #50606 from zdover23/wip-doc-2023-03-21-rados-operations-erasure...
zdover23 [Tue, 21 Mar 2023 22:09:20 +0000 (08:09 +1000)]
Merge pull request #50606 from zdover23/wip-doc-2023-03-21-rados-operations-erasure-code

doc/rados: line-edit erasure-code.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agorgw: add unordered listing to reindex to force stats update 50617/head
J. Eric Ivancich [Sat, 18 Mar 2023 18:35:39 +0000 (14:35 -0400)]
rgw: add unordered listing to reindex to force stats update

By including an unordered listing in the script, we will complete
placing objects in the bucket index and allow stats to be updated
rather than waiting for this to happen organically at a user's
request. Unordered is preferred as it can run more efficiently.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2 years agoMerge pull request #50486 from linuxbox2/wip-dedup-opsqs
Casey Bodley [Tue, 21 Mar 2023 21:52:28 +0000 (17:52 -0400)]
Merge pull request #50486 from linuxbox2/wip-dedup-opsqs

Do not duplicate query-string in ops-log

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoMerge pull request #50468 from aclamk/wip-aclamk-harmonize-bluefs-log
Laura Flores [Tue, 21 Mar 2023 20:42:52 +0000 (15:42 -0500)]
Merge pull request #50468 from aclamk/wip-aclamk-harmonize-bluefs-log

os/bluestore: BlueFS: harmonize log read and writes modes

2 years agorgw : fix python script using s3cmd with error code 403 w/https 49328/head
Mark Kogan [Tue, 21 Mar 2023 16:30:12 +0000 (18:30 +0200)]
rgw : fix python script using s3cmd with error code 403 w/https

Fixes: https://tracker.ceph.com/issues/54104
Signed-off-by: Mark Kogan <mkogan@redhat.com>
2 years agodoc/rados: line-edit erasure-code.rst 50606/head
Zac Dover [Tue, 21 Mar 2023 12:27:15 +0000 (22:27 +1000)]
doc/rados: line-edit erasure-code.rst

Line-edit doc/rados/operations/erasure-code.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2 years agorbd-mirror: add and rename perf counters for journal and snapshot mirroring 50302/head
Divyansh Kamboj [Fri, 17 Mar 2023 09:19:14 +0000 (14:49 +0530)]
rbd-mirror: add and rename perf counters for journal and snapshot mirroring

This commit renames the existing snapshot and journal based performance
counters in the rbd-mirror daemon to better reflect their purpose.
Additionally, new perf counters have been added to provide more detailed
information about the synchronization of snapshots between the source
and destination clusters.

This commit also switches to use labels instead of having the image spec
in the name of the counter.

Signed-off-by: Divyansh Kamboj <dkamboj@redhat.com>
2 years agoMerge pull request #50602 from xxhdx1985126/wip-invalid-clean-pending-extents
Yingxin [Tue, 21 Mar 2023 07:55:27 +0000 (15:55 +0800)]
Merge pull request #50602 from xxhdx1985126/wip-invalid-clean-pending-extents

crimson/os/seastore: don't set INVALID extents to CLEAN when reading extents

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agocrimson/os/seastore: don't set INVALID extents to CLEAN when reading 50602/head
Xuehan Xu [Tue, 21 Mar 2023 05:52:43 +0000 (05:52 +0000)]
crimson/os/seastore: don't set INVALID extents to CLEAN when reading
extents

CLEAN_PENDING extents may be invalidated before read completes:

1. transaction A retired an laddr, which lead to a RetirePlaceHolder in
   Cache
2. transaction B try to read that extent, and replace A's
   RetirePlaceHolder with it;
3. transaction A commits and invalidate that extent;
4. transaction B complete reading that extent;

In this case, we shouldn't set the extent's state to CLEAN

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2 years agoMerge pull request #50488 from zdover23/wip-doc-2023-03-13-rados-bluestore-migration...
zdover23 [Mon, 20 Mar 2023 00:59:21 +0000 (10:59 +1000)]
Merge pull request #50488 from zdover23/wip-doc-2023-03-13-rados-bluestore-migration-2-of-x

doc/rados: edit operations/bs-migration (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agoMerge pull request #50487 from zdover23/wip-doc-2023-03-12-rados-bluestore-migration
zdover23 [Mon, 20 Mar 2023 00:58:55 +0000 (10:58 +1000)]
Merge pull request #50487 from zdover23/wip-doc-2023-03-12-rados-bluestore-migration

doc/rados: edit operations/bs-migration (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agoMerge pull request #50579 from zdover23/wip-doc-2023-03-18-rados-configuration-mon...
zdover23 [Sun, 19 Mar 2023 07:56:22 +0000 (17:56 +1000)]
Merge pull request #50579 from zdover23/wip-doc-2023-03-18-rados-configuration-mon-lookup-dns

doc/rados: line edit mon-lookup-dns top matter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/rados: line edit mon-lookup-dns top matter 50579/head
Zac Dover [Sat, 18 Mar 2023 05:27:54 +0000 (15:27 +1000)]
doc/rados: line edit mon-lookup-dns top matter

Improve the syntax of the top matter of
doc/rados/operations/mon-lookup-dns.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2 years agoMerge pull request #49443 from cbodley/wip-qa-supported-distros
Casey Bodley [Fri, 17 Mar 2023 19:36:41 +0000 (15:36 -0400)]
Merge pull request #49443 from cbodley/wip-qa-supported-distros

qa/distros: add ubuntu 22 as supported distro

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2 years agoMerge pull request #50498 from rishabh-d-dave/fs-qa-caps-helper-minor
Rishabh Dave [Fri, 17 Mar 2023 18:23:27 +0000 (23:53 +0530)]
Merge pull request #50498 from rishabh-d-dave/fs-qa-caps-helper-minor

qa/cephfs: minor corrections in caps_helper.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2 years agoMerge pull request #50556 from aravind-wdc/fix-header
Yingxin [Fri, 17 Mar 2023 14:07:40 +0000 (22:07 +0800)]
Merge pull request #50556 from aravind-wdc/fix-header

crimson: fix build error

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agoMerge pull request #50559 from petrutlucian94/improve_win32_clock_check
Ilya Dryomov [Fri, 17 Mar 2023 12:30:12 +0000 (13:30 +0100)]
Merge pull request #50559 from petrutlucian94/improve_win32_clock_check

common: avoid redefining clock type on Windows

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agoMerge PR #50544 into main
Venky Shankar [Fri, 17 Mar 2023 08:55:42 +0000 (14:25 +0530)]
Merge PR #50544 into main

* refs/pull/50544/head:
test/libcephfs: skip flaky timestamp assertion on Windows

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
2 years agoMerge pull request #50106 from rhcs-dashboard/new-landing-page-and-toggle-button
Nizamudeen A [Fri, 17 Mar 2023 08:31:29 +0000 (14:01 +0530)]
Merge pull request #50106 from rhcs-dashboard/new-landing-page-and-toggle-button

mgr/dashboard: add button toggle to switch between landing pages

2 years agocommon: avoid redefining clock type on Windows 50559/head
Lucian Petrut [Thu, 16 Mar 2023 14:25:50 +0000 (14:25 +0000)]
common: avoid redefining clock type on Windows

mingw >= 8.0.1 defines CLOCK_REALTIME_COARSE, so we'll avoid
overriding it if already set.

Clock precision [1]:

mingw < 8.0.1:
  * CLOCK_REALTIME: ~10-55ms (GetSystemTimeAsFileTime)
mingw >= 8.0.1:
  * CLOCK_REALTIME: <1us (GetSystemTimePreciseAsFileTime)
  * CLOCK_REALTIME_COARSE: ~10-55ms (GetSystemTimeAsFileTime)

* CLOCK_MONOTONIC: <1us if TSC is usable, ~10-55ms otherwise
                   (QueryPerformanceCounter)

[1] https://github.com/mirror/mingw-w64/commit/dcd990ed423381cf35702df9495d44f1979ebe50

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2 years agoMerge pull request #50563 from zdover23/wip-doc-2023-03-17-rados-operations-upmap
zdover23 [Fri, 17 Mar 2023 05:54:32 +0000 (15:54 +1000)]
Merge pull request #50563 from zdover23/wip-doc-2023-03-17-rados-operations-upmap

doc/rados: line-edit upmap.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agodoc/rados: line-edit upmap.rst 50563/head
Zac Dover [Thu, 16 Mar 2023 17:41:08 +0000 (03:41 +1000)]
doc/rados: line-edit upmap.rst

Edit all of doc/rados/upmap.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2 years agoMerge pull request #50564 from Thingee/update-foundation-202303
Anthony D'Atri [Thu, 16 Mar 2023 22:32:42 +0000 (18:32 -0400)]
Merge pull request #50564 from Thingee/update-foundation-202303

docs: Update Foundation members for March 2023

2 years agodocs: Update Foundation members for March 2023 50564/head
Mike Perez [Thu, 16 Mar 2023 22:23:55 +0000 (15:23 -0700)]
docs: Update Foundation members for March 2023

Signed-off-by: Mike Perez <thingee@gmail.com>
2 years agocrimson: fix build error 50556/head
Aravind Ramesh [Thu, 16 Mar 2023 07:02:39 +0000 (12:32 +0530)]
crimson: fix build error

With -DWITH_SEASTAR=ON, build is failing.

src/crimson/osd/osd_operations/snaptrim_event.cc:174:29: error: ‘sleep’ is not a member of ‘seastar’

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>
2 years agotest/libcephfs: skip flaky timestamp assertion on Windows 50544/head
Lucian Petrut [Wed, 15 Mar 2023 09:04:40 +0000 (09:04 +0000)]
test/libcephfs: skip flaky timestamp assertion on Windows

There's a new libcephfs test that creates a snapshot and
compares ctime/mtime. The issue is that one of the assertion
fails on Windows, potentially due to reduced timestamp
precision.

For now, we'll skip those assertions on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2 years agomgr/dashboard: add button toggle to switch between new and old landing page 50106/head
Pedro Gonzalez Gomez [Tue, 14 Feb 2023 10:21:57 +0000 (11:21 +0100)]
mgr/dashboard: add button toggle to switch between new and old landing page

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
2 years agoMerge pull request #50534 from neha-ojha/wip-release-cl
Neha Ojha [Wed, 15 Mar 2023 19:50:42 +0000 (12:50 -0700)]
Merge pull request #50534 from neha-ojha/wip-release-cl

doc/dev/release-checklists.rst: add after dev freeze section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agotest/librados/snapshots_cxx: Enable OrderSnap for Crimson 50546/head
Matan Breizman [Wed, 15 Mar 2023 16:04:42 +0000 (16:04 +0000)]
test/librados/snapshots_cxx: Enable OrderSnap for Crimson

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2 years agocrimson/osd/osd_operations/client_request: Support librados::OPERATION_ORDERSNAP
Matan Breizman [Wed, 15 Mar 2023 15:52:16 +0000 (15:52 +0000)]
crimson/osd/osd_operations/client_request: Support librados::OPERATION_ORDERSNAP

Signed-off-by: Matan Breizman <mbreizma@redhat.com>