Sage Weil [Tue, 2 Nov 2021 15:41:53 +0000 (11:41 -0400)]
Merge PR #42762 into master
* refs/pull/42762/head:
ceph_test_objectstore: skip BlueStoreUnshareBlobTest with SMR
os/bluestore: debug ExtentMap::update()
os/bluestore: _txc_create inside of alloc_and_submit_lock
os/bluestore: fix cleaner race with collection removal
os/bluestore: add missing ' ' to LruOnodeCacheShare _[un]pin
os/bluestore: use simpler map<> to track (onode, zone) -> offset
os/bluestore: avoid casting zoned implementations again
os/bluestore/ZonedFreelistManager: remove sanity checks
os/bluestore/ZonedAllocator: fix allocate() search
os/bluestore: drain transactions on cleaner zone finish
os/bluestore/ZonedFreelistManager: simplify freelist merge update vs zone reset
os/bluetore: configurable sleep period for cleaner
blk/zoned: make discard a no-op
os/bluestore/ZonedAllocator: count sequential only as 'free'
os/bluestore: expect smr fields IFF device is smr
ceph_test_objectstore: Test for fixing write pointer
ceph_test_objectstore: complain if SMR support not compiled in
test/objectstore/run_smr_bluestore_test.sh
os/bluestore/ZonedAllocator: handle alloc/release spanning zones
os/bluestore: simple cleaner
os/bluestore: be smarter about picking a zone to clean
os/bluestore: avoid writes to cleaning zone
os/bluestore/HybridAllocator: whitespace in debug output
os/bluestore: give conventional region of SMR to bluefs
os/bluestore: separate alloc pointer from shared_alloc.a
test/objectstore/run_smr_bluestore_test.sh
ceph_test_objectstore: skip tests that don't work on SMR
os/bluestore: disable cleaner thread until it is implemented
os/bluestore: fsck verify zone refs
os/bluestore: include object in zone ref keys
os/bluestore: refactor object key helpers a bit
ceph_test_objectstore: skip failing tests on SMR
os/bluestore: report mismatch write pointer during fsck
os/bluestore: simplify zone to clean selection
ceph_test_objectstore: add trivial fsck test
os/bluestore: fsck smr allocations (verify num_dead_bytes, alloc past write pointer)
os/bluestore: duplicate zone refs when cloning
os/bluestore: correct zoned freelist when device write pointers are ahead
os/bluestore/ZonedFreelistManager: whitespace
os/bluestore: fix startup vs device write pointers
blk/zoned: add get_zones() to fetch write pointers
os/bluestore: use 64 bit values for zone_state_t
os/bluestore: reimplement zone backrefs
os/bluestore: fix smr allocator init
os/bluestore: do not use null freelist with SMR
blk/zones: implement HMSMRDevice has KernelDevice child
os/bluestore: fix/simplify zoned_cleaner thread start error handling
os/bluestore: properly reset zoned allocator on startup
os/bluestore: force prefer_deferred_size=0 for smr
os/bluestore: drop SMR 64K min_alloc_size restriction
os/bluestore/ZonedAllocator: less verbose
os/bluestore/ZonedAllocator: simplify debug output prefix
os/bluestore/ZonedAllocator: be consistent with hex debug output
os/bluestore/ZonedAllocator: whitespace
blk/zoned: remove dead VDO code
blk/zoned: add reset_all_zones()
blk/zoned: print error during init
os/bluestore: adjust allocator+freelist interfaces for smr params
os/bluestore: select 'zoned' freelistmanager during mkfs, not mount
Laura Flores [Tue, 2 Nov 2021 00:31:16 +0000 (00:31 +0000)]
doc/dev/developer_guide/testing_integration_tests: update "frequently used options"
The `subset` option is important in Teuthology runs for reducing the number of tests that are triggered. This option is outlined in another part of the Teuthology documentation, but I think it's important to place here as well.
Also, -n (for how many times the job will run) is incorrect; it should be -N.
Curt Bruns [Thu, 21 Oct 2021 21:53:31 +0000 (17:53 -0400)]
rgw/lua: Example read/write of StorageClass field
Admins may setup different pools for RGW objects and
having the StorageClass field mutable allows the steering
of RGW objects to the proper pools. This Lua example shows
how a user can modify the StorageClass header when
it is empty on a PUT request and steer objects to different
pools based on size (Request.ContentLength).
Jianpeng Ma [Mon, 1 Nov 2021 01:25:52 +0000 (09:25 +0800)]
librbd/cache/pwl: fix reorder when flush cache-data to osd.
Consider the following workload:
writeA(0, 4096)
writeB(0, 512).
pwl can makre sure writeA persist to cache before writeB.
But when flush to osd, it use async-read to read data from cache and in
the callback function they issue write to osd.
So although we by order issue aio-read(4096), aio-read(512). But we
can't make sure the return order.
If aio-read(512) firstly return, the write order to next layer is
writeB(0, 512)
writeA(0, 4096).
This is wrong from the user point.
To avoid this occur, we should firstly read all data from cache. And
then send write by order.
Fiexs: https://tracker.ceph.com/issues/52511
Tested-by: Feng Hualong <hualong.feng@intel.com> Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Jianpeng Ma [Mon, 1 Nov 2021 00:33:23 +0000 (08:33 +0800)]
librbd: send FLUSH_SOURCE_INTERNAL when do copy/deep_copy.
copy/deep_copy use object_map to judge whether object exist.
If w/ librbdo pwl cache, flush can't flush data to osd which
change objectmap state. So we should send flush w/ FLUSH_SOURCE_INTERNAL
to make data flush to osd.
Fixes:https://tracker.ceph.com/issues/53057 Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Zac Dover [Sat, 30 Oct 2021 15:56:44 +0000 (01:56 +1000)]
doc/cephadm: add admonition for tracker 53062
NOTE: This commit also adds an admonition to the pacific.rst
page, in the /releases directory.
This commit warns users not to upgrade to Ceph v16 (Pacific). Upgrading
to v16 (Pacific) can cause data corruption.
This commit has been made in response to https://tracker.ceph.com/issues/53062
and in response to Stefan Kooman's urgent request that we update the
documentation, adding a warning to the documentation to prevent users
from upgrading to v16.2.6.
Sage Weil [Thu, 23 Sep 2021 15:23:11 +0000 (10:23 -0500)]
os/bluestore/ZonedAllocator: handle alloc/release spanning zones
The same txn may allocate the end of one zone and the start of the next.
These get combined in an interval_set in bluestore and submitted as one
extent to the freelist. Ensure that we spread the allocate (or release)
over the appropriate zones.
Sage Weil [Fri, 17 Sep 2021 19:14:51 +0000 (14:14 -0500)]
os/bluestore: simple cleaner
Rewrite object ranges to move by reusing _do_read and _do_write. Note that
this will rewrite shared/cloned extents multiple times, so it is
suitable only for workloads/datasets that do not use clone (snapshots).
Sage Weil [Thu, 16 Sep 2021 17:09:54 +0000 (12:09 -0500)]
os/bluestore: give conventional region of SMR to bluefs
Let bluefs use the conventional region of an SMR HDD. It won't be able to
use the sequential region if it runs out of space, but it does mean that
bluefs can function on an SMR drive without a dedicated db/wal device.
Sage Weil [Wed, 8 Sep 2021 16:37:26 +0000 (11:37 -0500)]
os/bluestore: simplify zone to clean selection
Only pick one zone to clean based on the current. Since the best victim
may change (maybe another zone gets a bunch of releases and new dead
bytes!) there is no reason (yet) to explicitly avoid the victim zone
during allocation. There is also no need to track which zones we are
cleaning on disk because we can choose to clean from any zone at any time,
and in general want to clean from the best candidate at the time, not the
one that looked the best some time in the past.
projects / ceph.git / log