os/bluestore: fix cleaner race with collection removal

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: add missing ' ' to LruOnodeCacheShare _[un]pin

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: use simpler map<> to track (onode, zone) -> offset

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: avoid casting zoned implementations again

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedFreelistManager: remove sanity checks

These were there when I was using the merge operator; not really needed
since it's just setting zeroed values now.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedAllocator: fix allocate() search

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: drain transactions on cleaner zone finish

Make sure all open transactions (the ones from the cleaner especially)
get drained before we clear the device zone.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedFreelistManager: simplify freelist merge update vs zone reset

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluetore: configurable sleep period for cleaner

Signed-off-by: Sage Weil <sage@newdream.net>

blk/zoned: make discard a no-op

Discard is meaningless on SMR or ZNS since we are always explicitly
managing the reset of entire zones.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedAllocator: count sequential only as 'free'

Completely ignore the conventional region of the device.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: expect smr fields IFF device is smr

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_test_objectstore: Test for fixing write pointer

Test for checking if bluestore mount() can nicely handle unexpected
change of SMR sequential zone write pointer.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

ceph_test_objectstore: complain if SMR support not compiled in

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

test/objectstore/run_smr_bluestore_test.sh

Added execution permissions.
Modified script to account for different lsscsi outputs (7th column was not device path on Ubuntu 20.04).
Added comments.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore/ZonedAllocator: handle alloc/release spanning zones

The same txn may allocate the end of one zone and the start of the next.
These get combined in an interval_set in bluestore and submitted as one
extent to the freelist.  Ensure that we spread the allocate (or release)
over the appropriate zones.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: simple cleaner

Rewrite object ranges to move by reusing _do_read and _do_write.  Note that
this will rewrite shared/cloned extents multiple times, so it is
suitable only for workloads/datasets that do not use clone (snapshots).

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: be smarter about picking a zone to clean

Base our selection on best "value" (benefit / cost).

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: avoid writes to cleaning zone

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/HybridAllocator: whitespace in debug output

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: give conventional region of SMR to bluefs

Let bluefs use the conventional region of an SMR HDD.  It won't be able to
use the sequential region if it runs out of space, but it does mean that
bluefs can function on an SMR drive without a dedicated db/wal device.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: separate alloc pointer from shared_alloc.a

For the moment, these are always the same, so no change in functionality
form this patch.  In a future patch these may be different.

Signed-off-by: Sage Weil <sage@newdream.net>

test/objectstore/run_smr_bluestore_test.sh

Needs sudo/root to set up.

Requires tcmu-runner and targetcli be installed (and a sufficiently
new kernel, not sure what the exact version is!).

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_test_objectstore: skip tests that don't work on SMR

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: disable cleaner thread until it is implemented

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: fsck verify zone refs

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: include object in zone ref keys

We may have multiple objects referencing the same offset in the same
zone if the object gets cloned.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: refactor object key helpers a bit

We want methods that work on partial keys.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_test_objectstore: skip failing tests on SMR

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: report mismatch write pointer during fsck

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: simplify zone to clean selection

Only pick one zone to clean based on the current.  Since the best victim
may change (maybe another zone gets a bunch of releases and new dead
bytes!) there is no reason (yet) to explicitly avoid the victim zone
during allocation.  There is also no need to track which zones we are
cleaning on disk because we can choose to clean from any zone at any time,
and in general want to clean from the best candidate at the time, not the
one that looked the best some time in the past.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph_test_objectstore: add trivial fsck test

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: fsck smr allocations (verify num_dead_bytes, alloc past write pointer)

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: duplicate zone refs when cloning

When we clone an object, add zone refs pointing to the new object for
any relevant extents.

This is only half of the solution; we will want to make sure the cleaner
is clever enough to keep shared blobs shared when they are relocated.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: correct zoned freelist when device write pointers are ahead

If we find that the device's write pointers are ahead of ours, then we
need to advance the allocator *and* update the ondisk freelist so that we
don't see the adjustment on the next bluestore restart too.

(Actually, we could perhaps not manage the freelist at all and rely
entirely on the device's write pointers and a metadata scan on startup
w/ the null freelist mode, but it is cheap to track, and I like having
some redundancy so that we see if/when the device does something
unexpected.)

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedFreelistManager: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: fix startup vs device write pointers

Compare freelist write pointers to the device on startup.

Signed-off-by: Sage Weil <sage@newdream.net>

blk/zoned: add get_zones() to fetch write pointers

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: use 64 bit values for zone_state_t

Simpler and future-proof for devices with zones >= 4GB.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: reimplement zone backrefs

Implement a (zone, offset) -> oid reference.  The offset is the first
offset referenced by this object in this zone; subsequent extents in the
same zone will not generate additional refs.  The ref will be removed only
when *all* refs for that object to that zone are gone.

- minimal metadata per zone (one ref to each object involved
- (zone, offset) keys are ordered for efficient sequential cleaning

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: fix smr allocator init

- initialize from device's pointers
- cross check against freelist manager's pointers (warn!)
- fix init_{add,rm}_free methods to do little/nothing (the other init
  method does most of what we need)

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: do not use null freelist with SMR

Signed-off-by: Sage Weil <sage@newdream.net>

blk/zones: implement HMSMRDevice has KernelDevice child

No need to duplicate so much code when we are just adding a few things.
Also, we want to track KernelDevice changes/improvements.

We could probably integrate these SMR capabilities directly into
KernelDevice too...

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: fix/simplify zoned_cleaner thread start error handling

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: properly reset zoned allocator on startup

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: force prefer_deferred_size=0 for smr

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: drop SMR 64K min_alloc_size restriction

used_blocks is only for fsck.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedAllocator: less verbose

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedAllocator: simplify debug output prefix

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedAllocator: be consistent with hex debug output

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore/ZonedAllocator: whitespace

Signed-off-by: Sage Weil <sage@newdream.net>

blk/zoned: remove dead VDO code

VDO won't work on an SMR device

Signed-off-by: Sage Weil <sage@newdream.net>

blk/zoned: add reset_all_zones()

Signed-off-by: Sage Weil <sage@newdream.net>

blk/zoned: print error during init

Otherwise it is easy to miss things like EPERM during testing.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: adjust allocator+freelist interfaces for smr params

Instead of shoehorning these fields into alloc_size, adjust the inferfaces
to explicitly pass in zone_size and first_sequential_zone for
Allocator and FreelistManager.

Signed-off-by: Sage Weil <sage@newdream.net>

os/bluestore: select 'zoned' freelistmanager during mkfs, not mount

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #43709 into master

* refs/pull/43709/head:
qa/tasks/cephfs/test_nfs: wait for fs to come up before exporting

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43602 from sebastian-philipp/cephadm-rgw-ipv6

mgr/cephadm: Fix RGW ipv6 frontend configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #43260 from cfsnyder/wip-43692-repave-osds

mgr/orchestrator: add --zap flag to 'orch osd rm'

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43539 from orozery/ms-nocrc

msg/async: support disabling data crc for protocol v2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #31877 from rosinL/wip-fix-dpdk-link

cmake: link against DPDK shared libraries to avoid DPDK EAL double initialization

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43693 from batrick/i53074

pybind/mgr/cephadm: upgrade MDS if no MDS is "up"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43293 from pcuzner/granular-alerts

mgr/prometheus: expose ceph healthchecks as metrics

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #43583 into master

* refs/pull/43583/head:
changes to logging code requested by Adam
remove allocation file after rollback to real-fm
bug-fix close db when umount failed to store allocation file
Bug-Fix: When restoring allocation from file use a temp allocator and only copy the allocation to the shared-allocator after the file was verified and all extents were cleared

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #43645 into master

* refs/pull/43645/head:
os/bluestore: Disable compaction then no-column-b is storing allocations to bluefs file

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Gabriel Benhanokh <gbenhano@redhat.com>

qa/tasks/cephfs/test_nfs: wait for fs to come up before exporting

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41325 from rhcs-dashboard/wip-cephadm_box-master

cephadm: add cephadm sandbox (Docker in Docker)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #43692 from liewegas/misc-rook-1

mgr/orchestrator: log exception from wrapper

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43330 from sebastian-philipp/CEPHADM_STRAY_DAEMON

doc/cephadm/operations: extend CEPHADM_STRAY_DAEMON

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #43601 from sebastian-philipp/doc-cephadm-osd-similar-disk-layout

doc/cephadm: improve "Deploying OSDs on multiple similar hosts"

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43622 from Daniel-Pivonka/cephadm-doc-osd-replace-host

doc/cephadm: note that replacment osd needs to be on same host as removed osd

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43681 from cyx1231st/wip-seastore-fix-test-setup-teardown

crimson/os/seastore: initialize tests in seastar reactor

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

cephadm/box: add no-tmpfs flag prepare osd bluestore

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

pybind/mgr/cephadm: upgrade MDS if no MDS is "up"

The upgrade process can get stuck if an MDS crashes. This should be rare
when straddling v16.2.5 where the compatset of the file system inherits
the FSMap "default". The MDS from pre-v16.2.5 do not yet share a
compatset with the mons so the mons will do no promotions, causing
upgrade task to get stuck.

Fixes: https://tracker.ceph.com/issues/53074
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43689 from rzarzynski/wip-auth-drop-get_supported_con_modes

auth: drop unused get_supported_con_modes() from AuthServer.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/orchestrator: log exception from wrapper

This lets us see a proper traceback when we are calling across
modules.

Signed-off-by: Sage Weil <sage@newdream.net>

auth: drop unused get_supported_con_modes() from AuthServer.

In 2d530938753313a776258ff9fa8208db637c9d96 its last user has
been removed.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43684 from ybwang0211/doc-host

doc: The description of "Host" on the page is inconsistent

Signed-off-by: wangyingbin ybwang0211@163.com
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #43581 from cbodley/wip-52800

rgw: fix lock scope in ObjectCache::get()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #43170 from thotz/removeusercaps-in-modifyuser

rgw: remove dead code for user-caps from RGWOp_User_Modify

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43153 from ivancich/wip-add-bucket-listing-log

rgw: add logging to bucket listing so calls are better understood

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42704 from wzbxqt327/patch-1

rgw,doc: fix command problem

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41323 from pkalever/cookie

rbd-nbd: generate and send device cookie with netlink connect request

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43584 from rhcs-dashboard/add-multiple-hosts-at-once

mgr/dashboard: Cluster Creation Add multiple hosts at once

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43605 from rhcs-dashboard/bonding-metrics

monitoring: ethernet bonding filter in Network Load.

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43603 from rhcs-dashboard/fix-gather-facts-fetch-UI

mgr/dashboard: fetch host facts only if get_facts orch feature available

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43661 from rhcs-dashboard/review-page-fix

mgr/dashboard: Cluster expansion review page minor bug fixes

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

crimson/os/seastore: initialize tests in seastar reactor

testing::Test from gtest is constructed and destructed outside seastar
application. It is unexpected for most seastore components, and causing
undefined behaviors.

For example, when registering metrics during construction, the logic
needs to be called inside seastar reactor thread because metrics are
maintained in reactor thread-local variables.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup transaction_manager_test_state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/randomblock-manager: cleanup unit test

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: get block-size from reader

Cache cannot get the correct block size before SegmentManager is
mounted.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/store-nbd: cleanup headers

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: proper close journal when test finishes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc: The description of "Host" on the page is inconsistent

In the "https://docs.ceph.com/en/latest/radosgw/adminops/#radosgw-admin-ops" page, the description of "Host" is not uniform, so I changed it.

Signed-off-by: wangyingbin <ybwang0211@163.com>

monitoring: ethernet bonding filter in Network Load

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43630 from kkang-wr/master

cmake: add support for python 3.10

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43648 from rzarzynski/wip-crimson-proper-addrs-for-hbmsgrs

crimson/osd: fix network address selection for hearbeat's messengers.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #43671 from javicacheiro/python_data_as_binary

doc/rbd: correct sample code to use byte string for data

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43674 from rzarzynski/wip-crimson-net-honor_listen_type

crimson/net: FixedCPUServerSocket::accept() respects the listening addr's type.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/prometheus: remove cmake tests

Temporary removal of the cmake test integration

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>