osd: refuse to boot if SORTBITWISE not set but backend cannot sort nibblewise

This prevenents OSDs with new backends from starting if the
cluster is still using the legacy sort order.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prevent old OSDs from starting if 'sortbitwise' is enabled

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: osd set/unset sortbitwise

Add monitor command to flip the switch on the OSD hobject_t sort
order.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: sort bitwise on new clusters

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: select interval sort order based on OSDMap

If the OSDMap says sort bitwise, sort bitwise... unless we have
enabled osd_debug_randomize_hobject_sort_order, in which case
sometimes we won't.

If the OSDMap doesn't say sort bitwise, don't.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pg_interval_t: a change in sort order triggers a new interval

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add a SORTBITWISE OSDMap flag

This flag will indicate that hobject_t's shall hence-forth be
sorted in a bitwise fashion.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: fix filter lifecycle in do_osd_ops

If we take an error path in [N]PGLS (objects_list_partial error
or bad handle) and FAILOK is set we may exec a second filter op
that reallocates a new filter without freeing the old one.

Simplify by freeing only at the end of the function or when a new
filter is (re)allocated.

Signed-off-by: Sage Weil <sage@redhat.com>

fix 'rados ls' issue if using keyvaluestore as objectstore backend

Fixes: #12637
Signed-off-by: xinxin shu <xinxin.shu@intel.com>

osd: re-sort object_contects when the sort order changes

Scrub uses the object_contexts sort order to check whether it is
safe to extend the scrub interval (verifying there aren't ops in
flight).  For this to work we need to use the correct sort order
for the object_context map (as seen by SharedLRU<>::get_next()).

Signed-off-by: Sage Weil <sage@redhat.com>

common/shared_cache: allow comparator to be adjusted

Allow the use of a comparator that has a value, and for that
comparator to be adjusted at runtime for the purposes for get_next
sort order.  Since get_next uses the weak_refs map, that is the
only map that needs to be rebuilt.

We assume, for simplicity, that the comparator can be constructed
with no arguments with some default... otherwise this is messier.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: fix forced backfill when last_backfill sort order doesn't match

We need to force a backfill restart even if last_update is up to date.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: starting or continuing backfill .. be more precise

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: do some new interval work on init() too

Some work we need to do on any new interval, *including* when the PG is
first created and there is no "interval change".  Factor out the (mostly
new) bits of start_peering_interval() that also need to happen on the
initial interval after the PG is first instantiated into on_new_interval().

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: assert map and set sort order is correct during backfill

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: better debug on recover_backfill

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: fix comparator resets

We need to take care not to clear() any of the maps or sets or else we'll lose
the comparator.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: dynamically vary sort order for backfill sets and maps

Reset the map and set so that the comparator matches the sort order at the
start of each backfill.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: dynamically vary sort order of BackfillInterval map

We initialize all BackfillIntervals when backfill starts.  At that point we
set the Comparator order via reset().

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: add decode_noclear() variant for map<> decode with comparator

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: note NIBBLEWISE in PG debug string

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: use hash instead of filestore key for scrub

It's doesn't matter, since it's a deterministic permutation of hte value
and we're checking for equality, but avoid the filestore-specific
nibblewise key to avoid confusion.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: fix a bit of whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add and use operator<< for BackfillInterval

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: drop old hobject_t encoding compatibility

This was introduced in 0.47 ... we don't need it any more!

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: conditional Comparator object

This one can go either bitwise or nibblewise, depending on what you give
pass to the STL container constructor.  E.g.,

 map<hobject_t, int, hobject_t::Comparator> mymap(hobject_t::Comparator(bitwise));

Signed-off-by: Sage Weil <sage@redhat.com>

use explicit ghobject_t comparators

Signed-off-by: Sage Weil <sage@redhat.com>

use explicit hobject_t comparators

Signed-off-by: Sage Weil <sage@redhat.com>

explicitly specify comparator for all ghobject_t maps and sets

We no longer have operator<

Signed-off-by: Sage Weil <sage@redhat.com>

explicitly specify comparator for all hobject_t maps and sets

This is necessary now that operator< is not defined.

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: typed MIN_* helpers

Signed-off-by: Sage Weil <sage@redhat.com>

common/simple_cache: parameterize map<> comparator

Signed-off-by: Sage Weil <sage@redhat.com>

common/sharedptr_registry: parameterize map<> comparator

Signed-off-by: Sage Weil <sage@redhat.com>

common/shared_cache: parameterize the map<> comparator

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: handle set<> with comparator

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: operator<< for set with comparator

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: map operator<< with comparator

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: map<> encoders when comparator is specified

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject_t: remove comparison operators for [g]hobject_t

*All* comparisions, including everything in the OSD, needs to
conditionally use either bitwise or nibblewise sort order.  Force the
issue by using the explicit cmp methods.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: ignore CRUSH_ITEM_NONE when calculating interval features

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow sort order to randomly revert to nibblewise via debug config

Note that this option is misused and applied to some but not all OSDs they
will disagree on the sort order and get confused, potentially losing data.
It must be global to the cluster in order to be useful.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: do not trust last_backfill for purposes of missing objects if sort order is wrong

If the peer has the wrong sort order, we cannot trust lsst_backfill and we
do not know if the object is there.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: restart backfill on peers if last_backfill sort is off

If a peer has a last_backfill value but the wrong sort order, we need to
restart backfill at the beginning.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use accessor to set last_backfill

Always will in last_backfill and last_backfill_bitwise together.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: set acting, upacting features from osdmap, not peers

Set in start_peering_interval based on the osdmap, so that it is defined
deterministically on primary, replica, and stray.

Note that peer features are still a primary-only construct.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: clean up pg_info_t comments a bit

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_backfill_bitwise flag to pg_info_t

We need to know what sort order last_backfill is based on.

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: pad hash to 8 digits

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: call+test new collection_list variants

In particular, make SimpleListTest check both sorting methods and asssert
that the result is sorted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/HashIndex: handle new bitwise sorting

Handle listing by hash with bitwise sorting.  This involves futzing with
the hex digit sort order by reversing the digit nibbles and so on.

Signed-off-by: Sage Weil <sage@redhat.com>

os/HashIndex: handle legacy nibblewise sort

We need to separate the bitwise and nibblewise implementations because
they diverge too much.  Start by converting the old code to a working
nibblewise sort (compensating for the new ghobject_t sort and new args).

Signed-off-by: Sage Weil <sage@redhat.com>

os/LFNIndex: pass through sort_bitwise

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: pass sort_bitwise down to CollectionIndex

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: use bitwise sort when we don't care (e.g., split)

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueStore: only support bitwise sort

Signed-off-by: Sage Weil <sage@redhat.com>

os/MemStore: only support bitwise sort order

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGBackend: ask PG which sort order to use

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use bitwise collection_list sort when we don't care

Signed-off-by: Sage Weil <sage@redhat.com>

qa: fix misc collection_list callers

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objecstore-tool: use new api

Signed-off-by: Sage Weil <sage@redhat.com>

os: add sort type to ObjectStore interface (incomplete)

Note that this doesn't build, but it will be easier to review this way.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add OSD_BITWISE_HOBJ_SORT feature

We will sort [g]hobject_t's bitwise (instead of nibblewise) if all
OSDs who have participated in peering support the feature.

Note that this means a latecomer PG who does a notify may not have
the feature while the acting set operates with the new sort order.
We will need to be careful about last_backfill in this case.. that patch
it coming.

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: restore default comparators ... bitwise!

Note that this breaks compatibility for the moment.

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: nibblewise and bitwise comparators

Signed-off-by: Sage Weil <sage@redhat.com>

os/LFNIndex: return vector from list_subdirs

We'll need this later when we sort the subdirs bitwise instead of
nibblewise.

Signed-off-by: Sage Weil <sage@redhat.com>

os/HashIndex: drop unused lower_bound arg

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: fix omap_rmkey_range

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5331 from liewegas/wip-collection-list

os: clean up collection_list

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5271 from wonzhq/cache-rate

mon: add cache tier IO rate in the 'osd pool stats' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5507 from athanatos/wip-12436

blkdev.cc::get_device_by_uuid: do not leak cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5506 from athanatos/wip-12338

OSD: break connection->session->waiting message->connection cycle

Reviewed-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop get_ideal_list_min()

Unused.  Fix comment on get_ideal_list_max while we are here.

Signed-off-by: Sage Weil <sage@redhat.com>

os: drop snapid_t arg to collection_list

All this does is filter out objects with old snapids, but this can be
just as easily done in the one caller that specifies a non-zero value
(PGLS and PGNLS), which in fact already do that filtering.

Signed-off-by: Sage Weil <sage@redhat.com>

s/collection_list_impl/collection_list/

Signed-off-by: Sage Weil <sage@redhat.com>

Kill collection_list in CollectionIndex

Now we use CollectionIndex.collection_list_partial.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Kill Flat_index.

Too old, remove it entirely.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Kill collection_list

use collection_list_impl as well.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Kill collection_list_partial

Use collection_list_impl directly.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Kill collection_list_range

We can directly use collection_list_impl.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/KeyValuestore:Refactor collection_list_range and collection_list_partial

Add collection_list_impl which abstract the common process
of collection list behavior.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/CollectionIndex: use const ref for end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5509 from ceph/wip-12536

common/hobject_t: correctly decode pre-infernalis hobject_t min

Reviewed-by: Samuel Just <sjust@redhat.com>

os/Filestore:Refactor collection_list_range and collection_list_partial

Add collection_list_impl which abstract the common process
of coleection list behavior.

Todo: Refactor Index::collection_list_partial as
well.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/Memstore:Refactor collection_list_range and collection_list_partial

Add collection_list_impl which abstract the common process
of collection list behavior.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #5508 from athanatos/wip-12410

OSDMonitor::preprocess_get_osdmap: send the last map as well

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5413 from guangyy/wip-12523

osd: copy the RecoveryCtx::handle for when creating RecoveryCtx instance from another one

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5488 from oritwas/wip-cmake-fix-rbd

cmake: add global lib to rbd

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5475 from cxwshawn/com-fix

rgw_user.h: modify interface comments.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5483 from oddomatik/patch-1

Fix location of --access=full in SWIFT user creation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5484 from cxwshawn/de-fix

ceph-dencoder: add RGWRegion, RGWZoneParams, RGWOLHInfo support.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5421 from ceph/wip-selinux-policy-no-user

SELinux support for ceph

Reviewed-by: Kefu Chai <kchai@redhat.com>
Tested-by: Yuri Weinstein <yweinstei@redhat.com>

Merge pull request #5487 from ceph/wip-12619

mds: initialize InodeStoreBase::damage_flags

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: add global lib to rbd

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

mds: initialize InodeStoreBase::damage_flags

Fixes: #12619
Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph-dencoder: add RGWRegion, RGWZoneParams, RGWOLHInfo support.

    since ceph-dencoder does not support the up-list structures, so
    I add these three supports to analyze data more quickly.

Signed-off-by: shawn chen <cxwshawn@gmail.com>

Fix location of --access=full in SWIFT user creation

--access=full was incorrectly placed in the output of the command, while it was supposed to have been a flag run with the command.

ceph.spec.in: Make SELinux opt-out, not opt-in

We should enable SELinux support by default on rpm-based systems that
build with SELinux support to get better QE coverage in future runs.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5478 from dachary/wip-test-rados-striper

tests: rados striper tests use 7116 instead of 7113

Reviewed-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #5459 from cxwshawn/master

radosgw-admin: use cout instead of cerr to print help message.

Reviewed-by: Sage Weil <sage@redhat.com>

selinux: Add .gitignore file

The gitbuilders release script needs this. Otherwise, the ceph-release
build will fail because there were some untracked files.

Signed-off-by: Boris Ranto <branto@redhat.com>