osd: make coll_t structured instead of a string

coll_t now has a restricted set of representable values:

- meta
- pg
- pg temp
- forremoval

The forremoval is for legacy compatibility only; we will be able to remove
it soonish.

Signed-off-by: Sage Weil <sage@redhat.com>

os/MemStore: fix warnings

Signed-off-by: Sage Weil <sage@redhat.com>

osd: simplify load_pgs

No more snap collections.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change coll_t::is_pg() and is_temp() to take a pointer; drop snap

The snap arg isn't used anymore.  The spg_t should be a pointer (not ref)
since it is an output argument.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop old snap_colls arg from PG::upgrade()

No longer used.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: simplify coll_t::is_removal(); fix removal (pass pgid)

We just need a binary check, no metadata or output.

Fix the removal to take a pgid while we are here.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop pre-snapmapper upgrade

We dropped support for this back in 9d1466cb6e1f8e81b6a225279fe3aeb7878692b3.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: drop coll_t string ctor

No more users!

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueStore: change collection master list strategy

Don't use a pseudo-collection COLLECTIONS.  We aren't allowed to make
strangely-named coll_t's any more.  Instead, store the collection list in
a simple encoded buffer that's stored as a single key in the backend db.

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueStore: drop collection attr support

Deprecated!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_smalliobenchfs: use valid collection names

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: only recognized valid collection names

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add coll_t::parse() method

This will explicitly validate the form of the input string to make sure it
is a recognized collection.  (Eventually we can then store things
internally as something other than a string.)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: adjust tests to new sorting regime

A few things here:

 - try to use pg-based collections throughout
 - try to match object names to collection pool and shard

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: be better about collections

Use meta collection or pg collections.  Try to make ghobject's pools match
the pg collections.

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueStore: change naming scheme to work with new ghobject_t sorting

Note that this is definitely not backward compatible!

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: fix remove_collection return value

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: better debug output for destroy_collection

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: force temp objects into _TEMP temp collection

We've removed the temp collection concept from the ObjectStore interface,
but the FileStore HashIndex will mix objects with different pools together
by hash id, which breaks the ordering.

Compensate by forcing objects with 'temp' pool ids (anything < -1) into
a parallel temp collection.  Hide this detail isnide FileStore, below the
ObjectStore interface and above the HashIndex backend.

Signed-off-by: Sage Weil <sage@redhat.com>

shard_id_t: change NO_SHARD to sort before 0 (min instead of max)

The min ghobject has shard NO_SHARD, and is the default constructed value.
That initial value is assumed in uncounted ways across the code base when
users do

  ghobject_t foo;
  foo.this = that;

such that changing it is dangerous.  It is safer to change the shard_id_t
sort order such that NO_SHARD is signed instead of unsigned.  The value
doesn't actually change (still 0xff), but the sorting does.  Note that
only a single comparison triggers a signed/unsigned warning from this
change, and it assumes that the shard is not NO_SHARD (ec pool) and we
case it to preserve the old behavior anyway.

In PGBackend we change the minimum value for the objects_list_partial()
method to start with NO_SHARD.

Signed-off-by: Sage Weil <sage@redhat.com>

ghobject_t: is_min()

Signed-off-by: Sage Weil <sage@redhat.com>

ghobject_t: make operator<< put shard at front

Signed-off-by: Sage Weil <sage@redhat.com>

osd: some simple osd type sanity tests

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add coll_t is_temp() and is_meta()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use explicit coll_t::make_string_coll() ctor

Call out those creating collections with strings.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use non-string coll_t's

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use coll_t() for meta collection

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add coll_t::get_temp() helper

Generate the temp collection for a pg collection.

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: remove unused legacy split/create collection methods

These are no longer needed or called.

Signed-off-by: Sage Weil <sage@redhat.com>

ghobject_t: MIN for default

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: MIN for default

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_objectstore_test: a few simple collection_list_partial tests

Add a simple list test, and a second one that mixes different pool ids
into the same collection.  The latter confirms that we can deal with
ghobject_t's in the temp pool space within a single collection.

Signed-off-by: Sage Weil <sage@redhat.com>

ghobject_t: fix max to make embedded hobject_t also max

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: fix get_boundary to work with new sorting regime

The hash is no longer the most significant field; set everything that is
more significant, too.

Signed-off-by: Sage Weil <sage@redhat.com>

ghobject_t: change sort order (max, shard, hobj, gen)

Go from (hobj, shard, gen) -> (max, shard, hobj, gen)

This makes the ghobject_t's sort in a way that groups them by PG, which
will likely be useful in the future for ObjectStore implementations.
Notably, we get a ghobject_t MAX value that is distinct from the
hobject_t one.

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: ignore store_test_temp_dir

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: adjust comparator to check most-likely-different fields first

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: modify operator<<

Put the most significant fields to the left so that it matches the sort
order.  Also use unambiguous separator when the nspace is present
(like we do with the key).

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: adjust sort order (pool position)

Make pool more significant than hash (and nspace, object, and everything
else).

This won't break anything because we only care about hobject_t sort order
within the context of a PG (or possibly split) where pool is always
fixed.

Signed-off-by: Sage Weil <sage@redhat.com>

hobject_t: change default pool id to INT64_MIN

We are including pool in a (more) significant part of the sort and using
negative pool IDs; the default must be the most negative (min) so that
it is a useful starting point for a search.

Signed-off-by: Sage Weil <sage@redhat.com>

os/HashIndex: use ghobject_t::get_max() instead of hobject_t one

We are working with ghobject_t's; use that max!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: eliminate temp collections

The temp objects have distinct pool ids.  Old temp objects are already
blown away on OSD restart.  This patch removes all the futzing with
temp_coll and puts the temp objects in the same collection as everything
else.

Interesting, collection_move_rename is now always using the same source
and dest collection.  Hmm!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove ancient CEPH_*_NS cruft

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use per-pool temp poolid for temp objects

Previously, all temp objects had poolid == -1.  Instead, use -2 - poolid.
This, when combined with the PG hash, provides a unique temp namespace
per pg.

This has no impact on upgrade, since we delete all temp objects on startup
by collection (coll_t::is_temp()).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add --balance-reads option

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: avoid spurious omap_setkeys, setattrs during recovery

Only call these if we have values to set on this iteration.  This is
rarely true for omap.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: deprecated collection_move

You will not be missed!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use a temporary object for recovery

Currently we recover objects directly into position by deleting and then
overwriting the target object.  This means that we lose the object if we
are recovering in multiple steps and we fail partway through.

This is also the last user of collection_move(), which we would like to
deprecate.

Instead, generate a unique temp object name (pgid, object version, snap
is unique), and recover to that.  Use the existing temp object cleanup
machinery to throw out a partial recovery result.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5002 from SUSE/wip_fixes_to_rcceph

Fixes to rcceph script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4983 from yuyuyu101/wip-setallochint-option

librbd: Add option to allow disabling issuing alloc hint

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4993 from ceph/wip-12064

mon: only send MMonMetadata to peer mons that support it

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5005 from maxime1992/master

doc: Document include/str_list.h

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Document include/str_list.h
Fixes: #12050.
Signed-off-by: Claire MASSOT <claire.massot93@gmail.com>
Signed-off-by: Jordan DORNE <jordan.dorne@gmail.com>
Signed-off-by: Kévin CARADANT <kevin.caradant@gmail.com>
Signed-off-by: Gabriel SENTUCQ <perso@kazhord.fr>
Signed-off-by: Maxime ROBERT <maxime.robert1992@gmail.com>

Merge pull request #5017 from JordanDorne/master

doc: Replace requries with requires in python API doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replace requries with requires in python API doc
Fixes: #12076
Signed-off-by: Jordan DORNE <jordan.dorne@gmail.com>

Merge pull request #4989 from ltoto2/master

 doc: Adding a link to documentation on admin socket

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adding a link to documentation on admin socket

Fixes: #11845
Signed-off-by: Thomas Laumondais <thomas.laumondais@gmail.com>

Bug fix to ceph systemV compatability script.

Was failing with more than one OSD / MON deamon on a single node.
Fixes suse bugzilla #927862

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #5011 from robin974/master

doc: Fixes a missing dot

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5006 from Claire1293/master

doc: dev/differences-from-posix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4987 from Aeryax/master

doc: Replaced "disk or drive" by "disk" in ceph-disk.rs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Fixes a missing dot

Fixes: #12072
Signed-off-by: Robin Tang <robintang974@gmail.com>

Merge pull request #5008 from jr31/master

doc: Unify ID format

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5007 from Cleminou/master

doc: Fixes a missing word in a printed string

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: dev/differences-from-posix
Fixes: #11108
Signed-off-by: Claire MASSOT <claire.massot93@gmail.com>

doc: Unify ID format

Fixes: #12071
Signed-off-by: Jean-Rémi Deveaux <jeanremi.deveaux@gmail.com>

doc: Fixes a missing word in a printed string

Fixes: #11106
Signed-off-by: Clement Lebrun <clement.lebrun.31@gmail.com>

Merge pull request #5004 from arthurgorjux/master

doc: Remove `` before and after 'ceph -s' in titles

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Replaced "disk or drive" by "disk" in ceph-disk.rs
Fixes: #12062
Signed-off-by: Gabriel SENTUCQ <perso@kazhord.fr>

Merge pull request #5003 from Oxygen31/master

doc : Change of "create" in "add"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4954 from JevonQ/add-iov-support-in-libcephfs

libcephfs: add ceph_p{read,write}v

Reviewed-by: John Spray <john.spray@redhat.com>

doc: Remove `` before and after 'ceph -s' in titles
Fixes: #12059
Signed-off-by: Arthur Gorjux <arthurgorjux@gmail.com>

doc : Change of "ceph-deploy mon create {ceph-node}" with "ceph-deploy mon add {ceph-node}" in "Quick-ceph-deploy.rst"

Fixes : #10293

Signed-off-by : Germain CHIPAUX <germain.chipaux@gmail.com>

Merge pull request #4986 from nicolasyong/master

doc: Modification of a sentence

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Modification of a sentence.

Fixes : #12057

Signed-off-by: Nicolas Yong <nicolas.yong93@gmail.com>

Merge pull request #4988 from kevincaradant/master

doc: ceph-deploy man page: typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fixes to rcceph script

 - only start OSDs if mon daemons are also present
 - adds support for mask and unmask
 - removes support for cluster with non default cluster name,
   as this was very limited and inconsistent
 - Reapplied from a patch as could not cherry-pick
   66cb46c411d874be009c225450eea5021cf1219b from Mon Jan 12
   as this produced issues with src/gmock

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #4931 from yuyuyu101/wip-perf-msgr

PerfMsgr: A tool to benchmark messenger module

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4994 from liewegas/wip-mon-fixes

mon: fix warning, and get_bucket_utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3566 from majianpeng/librbd-cache-iohint

Librbd cache iohint

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'next'

Merge remote-tracking branch 'origin/wip-11579' into next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'next'

librbd: flush operations need to acquire owner lock

Cache writeback operations will expect the owner lock to be held.

Fixes: #11938
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4990 from ilc/f22-build

Fix the build on Fedora 22.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'next'

Merge remote-tracking branch 'origin/wip-11537' into next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4992 from maxime1992/master

doc: Change the type of list in doc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon/OSDMonitor: fix get_bucket_utilization return value

Broken in cc5f144a51d10f30e71b96c888a8d2a125e7c9b1.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: only send MMonMetadata to peer mons that support it

Avoid sending this to mons that don't support the feature or else they
will reset the connection and we'll go into a busy reconnect loop.

Fixes: #12064
Signed-off-by: Sage Weil <sage@redhat.com>

configure.ac: Fix JUnit 4 detection on Fedora 22.

This allows the build to detect JUnit 4 on Fedora 22, by adding
junit.jar as a valid jar name for JUnit 4.

Signed-off-by: Ira Cooper <ira@samba.org>

doc: Change the type of list in doc
Fixes: #12061.
Signed-off-by: Maxime ROBERT <maxime.robert1992@gmail.com>

Merge pull request #4975 from mikulely/master

Fix typos in librados example code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: ceph-deploy man page: typo #12063
Replace is initial monitor hostname with is the initial monitor hostname
Fixes : #12063
Signed-off-by: CARADANT Kevin <kevin.caradant@gmail.com>

Fix typos in librados example code

Signed-off-by: Jiaying Ren <mikulely@gmail.com>

Merge pull request #4985 from yannick-ad/master

doc: replace a sentence by another

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4979 from GaelFG/patch-1

doc: Fixes the description of the purpose of a cache pool

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4984 from LucasFantinel/master

doc: Wrong restriction for a daemon's ID

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: replace a sentence by another
Fixes: #12060
Signed-off-by: Yannick Atchy-Dalama <yannick.atchy.dalama@gmail.com>

doc: Purpose of a cache pool

Completing the description of the cache purpose.
Fixes : #11096
Signed-off-by: Gael Fenet-Garde <gael.fenet.garde@gmail.com>