PG: use !empty() instead of size() > 0

[src/osd/PG.cc:543]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:1480]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:1774]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:1810]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:2229]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:2599]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:4345]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:6067]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:6264]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:6807]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:7014]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

RPCRecPred: use !empty() instead of size() >= 1

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

KeyValueStore: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

KeyValueStore: use !empty() instead of size()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

FileStore: use !empty() instead of size()

[src/os/FileStore.cc:3635]: (performance) Possible inefficient checking
  for 'omap_set' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

DBObjectMap: use !empty() instead of size()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

OSDMonitor: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

OSDMonitor: use !empty() instead of size() > 0

[src/mon/OSDMonitor.cc:730]: (performance) Possible inefficient checking
  for 'osd_weight' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MDCache: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

CrushWrapper::is_valid_crush_loc(): pass function parameter by reference

[src/crush/CrushWrapper.cc:1412]: (performance) Function parameter 'loc'
  should be passed by reference.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

CrushWrapper: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PGLog.h: prefer prefix ++operator for non-trivial iterator

[src/osd/PGLog.h:363]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.
[src/osd/PGLog.h:375]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.
[src/osd/PGLog.h:380]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.
[src/osd/PGLog.h:382]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Client: remove a std::string::c_str() calls

[src/client/Client.cc:6821]: (performance) Passing the result of c_str() to a
  function that takes std::string as argument no. 2 is slow and redundant.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/config.cc: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

str_map.cc: prefer prefix ++operator for non-trivial iterator

[src/common/str_map.cc:55]: (performance) Prefer prefix ++/-- operators
  for non-primitive types.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Client.cc: use !empty() instead of size()

[src/client/Client.cc:2275]: (performance) Possible inefficient
  checking for 'cap_snaps' emptiness.
[src/client/Client.cc:2458]: (performance) Possible inefficient
  checking for 'cap_snaps' emptiness.
[src/client/Client.cc:2986]: (performance) Possible inefficient
  checking for 'caps' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.cc: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Monitor: pass function parameter by reference

[src/mon/Monitor.h:619]: (performance) Function parameter 'param_str_map'
  should be passed by reference.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

BarrierContext: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

BarrierContext: reduce scope of 'done' variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MonClient: pass function parameter by reference

[src/mon/MonClient.h:412]: (performance) Function parameter 'mon_name'
  should be passed by reference.

Changed also parameter name in function implementation to match name
in declaration.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PG: insert into stray set if !us_up() && !is_acting()

This is simpler than checking actingbackfill
since it may not yet be filled in.

Fixes: #7470
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1257 from ceph/wip-agent

osd: a few agent fixes

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd/ReplicatedPG: EOPNOTSUPP on hit_set_get on non-replicated pools

We need to use the async read stuff for this to work.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: load older HitSets into memory

If our evict_mode is non-idle, load older HitSets into memory in the agent
work thread.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: strengthen agent_work assertions

Signed-off-by: Sage Weil <sage@inktank.com>

OSD: consider up/acting primary for new intervals

Both acting_up_affected and start_peering_interval need
to consider primary changes as well as acting/up changes.

Fixes: #7469
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1259 from ceph/wip-6677

rgw: add compatibility for MultipartUpload

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1258 from ceph/wip-rgw-gc

radosgw-admin: gc list --include-all

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

radosgw-admin: gc list --include-all

A new option to the radosgw-admin gc list command that dumps *all* gc
entries, and not just the expired ones. This is useful for debugging.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Throw a Timeout exception on timeout.

Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1214 from ceph/wip-rgw-manifest-2

Wip rgw manifest 2
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

dencoder: fix for new rgw manifest code

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls/rgw: fix debug output

print the correct error value

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test/rgw: manifest unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: scalable manifest object

Define a new manifest structure. The idea is that the manifest defines a
set of rules for structuring the object parts. There are a few terms to
note:
 - head: the head part of the object, which is the part that contains
   the first chunk of data. An object might not have a head (as in the
   case of multipart-part objects.
 - stripe: data portion of a single rgw object that resides on a single
   rados object.
 - part: a collection of stripes that make a contiguous part of an
   object. A regular object will only have one part (although might have
   many stripes), a multipart object might have many parts. Each part
   has a fixed stripe size, although the last stripe of a part might
   be smaller than that. Consecutive parts may be merged if their stripe
   value is the same.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1227 from ceph/wip-7394

Improve the OSD and generic monitor timeout mechanisms

Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'upstream/wip-ecbackend-for-review'

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1255 from ceph/wip-cache-perf

osd: add perfcounters for new cache and agent events

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

osd/ReplicatedPG: clean up agent skip debug output

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_agent_{wake,skip,flush,evict}

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_[dirty|clean]

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_whiteout

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_evict

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_[try_]flush[_fail]

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_copyfrom

Signed-off-by: Sage Weil <sage@inktank.com>

RadosModel: only output if missing header is actually a problem

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: track primary explicitly to detect changing primaries

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: add some debugging if waiting_for_commit is non-empty

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: instantiate the right backend based on pool

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECBackend: flesh out ECBackend implementation

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/ECTransaction: add type encapsulating EC transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/ECUtil: add ec encode/decode util helpers

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: add bufferlist/string getattr

Signed-off-by: Samuel Just <sam.just@inktank.com>

buffer: allow caller to initialize bufferhash state

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add swap, claim_append to ObjectModDesc

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,PG: pass through EC messages

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages,osd: add EC messages and associated types

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/osd/: add ECBackend stubs

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: make getattrs virtual

This way, the backend can choose to filter private attrs.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fix size checking for ECBackend pools

Allow the backend to adjust the logical object size before
checking against the ondisk size.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: factor out logical to ondisk size mapping

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: move the backend agnostic code into PGBackend for scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: factor out be_deep-scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: replace bufferptr with bufferlist for attrs in recovery

Now we won't have to keep converting back and forth!

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::discover_all_missing: request a FULLLOG

Otherwise, we cannot correct for divergent objects in the missing
set.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: Generalize missing_loc for ECBackend

Prior to EC pools, unfound => missing.  Now, unfound (unreadable,
really) is dependent on the PGBackend requirements for reconstituting
an object.  This also means recovering an object missing on a replica
but not the primary requires tracking the missing_loc set.

Thus, rather than maintaining missing_loc only for objects missing
on the primary, the MissingLoc structure will track all missing
objects actingbackfill-wide until each object is recovered.

For simplicity, since we don't really know what objects need recovery
until activation (and since we can't do anything with that information
prior to activation anyway), we defer populating the missing_loc
information until activation.

We need peers to rollback divergent log entries before we attempt to
read the relevant objects.  The simplest way to accomplish this seems to
be the simply choose to always activate peers if search_for_missing
turns up missing objects.

Due to EC pools, missing is necessary, but not sufficient for readability.
Thus, we instead check is_unreadable for cases where we need to read the object
and reserve is_missing for cases where we need the object context.

wait_for_missing_object becomes waiting_for_unreadable_object in order to avoid
having another layer of waiting_for_* maps.  These ops may be requeued
either when the primary is recovered or when the object is no longer degraded,
depending on when the object becomes readable.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: allow PGBackend to set criteria for PG up-ness

ECBackend needs to be able to require that a readable
set of the most recent interval to write be available
in order to ensure that it rolls back the log far
enough.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: add some additional helpers.

ECBackend's primary specific logic mostly won't treat the
primary shard specially, so it'll be handy to have primary
agnostic helpers for get_shard_info and get_shard_missing.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,PGBackend: expose acting and backfill_targets

Signed-off-by: Samuel Just <sam.just@inktank.com>

os/: allow objects with NO_GEN but non-NO_SHARD shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: skip CRUSH_ITEM_NONE in acting set in various places

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: extend pg_interval_t to include primary

Otherwise, we cannot correctly determine up_from/up_thru for
old intervals.  Also, we need this information to determine
when a new interval starts due to a new primary without a
change in the acting set.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: handle primary other than acting[0]

EC pool particularly might have CRUSH_ITEM_NONE as acting[0] which
is not a valid osd.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap: handle CRUSH_ITEM_NONE in acting sets

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap: fill in get_primary_shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: introduce calc_ec_acting

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: refactor calc_acting to handle pg_shard_t changes

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: refer to pg by spg_t and replica by pg_shard_t

We may have multiple pg shards on the same osd for an
ec pool.  Thus, replicas must be referred to by <osd, shard>
and pgs by <pgid, shard>.

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages/: include shard information in various pg messages

We can no longer use the messenger source information to determine
the origin of the message since an osd might have more than one
shard of a particular pg.  Thus, we need to include a pg_shard_t
from field to indicate origin.  Similarly, pg_t is no longer
sufficient to specify the destination pg, we instead use spg_t.
In the event that we get a message from an old peer, we default
from to pg_shard_t(get_source().num(), ghobject_t::no_shard())
and spg_t to spg_t(pgid, ghobject_t::no_shard()).  This suffices
because non-NO_SHARD shards can only appear once ec pools have
been enabled -- and doing that bans unenlightened osds.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: excise OSDService*

This should eventually make it easier to mock out a PGBackend::Listener.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: move temp handling into PGBackend

Temp handling is also the same in ReplicatedBackend as in ECBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: use whoami_shard().shard in rollback*, objects*

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend/PGBackend: move objects_* methods into PGBackend

These also are unchanged in ECBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: move rollback methods into PGBackend

These will end up essentially unchanged in ECBackend as it turns out.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: introduce spg_t, pg_shard_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: use ghobject_t for setattrs

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/osd: pass in version to recover to recover_object

Otherwise we need to do some annoying contortions to
get it out of the missing sets.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: Add a shard parameter to SnapMapper to handle multiple pg shards

Signed-off-by: Samuel Just <sam.just@inktank.com>

TestPGLog: remove test with DELETE op with prior_version = eversion_t()

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog::merge_old_entry: simplify the oe.version > ne.version case

If ne.version < oe.version, the correct answer is to rollback oe.version
if possible regardless of what the entries are.

Also, rearrange to deal with the fact that we cannot roll back a missing
object.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: l_osd_tier_promote

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1128 from ceph/wip-dirfrags

Fixes for mds cluster + directory fragments

PG: just ignore rollbacks on objects we have already failed to rollback

The relevant changes to missing and/or the filestore will have
already been made.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::remove_snap_mapped_object: use pg_whoami.shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: take snapset_obc write lock where appropriate

Otherwise, we might read it for backfill before it's fully created
on a peer.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: release backfill state on_global_recover not peer recover

There may now be multiple backfill peers.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: take clone write lock as well in make_writeable

Otherwise, we might start backfilling the clone before the op is
complete.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,osd_types: seperate require_rollback from ec_pool

It's handy to allow a pool to answer false to ec_pool() and
true to require_rollback() in order to allow a replicated
pool to test the rollback mechanisms without allowing
non-NO_SHARD shards.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG:: remove_snap_mapped_object already removes the object

Signed-off-by: Samuel Just <sam.just@inktank.com>