rgw: fix swift range response

Fixes: #7099
Backport: dumpling
The range response header was broken in swift.

Reported-by: Julien Calvet <julien.calvet@neurea.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1259 from ceph/wip-6677

rgw: add compatibility for MultipartUpload

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1258 from ceph/wip-rgw-gc

radosgw-admin: gc list --include-all

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

radosgw-admin: gc list --include-all

A new option to the radosgw-admin gc list command that dumps *all* gc
entries, and not just the expired ones. This is useful for debugging.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Throw a Timeout exception on timeout.

Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1214 from ceph/wip-rgw-manifest-2

Wip rgw manifest 2
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

dencoder: fix for new rgw manifest code

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls/rgw: fix debug output

print the correct error value

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test/rgw: manifest unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: scalable manifest object

Define a new manifest structure. The idea is that the manifest defines a
set of rules for structuring the object parts. There are a few terms to
note:
 - head: the head part of the object, which is the part that contains
   the first chunk of data. An object might not have a head (as in the
   case of multipart-part objects.
 - stripe: data portion of a single rgw object that resides on a single
   rados object.
 - part: a collection of stripes that make a contiguous part of an
   object. A regular object will only have one part (although might have
   many stripes), a multipart object might have many parts. Each part
   has a fixed stripe size, although the last stripe of a part might
   be smaller than that. Consecutive parts may be merged if their stripe
   value is the same.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1227 from ceph/wip-7394

Improve the OSD and generic monitor timeout mechanisms

Reviewed-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'upstream/wip-ecbackend-for-review'

Reviewed-by: Sage Weil <sage@inktank.com>

Merge pull request #1255 from ceph/wip-cache-perf

osd: add perfcounters for new cache and agent events

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

osd/ReplicatedPG: clean up agent skip debug output

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_agent_{wake,skip,flush,evict}

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_[dirty|clean]

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_whiteout

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_evict

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_tier_[try_]flush[_fail]

Signed-off-by: Sage Weil <sage@inktank.com>

osd: l_osd_copyfrom

Signed-off-by: Sage Weil <sage@inktank.com>

RadosModel: only output if missing header is actually a problem

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: track primary explicitly to detect changing primaries

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: add some debugging if waiting_for_commit is non-empty

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: instantiate the right backend based on pool

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECBackend: flesh out ECBackend implementation

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/ECTransaction: add type encapsulating EC transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/ECUtil: add ec encode/decode util helpers

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: add bufferlist/string getattr

Signed-off-by: Samuel Just <sam.just@inktank.com>

buffer: allow caller to initialize bufferhash state

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: add swap, claim_append to ObjectModDesc

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD,PG: pass through EC messages

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages,osd: add EC messages and associated types

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/osd/: add ECBackend stubs

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: make getattrs virtual

This way, the backend can choose to filter private attrs.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fix size checking for ECBackend pools

Allow the backend to adjust the logical object size before
checking against the ondisk size.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: factor out logical to ondisk size mapping

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: move the backend agnostic code into PGBackend for scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: factor out be_deep-scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: replace bufferptr with bufferlist for attrs in recovery

Now we won't have to keep converting back and forth!

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::discover_all_missing: request a FULLLOG

Otherwise, we cannot correct for divergent objects in the missing
set.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,ReplicatedPG: Generalize missing_loc for ECBackend

Prior to EC pools, unfound => missing.  Now, unfound (unreadable,
really) is dependent on the PGBackend requirements for reconstituting
an object.  This also means recovering an object missing on a replica
but not the primary requires tracking the missing_loc set.

Thus, rather than maintaining missing_loc only for objects missing
on the primary, the MissingLoc structure will track all missing
objects actingbackfill-wide until each object is recovered.

For simplicity, since we don't really know what objects need recovery
until activation (and since we can't do anything with that information
prior to activation anyway), we defer populating the missing_loc
information until activation.

We need peers to rollback divergent log entries before we attempt to
read the relevant objects.  The simplest way to accomplish this seems to
be the simply choose to always activate peers if search_for_missing
turns up missing objects.

Due to EC pools, missing is necessary, but not sufficient for readability.
Thus, we instead check is_unreadable for cases where we need to read the object
and reserve is_missing for cases where we need the object context.

wait_for_missing_object becomes waiting_for_unreadable_object in order to avoid
having another layer of waiting_for_* maps.  These ops may be requeued
either when the primary is recovered or when the object is no longer degraded,
depending on when the object becomes readable.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: allow PGBackend to set criteria for PG up-ness

ECBackend needs to be able to require that a readable
set of the most recent interval to write be available
in order to ensure that it rolls back the log far
enough.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: add some additional helpers.

ECBackend's primary specific logic mostly won't treat the
primary shard specially, so it'll be handy to have primary
agnostic helpers for get_shard_info and get_shard_missing.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG,PGBackend: expose acting and backfill_targets

Signed-off-by: Samuel Just <sam.just@inktank.com>

os/: allow objects with NO_GEN but non-NO_SHARD shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: skip CRUSH_ITEM_NONE in acting set in various places

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: extend pg_interval_t to include primary

Otherwise, we cannot correctly determine up_from/up_thru for
old intervals.  Also, we need this information to determine
when a new interval starts due to a new primary without a
change in the acting set.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: handle primary other than acting[0]

EC pool particularly might have CRUSH_ITEM_NONE as acting[0] which
is not a valid osd.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap: handle CRUSH_ITEM_NONE in acting sets

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap: fill in get_primary_shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: introduce calc_ec_acting

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: refactor calc_acting to handle pg_shard_t changes

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: refer to pg by spg_t and replica by pg_shard_t

We may have multiple pg shards on the same osd for an
ec pool.  Thus, replicas must be referred to by <osd, shard>
and pgs by <pgid, shard>.

Signed-off-by: Samuel Just <sam.just@inktank.com>

messages/: include shard information in various pg messages

We can no longer use the messenger source information to determine
the origin of the message since an osd might have more than one
shard of a particular pg.  Thus, we need to include a pg_shard_t
from field to indicate origin.  Similarly, pg_t is no longer
sufficient to specify the destination pg, we instead use spg_t.
In the event that we get a message from an old peer, we default
from to pg_shard_t(get_source().num(), ghobject_t::no_shard())
and spg_t to spg_t(pgid, ghobject_t::no_shard()).  This suffices
because non-NO_SHARD shards can only appear once ec pools have
been enabled -- and doing that bans unenlightened osds.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend: excise OSDService*

This should eventually make it easier to mock out a PGBackend::Listener.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: move temp handling into PGBackend

Temp handling is also the same in ReplicatedBackend as in ECBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend: use whoami_shard().shard in rollback*, objects*

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedBackend/PGBackend: move objects_* methods into PGBackend

These also are unchanged in ECBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGBackend/ReplicatedBackend: move rollback methods into PGBackend

These will end up essentially unchanged in ECBackend as it turns out.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: introduce spg_t, pg_shard_t

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: use ghobject_t for setattrs

Signed-off-by: Samuel Just <sam.just@inktank.com>

src/osd: pass in version to recover to recover_object

Otherwise we need to do some annoying contortions to
get it out of the missing sets.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd/: Add a shard parameter to SnapMapper to handle multiple pg shards

Signed-off-by: Samuel Just <sam.just@inktank.com>

TestPGLog: remove test with DELETE op with prior_version = eversion_t()

Signed-off-by: Samuel Just <sam.just@inktank.com>

PGLog::merge_old_entry: simplify the oe.version > ne.version case

If ne.version < oe.version, the correct answer is to rollback oe.version
if possible regardless of what the entries are.

Also, rearrange to deal with the fact that we cannot roll back a missing
object.

Signed-off-by: Samuel Just <sam.just@inktank.com>

osd: l_osd_tier_promote

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1128 from ceph/wip-dirfrags

Fixes for mds cluster + directory fragments

PG: just ignore rollbacks on objects we have already failed to rollback

The relevant changes to missing and/or the filestore will have
already been made.

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG::remove_snap_mapped_object: use pg_whoami.shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: take snapset_obc write lock where appropriate

Otherwise, we might read it for backfill before it's fully created
on a peer.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: release backfill state on_global_recover not peer recover

There may now be multiple backfill peers.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: take clone write lock as well in make_writeable

Otherwise, we might start backfilling the clone before the op is
complete.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG,osd_types: seperate require_rollback from ec_pool

It's handy to allow a pool to answer false to ec_pool() and
true to require_rollback() in order to allow a replicated
pool to test the rollback mechanisms without allowing
non-NO_SHARD shards.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG:: remove_snap_mapped_object already removes the object

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: find_object_context debugging: output oi

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: handle xattr cache on rollback and delete_head

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: release_op_ctx_locks in all-applied, not all-committed

Signed-off-by: Samuel Just <sam.just@inktank.com>

FileStore: permit NULL onreadable callback

Signed-off-by: Samuel Just <sam.just@inktank.com>

ErasureCodeJerasure: fill in decoded if all shards are included

This causes it to match minimum_to_decode.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ErasureCodeJerasure: fix alignement issues in encode/decode

Signed-off-by: Samuel Just <sam.just@inktank.com>

ObjectStore: fix OP_COLL_ADD dump output

Signed-off-by: Samuel Just <sam.just@inktank.com>

TestRados: send aligned appends on erasure pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: reject unaligned writes on ec pools

Signed-off-by: Samuel Just <sam.just@inktank.com>

librados: add calls to expose required write alignment

Signed-off-by: Samuel Just <sam.just@inktank.com>

SimpleMessenger: init_local_connection whenever my_inst changes

This is necessary to correctly handle messages to self.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #1139 from ceph/wip-agent

osd: basic cache tier agent

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd: improve whiteout debug prints

Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: make agent skip blocked obcs

Among other things, this can interfere with an in-flight copy-from and
corrupt the pg stats.

Signed-off-by: Sage Weil <sage@inktank.com>

ReplicatedBackend: print char ack_type as int

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #1254 from ceph/wip-barrier

libcephfs: disable barriers for now

Merge remote-tracking branch 'gh/wip-7212-sage-b'

We should carefully backport some or all of this.

Backport: emperor, dumpling
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>

client: disable barrier support

The boost interval_set class is not available on centos6/rhel6.  Until that
dependency is sorted out, fix the build.

Signed-off-by: Sage Weil <sage@inktank.com>

client: fix barrier interval

(start, end) not (start, length)

Signed-off-by: Sage Weil <sage@inktank.com>

client/barrier: drop unused active_commit_interval

Signed-off-by: Sage Weil <sage@inktank.com>

client: don't populate Client::barriers from read accesses

If the entry doesn't exist, no need to create it.

Signed-off-by: Sage Weil <sage@inktank.com>

qa/workunits/suites/pjd: use test suite with acl tweak

Test 45 in xacl/00.t fails on ext4 and cephfs.  Just disable it.
Everything else passes.

Signed-off-by: Sage Weil <sage@inktank.com>

mds: bump the protocol version

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

osd/ReplicatedPG: allow is_degraded_object() to be called outside of backfil

The agent does this.

Signed-off-by: Sage Weil <sage@inktank.com>

mds: open sessions for rejoin imported caps

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>

mds: fix slave rename rollback

use rollback bufferlist to decide if the inode is being exported.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>