PGLog: test for missing using the correct shard value

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: fix query for ec pgs

We need to parse a pg_t and translate it into an spg_t.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: make is_pg_changed static and const for old/newacting

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap: make calc_pg_rank/role const for acting

Signed-off-by: Samuel Just <sam.just@inktank.com>

Objecter: resend op if primary role changed

Fixes: #7508
Signed-off-by: Samuel Just <sam.just@inktank.com>

ECTransaction: require hash_infos for deleted objects

Otherwise, an append preceded by a delete may not see the
correct HashInfo state since it won't have been cached
until the delete is applied.

Fixes: #7506
Signed-off-by: Samuel Just <sam.just@inktank.com>

ECUtil: clear() should reset hashes to -1, not 0

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::cancel_pull: ECBackend might cancel_pull on a non-missing object

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECBackend::filter_read_op: clean up read state properly

Fixes: #7494
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge branch 'wip-6685-firefly' into firefly

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'origin/wip-da-SCA-20140218' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Added evaluation of cluster and fixed evaluation when setting initial weight on start in verbose mode

Signed-off-by: Christopher O'Connell <jwriteclub@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1286 from ceph/wip-corpus

ceph-object-corpus: prune some old releases

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Revert "ReplicatedPG::recover_backfill: adjust last_backfill to HEAD if snapdir"

This reverts commit 545135f3e1c75318940caa2c82ac32a53fc4f957.

Conflicts:

src/osd/ReplicatedPG.cc

Merge remote-tracking branch 'origin/port/updates' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

ceph-object-corpus: prune some old releases

We do not need to test these old object encodings as they are covered by
later object collections.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge remote-tracking branch 'origin/wip-6936' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-7099' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-6830' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-6951' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-7064' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge remote-tracking branch 'origin/wip-7271' into firefly

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

dencoder: check for radosgw build option

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

osd: use ceph scoped shared_ptr

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge remote-tracking branch 'upstream/next'

Merge remote-tracking branch 'upstream/wip-copyget' into next

Fixes: #7471
Reviewed-by: Greg Farnum <greg@inktank.com>

PG::build_might_have_unfound: skip CRUSH_ITEM_NONE

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECBackend: deal with temp collection details in handle_sub_write

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::on_global_recover: requeue degraded, then unreadable

We need to requeue in reverse order since we are requeueing at the
front.

Signed-off-by: Samuel Just <sam.just@inktank.com>

rgw: minor cleanup

Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Fix generate keystone token from credentials

Request to v2.0/tokens must be a POST request and the request
should specify JSON in the content-type header and fix request
body JSON generation to have the right JSON format.

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Mouad Benchchaoui <m.benchchaoui@x-ion.de>

Merge branch 'next'

Conflicts:
src/test/admin_socket.cc

Signed-off-by: Ken Dreyer <ken.dreyer@inktank.com>

RadosModel: copyfrom should result in a dirty object

Signed-off-by: Samuel Just <sam.just@inktank.com>

PG: fix scrub feature check

acting[0] isn't necessarily the primary.  Start at
0 and explicitely skip the primary.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: assert no omap for ec pool in _write_copy_chunk and skip

Signed-off-by: Samuel Just <sam.just@inktank.com>

ECBackend: handle short reads

async reads may have a length past the end of the object.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: ensure that ec appends are aligned

When not writing out the last chunk, _write_copy_chunk
will trim the chunk to a stripe boundary for ec pools
and adjust cursor.data_offset to compensate.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG::fill_in_copy_get, use getattr_maybe_cache

Also, add user_only flag to getattr_maybe_cache and cleanup
the other user.

Signed-off-by: Samuel Just <sam.just@inktank.com>

ReplicatedPG: fix stat uses to use the correct shard

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSDMap::_apply_primary_affinity: skip CRUSH_ITEM_NONE in non-default check

Fixes: #7482
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>

doc/release-notes: v0.77

Also pending notes for v0.78

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.67.7

Signed-off-by: Sage Weil <sage@inktank.com>

v0.77

osd/,mon/: add (up|acting)_primary to pg_stat_t

We use pg_stat_t information to determine pg create targeting.

Fixes: #7481
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1256 from ceph/wip-librados-ops-and-rvals

librados c api for object operations and a few bug fixes

Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage.weil@inktank.com>

rgw: fix swift range response

Fixes: #7099
Backport: dumpling
The range response header was broken in swift.

Reported-by: Julien Calvet <julien.calvet@neurea.com>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: fix etag in multipart complete

Fixes: #6830
The etag member variable was not set correctly.

Reported-by: Benjamin Gilbert <bgilbert@backtick.net>
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: reset objv tracker on bucket recreation

Fixes: #6951
If we cannot create a new bucket (as it already existed), we need to
read the old bucket's info. However, this was failing as we were holding
the objv tracker that we created for the bucket creation. We need to
clear it, as subsequent read using it will fail.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

radosgw-admin: don't generate access key if user exists

Fixes: #6936
We want the user creation operation to be idempotent, so if user already
exists don't generate a new access key.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

BackedUpObject::get_xattr()  pass function parameter by reference

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

TestRados.cc: use !empty() instead of size()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ErasureCodeBench: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_erasure_code.cc: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

RGWObjManifest: prefer prefix ++/--operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ReplicatedPG: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

OSD: use !empty() instead of size() > 0

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PG: use !empty() instead of size() > 0

[src/osd/PG.cc:543]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:1480]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:1774]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:1810]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:2229]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:2599]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:4345]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:6067]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:6264]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:6807]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.
[src/osd/PG.cc:7014]: (performance) Possible inefficient checking
  for 'actingbackfill' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

RPCRecPred: use !empty() instead of size() >= 1

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

KeyValueStore: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

KeyValueStore: use !empty() instead of size()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

FileStore: use !empty() instead of size()

[src/os/FileStore.cc:3635]: (performance) Possible inefficient checking
  for 'omap_set' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

DBObjectMap: use !empty() instead of size()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

OSDMonitor: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

OSDMonitor: use !empty() instead of size() > 0

[src/mon/OSDMonitor.cc:730]: (performance) Possible inefficient checking
  for 'osd_weight' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MDCache: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

CrushWrapper::is_valid_crush_loc(): pass function parameter by reference

[src/crush/CrushWrapper.cc:1412]: (performance) Function parameter 'loc'
  should be passed by reference.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

CrushWrapper: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PGLog.h: prefer prefix ++operator for non-trivial iterator

[src/osd/PGLog.h:363]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.
[src/osd/PGLog.h:375]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.
[src/osd/PGLog.h:380]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.
[src/osd/PGLog.h:382]: (performance) Prefer prefix ++/-- operators for
  non-primitive types.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Client: remove a std::string::c_str() calls

[src/client/Client.cc:6821]: (performance) Passing the result of c_str() to a
  function that takes std::string as argument no. 2 is slow and redundant.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/config.cc: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

str_map.cc: prefer prefix ++operator for non-trivial iterator

[src/common/str_map.cc:55]: (performance) Prefer prefix ++/-- operators
  for non-primitive types.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Client.cc: use !empty() instead of size()

[src/client/Client.cc:2275]: (performance) Possible inefficient
  checking for 'cap_snaps' emptiness.
[src/client/Client.cc:2458]: (performance) Possible inefficient
  checking for 'cap_snaps' emptiness.
[src/client/Client.cc:2986]: (performance) Possible inefficient
  checking for 'caps' emptiness.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_argparse.cc: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Monitor: pass function parameter by reference

[src/mon/Monitor.h:619]: (performance) Function parameter 'param_str_map'
  should be passed by reference.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

BarrierContext: prefer prefix ++operator for non-trivial iterator

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

BarrierContext: reduce scope of 'done' variable

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MonClient: pass function parameter by reference

[src/mon/MonClient.h:412]: (performance) Function parameter 'mon_name'
  should be passed by reference.

Changed also parameter name in function implementation to match name
in declaration.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PG: insert into stray set if !us_up() && !is_acting()

This is simpler than checking actingbackfill
since it may not yet be filled in.

Fixes: #7470
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1257 from ceph/wip-agent

osd: a few agent fixes

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd/ReplicatedPG: EOPNOTSUPP on hit_set_get on non-replicated pools

We need to use the async read stuff for this to work.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: load older HitSets into memory

If our evict_mode is non-idle, load older HitSets into memory in the agent
work thread.

Signed-off-by: Sage Weil <sage@inktank.com>

osd/ReplicatedPG: strengthen agent_work assertions

Signed-off-by: Sage Weil <sage@inktank.com>

OSD: consider up/acting primary for new intervals

Both acting_up_affected and start_peering_interval need
to consider primary changes as well as acting/up changes.

Fixes: #7469
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>

rgw: don't try to read bucket's policy if it doesn't exist

It's going to fail anyway, just build a default policy.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: return error if accessing object in non-existent bucket

Fixes: #7064
Instead of trying to access the object, which is impossible as we don't
even have a proper bucket info. Up until now we ended up creating an
empty pool and eventually returning ENOENT, this fix catches the issue
earlier in the process.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1259 from ceph/wip-6677

rgw: add compatibility for MultipartUpload

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1258 from ceph/wip-rgw-gc

radosgw-admin: gc list --include-all

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

radosgw-admin: gc list --include-all

A new option to the radosgw-admin gc list command that dumps *all* gc
entries, and not just the expired ones. This is useful for debugging.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Merge pull request #1261 from ceph/wip-7463

ceph_rest_api.py: don't fail if no up OSDs found on startup

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Throw a Timeout exception on timeout.

Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #1214 from ceph/wip-rgw-manifest-2

Wip rgw manifest 2
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

dencoder: fix for new rgw manifest code

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

cls/rgw: fix debug output

print the correct error value

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

test/rgw: manifest unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

rgw: scalable manifest object

Define a new manifest structure. The idea is that the manifest defines a
set of rules for structuring the object parts. There are a few terms to
note:
 - head: the head part of the object, which is the part that contains
   the first chunk of data. An object might not have a head (as in the
   case of multipart-part objects.
 - stripe: data portion of a single rgw object that resides on a single
   rados object.
 - part: a collection of stripes that make a contiguous part of an
   object. A regular object will only have one part (although might have
   many stripes), a multipart object might have many parts. Each part
   has a fixed stripe size, although the last stripe of a part might
   be smaller than that. Consecutive parts may be merged if their stripe
   value is the same.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

ceph_rest_api.py: don't fail if no up OSDs found on startup

allow find_up_osd() to return None, and have caller check for it

Fixes: #7463
Signed-off-by: Dan Mick <dan.mick@inktank.com>

test/librados: remove unused lines and fix return code for cmp guards

Now that return values are actually set by the osd and client, fix up
the tests that were checking the old behavior.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ReplicatedPG: fix successful write condition

ctx->modify no longer implies that the operation is a write,
if it ever did. These days op->may_write() is checked reliably
on the OSD, so just use that instead.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ReplicatedPG: clear osd op reply output for writes

Since the vector of OSDOps used by the reply is the same
as the one processed by do_osd_ops() now, any output data
needs to be cleared for writes. To be compatible with current behavior,
allow writes that aren't applying anything or have failed to return data
still.

Add a new parameter to the MOSDOpReply constructor to determine
whether the output data should be cleared. Clear it for successful
writes, and remove a redundant result < 0 -> result > 0 check in the
process.  This was caught by ceph_test_cls_hello and its
writes_dont_return_data method.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

ReplicatedPG: set individual osd op rvals

This takes care of things that can fail before transaction is
executed, like omap comparison. Getting individual rvals from a
transaction requires more refactoring, so I'm leaving it for a later
cleanup.

Fixes: #6483
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>