rgw, cls_rgw: keep shard ids with oids

Instead of just having the list of oids, keep the shard ids together, so
that we can know on which shard the operation happened.
Bucket markers are just using the shard numeric id, instead of the
bucket instance shard id. This makes it easier to parse the markers
appropriately.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: clean up CLSRGWConcurrentIO

Class is no longer a template, and keeps a map of oids by shard_id. Call
issue_op() using both shard_id and oids. Shard id is used for mapping
the results in the derived classes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify bucket instance shard marker ids

Instead of having the markers prefixed by the oids, use the bucket
instance id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket replica log, handle shard ids

bucket replica log can now save entries by the specified shard id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: list bi log should not return marker entry

The marker should be served as a lower bound, but should not be
returned.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket_index_shard_hash_type fixes

add initializations, json encode /decode.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode the req_state bucket instance id if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: improve bucket sharding hashing

Amplify small source changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data changes log, log info by bucket shard id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use new BucketShard structure for index manipulation calls

Instead of recalculating the hash every call, do it once, and pass this
structure around. Also, will be used for logging changes into the data
log.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bi log list/trim can get specific bucket shard

bucket shard can be specified on the bucket instance param. It can be
added like this: <bucket-instance>[:shard-id]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Fix the multipart uploads functional test failures due to bucket index sharding.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Fix get_bucket_instance_info, only build the oid if it is empty.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Adjust bi log trim implementation to work with multiple bucket shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Adjust bi log listing to work with multiple bucket shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

cls_rgw, rgw: switch different ops to new concurrent infrastructure

Make all the relevant ops use the CLSRGWConcurrentIO infrastructure,
which simplifies things.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: generalize container type for concurrent IO base class

Turned the ConcurrentIO class a template, so that we could use different
kind of containers that are needed for the different operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw, rgw: create base class for common bucket shard operations

Instead of copy pasting the same code all over again, create a base
class for the needed concurrent IO operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Adjust bucket stats/index checking/index rebuild/tag timeout implementation to work with multiple shards.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Adjust bucket listing to work with multiple shards.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Adjust rgw bucket prepare/complete OP to work with multiple bucket index shards.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Implement sharding for bucket creation.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Add a new field to bucket info indicating the number of shards of this bucket and make it configurable.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Merge pull request #3316 from ceph/wip-10471

rgw: index swift keys appropriately

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3327 from ceph/wip-peeringqueue

osd: fix peering queue bug

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: return InvalidAccessKeyId instead of AccessDenied

Fixes: #10334
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 56af795b1046a4c1bfba59e1fefde272bb0e5c1e)

rgw: return SignatureDoesNotMatch instead of AccessDenied

Fixes: #10329
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit ef75d720f289ce2e18c0047380a16b7688864560)

Merge pull request #3250 from ceph/wip-10372

osdc/Objecter: improve pool deletion detection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "rgw: switch to new watch/notify API"

This reverts commit dc67cd69604ec4e4df846b818ec739dc7b09a537.

Conflicts:
src/rgw/rgw_rados.cc

osd: assert there is a peering event

This became conditional way back in 12e22b3d44eba51a70d8babebc2684f0c46575a7
for unclear reasons.  It probably predates the in_use checks.  In any case,
at this point, we should only arrive here if the PG was queued, implying
that there will always be an event to process.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: requeue PG when we skip handling a peering event

If we don't handle the event, we need to put the PG back into the peering
queue or else the event won't get processed until the next event is
queued, at which point we'll be processing events with a delay.

The queue_null is not necessary (and is a waste of effort) because the
event is still in pg->peering_queue and the PG is queued.

Note that this only triggers when we exceeed osd_map_max_advance, usually
when there is a lot of peering and recovery activity going on.  A
workaround is to increase that value, but if you exceed osd_map_cache_size
you expose yourself to crache thrashing by the peering work queue, which
can cause serious problems with heavily degraded clusters and bit lots of
people on dumpling.

Backport: giant, firefly
Fixes: #10431
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: index swift keys appropriately

Fixes: #10471
Backport: firefly, giant

We need to index the swift keys by the full uid:subuser when decoding
the json representation, to keep it in line with how we store it when
creating it through other mechanism.

Reported-by: hemant burman <hemant.burman@gmail.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3277 from ceph/wip-watch-leak

librados: fix a memory leak in watch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3247 from ceph/wip-10422

mon: provide encoded canonical full OSDMap from primary

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

librados: fix leak of WatchContext on unwatch

The lifecycle matches that of the watch linger_op.  Note that it is NULL
for notify.

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile: include radosgw-admin in base

Signed-off-by: Sage Weil <sage@redhat.com>

client: fix quota signed/unsigned warning

client/Client.cc: In member function 'bool Client::is_quota_bytes_exceeded(Inode*, uint64_t)':
client/Client.cc:10393:66: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (quota->max_bytes && (rstat->rbytes + new_bytes) > quota->max_bytes)

Signed-off-by: Sage Weil <sage@redhat.com>

mon: provide encoded canonical full OSDMap from primary

Currently we make each monitor apply the incremental and encode the full
map locally.  The original motivation was to save bandwidth, but the
savings are minimal to modest and the complexity associated with doing this
is huge.

This strategy also causes problems now that we have OSDMap crc's and old
mons/clusters may have diverging full OSDMaps due to mixed version
clusters.  See #10422

Instead, include the encoded full map in the paxos transaction.  We will
still apply the incremental and check the crc, but if it fails and we have
the correct version, reload it from disk and move on.  If we don't, we
will continue as we have before--the primary mon doesn't have support for
crc's yet.  When it does we will start verifying and/or get our
full map back into sync.

Fixes: #10422
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: improve pool deletion detection

Currently we can have a race like so:

 - send op in epoch X
 - osd replies
 - pool deleted in epoch X+1
 - client gets X+1, sends map epoch check
 - client gets reply
   -> fails assert that there is no map check in flight

Avoid this situation by inferring that the pool is deleted when we see
that we previously sent the request but the pool is no longer present.
Since pool ids are not reused there is no point in doing a synchronous
map check at all.

Backport: giant
Fixes: #10372
Signed-off-by: Sage Weil <sage@redhat.com>

librados: only call watch_flush if necessary

Fix bug #10424
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
(cherry picked from commit 926a1b7468550e0c83bfa68bd45df1a6af599727)

mds: add default ctor for quota_info_t

http://tracker.ceph.com/issues/10400 Fixes: #10400

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7f1e510165f284aad3f341ff89f64d5c3e8fe60e)

librados: warn about rados_watch_flush() prior to ioctx shutdown

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1fbe9b6f3a67584829fe17b6bb08cfa1b787d82c)

librados: watch_flush() on shutdown

Users can easily forget this. It makes shutdown potentially block, but if
they have racing callbacks they get what they ask for.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 30678f6daf0a71b6074c061c6b4aec6386372368)

librados: add rados_watch_flush() call

Add a call so that callers can make sure all queued callbacks have
completed before shutting down the ioctx.  This avoids a segv triggered
by the LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1
test due to the ioctx being destroyed when the in-progress callback
does a notify_ack.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4ebd4b4280cb048547842351f41b38658fb21a6e)

osdc/Objecter: do notify completion callback in fast-dispatch context

The notify completion has exactly one user, the librados caller which
does nothing but take a local (inner) lock and signal a Cond.  Do this
in the fast-dispatch context for simplicity.

Notably, this makes the notify completion (and timeout) trigger a
notify2() return (with ETIMEDOUT) even when the finisher queue that
normally delivers notify is busy.. for example with a notify that is
being very slow.  In our case, the unit test is doing a sleep(3) to
test timeouts but also prevented the ETIMEDOUT notification from
being delivered to the caller.  This patch resolves that.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9b78dafd4a80ad9a557cfdf17d10fb2bff236f1a)

Merge pull request #3229 from ceph/wip-osd-ctor

osd/ReplicatedPG: initialize new_backfill in ctor

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3230 from ceph/wip-pg-stat

mon/PGMap: restructure 'pg stat' formatted output

Reviewed-by: John Spray <jspray@redhat.com>

mon/PGMap: restructure 'pg stat' formatted output

The + character, which appears in state names, is not a valid XML token.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: initialize new_backfill in ctor

*** CID 1260213:  Uninitialized scalar field  (UNINIT_CTOR)
/osd/ReplicatedPG.cc: 1242 in ReplicatedPG::ReplicatedPG(OSDService *,
std::tr1::shared_ptr<const OSDMap>, const PGPool &, spg_t)()
1236       snap_trimmer_machine(this)
1237     {
1238       missing_loc.set_backend_predicates(
1239         pgbackend->get_is_readable_predicate(),
1240         pgbackend->get_is_recoverable_predicate());
1241       snap_trimmer_machine.initiate();
>>>     CID 1260213:  Uninitialized scalar field  (UNINIT_CTOR)
>>>     Non-static class member "new_backfill" is not initialized in this
constructor nor in any functions that it calls.
1242     }
1243
1244     void ReplicatedPG::get_src_oloc(const object_t& oid, const
object_locator_t& oloc, object_locator_t& src_oloc)
1245     {
1246       src_oloc = oloc;
1247       if (oloc.key.empty())

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3121 from ceph/wip-10277

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3199 from ceph/wip-qa-empty-xattr

qa: test zero size xattr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/wip-fs-quota'

Conflicts:
src/client/Client.cc

Merge pull request #3218 from dachary/wip-10383-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10383 Refs: #10383

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2976 from ceph/wip-pgmeta

osd: move PG metadata to a per-PG object

Passed rados and upgrade tests.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3212 from ceph/wip-10255-wusui

Remove sepia dependency (use fqdn)

Backport: giant
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3214 from xinxinsh/wip-cleanup

cleanup : remove sync_epoch

Reviewed-by: Sage Weil <sage@redhat.com>

0.90

doc: Instead of using admin socket, use 'ceph daemon' command.

Merge pull request #3193 from nilamdyuti/wip-doc-ceph-disk

Changes format style in ceph-disk to improve readability as html

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3210 from ceph/wip-osdmap

osd: only verify OSDMap crc if it is known

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3216 from cstavr/master

ceph-disk: Fix wrong string formatting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: Fix wrong string formatting

Signed-off-by: Christos Stavrakakis <stavr.chris@gmail.com>

cleanup : remove sync_epoch

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Remove sepia dependency (use fqdn)

Fixes: #10255
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #3209 from dmick/wip-stop.sh-me

stop.sh: only try killing processes that belong to me

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: only verfy OSDMap crc if it is known

Only verify we encode a full map with the correct CRC if we actually
have the value in the Incremental.  Otherwise, any map from an old
mon will fail.  And we'll try to request a full map with a message the
old mon doesn't understand.

Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: only try killing processes that belong to me

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3207 from ceph/wip-cot-rename

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3208 from ceph/wip-10326

qa/workunits/rest/test.py: fix pg stat test

qa/workunits/rest/test.py: fix pg stat test

Back in e27b0d9e42b7ac3de896c3398689bbdbe03f1bbb we changed pg stat to
not dump everything when a formatter is used.

Fixes: #10326
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3195 from nilamdyuti/wip-doc-ceph-deploy

doc: Changes format style in ceph-deploy to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3201 from nilamdyuti/wip-doc-ceph

doc: Changes format style in ceph to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3057 from ceph/wip-crush-straw

crush: fix straw bucket scaler bugs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

.gitmodules: update ceph-object-corpus auth repo to github

It's annoying having ceph.com for only this repo, and it's not that big.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3101 from yuyuyu101/wip-10147

Messenger Unit Tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3205 from ceph/wip-caps-init

messages/MClientCaps: init peer.flags

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3123 from linuxbox2/wip-claim-3

Wip claim 3

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3111 from yuyuyu101/wip-10193

WBThrottle: make bytes/ios/inode_wb's perf counter effective

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3133 from XinzeChi/wip-2

OSD: FileJournal: call writeq_cond.Signal if necessary in submit_entry

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3132 from mslovy/optimazation_collection_map

os: IndexManager:: optimize map<coll_t, CollectionIndex* > col_indices to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3204 from ceph/wip-osdmap-leak

osd: fix leaked OSDMap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: Fix time event is called after AsyncMessenger destruction

AsyncConnection uses time event to handle async event partially, but it's not
very effective so here use dispatch_event_external instead.

And if client try to connect to server after a period, it may be call
AsyncConnection::process which will reference to AsyncMessenger. Since
AsyncMessenger doesn't use reference count, it will result in segment
fault. Now we record time event id and delete these registered time events
when stopping connection.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

PipeConnection: Avoid deadlock when calling is_connected

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test/msgr/test_msgr: Fix potential unsafe cond wakeup and wrap check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

messages/MClientCaps: init peer.flags

*** CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
/messages/MClientCaps.h: 132 in MClientCaps::MClientCaps(int, inodeno_t,
inodeno_t, unsigned long, int, unsigned int)()
126         head.ino = ino;
127         head.realm = realm;
128         head.cap_id = id;
129         head.migrate_seq = mseq;
130         peer.cap_id = 0;
131         inline_version = 0;
>>>     CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
>>>     Non-static class member field "peer.flags" is not initialized in this
constructor nor in any functions that it calls.
132       }
133     private:
134       ~MClientCaps() {}
135
136     public:
137       const char *get_type_name() const { return "Cfcap";}

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix leaked OSDMap

*** CID 1258787:  Resource leak  (RESOURCE_LEAK)
/osd/OSD.cc: 6118 in OSD::handle_osd_map(MOSDMap *)()
6112                    << dendl;
6113            clog->warn() << "failed to encode map e" << e << " with expecte
crc\n";
6114            MMonGetOSDMap *req = new MMonGetOSDMap;
6115            req->request_full(e, last);
6116            monc->send_mon_message(req);
6117            last = e - 1;
>>>     CID 1258787:  Resource leak  (RESOURCE_LEAK)
>>>     Variable "o" going out of scope leaks the storage it points to.
6118            break;
6119           }
6120
6121
6122           hobject_t fulloid = get_osdmap_pobject_name(e);
6123           t.write(META_COLL, fulloid, 0, fbl.length(), fbl);

Signed-off-by: Sage Weil <sage@redhat.com>

Messenger: Create an Messenger implementation by name.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3202 from ceph/wip-fix-asok

mds: fix asok on rank 0

mds: fix asok on rank 0

Broken by:
1b2b34431 (MDS: do not allow invocation of most commands on an inactive MDS)

MDS::whoami is a rank, so check for inactive is <0, not <=0

Signed-off-by: John Spray <john.spray@redhat.com>

doc: Adds updated man page for ceph under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Changes format style in ceph to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-disk under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-disk with suggested changes.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-disk under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-disk.

Changes format style to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Show how new data pools can be added to CephFS for file layouts

packaging: package ceph-deploy(8)

Fixes merge conflict.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-deploy under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>