mon: provide encoded canonical full OSDMap from primary

Currently we make each monitor apply the incremental and encode the full
map locally.  The original motivation was to save bandwidth, but the
savings are minimal to modest and the complexity associated with doing this
is huge.

This strategy also causes problems now that we have OSDMap crc's and old
mons/clusters may have diverging full OSDMaps due to mixed version
clusters.  See #10422

Instead, include the encoded full map in the paxos transaction.  We will
still apply the incremental and check the crc, but if it fails and we have
the correct version, reload it from disk and move on.  If we don't, we
will continue as we have before--the primary mon doesn't have support for
crc's yet.  When it does we will start verifying and/or get our
full map back into sync.

Fixes: #10422
Signed-off-by: Sage Weil <sage@redhat.com>

librados: only call watch_flush if necessary

Fix bug #10424
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
(cherry picked from commit 926a1b7468550e0c83bfa68bd45df1a6af599727)

mds: add default ctor for quota_info_t

http://tracker.ceph.com/issues/10400 Fixes: #10400

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7f1e510165f284aad3f341ff89f64d5c3e8fe60e)

librados: warn about rados_watch_flush() prior to ioctx shutdown

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1fbe9b6f3a67584829fe17b6bb08cfa1b787d82c)

librados: watch_flush() on shutdown

Users can easily forget this. It makes shutdown potentially block, but if
they have racing callbacks they get what they ask for.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 30678f6daf0a71b6074c061c6b4aec6386372368)

librados: add rados_watch_flush() call

Add a call so that callers can make sure all queued callbacks have
completed before shutting down the ioctx.  This avoids a segv triggered
by the LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1
test due to the ioctx being destroyed when the in-progress callback
does a notify_ack.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4ebd4b4280cb048547842351f41b38658fb21a6e)

osdc/Objecter: do notify completion callback in fast-dispatch context

The notify completion has exactly one user, the librados caller which
does nothing but take a local (inner) lock and signal a Cond.  Do this
in the fast-dispatch context for simplicity.

Notably, this makes the notify completion (and timeout) trigger a
notify2() return (with ETIMEDOUT) even when the finisher queue that
normally delivers notify is busy.. for example with a notify that is
being very slow.  In our case, the unit test is doing a sleep(3) to
test timeouts but also prevented the ETIMEDOUT notification from
being delivered to the caller.  This patch resolves that.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9b78dafd4a80ad9a557cfdf17d10fb2bff236f1a)

Merge pull request #3229 from ceph/wip-osd-ctor

osd/ReplicatedPG: initialize new_backfill in ctor

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3230 from ceph/wip-pg-stat

mon/PGMap: restructure 'pg stat' formatted output

Reviewed-by: John Spray <jspray@redhat.com>

mon/PGMap: restructure 'pg stat' formatted output

The + character, which appears in state names, is not a valid XML token.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: initialize new_backfill in ctor

*** CID 1260213:  Uninitialized scalar field  (UNINIT_CTOR)
/osd/ReplicatedPG.cc: 1242 in ReplicatedPG::ReplicatedPG(OSDService *,
std::tr1::shared_ptr<const OSDMap>, const PGPool &, spg_t)()
1236       snap_trimmer_machine(this)
1237     {
1238       missing_loc.set_backend_predicates(
1239         pgbackend->get_is_readable_predicate(),
1240         pgbackend->get_is_recoverable_predicate());
1241       snap_trimmer_machine.initiate();
>>>     CID 1260213:  Uninitialized scalar field  (UNINIT_CTOR)
>>>     Non-static class member "new_backfill" is not initialized in this
constructor nor in any functions that it calls.
1242     }
1243
1244     void ReplicatedPG::get_src_oloc(const object_t& oid, const
object_locator_t& oloc, object_locator_t& src_oloc)
1245     {
1246       src_oloc = oloc;
1247       if (oloc.key.empty())

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3121 from ceph/wip-10277

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3199 from ceph/wip-qa-empty-xattr

qa: test zero size xattr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/wip-fs-quota'

Conflicts:
src/client/Client.cc

Merge pull request #3218 from dachary/wip-10383-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10383 Refs: #10383

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2976 from ceph/wip-pgmeta

osd: move PG metadata to a per-PG object

Passed rados and upgrade tests.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3212 from ceph/wip-10255-wusui

Remove sepia dependency (use fqdn)

Backport: giant
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3214 from xinxinsh/wip-cleanup

cleanup : remove sync_epoch

Reviewed-by: Sage Weil <sage@redhat.com>

0.90

doc: Instead of using admin socket, use 'ceph daemon' command.

Merge pull request #3193 from nilamdyuti/wip-doc-ceph-disk

Changes format style in ceph-disk to improve readability as html

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3210 from ceph/wip-osdmap

osd: only verify OSDMap crc if it is known

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3216 from cstavr/master

ceph-disk: Fix wrong string formatting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: Fix wrong string formatting

Signed-off-by: Christos Stavrakakis <stavr.chris@gmail.com>

cleanup : remove sync_epoch

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Remove sepia dependency (use fqdn)

Fixes: #10255
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #3209 from dmick/wip-stop.sh-me

stop.sh: only try killing processes that belong to me

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: only verfy OSDMap crc if it is known

Only verify we encode a full map with the correct CRC if we actually
have the value in the Incremental.  Otherwise, any map from an old
mon will fail.  And we'll try to request a full map with a message the
old mon doesn't understand.

Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: only try killing processes that belong to me

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3207 from ceph/wip-cot-rename

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3208 from ceph/wip-10326

qa/workunits/rest/test.py: fix pg stat test

qa/workunits/rest/test.py: fix pg stat test

Back in e27b0d9e42b7ac3de896c3398689bbdbe03f1bbb we changed pg stat to
not dump everything when a formatter is used.

Fixes: #10326
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3195 from nilamdyuti/wip-doc-ceph-deploy

doc: Changes format style in ceph-deploy to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3201 from nilamdyuti/wip-doc-ceph

doc: Changes format style in ceph to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3057 from ceph/wip-crush-straw

crush: fix straw bucket scaler bugs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

.gitmodules: update ceph-object-corpus auth repo to github

It's annoying having ceph.com for only this repo, and it's not that big.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3101 from yuyuyu101/wip-10147

Messenger Unit Tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3205 from ceph/wip-caps-init

messages/MClientCaps: init peer.flags

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3123 from linuxbox2/wip-claim-3

Wip claim 3

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3111 from yuyuyu101/wip-10193

WBThrottle: make bytes/ios/inode_wb's perf counter effective

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3133 from XinzeChi/wip-2

OSD: FileJournal: call writeq_cond.Signal if necessary in submit_entry

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3132 from mslovy/optimazation_collection_map

os: IndexManager:: optimize map<coll_t, CollectionIndex* > col_indices to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3204 from ceph/wip-osdmap-leak

osd: fix leaked OSDMap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: Fix time event is called after AsyncMessenger destruction

AsyncConnection uses time event to handle async event partially, but it's not
very effective so here use dispatch_event_external instead.

And if client try to connect to server after a period, it may be call
AsyncConnection::process which will reference to AsyncMessenger. Since
AsyncMessenger doesn't use reference count, it will result in segment
fault. Now we record time event id and delete these registered time events
when stopping connection.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

PipeConnection: Avoid deadlock when calling is_connected

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test/msgr/test_msgr: Fix potential unsafe cond wakeup and wrap check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

messages/MClientCaps: init peer.flags

*** CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
/messages/MClientCaps.h: 132 in MClientCaps::MClientCaps(int, inodeno_t,
inodeno_t, unsigned long, int, unsigned int)()
126         head.ino = ino;
127         head.realm = realm;
128         head.cap_id = id;
129         head.migrate_seq = mseq;
130         peer.cap_id = 0;
131         inline_version = 0;
>>>     CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
>>>     Non-static class member field "peer.flags" is not initialized in this
constructor nor in any functions that it calls.
132       }
133     private:
134       ~MClientCaps() {}
135
136     public:
137       const char *get_type_name() const { return "Cfcap";}

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix leaked OSDMap

*** CID 1258787:  Resource leak  (RESOURCE_LEAK)
/osd/OSD.cc: 6118 in OSD::handle_osd_map(MOSDMap *)()
6112                    << dendl;
6113            clog->warn() << "failed to encode map e" << e << " with expecte
crc\n";
6114            MMonGetOSDMap *req = new MMonGetOSDMap;
6115            req->request_full(e, last);
6116            monc->send_mon_message(req);
6117            last = e - 1;
>>>     CID 1258787:  Resource leak  (RESOURCE_LEAK)
>>>     Variable "o" going out of scope leaks the storage it points to.
6118            break;
6119           }
6120
6121
6122           hobject_t fulloid = get_osdmap_pobject_name(e);
6123           t.write(META_COLL, fulloid, 0, fbl.length(), fbl);

Signed-off-by: Sage Weil <sage@redhat.com>

Messenger: Create an Messenger implementation by name.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3202 from ceph/wip-fix-asok

mds: fix asok on rank 0

mds: fix asok on rank 0

Broken by:
1b2b34431 (MDS: do not allow invocation of most commands on an inactive MDS)

MDS::whoami is a rank, so check for inactive is <0, not <=0

Signed-off-by: John Spray <john.spray@redhat.com>

doc: Adds updated man page for ceph under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Changes format style in ceph to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-disk under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-disk with suggested changes.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-disk under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-disk.

Changes format style to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Show how new data pools can be added to CephFS for file layouts

packaging: package ceph-deploy(8)

Fixes merge conflict.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-deploy under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-deploy.

Changes format style in ceph-deploy to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

qa: test zero size xattr

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3198 from ceph/wip-10335

MDS: do not allow invocation of most commands on an inactive MDS

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

MDS: do not allow invocation of most commands on an inactive MDS

They just cause all kinds of trouble!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2836 from ceph/wip-7317

#7317 ENOSPC and OSD epoch barrier handling

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3197 from ktdreyer/rpm-spec-comment

ceph.spec.in: quote %files macro in comment

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: quote %files macro in comment

Prior to this commit, RPM would expand the %files macro that was present
in the comment.

Use a double percent sign to quote the macro so that RPM will not expand
it.

This fixes an rpmlint warning, "W: macro-in-comment %files"

More information from rpmlint's "-I" (help) command:

  $ rpmlint -I macro-in-comment
  macro-in-comment:
  There is a unescaped macro after a shell style comment in the
  specfile.  Macros are expanded everywhere, so check if it can cause a
  problem in this case and escape the macro with another leading % if
  appropriate.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3171 from tchaikov/refactor-rados-class

python-rados: refactor class Rados a little bit

Reviewed-by: Josh During <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

python-rados: refactor class Rados a little bit

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

pybind/cephfs: Fix setxattr function. Pass value as character pointer

This passes the value of the xattr correctly and makes it work.

Merge pull request #3190 from dachary/wip-test-helpers

tests: ignore xmlstarlet extra empty lines

tests: ignore xmlstarlet extra empty lines

With xmlstarlet 1.3.0 (which is on Ubuntu precise)

xmlstarlet sel -t -m '//pg_stat/state[(contains(.,'\''active'\'') and
contains(.,'\''clean'\'') and not(contains(.,'\''stale'\'')))]' -v . -n

will output an extra empty line which does not show on xmlstarlet
1.5.0 (which is on Ubuntu trusty and Fedora 20).

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os/FileStore: better debug on omap_setkeys

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: better debug on unlink

Signed-off-by: Sage Weil <sage@redhat.com>

os/MemStore: drop support for collection attrs

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: deprecated collection_getattr et al

Drop the functional tests for this too.

Return EOPNOTSUPP by default, and tolerate this in the callers.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore_workloadgen: set pgmeta omap instead of collection xattr

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop PG/collection attrs from ScrubMap

The collection attrs are often version specific.  We should compare the
semantic PG metadata, if anything, not these!

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: deprecated collection attr operations

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop PG::read_info support for ancient PGs

This is rewritten regularly by any vaguely-active OSD; no need for this
any more.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop snap_collections

This is obsolete since cuttlefish.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop infos_oid arg from PG::read_info()

Only needed for legacy.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop legacy arg to PG::peek_map_epoch

The function can figure it out itself, and it's only needed for legacy.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_objectstore_tool: use '_remove' pgmeta omap key to indicate removal too

We could conceivably mark the PG for backfill, but to do this correctly
we have to do the log rollback, which uses various methods in PG to update
snap metadata that we simply don't have access to from this tool.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move pg removal flag into a static helper

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_objectstore_tool: Prevent tool from transitioning struct_v on rm-past-intervals

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Clean-up unused struct_ver in tool's metadata section

Exported struct_ver not used by write_info() or write_pg()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: retire meta/infos object entirely

Do not create it; remove it on upgrade.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: prevent upgrade from older than v7

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: drop support for ancient logs

Assume that users have at least upgraded to dumpling first.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-test-helper.sh: enable filestore log

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move pg log and info to a per-pg object

Move all PG metadata (info, biginfo, epoch, and log) to a single per-PG
object.  This should make it easier for ObjectStore implementations to
avoid lock contention in the write path.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: explicitly pass log_oid collection

Do not assume the log_oid is ine the 'meta' collection.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: take ghobject_t instead of hobject_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move PG collection creation into static PG method

This way we properly create the pgmeta object and set the infokey attr on
it.  Break it into two steps, one part before split, one after.

Disable the collection_empty() check in split so that it doesn't trip over
this object.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: don't need to pass object names via ctor

Calculate biginfo on the fly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: define pgmeta_oid member

Signed-off-by: Sage Weil <sage@redhat.com>