tests: use port 7111 for osd-copy-from.sh

Because port 7110 is already used by mkfs.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3134 from ceph/wip-9059-checksums

osd: record and validate whole-object checksums during deep scrub

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: scrub: wait for digest updates to apply before next scrub chunk

Wait for any digest updates to apply before we scrub the next chunk.  This
bounds the number of repops we initiate by the size of the scrub chunk, and
it generally nicer to the cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change omap data encoding in object_copy_data_t

Pass the omap data in the struct as an opaque (encoded) bufferlist.  This
avoids a decode into an STL map, simplifying the crc calculation.  The
win isn't huge (yet) because we still turn it into a map to pass it down
to the ObjectStore method, but that too will change in time.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-copy-from: simple test of copy-from and error injection

Signed-off-by: Sage Weil <sage@redhat.com>

rados: use copy_from for rados cp (and cppool) command

A hammer CLI won't be able to run against a pre-firefly OSD, but I don't
think we care.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add copyfrom error injection

Unfortunately it's nontrivial to inject this from the usual
ceph_test_rados_api_* tests which need to run despite thrashing.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: pass and verify data+omap digest on copyfrom

Two things here:

 1- Pass the original digest from the source across the wire, if it is
    present.
 2- Calculate a new digest as we receive it, and record that.

If there is a mismatch, we currently crash; need to turn this into an
EIO, most likely.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: EIO if a full-object read produces a bad digest

Add a tunable to control this.  Returning an EIO instead of garbled data
may not be the best approach in all environments.

Eventually we want to do something better here, like read from a replica
instead.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: set a min age before we update whole-object digest

If an object is being actively updated, the whole-object digest will
quickly be invalidated.  On deep scrub, only record that digest if the
object is a few hours old.  Otherwise, we are wasting an IO.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: record whole-object digest on scrub

If we do not have a whole-object digest, record one after a deep scrub.

Note that we make no particular attempt to avoid this on frequently
changing objects where the digest will quickly be invalidated.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: complain about bad digest in final pass auth check

This will only trigger if we choose an auth that is known bad, but that
currently can happen, so compalin here too.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: be explicit about whether bad digest is known bad

The digest may just disagree with our best guess 'auth' shard, or it may
also disagree with the recorded oi digest.  Clarify which.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: disqualify shards with digests that disagree with oi digest

The auth may or may not know that the digest is truly bad.  Note that we
can only relate scrub digests to those in oi if the scrub used a seed of
-1; older OSDs use 0.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: share auth oi with caller and be_compare_scrub_objects

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move scrub no good auth check out of helper

This helps call out that this is an exceptional case and simplifies the
helper a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove more (most? all?) of classic scrub

Hopefully this is everything

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use -1 for deep scrub digest seed on new OSDs

0 is a weak initial value for a CRC since it doesn't change with a sequence
of 0 bytes (which are relatively common).  -1 is better.  Use -1 when
everyone in the acting set supports it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop vestigal invalid_snapcolls fields from scrub

This isn't used.  Snapcolls are ancient.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up scrub method debug output

Use __func__ instead of (usually inaccurate) names.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: kill some dead scrub code

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: set and invalidate data/omap digests on osd ops

When we create an empty object, we have a known digest of -1.  In a few
cases, we also know the digest from the op (e.g., writefull).  The rest
of the time we invalidate any existing digest.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add {data,omap}_digest to object_info_t

Also add flags to indicate whether the fields are valid.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3227 from nilamdyuti/wip-doc-ceph-deploy

Fixes a typo in ceph-deploy man page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adds updated ceph-deploy man page under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Fixes a typo in ceph-deploy man page.

Lack of a single space was restricting the proper rendering of an option in
html. Fixes the error.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3225 from dachary/wip-10400-quota-info-t

mds: add default ctor for quota_info_t

mds: add default ctor for quota_info_t

http://tracker.ceph.com/issues/10400 Fixes: #10400

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: v0.90

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3199 from ceph/wip-qa-empty-xattr

qa: test zero size xattr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/wip-fs-quota'

Conflicts:
src/client/Client.cc

Merge pull request #3218 from dachary/wip-10383-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10383 Refs: #10383

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2976 from ceph/wip-pgmeta

osd: move PG metadata to a per-PG object

Passed rados and upgrade tests.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3212 from ceph/wip-10255-wusui

Remove sepia dependency (use fqdn)

Backport: giant
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3214 from xinxinsh/wip-cleanup

cleanup : remove sync_epoch

Reviewed-by: Sage Weil <sage@redhat.com>

0.90

doc: Instead of using admin socket, use 'ceph daemon' command.

Merge pull request #3193 from nilamdyuti/wip-doc-ceph-disk

Changes format style in ceph-disk to improve readability as html

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3210 from ceph/wip-osdmap

osd: only verify OSDMap crc if it is known

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3216 from cstavr/master

ceph-disk: Fix wrong string formatting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: Fix wrong string formatting

Signed-off-by: Christos Stavrakakis <stavr.chris@gmail.com>

cleanup : remove sync_epoch

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Remove sepia dependency (use fqdn)

Fixes: #10255
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #3209 from dmick/wip-stop.sh-me

stop.sh: only try killing processes that belong to me

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: only verfy OSDMap crc if it is known

Only verify we encode a full map with the correct CRC if we actually
have the value in the Incremental.  Otherwise, any map from an old
mon will fail.  And we'll try to request a full map with a message the
old mon doesn't understand.

Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: only try killing processes that belong to me

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3207 from ceph/wip-cot-rename

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3208 from ceph/wip-10326

qa/workunits/rest/test.py: fix pg stat test

qa/workunits/rest/test.py: fix pg stat test

Back in e27b0d9e42b7ac3de896c3398689bbdbe03f1bbb we changed pg stat to
not dump everything when a formatter is used.

Fixes: #10326
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3195 from nilamdyuti/wip-doc-ceph-deploy

doc: Changes format style in ceph-deploy to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3201 from nilamdyuti/wip-doc-ceph

doc: Changes format style in ceph to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3057 from ceph/wip-crush-straw

crush: fix straw bucket scaler bugs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

.gitmodules: update ceph-object-corpus auth repo to github

It's annoying having ceph.com for only this repo, and it's not that big.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3101 from yuyuyu101/wip-10147

Messenger Unit Tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3205 from ceph/wip-caps-init

messages/MClientCaps: init peer.flags

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3123 from linuxbox2/wip-claim-3

Wip claim 3

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3111 from yuyuyu101/wip-10193

WBThrottle: make bytes/ios/inode_wb's perf counter effective

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3133 from XinzeChi/wip-2

OSD: FileJournal: call writeq_cond.Signal if necessary in submit_entry

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3132 from mslovy/optimazation_collection_map

os: IndexManager:: optimize map<coll_t, CollectionIndex* > col_indices to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3204 from ceph/wip-osdmap-leak

osd: fix leaked OSDMap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: Fix time event is called after AsyncMessenger destruction

AsyncConnection uses time event to handle async event partially, but it's not
very effective so here use dispatch_event_external instead.

And if client try to connect to server after a period, it may be call
AsyncConnection::process which will reference to AsyncMessenger. Since
AsyncMessenger doesn't use reference count, it will result in segment
fault. Now we record time event id and delete these registered time events
when stopping connection.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

PipeConnection: Avoid deadlock when calling is_connected

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test/msgr/test_msgr: Fix potential unsafe cond wakeup and wrap check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

messages/MClientCaps: init peer.flags

*** CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
/messages/MClientCaps.h: 132 in MClientCaps::MClientCaps(int, inodeno_t,
inodeno_t, unsigned long, int, unsigned int)()
126         head.ino = ino;
127         head.realm = realm;
128         head.cap_id = id;
129         head.migrate_seq = mseq;
130         peer.cap_id = 0;
131         inline_version = 0;
>>>     CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
>>>     Non-static class member field "peer.flags" is not initialized in this
constructor nor in any functions that it calls.
132       }
133     private:
134       ~MClientCaps() {}
135
136     public:
137       const char *get_type_name() const { return "Cfcap";}

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix leaked OSDMap

*** CID 1258787:  Resource leak  (RESOURCE_LEAK)
/osd/OSD.cc: 6118 in OSD::handle_osd_map(MOSDMap *)()
6112                    << dendl;
6113            clog->warn() << "failed to encode map e" << e << " with expecte
crc\n";
6114            MMonGetOSDMap *req = new MMonGetOSDMap;
6115            req->request_full(e, last);
6116            monc->send_mon_message(req);
6117            last = e - 1;
>>>     CID 1258787:  Resource leak  (RESOURCE_LEAK)
>>>     Variable "o" going out of scope leaks the storage it points to.
6118            break;
6119           }
6120
6121
6122           hobject_t fulloid = get_osdmap_pobject_name(e);
6123           t.write(META_COLL, fulloid, 0, fbl.length(), fbl);

Signed-off-by: Sage Weil <sage@redhat.com>

Messenger: Create an Messenger implementation by name.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3202 from ceph/wip-fix-asok

mds: fix asok on rank 0

mds: fix asok on rank 0

Broken by:
1b2b34431 (MDS: do not allow invocation of most commands on an inactive MDS)

MDS::whoami is a rank, so check for inactive is <0, not <=0

Signed-off-by: John Spray <john.spray@redhat.com>

doc: Adds updated man page for ceph under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Changes format style in ceph to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-disk under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-disk with suggested changes.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-disk under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-disk.

Changes format style to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Show how new data pools can be added to CephFS for file layouts

packaging: package ceph-deploy(8)

Fixes merge conflict.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds updated man page for ceph-deploy under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Updates man page for ceph-deploy.

Changes format style in ceph-deploy to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

qa: test zero size xattr

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3198 from ceph/wip-10335

MDS: do not allow invocation of most commands on an inactive MDS

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

MDS: do not allow invocation of most commands on an inactive MDS

They just cause all kinds of trouble!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2836 from ceph/wip-7317

#7317 ENOSPC and OSD epoch barrier handling

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3197 from ktdreyer/rpm-spec-comment

ceph.spec.in: quote %files macro in comment

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: quote %files macro in comment

Prior to this commit, RPM would expand the %files macro that was present
in the comment.

Use a double percent sign to quote the macro so that RPM will not expand
it.

This fixes an rpmlint warning, "W: macro-in-comment %files"

More information from rpmlint's "-I" (help) command:

  $ rpmlint -I macro-in-comment
  macro-in-comment:
  There is a unescaped macro after a shell style comment in the
  specfile.  Macros are expanded everywhere, so check if it can cause a
  problem in this case and escape the macro with another leading % if
  appropriate.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3171 from tchaikov/refactor-rados-class

python-rados: refactor class Rados a little bit

Reviewed-by: Josh During <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

python-rados: refactor class Rados a little bit

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

pybind/cephfs: Fix setxattr function. Pass value as character pointer

This passes the value of the xattr correctly and makes it work.

Merge pull request #3190 from dachary/wip-test-helpers

tests: ignore xmlstarlet extra empty lines

tests: ignore xmlstarlet extra empty lines

With xmlstarlet 1.3.0 (which is on Ubuntu precise)

xmlstarlet sel -t -m '//pg_stat/state[(contains(.,'\''active'\'') and
contains(.,'\''clean'\'') and not(contains(.,'\''stale'\'')))]' -v . -n

will output an extra empty line which does not show on xmlstarlet
1.5.0 (which is on Ubuntu trusty and Fedora 20).

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os/FileStore: better debug on omap_setkeys

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: better debug on unlink

Signed-off-by: Sage Weil <sage@redhat.com>

os/MemStore: drop support for collection attrs

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: deprecated collection_getattr et al

Drop the functional tests for this too.

Return EOPNOTSUPP by default, and tolerate this in the callers.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore_workloadgen: set pgmeta omap instead of collection xattr

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop PG/collection attrs from ScrubMap

The collection attrs are often version specific.  We should compare the
semantic PG metadata, if anything, not these!

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: deprecated collection attr operations

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop PG::read_info support for ancient PGs

This is rewritten regularly by any vaguely-active OSD; no need for this
any more.

Signed-off-by: Sage Weil <sage@redhat.com>