client: fix uninitialized member

Fixes: #10414
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3237 from dachary/wip-10408-something-is-better-than-nothing

osd: be_compare_scrubmaps uses incorrect j iterator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3236 from ceph/wip-10409

osd: fix 10409 (object checksums for ec backends)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: be_compare_scrubmaps uses incorrect j iterator

The code moved from be_select_auth_object to be_compare_scrubmaps
74bd8708dfbfd3c8e7ba3f41d8534609dcbc1237 but the j iterator is use
differently although it has the same type. Use map.begin() as a
fallback instead.

http://tracker.ceph.com/issues/10408 Fixes: #10408

Signed-off-by: Loic Dachary <ldachary@dachary.org>

osd: scrub: only assume shard digest == oi digest for replicated pools

For an EC object, the digest we get from scrub is for the *shard*, and that
is not the same as the *object* digest in the object_info_t.  Skip these
checks; we already have the per-shard digest that is verified in the EC
backend.

Fixes: #10409
Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up use of hex for digests

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: use correct seed for (empty) omap digest

This will be 0 on mixed OSD version PGs, -1 on current PGs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-10409-removed-object' of git://github.com/dachary/ceph

Merge remote-tracking branch 'gh/next'

Conflicts:
PendingReleaseNotes

Merge pull request #3230 from ceph/wip-pg-stat

mon/PGMap: restructure 'pg stat' formatted output

Reviewed-by: John Spray <jspray@redhat.com>

mon/PGMap: restructure 'pg stat' formatted output

The + character, which appears in state names, is not a valid XML token.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3235 from dachary/wip-10410-port-7110

tests: use port 7111 for osd-copy-from.sh

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3234 from majianpeng/bug-fix

objectstore: add fadvise_flags operations in ObjectStore::Transaction::append func.

Reviewed-by: Sage Weil <sage@redhat.com>

tests: use port 7111 for osd-copy-from.sh

Because port 7110 is already used by mkfs.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: recovery of a lost object in erasure coded pools

http://tracker.ceph.com/issues/10409 Refs: #10409

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore: add fadvise_flags operations in
ObjectStore::Transaction::append func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

tests: remove spurious lines from osd-scrub-repair.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3228 from dachary/wip-10401-docker-install-deps

tests: oneliner to run make check in a container

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3231 from ceph/wip-mon-health

mon: make 'ceph -s' show pg state counts in reverse descending order

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: reduce centos docker intermediate steps

Collapse multiple RUN statements related to systemd into a single line
to reduce the number of steps that are preserved by docker.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add docker-test.sh --ref giant|firefly etc.

Add the --ref option to chose the git ref to which the working tree must
be reset instead of HEAD.

   test/docker-test.sh --ref giant make

is equivalent to

   test/docker-test.sh bash -c 'git reset --hard giant ; make'

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: avoid bash == --shell confusion in docker-test-helper.sh

Do not force interactive mode when the bash script is specified by the
user because it should be usable to run a small script instead of
starting an interactive shell session.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add xmlstarlet in containers

It can be removed once https://github.com/ceph/ceph/pull/3228 is merged
and xmlstarlet is found in ceph.spec.in and debian/control.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: helpers to run make check in containers

Add scripts that run make check in a container for a given operating
system version. They are a little more than oneliners and serve two
purposes:

  * help new developers who do not have to figure out which options
    should be used

  * run make check in containers in parallel with make -j2

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: run-make-check.sh arguments are passed to configure

All run-make-check.sh arguments are transparently given to the configure
command. For instance:

    run-make-check.sh --enable-root-make-check

implies

    ./configure --enable-root-make-check

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: replace --enable-docker with --enable-root-make-check

The --enable-docker logic was that each test that needs privileges to
run uses a container to do so. The problem with this approach and
make -j8 check is that such tests compete for a single container.

The --enable-root-make-check activates tests that require privileges and
assume it is ok to use sudo to acquire privileges. The decision to run
in a container is then taken by the caller who will run something like:

   docker-tests.sh ./configure --enable-root-make-check
   docker-tests.sh make -j8 check

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: docker-test.sh commands are relative to git root

Instead of being in the src directory. Forcing the working directory to
src is convenient to run unittests individually without the need to
change directory, but it is confusing to the user.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: deprecate docker-test.sh --compile option

It is redundant with the run-make-check.sh script and imposes additional
limitations.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: enable password-less sudo in containers

If compilation is run with ./configuire --enable-root-make-check, sudo
is expected to work without human interaction.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph.spec.in or debian/control rebuild docker images

If any of the files in test/$os_type is newer than the creation date of
a docker image, the image is removed and rebuilt from scratch. For
instance, when a package is added to debian/control, the debian based
images are rebuilt.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: docker images must use install-deps.sh

The dockerfiles must run install-deps.sh instead of duplicating the
package list that is found in ceph.spec.in and debian/control.

A directory is created for each os_type and provided as a context for
docker build to use. The former $os_type.dockerfile is moved into
$os_type/Dockerfile.in (the .in as a reminder that it will be variable
substituted).

http://tracker.ceph.com/issues/10401 Fixes: #10401

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: run-make-check.sh install jq

jq is useful to parse json from the command line. It is however not
packaged for all supported distributions (precise has it in the backport
repository which is usually not active) and cannot be conveniently added
to debian/control because it targets all distributions and has no
conditionals.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

install-deps.sh: do not require sudo when root

If the user is root, do not use sudo. The sudo package is not installed
by default on centos by default and when building from script it may be
that root is running install-deps.sh on a freshly install distribution.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3134 from ceph/wip-9059-checksums

osd: record and validate whole-object checksums during deep scrub

Reviewed-by: Samuel Just <sjust@redhat.com>

mon: make 'ceph -s' show pg state counts in reverse descending order

Because these are in an unordered_map we print this is random order.
Instead sort descending by count.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: wait for digest updates to apply before next scrub chunk

Wait for any digest updates to apply before we scrub the next chunk.  This
bounds the number of repops we initiate by the size of the scrub chunk, and
it generally nicer to the cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change omap data encoding in object_copy_data_t

Pass the omap data in the struct as an opaque (encoded) bufferlist.  This
avoids a decode into an STL map, simplifying the crc calculation.  The
win isn't huge (yet) because we still turn it into a map to pass it down
to the ObjectStore method, but that too will change in time.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-copy-from: simple test of copy-from and error injection

Signed-off-by: Sage Weil <sage@redhat.com>

rados: use copy_from for rados cp (and cppool) command

A hammer CLI won't be able to run against a pre-firefly OSD, but I don't
think we care.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add copyfrom error injection

Unfortunately it's nontrivial to inject this from the usual
ceph_test_rados_api_* tests which need to run despite thrashing.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: pass and verify data+omap digest on copyfrom

Two things here:

 1- Pass the original digest from the source across the wire, if it is
    present.
 2- Calculate a new digest as we receive it, and record that.

If there is a mismatch, we currently crash; need to turn this into an
EIO, most likely.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: EIO if a full-object read produces a bad digest

Add a tunable to control this.  Returning an EIO instead of garbled data
may not be the best approach in all environments.

Eventually we want to do something better here, like read from a replica
instead.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: set a min age before we update whole-object digest

If an object is being actively updated, the whole-object digest will
quickly be invalidated.  On deep scrub, only record that digest if the
object is a few hours old.  Otherwise, we are wasting an IO.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: record whole-object digest on scrub

If we do not have a whole-object digest, record one after a deep scrub.

Note that we make no particular attempt to avoid this on frequently
changing objects where the digest will quickly be invalidated.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: complain about bad digest in final pass auth check

This will only trigger if we choose an auth that is known bad, but that
currently can happen, so compalin here too.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: be explicit about whether bad digest is known bad

The digest may just disagree with our best guess 'auth' shard, or it may
also disagree with the recorded oi digest.  Clarify which.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: disqualify shards with digests that disagree with oi digest

The auth may or may not know that the digest is truly bad.  Note that we
can only relate scrub digests to those in oi if the scrub used a seed of
-1; older OSDs use 0.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: scrub: share auth oi with caller and be_compare_scrub_objects

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move scrub no good auth check out of helper

This helps call out that this is an exceptional case and simplifies the
helper a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove more (most? all?) of classic scrub

Hopefully this is everything

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use -1 for deep scrub digest seed on new OSDs

0 is a weak initial value for a CRC since it doesn't change with a sequence
of 0 bytes (which are relatively common).  -1 is better.  Use -1 when
everyone in the acting set supports it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop vestigal invalid_snapcolls fields from scrub

This isn't used.  Snapcolls are ancient.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up scrub method debug output

Use __func__ instead of (usually inaccurate) names.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add {data,omap}_digest to object_info_t

Also add flags to indicate whether the fields are valid.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: kill some dead scrub code

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: set and invalidate data/omap digests on osd ops

When we create an empty object, we have a known digest of -1.  In a few
cases, we also know the digest from the op (e.g., writefull).  The rest
of the time we invalidate any existing digest.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3227 from nilamdyuti/wip-doc-ceph-deploy

Fixes a typo in ceph-deploy man page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Adds updated ceph-deploy man page under man/

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Fixes a typo in ceph-deploy man page.

Lack of a single space was restricting the proper rendering of an option in
html. Fixes the error.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #3225 from dachary/wip-10400-quota-info-t

mds: add default ctor for quota_info_t

mds: add default ctor for quota_info_t

http://tracker.ceph.com/issues/10400 Fixes: #10400

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3121 from ceph/wip-10277

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/release-notes: v0.90

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3199 from ceph/wip-qa-empty-xattr

qa: test zero size xattr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/wip-fs-quota'

Conflicts:
src/client/Client.cc

Merge pull request #3218 from dachary/wip-10383-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10383 Refs: #10383

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2976 from ceph/wip-pgmeta

osd: move PG metadata to a per-PG object

Passed rados and upgrade tests.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3212 from ceph/wip-10255-wusui

Remove sepia dependency (use fqdn)

Backport: giant
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3214 from xinxinsh/wip-cleanup

cleanup : remove sync_epoch

Reviewed-by: Sage Weil <sage@redhat.com>

0.90

doc: Instead of using admin socket, use 'ceph daemon' command.

Merge pull request #3193 from nilamdyuti/wip-doc-ceph-disk

Changes format style in ceph-disk to improve readability as html

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3210 from ceph/wip-osdmap

osd: only verify OSDMap crc if it is known

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3216 from cstavr/master

ceph-disk: Fix wrong string formatting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: Fix wrong string formatting

Signed-off-by: Christos Stavrakakis <stavr.chris@gmail.com>

cleanup : remove sync_epoch

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Remove sepia dependency (use fqdn)

Fixes: #10255
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #3209 from dmick/wip-stop.sh-me

stop.sh: only try killing processes that belong to me

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: only verfy OSDMap crc if it is known

Only verify we encode a full map with the correct CRC if we actually
have the value in the Incremental.  Otherwise, any map from an old
mon will fail.  And we'll try to request a full map with a message the
old mon doesn't understand.

Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: only try killing processes that belong to me

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3207 from ceph/wip-cot-rename

ceph_objectstore_tool: Rename generated binary to ceph-objectstore-tool

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3208 from ceph/wip-10326

qa/workunits/rest/test.py: fix pg stat test

qa/workunits/rest/test.py: fix pg stat test

Back in e27b0d9e42b7ac3de896c3398689bbdbe03f1bbb we changed pg stat to
not dump everything when a formatter is used.

Fixes: #10326
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3195 from nilamdyuti/wip-doc-ceph-deploy

doc: Changes format style in ceph-deploy to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3201 from nilamdyuti/wip-doc-ceph

doc: Changes format style in ceph to improve readability as html.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3057 from ceph/wip-crush-straw

crush: fix straw bucket scaler bugs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

.gitmodules: update ceph-object-corpus auth repo to github

It's annoying having ceph.com for only this repo, and it's not that big.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3101 from yuyuyu101/wip-10147

Messenger Unit Tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3205 from ceph/wip-caps-init

messages/MClientCaps: init peer.flags

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3123 from linuxbox2/wip-claim-3

Wip claim 3

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3111 from yuyuyu101/wip-10193

WBThrottle: make bytes/ios/inode_wb's perf counter effective

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3133 from XinzeChi/wip-2

OSD: FileJournal: call writeq_cond.Signal if necessary in submit_entry

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3132 from mslovy/optimazation_collection_map

os: IndexManager:: optimize map<coll_t, CollectionIndex* > col_indices to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3204 from ceph/wip-osdmap-leak

osd: fix leaked OSDMap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: Fix time event is called after AsyncMessenger destruction

AsyncConnection uses time event to handle async event partially, but it's not
very effective so here use dispatch_event_external instead.

And if client try to connect to server after a period, it may be call
AsyncConnection::process which will reference to AsyncMessenger. Since
AsyncMessenger doesn't use reference count, it will result in segment
fault. Now we record time event id and delete these registered time events
when stopping connection.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

PipeConnection: Avoid deadlock when calling is_connected

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test/msgr/test_msgr: Fix potential unsafe cond wakeup and wrap check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

messages/MClientCaps: init peer.flags

*** CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
/messages/MClientCaps.h: 132 in MClientCaps::MClientCaps(int, inodeno_t,
inodeno_t, unsigned long, int, unsigned int)()
126         head.ino = ino;
127         head.realm = realm;
128         head.cap_id = id;
129         head.migrate_seq = mseq;
130         peer.cap_id = 0;
131         inline_version = 0;
>>>     CID 1258789:  Uninitialized scalar field  (UNINIT_CTOR)
>>>     Non-static class member field "peer.flags" is not initialized in this
constructor nor in any functions that it calls.
132       }
133     private:
134       ~MClientCaps() {}
135
136     public:
137       const char *get_type_name() const { return "Cfcap";}

Signed-off-by: Sage Weil <sage@redhat.com>