erasure-code: initialize all data members

As a safeguard against unitialized memory in isa and jerasure
plugins. Since the init method is supposed to be called before the
plugin is used, it is mostly not relevant. However it can lead to
problems that are non trivial to figure out.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/master' into hammer

Merge pull request #3092 from abartlet/ceph-dmcrypt-plain

Ceph dmcrypt with LUKS (split into 4 patches)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge branch 'hammer' of github.com:ceph/ceph into hammer

mon bits
Reviewed-by: Sage Weil <sage@redhat.com>
cli bits
Reviewed-by: Dan Mick <dmick@redhat.com>

mon/PGMonitor: drop pg ls debug line

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3691 from athanatos/wip-6003

FileJournal: fix journalq population in do_read_entry()

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3690 from athanatos/wip-10817

Wip 10817

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3689 from athanatos/wip-10771

ReplicatedPG::eval_repop: check waiting_for_* even if !m

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-7701' of git://github.com/xinxinsh/ceph into hammer

Conflicts:
src/osd/osd_types.h

Merge branch 'dachary-wip-10728-fedora-junit'

ceph.spec.in: junit always except for EPEL 6

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: junit always except for EPEL 6

The package was renamed a long time ago (around the Fedora 15
timeframe). The "junit4" name is only relevant for EPEL 6. For EPEL 7
and Fedora 20, the "junit" package has "Provides: junit4". And most
recently, in the junit package that ships in Fedora 21 and 22, the
package maintainer dropped the old Provides: line.

http://tracker.ceph.com/issues/10728 Fixes: #10728

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

rgw: Swift API. The second way of specifying desirable response format.

OpenStack Object Storage API v1 defines two ways for the client to tell which response format it understands.
Until now RGW looked at query variable 'format' only. This commit implements the second way of setting the format by
providing 'Accept' HTTP-header with the desirable response mime-type.

Backport: hammer
Fixes: #10746
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

Merge pull request #3682 from ceph/wip-10719

Wip 10719

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3680 from ceph/wip-10827

osdc/Objecter: linger_register now acquires rwlock

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove multipart entries for bucket index when aborting

Fixes #10719

Since we now let the gc clean the aborter multipart uploads, we need to
clear the multipart entries from the bucket index when cleaning up the
meta object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: encode rgw_obj::orig_obj

We weren't encoding orig_obj, however, it's now needed so that we can
call get_index_key() on decoded objects. Only encode/decode it if ns or
instance are not empty.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3672 from ceph/wip-old-gcc

fix build and warnings on older gcc

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge branch 'wip-diurchenko-master' into hammer

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Swift API. Support for X-Remove-Container-Meta-{key} header.

Fixes: #10475
Backport: hammer, firefly
Reported-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

rgw: fix doc, A typo in command line argument name

Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

rgw: Swift API. Dump container's custom metadata.

Fixes: #10665
Backport: hammer, firefly
Reported-by: Ahmad Faheem <ahmad.faheem@ril.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

Merge pull request #3525 from qiushanggao/master

Fix bug: When run Test_filejournal  testcase with gtest argument, all of testcases is failed.

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3677 from tchaikov/fix-sign-unsigned-comparison-warnings

erasure-code: fix compilation warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix bug: When run Test_filejournal testcase with gtest argument, all of testcases is failed.

when run testcase with gtest argument, for example, with argument --gtest_output=xml:/root/reports/ceph_test_cls_version.xml,
then the test result is failed, because of the test program use the first argument as the journal file name.
Signed-off-by: shanggao qiu <qiushanggao@qq.com>

erasure-code: fix compilation warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3673 from tchaikov/cosmetic-init-with-false-no-0

osd/PG.c: use boolean to init `bool` variables

Reviewed-by: Loic Dachary <ldachary@redhat.com>

fsync-tester: print info about PATH and locations of lsof lookup

We're seeing the lsof invocation fail (as not found) in testing and nobody can
identify why. Since attempting to reproduce the issue has not worked, this
patch will gather data from a genuinely in-vitro location.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ShardedThreadPool: make wait timeout on empty queue configurable

Fixes: 10818
Backport: giant
Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PG.c: use boolean to init `bool` variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

WorkQueue: make wait timeout on empty queue configurable

Fixes: 10817
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>

add tests for ceph pg ls-by-pool

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls-by-pool

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add tests for ceph pg ls-by-osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls-by-osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add tests for ceph pg ls-by-primary

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

fix error : ceph pg ls 0

no valid command found; 10 closest matches:
pg ls-by-primary <osdname (id|osd.id)> {<int>} {active|clean|down|replay|splitting|scrubbing|scrubq|degraded|inconsistent|peering|repair|recovery|backfill_wait|incomplete|stale|remapped|deep_scrub|backfill|backfill_toofull|recovery_wait|undersized [active|clean|down|replay|splitting|scrubbing|scrubq|degraded|inconsistent|peering|repair|recovery|backfill_wait|incomplete|stale|remapped|deep_scrub|backfill|backfill_toofull|recovery_wait|undersized...]}
Error EINVAL: invalid command

this is due to partial compare of prefix

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

locally disable pragma warnings on gcc < 4.6

diagnostic push|pop are not present before 4.6, so each use gives a
warning. Temporarily ignore these by disabling/reenabling -Wpragmas.
Note that this means the other disabled warnings are not scoped on
old gcc. I don't think this is worth fixing, since newer gcc will
respect the scope and show us the warnings outside of it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Pipe: conditionally compile IPTOS setting

This doesn't exist on older platforms.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

librbd: consolidate all async operation flush logic

librbd has three different methods for flushing asynchronous
operations. These have all been consolidated down to a single,
shared method to fix an existing issue and simplify maintenance
going forward.

Fixes: #10783
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xlist: added missing include

The xlist header references NULL without including
the header which defines it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG::eval_repop: check waiting_for_* even if !m

There can be client ops waiting on a promote now.

Fixes: 10771
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3602 from dachary/wip-10730-fedora-docker

tests: add Dockerfile for fedora

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3656 from dachary/wip-osd-scrub-repair

tests: run osd-scrub-repair.sh with make check

Reviewed-by: Sebastien Ponce <Sebastien.Ponce@cern.ch>

Merge pull request #3631 from liewegas/wip-mon-dup-down

mon: fix unnecessary osd mark down

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3605 from ceph/wip-da-SCA-20150129

Various SCA and smaller bugfixes

Reviewed-by: Sage Weil <sage@redhat.com>

mon: have two COMMAND two COMMAND_WITH_FLAG macros

This avoids 2 separate calls to define each item and keeps the blame
info intact.

Signed-off-by: Sage Weil <sage@redhat.com>

tests: make ceph_watch_wait output watchfile content on failure

It should help in troubleshooting remote test failures.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

mon: noforward flag for commands that are not supposed to be forwarded

For not a monitor leader, if a received command is not supported
locally, but is supported by the leader, it is forwarded to the
leader.

For some commands that may have undesirable behaviour. E.g. for the
recently added "ceph tell mon.x version", if the mon.x is not a leader
and does not support "version" command yet, but the leader does, the
user will receive the version of the leader, and can't be actually
sure about a non leader version.

Fix this by adding noforward flag to commands that are not supposed to
be forwarded. Set the flag for "version" command. Although there are
other non-forwardable commands (like injectargs) it is not necessary
to add the flag to them, as the commands are too old to suffer from
the describe problem.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

mon: fix Monitor::_get_moncommand()

It should match command prefix, not any substring.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge branch 'wip-mshec-r44' of git://github.com/t-miyamae/ceph into hammer

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3667 from ceph/wip-doc-ports

doc: updates for port ranges

Reviewed-by: Sage Weil <sage@redhat.com>

doc: remind user to calculate the correct max port

This was kind of easy to miss, and if users
just paste in the command as given then their
first few OSDs will work, and from their
4th OSD onwards things will get weird.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: fix "7810" port typo

The correct value is now 7300, used to be
7100, was never 7810.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: correct default of ms_bind_port_max

It's 7300 in the code, whereas these docs
had it as 7100.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3665 from yuyuyu101/wip-fix-qa

AsyncConnection: Fix several bugs from qa test

Reviewed-by: Sage Weil <sage@redhat.com>

AsyncConnection: fix incorrect condition for exchanging in_seq

Originally we use "in_seq==0" to judge whether need to exchange in_seq,
it's wrong when peer side already receive message and need to reply new
in_seq to this side.

Now use "is_reset_from_peer" to indicate whether not need to exchange

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #3660 from SUSE/wip-fix-suse-tests

Handle differently-named xmlstarlet binary for *suse

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Handle differently-named xmlstarlet binary for *suse

Seems *SUSE sticks to the upstream binary name of 'xml'
for the xmlstarlet package.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #3657 from ceph/wip-10782

pybind: fixed runtime errors with librbdpy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: run osd-scrub-repair.sh with make check

Signed-off-by: Loic Dachary <ldachary@redhat.com>

FileJournal: fix journalq population in do_read_entry()

Fixes: 6003
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3627 from tchaikov/check_output-on-centos6

ceph_objectstore_tool: fix check_output on python2.6

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: fix check_output on python2.6

* backported the subprocess.check_output from python2.7

Fixes: #10756
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3652 from dachary/wip-mailmap

mailmap: UMD is University of Maryland

Merge pull request #3131 from mslovy/fix_init_filejournal_throttle

os: FileJournal:: fix, uninitialization of FileJournal throttle

Reviewed-by: Sage Weil <sage@redhat.com>

mailmap: UMD is University of Maryland

And not University of Mississippi

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3468 from ceph/wip-cephfs-snap1

Wip cephfs snap1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3649 from ceph/wip-opath-setattr

client: handle O_PATH fds on setattr too

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3633 from sponce/wip-10758

Fixed issue #10758 : write_full behavior in libradosstriper

Reviewed-by: Loic Dachary <ldachary@redhat.com>

libradosstriper: fixed write_full when ENOENT

Fixes: #10758
write_full was returning ENOENT when the file did not exists, while it should just have created it without complaining.

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #3624 from ktdreyer/wip-doc-submitting-patches-branches

SubmittingPatches: clarify the use of branches in PRs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: handle O_PATH fds on setattr too

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3626 from majianpeng/misc

rbd.py: Add rbd_read2/rbd_write2 funcion which can handle fadvise flags.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: obj delete operation can use remove_objs param

The remove_objs param is needed to atomically remove multiple entries
from the bucket index, e.g., when cancelling a multipart upload.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rbd.py: Add rbd_read2/rbd_write2 funcion which can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3642 from vuhuong/wip-hammer

Wip xio hammer

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3639 from ceph/wip-10770

rgw: send appropriate op to cancel bucket index pending operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3641 from athanatos/wip-10731

ReplicatedPG: block writes on degraded objects unless all peers support ...

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: send appropriate op to cancel bucket index pending operation

Fixes: #10770
Backport: firefly, giant

Reported-by: baijiaruo <baijiaruo@126.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3618 from tchaikov/add-norebalance-flag

Add norebalance flag

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: avoid propagrating unnecessary snap rstat to parent

if parent is not snapshotted, avoid propagrating snap rstat to it

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: update client_need_snapflush when splitting snap inode

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix rdlock snap inode

make sure the "sync head inode" code  execute

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: save the oldest snapid of inode

When moving a inode to different snapshot realm, we need to check if
the inode is snapshotted. If the inode is snapshotted, we create a
snapshot realm for it. The snapshot realm records inode's past snapshot
realm.

Currently there is no reliable way to check if a inode is snapshotted.
So add a new field to CInode the store the oldest snapid, it's used to
check if inode is snapshotted.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

qa/workunits/snaps: remove snapshots when test finishes

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: split old_inode when needed

Signed-off-by: Yan, Zheng <zyan@redhat.com>

qa/workunits/snaps: don't assume snapshot is disabled

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph: don't count snap dentries when calculating new fragstat/neststat

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: add snaprealm to open_past_parents after it's opened

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid infinite loop of SnapRealm::_open_parents

handle the case that some past parents are missing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly commit snaprealm to dirfrag

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't manully create snaprealm during rename

project_past_snaprealm_parent() will do the same job. Besides
MDCache::_snaprealm_create_finish() is buggy, it updates snaprealm's
current_parent_since incorrectly

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't mark dirfrag dirty after dropping stale dentries

now MDS use omap to store dirfrag, and always does partial commit.
marking dirfrag dirty after dropping stale dentries is useless.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: remove snapshot dentry during rmsnap

reply of RMSNAP request has no trace, we need to invalidate the
snapshot dentry manually

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: remove snaprealm from parent when closing it

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: check xattr version independent of inode version

Fixes: #10343
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove superfluous assertion

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: temporarily disable recovery for snapshotted files

recovery of snapshots is very inefficient, which requires
NR_SNAPS * NR_OBJECTS operations.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: verify backtrace by auth MDS only

Signed-off-by: Yan, Zheng <zyan@redhat.com>