tests: ceph-disk: multipath now auto activates

http://tracker.ceph.com/issues/12786 Refs: #12786

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: use sh() instead of helper() when possible

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: doc: ceph-disk workunit reminders

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: reduce debug verbosity

There is no need for debug traces from which() and it is very
noisy. Just get rid of it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: ensure udev add on the data partition is last

When calling partprobe, we make sure there is at least one udev add
called for each partition created when preparing a device. But there is
no guarantee that the udev add for data partition will be last and the
following scenario can happen:

 - udev add data partition fails because the journal partition is owned
   by root
 - udev add journal partition chown the journal partition
 - no other udev add event is sent and the OSD does not activate

An additional, possibly redundant, udev add event is fired after
partprobe is run and after udevadm settles, to guarantee there is at
least one udev add data partition after the last udev add journal
partition.

http://tracker.ceph.com/issues/12787 Fixes: #12787

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: move update_partition from main_prepare to prepare_dev

The update_partition call in main_prepare happens immediately after
prepare_dev but only if the data argument is a block device. There is no
reason for this separation: it is more sensible to call it from within
prepare_dev.

There is an additional test in prepare_dev that verifies partprobe won't
be called on a partition because it would not make sense.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: guard partprobe with udevadm settle

Call udevadm settle before and after partprobe.

A side effect of partprobe is to remove partitions and add them again.
The first udevadm settle waits for ongoing udev events to complete, just
in case one of them rely on an existing partition on dev.

The second udevadm settle guarantees to the caller that all udev events
related to the partition table change have been processed, i.e. the
95-ceph-osd.rules actions and mode changes, group changes etc. are
complete.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: do not hide ceph-disk prepare stack trace

raise without an argument so the original stack trace is shown when
ceph-disk prepare fails.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: support --verbose with --log-stdout

Set the LOG level as well as the channel level, otherwise the debug
messages are trimmed before they reach the channel. Also set the prefix
while we're at it.

http://tracker.ceph.com/issues/13180 Fixes: #13180

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: prefer sgdisk to blkid to retrieve partition UUID

blkid 2.23.2 which is the default for official CentOS 7 cloud images
fails on journal device. It would be better to use blkid because it does
not trigger udev events, but it is more important to get reliable
results.

http://tracker.ceph.com/issues/13153 Fixes: #13153

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: make ceph-disk list /dev/vdb equivalent to list vdb

The ceph-disk list argument must be the device name without the leading
/dev/. This is error prone and silently does nothing. Strip the /dev/
prefix of ceph-disk list arguments so that it behaves as expected.

http://tracker.ceph.com/issues/13154 Fixes: #13154

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: - is translated into / with systemd

When running ceph-disk trigger /dev/dm-1 with systemd, the path name is
translated into /dev/dm/1 because of systemd escape rules. Explicitly
translate - into \x2d for systemd to preserve the -.

It would be better to use systemd-escape
http://www.freedesktop.org/software/systemd/man/systemd-escape.html
but it does not appear to be generally available on CentOS 7 and
probably other distributions.

http://tracker.ceph.com/issues/13174 Fixes: #13174

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: a journal partition may survive a data partition

When a data partition is removed and the journal partition is not
removed, ceph-disk list will not find a journal_for information and
should just ignore it.

http://tracker.ceph.com/issues/13157 Fixes: #13157

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: upstart must not run ceph-disk activate concurrently

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: systemd must not kill a running ceph-disk

When activating a device, ceph-disk trigger restarts the ceph-disk
systemd service. Two consecutive udev add on the same device will
restart the ceph-disk systemd service and the second one may kill the
first one, leaving the device half activated.

The ceph-disk systemd service is instructed to not kill an existing
process when restarting. The second run waits (via flock) for the second
one to complete before running so that they do not overlap.

http://tracker.ceph.com/issues/13160 Fixes: #13160

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: fix typos in udev rules

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: ensure ceph owner on udev change

On udev change the owner of the device switch back to the default. If
that happens on a journal while an OSD is being activated, it will fail
with permission denied.

Make sure all ceph device types are chown to ceph on udev change.

http://tracker.ceph.com/issues/13000 Fixes: #13000

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5750 from wonzhq/rbd-write-full

librbd: do write_full for whole object write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5416 from zhouyuan/ceph_fuse_12204

ceph-fuse: warn and shut down when there is no MDS present

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5826 from ukernel/wip-osx-port

fix compilation on OSX/FreeBSD

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-12971' of git://github.com/jcsp/ceph

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Conflicts:
src/osdc/Objecter.cc

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5644 from majianpeng/mds-support-iohint

Mds journal support iohint

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5966 from ukernel/wip-12506

Wip 12506

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5962 from majianpeng/client-fix

Client fix mini bugs.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6014 from athanatos/wip-13108

Revert "osd: purge the object from the cache when proxying and not pr…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6015 from athanatos/wip-13143

osd/: find_object_context: return obc for head for snapdir if non null

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6017 from athanatos/wip-13150

ReplicatedPG::get_snapset_context: set exists to false for new ssc wi…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6013 from athanatos/wip-12828

Wip 12828

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5918 from oritwas/wip-5378

rgw: make radosgw-admin user rm idempotent

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5996 from rzarzynski/wip-12780

rgw: don't read actual data on user manifest HEAD

Merge pull request #5974 from Sandy4999/wip-rgw-errmsg

rgw: add a new error message for user conflict when using Admin Ops API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6003 from ukernel/wip-13166

Wip 13166

Reviewed-by: John Spray <john.spray@redhat.com>

mds: adjust MDSRank::incarnation according to mdsmap

When a standby-replay MDS replace failed MDS, we need update its
incarnation.

Fixes: #13166
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #6001 from dachary/wip-13182-crushtool

tests: update to match crushmap validation message

tests: update to match crushmap validation message

http://tracker.ceph.com/issues/13182 Fixes: #13182

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5979 from ceph/wip-crushtest

mon: fix crush testing for new pools

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5950 from tchaikov/wip-13112-infernalis

mon/PGMap: calc min_last_epoch_clean when decode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5998 from dachary/wip-13136-python3-coverage

install-deps: enable python3

install-deps: enable python3

The upstream regression has been fixed, we can re-enable python3
support.

http://tracker.ceph.com/issues/13136 Fixes: #13136

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5994 from SUSE/wip-13175

doc: do not promise backports to Dumpling

Reviewed-by: Sage Weil <sage@redhat.com>

doc: do not promise backports to Dumpling

Dumpling was retired in May 2015.

http://tracker.ceph.com/issues/13175 Fixes: #13175

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #5965 from jcsp/wip-get-or-create-output

mon: fix auth get-or-create output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5984 from ceph/wip-packages

Fixes locations from ceph.com to download.ceph.com

(For the record, this is to fix http://tracker.ceph.com/issues/13162)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: remove mention of ceph-extra as a requirement

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: remove ceph-extras

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: correct links to download.ceph.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: fix auth get-or-create output

Previously the caps were omitted from the
output: they should be present for this
to fulfil the 'get' part of get-or-create.

Signed-off-by: John Spray <john.spray@redhat.com>

ReplicatedPG::get_snapset_context: set exists to false for new ssc with can_create

Otherwise, if we do a proxy_write and don't promote it, the ssc will
stick around with exists = true and cause future reads on snaps for the
object to fail to fall through to the base tier.

Fixes: #13150
Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: don't read actual data on user manifest HEAD

Fixes: #12780
We unconditionally read all the data, which is not needed
when doing HEAD operation on user manifest objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ceph-osd-prestart.sh: no ceph-disk chown

chown -R ceph:ceph $1 is easy enough.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5980 from liewegas/wip-13059

ceph.spec: fix up /var/run/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: Fix up (/var)/run/ceph creation

Fixes: #13059
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5977 from tchaikov/wip-size_t-monstore-txn

mon: return size_t from MonitorDBStore::Transaction::size()

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: fix crush injection error message

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: only test crush ruleset for the newly created pool

Otherwise, we test *all* crush rules.. which might be a lot, and
which is a big waste of time and effort.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushTester: allow testing by ruleset

Signed-off-by: Sage Weil <sage@redhat.com>

mon: return size_t from MonitorDBStore::Transaction::size()

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add a new error message for user conflict when using Admin Ops API

Add a new error message to indicate user conflict, which was using the error code for bucket conflict.

Signed-off-by: Sangdi Xu <xu.sangdi@h3c.com>

Merge pull request #5948 from flyd1005/master

doc: delete wrong description of installing RPMs

osd/: find_object_context: return obc for head for snapdir if non null

The head obc may have an in progress operation holding a lock, we
need to block on the lock for ordering reasons.

Fixes: #13143
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #5926 from dillaman/wip-13082

include/inline_memory: out-of-bounds read on unaligned memory

Reviewed-by: Sage Weil <sage@redhat.com>

mds: Make sure wanted_state of first MDSBeacon is MDSMap::STATE_BOOT

Fixes: #12506
Signed-off-by: Yan, Zheng <zyan@redhat.com>

new release key

The previous release key,

pub   4096R/460F3994 2015-09-15
uid                  Ceph.com (release key) <security@ceph.com>

may have been compromised.

The new release key is

pub   4096R/460F3994 2015-09-15
uid                  Ceph.com (release key) <security@ceph.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5947 from weiqian/master

doc: replaces 'osd host' with 'host' in ceph.conf docs.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5964 from jcsp/wip-13138

libcephfs: fix calling init() then mount()

libcephfs: fix calling init() then mount()

Previously only ever called these separately, but
it should be allowed for callers to use one after
the other.

Fixes: #13138
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5963 from jcsp/wip-cmake-cephfs

CMake: fix libcephfs shared lib generation

Merge pull request #5857 from ceph/wip-da-SCA-20150812

SCA and compiler warning fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5870 from s09816/master

osd:the fuction osd::shutdown Lock failed.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5096 from tchaikov/wip-add-bl-const-iterator

common/buffer: add bufferlist::const_iterator

Reviewed-by: Sage Weil <sage@redhat.com>

CMake: fix libcephfs shared lib generation

Previously weren't generating versioned symlinks
etc, so python bindings didn't find it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5920 from caibo2014/fix-tool-bug

tools: ceph_monstore_tool: remove the output file when get map fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5960 from dachary/wip-13136-python3-coverage

install-deps.sh: disable python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: disable python3

Disable python3 support until
https://bugs.launchpad.net/ubuntu/+source/python-coverage/+bug/1496715
is fixed.

Nothing in the build process depends on python3 right now, there is no
harm disabling it.

http://tracker.ceph.com/issues/13136 Fixes: #13136

Signed-off-by: Loic Dachary <ldachary@redhat.com>

client/MetaRequest: optimize func can_forward/auth_is_best.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

client/MetaRequest: open w/ O_CREAT|O_TRUNC is write.

From the function name is_write, it judge MetaRequest whether write.
For open, if it w/ O_CREAT|O_TRUNC, it's a write op and vice versa.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

clinet/MetaRequest: Remove the useless condition.

"head.op & CEPH_MDS_OP_WRITE" include "head.op == CEPH_MDS_OP_CREATE".
So remove the conditon statement "head.op == CEPH_MDS_OP_CREATE".

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon/PGMap: calc min_last_epoch_clean when decode

Fixes: #13112
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5841 from ceph/wip-12994-rbd-replay-packaging

#12994: packaging: move rbd-replay* to ceph-common

Reviewed-by: James Page <jamespage@ubuntu.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5946 from liewegas/wip-rgw-logdir

radosgw: log to /var/log/ceph instead of /var/log/radosgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: delete wrong description of installing RPMs

In CentOS/RHEL 6/7, gitk and git-gui are available in default repository actually.

Signed-off-by: Li Peng <lip@dtdream.com>

doc:Replaces 'osd host' with 'host'

Signed-off-by: weiqian <weiq@dtdream.com>

radosgw: log to /var/log/ceph instead of /var/log/radosgw

This is simpler.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5725 from liewegas/wip-scrub-stats

osd/ReplicatedPG: snaptimmer: adjust stats through ctx->delta_stats

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5925 from liewegas/wip-mon-leaks

fix various mon leaks

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5929 from liewegas/wip-osd-leaks

misc osd leaks

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: permit MDS to send writes on full pools

Same behaviour as the global full flag, for the
per-pool full flag.  The FS was broken by 67de12bf,
which added the pool flag handling.

Fixes: #12971
Signed-off-by: John Spray <john.spray@redhat.com>

Revert "osd: purge the object from the cache when proxying and not promoting the op"

Fixes: #13108
This reverts commit da68bb371f8b06b85a73077cd069a294193ac6c0.

Merge pull request #5944 from tchaikov/wip-13090

mds: cast numbers for mds health to string when print

Reviewed-by: John Spray <john.spray@redhat.com>

osdc/Objecter: make unlocked 'full' helpers private

These never needed to be public, let's be safe.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc: fix honor_osdmap_full

This was being ignored in new code that tested
the per-pool full flags.  Refactor it so that
all the tests of full flags go through _osdmap_pool_full
and thereby experience the honor_osdmap_full policy.

Fixes: #12971
Signed-off-by: John Spray <john.spray@redhat.com>

packaging: move rbd-replay* to ceph-common

The rbd-replay* utilities are useful for Ceph users with RBD clients.
Currently the rbd-replay* utilities ship in the "ceph-test" package, and
we intend this ceph-test package for Ceph developers and contributors,
not normal users.

Move the rbd-replay* utilities to "ceph-common".

http://tracker.ceph.com/issues/12994 Fixes: #12994

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

src/test/osd/Object: allow a hole at the end of the object

This way, we test the case with holes at the end of the file more
often.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types.cc: add size to ObjectRecoveryInfo print

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend::prepare_pull: set recover_info.size for clones

Fixes: #12828
Signed-off-by: Samuel Just <sjust@redhat.com>

mon: debug refs on output replies

Signed-off-by: Sage Weil <sage@redhat.com>

msg/simple: discard local queued messages on shutdown

Otherwise these generate leak detection noise.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Monitor: fix MonSession Leak

dispatch() does not need to keep a ref; we hold one indirectly
via the MonOpRequest.

Signed-off-by: Sage Weil <sage@redhat.com>