build/ops: ceph-detect-init must ignore .cache

In case the test fails, it may leave files behind:

src/ceph-detect-init/.cache/v/cache/lastfailed

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/wip-aarch64'

Merge remote-tracking branch 'gh/infernalis'

arch/arm: s/false/0/

arch/arm.c: In function 'ceph_arch_arm_probe':
arch/arm.c:54:28: error: 'false' undeclared (first use in this function)
  ceph_arch_aarch64_crc32 = false;  // sorry!
                            ^
arch/arm.c:54:28: note: each undeclared identifier is reported only once for each function it appears in

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6037 from liewegas/wip-12407

init-radosgw: fix restart w/ multiple instances

Reviewed-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

Merge pull request #5928 from ceph/wip-12819-infernalis

Wip 12819 infernalis

Merge pull request #6006 from dachary/wip-12936-shec

erasure-code: workaround i386 optimization bug with SHEC

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw: specify pid file to start-stop-daemon

This fixes restart when multiple instances are running.

Fixes: #12407
Tested-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5922 from trociny/wip-pool-set-noscrub

osd: per pool control for scrub and deep-scrub

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5846 from xiexingguo/xxg-wip-13002

msg/simple: start over after fails to bind a port in specified range

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code: workaround i386 optimization bug with SHEC

http://tracker.ceph.com/issues/12936 Fixes: #12936

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5999 from dachary/wip-13000-ceph-disk

ceph-disk: fixes various race conditions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6008 from liewegas/wip-13190

osd/ReplicatedPG: tolerate promotion completion with stopped agent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6011 from liewegas/wip-systemd

systemd: increase nproc ulimit

Merge pull request #6019 from liewegas/wip-mon-dispatch-debug

mon/PaxosService: include msg and con ptr in dispatch debug

Merge pull request #5956 from xinxinsh/wip-13124

unify order limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: ceph-disk: workunit must fail when test fail

Signed-off-by: Joe Julian <me@joejulian.name>

tests: ceph-disk: only install multipath on CentOS

If installed on Ubuntu where multipath does not activate properly, it
interferes with the other tests.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: inline run_osd

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: wait for the OSD to be up

After preparing an OSD, wait for the corresponding OSD to be up
according to ceph osd dump before asserting the devices are in the
expected state. Otherwise the test races with ceph-disk activate which
is run asynchronously via udev / upstart / system.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: race condition is fixed

It turns out it was not CentOS 7 specific. There is no excuse to skip
the tests anymore.

http://tracker.ceph.com/issues/12787 Refs: #12787

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: multipath now auto activates

http://tracker.ceph.com/issues/12786 Refs: #12786

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: use sh() instead of helper() when possible

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: doc: ceph-disk workunit reminders

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: reduce debug verbosity

There is no need for debug traces from which() and it is very
noisy. Just get rid of it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: ensure udev add on the data partition is last

When calling partprobe, we make sure there is at least one udev add
called for each partition created when preparing a device. But there is
no guarantee that the udev add for data partition will be last and the
following scenario can happen:

 - udev add data partition fails because the journal partition is owned
   by root
 - udev add journal partition chown the journal partition
 - no other udev add event is sent and the OSD does not activate

An additional, possibly redundant, udev add event is fired after
partprobe is run and after udevadm settles, to guarantee there is at
least one udev add data partition after the last udev add journal
partition.

http://tracker.ceph.com/issues/12787 Fixes: #12787

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: move update_partition from main_prepare to prepare_dev

The update_partition call in main_prepare happens immediately after
prepare_dev but only if the data argument is a block device. There is no
reason for this separation: it is more sensible to call it from within
prepare_dev.

There is an additional test in prepare_dev that verifies partprobe won't
be called on a partition because it would not make sense.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: guard partprobe with udevadm settle

Call udevadm settle before and after partprobe.

A side effect of partprobe is to remove partitions and add them again.
The first udevadm settle waits for ongoing udev events to complete, just
in case one of them rely on an existing partition on dev.

The second udevadm settle guarantees to the caller that all udev events
related to the partition table change have been processed, i.e. the
95-ceph-osd.rules actions and mode changes, group changes etc. are
complete.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: do not hide ceph-disk prepare stack trace

raise without an argument so the original stack trace is shown when
ceph-disk prepare fails.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: support --verbose with --log-stdout

Set the LOG level as well as the channel level, otherwise the debug
messages are trimmed before they reach the channel. Also set the prefix
while we're at it.

http://tracker.ceph.com/issues/13180 Fixes: #13180

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: prefer sgdisk to blkid to retrieve partition UUID

blkid 2.23.2 which is the default for official CentOS 7 cloud images
fails on journal device. It would be better to use blkid because it does
not trigger udev events, but it is more important to get reliable
results.

http://tracker.ceph.com/issues/13153 Fixes: #13153

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: make ceph-disk list /dev/vdb equivalent to list vdb

The ceph-disk list argument must be the device name without the leading
/dev/. This is error prone and silently does nothing. Strip the /dev/
prefix of ceph-disk list arguments so that it behaves as expected.

http://tracker.ceph.com/issues/13154 Fixes: #13154

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: - is translated into / with systemd

When running ceph-disk trigger /dev/dm-1 with systemd, the path name is
translated into /dev/dm/1 because of systemd escape rules. Explicitly
translate - into \x2d for systemd to preserve the -.

It would be better to use systemd-escape
http://www.freedesktop.org/software/systemd/man/systemd-escape.html
but it does not appear to be generally available on CentOS 7 and
probably other distributions.

http://tracker.ceph.com/issues/13174 Fixes: #13174

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: a journal partition may survive a data partition

When a data partition is removed and the journal partition is not
removed, ceph-disk list will not find a journal_for information and
should just ignore it.

http://tracker.ceph.com/issues/13157 Fixes: #13157

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: upstart must not run ceph-disk activate concurrently

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: systemd must not kill a running ceph-disk

When activating a device, ceph-disk trigger restarts the ceph-disk
systemd service. Two consecutive udev add on the same device will
restart the ceph-disk systemd service and the second one may kill the
first one, leaving the device half activated.

The ceph-disk systemd service is instructed to not kill an existing
process when restarting. The second run waits (via flock) for the second
one to complete before running so that they do not overlap.

http://tracker.ceph.com/issues/13160 Fixes: #13160

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: fix typos in udev rules

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: ensure ceph owner on udev change

On udev change the owner of the device switch back to the default. If
that happens on a journal while an OSD is being activated, it will fail
with permission denied.

Make sure all ceph device types are chown to ceph on udev change.

http://tracker.ceph.com/issues/13000 Fixes: #13000

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5976 from yuyuyu101/wip-fix-metadata-conf

Librbd: Fix incorrect metadata filter behavior

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5750 from wonzhq/rbd-write-full

librbd: do write_full for whole object write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5416 from zhouyuan/ceph_fuse_12204

ceph-fuse: warn and shut down when there is no MDS present

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5826 from ukernel/wip-osx-port

fix compilation on OSX/FreeBSD

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-12971' of git://github.com/jcsp/ceph

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Conflicts:
src/osdc/Objecter.cc

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5644 from majianpeng/mds-support-iohint

Mds journal support iohint

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5966 from ukernel/wip-12506

Wip 12506

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5962 from majianpeng/client-fix

Client fix mini bugs.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon/PaxosService: include msg and con ptr in dispatch debug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6014 from athanatos/wip-13108

Revert "osd: purge the object from the cache when proxying and not pr…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6015 from athanatos/wip-13143

osd/: find_object_context: return obc for head for snapdir if non null

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6017 from athanatos/wip-13150

ReplicatedPG::get_snapset_context: set exists to false for new ssc wi…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6013 from athanatos/wip-12828

Wip 12828

Reviewed-by: Sage Weil <sage@redhat.com>

systemd: increase nproc ulimit

We were observed to be hitting the limit on centos7
(triggering pthread_create failures) on a ~2000 OSD cluster.

Increasing this resolves it!

Reported-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: tolerate promotion completion with stopped agent

We may start a promotion, then get a pool update that disables the
agent, and then complete it.  We should not segfault in this case.

Fixes: #13190
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5918 from oritwas/wip-5378

rgw: make radosgw-admin user rm idempotent

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5996 from rzarzynski/wip-12780

rgw: don't read actual data on user manifest HEAD

Merge pull request #5974 from Sandy4999/wip-rgw-errmsg

rgw: add a new error message for user conflict when using Admin Ops API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6003 from ukernel/wip-13166

Wip 13166

Reviewed-by: John Spray <john.spray@redhat.com>

mds: adjust MDSRank::incarnation according to mdsmap

When a standby-replay MDS replace failed MDS, we need update its
incarnation.

Fixes: #13166
Signed-off-by: Yan, Zheng <zyan@redhat.com>

osd: new pool settings: scrub intervals

 ceph osd pool set $POOL scrub_min_interval N
 ceph osd pool set $POOL scrub_max_interval N
 ceph osd pool set $POOL deep_scrub_interval N

If N > 0, this value is used for the pool instead of
the corresponding global parameter from the config
(osd_scrub_min_interval, osd_scrub_max_interval or
osd_deep_scrub_interval).

Fixes: #13077
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: new pool flags: noscrub, nodeep-scrub

This allows to disable periodic scrub on per pool basis.

Fixes: #13076
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: make 'ceph osd pool get' work for all settable pool flags

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: refactor setting write_fadvise_dontneed pool flag

Handle together with other flags.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6001 from dachary/wip-13182-crushtool

tests: update to match crushmap validation message

tests: update to match crushmap validation message

http://tracker.ceph.com/issues/13182 Fixes: #13182

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5979 from ceph/wip-crushtest

mon: fix crush testing for new pools

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5950 from tchaikov/wip-13112-infernalis

mon/PGMap: calc min_last_epoch_clean when decode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5998 from dachary/wip-13136-python3-coverage

install-deps: enable python3

install-deps: enable python3

The upstream regression has been fixed, we can re-enable python3
support.

http://tracker.ceph.com/issues/13136 Fixes: #13136

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5994 from SUSE/wip-13175

doc: do not promise backports to Dumpling

Reviewed-by: Sage Weil <sage@redhat.com>

doc: do not promise backports to Dumpling

Dumpling was retired in May 2015.

http://tracker.ceph.com/issues/13175 Fixes: #13175

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #5965 from jcsp/wip-get-or-create-output

mon: fix auth get-or-create output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5984 from ceph/wip-packages

Fixes locations from ceph.com to download.ceph.com

(For the record, this is to fix http://tracker.ceph.com/issues/13162)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: remove mention of ceph-extra as a requirement

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: remove ceph-extras

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: correct links to download.ceph.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: fix auth get-or-create output

Previously the caps were omitted from the
output: they should be present for this
to fulfil the 'get' part of get-or-create.

Signed-off-by: John Spray <john.spray@redhat.com>

ReplicatedPG::get_snapset_context: set exists to false for new ssc with can_create

Otherwise, if we do a proxy_write and don't promote it, the ssc will
stick around with exists = true and cause future reads on snaps for the
object to fail to fall through to the base tier.

Fixes: #13150
Signed-off-by: Samuel Just <sjust@redhat.com>

rgw: don't read actual data on user manifest HEAD

Fixes: #12780
We unconditionally read all the data, which is not needed
when doing HEAD operation on user manifest objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

ceph-osd-prestart.sh: no ceph-disk chown

chown -R ceph:ceph $1 is easy enough.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5980 from liewegas/wip-13059

ceph.spec: fix up /var/run/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: Fix up (/var)/run/ceph creation

Fixes: #13059
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5977 from tchaikov/wip-size_t-monstore-txn

mon: return size_t from MonitorDBStore::Transaction::size()

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: fix crush injection error message

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: only test crush ruleset for the newly created pool

Otherwise, we test *all* crush rules.. which might be a lot, and
which is a big waste of time and effort.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushTester: allow testing by ruleset

Signed-off-by: Sage Weil <sage@redhat.com>

Librbd: Fix incorrect metadata filter behavior

Signed-off-by: Haomai Wang <haomai@xsky.com>

mon: return size_t from MonitorDBStore::Transaction::size()

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add a new error message for user conflict when using Admin Ops API

Add a new error message to indicate user conflict, which was using the error code for bucket conflict.

Signed-off-by: Sangdi Xu <xu.sangdi@h3c.com>

Merge pull request #5948 from flyd1005/master

doc: delete wrong description of installing RPMs

osd/: find_object_context: return obc for head for snapdir if non null

The head obc may have an in progress operation holding a lock, we
need to block on the lock for ordering reasons.

Fixes: #13143
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #5926 from dillaman/wip-13082

include/inline_memory: out-of-bounds read on unaligned memory

Reviewed-by: Sage Weil <sage@redhat.com>

mds: Make sure wanted_state of first MDSBeacon is MDSMap::STATE_BOOT

Fixes: #12506
Signed-off-by: Yan, Zheng <zyan@redhat.com>

new release key

The previous release key,

pub   4096R/460F3994 2015-09-15
uid                  Ceph.com (release key) <security@ceph.com>

may have been compromised.

The new release key is

pub   4096R/460F3994 2015-09-15
uid                  Ceph.com (release key) <security@ceph.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5947 from weiqian/master

doc: replaces 'osd host' with 'host' in ceph.conf docs.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5964 from jcsp/wip-13138

libcephfs: fix calling init() then mount()

libcephfs: fix calling init() then mount()

Previously only ever called these separately, but
it should be allowed for callers to use one after
the other.

Fixes: #13138
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5963 from jcsp/wip-cmake-cephfs

CMake: fix libcephfs shared lib generation

Merge pull request #5857 from ceph/wip-da-SCA-20150812

SCA and compiler warning fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>