os/fs: fix aio submit method

continue in a do while(false) will always eval false and break
out.  To repeat, we need while (true) and an explicit break.

Reported-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6078 from H3C/wip-13208

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6064 from H3C/wip-rbd-bugfix1

Objecter: misc fix for segmentation fault.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6084 from liewegas/wip-revert-scrub

Revert "osd: new pool settings: scrub intervals"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5832 from SUSE/wip-drop-dup-rcceph

ceph.spec.in: refrain from duplicating %{_sbindir}/rcceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6098 from liewegas/wip-crash-replay-interval

don't assume crash_replay_itnerval == 45

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45

e.g., it's 5 in teuthology's ceph.conf.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6076 from liewegas/wip-scrub-fix

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6086 from stiopaa1/librados_missingsemicolon

examples/librados/hello_world.cc:missing semicolon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

examples/librados/hello_world.cc:missing semicolon

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #6052 from liewegas/wip-13098

osd: fix full request handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6075 from liewegas/wip-osd-getmap

osd: avoid duplicate MMonGetOSDMap requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6077 from liewegas/wip-qa

qa: http://ceph.com/qa -> http://download.ceph.com/qa
also some ceph.com/git -> git.ceph.com and a few line-endings - dmick

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "osd: new pool settings: scrub intervals"

This reverts commit 30810da4b573b3c7e679bc66eaa29b70246538cb.

After some discussion we have decided it is better to build a generic
dictionary in pg_pool_t to store infrequently used per-pool properties.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6083 from jcsp/wip-mds-allow-star

ceph-create-keys: set mds "allow *"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-create-keys: set mds "allow *"

So that 'tell' works out of the box.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5959 from mslovy/wip-current-dev

mon: Fix unnecessary at/near max target warn in ceph -s

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5356 from majianpeng/code-cleanup

osd: cleanup code when met crc error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5972 from Jerry7X/master

unsigned type is short for journal max_size,use uint64_t instead

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #6048 from XinzeChi/wip-filestore-qlock

filestore: fix peek_queue for OpSequencer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6040 from liewegas/wip-mon-kv

mon: some fixes for kv backend failures

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6022 from jdurgin/wip-13032

msg: randomize client nonces

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6012 from liewegas/wip-mon

mon: misc scaling fixes

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6027 from wido/mirror-docs

docs: Update docs to make it clear that there are mirrors worldwide

Reviewed-by: Kefu Chai <kchai@redhat.com>

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

osd/ReplicatedPG: fix ENOSPC checking

1. We need to return ENOSPC *before* we apply our side-effects to the obc
cache in finish_ctx().

2. Consider object count changes too, not just bytes.

3. Consider cluster full flag, not just the pool flag.

4. Reply only if FULL_TRY; silently drop ops that were sent despite the
full flag.

Fixes: #13098
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove [pool_]last_epoch_marked_full from OSDSuperblock

This is no longer used or needed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove buggy [pool_]last_map_marked_full tracking

This is buggy (we reset the pool epoch each time it remains full instead
of only on the initial transition from non-full to full) and now unused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do full check in do_op

1. The current pool_last_map_marked_full tracking is buggy.
2. We need to recheck this each time we consider the op, not just when it
is received off the wire.  Otherwise, we might get a message, queue it
for some reason, get a map indicating the cluster or pool is full, and
then requeue and process the op instead of discarding it.
3. For now, silently drop ops when failsafe check fails. This will lead to
stalled client IO.  This needs a more robust fix.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: set last_epoch_marked_full on map advance

This will get persisted and shared.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_epoch_marked_full to pg_history_t

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: set FULL_FORCE flag when honor_full is false

This currenty only applies to the MDS.  Eventually we can remove the
OSD MDS checks once we are confident all MDS instances are new enough
to set this flag.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add FULL_TRY and FULL_FORCE rados op flags

FULL_TRY = try to do the op; ENOSPC if it results in a net increase is
space.  client must cope.

FULL_FORCE = do the op even if it consumes space.  The MDS will use this.
We should restrict this based on the OSD cap (* vs w, probably).

Signed-off-by: Sage Weil <sage@redhat.com>

qa: https://ceph.com/git -> https://git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6031 from s09816/master

librbd:reads larger than cache size hang.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: http://ceph.com/qa -> http://download.ceph.com/qa

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Hammer is sloppy about the hobject_t's it uses for the scrub bounds in that
the pool isn't set.  (Hammer FileStore doesn't care, but post-hammer is
much more careful about this sort of thing.)

Compensate by setting the pool on any scrub messages we receive.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: avoid duplicate MMonGetOSDMap requests

These are relatively expensive (we grab the full map from the mon) so we
should avoid duplicating our requests.

Track which requests are in flight.  Only send a new request when new
maps are asked for.  Resend requests when there is a new mon session.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6072 from yuyuyu101/update-org

Update org

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Update Xinze affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6066 from dachary/wip-ceph-detect-init

build/ops: ceph-detect-init must ignore .cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Update Chen Min affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update Tianshan affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update Haomai Wang affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

docs: Update docs to make it clear that there are mirrors worldwide

Merge pull request #6063 from dachary/wip-tar-v7

build/ops: make dist needs files with names > 99 characters

Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: ceph-detect-init must ignore .cache

In case the test fails, it may leave files behind:

src/ceph-detect-init/.cache/v/cache/lastfailed

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5694 from guce/rgw_improve_key_operate

rgw: improve convenience for key operate.

librbd:reads larger than cache size hang.

Fixes:#13164

Signed-off-by: Lu Shi <shi.lu@h3c.com>

build/ops: make dist needs files with names > 99 characters

When running make distdir=ceph-9.0.3-1870-gfd861bb dist, a few files
have names longer than 99 characters and discarded, which then causes
the resulting tarbal to be incomplete:

tar: ceph-9.0.3-1870-gfd861bb/src/rocksdb/utilities/write_batch_with_index/write_batch_with_index_internal.cc: file name is too long (max 99); not dumped
tar: ceph-9.0.3-1870-gfd861bb/src/rocksdb/utilities/write_batch_with_index/write_batch_with_index_internal.h: file name is too long (max 99); not dumped

Use the tar-ustar format instead of the legacy v7
format (http://www.gnu.org/software/automake/manual/automake.html#Options). It
is unlikely machines with a C++11 compiler also have an antique tar
binary that would only support v7.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

filestore: fix peek_queue for OpSequencer

Fixes: #13209
Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6053 from ceph/wip-cmake-dang

Wip cmake dang

Merge pull request #6056 from guangyy/wip-13198

osd: print min_last_epoch_clean along with pg dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-aarch64'

osd: print min_last_epoch_clean along with pg dump

Fixes: 13198
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

mon/Elector: do a trivial write on every election cycle

Currently we already do a small write when the *first* election in
a round happens (to update the election epoch).  If the backend
happens to fail while we are already in the midst of elections,
however, we may continue to call elections without verifying we
are still writeable.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: assert/crash if there is a write error

Do this globally intead of relying on teh zillion mon callers to
check the error code.  There are no cases where we want to
tolerate a commit failure.

Fixes: #13089
Signed-off-by: Sage Weil <sage@redhat.com>

CMake - fix check for UDEV

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for FUSE

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libedit

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for Snappy

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for XFS

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for AIO

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for blkid

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libcurl

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for fcgi

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libuuid

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for leveldb

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for expat

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - Add check for keyutils

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for NSS

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix libatomic_ops and gperftools checks

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge remote-tracking branch 'gh/infernalis'

arch/arm: s/false/0/

arch/arm.c: In function 'ceph_arch_arm_probe':
arch/arm.c:54:28: error: 'false' undeclared (first use in this function)
  ceph_arch_aarch64_crc32 = false;  // sorry!
                            ^
arch/arm.c:54:28: note: each undeclared identifier is reported only once for each function it appears in

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: improve convenience for key operate.

1.key-type assignments based on context if it wasn't specified
In user    operate context, key-type assignment to KEY_TYPE_S3
In subuser operate context, key-type assignment to KEY_TYPE_SWIFT
In key     operate context, key-type assignment based on user type

2.fix RGWSubUserPool::add()
When create subuser generate secret by default

3.fix RGWAccessKeyPool::generate_key()
Avoid wrong key's username when create user and subuser at the same time
Check empty secret

Signed-off-by: Ce Gu <guce@h3c.com>

Merge pull request #6037 from liewegas/wip-12407

init-radosgw: fix restart w/ multiple instances

Reviewed-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

Merge pull request #5928 from ceph/wip-12819-infernalis

Wip 12819 infernalis

Merge pull request #6006 from dachary/wip-12936-shec

erasure-code: workaround i386 optimization bug with SHEC

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw: specify pid file to start-stop-daemon

This fixes restart when multiple instances are running.

Fixes: #12407
Tested-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5922 from trociny/wip-pool-set-noscrub

osd: per pool control for scrub and deep-scrub

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5846 from xiexingguo/xxg-wip-13002

msg/simple: start over after fails to bind a port in specified range

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code: workaround i386 optimization bug with SHEC

http://tracker.ceph.com/issues/12936 Fixes: #12936

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5999 from dachary/wip-13000-ceph-disk

ceph-disk: fixes various race conditions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6008 from liewegas/wip-13190

osd/ReplicatedPG: tolerate promotion completion with stopped agent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6011 from liewegas/wip-systemd

systemd: increase nproc ulimit

Merge pull request #6019 from liewegas/wip-mon-dispatch-debug

mon/PaxosService: include msg and con ptr in dispatch debug

Merge pull request #5956 from xinxinsh/wip-13124

unify order limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: ceph-disk: workunit must fail when test fail

Signed-off-by: Joe Julian <me@joejulian.name>

tests: ceph-disk: only install multipath on CentOS

If installed on Ubuntu where multipath does not activate properly, it
interferes with the other tests.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: inline run_osd

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: wait for the OSD to be up

After preparing an OSD, wait for the corresponding OSD to be up
according to ceph osd dump before asserting the devices are in the
expected state. Otherwise the test races with ceph-disk activate which
is run asynchronously via udev / upstart / system.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: race condition is fixed

It turns out it was not CentOS 7 specific. There is no excuse to skip
the tests anymore.

http://tracker.ceph.com/issues/12787 Refs: #12787

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: multipath now auto activates

http://tracker.ceph.com/issues/12786 Refs: #12786

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: use sh() instead of helper() when possible

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: doc: ceph-disk workunit reminders

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: reduce debug verbosity

There is no need for debug traces from which() and it is very
noisy. Just get rid of it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: ensure udev add on the data partition is last

When calling partprobe, we make sure there is at least one udev add
called for each partition created when preparing a device. But there is
no guarantee that the udev add for data partition will be last and the
following scenario can happen:

 - udev add data partition fails because the journal partition is owned
   by root
 - udev add journal partition chown the journal partition
 - no other udev add event is sent and the OSD does not activate

An additional, possibly redundant, udev add event is fired after
partprobe is run and after udevadm settles, to guarantee there is at
least one udev add data partition after the last udev add journal
partition.

http://tracker.ceph.com/issues/12787 Fixes: #12787

Signed-off-by: Loic Dachary <ldachary@redhat.com>