osdc/Objecter: send FULL_TRY and FULL_FORCE ops despite full flag

If a request has the FULL_TRY or FULL_FORCE flag, send it despite a full
pool or cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: document new flag

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: exempt MDS from the failsafe check, too

This mimics the prior behavior.  We probably want to change this at some
point!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_aio: test pool full gets EDQUOT when FULL_TRY flag is set

Signed-off-by: Sage Weil <sage@redhat.com>

librados: expose OPERATION_FULL_TRY flag

Allow librados users to opt to receive ENOSPC or EDQUOT when they submit
an operation against a full cluster.  This should only be used if the
librados app can handle those errors gracefully (librbd, for example,
cannot).

Also note that this allows savvy librados users to send delete operations;
they will get either a success or EDQUOT, depending on whether the
operation results in a net drop in space utilization.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: return -EDQUOT instead of -ENOSPC if it is a pool quota

The pool quota error should return EDQUOT intead of ENOSPC.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-add-async-log' of git://github.com/yuyuyu101/ceph into infernalis

Merge pull request #6117 from athanatos/wip-12990

Wip 12990

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6120 from athanatos/wip-13185

Wip 13185

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6122 from athanatos/wip-13234

OSDMap: fill in known encode_features where possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-12983' of git://github.com/dzafman/ceph into infernalis

Merge pull request #6119 from athanatos/wip-13192

Wip 13192

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6121 from athanatos/wip-13199

OSDService::agent_entry: don't use PG::operator<< without pg lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6113 from dachary/wip-13176-objectstore-tool

ceph-objectstore-tool: delete ObjectStore::Sequencer after umount

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6036 from liewegas/wip-13116

osd: fix replay requeue when pg is activating

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

ceph-objectstore-tool: delete ObjectStore::Sequencer after umount

An ObjectStore::Sequencer provided to an ObjectStore must not be
deallocated before umount. The ObjectStore::Sequencer may hold a pointer
to the instance with no reference counting. If a Context completes after
the ObjectStore::Sequencer is deleted, it could try to use it and fail.

http://tracker.ceph.com/issues/13176 Fixes: #13176

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6101 from liewegas/wip-fs

os/fs: fix do loop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6103 from SUSE/wip-re-redrop-fdupes

ceph.spec.in: re-re-drop fdupes

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: re-re-drop fdupes

The %fdupes call was dropped by 53072b9019caf72e0313b2804ea174237ed7da33
Later, it got accidentally reinstated by ceb93e8e69e125c9358f63b0099e7509b6624bcf

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6025 from ukernel/wip-13167

journaler: detect unexpected holes in journal objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5849 from ukernel/wip-32bits-ceph-fuse

Wip 32bits ceph fuse

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4791 from ceph/wip-11783

cephfs: detect completed cap flushes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6070 from jcsp/wip-reformat-subtreemaptest

Fix reformatting journals containing SUBTREEMAP_TEST events

Merge pull request #6082 from jcsp/wip-client-quota-roots

Fix quota enforcement on subdir mounts

os/fs: fix aio submit method

continue in a do while(false) will always eval false and break
out.  To repeat, we need while (true) and an explicit break.

Reported-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6078 from H3C/wip-13208

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6064 from H3C/wip-rbd-bugfix1

Objecter: misc fix for segmentation fault.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6084 from liewegas/wip-revert-scrub

Revert "osd: new pool settings: scrub intervals"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5832 from SUSE/wip-drop-dup-rcceph

ceph.spec.in: refrain from duplicating %{_sbindir}/rcceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ECBackend::handle_recovery_read_complete: do not expose the hash_info when getting the obc

Aside from expose an ECBackend internal detail, it causes ReplicatedPG
to cache the value causing trouble with WRITEFULL.

Partially fixes: #12983
Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend::handle_sub_read: restructure hash check and fix part of 12983

The previous code erroneously assumed that the reads on a single object
would always be in order, and not overlap.

Partially fixes: #12983
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6098 from liewegas/wip-crash-replay-interval

don't assume crash_replay_itnerval == 45

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45

e.g., it's 5 in teuthology's ceph.conf.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6076 from liewegas/wip-scrub-fix

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6086 from stiopaa1/librados_missingsemicolon

examples/librados/hello_world.cc:missing semicolon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

examples/librados/hello_world.cc:missing semicolon

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #6068 from dachary/wip-12936-shec

erasure-code: shec must compare for equality with epsilon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6052 from liewegas/wip-13098

osd: fix full request handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6075 from liewegas/wip-osd-getmap

osd: avoid duplicate MMonGetOSDMap requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6077 from liewegas/wip-qa

qa: http://ceph.com/qa -> http://download.ceph.com/qa
also some ceph.com/git -> git.ceph.com and a few line-endings - dmick

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "osd: new pool settings: scrub intervals"

This reverts commit 30810da4b573b3c7e679bc66eaa29b70246538cb.

After some discussion we have decided it is better to build a generic
dictionary in pg_pool_t to store infrequently used per-pool properties.

Signed-off-by: Sage Weil <sage@redhat.com>

OSDMap: fill in known encode_features where possible

Otherwise, if we get an incremental from hammer (struct_v = 6) we will
encode the full map as if it were before CEPH_FEATURE_PGID64, which
was actually pre-argonaut.  Similarly, if struct_v >= 7, we know it
was encoded with CEPH_FEATURE_OSDMAP_ENC.

Fixes: #13234
Backport: hammer
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6083 from jcsp/wip-mds-allow-star

ceph-create-keys: set mds "allow *"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-create-keys: set mds "allow *"

So that 'tell' works out of the box.

Signed-off-by: John Spray <john.spray@redhat.com>

erasure-code: shec must compare for equality with epsilon

When comparing two double for strict inequality, a test for equality
using epsilon must also be done to ensure the results do not vary cross
platform.

http://tracker.ceph.com/issues/12936 Fixes: #12936

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5959 from mslovy/wip-current-dev

mon: Fix unnecessary at/near max target warn in ceph -s

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5356 from majianpeng/code-cleanup

osd: cleanup code when met crc error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5972 from Jerry7X/master

unsigned type is short for journal max_size,use uint64_t instead

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #6048 from XinzeChi/wip-filestore-qlock

filestore: fix peek_queue for OpSequencer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6040 from liewegas/wip-mon-kv

mon: some fixes for kv backend failures

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6022 from jdurgin/wip-13032

msg: randomize client nonces

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6012 from liewegas/wip-mon

mon: misc scaling fixes

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: fix quote enforcement on subdir mounts

While we don't allow setting quotas on the global
filesystem root, the inode which is root from the
client mount's point of view might have a quota.

Signed-off-by: John Spray <john.spray@redhat.com>

client: refactor quota check functions

Generalise the path traversal into check_quota_condition
and then call that from each of file_exceeded, bytes_exceeded,
bytes_approaching with the appropriate lambda function.

Motivated by fixing the path traversal and wanting to
only fix it in one place.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6027 from wido/mirror-docs

docs: Update docs to make it clear that there are mirrors worldwide

Reviewed-by: Kefu Chai <kchai@redhat.com>

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

AsyncConnection: Add new debug log

Signed-off-by: Haomai Wang <haomai@xsky.com>

osd/ReplicatedPG: fix ENOSPC checking

1. We need to return ENOSPC *before* we apply our side-effects to the obc
cache in finish_ctx().

2. Consider object count changes too, not just bytes.

3. Consider cluster full flag, not just the pool flag.

4. Reply only if FULL_TRY; silently drop ops that were sent despite the
full flag.

Fixes: #13098
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove [pool_]last_epoch_marked_full from OSDSuperblock

This is no longer used or needed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove buggy [pool_]last_map_marked_full tracking

This is buggy (we reset the pool epoch each time it remains full instead
of only on the initial transition from non-full to full) and now unused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do full check in do_op

1. The current pool_last_map_marked_full tracking is buggy.
2. We need to recheck this each time we consider the op, not just when it
is received off the wire.  Otherwise, we might get a message, queue it
for some reason, get a map indicating the cluster or pool is full, and
then requeue and process the op instead of discarding it.
3. For now, silently drop ops when failsafe check fails. This will lead to
stalled client IO.  This needs a more robust fix.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: set last_epoch_marked_full on map advance

This will get persisted and shared.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_epoch_marked_full to pg_history_t

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: set FULL_FORCE flag when honor_full is false

This currenty only applies to the MDS.  Eventually we can remove the
OSD MDS checks once we are confident all MDS instances are new enough
to set this flag.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add FULL_TRY and FULL_FORCE rados op flags

FULL_TRY = try to do the op; ENOSPC if it results in a net increase is
space.  client must cope.

FULL_FORCE = do the op even if it consumes space.  The MDS will use this.
We should restrict this based on the OSD cap (* vs w, probably).

Signed-off-by: Sage Weil <sage@redhat.com>

qa: https://ceph.com/git -> https://git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6031 from s09816/master

librbd:reads larger than cache size hang.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: http://ceph.com/qa -> http://download.ceph.com/qa

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Hammer is sloppy about the hobject_t's it uses for the scrub bounds in that
the pool isn't set.  (Hammer FileStore doesn't care, but post-hammer is
much more careful about this sort of thing.)

Compensate by setting the pool on any scrub messages we receive.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: avoid duplicate MMonGetOSDMap requests

These are relatively expensive (we grab the full map from the mon) so we
should avoid duplicating our requests.

Track which requests are in flight.  Only send a new request when new
maps are asked for.  Resend requests when there is a new mon session.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6072 from yuyuyu101/update-org

Update org

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Update Xinze affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6066 from dachary/wip-ceph-detect-init

build/ops: ceph-detect-init must ignore .cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Update Chen Min affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update Tianshan affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update Haomai Wang affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

docs: Update docs to make it clear that there are mirrors worldwide

mds: fix error reformatting subtreemap_test events

Rare outside of vstart clusters, but if someone did
ever have one of these events in their journal and
try to update to latest ceph, they would end up
with bogus expire_pos on the reformatted events.

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: detect malformed ESubtreeMap::expire_pos

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6063 from dachary/wip-tar-v7

build/ops: make dist needs files with names > 99 characters

Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: ceph-detect-init must ignore .cache

In case the test fails, it may leave files behind:

src/ceph-detect-init/.cache/v/cache/lastfailed

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5694 from guce/rgw_improve_key_operate

rgw: improve convenience for key operate.

librbd:reads larger than cache size hang.

Fixes:#13164

Signed-off-by: Lu Shi <shi.lu@h3c.com>

build/ops: make dist needs files with names > 99 characters

When running make distdir=ceph-9.0.3-1870-gfd861bb dist, a few files
have names longer than 99 characters and discarded, which then causes
the resulting tarbal to be incomplete:

tar: ceph-9.0.3-1870-gfd861bb/src/rocksdb/utilities/write_batch_with_index/write_batch_with_index_internal.cc: file name is too long (max 99); not dumped
tar: ceph-9.0.3-1870-gfd861bb/src/rocksdb/utilities/write_batch_with_index/write_batch_with_index_internal.h: file name is too long (max 99); not dumped

Use the tar-ustar format instead of the legacy v7
format (http://www.gnu.org/software/automake/manual/automake.html#Options). It
is unlikely machines with a C++11 compiler also have an antique tar
binary that would only support v7.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

filestore: fix peek_queue for OpSequencer

Fixes: #13209
Signed-off-by: Xinze Chi <xinze@xsky.com>

osd/: eliminate unnecessary pg_hit_set_history_t::current_info

The only field actually relevant from this structure is .begin, which
duplicates the information in hit_set_start_stamp less well.  The problem
is that the starting point of the currently open hit set is ephemeral
state which shouldn't go into the pg_info_t structure.

This also caused 13185  since pg_info_t.hit_set.current_info gets default
constructed with use_gmt = true regardless of the pool setting.  This
becomes a problem in hit_set_persist since the oid is generated using
the pool setting, rather than the use_gmt value in current_info which
is placed into the history list.  That discrepancy then causes a crash
in hit set trim.  There would also be a related bug if the pool setting
is changed between when current_info is constructed and when it is
written out.

Since current_info isn't actually useful, I'm removing it so that we
don't later rely on invalid fields.

Fixes: 13185
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6053 from ceph/wip-cmake-dang

Wip cmake dang

Merge pull request #6056 from guangyy/wip-13198

osd: print min_last_epoch_clean along with pg dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-aarch64'

osd: print min_last_epoch_clean along with pg dump

Fixes: 13198
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

mon/Elector: do a trivial write on every election cycle

Currently we already do a small write when the *first* election in
a round happens (to update the election epoch).  If the backend
happens to fail while we are already in the midst of elections,
however, we may continue to call elections without verifying we
are still writeable.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: assert/crash if there is a write error

Do this globally intead of relying on teh zillion mon callers to
check the error code.  There are no cases where we want to
tolerate a commit failure.

Fixes: #13089
Signed-off-by: Sage Weil <sage@redhat.com>

CMake - fix check for UDEV

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for FUSE

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libedit

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for Snappy

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for XFS

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>