librbd: invalidate object map on error even w/o holding lock

If there is a deep bug that prevents the object map from owning
the lock before objects are updated, flag the object map as invalid
regardless.

Fixes: #13372
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/release-notes: initial v9.1.0 notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6110 from ceph/wip-13239-infernalis

rgw: set default value for env->get() call

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6142 from ceph/wip-13061

ceph.spec.in: Do not always restart the daemons on upgrades/removal

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6105 from liewegas/wip-13262

mon: ignore messages from disconnected clients; clean up session checks

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6176 from liewegas/wip-13283

osdc/Objecter: send FULL_TRY and FULL_FORCE ops despite full flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdc/Objecter: send FULL_TRY and FULL_FORCE ops despite full flag

If a request has the FULL_TRY or FULL_FORCE flag, send it despite a full
pool or cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: allow ping through despite synch/quorum status, with session

Attach sessions to ping messages, too.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: drop ops on closed sessions early

We will throw it out in dispatch_op() anyway.  Remove the ancient (~2009)
reuse_caps logic too, which doesn't make any sense nowadays anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: clean up _ms_dispatch

Lots of cruft here.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: all MonOpRequests have an attached Session

...except in dispatch, where we initially set it up.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: drop any ops from closed sessions in dispatch_op

If we've closed the session we shouldn't waste our time on this op.

Fixes: #13262 (we should not add a subscription to a disconnected session)
Signed-off-by: Sage Weil <sage@redhat.com>

mon: always set up session; move waitlist logic

Unconditionally attach a Session, so that when we push an op back through
dispatch_op it isn't in a weird broken state.

Move the waitinglist check to the bottom.  Use s->global_id == 0 to tell
whether a client has authenticated, and zap sessions based on that.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix msg leak in resend_routed_requests

MForward takes its own ref so we can drop ours.

Signed-off-by: Sage Weil <sage@redhat.com>

Mon: Fix decoded message leak when this monitor is leader

Fix #13109
Signed-off-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6139 from liewegas/wip-13283

librados: add FULL_TRY flag; EDQUOT != ENOSPC

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librados: document new flag

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6169 from rzarzynski/wip-13326

rgw: improve handling of already removed buckets in object expirer.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6168 from SUSE/wip-13333-infernalis

ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 7b1882f87509092380d11ec68b504a7bc1f77830)

osd/ReplicatedPG: exempt MDS from the failsafe check, too

This mimics the prior behavior.  We probably want to change this at some
point!

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: improve handling of already removed buckets in object expirer.

Fixes: #13326
Backport: infernalis
Signed-off-by: Radoslaw Zarzynski <rzarzynski@github.com>

Merge pull request #6096 from liewegas/wip-13260

osd/ReplicatedPG: preserve (some) flags when proxying reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6136 from liewegas/wip-13114

osd/librados: fix notify completion race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph.spec.in: Do not always restart the daemons on removal

There is no need to restart the daemons when removing ceph-selinux
package if SELinux is not enabled.

Fixes: #13061
Signed-off-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: Do not always restart the daemons on upgrades

This patch minimizes the amount of daemon stop/start procedures when
upgrading ceph-selinux package. With this patch, the daemons get
restarted only if SELinux is enabled and the SELinux policy version
changed in the meantime.

Fixes: #13061
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #6137 from SUSE/wip-13324-infernalis

install-deps.sh: use %bcond_with selinux on SLE/openSUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'jdurgin/wip-13281' into infernalis

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

install-deps.sh: use %bcond_with selinux on SLE/openSUSE

http://tracker.ceph.com/issues/13324 Fixes: #13324

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph_test_rados_api_aio: test pool full gets EDQUOT when FULL_TRY flag is set

Signed-off-by: Sage Weil <sage@redhat.com>

librados: expose OPERATION_FULL_TRY flag

Allow librados users to opt to receive ENOSPC or EDQUOT when they submit
an operation against a full cluster.  This should only be used if the
librados app can handle those errors gracefully (librbd, for example,
cannot).

Also note that this allows savvy librados users to send delete operations;
they will get either a success or EDQUOT, depending on whether the
operation results in a net drop in space utilization.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: return -EDQUOT instead of -ENOSPC if it is a pool quota

The pool quota error should return EDQUOT intead of ENOSPC.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: distinguish between multiple notify completions

We may send a notify to the cluster multiple times due to OSDMap
changes.  In some cases, earlier notify attempts may complete with
an error, while later attempts succeed.  We need to only pay
attention to the most-recently send notify's completion.

Do this by making note of the notify_id in the initial ACK (only
present when talking to newer OSDs).  When we get a notify
completion, match it against our expected notify_id (if we have
one) or else discard it.

This is important because in some cases an early notify completion
may be an error while a later one succeeds.

Note that if we are talking to an old cluster we will simply not record a
notify_id and our behavior will be the same as before (we will trust any
notify completion we get).

Fixes: #13114
Signed-off-by: Sage Weil <sage@redhat.com>

osd: reply to notify request with our unique notify_id

The OSD assigns a unique ID to each notify it queues for
processing.  Include this in the reply to the notifier so that
they can match it up with the eventual completions they receive.

This is necessary to distinguish between multiple completions
they may receive if there is PG peering and the notify is resent.
In particular, an earlier notify may return an error when a later
attempt succeeds.

This is forwards and backwards compatible: new clients will make use of
this reply payload but older clients ignore it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6134 from SUSE/wip-13318-infernalis

install-deps.sh: openSUSE-release/sles-release/sled-release are always present

install-deps.sh: openSUSE-release/sles-release/sled-release are always present

http://tracker.ceph.com/issues/13318 Fixes: #13318

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge branch 'wip-add-async-log' of git://github.com/yuyuyu101/ceph into infernalis

Merge pull request #6117 from athanatos/wip-12990

Wip 12990

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6120 from athanatos/wip-13185

Wip 13185

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6122 from athanatos/wip-13234

OSDMap: fill in known encode_features where possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-12983' of git://github.com/dzafman/ceph into infernalis

Merge pull request #6119 from athanatos/wip-13192

Wip 13192

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6121 from athanatos/wip-13199

OSDService::agent_entry: don't use PG::operator<< without pg lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6113 from dachary/wip-13176-objectstore-tool

ceph-objectstore-tool: delete ObjectStore::Sequencer after umount

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6036 from liewegas/wip-13116

osd: fix replay requeue when pg is activating

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

ReplicatedPG: clearing a whiteout should create the object

This was uncovered by 75321943729f1d5dfacb68645e3c5483740d66f8. Since
rbd_create() does a stat, the obc is cached as a whiteout, and the
subsequent create(EXCL) would fall through to return false from
maybe_create_new_object(). This would then skip adding a touch() to
the transaction.

Fixes: #13281
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph-objectstore-tool: delete ObjectStore::Sequencer after umount

An ObjectStore::Sequencer provided to an ObjectStore must not be
deallocated before umount. The ObjectStore::Sequencer may hold a pointer
to the instance with no reference counting. If a Context completes after
the ObjectStore::Sequencer is deleted, it could try to use it and fail.

http://tracker.ceph.com/issues/13176 Fixes: #13176

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6101 from liewegas/wip-fs

os/fs: fix do loop

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: set default value for env->get() call

Fixes: #13239
This fixes a regression introduced at commit abe4ec293d08b0314bf5c081ace2456073f3a22c.
The host var is a string, env->get() returns a char pointer, shouldn't
pass in NULL.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon: combine _ms_dispatch and dispatch

_ms_dispatch() is the only caller, and it is confusing to have both
dispatch() and dispatch_op() that take the same argument type.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6103 from SUSE/wip-re-redrop-fdupes

ceph.spec.in: re-re-drop fdupes

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: re-re-drop fdupes

The %fdupes call was dropped by 53072b9019caf72e0313b2804ea174237ed7da33
Later, it got accidentally reinstated by ceb93e8e69e125c9358f63b0099e7509b6624bcf

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6025 from ukernel/wip-13167

journaler: detect unexpected holes in journal objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5849 from ukernel/wip-32bits-ceph-fuse

Wip 32bits ceph fuse

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4791 from ceph/wip-11783

cephfs: detect completed cap flushes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6070 from jcsp/wip-reformat-subtreemaptest

Fix reformatting journals containing SUBTREEMAP_TEST events

Merge pull request #6082 from jcsp/wip-client-quota-roots

Fix quota enforcement on subdir mounts

os/fs: fix aio submit method

continue in a do while(false) will always eval false and break
out.  To repeat, we need while (true) and an explicit break.

Reported-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6078 from H3C/wip-13208

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6064 from H3C/wip-rbd-bugfix1

Objecter: misc fix for segmentation fault.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6084 from liewegas/wip-revert-scrub

Revert "osd: new pool settings: scrub intervals"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5832 from SUSE/wip-drop-dup-rcceph

ceph.spec.in: refrain from duplicating %{_sbindir}/rcceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ECBackend::handle_recovery_read_complete: do not expose the hash_info when getting the obc

Aside from expose an ECBackend internal detail, it causes ReplicatedPG
to cache the value causing trouble with WRITEFULL.

Partially fixes: #12983
Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend::handle_sub_read: restructure hash check and fix part of 12983

The previous code erroneously assumed that the reads on a single object
would always be in order, and not overlap.

Partially fixes: #12983
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6098 from liewegas/wip-crash-replay-interval

don't assume crash_replay_itnerval == 45

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45

e.g., it's 5 in teuthology's ceph.conf.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6076 from liewegas/wip-scrub-fix

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: preserve (some) flags when proxying reads

In particular, proxy RWORDERED as clients in the real world set
that.

Fixes: #13260
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6086 from stiopaa1/librados_missingsemicolon

examples/librados/hello_world.cc:missing semicolon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

examples/librados/hello_world.cc:missing semicolon

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #6068 from dachary/wip-12936-shec

erasure-code: shec must compare for equality with epsilon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6052 from liewegas/wip-13098

osd: fix full request handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6075 from liewegas/wip-osd-getmap

osd: avoid duplicate MMonGetOSDMap requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6077 from liewegas/wip-qa

qa: http://ceph.com/qa -> http://download.ceph.com/qa
also some ceph.com/git -> git.ceph.com and a few line-endings - dmick

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "osd: new pool settings: scrub intervals"

This reverts commit 30810da4b573b3c7e679bc66eaa29b70246538cb.

After some discussion we have decided it is better to build a generic
dictionary in pg_pool_t to store infrequently used per-pool properties.

Signed-off-by: Sage Weil <sage@redhat.com>

OSDMap: fill in known encode_features where possible

Otherwise, if we get an incremental from hammer (struct_v = 6) we will
encode the full map as if it were before CEPH_FEATURE_PGID64, which
was actually pre-argonaut.  Similarly, if struct_v >= 7, we know it
was encoded with CEPH_FEATURE_OSDMAP_ENC.

Fixes: #13234
Backport: hammer
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6083 from jcsp/wip-mds-allow-star

ceph-create-keys: set mds "allow *"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-create-keys: set mds "allow *"

So that 'tell' works out of the box.

Signed-off-by: John Spray <john.spray@redhat.com>

erasure-code: shec must compare for equality with epsilon

When comparing two double for strict inequality, a test for equality
using epsilon must also be done to ensure the results do not vary cross
platform.

http://tracker.ceph.com/issues/12936 Fixes: #12936

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5959 from mslovy/wip-current-dev

mon: Fix unnecessary at/near max target warn in ceph -s

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5356 from majianpeng/code-cleanup

osd: cleanup code when met crc error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5972 from Jerry7X/master

unsigned type is short for journal max_size,use uint64_t instead

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #6048 from XinzeChi/wip-filestore-qlock

filestore: fix peek_queue for OpSequencer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6040 from liewegas/wip-mon-kv

mon: some fixes for kv backend failures

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6022 from jdurgin/wip-13032

msg: randomize client nonces

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6012 from liewegas/wip-mon

mon: misc scaling fixes

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: fix quote enforcement on subdir mounts

While we don't allow setting quotas on the global
filesystem root, the inode which is root from the
client mount's point of view might have a quota.

Signed-off-by: John Spray <john.spray@redhat.com>

client: refactor quota check functions

Generalise the path traversal into check_quota_condition
and then call that from each of file_exceeded, bytes_exceeded,
bytes_approaching with the appropriate lambda function.

Motivated by fixing the path traversal and wanting to
only fix it in one place.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6027 from wido/mirror-docs

docs: Update docs to make it clear that there are mirrors worldwide

Reviewed-by: Kefu Chai <kchai@redhat.com>

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

AsyncConnection: Add new debug log

Signed-off-by: Haomai Wang <haomai@xsky.com>

osd/ReplicatedPG: fix ENOSPC checking

1. We need to return ENOSPC *before* we apply our side-effects to the obc
cache in finish_ctx().

2. Consider object count changes too, not just bytes.

3. Consider cluster full flag, not just the pool flag.

4. Reply only if FULL_TRY; silently drop ops that were sent despite the
full flag.

Fixes: #13098
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove [pool_]last_epoch_marked_full from OSDSuperblock

This is no longer used or needed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove buggy [pool_]last_map_marked_full tracking

This is buggy (we reset the pool epoch each time it remains full instead
of only on the initial transition from non-full to full) and now unused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do full check in do_op

1. The current pool_last_map_marked_full tracking is buggy.
2. We need to recheck this each time we consider the op, not just when it
is received off the wire.  Otherwise, we might get a message, queue it
for some reason, get a map indicating the cluster or pool is full, and
then requeue and process the op instead of discarding it.
3. For now, silently drop ops when failsafe check fails. This will lead to
stalled client IO.  This needs a more robust fix.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: set last_epoch_marked_full on map advance

This will get persisted and shared.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_epoch_marked_full to pg_history_t

Signed-off-by: Sage Weil <sage@redhat.com>