ceph.spec.in: remove comments regarding ceph UID/GID in SUSE

It is possible that the ceph user/group will not have fixed UID/GID in SUSE.
Instead, it is possible that the ceph package will depend on a separate package
whose sole purpose will be to create the ceph user/group if they do not exist.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: enable OBS post-build-checks to find systemd-tmpfiles

The openSUSE Build Service runs a number of "post-build checks" after the RPMs
have been generated. One of these tests the RPM scriptlets for idempotence.
Without this line in the specfile, the check fails on SLE_12 because it cannot
find the systemd-tmpfiles binary.

http://tracker.ceph.com/issues/13375 Fixes: #13375

Signed-off-by: Nathan Cutler <ncutler@suse.com>

etc/sysconfig/ceph: add CEPH_AUTO_RESTART_ON_UPGRADE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: Standardize systemd preun and postun scripts

Currently, the main ceph package and the ceph-radosgw behave
differently on upgrade. This commit unifies their behavior
to the following:

On package removal, disable and stop all related systemd units.

On package upgrade, do nothing unless there is a file /etc/sysconfig/ceph
containing a parameter CEPH_AUTO_RESTART_ON_UPGRADE. If parameter is set
to "yes", restart the systemd units iff they are running.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: fix for out-of-memory errors in OBS

Add "--param ggc-min-expand=20 --param ggc-min-heapsize=32768"
to RPM_OPT_FLAGS, ensuring gcc does not add debug symbols and is
more aggressive about garbage collection.

Thanks to Berthold Gunreben for debugging this issue.

http://tracker.ceph.com/issues/13352 Fixes: #13352

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: Use _smp_mflags instead of _NPROCESSORS_ONLN

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "install-deps.sh: use %bcond_with selinux on SLE/openSUSE"

This reverts commit 1df2cc2bf625a1f65ef4968d9cf313cc8ee9acc5.

ceph.spec.in: selinux by default on RHEL/Fedora, non-default on SUSE

Surround it with an appropriate conditional.

http://tracker.ceph.com/issues/13324 Fixes: #13324

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: use %fillup_only macro in SLE/openSUSE

SUSE does not allow /etc/sysconfig/ceph to be owned by the ceph
package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: distro-conditional groupadd and useradd

The -o option to groupadd/useradd is not recognized in SLE12. For
SLE/openSUSE, follow the openSUSE packaging guidelines.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: move systemd-tmpfiles stuff to ceph-common

Since the ceph user and group are created in the ceph-common
%pre scriptlet, the systemd-tmpfiles --create command needs
to run in that package.

tracker.ceph.com/issues/13346 Fixes: #13346

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: remove SUSE-specific curses hack

Unclear what this was supposed to fix.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: lttng in SLES12 only

Over in the SUSE sector, we are trying to enable the SLE_12 and openSUSE_13.2
build targets. The lttng/babeltrace stuff is currently available only in
SLE_12.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: drop %insserv_prereq (obsoleted by systemd)

The %post, etc. scriptlets are no longer calling %insserv, et al.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: fix boost-random build dependency for SLE/openSUSE

In Fedora, RHEL and CentOS there is a package called boost-random.
In SLE/openSUSE, there is no such package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6139 from liewegas/wip-13283

librados: add FULL_TRY flag; EDQUOT != ENOSPC

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librados: document new flag

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6169 from rzarzynski/wip-13326

rgw: improve handling of already removed buckets in object expirer.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6168 from SUSE/wip-13333-infernalis

ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: correctly declare systemd dependency for SLE/openSUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 7b1882f87509092380d11ec68b504a7bc1f77830)

osd/ReplicatedPG: exempt MDS from the failsafe check, too

This mimics the prior behavior.  We probably want to change this at some
point!

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: improve handling of already removed buckets in object expirer.

Fixes: #13326
Backport: infernalis
Signed-off-by: Radoslaw Zarzynski <rzarzynski@github.com>

Merge pull request #6096 from liewegas/wip-13260

osd/ReplicatedPG: preserve (some) flags when proxying reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6136 from liewegas/wip-13114

osd/librados: fix notify completion race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6137 from SUSE/wip-13324-infernalis

install-deps.sh: use %bcond_with selinux on SLE/openSUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'jdurgin/wip-13281' into infernalis

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

install-deps.sh: use %bcond_with selinux on SLE/openSUSE

http://tracker.ceph.com/issues/13324 Fixes: #13324

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph_test_rados_api_aio: test pool full gets EDQUOT when FULL_TRY flag is set

Signed-off-by: Sage Weil <sage@redhat.com>

librados: expose OPERATION_FULL_TRY flag

Allow librados users to opt to receive ENOSPC or EDQUOT when they submit
an operation against a full cluster.  This should only be used if the
librados app can handle those errors gracefully (librbd, for example,
cannot).

Also note that this allows savvy librados users to send delete operations;
they will get either a success or EDQUOT, depending on whether the
operation results in a net drop in space utilization.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: return -EDQUOT instead of -ENOSPC if it is a pool quota

The pool quota error should return EDQUOT intead of ENOSPC.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: distinguish between multiple notify completions

We may send a notify to the cluster multiple times due to OSDMap
changes.  In some cases, earlier notify attempts may complete with
an error, while later attempts succeed.  We need to only pay
attention to the most-recently send notify's completion.

Do this by making note of the notify_id in the initial ACK (only
present when talking to newer OSDs).  When we get a notify
completion, match it against our expected notify_id (if we have
one) or else discard it.

This is important because in some cases an early notify completion
may be an error while a later one succeeds.

Note that if we are talking to an old cluster we will simply not record a
notify_id and our behavior will be the same as before (we will trust any
notify completion we get).

Fixes: #13114
Signed-off-by: Sage Weil <sage@redhat.com>

osd: reply to notify request with our unique notify_id

The OSD assigns a unique ID to each notify it queues for
processing.  Include this in the reply to the notifier so that
they can match it up with the eventual completions they receive.

This is necessary to distinguish between multiple completions
they may receive if there is PG peering and the notify is resent.
In particular, an earlier notify may return an error when a later
attempt succeeds.

This is forwards and backwards compatible: new clients will make use of
this reply payload but older clients ignore it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6134 from SUSE/wip-13318-infernalis

install-deps.sh: openSUSE-release/sles-release/sled-release are always present

install-deps.sh: openSUSE-release/sles-release/sled-release are always present

http://tracker.ceph.com/issues/13318 Fixes: #13318

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge branch 'wip-add-async-log' of git://github.com/yuyuyu101/ceph into infernalis

Merge pull request #6117 from athanatos/wip-12990

Wip 12990

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6120 from athanatos/wip-13185

Wip 13185

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6122 from athanatos/wip-13234

OSDMap: fill in known encode_features where possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-12983' of git://github.com/dzafman/ceph into infernalis

Merge pull request #6119 from athanatos/wip-13192

Wip 13192

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6121 from athanatos/wip-13199

OSDService::agent_entry: don't use PG::operator<< without pg lock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6113 from dachary/wip-13176-objectstore-tool

ceph-objectstore-tool: delete ObjectStore::Sequencer after umount

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #6036 from liewegas/wip-13116

osd: fix replay requeue when pg is activating

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

ReplicatedPG: clearing a whiteout should create the object

This was uncovered by 75321943729f1d5dfacb68645e3c5483740d66f8. Since
rbd_create() does a stat, the obc is cached as a whiteout, and the
subsequent create(EXCL) would fall through to return false from
maybe_create_new_object(). This would then skip adding a touch() to
the transaction.

Fixes: #13281
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph-objectstore-tool: delete ObjectStore::Sequencer after umount

An ObjectStore::Sequencer provided to an ObjectStore must not be
deallocated before umount. The ObjectStore::Sequencer may hold a pointer
to the instance with no reference counting. If a Context completes after
the ObjectStore::Sequencer is deleted, it could try to use it and fail.

http://tracker.ceph.com/issues/13176 Fixes: #13176

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6101 from liewegas/wip-fs

os/fs: fix do loop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6103 from SUSE/wip-re-redrop-fdupes

ceph.spec.in: re-re-drop fdupes

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: re-re-drop fdupes

The %fdupes call was dropped by 53072b9019caf72e0313b2804ea174237ed7da33
Later, it got accidentally reinstated by ceb93e8e69e125c9358f63b0099e7509b6624bcf

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6025 from ukernel/wip-13167

journaler: detect unexpected holes in journal objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5849 from ukernel/wip-32bits-ceph-fuse

Wip 32bits ceph fuse

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4791 from ceph/wip-11783

cephfs: detect completed cap flushes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6070 from jcsp/wip-reformat-subtreemaptest

Fix reformatting journals containing SUBTREEMAP_TEST events

Merge pull request #6082 from jcsp/wip-client-quota-roots

Fix quota enforcement on subdir mounts

os/fs: fix aio submit method

continue in a do while(false) will always eval false and break
out.  To repeat, we need while (true) and an explicit break.

Reported-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6078 from H3C/wip-13208

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6064 from H3C/wip-rbd-bugfix1

Objecter: misc fix for segmentation fault.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6084 from liewegas/wip-revert-scrub

Revert "osd: new pool settings: scrub intervals"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5832 from SUSE/wip-drop-dup-rcceph

ceph.spec.in: refrain from duplicating %{_sbindir}/rcceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ECBackend::handle_recovery_read_complete: do not expose the hash_info when getting the obc

Aside from expose an ECBackend internal detail, it causes ReplicatedPG
to cache the value causing trouble with WRITEFULL.

Partially fixes: #12983
Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend::handle_sub_read: restructure hash check and fix part of 12983

The previous code erroneously assumed that the reads on a single object
would always be in order, and not overlap.

Partially fixes: #12983
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6098 from liewegas/wip-crash-replay-interval

don't assume crash_replay_itnerval == 45

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45

e.g., it's 5 in teuthology's ceph.conf.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6076 from liewegas/wip-scrub-fix

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: preserve (some) flags when proxying reads

In particular, proxy RWORDERED as clients in the real world set
that.

Fixes: #13260
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6086 from stiopaa1/librados_missingsemicolon

examples/librados/hello_world.cc:missing semicolon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

examples/librados/hello_world.cc:missing semicolon

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #6068 from dachary/wip-12936-shec

erasure-code: shec must compare for equality with epsilon

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6052 from liewegas/wip-13098

osd: fix full request handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6075 from liewegas/wip-osd-getmap

osd: avoid duplicate MMonGetOSDMap requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6077 from liewegas/wip-qa

qa: http://ceph.com/qa -> http://download.ceph.com/qa
also some ceph.com/git -> git.ceph.com and a few line-endings - dmick

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "osd: new pool settings: scrub intervals"

This reverts commit 30810da4b573b3c7e679bc66eaa29b70246538cb.

After some discussion we have decided it is better to build a generic
dictionary in pg_pool_t to store infrequently used per-pool properties.

Signed-off-by: Sage Weil <sage@redhat.com>

OSDMap: fill in known encode_features where possible

Otherwise, if we get an incremental from hammer (struct_v = 6) we will
encode the full map as if it were before CEPH_FEATURE_PGID64, which
was actually pre-argonaut.  Similarly, if struct_v >= 7, we know it
was encoded with CEPH_FEATURE_OSDMAP_ENC.

Fixes: #13234
Backport: hammer
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #6083 from jcsp/wip-mds-allow-star

ceph-create-keys: set mds "allow *"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-create-keys: set mds "allow *"

So that 'tell' works out of the box.

Signed-off-by: John Spray <john.spray@redhat.com>

erasure-code: shec must compare for equality with epsilon

When comparing two double for strict inequality, a test for equality
using epsilon must also be done to ensure the results do not vary cross
platform.

http://tracker.ceph.com/issues/12936 Fixes: #12936

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5959 from mslovy/wip-current-dev

mon: Fix unnecessary at/near max target warn in ceph -s

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5356 from majianpeng/code-cleanup

osd: cleanup code when met crc error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5972 from Jerry7X/master

unsigned type is short for journal max_size,use uint64_t instead

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #6048 from XinzeChi/wip-filestore-qlock

filestore: fix peek_queue for OpSequencer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6040 from liewegas/wip-mon-kv

mon: some fixes for kv backend failures

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6022 from jdurgin/wip-13032

msg: randomize client nonces

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6012 from liewegas/wip-mon

mon: misc scaling fixes

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: fix quote enforcement on subdir mounts

While we don't allow setting quotas on the global
filesystem root, the inode which is root from the
client mount's point of view might have a quota.

Signed-off-by: John Spray <john.spray@redhat.com>

client: refactor quota check functions

Generalise the path traversal into check_quota_condition
and then call that from each of file_exceeded, bytes_exceeded,
bytes_approaching with the appropriate lambda function.

Motivated by fixing the path traversal and wanting to
only fix it in one place.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6027 from wido/mirror-docs

docs: Update docs to make it clear that there are mirrors worldwide

Reviewed-by: Kefu Chai <kchai@redhat.com>

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

AsyncConnection: Add new debug log

Signed-off-by: Haomai Wang <haomai@xsky.com>

osd/ReplicatedPG: fix ENOSPC checking

1. We need to return ENOSPC *before* we apply our side-effects to the obc
cache in finish_ctx().

2. Consider object count changes too, not just bytes.

3. Consider cluster full flag, not just the pool flag.

4. Reply only if FULL_TRY; silently drop ops that were sent despite the
full flag.

Fixes: #13098
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove [pool_]last_epoch_marked_full from OSDSuperblock

This is no longer used or needed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove buggy [pool_]last_map_marked_full tracking

This is buggy (we reset the pool epoch each time it remains full instead
of only on the initial transition from non-full to full) and now unused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do full check in do_op

1. The current pool_last_map_marked_full tracking is buggy.
2. We need to recheck this each time we consider the op, not just when it
is received off the wire.  Otherwise, we might get a message, queue it
for some reason, get a map indicating the cluster or pool is full, and
then requeue and process the op instead of discarding it.
3. For now, silently drop ops when failsafe check fails. This will lead to
stalled client IO.  This needs a more robust fix.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: set last_epoch_marked_full on map advance

This will get persisted and shared.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_epoch_marked_full to pg_history_t

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: set FULL_FORCE flag when honor_full is false

This currenty only applies to the MDS.  Eventually we can remove the
OSD MDS checks once we are confident all MDS instances are new enough
to set this flag.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add FULL_TRY and FULL_FORCE rados op flags

FULL_TRY = try to do the op; ENOSPC if it results in a net increase is
space.  client must cope.

FULL_FORCE = do the op even if it consumes space.  The MDS will use this.
We should restrict this based on the OSD cap (* vs w, probably).

Signed-off-by: Sage Weil <sage@redhat.com>

qa: https://ceph.com/git -> https://git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6031 from s09816/master

librbd:reads larger than cache size hang.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>