ReplicatedPG: clearing a whiteout should create the object

This was uncovered by 75321943729f1d5dfacb68645e3c5483740d66f8. Since
rbd_create() does a stat, the obc is cached as a whiteout, and the
subsequent create(EXCL) would fall through to return false from
maybe_create_new_object(). This would then skip adding a touch() to
the transaction.

Fixes: #13281
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6103 from SUSE/wip-re-redrop-fdupes

ceph.spec.in: re-re-drop fdupes

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph.spec.in: re-re-drop fdupes

The %fdupes call was dropped by 53072b9019caf72e0313b2804ea174237ed7da33
Later, it got accidentally reinstated by ceb93e8e69e125c9358f63b0099e7509b6624bcf

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6025 from ukernel/wip-13167

journaler: detect unexpected holes in journal objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5849 from ukernel/wip-32bits-ceph-fuse

Wip 32bits ceph fuse

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4791 from ceph/wip-11783

cephfs: detect completed cap flushes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6070 from jcsp/wip-reformat-subtreemaptest

Fix reformatting journals containing SUBTREEMAP_TEST events

Merge pull request #6082 from jcsp/wip-client-quota-roots

Fix quota enforcement on subdir mounts

Merge pull request #6078 from H3C/wip-13208

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6064 from H3C/wip-rbd-bugfix1

Objecter: misc fix for segmentation fault.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6084 from liewegas/wip-revert-scrub

Revert "osd: new pool settings: scrub intervals"

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #5832 from SUSE/wip-drop-dup-rcceph

ceph.spec.in: refrain from duplicating %{_sbindir}/rcceph

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6098 from liewegas/wip-crash-replay-interval

don't assume crash_replay_itnerval == 45

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45

e.g., it's 5 in teuthology's ceph.conf.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6076 from liewegas/wip-scrub-fix

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6086 from stiopaa1/librados_missingsemicolon

examples/librados/hello_world.cc:missing semicolon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

examples/librados/hello_world.cc:missing semicolon

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #6052 from liewegas/wip-13098

osd: fix full request handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6075 from liewegas/wip-osd-getmap

osd: avoid duplicate MMonGetOSDMap requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6077 from liewegas/wip-qa

qa: http://ceph.com/qa -> http://download.ceph.com/qa
also some ceph.com/git -> git.ceph.com and a few line-endings - dmick

Reviewed-by: Dan Mick <dmick@redhat.com>

Revert "osd: new pool settings: scrub intervals"

This reverts commit 30810da4b573b3c7e679bc66eaa29b70246538cb.

After some discussion we have decided it is better to build a generic
dictionary in pg_pool_t to store infrequently used per-pool properties.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6083 from jcsp/wip-mds-allow-star

ceph-create-keys: set mds "allow *"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-create-keys: set mds "allow *"

So that 'tell' works out of the box.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #5959 from mslovy/wip-current-dev

mon: Fix unnecessary at/near max target warn in ceph -s

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5356 from majianpeng/code-cleanup

osd: cleanup code when met crc error.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5972 from Jerry7X/master

unsigned type is short for journal max_size,use uint64_t instead

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #6048 from XinzeChi/wip-filestore-qlock

filestore: fix peek_queue for OpSequencer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6040 from liewegas/wip-mon-kv

mon: some fixes for kv backend failures

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6022 from jdurgin/wip-13032

msg: randomize client nonces

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6012 from liewegas/wip-mon

mon: misc scaling fixes

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: fix quote enforcement on subdir mounts

While we don't allow setting quotas on the global
filesystem root, the inode which is root from the
client mount's point of view might have a quota.

Signed-off-by: John Spray <john.spray@redhat.com>

client: refactor quota check functions

Generalise the path traversal into check_quota_condition
and then call that from each of file_exceeded, bytes_exceeded,
bytes_approaching with the appropriate lambda function.

Motivated by fixing the path traversal and wanting to
only fix it in one place.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6027 from wido/mirror-docs

docs: Update docs to make it clear that there are mirrors worldwide

Reviewed-by: Kefu Chai <kchai@redhat.com>

Objecter: repeated free op->ontimeout.

repeated free op->ontimeout in SafeTimer::timer_thread::callback->complete

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

Objecter: maybe access wild pointer(op) in _op_submit_with_budget.

look at "after giving up session lock it can be freed at any time by response handler" in _op_submit,
so the _op_submit_with_budget::op maybe is wild after call _op_submit.

Fixes: #13208
Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>

osd/ReplicatedPG: fix ENOSPC checking

1. We need to return ENOSPC *before* we apply our side-effects to the obc
cache in finish_ctx().

2. Consider object count changes too, not just bytes.

3. Consider cluster full flag, not just the pool flag.

4. Reply only if FULL_TRY; silently drop ops that were sent despite the
full flag.

Fixes: #13098
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove [pool_]last_epoch_marked_full from OSDSuperblock

This is no longer used or needed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove buggy [pool_]last_map_marked_full tracking

This is buggy (we reset the pool epoch each time it remains full instead
of only on the initial transition from non-full to full) and now unused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do full check in do_op

1. The current pool_last_map_marked_full tracking is buggy.
2. We need to recheck this each time we consider the op, not just when it
is received off the wire.  Otherwise, we might get a message, queue it
for some reason, get a map indicating the cluster or pool is full, and
then requeue and process the op instead of discarding it.
3. For now, silently drop ops when failsafe check fails. This will lead to
stalled client IO.  This needs a more robust fix.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: set last_epoch_marked_full on map advance

This will get persisted and shared.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_epoch_marked_full to pg_history_t

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: set FULL_FORCE flag when honor_full is false

This currenty only applies to the MDS.  Eventually we can remove the
OSD MDS checks once we are confident all MDS instances are new enough
to set this flag.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add FULL_TRY and FULL_FORCE rados op flags

FULL_TRY = try to do the op; ENOSPC if it results in a net increase is
space.  client must cope.

FULL_FORCE = do the op even if it consumes space.  The MDS will use this.
We should restrict this based on the OSD cap (* vs w, probably).

Signed-off-by: Sage Weil <sage@redhat.com>

qa: https://ceph.com/git -> https://git.ceph.com

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6031 from s09816/master

librbd:reads larger than cache size hang.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: http://ceph.com/qa -> http://download.ceph.com/qa

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: compensate for sloppy hobject scrub bounds from hammer

Hammer is sloppy about the hobject_t's it uses for the scrub bounds in that
the pool isn't set.  (Hammer FileStore doesn't care, but post-hammer is
much more careful about this sort of thing.)

Compensate by setting the pool on any scrub messages we receive.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: avoid duplicate MMonGetOSDMap requests

These are relatively expensive (we grab the full map from the mon) so we
should avoid duplicating our requests.

Track which requests are in flight.  Only send a new request when new
maps are asked for.  Resend requests when there is a new mon session.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6072 from yuyuyu101/update-org

Update org

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Update Xinze affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6066 from dachary/wip-ceph-detect-init

build/ops: ceph-detect-init must ignore .cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Update Chen Min affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update Tianshan affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

Update Haomai Wang affinity

Signed-off-by: Haomai Wang <haomai@xsky.com>

docs: Update docs to make it clear that there are mirrors worldwide

mds: fix error reformatting subtreemap_test events

Rare outside of vstart clusters, but if someone did
ever have one of these events in their journal and
try to update to latest ceph, they would end up
with bogus expire_pos on the reformatted events.

Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: detect malformed ESubtreeMap::expire_pos

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6063 from dachary/wip-tar-v7

build/ops: make dist needs files with names > 99 characters

Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: ceph-detect-init must ignore .cache

In case the test fails, it may leave files behind:

src/ceph-detect-init/.cache/v/cache/lastfailed

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5694 from guce/rgw_improve_key_operate

rgw: improve convenience for key operate.

librbd:reads larger than cache size hang.

Fixes:#13164

Signed-off-by: Lu Shi <shi.lu@h3c.com>

build/ops: make dist needs files with names > 99 characters

When running make distdir=ceph-9.0.3-1870-gfd861bb dist, a few files
have names longer than 99 characters and discarded, which then causes
the resulting tarbal to be incomplete:

tar: ceph-9.0.3-1870-gfd861bb/src/rocksdb/utilities/write_batch_with_index/write_batch_with_index_internal.cc: file name is too long (max 99); not dumped
tar: ceph-9.0.3-1870-gfd861bb/src/rocksdb/utilities/write_batch_with_index/write_batch_with_index_internal.h: file name is too long (max 99); not dumped

Use the tar-ustar format instead of the legacy v7
format (http://www.gnu.org/software/automake/manual/automake.html#Options). It
is unlikely machines with a C++11 compiler also have an antique tar
binary that would only support v7.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

filestore: fix peek_queue for OpSequencer

Fixes: #13209
Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6053 from ceph/wip-cmake-dang

Wip cmake dang

Merge pull request #6056 from guangyy/wip-13198

osd: print min_last_epoch_clean along with pg dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-aarch64'

osd: print min_last_epoch_clean along with pg dump

Fixes: 13198
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

mon/Elector: do a trivial write on every election cycle

Currently we already do a small write when the *first* election in
a round happens (to update the election epoch).  If the backend
happens to fail while we are already in the midst of elections,
however, we may continue to call elections without verifying we
are still writeable.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: assert/crash if there is a write error

Do this globally intead of relying on teh zillion mon callers to
check the error code.  There are no cases where we want to
tolerate a commit failure.

Fixes: #13089
Signed-off-by: Sage Weil <sage@redhat.com>

CMake - fix check for UDEV

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for FUSE

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libedit

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for Snappy

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for XFS

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for AIO

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for blkid

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libcurl

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for fcgi

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - add check for libuuid

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for leveldb

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for expat

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - Add check for keyutils

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix check for NSS

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

CMake - fix libatomic_ops and gperftools checks

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge remote-tracking branch 'gh/infernalis'

arch/arm: s/false/0/

arch/arm.c: In function 'ceph_arch_arm_probe':
arch/arm.c:54:28: error: 'false' undeclared (first use in this function)
  ceph_arch_aarch64_crc32 = false;  // sorry!
                            ^
arch/arm.c:54:28: note: each undeclared identifier is reported only once for each function it appears in

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: improve convenience for key operate.

1.key-type assignments based on context if it wasn't specified
In user    operate context, key-type assignment to KEY_TYPE_S3
In subuser operate context, key-type assignment to KEY_TYPE_SWIFT
In key     operate context, key-type assignment based on user type

2.fix RGWSubUserPool::add()
When create subuser generate secret by default

3.fix RGWAccessKeyPool::generate_key()
Avoid wrong key's username when create user and subuser at the same time
Check empty secret

Signed-off-by: Ce Gu <guce@h3c.com>

Merge pull request #6037 from liewegas/wip-12407

init-radosgw: fix restart w/ multiple instances

Reviewed-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

Merge pull request #5928 from ceph/wip-12819-infernalis

Wip 12819 infernalis

Merge pull request #6006 from dachary/wip-12936-shec

erasure-code: workaround i386 optimization bug with SHEC

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw: specify pid file to start-stop-daemon

This fixes restart when multiple instances are running.

Fixes: #12407
Tested-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #5922 from trociny/wip-pool-set-noscrub

osd: per pool control for scrub and deep-scrub

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5846 from xiexingguo/xxg-wip-13002

msg/simple: start over after fails to bind a port in specified range

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code: workaround i386 optimization bug with SHEC

http://tracker.ceph.com/issues/12936 Fixes: #12936

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5999 from dachary/wip-13000-ceph-disk

ceph-disk: fixes various race conditions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6008 from liewegas/wip-13190

osd/ReplicatedPG: tolerate promotion completion with stopped agent

Reviewed-by: Samuel Just <sjust@redhat.com>

journaler: detect unexpected holes in journal objects

Fixes: #13167
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #6011 from liewegas/wip-systemd

systemd: increase nproc ulimit