os/bluestore/BlueFS: dump allocator freelist on failure in reclaim_blocks

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 15ff48518a65e0a3add97c5b7f650d329abf1dd1)

os/bluestore/BlueFS: handle failure to reclaim blocks without crashing

We shouldn't fail to reclaim space in general, but if we do, do not treat
it as a fatal error.  Log loudly.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6dc1bcd9e72b01c3fbf09c7eb91cd7cf4e01f94a)

unitest_bit_alloc, unittest_alloc: fixes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7492aef0f717f1fb5c09d55c99ded2adce7997d6)

os/bluestore: prealloc/reserve extent vector for common alloc path

No need to worry about the gift/reclaim path--those are very rare.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b0c243875fe963e2958828bef62f3ba55e123434)

os/bluestore/Allocator: drop unused and goofy release_extents

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 14676ac8cd69a1b4c06c728b0c00b00dd6a71a7a)

os/bluestore: drop useless count arg to allocate

The vector<> has a size.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 0aa402fba19f5c83806304b2ecef6902c481eff4)

os/bluestore: return blocks allocated from allocate()

Instead of having a separate output argument with the number of
blocks allocated, just return it via the return value.  Simplifies
the calling convention.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 95c78706317f2fe61b8f48de5e8539ed49221d64)

os/bluestore: manage vector from ExtentList

ExtentList was previous relying the caller to preallocate/size the
vector to be large enough for the worst case allocation of extents,
and keeping it's own manual count of the extent list size.  Instead,
manage that from ExtentList, and remove the preallocation from the
callers.

Fixes: http://tracker.ceph.com/issues/18573
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 65502891dd66569c8d99384e63b68decdab1c8e2)

Merge tag 'v11.1.1' into kraken

v11.1.1

Merge pull request #12960 from SUSE/wip-18546-kraken

kraken: [teuthology] update Ubuntu image url after ceph.com refactor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6d17befb3bbc3d83c9d23d763ad95e1e7b2e4be0)

Merge pull request #12949 from cbodley/wip-rgw-kraken

rgw: valgrind fixes for kraken

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: RGWCloneMetaLogCoroutine uses RGWMetadataLogInfoCompletion

Fixes: http://tracker.ceph.com/issues/18412
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 34a2edbf53f26a21e3206027dc61efc70d9c6aad)

rgw: expose completion for RGWMetadataLog::get_info_async()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 7b3532e2ad0f478a1670ceb61fd68209df87aa8b)

rgw: RGWMetaSyncShardCR drops stack refs on destruction

if the coroutine is canceled before collect_children() can clean up
all of its child stacks, those stack refs will leak. store these
stacks as boost::intrusive_ptr so the ref is dropped automatically on
destruction

Fixes: http://tracker.ceph.com/issues/18300
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 060fe72faf6a483a36d481207c6624c46a414231)

rgw: RGWAsyncRadosRequest drops notifier ref on cancel

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1d586f76a11fed937fc7bb0f7cf6a44ca0506881)

rgw: release RGWAioCompletionNotifier refs on destruction

Fixes: http://tracker.ceph.com/issues/18407
Fixes: http://tracker.ceph.com/issues/18414
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 7a4975adba8b890e211d47662896e445a9822970)

Merge pull request #12889 from dillaman/wip-18452-kraken

kraken: librbd: new API method to force break a peer's exclusive lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw_rados: sanitize dout print in GWRados::get_obj_iterate_cb(...)

We cannot assume pbl may be deferenced.  Per review, move the dout
print into the r < 0 condition--since it's now an error, make it's
trace level 0.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c0233c4abb9f99304e3d82272bbb4385086bbfc0)

rgw_rados: add guard assert in add_io()

Use the iterator-returning insert operation in std::map, check
assert the insert case.  As a side effect, this makes use of the
inserted object record more clear.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit d10c37adf925d8b34daa81b419463ea376ec27aa)

Merge pull request #12911 from athanatos/wip-18165

osd/PrimaryLogPG::failed_push: update missing as well

Merge pull request #12910 from athanatos/wip-18358

osdc/Objecter: resend pg commands on interval change

Objecter: resend pg commands on interval change

mark_lost_unfound* are now async since the rework, so we need
the Objecter to be able to resend on interval change.  This
is preferable to somehow requeueing the Command because they
don't use the normal op queue.

Fixes: http://tracker.ceph.com/issues/18358
Signed-off-by: Samuel Just <sjust@redhat.com>

Objecter: fold last_force_resend into op_target_t

It's always passed along with op_target_t, this seems simpler.

Signed-off-by: Samuel Just <sjust@redhat.com>

librados: blacklist_add should wait for latest OSD map

This ensures that future operations against the OSDs force
a OSD map update to notice the blacklisted client.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9242a2e4e1a5a9fcea48d8135b1589493fc28242)

librbd: prevent assertion failure when journal IO is blacklisted

Fixes: http://tracker.ceph.com/issues/18429
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c720f6e3704ed7e8cf41dffdb931dbb05d59a003)

librbd: ignore blacklist error when releasing exclusive lock

This ensures the journal and object map are properly closed so that the
image can be properly closed w/o failing any assertions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 418dcf29cb8c450049047e09a4dad2941af87018)

librbd: fail immediately if the exclusive lock cannot be acquired

Fixes: http://tracker.ceph.com/issues/16988
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 47b89f4d48a0b99876462167389df28893a8d418)

librbd: add new lock_get_owners / lock_break_lock API methods

If the client application supports failover, let the application
force break the current lock and blacklist the owner. This is
required in case the current lock owner is alive from the point-of-view
of librbd but failover was required due to a higher level reason.

Fixes: http://tracker.ceph.com/issues/18327
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9a5a8c75a025143cee6f92f3dbc3a12f2b6a9ad7)

librbd: separate break lock logic into standalone state machine

The current lockers are now queried before the lock is attempted to
prevent any possible race conditions when one or more clients attempt
to break the lock of a dead client.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 23f60fee86d1ff9b261fbb6411746a2a9479cf19)

librbd: separate locker query into standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 03533b912c59d5e433d0f006e1a063e014468ca5)

PrimaryLogPG::failed_push: update missing as well

This way, pick_newest_available should do the right thing.

Fixes: http://tracker.ceph.com/issues/18165
Signed-off-by: Samuel Just <sjust@redhat.com>

osd: _exit() intead of exit() for failure injection

This avoids a segv from code that doesn't easily shut
down (e.g., bluestore).

Fixes: http://tracker.ceph.com/issues/18372
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c2fac9c9a92a63b9a456c9e301c4b393de096a97)

os/bluestore/BlueFS: fix reclaim_blocks

We need to return all extents to the caller.  The current code
fails to assign *offset so it appears like a single extent from
the start of the device, which is very wrong.

Fixes: http://tracker.ceph.com/issues/18368
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 882ad33b527c9ea2224dcdc28f9fa3f459ee4712)

os/bluestore/bluestore_types: operator<< for AllocExtent

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 8e313a24965b6ab75e6f994bbd9ddf248e382f0d)

Merge pull request #12823 from SUSE/wip-18451-kraken

kraken: tests: update rbd/singleton/all/formatted-output.yaml to support ceph-ci

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: subst repo and branch in qemu test urls

References: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 2484a5546160560a4050d35400a6edce37ae48ad)

tests: subst branch and repo in qa/tasks/qemu.py

References: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 74689df754561e11a3377998840efcea9e780755)

tests: subst repo name in krbd/unmap/tasks/unmap.yaml

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit ed0e8be0b2c4d7a3e6e0716a0211d19e8b93f125)

tests: subst repo name in qa/tasks/cram.py

Inspired by bcbe45d948f1c4da02e27b3be5f29a8b52745e28

Fixes: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 56e37e41f4dddd289dd3c1886b192cd328ed311b)

Merge pull request #12814 from yuriw/kraken

Kraken

Using centos 7.2 as latest version

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 6c00a64e8723e06b98c2d7a28d6fc28260e087bc)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa: Cleaned up distros to use `latest` versions

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit b29faef354ec47a27178716f6e1caeb796fbf844)

Merge pull request #12803 from athanatos/wip-oi-mismatch

PrimaryLogPG: don't update digests for objects with mismatched names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'kraken' into wip-oi-mismatch

Merge pull request #12773 from yuriw/kraken

qa: fixed distros links

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: fixed distros links

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit fbb560c90101937317380a6621a70564f10e0ae3)

Merge pull request #12768 from yuriw/wip_fix_distros

qa: Fixed link to centos distro

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: Fixed link to centos distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

PrimaryLogPG: don't update digests for objects with mismatched names

I've only seen this on one cluster, but let's not issue repops during
scrub on objects where the object_info_t::soid value is not correct.
The cluster in question has been through many different non-release
kernels and osd versions, so the objects presumably came about due to an
old xfs or filestore bug.  They recently became fatal since we made
filestore crash on ENOENT for setattrs.  In the past, the cluster just
silently tolerated them.

http://tracker.ceph.com/issues/18409 is a larger feature to detect these
better and repair them automatically.

Related: http://tracker.ceph.com/issues/18409
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12752 from SUSE/wip-18392-kraken

kraken: qa/workunits/rbd/test_lock_fence.sh fails

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: rbd/test_lock_fence.sh: fix rbdrw.py relative path

This commit fixes a regression introduced by
cf294777ea92f0911813a7132068584d4f73a65a

Fixes: http://tracker.ceph.com/issues/18388
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 91231de16dbe4d0e493ec617165a2b38078d122b)

os/bluestore: _txc_release_alloc when do wal cleaning

Signed-off-by: Xinze Chi <xinze@xsky.com>
(cherry picked from commit 59fdcc199039f875bff5bc218b61b2e6857e885c)

Merge pull request #12475 from dillaman/wip-18231-kraken

librbd: remove consistency group rbd cli and API support

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mon/OSDMonitor: fixup sortbitwise flag warning

"ceph -s" does not report warning when using
command "ceph osd unset sortbitwise" to drop
sortbitwise flag.
we should use "osdmap.get_up_osd_features() &
CEPH_FEATURE_OSD_BITWISE_HOBJ_SORT"
instead of "(osdmap.get_features(CEPH_ENTITY_TYPE_OSD, NULL) &
CEPH_FEATURE_OSD_BITWISE_HOBJ_SORT)",
because osdmap.get_features only get local "features"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit c25ee187e28724846d0011cd8145e16956d3636e)

Merge pull request #12630 from liewegas/wip-workunit-retry

qa/tasks/workunit: clear clone dir before retrying checkout

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12614 from liewegas/wip-task-peer

qa/tasks/peer: update task based on current peering behavior

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12632 from liewegas/wip-centos-73

qa/suites: switch from centos 7.2 to 7.x

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/distros: centos_7.yaml -> centos.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: centos_7.2.yaml -> centos_7.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: add centos 7.3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/distros: add centos 7 yaml; use that instead

No need to be picky about the point release here.

Signed-off-by: Sage Weil <sage@redhat.com>

11.1.1

Merge pull request #12629 from liewegas/wip-bluestore-shard-extents

os/bluestore: remove 'extents' from shard_info

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

qa/tasks/workunit: clear clone dir before retrying checkout

If we checkout ceph-ci.git, and don't find a branch,
we'll try again from ceph.git. But the checkout will
already exist and the clone will fail, so we'll still
fail to find the branch.

The same can happen if a previous workunit task already
checked out the repo.

Fix by removing the repo before checkout (the first and
second times).  Note that this may break if there are
multiple workunit tasks running in parallel on the same
role.  That is already racy, so if it's happening, we'll
want to switch to using a truly unique clonedir for each
instantiation.

Fixes: http://tracker.ceph.com/issues/18336
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove 'extents' from shard_info

We don't use this field; no need to consume the space
on disk.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluefs: make bluefs_preextend_wal_files default is false.

This make rocksdb log corruption. Before fix, disable this function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon: MonMap: clear 'mon_info' during decode

It so happens that it's not safe to assume the monmap will be in an
empty state upon decoding.

Turns out the MonClient will reuse the MonMap instance when decoding
the just received map from the monitors. Should the monitors be on an
older version that do not support 'mon_info', this field will not be
decoded (after all, there's no field to decode from); but by this time,
the MonClient would already have a built monmap, which could have
populated 'mon_info' with temporary mon names from 'mon initial
members'.

Given the existing entries in 'mon_info', and the conflicting entries in
'mon_addr', we would end up asserting in 'sanitize_mons()'. This becomes
a non-issue if 'mon_info' is empty, as was unfortunately presumed.

Fixes: http://tracker.ceph.com/issues/18265
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge branch 'kraken' into wip-18314

qa/tasks/peer: update task based on current peering behavior

This changed in 0be3f5f72e169fad08dcb0240de45287b567bd49.

Fixes: http://tracker.ceph.com/issues/18330
Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: output recovery_state records for Down and Incomplete states

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12603 from liewegas/wip-fix-rwordered

osd: fix CEPH_OSD_FLAG_RWORDERED

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: fix CEPH_OSD_FLAG_RWORDERED

Broke this in 3a2ed3b136de68192f661b6992cbb96ce6aafd20.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: enlarge omap request bytes limitation

For cephfs, 4M buffer can only encode about 5k dentries. It's too
small for directory.

Fixes: http://tracker.ceph.com/issues/18314
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12587 from yuriw/wip-fix-subset-script-master

qa: fixed script to schedule rados and other suites with --subset option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

changed script name
added kraken
parameterized suite name to be able to use for rados, rbd etc.

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #12590 from athanatos/wip-ping-write-ordered

osd: force watch PING to be write ordered

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12593 from wjwithagen/wip-wjw-freebssd-SO_reuseaddr

msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12584 from mkoderer/openstack_nova_doc

doc: mention corresponding libvirt section in nova.conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

osd/: treat PINGs as RWORDERED

89fd030bf9436dc4e37cc3a0f935ec077455d9d5 switched them to show up
as reads to avoid logging them, but we still pipeline them with
reconnects.  Thus, also force them to be rwordered.

Fixes: http://tracker.ceph.com/issues/18310
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12540 from dmick/wip-18297

ceph.in: allow 'flags' to not be present in cmddescs

Merge pull request #12577 from liewegas/wip-osd-backfill

qa/tasks/osd_backfill.py: wait for osd.[12] to start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12574 from liewegas/wip-bluestore-split

os/bluestore: preserve source collection cache during split

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

DOC: Mention corresponding section in nova.conf

rbd_user and rbd_secret_uuid are part of the libvirt configuration.

Signed-off-by: Marc Koderer <marc@koderer.com>

Merge pull request #12472 from dillaman/wip-18244

librbd: remove image header lock assertions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12532 from ifed01/wip-bluestore-large-onode-test

os/bluestore: add UT for an estimation of Onode in-memory size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12505 from ceph/wip-jlayton-submount

client: set metadata["root"] from mount method when it's called with …

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12570 from wjwithagen/patch-9

common/str_list.h: fix clang warning about std::move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12541 from liewegas/wip-bluestore-omap-flush

os/bluestore: include modified objects in flush list even if onode unchanged

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/tasks/osd_backfill.py: wait for osd.[12] to start

...before sending a tell command.  Otherwise osd.2 might
start without 1, the io unblocks, and the tell fails
because osd.1 is still down.

Fixes: http://tracker.ceph.com/issues/18303
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: preserve source collection cache during split

OSD split transactions look something like

 mkcoll new
 split old
 ...
 omap_rmkey_range old
 omap_setkeys old
 omap_setkeys new

The last part splits the log into two pieces.  The
problem is that the rmkey_range needs to wait on old
omap transactions to flush, and those are linked to the
old onode, and split clears the cache.  The result is
that we don't wait, rmkeyrange leaves some recent pg log
keys behind, and on OSD restart we get an error because
the object doesn't belong to the (old) collection.

Fix this by preserving objects in the old collection and
only clear out objects that are moving to the newly
split collections.  This will include the pgmeta object
that we care about.

(Note that we are one step closer to preserving the
cache contents across the split, but not quite there
yet: at this point we don't have all of the destination
collections.  A change in the ObjectStore interface is
probably needed to make that not be extremely awkward.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12571 from athanatos/wip-15943

osd: Fix map gaps again (bug 15943)

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12572 from liewegas/wip-python-rados-test

qa/workunits: include extension for nose tests

Merge pull request #12568 from ifed01/wip-bluestore-onode-diet2

os/bluestore: reduce Onode in-memory footprint

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12555 from socketpair/usr_bin

systemd: Fix startup of ceph-mgr on Debian 8

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12355 from chhabaramesh/bit_alloc_mempool

os/bluestore: simplified allocator interfaces to single apis

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits: include extension for nose tests

When you have a relative path you have to include the extension.
Weird.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12412 from ceph/wip-jlayton-suid

client/mds: Clear setuid bits when writing or truncating

Clang/str_list.h: fix clang warning about std::move

/home/jenkins/workspace/ceph-master/src/include/str_list.h:99:10: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
  return std::move(str_vec);
         ^
/home/jenkins/workspace/ceph-master/src/include/str_list.h:99:10: note: remove std::move call here
  return std::move(str_vec);
         ^~~~~~~~~~       ~
1 warning generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>