OSD: break connection->session->waiting message->connection cycle

Fixes: #12338
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 7f15e22be291df139c85909b96b2366f6dbadf37)

Merge pull request #5527 from SUSE/wip-12585-hammer

OSD crash creating/deleting pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5551 from ceph/wip-corpus-hammer

ceph-object-corpus: add 0.94.2-207-g88e7ee7 hammer objects

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-object-corpus: add 0.94.2-207-g88e7ee7 hammer objects

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5365 from dachary/wip-12491-hammer

buffer: critical bufferlist::zero bug

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5369 from dachary/wip-12498-hammer

get pools health'info have error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5370 from dachary/wip-12496-hammer

pgmonitor: wrong at/near target max“ reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5378 from dachary/wip-12394-hammer

Memory leak in Mutex.cc, pthread_mutexattr_init without pthread_mutexattr_destroy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5372 from dachary/wip-12493-hammer

the output is wrong when runing ceph osd reweight

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5374 from dachary/wip-12487-hammer

ceph osd crush reweight-subtree does not reweight parent node

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5380 from dachary/wip-12390-hammer

PGLog::proc_replica_log: correctly handle case where entries between olog.head and log.tail were split out

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5366 from dachary/wip-12303-hammer

arm: all programs that link to librados2 hang forever on startup

Merge pull request #5202 from SUSE/wip-12293-hammer

ceph.spec.in: rpm: cephfs_java not fully conditionalized

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5203 from SUSE/wip-11997-hammer

 ceph.spec.in: rpm: not possible to turn off Java

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5204 from SUSE/wip-11629-hammer

ceph.spec.in: SUSE/openSUSE builds need libbz2-devel

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5207 from SUSE/wip-12267-hammer

ceph.spec.in: 50-rbd.rules conditional is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5216 from SUSE/wip-12269-hammer

ceph.spec.in: ceph-common needs python-argparse on older distros, but doesn't require it

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5264 from SUSE/wip-12361-hammer

ceph.spec.in: snappy-devel for all supported distros

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5368 from dachary/wip-12331-hammer

ceph: cli throws exception on unrecognized errno

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5371 from dachary/wip-12494-hammer

ceph tell: broken error message / misleading hinting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5385 from dachary/wip-12501-hammer

error in ext_mime_map_init() when /etc/mime.types is missing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5411 from SUSE/wip-12446-hammer

ceph.spec.in: radosgw requires apache for SUSE only -- makes no sense

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5412 from SUSE/wip-12448-hammer

ceph.spec.in: useless %py_requires breaks SLE11-SP3 build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5318 from ceph/wip-12236-hammer

Possible crash while concurrently writing and shrinking an image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5319 from ceph/wip-12235-hammer

librbd: deadlock during cooperative exclusive lock transition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5296 from ceph/wip-12345-hammer

librbd: correct issues discovered via lockdep / helgrind

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5559 from ceph/wip-12682-hammer

object_map_update fails with -EINVAL return code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5468 from ceph/wip-12577-hammer

osd: fix repair when recorded digest is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5376 from dachary/wip-12433-hammer

Show osd as NONE in ceph osd map <pool> <object>  output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5359 from ceph/wip-11470.hammer

mon: PaxosService: call post_refresh() instead of post_paxos_update()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5691 from jdurgin/wip-10399-hammer

is_new_interval() fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Objecter: pg_interval_t::is_new_interval needs pgid from previous pool

When increasing the pg_num of a pool, an assert would fail since the
calculated pgid seed would be for the pool's new pg_num value instead
of the previous pg_num value.

Fixes: #10399
Backport: infernalis, hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f20f7a23e913d09cc7fc22fb3df07f9938ddc144)

Conflicts: (hobject_t sort order not backported, trivial resolution)
src/osdc/Objecter.cc
src/osdc/Objecter.h

osd_types::is_new_interval: size change triggers new interval

Fixes: 11771
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit ff79959c037a7145f7104b06d9e6a64492fdb95f)

Merge remote-tracking branch 'gh/wip-12536-hammer' into hammer

This includes wip-osd-compat-hammer.

0.94.3

Merge remote-tracking branch 'gh/wip-osd-compat-hammer' into wip-12536-hammer

Merge pull request #5589 from ceph/hammer-12709

Workunits : fs/misc/chmod.sh : Include ACL characters in permission check

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Workunits : fs/misc/chmod.sh : Include ACL characters in permission check.

Signed-off-by: Yazen Ghannam <yazen.ghannam@linaro.org>
(cherry picked from commit d3dbfffefb0ae53583350f53258dc902670da659)

tests: increase test coverage for partial encodes/decodes

Multiple combinations of offsets/lengths are now tested when
performing partial encodes/decodes of the bit vector.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3e145f714ac9b2d599b45a058c6b93595e38f424)

common: bit_vector extent calculation incorrect for last page

It's highly probable that the last page in the bit vector will not
be a full page size.  As a result, the computed extents will extend
beyond the data portion of the bit vector, resulting in a end_of_buffer
exception.

Fixes: #12611
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c6d98992691683524d3d96def83a90a6f5fe7f93)

osd/OSDMap: handle incrementals that modify+del pool

Because new_pools was processed after old_pools, if something
was modified and then deleted in the same incremental
map, then the resulting state would be a pool in the map
that had no entry in pool_name.

Fixes: #12429
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8d4932e721e32a33c1bada54e11d4ecccd868618)

hobject_t: fix get_boundary to work with new sorting regime

The hash is no longer the most significant field; set everything that is
more significant, too.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 45281efad1ae6b2b5ddd5549c56dbdd46671e5c5)

mon: disallow post-hammer OSDs if there are up pre-hammer OSDs

Force *all* OSDs to upgrade to hammer before allowing post-hammer
OSDs to join.  This prevents any pre-hammer OSDs from running at
the same time as a post-hammer OSD.

This commit, as well as the definition of the sentinal post-hammer
feature, should get backported to hammer stable series.

Backport: hammer
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e38d60e94503c7e8511a60a8dceceb3cd39c2bd8)

# Conflicts:
# src/mon/OSDMonitor.cc

include/ceph_features: define MON_METADATA feature

This was added in master in 40307748b9b8f84c98354a7141da645a3c556ba9.  We
define it but notably do not support it!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: include newlines in scrub errors

We may log more than one of these in a scrubmap; make sure they are
not concatenated.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2b3dd1bb7ab360e8f467a47b7ce3a0e4d3a979b2)

osd: fix condition for loggin scrub errors

We may have an error to log even though the authoritative map is empty:
when we have a digest that doesn't match all replicas.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d23766be902080ffb29528acb5a00d70beb97a9f)

osd: fix fallback logic; move into be_select_auth_object

The fallback behavior was printing a promising message but then doing
a 'continue' and not actually scrubbing the object.  Instead, fall back to
a less-bad auth inside be_select_auth_object.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b597db59fedc9a96dbe4b37f03b819dcf8fdb1bb)

osd: log a scrub error when we can't pick an auth object

If we really can't find a suitable auth, log an error.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a476d8f3903ca78b5cf36d63059f13cb5268efe5)

osd: repair record digest if all replicas match but do not match

If the recorded digest does not match the replicas, and all replicas
match, then (on repair) fix the recorded digest.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f581fecd8a16b9250fa96beec5aa26074f401028)

osd: move recorded vs on disk digest warning into be_compare_scrubmaps

This is a better place for it.  While we are here, change the wording to
clearly call out the 'recorded' digest (from object_info_t) vs the 'on
disk' digest (what we observed during scrub).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cf349ff35e9111ca52db1b0aad3753d0680b486d)

osd: be slightly paranoid about value of okseed

The scrubber.seed value is set based on the peer feature; check for that
too explicitly, and assert the scrubber.seed value matches.

No change in behavior here.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e34d31b7a9a71132a17b95cf5ad60255ece60243)

osd: be precise about "known" vs "best guess"

We cannot assume that the auth info has the "known" digest; all replicas
may have a matching digest that does not match the oi, or we may choose
a different auth info for some other reason.  Verify that the digest
matches the oi before calling it "known".

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 546d1c7b8224d96d0cc93dc3c96f52787cd69413)

osd: record digest if object is clean (vs entire scrub chunk)

If an individual object is clean, record it in the missing_digest map.
If not, leave it out.  This lets us drop the old condition that only
recorded any digests if the entire scrub chunk was clean.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c39e0e5f126028f552d410ab6924cac6e1f714fb)

Merge pull request #5160 from theanalyst/wip-11910-hammer

mon: pg ls is broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5384 from dachary/wip-12502-hammer

rgw: need conversion tool to handle fixes following #11974

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5405 from ceph/wip-12465-hammer

Log::reopen_log_file: take m_flush_mutex

Reviewed-by: Loic Dachary <ldachary@redhat.com>

hobject_t: decode future hobject_t::get_min() properly

The post-hammer wip-temp branch changed hobject_t::get_min() so that pool
is INT64_MIN instead of -1 and neglected to deal with the encoding compat
with older versions.  Compensate on hammer by mapping INT64_MIN to -1
locally.  See commit ff99af38df830e215359bfb8837bf310a2023a4d.

Note that this means upgrades from hammer to post-hammer *must* include
this fix prior to the upgrade.  This will need to be well-documented in the
release notes.

Master gets a similar fix so that they map our min value to the new
INT64_MIN one on decode.

Fixes: #12536 (for hammer)
Signed-off-by: Sage Weil <sage@redhat.com>

OSDMonitor::preprocess_get_osdmap: send the last map as well

Fixes: #12410
Backport: hammer
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1320e29dfaee9995409a6d99b9ccaa748dc67b90)

Merge pull request #5121 from theanalyst/wip-11983-hammer

FAILED assert(!old_value.deleted()) in upgrade:giant-x-hammer-distro-basic-multi run

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5269 from dachary/wip-12362-hammer

stuck incomplete

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5117 from theanalyst/wip-12099-hammer

rgw: rados objects wronly deleted

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5118 from theanalyst/wip-12042-hammer

DragonDisk fails to create directories via S3: MissingContentLength

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5214 from SUSE/wip-12299-hammer

RGW Swift API: support for 202 Accepted response code on container creation

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5226 from SUSE/wip-12322-hammer

rgw: keystone does not support chunked input

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5227 from SUSE/wip-12323-hammer

RGW Swift API: XML document generated in response for GET on account does not contain account name

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5228 from theanalyst/wip-11872-hammer

RGW does not send Date HTTP header when civetweb frontend is used

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5229 from theanalyst/wip-12242-hammer

Fix tool for #11442 does not correctly fix objects created via multipart uploads

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5237 from theanalyst/wip-12245-hammer

rgw: empty json response when getting user quota

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5284 from SUSE/wip-12398-hammer

rgw: Properly respond to the Connection header with Civetweb

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5285 from SUSE/wip-12399-hammer

rgw: multipart list part response returns incorrect field

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5286 from SUSE/wip-12400-hammer

rgw: radosgw-admin dumps user info twice

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5427 from dachary/wip-12088-hammer-part-2

Fh ref count will leak if readahead does not need to do read from osd

Fh ref count will leak if readahead does not need to do read from osd

The 3c8cdeacf46ae4031189d2ef6948aa3b6ab4ae43 backport introduced a leak.

http://tracker.ceph.com/issues/12319 Fixes: #12319

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #5120 from theanalyst/wip-11999-hammer

cephfs Dumper tries to load whole journal into memory at once

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5119 from theanalyst/wip-12098-hammer

kernel_untar_build fails on EL7

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5417 from dachary/wip-11998-hammer

debian/control: ceph-common (>> 0.94.2) must be >= 0.94.2-2

Reviewed-by: Sage Weil <sage@redhat.com>

debian/control: ceph-common (>> 0.94.2) must be >= 0.94.2-2

The d8733be2ef8874b9a858a7ffddfb81b9b656e9a6 backport introduced a
regression by adding an incorrect Depends / Break combo supposed to
reflect the fact that ceph_argparse moved from ceph to ceph-common after
v0.94.2. It assumed the package is released under the 0.94.2 version
where in reality it is released under the 0.94.2-1xxx version (where xxx
is trusty, jessie etc.).

The Depends / Break combo is changed to use 0.94.2-2 instead.

See also http://tracker.ceph.com/issues/12529 for a larger discussion.

http://tracker.ceph.com/issues/11998 Fixes: #11998

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: drop SUSE-specific %py_requires macro

%py_requires expands to

BuildRequires: /usr/bin/python
PreReq: python = 2.7

The BuildRequires: is already provided, and the PreReq is wrong because
e.g. SLE11-SP3 (a platform we are trying to support) has Python 2.6.

http://tracker.ceph.com/issues/12351 Fixes: #12351

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit cbf6c7b46b42e163950bf7956e5b7565fca961c9)

ceph.spec.in: remove SUSE-specific apache2-mod_fcgid dependency

This package is no longer required for RGW to work in SUSE.

http://tracker.ceph.com/issues/12358 Fixes: #12358

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit f3f8d3660d0ece3139e5f3dc7881ccceadcf648d)

Conflicts:
ceph.spec.in
            In master, the "%if %else %endif" has been changed
            into two "%if %endif"s

tests: verify that image shrink properly handles flush op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8dfcc72071d63c7867a5a59abe871cf5db673a0a)

librbd: invalidate cache outside cache callback context

When shrinking an image, it's possible that the op flush callback
will be from within the cache callback context.  This would result
in a deadlock when attempting to re-lock the cache lock in order to
invalidate the cache.

Fixes: #11743
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 726d699b7790c7e371279281ab32cd3aeb8ece7b)

librbd: don't cancel request lock early

It's possible that a stale notice is received and will
be discarded after the request lock has been canceled.
This will stale the client.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d9dd5c5890029107df40ed438f23fb9865d7de29)

tests: new test for transitioning exclusive lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f97ce46ea4c9c7cdc6e37e3759871d5b41b25769)

tests: verify that librbd will periodically resend lock request

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d2a1c226ab3f7e202f62896f0c80c4cf3607cdaf)

common: Mutex shouldn't register w/ lockdep if disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 879b8a73e2452332b26b8f3428ff5e3e0af8ddad)

librbd: improve debugging output for ImageWatcher

Include the instance pointer so that different images
can be differentiated in the logs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b951a7398e3e749cb64a3f6a604212a5627e82a0)

librados_test_stub: watcher id should be the instance id (gid)

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3e1e561cc29043eb64dff8afb3a2c1ed77788e93)

librbd: retry lock requests periodically until acquired

If the exclusive lock owner acks the lock release request but crashes
before it actually releases the lock, the requestor will wait forever.
Therefore, after a certain timeout, retry the request again until it
succeeds.

Fixes: #11537
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 37c74e6e5274208d1b6efaf315afec03ea7eaa82)

librbd: don't hold owner_lock for write during flush

The various IO callback codepaths will attempt to take
the lock, which will result in deadlock since the flush
cannot complete.

Backport: hammer
Fixes: #11537
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2b6d0633d5c89de3a557cdd72621dfc19f0540de)

Merge pull request #5248 from ceph/wip-11833-hammer

mon: add an "osd crush tree" command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5222 from ceph/hammer-12088

client: reference counting 'struct Fh'

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5231 from theanalyst/wip-12243-hammer

Civetweb RGW appears to report full size of object as downloaded when only partially downloaded

Merge pull request #5243 from theanalyst/wip-12239-hammer

librbd/internal.cc: 1967: FAILED assert(watchers.size() == 1)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5241 from theanalyst/wip-12238-hammer

[  FAILED  ] TestLibRBD.ExclusiveLockTransition

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

lockdep: do not automatically collect all backtraces

It is expensive to collect backtraces every time a lock is
checked in order to provide cycle backtraces.  The backtraces
can be forced on for specific locks or globally via the new
config option "lockdep_force_backtrace".

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7354d25f56c5e004f288febdee2b6961c069163b)

librbd: flush operations need to acquire owner lock

Cache writeback operations will expect the owner lock to be held.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit adfa2e0070ccca7b6556d3bfc5fac7ce4d43a4d0)

librbd: avoid infinite loop if copyup fails

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 43e0e3cd63f0067217ed0811d73f6c546f3027be)

librbd: flush pending ops while not holding lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3d5cef38c37e5dda6b23751ad560851f1304d86d)

tests: fix possible deadlock in librbd ImageWatcher tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 742a85d667b26b3490d96270b5c500b08f2a5283)

tests: enable lockdep for librbd unit tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 45cb9cb5980e90235b8244d0d61ece40bde4784a)