lockdep: allow lockdep to be dynamically enabled/disabled

librbd test cases attempt to enable lockdep coverage via the librados
API.  Use a configuration observer to register/unregister lockdep
support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit af0cade0293be22e74a3293765676c7b227b06c4)

tests: librbd API test cannot use private md_config_t struct

Remove all depencencies on md_config_t and instead use librados API
methods to get/set configuration values.

Fixes: #12479
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c51aadb447d3447b5dace19c5cb6268e94247c8)

tests: ensure old-format RBD tests still work

Override the RBD default image format back to version 1
to ensure tests properly cover the old format.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b95edb5bd373bd05f83516eccfd30a6b5cf4693)

librados_test_stub: implement conf get/set API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4d03c664f2a9362009c7e3077ed65e2989f0f21b)

crypto: use NSS_InitContext/NSS_ShutdownContex to avoid memory leak

Switched to context-aware NSS init/shutdown functions to avoid conflicts
with parent application.  Use a reference counter to properly shutdown the
NSS crypto library when the last CephContext is destroyed.  This avoids
memory leaks with the NSS library from users of librados.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9fa0112dd9286178da1d76020158503b2062d252)

auth: use crypto_init_mutex to protect NSS_Shutdown()

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit cf058174fe6be26129eb52f4fb24d0f7146ef7af)

auth: reinitialize NSS modules after fork()

Fixes: #11128
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 4c24d0cc074462ae258b5cf901cd884bb4f50a53)

Merge pull request #5223 from SUSE/wip-12305-hammer

ceph.spec.in: running fdupes unnecessarily

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5716 from dachary/wip-12851-hammer

Ensure that swift keys don't include backslashes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5717 from dachary/wip-12591-hammer

rgw: create a tool for orphaned objects cleanup

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5755 from dachary/wip-12589-hammer

ceph-disk zap should ensure block device

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5732 from ceph/wip-11455-hammer

rgw: init some manifest fields when handling explicit objs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5721 from dachary/wip-12853-hammer

RGW Swift API: X-Trans-Id header is wrongly formatted

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #5498 from ceph/wip-12432-hammer

rgw: HTTP return code is not being logged by CivetWeb

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5527 from SUSE/wip-12585-hammer

OSD crash creating/deleting pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: always check zap is applied on a full device

zap only makes sense on a full device and this is verified for

    ceph-disk prepare --zap-disk

but not for

    ceph-disk zap

Unify the two and add a test to check that it fails as expected when
trying to zap a directory.

http://tracker.ceph.com/issues/11272 Fixes: #11272

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 8e1bcf21f090fc98ac9b495d76614e3012f5872a)

Conflicts:
src/test/ceph-disk.sh : trivial resolution

Merge pull request #5551 from ceph/wip-corpus-hammer

ceph-object-corpus: add 0.94.2-207-g88e7ee7 hammer objects

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-object-corpus: add 0.94.2-207-g88e7ee7 hammer objects

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: init some manifest fields when handling explicit objs

Fixes: #11455
When dealing with old manifest that has explicit objs, we also
need to set the head size and head object correctly so that
code that relies on this info doesn't break.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d7bf8cb594e5276d1c80544f5ec954d52b159750)

Merge pull request #5365 from dachary/wip-12491-hammer

buffer: critical bufferlist::zero bug

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5369 from dachary/wip-12498-hammer

get pools health'info have error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5370 from dachary/wip-12496-hammer

pgmonitor: wrong at/near target max“ reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5378 from dachary/wip-12394-hammer

Memory leak in Mutex.cc, pthread_mutexattr_init without pthread_mutexattr_destroy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5372 from dachary/wip-12493-hammer

the output is wrong when runing ceph osd reweight

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: rework X-Trans-Id header to be conform with Swift API.

Fixes: #12108
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit d1735a4e8ea2056d333a5001a7a410fc03fe1b9e)

Transaction Id added in response

Signed-off-by: Abhishek Dixit dixitabhi@gmail.com
(cherry picked from commit b711e3124f8f73c17ebd19b38807a1b77f201e44)

Conflicts:
src/rgw/rgw_rest.cc
             the prototype of the end_header( function
             in the context of the diff changed

rgw: api adjustment following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7cf1f376e7dbc98a44f4a8e78bfa11f84291a941)

rgw: orphans, fix check on number of shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 47edec359be2a4d230a4a54c0c670d0aa711fa1e)

rgw: orphans, change default number of shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5528f21d0861d42378a8eb10759a1e55ddd66c09)

rgw: change error output related to orphans

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit cac57caf639236adc9a641a85352eb729b58725a)

rgw: orphan, fix truncated detection

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d8ce04db8542dae0027423f3c50355e5de74dc9a)

radosgw-admin: simplify orphan command

No need for --init-search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4980cbd04ff650d79a033bc5298e6c1abe985133)

radosgw-admin: stat orphan objects before reporting leakage

We don't want to report new objects as leaked, because they might just
be written, and waiting to be linked to a new object. The number of
seconds prior to the job initialization can be cofigured through
--orphan-stale-secs (by default it's 24h).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 66b0090023859d97da2bbaca698b6a50c225abd0)

radosgw-admin: orphans finish command

A command to remove orphans data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 55d6f5ecf18f532c7f056f8b60c101843594b00c)

rgw: cannot re-init an orphan scan job

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f8993102731518153bb18eb74c3e6e4943a5dbac)

rgw: stat_async() sets the object locator appropriately

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d7d117121858155acc8548908e9459a48c676dd1)

rgw: list_objects() sets namespace appropriately

list_objects() now uses parse_raw_oid(), so that it can set the correct
namespace. It only affects users of the function that want to get all
objects in bucket, regardless to the namespace associated with it. This
makes it so that the orphan code actually works now with namespaced
objects, and with special named objects (namely, start with underscore).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 20bd490bebf9be3553d4e25322c9544b0b090086)

rgw: modify orphan search fingerprints

Now works with multipart uploads.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit fa61ea2ccc94f04afb39868ac34abc264806d8d4)

rgw: compare oids and dump leaked objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1bc63d98ff8a3aa180065153690b4f8a73658b79)

rgw: keep accurate state for linked objects orphan scan

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f19b2f087cfb5a89edf3b19487c0df3a214e350f)

rgw: iterate over linked objects, store them

only keep part of the oid name if it is in a namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 75902fdb12ce0413d3beb455df47134f3d7386e7)

rgw: add rgw_obj::parse_raw_oid()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7c1aa8373a090d5c1b2aa121582547723120a079)

rgw: iterate asynchronously over linked objects

Read objects manifest. So that we could keep the relevant info later.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 05a953d965a0a6a0e9feddaa62f7429e154e828e)

rgw: async object stat functionality

An async functionality that stats object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 13adf3cf4f5fd8a0feb7cc1d7a4ee4ded9f573db)

rgw-admin: build index of bucket indexes

for the orphan search tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 16a2dbd838b69fb445c06122a68fc65ec3a7b3de)

rgw: initial work of orphan detection tool implementation

So far doesn't do much, iterate through all objects in a specific pool
data, store it in a sharded index.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 767fc29aa62a0a4e4dd8a78785576e8f1b44ee5b)

Avoid an extra read on the atomic variable

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
(cherry picked from commit 7476e944af049aefdb8871cce62883b86f6b6f8d)

RGW: Make RADOS handles in RGW to be a configurable option

Having a configurable number of RADOS handles is resulting
in significant performance boost for all types of workloads.
Each RGW worker thread would now get to pick a RADOS handle
for its lifetime, from the available bunch.

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
(cherry picked from commit b3c3a24fe9c0fbb2bdbbccb08b40efc1dca207cc)

Merge pull request #5374 from dachary/wip-12487-hammer

ceph osd crush reweight-subtree does not reweight parent node

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5380 from dachary/wip-12390-hammer

PGLog::proc_replica_log: correctly handle case where entries between olog.head and log.tail were split out

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: avoid using slashes for generated secret keys

Just use plain alphanumeric characterset.

Fixes: #7647
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 3f04a6126fdbfb93304f798da3775c0eec9b7d44)

Merge pull request #5366 from dachary/wip-12303-hammer

arm: all programs that link to librados2 hang forever on startup

Merge pull request #5202 from SUSE/wip-12293-hammer

ceph.spec.in: rpm: cephfs_java not fully conditionalized

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5203 from SUSE/wip-11997-hammer

 ceph.spec.in: rpm: not possible to turn off Java

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5204 from SUSE/wip-11629-hammer

ceph.spec.in: SUSE/openSUSE builds need libbz2-devel

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5207 from SUSE/wip-12267-hammer

ceph.spec.in: 50-rbd.rules conditional is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5216 from SUSE/wip-12269-hammer

ceph.spec.in: ceph-common needs python-argparse on older distros, but doesn't require it

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5264 from SUSE/wip-12361-hammer

ceph.spec.in: snappy-devel for all supported distros

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5368 from dachary/wip-12331-hammer

ceph: cli throws exception on unrecognized errno

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5371 from dachary/wip-12494-hammer

ceph tell: broken error message / misleading hinting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5385 from dachary/wip-12501-hammer

error in ext_mime_map_init() when /etc/mime.types is missing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5411 from SUSE/wip-12446-hammer

ceph.spec.in: radosgw requires apache for SUSE only -- makes no sense

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5412 from SUSE/wip-12448-hammer

ceph.spec.in: useless %py_requires breaks SLE11-SP3 build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5318 from ceph/wip-12236-hammer

Possible crash while concurrently writing and shrinking an image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5319 from ceph/wip-12235-hammer

librbd: deadlock during cooperative exclusive lock transition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5296 from ceph/wip-12345-hammer

librbd: correct issues discovered via lockdep / helgrind

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5559 from ceph/wip-12682-hammer

object_map_update fails with -EINVAL return code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5468 from ceph/wip-12577-hammer

osd: fix repair when recorded digest is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5376 from dachary/wip-12433-hammer

Show osd as NONE in ceph osd map <pool> <object>  output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5359 from ceph/wip-11470.hammer

mon: PaxosService: call post_refresh() instead of post_paxos_update()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5691 from jdurgin/wip-10399-hammer

is_new_interval() fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Objecter: pg_interval_t::is_new_interval needs pgid from previous pool

When increasing the pg_num of a pool, an assert would fail since the
calculated pgid seed would be for the pool's new pg_num value instead
of the previous pg_num value.

Fixes: #10399
Backport: infernalis, hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f20f7a23e913d09cc7fc22fb3df07f9938ddc144)

Conflicts: (hobject_t sort order not backported, trivial resolution)
src/osdc/Objecter.cc
src/osdc/Objecter.h

osd_types::is_new_interval: size change triggers new interval

Fixes: 11771
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit ff79959c037a7145f7104b06d9e6a64492fdb95f)

Merge remote-tracking branch 'gh/wip-12536-hammer' into hammer

This includes wip-osd-compat-hammer.

0.94.3

Merge remote-tracking branch 'gh/wip-osd-compat-hammer' into wip-12536-hammer

Merge pull request #5589 from ceph/hammer-12709

Workunits : fs/misc/chmod.sh : Include ACL characters in permission check

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Workunits : fs/misc/chmod.sh : Include ACL characters in permission check.

Signed-off-by: Yazen Ghannam <yazen.ghannam@linaro.org>
(cherry picked from commit d3dbfffefb0ae53583350f53258dc902670da659)

tests: increase test coverage for partial encodes/decodes

Multiple combinations of offsets/lengths are now tested when
performing partial encodes/decodes of the bit vector.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3e145f714ac9b2d599b45a058c6b93595e38f424)

common: bit_vector extent calculation incorrect for last page

It's highly probable that the last page in the bit vector will not
be a full page size.  As a result, the computed extents will extend
beyond the data portion of the bit vector, resulting in a end_of_buffer
exception.

Fixes: #12611
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c6d98992691683524d3d96def83a90a6f5fe7f93)

osd/OSDMap: handle incrementals that modify+del pool

Because new_pools was processed after old_pools, if something
was modified and then deleted in the same incremental
map, then the resulting state would be a pool in the map
that had no entry in pool_name.

Fixes: #12429
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8d4932e721e32a33c1bada54e11d4ecccd868618)

rgw: set http status in civetweb

Need to set the http status in civetweb so that we report it correctly.
Fixes: #12432
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b8e28ab9f914bf48c9ba4f0def9a0deb9dbb93bc)

civetweb: update submodule to support setting of http status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

hobject_t: fix get_boundary to work with new sorting regime

The hash is no longer the most significant field; set everything that is
more significant, too.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 45281efad1ae6b2b5ddd5549c56dbdd46671e5c5)

mon: disallow post-hammer OSDs if there are up pre-hammer OSDs

Force *all* OSDs to upgrade to hammer before allowing post-hammer
OSDs to join.  This prevents any pre-hammer OSDs from running at
the same time as a post-hammer OSD.

This commit, as well as the definition of the sentinal post-hammer
feature, should get backported to hammer stable series.

Backport: hammer
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e38d60e94503c7e8511a60a8dceceb3cd39c2bd8)

# Conflicts:
# src/mon/OSDMonitor.cc

include/ceph_features: define MON_METADATA feature

This was added in master in 40307748b9b8f84c98354a7141da645a3c556ba9.  We
define it but notably do not support it!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: include newlines in scrub errors

We may log more than one of these in a scrubmap; make sure they are
not concatenated.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2b3dd1bb7ab360e8f467a47b7ce3a0e4d3a979b2)

osd: fix condition for loggin scrub errors

We may have an error to log even though the authoritative map is empty:
when we have a digest that doesn't match all replicas.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d23766be902080ffb29528acb5a00d70beb97a9f)

osd: fix fallback logic; move into be_select_auth_object

The fallback behavior was printing a promising message but then doing
a 'continue' and not actually scrubbing the object.  Instead, fall back to
a less-bad auth inside be_select_auth_object.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b597db59fedc9a96dbe4b37f03b819dcf8fdb1bb)

osd: log a scrub error when we can't pick an auth object

If we really can't find a suitable auth, log an error.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a476d8f3903ca78b5cf36d63059f13cb5268efe5)

osd: repair record digest if all replicas match but do not match

If the recorded digest does not match the replicas, and all replicas
match, then (on repair) fix the recorded digest.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f581fecd8a16b9250fa96beec5aa26074f401028)

osd: move recorded vs on disk digest warning into be_compare_scrubmaps

This is a better place for it.  While we are here, change the wording to
clearly call out the 'recorded' digest (from object_info_t) vs the 'on
disk' digest (what we observed during scrub).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cf349ff35e9111ca52db1b0aad3753d0680b486d)

osd: be slightly paranoid about value of okseed

The scrubber.seed value is set based on the peer feature; check for that
too explicitly, and assert the scrubber.seed value matches.

No change in behavior here.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e34d31b7a9a71132a17b95cf5ad60255ece60243)

osd: be precise about "known" vs "best guess"

We cannot assume that the auth info has the "known" digest; all replicas
may have a matching digest that does not match the oi, or we may choose
a different auth info for some other reason.  Verify that the digest
matches the oi before calling it "known".

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 546d1c7b8224d96d0cc93dc3c96f52787cd69413)

osd: record digest if object is clean (vs entire scrub chunk)

If an individual object is clean, record it in the missing_digest map.
If not, leave it out.  This lets us drop the old condition that only
recorded any digests if the entire scrub chunk was clean.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c39e0e5f126028f552d410ab6924cac6e1f714fb)

Merge pull request #5160 from theanalyst/wip-11910-hammer

mon: pg ls is broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5384 from dachary/wip-12502-hammer

rgw: need conversion tool to handle fixes following #11974

Reviewed-by: Yehuda Sadeh <ysadehwe@redhat.com>

Merge pull request #5405 from ceph/wip-12465-hammer

Log::reopen_log_file: take m_flush_mutex

Reviewed-by: Loic Dachary <ldachary@redhat.com>

hobject_t: decode future hobject_t::get_min() properly

The post-hammer wip-temp branch changed hobject_t::get_min() so that pool
is INT64_MIN instead of -1 and neglected to deal with the encoding compat
with older versions.  Compensate on hammer by mapping INT64_MIN to -1
locally.  See commit ff99af38df830e215359bfb8837bf310a2023a4d.

Note that this means upgrades from hammer to post-hammer *must* include
this fix prior to the upgrade.  This will need to be well-documented in the
release notes.

Master gets a similar fix so that they map our min value to the new
INT64_MIN one on decode.

Fixes: #12536 (for hammer)
Signed-off-by: Sage Weil <sage@redhat.com>

OSDMonitor::preprocess_get_osdmap: send the last map as well

Fixes: #12410
Backport: hammer
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1320e29dfaee9995409a6d99b9ccaa748dc67b90)