Merge pull request #5769 from dachary/wip-12850-hammer

Crash during TestInternal.MultipleResize

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5768 from dachary/wip-12849-hammer

[  FAILED  ] TestLibRBD.BlockingAIO

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5697 from tchaikov/wip-12638-hammer

mon: add a cache layer over MonitorDBStore

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5381 from dachary/wip-12499-hammer

ceph-fuse 0.94.2-1trusty segfaults / aborts

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5757 from dachary/wip-12836-hammer

WBThrottle::clear_object: signal on cond when we reduce throttle values

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5759 from dachary/wip-12841-hammer

recursive lock of md_config_t (0)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5761 from dachary/wip-12843-hammer

long standing slow requests: connection->session->waiting_for_map->connection ref cycle

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5762 from dachary/wip-12844-hammer

osd suicide timeout during peering - search for missing objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5763 from dachary/wip-12846-hammer

osd/PGLog.cc: 732: FAILED assert(log.log.size() == log_keys_debug.size())

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5764 from dachary/wip-12847-hammer

common: do not insert emtpy ptr when rebuild emtpy bufferlist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5373 from dachary/wip-12489-hammer

pg_interval_t::check_new_interval - for ec pool, should not rely on min_size to determine if the PG was active at the interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5383 from dachary/wip-12504-hammer

rest-bench common/WorkQueue.cc: 54: FAILED assert(_threads.empty())

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5765 from dachary/wip-12883-hammer

cache agent is idle although one object is left in the cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5754 from dachary/wip-12588-hammer

Change radosgw pools default crush ruleset

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5377 from dachary/wip-12396-hammer

register_new_pgs() should check ruleno instead of its index

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5758 from dachary/wip-12839-hammer

Mutex Assert from PipeConnection::try_get_pipe

tests: tiering agent and proxy read

Verify that an object promoted to a cache tier because of a proxy read
is evicted as expected.

http://tracker.ceph.com/issues/12673 Refs: #12673

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 7924231930732bd297d3bd034c8295e96cb81088)

osd: trigger the cache agent after a promotion

When a proxy read happens, the object promotion is done in parallel. The
agent_choose_mode function must be called to reconsider the situation
to protect against the following scenario:

  * proxy read
  * agent_choose_mode finds no object exists and the agent
    goes idle
  * object promotion happens
  * the agent does not reconsider and eviction does not happen
    although it should

http://tracker.ceph.com/issues/12673 Fixes: #12673

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit e1f58feb9b1d20b72f2eb2eefdea5982e0cddccd)

Merge pull request #5276 from dachary/wip-11824-hammer

implicit erasure code crush ruleset is not validated

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5382 from dachary/wip-12500-hammer

segfault launching ceph-fuse with bad --name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5367 from dachary/wip-12311-hammer

read on chunk-aligned xattr not handled

Reviewed-by: Sage Weil <sage@redhat.com>

lockdep: allow lockdep to be dynamically enabled/disabled

librbd test cases attempt to enable lockdep coverage via the librados
API.  Use a configuration observer to register/unregister lockdep
support.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit af0cade0293be22e74a3293765676c7b227b06c4)

tests: librbd API test cannot use private md_config_t struct

Remove all depencencies on md_config_t and instead use librados API
methods to get/set configuration values.

Fixes: #12479
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c51aadb447d3447b5dace19c5cb6268e94247c8)

tests: ensure old-format RBD tests still work

Override the RBD default image format back to version 1
to ensure tests properly cover the old format.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b95edb5bd373bd05f83516eccfd30a6b5cf4693)

librados_test_stub: implement conf get/set API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4d03c664f2a9362009c7e3077ed65e2989f0f21b)

crypto: use NSS_InitContext/NSS_ShutdownContex to avoid memory leak

Switched to context-aware NSS init/shutdown functions to avoid conflicts
with parent application.  Use a reference counter to properly shutdown the
NSS crypto library when the last CephContext is destroyed.  This avoids
memory leaks with the NSS library from users of librados.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9fa0112dd9286178da1d76020158503b2062d252)

auth: use crypto_init_mutex to protect NSS_Shutdown()

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit cf058174fe6be26129eb52f4fb24d0f7146ef7af)

auth: reinitialize NSS modules after fork()

Fixes: #11128
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 4c24d0cc074462ae258b5cf901cd884bb4f50a53)

librbd: prevent race condition between resize requests

It was possible that the same resize request could be sent twice
if a completed resize op started a newly created resize op while
it was also being concurrently started by another thread.

Fixes: #12664
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6ec431bac1cf9446f3af90209d193dfcf003f2c5)

Merge pull request #5223 from SUSE/wip-12305-hammer

ceph.spec.in: running fdupes unnecessarily

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5716 from dachary/wip-12851-hammer

Ensure that swift keys don't include backslashes

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5717 from dachary/wip-12591-hammer

rgw: create a tool for orphaned objects cleanup

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5755 from dachary/wip-12589-hammer

ceph-disk zap should ensure block device

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5732 from ceph/wip-11455-hammer

rgw: init some manifest fields when handling explicit objs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5721 from dachary/wip-12853-hammer

RGW Swift API: X-Trans-Id header is wrongly formatted

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #5498 from ceph/wip-12432-hammer

rgw: HTTP return code is not being logged by CivetWeb

Reviewed-by: Loic Dachary <ldachary@redhat.com>

OSD: break connection->session->waiting message->connection cycle

Fixes: #12338
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 7f15e22be291df139c85909b96b2366f6dbadf37)

osd/PGLog: dirty_to is inclusive

There are only two callers of mark_dirty_to who do not pass max,
and they are both in the merge_log extending tail path.  In that
case, we want to include the last version specified in the log
writeout.  Fix the tail extending code to always specify the
last entry added, inclusive.

Fixes: #12652
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f0ca14df0641daa04eee39d98d8bd0faf46e4e6d)

common: fix code format

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
(cherry picked from commit 15e5ebe14787bbbc6b906d72ccd2d418d762cfad)

test: add test case for insert empty ptr when buffer rebuild

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
(cherry picked from commit 2d2f0eb338906742d516a2f8e6d5d05505be8f47)

common: fix insert empty ptr when bufferlist rebuild

Fixes: #12775
Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
(cherry picked from commit fb1b6ddd7bfe40d4a38d5ec59b1c3f59f2a83f23)

Conflicts:
src/common/buffer.cc : because invalidate_crc did not exist
        in hammer

Merge pull request #5527 from SUSE/wip-12585-hammer

OSD crash creating/deleting pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: copy the RecoveryCtx::handle when creating a new RecoveryCtx instance from another one

Fixes: 12523
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>
(cherry picked from commit 6ab9efe799b947f2ef6113df8cc542c73760107d)

config: skip lockdep for intentionally recursive md_config_t lock

lockdep can't handle recursive locks, resulting in false positive
reports for certain set_val_or_die() calls, like via
md_config_t::parse_argv() passed "-m".

Fixes: #12614
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 415226995ebe841cc9e8639a3233150a75aecaed)

osd: Keep a reference count on Connection while calling send_message()

Fixes: #12437
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a140085f467889f2743294a3c150f13b62fcdf51)

WBThrottle::clear_object: signal if we cleared an object

Fixes: #12223
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 8447b08d60d5cf7dce20f8b839cc2cb5a93782de)

ceph-disk: always check zap is applied on a full device

zap only makes sense on a full device and this is verified for

    ceph-disk prepare --zap-disk

but not for

    ceph-disk zap

Unify the two and add a test to check that it fails as expected when
trying to zap a directory.

http://tracker.ceph.com/issues/11272 Fixes: #11272

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 8e1bcf21f090fc98ac9b495d76614e3012f5872a)

Conflicts:
src/test/ceph-disk.sh : trivial resolution

librados: Make librados pool_create respect default_crush_ruleset

Make pool_create in librados use the osd_pool_default_crush_replicated_ruleset

Fixes: #11640
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
(cherry picked from commit da96a89033590277460aef1c80f385bd93d625e1)

Merge pull request #5551 from ceph/wip-corpus-hammer

ceph-object-corpus: add 0.94.2-207-g88e7ee7 hammer objects

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-object-corpus: add 0.94.2-207-g88e7ee7 hammer objects

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: init some manifest fields when handling explicit objs

Fixes: #11455
When dealing with old manifest that has explicit objs, we also
need to set the head size and head object correctly so that
code that relies on this info doesn't break.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d7bf8cb594e5276d1c80544f5ec954d52b159750)

mon: test the crush ruleset when creating a pool

We want to fix the following scenario:

 * an erasure code plugin (or another part of the code) creates a
   ruleset
 * the ruleset crashes during mapping (for whatever reason)
 * ceph osd pool create uses the bugous ruleset
 * the monitors try to do mapping a crash

Having a bugous ruleset in the crush map is very difficult prevent. The
catastrophic event of using it with a newly created pool can however be
prevented by calling the CrushTester just before creating the pool and
after all implicit or explicit crush ruleset creation happened.

http://tracker.ceph.com/issues/11814 Fixes: #11814

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit f1e86be589803596e86acc964ac5c5c03b4038d8)

Conflicts:
src/test/mon/osd-crush.sh
          removed the run_mon because in hammer it is shared
          between all tests
        src/mon/OSDMonitor.cc
          prepare_new_pool changed stringstream to *ostream

erasure-code: set max_size to chunk_count() instead of 20 for shec

See 8b64fe93b088a3a33d357869c47e6bf928c3f0e4 for a detailed explanation.

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit a44499f7e6609d9a32254dc6f4303122de74db31)

vstart.sh: set PATH to include pwd

ceph-mon needs crushtool to be in PATH. Don't set if it is run
from ceph_vstart_wrapper, which already sets it as it needs.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit e08bccf377292d72463796ae1dff5d8be40d88e9)

Merge pull request #5365 from dachary/wip-12491-hammer

buffer: critical bufferlist::zero bug

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5369 from dachary/wip-12498-hammer

get pools health'info have error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5370 from dachary/wip-12496-hammer

pgmonitor: wrong at/near target max“ reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5378 from dachary/wip-12394-hammer

Memory leak in Mutex.cc, pthread_mutexattr_init without pthread_mutexattr_destroy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5372 from dachary/wip-12493-hammer

the output is wrong when runing ceph osd reweight

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: rework X-Trans-Id header to be conform with Swift API.

Fixes: #12108
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
(cherry picked from commit d1735a4e8ea2056d333a5001a7a410fc03fe1b9e)

Transaction Id added in response

Signed-off-by: Abhishek Dixit dixitabhi@gmail.com
(cherry picked from commit b711e3124f8f73c17ebd19b38807a1b77f201e44)

Conflicts:
src/rgw/rgw_rest.cc
             the prototype of the end_header( function
             in the context of the diff changed

rgw: api adjustment following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7cf1f376e7dbc98a44f4a8e78bfa11f84291a941)

rgw: orphans, fix check on number of shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 47edec359be2a4d230a4a54c0c670d0aa711fa1e)

rgw: orphans, change default number of shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5528f21d0861d42378a8eb10759a1e55ddd66c09)

rgw: change error output related to orphans

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit cac57caf639236adc9a641a85352eb729b58725a)

rgw: orphan, fix truncated detection

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d8ce04db8542dae0027423f3c50355e5de74dc9a)

radosgw-admin: simplify orphan command

No need for --init-search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4980cbd04ff650d79a033bc5298e6c1abe985133)

radosgw-admin: stat orphan objects before reporting leakage

We don't want to report new objects as leaked, because they might just
be written, and waiting to be linked to a new object. The number of
seconds prior to the job initialization can be cofigured through
--orphan-stale-secs (by default it's 24h).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 66b0090023859d97da2bbaca698b6a50c225abd0)

radosgw-admin: orphans finish command

A command to remove orphans data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 55d6f5ecf18f532c7f056f8b60c101843594b00c)

rgw: cannot re-init an orphan scan job

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f8993102731518153bb18eb74c3e6e4943a5dbac)

rgw: stat_async() sets the object locator appropriately

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d7d117121858155acc8548908e9459a48c676dd1)

rgw: list_objects() sets namespace appropriately

list_objects() now uses parse_raw_oid(), so that it can set the correct
namespace. It only affects users of the function that want to get all
objects in bucket, regardless to the namespace associated with it. This
makes it so that the orphan code actually works now with namespaced
objects, and with special named objects (namely, start with underscore).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 20bd490bebf9be3553d4e25322c9544b0b090086)

rgw: modify orphan search fingerprints

Now works with multipart uploads.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit fa61ea2ccc94f04afb39868ac34abc264806d8d4)

rgw: compare oids and dump leaked objects

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1bc63d98ff8a3aa180065153690b4f8a73658b79)

rgw: keep accurate state for linked objects orphan scan

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f19b2f087cfb5a89edf3b19487c0df3a214e350f)

rgw: iterate over linked objects, store them

only keep part of the oid name if it is in a namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 75902fdb12ce0413d3beb455df47134f3d7386e7)

rgw: add rgw_obj::parse_raw_oid()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7c1aa8373a090d5c1b2aa121582547723120a079)

rgw: iterate asynchronously over linked objects

Read objects manifest. So that we could keep the relevant info later.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 05a953d965a0a6a0e9feddaa62f7429e154e828e)

rgw: async object stat functionality

An async functionality that stats object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 13adf3cf4f5fd8a0feb7cc1d7a4ee4ded9f573db)

rgw-admin: build index of bucket indexes

for the orphan search tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 16a2dbd838b69fb445c06122a68fc65ec3a7b3de)

rgw: initial work of orphan detection tool implementation

So far doesn't do much, iterate through all objects in a specific pool
data, store it in a sharded index.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 767fc29aa62a0a4e4dd8a78785576e8f1b44ee5b)

Avoid an extra read on the atomic variable

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
(cherry picked from commit 7476e944af049aefdb8871cce62883b86f6b6f8d)

RGW: Make RADOS handles in RGW to be a configurable option

Having a configurable number of RADOS handles is resulting
in significant performance boost for all types of workloads.
Each RGW worker thread would now get to pick a RADOS handle
for its lifetime, from the available bunch.

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
(cherry picked from commit b3c3a24fe9c0fbb2bdbbccb08b40efc1dca207cc)

Merge pull request #5374 from dachary/wip-12487-hammer

ceph osd crush reweight-subtree does not reweight parent node

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5380 from dachary/wip-12390-hammer

PGLog::proc_replica_log: correctly handle case where entries between olog.head and log.tail were split out

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: avoid using slashes for generated secret keys

Just use plain alphanumeric characterset.

Fixes: #7647
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 3f04a6126fdbfb93304f798da3775c0eec9b7d44)

Merge pull request #5366 from dachary/wip-12303-hammer

arm: all programs that link to librados2 hang forever on startup

Merge pull request #5202 from SUSE/wip-12293-hammer

ceph.spec.in: rpm: cephfs_java not fully conditionalized

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5203 from SUSE/wip-11997-hammer

 ceph.spec.in: rpm: not possible to turn off Java

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5204 from SUSE/wip-11629-hammer

ceph.spec.in: SUSE/openSUSE builds need libbz2-devel

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5207 from SUSE/wip-12267-hammer

ceph.spec.in: 50-rbd.rules conditional is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5216 from SUSE/wip-12269-hammer

ceph.spec.in: ceph-common needs python-argparse on older distros, but doesn't require it

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5264 from SUSE/wip-12361-hammer

ceph.spec.in: snappy-devel for all supported distros

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5368 from dachary/wip-12331-hammer

ceph: cli throws exception on unrecognized errno

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5371 from dachary/wip-12494-hammer

ceph tell: broken error message / misleading hinting

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5385 from dachary/wip-12501-hammer

error in ext_mime_map_init() when /etc/mime.types is missing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5411 from SUSE/wip-12446-hammer

ceph.spec.in: radosgw requires apache for SUSE only -- makes no sense

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5412 from SUSE/wip-12448-hammer

ceph.spec.in: useless %py_requires breaks SLE11-SP3 build

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5318 from ceph/wip-12236-hammer

Possible crash while concurrently writing and shrinking an image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5319 from ceph/wip-12235-hammer

librbd: deadlock during cooperative exclusive lock transition

Reviewed-by: Loic Dachary <ldachary@redhat.com>