Merge pull request #4077 from dachary/wip-10150-giant

osd/ReplicatedPG.cc: 10853: FAILED assert(r >= 0) (in _scan_range)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4076 from dachary/wip-10153-giant

Rados.shutdown() dies with Illegal instruction (core dumped)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4074 from dachary/wip-10220-giant

mon/Paxos.cc: 1033: FAILED assert(mon->is_leader())

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #3548 from ceph/wip-10643

mon: MDSMonitor: missing backports for giant

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4053 from dachary/wip-8011-giant

osd/ReplicatedPG.cc: 5244: FAILED assert(soid < scrubber.start || soid >= scrubber.end)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4052 from dachary/wip-10844-giant

mon: caps validation should rely on EntityName instead of entity_name_t

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4050 from dachary/wip-10817-giant

WorkQueue: make timeout when calling WaitInterval configurable

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4049 from dachary/wip-10787-giant

mon: OSDMonitor::map_cache is buggy, send_incremental is not conservative

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4048 from dachary/wip-10770-giant

rgw: pending bucket index operations are not cancelled correctly

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4046 from dachary/wip-10723-giant

rados python binding leaks Ioctx objects

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4044 from dachary/wip-10617-giant

osd: pgs for deleted pools don't finish getting removed if osd restarts

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4034 from dachary/wip-10475-giant

rgw: Swift API. Support for X-Remove-Container-Meta-{key} header.

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4033 from dachary/wip-10471-giant

rgw: index swift keys appropriately

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #3971 from ceph/giant-11053

mds: fix assertion caused by system clock backwards

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Swift API. Support for X-Remove-Container-Meta-{key} header.

Fixes: #10475
Backport: hammer, firefly
Reported-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>
(cherry picked from commit f67bfa24fd6f69c2fcc0987eba8b6b426dd78320)

Conflicts:
src/rgw/rgw_rest.h
        trivial merge: prototype of an unrelated function changed
        src/rgw/rgw_op.cc
        s/is_object_op/!(s->object == NULL)/

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit dce6f288ad541fe7f0ef8374301cd712dd3bfa39)

common: do not unlock rwlock on destruction

According to pthread_rwlock_unlock(3p):

 Results are undefined if the read-write lock rwlock is not held
 by the calling thread.

and:

 https://sourceware.org/bugzilla/show_bug.cgi?id=17561

 Calling pthread_rwlock_unlock on an rwlock which is not locked
 is undefined.

calling pthread_rwlock_unlock on RWLock destruction could cause
an unknown behavior for two reasons:

- the lock is acquired by another thread (undefined)
- the lock is not acquired (undefined)

Moreover since glibc-2.20 calling pthread_rwlock_unlock on a
rwlock that is not locked results in a SIGILL that kills the
application.

This patch removes the pthread_rwlock_unlock call on destruction
and replaces it with an assertion to check that the RWLock is
not in use.

Any code that relied on the implicit release is now going to
break the assertion, e.g.:

 {
   RWLock l;
   l.get(for_write);
 } // implicit release, wrong.

Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
(cherry picked from commit cf2104d4d991361c53f6e2fea93b69de10cd654b)

mon: Paxos: reset accept timeout before submiting work to the store

Otherwise we may trigger the timeout while waiting for the work to be
committed to the store -- and it would only take the write to take a bit
longer than 10 seconds (default accept timeout).

We do wait for the work to be properly committed to the store before
extending the lease though.

Fixes: #10220
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 18534615f184ba56b441fd1d4242eb06debdfe13)

mon: MonitorDBStore: allow randomly injecting random delays on writes

Adds two new config options:

mon_inject_transaction_delay_probability : DOUBLE (0.0-1.0, default: 0.0)
mon_inject_transaction_delay_max : DOUBLE (seconds, default: 10.0)

If probability is set to a value greater than 0, just before applying
the transaction, the store will decide whether to inject a delay,
randomly choosing a value between 0 and the max.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit beaa04e4119765d5775a6c48fd072dd95c984e3b)

ShardedThreadPool: make wait timeout on empty queue configurable

Fixes: 10818
Backport: giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 7002f934e6664daa995ca0629c0ea3bae1c6bddf)

WorkQueue: make wait timeout on empty queue configurable

Fixes: 10817
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 5aa6f910843e98a05bfcabe6f29d612cf335edbf)

PGLog: include rollback_info_trimmed_to in (read|write)_log

Fixes: #10157
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1fe8b846641486cc294fe7e1d2450132c38d2dba)

mon: MonCap: take EntityName instead when expanding profiles

entity_name_t is tightly coupled to the messenger, while EntityName is
tied to auth.  When expanding profiles we want to tie the profile
expansion to the entity that was authenticated.  Otherwise we may incur
in weird behavior such as having caps validation failing because a given
client messenger inst does not match the auth entity it used.

e.g., running

ceph --name osd.0 config-key exists foo daemon-private/osd.X/foo

has entity_name_t 'client.12345' and EntityName 'osd.0'.  Using
entity_name_t during profile expansion would not allow the client access
to daemon-private/osd.X/foo (client.12345 != osd.X).

Fixes: #10844
Backport: firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 87544f68b88fb3dd17c519de3119a9ad9ab21dfb)

mon: Monitor: stash auth entity name in session

Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit ca8e1efc0be9bffcfbdce5593526d257aa498062)

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 9b26de3f3653d38dcdfc5b97874089f19d2a59d7)

Merge pull request #4042 from dachary/wip-10546-giant

ceph time check start round bug in monitor.cc

Merge pull request #4047 from dachary/wip-10762-giant

mon: osd gets marked down twice

Merge pull request #4041 from dachary/wip-10512-giant

osd: cancel_flush requeues blocked events after blocking event

Merge pull request #4031 from dachary/wip-10353-giant

crush: set_choose_tries = 100 for erasure code rulesets

Merge pull request #4029 from dachary/wip-9910-giant

msg/Pipe: discard delay queue before incoming queue

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4030 from dachary/wip-10351-giant

mount.ceph: avoid spurious error message

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #4028 from dachary/wip-10259-giant

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Merge pull request #4027 from dachary/wip-10257-giant

  mon: PGMonitor: several stats output error fixes

Merge pull request #3998 from dzafman/wip-10677

Fix ceph command manpage to match ceph -h (giant)

Reviewed-by: Xinxin Shu <xinxin.shu@intel.com>

Merge pull request #3921 from sponce/wip-11078-giant

Fix libstriprados::stat, use strtoll insdead of strtol

Merge pull request #3819 from tchaikov/giant-pg-leak-10421

osd: fix PG leak in SnapTrimWQ._clear()

Merge pull request #3771 from ceph/wip-10883-giant

osd: Fix FileJournal wrap to get header out first

Merge pull request #3637 from sponce/wip-10758-giant

Backport of pull request 3633 to giant : Fixed write_full behavior in libradosstriper

mon/OSDMonitor: do not trust small values in osd epoch cache

If the epoch cache says the osd has epoch 100 and the osd is asking for
epoch 200+, do not send it 100+.

Fixes: #10787
Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a5759e9b97107488a8508f36adf9ca1aba3fae07)

rgw: send appropriate op to cancel bucket index pending operation

Fixes: #10770
Backport: firefly, giant

Reported-by: baijiaruo <baijiaruo@126.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit dfee96e3aebcaeef18c721ab73f0460eba69f1c7)

Conflicts:
src/rgw/rgw_rados.cc
        resolved by manual s/ADD/CANCEL/

mon: ignore osd failures from before up_from

If the failure was generated for an instance of the OSD prior to when
it came up, ignore it.

This probably causes a fair bit of unnecessary flapping in the wild...

Backport: giant, firefly
Fixes: #10762
Reported-by: Dan van der Ster <dan@vanderster.com>
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 400ac237d35d0d1d53f240fea87e8483c0e2a7f5)

rados.py: keep reference to python callbacks

If we don't keep a reference to these, the librados aio calls will
segfault since the python-level callbacks will have been garbage
collected. Passing them to aio_create_completion() does not take a
reference to them. Keep a reference in the python Completion object
associated with the request, since they need the same lifetime.

This fixes a regression from 60b019f69aa0e39d276c669698c92fc890599f50.

Fixes: #10775
Backport: dumpling, firefly, giant
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 36d37aadbbbece28d70e827511f1a473d851463d)
(cherry picked from commit 5f1245e131e33a98572408c8223deed2c7cf7b75)

Fix memory leak in python rados bindings

A circular reference was inadvertently created when using the
CFUNCTYPE binding for callbacks for the asynchronous i/o callbacks.
This commit refactors the usage of the callbacks such that the
Ioctx object does not have a class reference to the callbacks.

Fixes: #10723
Backport: giant, firefly, dumpling
Signed-off-by: Billy Olsen <billy.olsen@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 60b019f69aa0e39d276c669698c92fc890599f50)

osd: do not ignore deleted pgs on startup

These need to get instantiated so that we can complete the removal process.

Fixes: #10617
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 879fd0c192f5d3c6afd36c2df359806ea95827b8)

mon: Monitor: fix timecheck rounds period

Fixes: #10546
Backports: dumpling?,firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 2e749599ac6e1060cf553b521761a93fafbf65bb)

osd: requeue blocked op before flush it was blocked on

If we have request A (say, cache-flush) that blocks things, and then
request B that gets blocked on it, and we have an interval change, then we
need to requeue B first, then A, so that the resulting queue will keep
A before B and preserve the order.

This was observed on this firefly run:

  ubuntu@teuthology:/a/sage-2015-01-09_21:43:43-rados-firefly-distro-basic-multi/694675

Backport: giant, firefly
Fixes: #10512
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 11bdfb4131ecac16d4a364d651c6cf5d1d28c702)

rgw: index swift keys appropriately

Fixes: #10471
Backport: firefly, giant

We need to index the swift keys by the full uid:subuser when decoding
the json representation, to keep it in line with how we store it when
creating it through other mechanism.

Reported-by: hemant burman <hemant.burman@gmail.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 478629bd2f3f32afbe6e93eaebb8a8fa01af356f)

crush: set_choose_tries = 100 for erasure code rulesets

It is common for people to try to map 9 OSDs out of a 9 OSDs total ceph
cluster. The default tries (50) will frequently lead to bad mappings for
this use case. Changing it to 100 makes no significant CPU performance
difference, as tested manually by running crushtool on one million
mappings.

http://tracker.ceph.com/issues/10353 Fixes: #10353

Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 2f87ac807f3cc7ac55d9677d2051645bf5396a62)

mount.ceph: avoid spurious error message

/etc/mtab in most modern distributions is a symbol link to
/proc/self/mounts.

Fixes: #10351
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit bdd0e3c4bda97fe18487a58dd173a7dff752e1a2)

msg/Pipe: discard delay queue before incoming queue

Shutdown the delayed delivery before the incoming queue in case the
DelayedDelivery thread is busy queuing messages.

Fixes: #9910
Signed-off-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit f7431cc3c25878057482007beb874c9d4473883e)

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::operator==

Fixes: 10259
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1ac17c0a662e6079c2c57edde2b4dc947f547f57)

mon: PGMonitor: skip zeroed osd stats on get_rule_avail()

Fixes: #10257
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit b311e7c36273efae39aa2602c1f8bd90d39e5975)

mon: PGMonitor: available size 0 if no osds on pool's ruleset

get_rule_avail() may return < 0, which we were using blindly assuming it
would always return an unsigned value.  We would end up with weird
values if the ruleset had no osds.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 8be6a6ab2aa5a000a39c73a98b11a0ab32fffa1c)

mon: PGMonitor: fix division by zero on stats dump

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 50547dc3c00b7556e26b9a44ec68640c5c3a2384)

doc: Fix ceph command manpage to match ceph -h (giant)

Fixes: #10677
Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Minor fixes to ceph command manpage

Fixes: #10676
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7e85722fd4c89715fc2ed79697c82d65d7ebf287)

doc: Fix ceph command manpage to match ceph -h (firefly)

Improve synopsis section
Fixes: #10676
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9ac488c1eb0e30511079ba05aaf11c79615b3940)

Conflicts:

man/ceph.8 (took incoming version)

doc: Changes format style in ceph to improve readability as html.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>
(cherry picked from commit 8b796173063ac9af8c21364521fc5ee23d901196)

mds: fix assertion caused by system clock backwards

Fixes: #11053
Signed-off-by: Yan, Zheng <zyan@redhat.com>

doc: Adds man page for ceph.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>
(cherry picked from commit 76da87a64ca6b3cc0ceeaf63e19a9f440d6f4161)

Fix libstriprados::stat, use strtoll insdead of strtol

The return value(long int) of strict_strtol is too small for unstriped
object.

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>
(cherry picked from commit fe6679dca479fc24806d7e57ab0108a516cd6d55)

Fix libstriprados::remove, use strtoll insdead of strtol

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>
(cherry picked from commit 78a15ee4c61fdadccb1921e861748400cc651862)

Objecter::_op_submit_with_budget: add timeout before call

Objecter::_send_op depends on the ontimeout field being filled in
to avoid 10340 and 9582.

Fixes: 10340
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit cfcfafcb0f33994dbda1efe478ef3ab822ff50d4)

osd: fix PG leak in SnapTrimWQ._clear()

Fixes: #10421
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 01e154d592d6cdbf3f859cf1b4357e803536a6b4)

0.87.1

osd: Fix FileJournal wrap to get header out first

Correct and restore assert that was removed

Cause by f46b1b473fce0322a672b16c7739e569a45054b6
Fixes: #10883
Backport: dumpling, firefly, giant

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 970bb4901f93575709421b5b25c3eff213de61b8)

Merge pull request #3731 from liewegas/wip-10834-giant

osd: tolerate sessionless con in fast dispatch path

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: tolerate sessionless con in fast dispatch path

We can now get a session cleared from a Connection at any time.  Change
the assert to an if in ms_fast_dispatch to cope.  It's pretty rare, but it
can happen, especially with delay injection.  In particular, a racing
thread can call mark_down() on us.

Fixes: #10209
Backport: giant
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 01df2227125abf94571b4b0c7bccca57098ed2dc)

qa: use correct binary path on rpm-based systems

Fixes: #10715
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 05ce2aa1bf030ea225300b48e7914577a412b38c)

Merge pull request #3407 from ceph/wip-9854-giant

osdc: Constrain max number of in-flight read requests

Merge pull request #3568 from dachary/wip-10471-rgw-giant

rgw: use s->bucket_attrs instead of trying to read obj attrs

Merge pull request #3263 from dachary/wip-jerasure-giant

erasure-code: update links to jerasure upstream (giant)

Merge pull request #3191 from ceph/giant-10277

Giant 10277

Merge pull request #3186 from ceph/wip-giant-mon-backports

mon: backports for #9987 against giant

Merge pull request #3185 from dachary/wip-10325-cauchy-giant

erasure-code: relax cauchy w restrictions (giant)

Merge pull request #3178 from dachary/wip-9998-crush-underfloat-giant

crush: fix weight underfloat issue (giant)

Merge pull request #3579 from dachary/wip-9877-rgw-giant

rgw: RGWRados::get_obj() returns wrong len if len == 0

Merge pull request #3168 from ceph/wip-8797-giant

Wip 8797 giant

Merge pull request #3582 from dachary/wip-10062-rgw-giant

rgw: s3 keystone auth fixes

Merge pull request #3581 from dachary/wip-9918-rgw-giant

rgw: update swift subuser perm masks when authenticating

Merge pull request #3580 from dachary/wip-9907-rgw-giant

fix can not disable max_size quota

Merge pull request #3083 from dachary/wip-10211-erasure-code-buffer-alignement-giant

erasure-code: enforce chunk size alignment (giant)

Merge pull request #3577 from dachary/wip-9587-rgw-giant

init-radosgw.sysv: set ulimit -n before starting daemon

Merge pull request #3576 from dachary/wip-5595-9576-rgw-giant

update object content-length doesn't work correctly

Merge pull request #3575 from dachary/wip-9479-rgw-giant

rgw: send back ETag on S3 object copy

Merge pull request #3574 from dachary/wip-9478-rgw-giant

rgw: S3 object copy content type fix

Merge pull request #3573 from dachary/wip-9254-rgw-giant

rgw: http headers need to end with \r\n

Merge pull request #3572 from dachary/wip-9973-rgw-giant

rgw: remove swift user manifest (DLO) hash calculation

Merge pull request #3571 from dachary/wip-8911-rgw-giant

rgw: swift GET / HEAD object returns X-Timestamp field

Merge pull request #3570 from dachary/wip-10701-rgw-giant

rgw: use strict_strtoll() for content length

Merge pull request #3569 from dachary/wip-10103-rgw-giant

rgw-admin: create subuser if needed when creating user

Merge pull request #3567 from dachary/wip-10307-rgw-giant

rgw: use s->bucket_attrs instead of trying to read obj attrs

Merge pull request #3443 from ceph/wip-10590-giant

rbd: ensure aio_write buffer isn't invalidated during image import

Merge pull request #3557 from dachary/wip-10688-boost-157-giant

support Boost 1.57.0

Merge pull request #2954 from sponce/giant

Fixed trivial locking issue in the trunc method of libradosstriper - Giant branch

Merge pull request #3405 from ceph/wip-10299-giant

librbd: complete all pending aio ops prior to closing image

Merge pull request #3403 from ceph/wip-10270-giant

librbd: gracefully handle deleted/renamed pools

Merge pull request #3356 from liewegas/wip-msgr-giant

msgr: fast dispatch backports for giant

fsync-tester: print info about PATH and locations of lsof lookup

We're seeing the lsof invocation fail (as not found) in testing and nobody can
identify why. Since attempting to reproduce the issue has not worked, this
patch will gather data from a genuinely in-vitro location.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit a85051483874ff5b8b0fb50426a3577040457596)

libradosstriper: fixed write_full when ENOENT

Fixes: #10758
write_full was returning ENOENT when the file did not exists, while it should just have created it without complaining.

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>
(cherry picked from commit 6a91f2bb82035b3c8021a7dc7b23548ce3d61eb6)