Merge pull request #6055 from ktdreyer/wip-giant-13162

doc: rm another reference to Ceph Extras

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: rm another reference to Ceph Extras

http://tracker.ceph.com/issues/13162 Refs: #13162

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5983 from ceph/wip-13162

remove references to ceph.com repos

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: remove ceph-extras from vm-cloud

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update references to packages.ceph and git.ceph

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rados/test-upgarde-v9.0.1: fix exclude syntax

It's -, then a list of all exclusions separated by :.  There are just 2.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 78d894a634d727a9367f809a1f57234e5e6935be)

qa/workunits/rados/test-upgrade-v9.0.1: skip one more evict test

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e8d60a80ce31860eac76a1f6489a35e1795a0c0)

qa: add compatibility filtered rados api tests for upgrades

Post-9.0.1, the evict op returns success when an object doesn't exist
in the cache tier. Skip the tests that are incompatible across
versions.

Fixes: #11548
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 348a3d3c9880e7d022e71a2faafe51c8f771406e)

Merge pull request #4865 from dachary/wip-11888-giant

rpm: unconditionally package rbd-replay-prep(8)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rpm: unconditionally package rbd-replay-prep(8)

The build system will conditionally build and install the
rbd-replay-prep utility based on the presence of lttng.

Commit 230c5b8e7f58e2e8be32c3f31c6abcc5bdabc5ca adjusted the RPM
packaging so that we only package and ship the rbd-replay-prep utility
and its man page when we're on a platform that has lttng.

The problem is that this is only half-correct. The build system
unconditionally installs the man page for rbd-replay-prep, regardless of
whether lttng was present or absent, and (by extension,) without regard
for whether we've built and installed the rbd-replay-prep binary or not.

We have two options: either the build system should be adjusted to
conditionally skip installing rbd-replay-prep(8), or else we should just
adjust the RPM packaging to always ship the man page. This commit does
the latter.

Backport: firefly
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 6a98dacde1d42f5c599feb53b45c3c5ee8515e51)

Conflicts:
ceph.spec.in
        because rbd-replay-many does not exist in giant

Added a "ceph hello world" for a simple check for ceph-deploy qa suite

Signed-off-by: Yuri Weinstein <yweinstein@redhat.com>
(cherry picked from commit 13abae186357f4e9bb40990a7a212f93ec2e1e79)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #4477 from dachary/wip-release-notes-giant

doc/release-notes: v0.87.2

Reviewed-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.87.2

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fd11e329937cc091758302c32a00bb78aecf9c7d)

Conflicts:
doc/release-notes.rst
        because giant release notes have never been backported

0.87.2

Merge pull request #4214 from dachary/wip-10430-giant

osd/osd_types.cc: 456: FAILED assert(m_seed < old_pg_num)

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4258 from ceph/wip-10643-v2

mon: MDSMonitor: additional backports for #10643

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4261 from ceph/wip-11303-giant

allow -L to disable lttng. Enable it by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

allow -L to disable lttng. Enable it by default

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: MDSMonitor: wait for osdmon to be writable when requesting proposal

Otherwise we may end up requesting the osdmon to propose while it is
mid-proposal.  We can't simply return EAGAIN to the user either because
then we would have to expect the user to be able to successfully race
with the whole cluster in finding a window in which 'mds fs new' command
would succeed -- which is not a realistic expectation.  Having the
command to osdmon()->wait_for_writable() guarantees that the command
will be added to a queue and that we will, eventually, tend to it.

Fixes: #9794
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 2ae1cba595d9c56a0a4c534b34fe25250e7eb2d5)

mon: MDSMonitor: have management_command() returning int instead of bool

We can more easily differentiate between errors we get out of this
function, which makes the code a bit more versatile and readable.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 0dd473cbad4f9ea403fe60badffdc6da4dd3aa3c)

osd: Get pgid ancestor from last_map when building past intervals

Fixed OSD::build_past_intervals_parallel() and PG::generate_past_intervals()

Fixes: #10430
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0c5b66da7a9ba516340d06d9e806beb9d1040d0e)

Merge pull request #4175 from wonzhq/objecter-timer-2

Objecter: failed assert(tick_event==NULL) at osdc/Objecter.cc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Objecter: failed assert(tick_event==NULL) at osdc/Objecter.cc

When the Objecter timer erases the tick_event from its events queue and
calls tick() to dispatch it, if the Objecter::rwlock is held by shutdown(),
it waits there to get the rwlock. However, inside the shutdown function,
it checks the tick_event and tries to cancel it. The cancel_event function
returns false since tick_event is already removed from the events queue. Thus
tick_event is not set to NULL in shutdown(). Later the tick function return
ealier and doesn't set tick_event to NULL as well. This leads to the assertion
failure.

This is a regression introduced by an incorrect conflict resolution when
d790833 was backported.

Fixes: #11183
Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4127 from dzafman/wip-11176-giant

ceph-objectstore-tool: Output only unsupported features when incomatible

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4097 from dachary/wip-10497-giant

librados: c api does not translate op flag

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4096 from dachary/wip-9617-giant

objecter shutdown races with msg dispatch

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4095 from dachary/wip-9675-giant

splitting a pool doesn't start when rule_id != ruleset_id

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4094 from dachary/wip-9891-giant

Assertion: os/DBObjectMap.cc: 1214: FAILED assert(0)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4093 from dachary/wip-9915-giant

osd: eviction logic reversed

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4092 from dachary/wip-9985-giant

osd: incorrect atime calculation

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4091 from dachary/wip-9986-giant

objecter: map epoch skipping broken

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4090 from dachary/wip-10059-giant

osd/ECBackend.cc: 876: FAILED assert(0)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4089 from dachary/wip-10080-giant

Pipe::connect() cause osd crash when osd reconnect to its peer

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4088 from dachary/wip-6003-giant

journal Unable to read past sequence 406 ...

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4082 from dachary/wip-10106-giant

rgw acl response should start with <?xml version=1.0 ?>

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4078 from dachary/wip-11157-giant

doc,tests: force checkout of submodules

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4077 from dachary/wip-10150-giant

osd/ReplicatedPG.cc: 10853: FAILED assert(r >= 0) (in _scan_range)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4076 from dachary/wip-10153-giant

Rados.shutdown() dies with Illegal instruction (core dumped)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4074 from dachary/wip-10220-giant

mon/Paxos.cc: 1033: FAILED assert(mon->is_leader())

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #3548 from ceph/wip-10643

mon: MDSMonitor: missing backports for giant

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4053 from dachary/wip-8011-giant

osd/ReplicatedPG.cc: 5244: FAILED assert(soid < scrubber.start || soid >= scrubber.end)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4052 from dachary/wip-10844-giant

mon: caps validation should rely on EntityName instead of entity_name_t

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4050 from dachary/wip-10817-giant

WorkQueue: make timeout when calling WaitInterval configurable

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4049 from dachary/wip-10787-giant

mon: OSDMonitor::map_cache is buggy, send_incremental is not conservative

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4048 from dachary/wip-10770-giant

rgw: pending bucket index operations are not cancelled correctly

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4046 from dachary/wip-10723-giant

rados python binding leaks Ioctx objects

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4044 from dachary/wip-10617-giant

osd: pgs for deleted pools don't finish getting removed if osd restarts

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4034 from dachary/wip-10475-giant

rgw: Swift API. Support for X-Remove-Container-Meta-{key} header.

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #4033 from dachary/wip-10471-giant

rgw: index swift keys appropriately

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

ceph-objectstore-tool: Output only unsupported features when incomatible

Fixes: #11176
Backport: firefly, giant

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5b23f5b5892b36fb7d06efc0d77e64a24ef6e8c9)

Merge pull request #3971 from ceph/giant-11053

mds: fix assertion caused by system clock backwards

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Swift API. Support for X-Remove-Container-Meta-{key} header.

Fixes: #10475
Backport: hammer, firefly
Reported-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>
(cherry picked from commit f67bfa24fd6f69c2fcc0987eba8b6b426dd78320)

Conflicts:
src/rgw/rgw_rest.h
        trivial merge: prototype of an unrelated function changed
        src/rgw/rgw_op.cc
        s/is_object_op/!(s->object == NULL)/

librados: Translate operation flags from C APIs

The operation flags in the public C API are a distinct enum
and need to be translated to Ceph OSD flags, like as happens in
the C++ API. It seems like the C enum and the C++ enum consciously
use the same values, so I reused the C++ translation function.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>
(cherry picked from commit 49d114f1fff90e5c0f206725a5eb82c0ba329376)

Objecter: check the 'initialized' atomic_t safely

shutdown() resets initialized to 0, but we can still receive messages
after this point, so fix message handlers to skip messages in this
case instead of asserting.

Also read initialized while holding Objecter::rwlock to avoid races
where e.g. handle_osd_map() checks initialized -> 1, continues,
shutdown() is called, sets initialized to 0, then handle_osd_map()
goes about its business and calls op_submit(), which would fail the
assert(initialized.read()) check. Similar races existed in other
message handlers which change Objecter state.

The Objecter is not destroyed until after its Messenger in
the MDS, OSD, and librados, so this should be safe.

Fixes: #9617
Backport: giant
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit e506f896a9217324ab7a7865989f4454562aed5f)

Conflicts:
src/osdc/Objecter.cc
        context changed: Objecter::tick() did not have
        assert(initialized.read())

Objecter: init with a constant of the correct type

Just a tiny cleanup.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 1feba200aae7d9a042cda705c3de8fba2fc82331)

CrushWrapper: pick a ruleset same as rule_id

Originally in the add_simple_ruleset funtion, the ruleset_id
is not reused but rule_id is reused. So after some add/remove
against rules, the newly created rule likely to have
ruleset!=rule_id.

We dont want this happen because we are trying to hold the constraint
that ruleset == rule_id.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>
(cherry picked from commit 78e84f34da83abf5a62ae97bb84ab70774b164a6)

DBObjectMap: lock header_lock on sync()

Otherwise, we can race with another thread updating state.seq
resulting in the old, smaller value getting persisted.  If there
is a crash at that time, we will reuse a sequence number, resulting
in an inconsistent node tree and bug #9891.

Fixes: 9891
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 2b63dd25fc1c73fa42e52e9ea4ab5a45dd9422a0)

Conflicts:
src/os/DBObjectMap.cc
        because we have state.v = 1; instead of state.v = 2;

osd: cache tiering: fix the atime logic of the eviction

Reported-by: Xinze Chi <xmdxcxz@gmail.com>
Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
(cherry picked from commit 622c5ac41707069ef8db92cb67c9185acf125d40)

osd/ReplicatedPG: fix compile error

From 1fef4c3d541cba360738437420ebfa2447d5802e.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4a9ad7dc2da6f4fa6a64235776a3f1d2799aef60)

Get the currently atime of the object in cache pool for eviction

Because if there are mutiple atime in agent_state for the same object, we should use the recently one.

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
(cherry picked from commit 1fef4c3d541cba360738437420ebfa2447d5802e)

osdc/Objecter: Fix a bug of dead looping in Objecter::handle_osd_map

If current map epoch is less than oldest epoch, current map epoch
should step up to oldest epoch.

Fixes: #9986
Signed-off-by: Ding Dinghua <dingdinghua85@gmail.com>
(cherry picked from commit e0166a23c2cf655bfb4cf873be021a14d9b9be27)

osdc/Objecter: e shouldn't be zero in Objecter::handle_osd_map

Signed-off-by: Ding Dinghua <dingdinghua85@gmail.com>
(cherry picked from commit 31c584c8ba022cd44fe2872d221f3026618cefab)

PG: always clear_primary_state on new interval, but only clear pg temp if not primary

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit f692bfe076b8ddb679c6d1a6ea78cc47f0876326)

PG: always clear_primary_state when leaving Primary

Otherwise, entries from the log collection process might leak into the next
epoch, where we might end up choosing a different authoritative log.  In this
case, it resulted in us not rolling back to log entries on one of the replicas
prior to trying to recover from an affected object due to the peer_missing not
being cleared.

Fixes: #10059
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit c87bde64dfccb5d6ee2877cc74c66fc064b1bcd7)

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

In the past (e229f8451d37913225c49481b2ce2896ca6788a2) we decided to disable
reset of lossless Pipes, because lossless peers resetting caused trouble and
they can't forget about each other. But they actually can: if mark_down()
is called.

I can't figure out how else we could forget about a remote endpoint, so I think
it's okay if we tell them we reset in order to clean up state. That's desirable
so that we don't get into strange situations with out-of-whack counters.

Fixes: #10080
Backport: giant, firefly, dumpling

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 8cd1fdd7a778eb84cb4d7161f73bc621cc394261)

FileJournal: fix journalq population in do_read_entry()

Fixes: 6003
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit bae1f3eaa09c4747b8bfc6fb5dc673aa6989b695)

Conflicts:
src/os/FileJournal.cc
        because reinterpret_cast was added near two hunks after firefly

rgw: flush xml header on get acl request

Fixes: #10106
Backport: firefly, giant

dump_start() updates the formatter with the appropriate prefix, however,
we never flushed the formatter.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit eb45f861343162e018968b8c56693a8c6f5b2cab)

doc,tests: force checkout of submodules

When updating submodules, always checkout even if the HEAD is the
desired commit hash (update --force) to avoid the following:

    * a directory gmock exists in hammer
    * a submodule gmock replaces the directory gmock in master
    * checkout master + submodule update : gmock/.git is created
    * checkout hammer : the gmock directory still contains the .git from
      master because it did not exist at the time and checkout won't
      remove untracked directories
    * checkout master + submodule update : git rev-parse HEAD is
      at the desired commit although the content of the gmock directory
      is from hammer

http://tracker.ceph.com/issues/11157 Fixes: #11157

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit dce6f288ad541fe7f0ef8374301cd712dd3bfa39)

common: do not unlock rwlock on destruction

According to pthread_rwlock_unlock(3p):

 Results are undefined if the read-write lock rwlock is not held
 by the calling thread.

and:

 https://sourceware.org/bugzilla/show_bug.cgi?id=17561

 Calling pthread_rwlock_unlock on an rwlock which is not locked
 is undefined.

calling pthread_rwlock_unlock on RWLock destruction could cause
an unknown behavior for two reasons:

- the lock is acquired by another thread (undefined)
- the lock is not acquired (undefined)

Moreover since glibc-2.20 calling pthread_rwlock_unlock on a
rwlock that is not locked results in a SIGILL that kills the
application.

This patch removes the pthread_rwlock_unlock call on destruction
and replaces it with an assertion to check that the RWLock is
not in use.

Any code that relied on the implicit release is now going to
break the assertion, e.g.:

 {
   RWLock l;
   l.get(for_write);
 } // implicit release, wrong.

Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
(cherry picked from commit cf2104d4d991361c53f6e2fea93b69de10cd654b)

mon: Paxos: reset accept timeout before submiting work to the store

Otherwise we may trigger the timeout while waiting for the work to be
committed to the store -- and it would only take the write to take a bit
longer than 10 seconds (default accept timeout).

We do wait for the work to be properly committed to the store before
extending the lease though.

Fixes: #10220
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 18534615f184ba56b441fd1d4242eb06debdfe13)

mon: MonitorDBStore: allow randomly injecting random delays on writes

Adds two new config options:

mon_inject_transaction_delay_probability : DOUBLE (0.0-1.0, default: 0.0)
mon_inject_transaction_delay_max : DOUBLE (seconds, default: 10.0)

If probability is set to a value greater than 0, just before applying
the transaction, the store will decide whether to inject a delay,
randomly choosing a value between 0 and the max.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit beaa04e4119765d5775a6c48fd072dd95c984e3b)

ShardedThreadPool: make wait timeout on empty queue configurable

Fixes: 10818
Backport: giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 7002f934e6664daa995ca0629c0ea3bae1c6bddf)

WorkQueue: make wait timeout on empty queue configurable

Fixes: 10817
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 5aa6f910843e98a05bfcabe6f29d612cf335edbf)

PGLog: include rollback_info_trimmed_to in (read|write)_log

Fixes: #10157
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1fe8b846641486cc294fe7e1d2450132c38d2dba)

mon: MonCap: take EntityName instead when expanding profiles

entity_name_t is tightly coupled to the messenger, while EntityName is
tied to auth.  When expanding profiles we want to tie the profile
expansion to the entity that was authenticated.  Otherwise we may incur
in weird behavior such as having caps validation failing because a given
client messenger inst does not match the auth entity it used.

e.g., running

ceph --name osd.0 config-key exists foo daemon-private/osd.X/foo

has entity_name_t 'client.12345' and EntityName 'osd.0'.  Using
entity_name_t during profile expansion would not allow the client access
to daemon-private/osd.X/foo (client.12345 != osd.X).

Fixes: #10844
Backport: firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 87544f68b88fb3dd17c519de3119a9ad9ab21dfb)

mon: Monitor: stash auth entity name in session

Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit ca8e1efc0be9bffcfbdce5593526d257aa498062)

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 9b26de3f3653d38dcdfc5b97874089f19d2a59d7)

Merge pull request #4042 from dachary/wip-10546-giant

ceph time check start round bug in monitor.cc

Merge pull request #4047 from dachary/wip-10762-giant

mon: osd gets marked down twice

Merge pull request #4041 from dachary/wip-10512-giant

osd: cancel_flush requeues blocked events after blocking event

Merge pull request #4031 from dachary/wip-10353-giant

crush: set_choose_tries = 100 for erasure code rulesets

Merge pull request #4029 from dachary/wip-9910-giant

msg/Pipe: discard delay queue before incoming queue

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4030 from dachary/wip-10351-giant

mount.ceph: avoid spurious error message

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #4028 from dachary/wip-10259-giant

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Merge pull request #4027 from dachary/wip-10257-giant

  mon: PGMonitor: several stats output error fixes

Merge pull request #3998 from dzafman/wip-10677

Fix ceph command manpage to match ceph -h (giant)

Reviewed-by: Xinxin Shu <xinxin.shu@intel.com>

Merge pull request #3921 from sponce/wip-11078-giant

Fix libstriprados::stat, use strtoll insdead of strtol

Merge pull request #3819 from tchaikov/giant-pg-leak-10421

osd: fix PG leak in SnapTrimWQ._clear()

Merge pull request #3771 from ceph/wip-10883-giant

osd: Fix FileJournal wrap to get header out first

Merge pull request #3637 from sponce/wip-10758-giant

Backport of pull request 3633 to giant : Fixed write_full behavior in libradosstriper

mon/OSDMonitor: do not trust small values in osd epoch cache

If the epoch cache says the osd has epoch 100 and the osd is asking for
epoch 200+, do not send it 100+.

Fixes: #10787
Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a5759e9b97107488a8508f36adf9ca1aba3fae07)

rgw: send appropriate op to cancel bucket index pending operation

Fixes: #10770
Backport: firefly, giant

Reported-by: baijiaruo <baijiaruo@126.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit dfee96e3aebcaeef18c721ab73f0460eba69f1c7)

Conflicts:
src/rgw/rgw_rados.cc
        resolved by manual s/ADD/CANCEL/

mon: ignore osd failures from before up_from

If the failure was generated for an instance of the OSD prior to when
it came up, ignore it.

This probably causes a fair bit of unnecessary flapping in the wild...

Backport: giant, firefly
Fixes: #10762
Reported-by: Dan van der Ster <dan@vanderster.com>
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 400ac237d35d0d1d53f240fea87e8483c0e2a7f5)

rados.py: keep reference to python callbacks

If we don't keep a reference to these, the librados aio calls will
segfault since the python-level callbacks will have been garbage
collected. Passing them to aio_create_completion() does not take a
reference to them. Keep a reference in the python Completion object
associated with the request, since they need the same lifetime.

This fixes a regression from 60b019f69aa0e39d276c669698c92fc890599f50.

Fixes: #10775
Backport: dumpling, firefly, giant
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 36d37aadbbbece28d70e827511f1a473d851463d)
(cherry picked from commit 5f1245e131e33a98572408c8223deed2c7cf7b75)

Fix memory leak in python rados bindings

A circular reference was inadvertently created when using the
CFUNCTYPE binding for callbacks for the asynchronous i/o callbacks.
This commit refactors the usage of the callbacks such that the
Ioctx object does not have a class reference to the callbacks.

Fixes: #10723
Backport: giant, firefly, dumpling
Signed-off-by: Billy Olsen <billy.olsen@gmail.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 60b019f69aa0e39d276c669698c92fc890599f50)

osd: do not ignore deleted pgs on startup

These need to get instantiated so that we can complete the removal process.

Fixes: #10617
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 879fd0c192f5d3c6afd36c2df359806ea95827b8)

mon: Monitor: fix timecheck rounds period

Fixes: #10546
Backports: dumpling?,firefly,giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
(cherry picked from commit 2e749599ac6e1060cf553b521761a93fafbf65bb)

osd: requeue blocked op before flush it was blocked on

If we have request A (say, cache-flush) that blocks things, and then
request B that gets blocked on it, and we have an interval change, then we
need to requeue B first, then A, so that the resulting queue will keep
A before B and preserve the order.

This was observed on this firefly run:

  ubuntu@teuthology:/a/sage-2015-01-09_21:43:43-rados-firefly-distro-basic-multi/694675

Backport: giant, firefly
Fixes: #10512
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 11bdfb4131ecac16d4a364d651c6cf5d1d28c702)

rgw: index swift keys appropriately

Fixes: #10471
Backport: firefly, giant

We need to index the swift keys by the full uid:subuser when decoding
the json representation, to keep it in line with how we store it when
creating it through other mechanism.

Reported-by: hemant burman <hemant.burman@gmail.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 478629bd2f3f32afbe6e93eaebb8a8fa01af356f)