Merge pull request #2242 from majianpeng/fix4

utime: Because class Clock didn't exist,so remove the declaration in class utime_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2246 from ceph/wip-9064

ReplicatedPG::maybe_handle_cache: do not skip promote for write_ordered

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::maybe_handle_cache: do not skip promote for write_ordered

We cannot redirect a RW ordered read.

Fixes: #9064
Introduced: 0ed3adc1e0a74bf9548d1d956aece11f019afee0
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2245 from dachary/wip-9085-isa-link

erasure-code: isa plugin must link with ErasureCode.cc

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: isa plugin must link with ErasureCode.cc

Otherwise it will not get the methods it needs. A test is added to check
the plugin loads as expected, from the command line. The test is not run
if the isa plugin is not found, which happens on platforms that are not
supported.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc/changelog: v0.67.10 notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-8860'

Reviewed-by: Sage Weil <sage@redhat.com>

init-ceph: conditionally update  after argparsing

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

doc/release-notes: v0.67.10

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Conflicts:
PendingReleaseNotes

Merge pull request #2243 from ceph/wip-log-client

common/LogClient: fix sending dup log items

Reviewed-by: Loic Dachary <loic@dachary.org>

common/LogClient: fix sending dup log items

We need to skip even the most recently sent item in order to get to the
ones we haven't sent yet.

Fixes: #9080
Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2232 from dachary/wip-9044-use-ruleset

erasure-code: ErasureCodeJerasure::create_ruleset must return a ruleset

Reviewed-by: Xiaoxi Chen <xiaoxi.chen@intel.com>
Reviewed-by: Ma Jianpeng <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

utime: Because class Clock didn't exist,so remove the declaration in class utime_t.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

doc: Replace [default] with [global].

Fixes: #8955
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added yum-priorities procedure to manual install.

Fixes: #8734
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added priority = 2 to packages.

Fixes: #8734
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

rgw: Don't send error body when it's a HEAD request

The main 'operation' know not to send any body for HEAD requests.

However for errors, this was not the case, the formatter would be
flushed and would send the error 'message' in the body in all cases.

For the FastCGI case it doesn't seem to be an issue, it's possible
that the webserver (apache/lighttpd/...) cleans up the response
into shape.

But when using the new civetweb frontend this cause invalid HTTP.

Backport: firefly
Fixes #8539

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Sylvain Munaut <s.munaut@whatever-company.com>

Merge pull request #2214 from ceph/wip-rados-df-json

rados: fix 'rados df --format=json' field names

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

rados: remove {read,write}_kb fields from rados df json output

Redundant.  Do not backport.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: fix {read,write}_bytes value

Fill this in with kb * 1024 instead of the ops count.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2238 from ceph/wip-9057

msg/Pipe: do not wait for self in Pipe::stop_and_wait()

Reviewed-by: Samuel Just <sam.just@inktank.com>

msg/Pipe: do not wait for self in Pipe::stop_and_wait()

The fast dispatch code necessitated adding a wait for the fast dispatch
to complete when taking over sockets back in commit
2d5d3097c3998add1061ce253104154d72879237.  This included mark_down()
(although I am not certain mark_down was required to fix the previous set
of races).

In any case, if the fast dispatch thread itself tries to mark down its
own connection, it will deadlock in this method waiting for itself to
return and clear reader_dispatching.  Skip this wait if we are in fact
the reader thread.  This avoids the deadlock.

Alternatively, we could change mark_down() to not use stop_and_wait(), but
I am less clear about the potential races there, so I'm opting for the
minimal (though ugly) fix.

Fixes: #9057
Signed-off-by: Sage Weil <sage@redhat.com>

osd: improve ruleno/ruleset consistency

rename variables to reflect when they hold a ruleset and when they hold
a rule id / ruleno in the hope to avoid bugs in the future.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: ErasureCodeIsa::create_ruleset must return a ruleset

http://tracker.ceph.com/issues/9044 Refs: #9044

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: ErasureCodeJerasure::create_ruleset must return a ruleset

CrushWrapper::add_simple_ruleset does not return a ruleset, it returns
a ruleid that must be converted into a ruleset before being returned.

http://tracker.ceph.com/issues/9044 Fixes: #9044

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: OSDMonitor::crush_ruleset_create_erasure needs ruleset

When OSDMonitor::crush_ruleset_create_erasure checks the ruleset for
existence, it must convert the ruleid into a ruleset before assigning it
back to the *ruleset parameter.

http://tracker.ceph.com/issues/9044 Fixes: #9044

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #2201 from dachary/wip-8496-erasure-code-base-class

erasure code base class

Reviewed-by: Andreas-Joachim Peters <Andreas.Joachim.Peters@cern.ch>

os/FileStore: dump open fds before asserting

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'next'

Conflicts:
src/osd/OSD.cc

OSD: add require_same_peer_inst(OpRequestRef&,OSDMap&) helper

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit e99acf9810976b1fc74b84ad289773af43be973f)

OSD: introduce require_self_aliveness(OpRequestRef&,epoch_t) function

Take the self-aliveness checks out of require_same_or_newer_map() and use
the new function for that and for require_up_osd_peer().

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit e179e9227b4a4482d8359682092fd7f426b9a919)

OSD: use OpRequestRef& for a few require_* functions

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit eb2f1ea2c33647934af878b504383829f5a198ce)

OSD: introduce require_up_osd_peer() function for gating replica ops

This checks both that a Message originates from an OSD, and that the OSD
is up in the given map epoch.
We use it in handle_replica_op so that we don't inadvertently add operations
from down peers, who might or might not know it.

Signed-off-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit ccd0eec50103b919b3eb6eea96f7dc6438520ed3)

Merge remote-tracking branch 'gh/next'

Merge pull request #2228 from ceph/wip-pg-epoch

osd: fix pg epoch floor tracking

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2209 from somnathr/wip-sd-check-pool-existence

RadosClient: Adding pool existence checks for pool create interfaces.

Reviewed-by: Sage Weil <sage@redhat.com>

RadosClient: Enable pool existence check

Pool create interfaces would now refer to updated osdmap
for pool existence, such that an appropriate message is
thrown to the user when run via CLI.

Fixes: #8835
Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

osd: fix pg epoch floor tracking

If you call erase() on a multiset it will delete all instances of a value;
we only want to delete one of them.  Fix this by passing an iterator.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2226 from athanatos/wip-8396

Wip 8396

Reviewed-by: Sage Weil <sage@redhat.com>

OSD: move waiting_for_pg into the session structures

Each message belongs to a session.  Further, no ordering is implied
between messages which arrived on different sessions.  Breaking the
global waiting_for_pg structure into a per-session structure lets
us avoid the problem of taking a write lock on a global structure
(pg_map_lock) in get_pg_or_queue_for_pg at the cost of some
complexity in updating each session's waiting_for_pg structure when
we receive a new map (due to pg splits) or when we locally create
a pg.

Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2227 from athanatos/wip-8625

osd_types: s/stashed/rollback_info_completed and set on create

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Removed Debian reference from Upstart.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Incorporated user feedback to clarify upgrade doc.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Added configuration discussion at end of gateway install with links to config.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

OSD::shutdown: actually drop sessions waiting on map

There might be messages for which we still don't have the
map.  Dispatching waiting won't actually help.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: clear_session_waiting_on_map in ms_handle_reset

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: rename session_waiting_for_map_lock to session_waiting_lock

This lock will also protect the waiting_for_pg structures in each
session.

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: wake_pg_waiters outside of the pgmap write_lock, pg_lock

Signed-off-by: Samuel Just <sam.just@inktank.com>

OSD: fix wake_pg_waiters revert error in _open_lock_pg

231fe1b685bfbd3db9c81709ca39a29d696b13ad reintroduced erroneously
this call to wake_pg_waiters.  All _create_lock_pg callers handle
calling wake_pg_waiters after the pg lock has been dropped.

Fixes: #8691
Signed-off-by: Samuel Just <sam.just@inktank.com>

osd_types: s/stashed/rollback_info_completed and set on create

Originally, this flag indicated that the object had already been stashed and
that therefore recording subsequent changes is unecessary.  We want to set it
on create() as well since operations like [create, writefull] should not need
to stash the object.

Fixes: #8625
Signed-off-by: Samuel Just <sam.just@inktank.com>

Merge remote-tracking branch 'gh/wip-filestore-bigxattr'

make ceph-disk use the new init flag for cluster

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

allow passing a --cluster flag to the init script

It will fallback to looking at /etc/ceph/$cluster.conf
otherwise.

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

powerdns: Update README with better markdown

Merge pull request #2220 from somnathr/wip-lock-leak-fix

RadosClient: Fixing potential lock leaks.

Backport: firefly
Reviewed-by: Sage Weil <sage@redhat.com>

RadosClient: Fixing potential lock leaks.

In lookup_pool and pool_delete, a lock is taken
before invoking wait_for_osdmap, but is not
released for the failure case of the call. Fixing the same.

Fixes: #9022
Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

os/FileStore: force any new xattr into omap on E2BIG

If we have a huge xattr (or many little ones), the _fgetattrs() for the
inline_set will fail with E2BIG.  The conditions later where we decide
whether to clean up the old xattr will then also fail.  We *will* put
the xattr in omap, but the non-omap version isn't cleaned up.

Fix this by setting a flag if we get E2BIG that the inline_set is known
to be incomplete.  In that case, take the conservative step of assuming
the xattr might be present and chain_fremovexattr().  Ignore any error
because it might not be there.

This is clearly harmless in the general case because it won't be there.
If it is, we will hopefully remove enough xattrs that the E2BIG
condition will go away (usually by removing some really big chained
xattr).

See original bug #7779.  With this in place, we can repair objects in
the broken state if we know the rados attr(s) that are responsible.
Usually that is user.rgw.manifset, and a rados get + set of the attr
will repair things.

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2218 from ceph/wip-rados-xattr

rados: fix get/setxattr commands up

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rados: use STD{IN,OUT}_FILENO for magic values

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_rados_tool: add a few xattr tests

Signed-off-by: Sage Weil <sage@redhat.com>

rados: optionally read setxattr value from stdin

Signed-off-by: Sage Weil <sage@redhat.com>

rados: don't add \n to getxattr

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: fix 'ceph df ...' tests

Broken by ee2dbdb0f5e54fe6f9c5999c032063b084424c4c and friends.

Signed-off-by: Sage Weil <sage@redhat.com>

use cluster name when checking the host

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

use  name in /var/lib/ceph paths

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

when starting an OSD, pass in the cluster name

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

PendingReleaseNotes: make note about rbd cache default change

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2123 from ceph/wip-rbd-flush

librbd: enable rbd cache by default; writethrough until flush

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2205 from ceph/wip-librbd-snap-meta

librbd: fix crash with a chain of flattened images

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: fix i386 build

Signed-off-by: Sage Weil <sage@redhat.com>

rados: add {read,write}_ops to df json

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/wip-test-ceph-disk'

Merge pull request #2215 from ceph/wip-kb

mon: clean up _kb fields in json and perf counter output

Reviewed-by: John Spray <john.spray@redhat.com>

mon/PGMonitor: remove {rd,wr}_kb from pool stat dumps

These fields are replaced with corresponding *_bytes fields.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove *_kb perf counters

This is an incompatible change.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: add _bytes perf counters

Leave the _kb ones in place for now.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: add _bytes fields for all usage dumps

Leave the _kb ones in place for now.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2165 from dachary/wip-mailmap

mailmap updates

README.md: word wrap

Signed-off-by: Sage Weil <sage@redhat.com>

README: symlink from README.md

It looks better as markdown than rendered as text via the markdown tool,
so just symlink it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2202 from xinxinsh/enable-rocksdb-log-level

Enable rocksdb log level

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: rework ErasureCode*::parse methods

The ErasureCode::parse virtual function is derived in
ErasureCode{Jerasure,Isa}. It is reworked to share parsing arguments the
various techniques have in common. The logic is otherwise unmodified.

http://tracker.ceph.com/issues/8496 Refs: #8496

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: move to ErasureCode::decode_concat

It does not belong to ErasureCodeInterface which is supposed to not have
any implementation.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: move to ErasureCode::to_{int,bool}

The parameter parser helpers to_int and to_bool are moved from
ErasureCode{Jerasure,Isa} to ErasureCode.

The prototype is modified to return a status instead of the value. An
error ostream is provided as the last argument because ErasureCode
cannot use dout() or derr().

http://tracker.ceph.com/issues/8496 Refs: #8496

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: move to ErasureCode::minimum_to_decode*

The ErasureCode{Jerasure,Isa}::minimum_to_decode and
ErasureCode{Jerasure,Isa}::minimum_to_decode_with_cost methods are moved
verbatim to the ErasureCode base class.

http://tracker.ceph.com/issues/8496 Refs: #8496

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: move to ErasureCode::{encode,decode}{,chunk}

The bulk of ErasureCode{Jerasure,Isa}::encode and
ErasureCode{Jerasure,Isa}::decode is dealing with alignment and buffer
allocation. This is moved to ErasureCode::encode and ErasureCode::decode
respectively.

The jerasure/isa specific code is isolated in the new
ErasureCode{Jerasure,Isa}::encode_chunks and
ErasureCode{Jerasure,Isa}::decode_chunks virtual functions.

http://tracker.ceph.com/issues/8496 Refs: #8496

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: ErasureCode base class

Add the ErasureCode base class for ErasureCodeJerasure and
ErasureCodeIsa, for methods that can be shared among all plugins.

http://tracker.ceph.com/issues/8496 Fixes: #8496

Signed-off-by: Loic Dachary <loic@dachary.org>

test/osd/osd-test-helpers: mkdir -p for ceph-disk

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2208 from lpabon/osd_dev_doc

Developer quick start guide

Reviewed-by: Sage Weil <sage@redhat.com>

test/ceph-disk.sh: mkdir -p

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'upstream/next' into wip-sam-testing

Conflicts:
src/osd/OSD.cc

Renamed README to README.md to render in markdown

Signed-off-by: Luis Pabón <lpabon@redhat.com>

Developer quick start guide

Signed-off-by: Luis Pabón <lpabon@redhat.com>

enable info_log_level config option for rocksdb

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #2206 from ceph/wip-8875

doc: be a bit more explicit about 'ceph-deploy new' in quickstart

Reviewed-by: John Wilkins <john.wilkins@inktank.com>

be a bit more explicit about 'ceph-deploy new' in quickstart

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge branch 'master' of github.com:ceph/ceph

Merge remote-tracking branch 'gh/wip-8880'

Conflicts:
src/osd/OSD.cc

Merge pull request #2204 from osynge/wip-dont-mkdir-by-mistake2

Do not make directories by mistake.

Reviewed-by: Alfredo Deza <adeza@redhat.com>