Merge pull request #2169 from ceph/wip-double-pc

mon: s/%%/%/

Realized where these came from; it was an accident.

Merge branch 'wip-cache-second'

Reviewed-by: Samuel Just <sam.just@inktank.com>

ceph_test_rados_api_tier: test promote-on-second-read behavior

Signed-off-by: Zhiqiang Wang <wonzhq@hotmail.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osd: promotion on 2nd read for cache tiering

Signed-off-by: Zhiqiang Wang <wonzhq@hotmail.com>

Merge pull request #2183 from majianpeng/master

ECBackend: Using ROUND_UP_TO to refactor function get_recovery_chunk_size()

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2175 from majianpeng/fix1

ReplicatedPG: For async-read, set the real result after completing read.

Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #2180 from ceph/wip-ec-isa

osd: add support for intel ISA-L EC library

Merge remote-tracking branch 'gh/next'

Merge pull request #2190 from ceph/wip-osd-leaks

osd: do not leak Session* ref in _send_boot()

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2182 from ceph/wip-round

use llrintl when converting double to micro

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge branch 'wip-rocksdb'

rocksdb: -Wno-portability

Signed-off-by: Sage Weil <sage@redhat.com>

autogen.sh: debug with -x

Signed-off-by: Sage Weil <sage@redhat.com>

debian, rpm: build rocksdb statically if we can

Signed-off-by: Sage Weil <sage@redhat.com>

debian, rpm: build require bzip2 dev

Signed-off-by: Sage Weil <sage@redhat.com>

add rocksdb bz2 dep

Signed-off-by: Sage Weil <sage@redhat.com>

configure: enable static rocksdb if check and c++11

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile: include rocksdb in dist tarball always

Signed-off-by: Sage Weil <sage@redhat.com>

configure: verify c++11 is present for rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: require less shiny autoconf

precise has 2.59

Signed-off-by: Sage Weil <sage@redhat.com>

do_autogen.sh: build with static librocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: update submodule makefile

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>

unittest_erasure_code_[plugin_]isa: conditionally compile

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not leak Session* ref in _send_boot()

The get_priv() call returns a ref; make sure we drop it if it exists.
This doesn't happen on every run because usually it is NULL and we take
the other path; it's only after the OSD has been marked down that we reach
the second path.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2157 from ceph/wip-8937

rgw: call processor->handle_data() again if needed

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code/isa: fix signed/unsigned comparison

warning: erasure-code/isa/ErasureCodeIsa.cc:313: comparison between signed and unsigned integer expressions

Signed-off-by: Sage Weil <sage@redhat.com>

erasure-code/isa: don't use typename outside of template

error: erasure-code/isa/ErasureCodeIsa.h:141: using 'typename' outside of template

Signed-off-by: Sage Weil <sage@redhat.com>

ECBackend: Using ROUND_UP_TO to refactor function get_recovery_chunk_size()

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

ReplicatedPG: For async-read, set the real result after completing read.

When reading an object from replicated pool, ceph uses sync mode,
so it can set the results in execute_ctx correctly.
However, For the async-read in EC Pool, current code didn't set the
real results after read in complete_read_ctx.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

doc/release-notes: tweak quoting

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/release-notes.rst:22: Unknown target name: "leveldb".

Signed-off-by: Sage Weil <sage@redhat.com>

use llrintl when converting double to micro

This avoids rounding error (noticeable on i386).

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

configure: check for 'better' yasm that can build ISA-L

Signed-off-by: Sage Weil <sage@redhat.com>

erasure-code/isa/Makefile: add missing \, and missing headers

Signed-off-by: Sage Weil <sage@redhat.com>

yasm-wrapper: fix -f ...

Signed-off-by: Sage Weil <sage@redhat.com>

yasm-wrapper: turn -I foo into -i foo

Signed-off-by: Sage Weil <sage@redhat.com>

yasm-wrapper: echo original args

Signed-off-by: Sage Weil <sage@redhat.com>

EC: add plugin for Intel ISA-L library

Merge pull request #2179 from ceph/wip-8982

osd: fix cache pools without hit_sets configured

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon/OSDMonitor: warn when cache pools do not have hit_sets configured

Give users a clue when cache pools are enabled but the hit_set is not
configured.  Note that technically this will work, but not well, so for
now let's just steer them away.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: improve agent_choose_mode args

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: limit open files

Most users only get 1024 fds.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: evict blindly if there is no hit_set

If there is no hit set for a PG, blindly evict objects.  This avoids an
assert(hit_set) in agent_estimate_atime_temp().

Fixes: #8982
Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: check agent_mode if agent is enabled but hit_sets aren't

It is probably not a good idea to try to run the tiering agent without a
hit_set to inform its actions, but it is technically possible.  For
example, one could simply blindly evict when we reach the full point.
However, this doesn't work because the agent mode is guarded by a hit_set
check, even though agent_setup() is not.  Fix that.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2168 from ceph/wip-8972

cls_rgw: fix object name of objects removed on object creation

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2173 from ceph/wip-8969

fix i386 atomics

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #1990 from dachary/wip-erasure-code-profile-default

erasure-code: create default profile if necessary

Reviewed-by: Joao Eduardo Luis <jluis@redhat.com>

Merge pull request #2162 from ceph/wip-8586

rgw: fix crash in swift CORS preflight request

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix i386 floating point rounding error

This would round .123 to .122999 on i386; now it does not.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2171 from ceph/wip-cli-integration

fix rbd cli-integratoin tests

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2164 from ceph/wip-8944

ceph.in: Pass global args to ceph-conf for proper lookup

Backport: firefly
Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test_daemon.sh: verify ceph -c works with daemon

Signed-off-by: Dan Mick <dan.mick@inktank.com>

qa/workunits/cephtool/test_daemon.sh: typo

Signed-off-by: Dan Mick <dan.mick@inktank.com>

qa/workunits/cephtool/test_daemon.sh: allow local ceph command

atomic: fix read() on i386, clean up types

Among other things, fixes #8969

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: Pass global args to ceph-conf for proper lookup

Fixes: #8944
Signed-off-by: Dan Mick <dan.mick@inktank.com>

test/cli-integration/rbd: fix trailing space

Newer versions of json.tool remove the trailing ' ' after the comma.  Add
it back in with sed so that the .t works on both old and new versions, and
so that we don't have to remove the trailing spaces from all of the test
cases.

Backport: firefly
Fixes: #8920
Signed-off-by: Sage Weil <sage@redhat.com>

mon: s/%%/%/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2163 from ceph/wip-filestore-omap

clean up KeyValueDB instantiation

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

cls_rgw: fix object name of objects removed on object creation

Fixes: #8972
Backport: firefly, dumpling

Reported-by: Patrycja Szabłowska <szablowska.patrycja@gmail.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

erasure-code: create default profile if necessary

After an upgrade to firefly, the existing Ceph clusters do not have the
default erasure code profile. Although it may be created with

   ceph osd erasure-code-profile set default

it was not included in the release notes and is confusing for the
administrator.

The *osd pool create* and *osd crush rule create-erasure* commands are
modified to implicitly create the default erasure code profile if it is
not found.

In order to avoid code duplication, the default erasure code profile
code creation that happens when a new firefly ceph cluster is created is
encapsulated in the OSDMap::get_erasure_code_profile_default method.

Conversely, handling the pending change in OSDMonitor is not
encapsulated in a function but duplicated instead. If it was a function
the caller would need a switch to distinguish between the case when goto
wait is needed, or goto reply or proceed because nothing needs to be
done. It is unclear if having a function would lead to smaller or more
maintainable code.

http://tracker.ceph.com/issues/8601 Fixes: #8601

Backport: firefly
Signed-off-by: Loic Dachary <loic@dachary.org>

rocksdb backend optimization

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add --with-librocksdb-static configure options

with this options, ceph will build rocksdb from submodule

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

get a stable rocksdb (3.0)

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add rocksdb submodule

add rocksdb support

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

os/KeyValueStore: rename osd_keyvaluedb -> keyvaluestore_backend

This option should be keyvaluestore_*, not osd_*.

Clean up the backend instantiation.

Signed-off-by: xinxin shu <xinxin.shu@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: use generic KeyValueDB::create(); store omap_backend in superblock

The only time we set this is at mkfs time.  Thereafter, we stick with the
same backend.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: use generic KeyValueDB::create()

Signed-off-by: Sage Weil <sage@inktank.com>

os/KeyValueDB: make compaction interface generic

Signed-off-by: Sage Weil <sage@inktank.com>

os/KeyValueDB: generic create(), test_init()

Let us create an implemenetation by name.  Include a test_init() method
that will instantiate an instance and verify it could start up.

Signed-off-by: Sage Weil <sage@inktank.com>

config: allow unsafe setting of config values

Allow callers to set config values even when there is no observer.

Signed-off-by: Sage Weil <sage@inktank.com>

doc/release-notes: v0.83

Signed-off-by: Sage Weil <sage@redhat.com>

Merge tag 'v0.83'

v0.83

Merge pull request #2161 from ceph/wip-jcsp-test

Reviewed-by: Greg Farnum greg@inktank.com

rgw: fix crash in swift CORS preflight request

Fixes: #8586
This fixes error handling, in accordance with commit 6af5a537 that fixed
the same issue for the S3 case.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mds: remove some rogue "using namespace std;"

Signed-off-by: John Spray <john.spray@redhat.com>

mds: handle replaying old format journals

To get back to the reformatting procedure that otherwise
occurs during MDLog::open, introduce an MDLog::reopen call
that MDS can use in the standbyreplay->standby transition
for the special case where the journal is old.

Fixes: #8869
Signed-off-by: John Spray <john.spray@redhat.com>

mds: introduce explicit DaemonState instead of int

Signed-off-by: John Spray <john.spray@redhat.com>

mds: refactor MDS boot

* Make boot_start private.
* Define boot stages in enum, replace int with type.
* Merge steps 0 and 1, 0 always fell through to 1.
* starting_done was only ever reached by a fall through
  from the previous step, so call it directly from there.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: make MDS::replay_done clearer

... and add some assertions.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: remove unused purge_prealloc_ino

Signed-off-by: John Spray <john.spray@redhat.com>

mds: separate inode recovery queue from MDCache

Refactor to:
* have somewhere to put some logic for doing
  background recovery in future.
* trim a few lines from the oversized MDCache.cc
  whereever we can.

Signed-off-by: John Spray <john.spray@redhat.com>

python-ceph: require libcephfs.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

0.83

Merge pull request #2159 from ceph/wip-undump

tools/cephfs: fuller header in dump/undump

Reviewed-by: Greg Farnum <greg@inktank.com>

Remove reference from mkcephfs.

A bit of colission from spec changes for the rhel7/ceph-common
changes and alfredo's pull request for wip-die-ceph-mkcephfs.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
(cherry picked from commit 1526546ddcfd4230403d0d2364575c4e46970f8d)

Merge pull request #2156 from ceph/wip-upstart-nfile

upstart/ceph-osd.conf: bump nofile limit up by 10x

Reviewed-by: Greg Farnum <greg@inktank.com>

doc/release-notes: typo

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v0.80.5 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge branch 'origin/wip-osd-leaks'

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2139 from ceph/wip-journal-header

os/FileJournal: Update the journal header when closing journal

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2146 from ceph/wip-8932

ceph_test_rados_api_tier: do fewer writes in HitSetWrite

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2147 from ceph/wip-8931

osd: fix ops blocked by full cache tier dequeue

Reviewed-by: Greg Farnum <greg@inktank.com>

unittest_crush_wrapper: fix build

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2150 from ceph/wip-libs

don't link everything with blkid, udev, and boost_threads

Merge pull request #2153 from ceph/wip-fsx-overlap

librbd API fix + wip-fsx-overlap

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #2152 from xiaoxichen/fix_ceph_df

PGMonitor: fix bug in caculating pool avail space

Reviewed-by: Sage Weil <sage@redhat.com>

Remove reference from mkcephfs.

A bit of colission from spec changes for the rhel7/ceph-common
changes and alfredo's pull request for wip-die-ceph-mkcephfs.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Fix some style and checking issue

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>