Merge pull request #5636 from liewegas/wip-12747

make EC plugin path static

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5646 from liewegas/wip-12766

osd: wait for cleanup from bench

Merge pull request #5650 from liewegas/wip-12736

uuid: use boost::random:random_device

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5654 from ceph/wip-12614

config: skip lockdep for intentionally recursive md_config_t lock

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5647 from ceph/wip-12765

librbd: error closing image while set to invalid snapshot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4677 from tchaikov/wip-fix-type-error-in-crush-rule

pybind: Avoid type error in crush rule (str, not an int)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5659 from badone/doc_writeback_typo

doc: Fix typo in writeback throttle section

Merge pull request #5660 from badone/doc_snaps_typo

doc: Fix typo and redundant word in snaps section

Merge pull request #5648 from ceph/wip-10399

Objecter: pg_interval_t::is_new_interval needs pgid from previous pool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5663 from ritz303/wip-doc-12771

doc: Added "Hammer" in the list of major releases.

doc: Added "Hammer" in the list of major releases.

Fixes: #12771
Signed-off-by: ritz303 <ritz_303@yahoo.com>

Merge pull request #5661 from SUSE/wip-suse-sysvinit-cleanup

ceph.spec.in: drop sysvinit-specific macros that run only on SUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: drop sysvinit-specific macros that run only on openSUSE/SLE

All sysvinit-based versions of openSUSE/SLE are EOL as far as upstream Ceph is
concerned.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5657 from badone/master

doc: Fix typo in recovery reservation section

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix typo and redundant word in snaps section

Fix typo and remove identifying from "filtering identifying" in
doc/dev/osd_internals/snaps.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: Fix typo in writeback throttle section

Fix typo in doc/dev/osd_internals/wbthrottle.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: Fix typo in writeback throttle section

Fix typo in doc/dev/osd_internals/wbthrottle.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: Fix typo in recovery reservation section

Fix typo in doc/dev/osd_internals/recovery_reservation.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #5607 from rzarzynski/wip-12724

rgw: fix error handling during GET on object through Swift API.

Merge pull request #5563 from varadakari/wip-kvstore-keyformat

Change key format to preserve order of keys in backend db.

Reviewed-by: Sage Weil <sage@redhat.com>

config: skip lockdep for intentionally recursive md_config_t lock

lockdep can't handle recursive locks, resulting in false positive
reports for certain set_val_or_die() calls, like via
md_config_t::parse_argv() passed "-m".

Fixes: #12614
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5649 from ceph/wip-12764

tests: fixed rbd cli cram integration tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

uuid: use boost::random:random_device

The boost mt code uses uninitialized memory for extra randomness,
which is a bad idea in general but more importantly makes valgrind
unhappy.  Use /dev/urandom instead.

Unfortunately this introduces a link time dependency.. meh!

Fixes: #12736
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5617 from rzarzynski/wip-12728

rgw: url_decode bucket name and prefix from X-Object-Manifest during GET on Swift DLO.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

tests: fixed rbd cli cram integration tests

Fixes: #12764
Backport: infernalis
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Objecter: pg_interval_t::is_new_interval needs pgid from previous pool

When increasing the pg_num of a pool, an assert would fail since the
calculated pgid seed would be for the pool's new pg_num value instead
of the previous pg_num value.

Fixes: #10399
Backport: infernalis, hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: error closing image while set to invalid snapshot

With cache disabled, closing the image will result in a flush
which might result in the image being refreshed.  If this happens
while the image is set to an invalid snapshot, an error will be
returned while closing the image.

Fixes: #12765
Backport: infernalis
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: wait for cleanup from bench

We need to wait for cleanup to flush before we destroy the
sequencer or else we get a use-after-free.

Introduced e7bbafa3bfbd5e936a8be026a30b83a89f6121c3.

Fixes: #12766
Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v9.0.3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #5623 from yuyuyu101/wip-12729

Fix compile warning unused-result

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5493 from dachary/wip-10887-shec

erasure-code: shec plugin feature

Reviewed-by: Takeshi Miyamae <miyamae.takeshi@jp.fujitsu.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5631 from wonzhq/finish-ctx

osd: consolidate encoding snaps into pg log code in finish_ctx

Reviewed-by: Kefu Chai <kchai@redhat.com>

PendingReleaseNotes: make a note about KeyValueStore on-disk format change

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5611 from ceph/wip-newstore-misc

osd: final prereq patches for newstore

Reviewed-by: Samuel Just <sjust@redhat.com>

test/erasure-code: drop directory from profile

Signed-off-by: Sage Weil <sage@redhat.com>

do not include directory in ec profiles

Signed-off-by: Sage Weil <sage@redhat.com>

osd: always load erasure plugins from the configured directory

Ignore the profile 'directory' field.

This ensures that we can always find plugins even when teh cluster
is installed across a mix of distros.

Rename the option to have no osd_ (or mon_) prefix since anybody
may use the ec factory/plugin code.

We still hard-code .libs in the unit tests... sigh.

Signed-off-by: Sage Weil <sage@redhat.com>

9.0.3

erasure-code: shec plugin feature

There is one new plugin (shec). When upgrading a cluster, there
must be a protection against the following scenario:

   * the mon are upgraded but not the osd
   * a new pool is created using plugin shec
   * the osd fail to load the shec plugin because they have not been
     upgraded

A feature bit is added : PLUGINS_V3. The monitor will only agree to
create an erasure code profile for the shec plugin if all OSDs
supports PLUGINS_V3. Once such an erasure code profile is stored in the
OSDMap, an OSD can only boot if it supports the PLUGINS_V3 feature,
which means it is able to load the shec plugin.

The monitors will only activate the PLUGINS_V3 feature if all monitors
in the quorum support it. It protects against the following scenario:

   * the leader is upgraded the peons are not upgraded
   * the leader creates a pool with plugin=shec because all OSD have
     the PLUGINS_V3 feature
   * the leader goes down and a non upgraded peon becomes the leader
   * an old OSD tries to join the cluster
   * the new leader will let the OSD boot because it does not contain
     the logic that would excluded it
   * the old OSD will fail when required to load the plugin shec

This is going to be needed each time new plugins are added, which is
impractical. A more generic plugin upgrade support should be added
instead, as described in http://tracker.ceph.com/issues/7291.

See also 9687150ceac9cc7e506bc227f430d4207a6d7489 for the PLUGINS_V2
implementation.

http://tracker.ceph.com/issues/10887 Fixes: #10887

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5634 from ceph/wip-fix-FTBFS-boost

rbd: fix the FTBFS on old boost introduced by 2050d08

rbd: fix the FTBFS on old boost introduced by 2050d08

in boost 1.49, BOOST_SCOPE_EXIT() does not accept capture_tuple,
only `(capture) (capture) ...` is supported.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5583 from ceph/wip-12676

mds: check mds up before calling MDSMap::get_mds_info

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5587 from caibo2014/master

rbd: remove the output file if rbd export-diff fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5632 from ceph/wip-fix-ftbfs-4998

osd: fix the FTBFS introduced by be28319

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: fix the FTBFS introduced by be28319

the signature of Scrubber::write_blocked_by_scrub() was changed in #5470.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4998 from wonzhq/hitset-remove-all

osd: implement hit_set_remove_all

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: consolidate encoding snaps into pg log code in finish_ctx

'MODIFY', 'PROMOTE' and 'CLEAN' are sharing the same encoding snaps
code.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: only remove all the hit set objects on primary osd

Only the primary PG is allowed to remove all the hit set objects. And
the PG should be in the active or peered states.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: no need to check current hitset object is degraded when persisting
hit set

The current hit set object doesn't exist yet. We are about to create it.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: implement hit_set_remove_all

When hit set is not configured on startup or on a change, remove all
previous hitsets.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Fix compile warning unused-result

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #5315 from tchaikov/wip-10930

mon: remove osd_epoch to avoid out-dated osdmap_cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5579 from jbweber/s3_encoding_type

rgw: implement s3 encoding-type for get bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 encoding-type for get bucket

This change introduces handling for the encoding-type request
parameter on the get bucket operation. An object key may contain
characters which are not supported in XML. Passing the value "url" for
the encoding-type parameter will cause the key to be urlencoded in the
response.

Fixes: #12735
Signed-off-by: Jeff Weber <jweber@cofront.net>

Merge branch 'dont-cache-recovery' of git://github.com/yuyuyu101/ceph

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/osd/ReplicatedBackend.cc
# src/osd/ReplicatedBackend.h
# src/osd/ReplicatedPG.cc

Merge pull request #5259 from branch-predictor/bp-tuned-bufferlists

Bufferlist tuning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5575 from ceph/wip-12370

rgw: don't preserve acls when copying object

Merge pull request #5518 from ceph/wip-12652

osd/PGLog: dirty_to is inclusive

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5539 from wonzhq/flush-mode-stat

mon: show # of PGs at high flush mode in the command output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5556 from zaitcev/cleanup-dead-code

rgw: cleanup dead init_bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5598 from jasuarez/remove-useless-code

rgw: Remove useless code in calc_hmac_sha1()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5620 from dachary/wip-12734-uuid

build/ops: add uuid-runtime to debian/control

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: add uuid-runtime to debian/control

uuid-runtime was incorrectly removed by
62bfc7a1ab1587e81ed3bff0ddfbb1aa69d1c299, restore it.

http://tracker.ceph.com/issues/12734 Fixes: #12734

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5621 from ceph/revert-5613-wip-12729

Revert "Fix compile warning unused-result"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Revert "Fix compile warning unused-result"

Merge pull request #5613 from ceph/wip-12729

Fix compile warning unused-result

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5523 from ceph/wip-scan-use-layout

Striped layouts in cephfs-data-scan

Merge pull request #5272 from chenji-kael/work-branch

rados.py: wrap omap operation for librados

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5614 from liewegas/wip-logtomonitors

mds/MDSRank: less noisy about log_to_monitors

rbd:remove the local file when rbd export-diff fail
Signed-off-by: Bo Cai <cai.bo@h3c.com>

rgw: url_decode values from X-Object-Manifest during GET on Swift DLO.

Fixes: #12728
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rados.py: This module now supports omap operations

you can operate omap in python
test_rados.py: add unit-test cases

Signed-off-by: Ji Chen <insomnia@139.com>

Merge pull request #5601 from wonzhq/doc-peering

doc: fix the format of peering.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDSRank: less noisy about log_to_monitors

Signed-off-by: Sage Weil <sage@redhat.com>

Fix compile warning unused-result

Fix #12729
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

doc: fix the format of peering.rst

Fix an incorrent number in the ordered list and some indention issue.
Make the ordered list to use '1' or 'a' for the first item, and '#' for
the remaining items.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge remote-tracking branch 'origin/wip-sam-proxy-write'

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Conflicts:
src/include/ceph_features.h
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/PGLog: dirty_to is inclusive

There are only two callers of mark_dirty_to who do not pass max,
and they are both in the merge_log extending tail path.  In that
case, we want to include the last version specified in the log
writeout.  Fix the tail extending code to always specify the
last entry added, inclusive.

Fixes: #12652
Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: log any options we pass to rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: ensure sync() doesn't wait forever

Avoid

2015-08-19 01:46:37.586450 7f89a53ca980 10 filestore(b.10) start_sync
2015-08-19 01:46:37.586451 7f89a53ca980 10 filestore(b.10) sync waiting
2015-08-19 01:46:37.586528 7f899e2b1700 20 filestore(b.10) sync_entry woke after 5.398564
2015-08-19 01:46:37.586539 7f899e2b1700 20 filestore(b.10) sync_entry waiting for another 35994.601435 to reach min interval 36000.000000

by setting force_sync flag.

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: rename get_bitreverse_key* -> get_bitwise_key*

Signed-off-by: Sage Weil <sage@redhat.com>

os/GenericObjectMap: use bitwise hash, not nibblewise

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: rename get_filestore_key* -> get_nibblewise_key*

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: fix bench-write

Make each IO get a unique offset!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-kvstore-tool: take a db type too

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: make hobject_t::set_key avoid dups

If the key matches the oid name, clear it.

Signed-off-by: Sage Weil <sage@redhat.com>

test_objectstore_memstore.sh on make check

This'll give us some coverage from ceph_test_objectstore, albeit
on the memstore backend.  Upside is that this takes < 10s.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test many small writes

This exercises the overlay code better.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: strengthen clone test w/ new assert

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test omap clone

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: much cleanup, new tests

These changes are necessary to make newstore (which is more picky
about unique collection names and objects that land *inside* the
correct collection) pass.  There are also several new tests.

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: fix open_handle when name_to_handle not present

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: add zero / hole punch support

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: add simple FS abstraction layer

- open by handle
- xfs allocation hint

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop sync(Context *onsync)

No users.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop sync()

No callers.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop flush()

No more callers.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop sync_and_flush()

The semantics of this call were FileStore-specific; and there are no more
callers.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: improve osd bench

Signed-off-by: Sage Weil <sage@redhat.com>