PendingReleaseNotes: make a note about KeyValueStore on-disk format change

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5611 from ceph/wip-newstore-misc

osd: final prereq patches for newstore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5634 from ceph/wip-fix-FTBFS-boost

rbd: fix the FTBFS on old boost introduced by 2050d08

rbd: fix the FTBFS on old boost introduced by 2050d08

in boost 1.49, BOOST_SCOPE_EXIT() does not accept capture_tuple,
only `(capture) (capture) ...` is supported.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5583 from ceph/wip-12676

mds: check mds up before calling MDSMap::get_mds_info

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5587 from caibo2014/master

rbd: remove the output file if rbd export-diff fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5632 from ceph/wip-fix-ftbfs-4998

osd: fix the FTBFS introduced by be28319

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: fix the FTBFS introduced by be28319

the signature of Scrubber::write_blocked_by_scrub() was changed in #5470.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4998 from wonzhq/hitset-remove-all

osd: implement hit_set_remove_all

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: only remove all the hit set objects on primary osd

Only the primary PG is allowed to remove all the hit set objects. And
the PG should be in the active or peered states.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: no need to check current hitset object is degraded when persisting
hit set

The current hit set object doesn't exist yet. We are about to create it.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: implement hit_set_remove_all

When hit set is not configured on startup or on a change, remove all
previous hitsets.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #5315 from tchaikov/wip-10930

mon: remove osd_epoch to avoid out-dated osdmap_cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5579 from jbweber/s3_encoding_type

rgw: implement s3 encoding-type for get bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 encoding-type for get bucket

This change introduces handling for the encoding-type request
parameter on the get bucket operation. An object key may contain
characters which are not supported in XML. Passing the value "url" for
the encoding-type parameter will cause the key to be urlencoded in the
response.

Fixes: #12735
Signed-off-by: Jeff Weber <jweber@cofront.net>

Merge branch 'dont-cache-recovery' of git://github.com/yuyuyu101/ceph

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/osd/ReplicatedBackend.cc
# src/osd/ReplicatedBackend.h
# src/osd/ReplicatedPG.cc

Merge pull request #5259 from branch-predictor/bp-tuned-bufferlists

Bufferlist tuning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5518 from ceph/wip-12652

osd/PGLog: dirty_to is inclusive

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5539 from wonzhq/flush-mode-stat

mon: show # of PGs at high flush mode in the command output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5556 from zaitcev/cleanup-dead-code

rgw: cleanup dead init_bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5598 from jasuarez/remove-useless-code

rgw: Remove useless code in calc_hmac_sha1()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5620 from dachary/wip-12734-uuid

build/ops: add uuid-runtime to debian/control

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: add uuid-runtime to debian/control

uuid-runtime was incorrectly removed by
62bfc7a1ab1587e81ed3bff0ddfbb1aa69d1c299, restore it.

http://tracker.ceph.com/issues/12734 Fixes: #12734

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5621 from ceph/revert-5613-wip-12729

Revert "Fix compile warning unused-result"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Revert "Fix compile warning unused-result"

Merge pull request #5613 from ceph/wip-12729

Fix compile warning unused-result

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5523 from ceph/wip-scan-use-layout

Striped layouts in cephfs-data-scan

Merge pull request #5272 from chenji-kael/work-branch

rados.py: wrap omap operation for librados

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5614 from liewegas/wip-logtomonitors

mds/MDSRank: less noisy about log_to_monitors

rbd:remove the local file when rbd export-diff fail
Signed-off-by: Bo Cai <cai.bo@h3c.com>

rados.py: This module now supports omap operations

you can operate omap in python
test_rados.py: add unit-test cases

Signed-off-by: Ji Chen <insomnia@139.com>

Merge pull request #5601 from wonzhq/doc-peering

doc: fix the format of peering.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDSRank: less noisy about log_to_monitors

Signed-off-by: Sage Weil <sage@redhat.com>

Fix compile warning unused-result

Fix #12729
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

doc: fix the format of peering.rst

Fix an incorrent number in the ordered list and some indention issue.
Make the ordered list to use '1' or 'a' for the first item, and '#' for
the remaining items.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge remote-tracking branch 'origin/wip-sam-proxy-write'

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Conflicts:
src/include/ceph_features.h
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/PGLog: dirty_to is inclusive

There are only two callers of mark_dirty_to who do not pass max,
and they are both in the merge_log extending tail path.  In that
case, we want to include the last version specified in the log
writeout.  Fix the tail extending code to always specify the
last entry added, inclusive.

Fixes: #12652
Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: log any options we pass to rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: ensure sync() doesn't wait forever

Avoid

2015-08-19 01:46:37.586450 7f89a53ca980 10 filestore(b.10) start_sync
2015-08-19 01:46:37.586451 7f89a53ca980 10 filestore(b.10) sync waiting
2015-08-19 01:46:37.586528 7f899e2b1700 20 filestore(b.10) sync_entry woke after 5.398564
2015-08-19 01:46:37.586539 7f899e2b1700 20 filestore(b.10) sync_entry waiting for another 35994.601435 to reach min interval 36000.000000

by setting force_sync flag.

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: rename get_bitreverse_key* -> get_bitwise_key*

Signed-off-by: Sage Weil <sage@redhat.com>

os/GenericObjectMap: use bitwise hash, not nibblewise

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: rename get_filestore_key* -> get_nibblewise_key*

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: fix bench-write

Make each IO get a unique offset!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-kvstore-tool: take a db type too

Signed-off-by: Sage Weil <sage@redhat.com>

common/hobject: make hobject_t::set_key avoid dups

If the key matches the oid name, clear it.

Signed-off-by: Sage Weil <sage@redhat.com>

test_objectstore_memstore.sh on make check

This'll give us some coverage from ceph_test_objectstore, albeit
on the memstore backend.  Upside is that this takes < 10s.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test many small writes

This exercises the overlay code better.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: strengthen clone test w/ new assert

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test omap clone

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: much cleanup, new tests

These changes are necessary to make newstore (which is more picky
about unique collection names and objects that land *inside* the
correct collection) pass.  There are also several new tests.

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: fix open_handle when name_to_handle not present

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: add zero / hole punch support

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: add simple FS abstraction layer

- open by handle
- xfs allocation hint

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop sync(Context *onsync)

No users.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop sync()

No callers.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop flush()

No more callers.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: drop sync_and_flush()

The semantics of this call were FileStore-specific; and there are no more
callers.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: improve osd bench

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add prepare_iov() method

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop dead suicide() code

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop explicit sync_and_flush() calls

We implicitly sync on shutdown.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: drop useless sync_and_flush

Signed-off-by: Sage Weil <sage@redhat.com>

osd: sync (no flush) on osd suicide

Not that we even need to sync here!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop explicit sync/flush calls before umount

Do it implicity on umount.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix flush_journal command

We need to sync_and_flush *and* sync() again in order to retire any items
in the FileStore journal.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: explicitly use a 'meta' Sequencer

Use an explicit Sequencer for all transactions.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use osr for scan_snaps

Updates should be ordered with respect to PG metadata.  Not that it matters
that much since we are doing cleanup/repair work here.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: do not let Sequencer::flush_commit delete the context

Let the caller manage the lifecycle; this allows us to potentially put it
on the stack, for instance.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: ref count Sequencer_impl

This allows the implementation to have a lifecycle that extends beyond
the Sequencer interface object we expose to the user.

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueDB: return -ENOENT if key doesn't exist

Previous code return 0 in all cases, caller doesn't
know if the key is exists in the db.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/ObjectStore: pass bits to create_collection

This informs the backend how many bits of the hash range will be covered
by the collection.  Those bits are implicitly matched against the spg_t in
coll_t.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pg_t: add contains() helper to check objects is in a pg

This calls the ghobject_t method that does the same, along with ps().

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: return const refs for oid, cid

Should be a bit faster.

Signed-off-by: Sage Weil <sage@redhat.com>

common/WorkQueue: add wait()

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: allow rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDbStore: init pointers in ctor

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: make coll_t operator== less picky

For meta coll's do not compare pg.

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueDB: add sipler get() variant

Signed-off-by: Sage Weil <sage@redhat.com>

common/Formatter: add dump_object helper

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: clear cached fd for renamed-over object

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5470 from ceph/wip-newstore-sort

osd: change sort order for hobject_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5424 from lebauce/fix_swift_user_command_line

doc: fix command line for swift user creation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5048 from SUSE/wip-objecter-comment-cleanup

Objecter.cc: fix name of _take_op_budget() function in comment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5606 from liewegas/wip-scrub-errormsg

osd/ReplicatedPG: fix missing whitespace in scrub error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5604 from oritwas/wip-cmake-fix-rbd

cmake: add DiffIterate.cc to librbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

moved to use boost uuid implementation, based on commit 4fe89a7b14c97b2ed7f357132901beb2bdcec551

Signed-off-by: Rohan Mars <code@rohanmars.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #5433 from zqkkqz/master

common/Thread: pthread_attr_destroy(thread_attr) when done with it

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: fix missing whitespace in scrub error message

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5538 from ceph/wip-selinux-improve-update

selinux: Relabel files if and only if the policy version changed

Reviewed-by: Milan Broz <mbroz@redhat.com>

ceph.spec.in: Require awk for selinux post script

Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Relabel files if and only if the policy version changed

Currently, the ceph files are being relabelled every time the package is
rebuilt. Fix this by checking the policy versions and relabel the files
only if the policy actually changed (different policy version was
detected).

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5596 from mbroz/wip-selinux-systemd

Update ceph-selinux rpm spec

Reviewed-by: Boris Ranto <branto@redhat.com>

cmake: add DiffIterate.cc to librbd

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5600 from liewegas/wip-osd-bind

doc: ceph-osd binds to up to 4 ports, not 3

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG::cancel_pull: also finish_degraded_object

We might have degraded object waiters on a pull.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: treat object as degraded until on_global_recover

Otherwise, an op might slip by between on_local_recover and
on_global_recover.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: block writes on promote of rollback snap promotion

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: enforce write ordering on rollback

Previously, rollback ops could reorder w.r.t other writes due to waiting
on degraded snaps other than head.  To fix that, we'll introduce a new
map tracking objects blocked on degraded snaps.  A particular object can
only be blocked on one snap at a time (subsequent writes won't get far
enough to add another entry).

It might have been possible use the blocked_by machinery for this, but
it requires that the object have an extant obc, which we may not
have for a missing object.  Also, that machinery exists primarily to
support clone_range, which I hope to remove soon.

Signed-off-by: Samuel Just <sjust@redhat.com>

RadosModel: send writes before and after rollback to verify ordering

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "osd: set the blocked_by relationship when rolling back to a degraded"

This reverts commit 48dacd75566f64cb89f991ac4d05e2e50c4e73e0.