test/erasure-code: drop directory from profile

Signed-off-by: Sage Weil <sage@redhat.com>

do not include directory in ec profiles

Signed-off-by: Sage Weil <sage@redhat.com>

osd: always load erasure plugins from the configured directory

Ignore the profile 'directory' field.

This ensures that we can always find plugins even when teh cluster
is installed across a mix of distros.

Rename the option to have no osd_ (or mon_) prefix since anybody
may use the ec factory/plugin code.

We still hard-code .libs in the unit tests... sigh.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5634 from ceph/wip-fix-FTBFS-boost

rbd: fix the FTBFS on old boost introduced by 2050d08

rbd: fix the FTBFS on old boost introduced by 2050d08

in boost 1.49, BOOST_SCOPE_EXIT() does not accept capture_tuple,
only `(capture) (capture) ...` is supported.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5583 from ceph/wip-12676

mds: check mds up before calling MDSMap::get_mds_info

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5587 from caibo2014/master

rbd: remove the output file if rbd export-diff fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5632 from ceph/wip-fix-ftbfs-4998

osd: fix the FTBFS introduced by be28319

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: fix the FTBFS introduced by be28319

the signature of Scrubber::write_blocked_by_scrub() was changed in #5470.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4998 from wonzhq/hitset-remove-all

osd: implement hit_set_remove_all

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: only remove all the hit set objects on primary osd

Only the primary PG is allowed to remove all the hit set objects. And
the PG should be in the active or peered states.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: no need to check current hitset object is degraded when persisting
hit set

The current hit set object doesn't exist yet. We are about to create it.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: implement hit_set_remove_all

When hit set is not configured on startup or on a change, remove all
previous hitsets.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #5315 from tchaikov/wip-10930

mon: remove osd_epoch to avoid out-dated osdmap_cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5579 from jbweber/s3_encoding_type

rgw: implement s3 encoding-type for get bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 encoding-type for get bucket

This change introduces handling for the encoding-type request
parameter on the get bucket operation. An object key may contain
characters which are not supported in XML. Passing the value "url" for
the encoding-type parameter will cause the key to be urlencoded in the
response.

Fixes: #12735
Signed-off-by: Jeff Weber <jweber@cofront.net>

Merge branch 'dont-cache-recovery' of git://github.com/yuyuyu101/ceph

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/osd/ReplicatedBackend.cc
# src/osd/ReplicatedBackend.h
# src/osd/ReplicatedPG.cc

Merge pull request #5259 from branch-predictor/bp-tuned-bufferlists

Bufferlist tuning

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5518 from ceph/wip-12652

osd/PGLog: dirty_to is inclusive

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5539 from wonzhq/flush-mode-stat

mon: show # of PGs at high flush mode in the command output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5556 from zaitcev/cleanup-dead-code

rgw: cleanup dead init_bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5598 from jasuarez/remove-useless-code

rgw: Remove useless code in calc_hmac_sha1()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5620 from dachary/wip-12734-uuid

build/ops: add uuid-runtime to debian/control

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: add uuid-runtime to debian/control

uuid-runtime was incorrectly removed by
62bfc7a1ab1587e81ed3bff0ddfbb1aa69d1c299, restore it.

http://tracker.ceph.com/issues/12734 Fixes: #12734

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5621 from ceph/revert-5613-wip-12729

Revert "Fix compile warning unused-result"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Revert "Fix compile warning unused-result"

Merge pull request #5613 from ceph/wip-12729

Fix compile warning unused-result

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5523 from ceph/wip-scan-use-layout

Striped layouts in cephfs-data-scan

Merge pull request #5272 from chenji-kael/work-branch

rados.py: wrap omap operation for librados

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5614 from liewegas/wip-logtomonitors

mds/MDSRank: less noisy about log_to_monitors

rbd:remove the local file when rbd export-diff fail
Signed-off-by: Bo Cai <cai.bo@h3c.com>

rados.py: This module now supports omap operations

you can operate omap in python
test_rados.py: add unit-test cases

Signed-off-by: Ji Chen <insomnia@139.com>

Merge pull request #5601 from wonzhq/doc-peering

doc: fix the format of peering.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDSRank: less noisy about log_to_monitors

Signed-off-by: Sage Weil <sage@redhat.com>

Fix compile warning unused-result

Fix #12729
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

doc: fix the format of peering.rst

Fix an incorrent number in the ordered list and some indention issue.
Make the ordered list to use '1' or 'a' for the first item, and '#' for
the remaining items.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge remote-tracking branch 'origin/wip-sam-proxy-write'

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Conflicts:
src/include/ceph_features.h
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/PGLog: dirty_to is inclusive

There are only two callers of mark_dirty_to who do not pass max,
and they are both in the merge_log extending tail path.  In that
case, we want to include the last version specified in the log
writeout.  Fix the tail extending code to always specify the
last entry added, inclusive.

Fixes: #12652
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5470 from ceph/wip-newstore-sort

osd: change sort order for hobject_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #5424 from lebauce/fix_swift_user_command_line

doc: fix command line for swift user creation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5048 from SUSE/wip-objecter-comment-cleanup

Objecter.cc: fix name of _take_op_budget() function in comment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5606 from liewegas/wip-scrub-errormsg

osd/ReplicatedPG: fix missing whitespace in scrub error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5604 from oritwas/wip-cmake-fix-rbd

cmake: add DiffIterate.cc to librbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

moved to use boost uuid implementation, based on commit 4fe89a7b14c97b2ed7f357132901beb2bdcec551

Signed-off-by: Rohan Mars <code@rohanmars.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #5433 from zqkkqz/master

common/Thread: pthread_attr_destroy(thread_attr) when done with it

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: fix missing whitespace in scrub error message

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5538 from ceph/wip-selinux-improve-update

selinux: Relabel files if and only if the policy version changed

Reviewed-by: Milan Broz <mbroz@redhat.com>

ceph.spec.in: Require awk for selinux post script

Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Relabel files if and only if the policy version changed

Currently, the ceph files are being relabelled every time the package is
rebuilt. Fix this by checking the policy versions and relabel the files
only if the policy actually changed (different policy version was
detected).

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5596 from mbroz/wip-selinux-systemd

Update ceph-selinux rpm spec

Reviewed-by: Boris Ranto <branto@redhat.com>

cmake: add DiffIterate.cc to librbd

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #5600 from liewegas/wip-osd-bind

doc: ceph-osd binds to up to 4 ports, not 3

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG::cancel_pull: also finish_degraded_object

We might have degraded object waiters on a pull.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: treat object as degraded until on_global_recover

Otherwise, an op might slip by between on_local_recover and
on_global_recover.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: block writes on promote of rollback snap promotion

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: enforce write ordering on rollback

Previously, rollback ops could reorder w.r.t other writes due to waiting
on degraded snaps other than head.  To fix that, we'll introduce a new
map tracking objects blocked on degraded snaps.  A particular object can
only be blocked on one snap at a time (subsequent writes won't get far
enough to add another entry).

It might have been possible use the blocked_by machinery for this, but
it requires that the object have an extant obc, which we may not
have for a missing object.  Also, that machinery exists primarily to
support clone_range, which I hope to remove soon.

Signed-off-by: Samuel Just <sjust@redhat.com>

RadosModel: send writes before and after rollback to verify ordering

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "osd: set the blocked_by relationship when rolling back to a degraded"

This reverts commit 48dacd75566f64cb89f991ac4d05e2e50c4e73e0.

test: change the test_tiering test case accordingly for proxy write

When an object is first created, it's proxied to base tier, need to
change the behavior of the test_tiering test case accordingly.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: copy the reqids even if the object is deleted during promotion

If the object is deleted on the base tier, and the reqids are not copied
during promotion, this again leads to the 'ops not idempotent' problem.
For the copy-get op, this fix copies the reqids even if the object doesn't
exist.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: purge the object from the cache when proxying and not promoting the op

When proxying the write/cache op, if it is decided to not promote the
object, need to purge it from the object_contexts cache. Otherwise, it
causes problems for the later ops on this object.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: set the blocked_by relationship when rolling back to a degraded
object

In a scenario like below:
- A rollback op comes in, and is enqueued.
- Several other ops on the same object come in, and are enqueued.
- The rollback op dispatches, and finds the object which it rollbacks to is
degraded, then this op is pushbacked into a list to wait for the degraded
object to recover.
- The later ops are handled and responded back to client.
- The degraded object recovers. The rollback op is enqueued again and finally
responded to client.
This breaks the op order. Need to set the blocked_by relationship to enqueue
the later ops until the degraded object recovers.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: skip promotion when proxying a delete op

When the object is deleted, there is no need to promote it.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: rename SKIP_PROMOTE to SKIP_HANDLE_CACHE

To comply with what it really means.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: force promote for object overwrites on a ec base pool

When doing object overwrites on the ec base pool, the write op can't be
proxied. Always force promotion in this case.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: explicitly set the reqid when proxying the write op

This is needed as in the following scenario:
- Client sends 3 writes and a read on the same object to base tier
- Set up cache tiering
- Client retries ops and sends the 3 writes and 1 read to the cache tier
- The 3 writes finished on the base tier, say with versions v1, v2 and v3
- Cache tier proxies the 1st write, and start to promote the object for the 2nd
write, the 2nd and 3rd writes and the read are blocked
- The proxied 1st write finishes on the base tier with version v4, and returns
to cache tier. But somehow the cache tier fails to send the reply due to socket
failure injecting
- Client retries the writes and the read again, the writes are identified as
dup ops
- The promotion finishes, it copies the pg_log entries from the base tier and
put it in the cache tier's pg_log. This includes the 3 writes on the base tier
and the proxied write
- The writes dispatches after the promotion, they are identified as completed
dup ops. Cache tier replies these write ops with the version from the base tier
(v1, v2 and v3)
- In the last, the read dispatches, it reads the version of the proxied write
(v4) and replies to client
- Client complains that 'racing read got wrong version'

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Objecter: optionally setting the reqid in the mutate interface

The cache tier needs to set the reqid to the original reqid from client
when proxying the write op.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: add reqid in MOSDOp

The cache tier needs to set the reqid explicitly to the original reqid
from the client when proxying the write op to the base tier.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: turn on proxy write feature bit by default

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: add proxy write perf counter

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: add the proxy write feature bit support

For older versions which doesn't support proxy write, doing promote.
Otherwise, we can proxy the write.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: don't check order in finish_proxy_write

When proxy write comes back from base tier, the write op may or may not
sit at the front of the list.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: add helper function check_for_promote

This function is used to check if we need to initiate a promotion in
maybe_handle_cache.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc

osd/ReplicatedPG: minor updates on proxy write

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

mon: add osd pool set/get for min_write_recency_for_promote

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/mon/MonCommands.h
src/mon/OSDMonitor.cc

osd/ReplicatedPG: promote on 2nd write

If min_write_recency_for_promote is
- 0: Promote when there is a write.
- 1: Check if the object is in current hit set. Promote if yes.
- else: Check if the object is in current and other in memory hit sets.
Promote if yes.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc

osd/osd_types: add min_write_recency_for_promote in pg_pool_t

This field stands for the minimum number of hit sets to check before
promote on write.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: set the RWORDERED flag for the promote copy-from op

Set the RWORDERED flag when doing promote copy-from op. This is in case
there are proxy writes in flight.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc

osd: tiering: use proxy write in writeback mode

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc

osd/ReplicatedPG: remove the peer_type assertion in eval_repop

When moving to proxy write, this assertion doesn't hold any more.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc

osd: tiering: add proxy write support

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc

doc: ceph-osd binds to up to 4 ports, not 3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

rgw: Remove useless code in calc_hmac_sha1()

Hex conversion from buf is not used at all.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>

Merge pull request #5597 from dachary/wip-12718-restorecon

ceph-disk: only call restorecon when available

Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-disk: only call restorecon when available

9db80da12803d42bb676d67f37442c0c54d83448 added an unconditional call to
restorecon after mounting the filesystem. It fails when restorecon is
not available and must be made conditional.

http://tracker.ceph.com/issues/12718 Fixes: #12718

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Do not use hardcoded paths in spec post section for selinux commands.

Signed-off-by: Milan Broz <mbroz@redhat.com>

Add dependence to selinux-policy-base to fix installation in kickstart.

Signed-off-by: Milan Broz <mbroz@redhat.com>

Update rpm spec to properly restart systemd ceph.target.

If we are using systemd, ceph target should be used.
Also fail to restart service should not stop policy to load.

Signed-off-by: Milan Broz <mbroz@redhat.com>

Merge pull request #5595 from ceph/wip-12710

qa/fsstress.sh: fix 'cp not writing through dangling symlink'

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #5594 from ceph/wip-12711

mds: properly set client incarnation

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tools/cephfs: account for striping in size estimation

Signed-off-by: John Spray <john.spray@redhat.com>

qa/fsstress.sh: fix 'cp not writing through dangling symlink'

On some test machines, /usr/lib/ltp/testcases/bin/fsstress is
dangling symlink. 'cp -f' is impotent in this case.

Fixes: #12710
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly set client incarnation

Fixes: #12711
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #5593 from ceph/revert-4927-snapset-obc

Revert "osd/ReplicatedPG: snapset is not persisted"

Reviewed-by: Samuel Just <sjust@redhat.com>

Revert "osd/ReplicatedPG: snapset is not persisted"

Merge pull request #5577 from oritwas/wip-next-12363

rgw: we should not overide Swift sent content type

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5558 from s09816/rbd-fix

rbd:improve the error handle of rbd,check the return value.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/OSDMap: test_flag returns bool

Otherwise comparing the result to a bool will promote the bool to
an int and fail the comparison.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: restart listing this PG if sort order changes

If the cluster sort order changes mid-way through our listing, our
cursor within this pg is meaningless and we need to restart at the
beginning of the PG.

Signed-off-by: Sage Weil <sage@redhat.com>