test: Add testing for PGLog::filter_log()

Indirectly tests pg_log_t::filter_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Create a filter_log for PGLog

This could be tested as a way to test pg_log_t::filter_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool, osd: Filter the pg_log_t for objects no longer in pg

Fixes: #11184
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Fix message and make it debug only to stderr

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove bogus comment and eliminate a debug message

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4163 from athanatos/wip-revert-degraded-writes

Wip revert degraded writes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4152 from ceph/wip-11145

rgw: shut down timer before erroring out

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: shut down timer before erroring out

Fixes: #11145
Backport: hammer

Clean up initialization timer, otherwise if we error out, we'll assert.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4125 from dzafman/wip-11176-hammer

ceph-objectstore-tool: Output only unsupported features when incomatible

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4145 from ceph/wip-hammer-rgw

Wip hammer rgw

Reviewed-by: Sage Weil <sage@redhat.com>

osd/: s/is_degraded_object/is_degraded_or_backfilling_object

Currently, this method also returns true if the object is backfilling.
This commit was reverted earlier in the branch in order to make the
other reverts clean.  It's actually a nice rename though, so I'm
re-cherry-picking it.

Signed-off-by: Samuel Just <sjust@redhat.com>
Conflicts:
src/osd/ReplicatedPG.cc

Revert "osd/: update peer_missing and local missing if we write a degraded object"

This reverts commit c632fecb87e292eb73161f36d4e6df6f077fae77.

Revert "append_log: use remove_snap_mapped_object"

This reverts commit 964d17756f9e49002eb96c4f27a3f41da0880f71.

Revert "osd/: don't block writes on degraded objects"

This reverts commit f9abffbb2461b5988c0435ee2f7771eca028f29f.

Revert "ReplicatedPG: block writes on degraded objects for ec pools"

This reverts commit 1f165d2547af844de8406b35e89d371d56730f09.

Revert "Merge pull request #3641 from athanatos/wip-10731"

This reverts commit 54a22236a00ab18ec62c6e7df1117055d5fd0ef1, reversing
changes made to 7eca03e290f278bdf3a9613e0f383ec49f40b645.

Conflicts:
src/include/ceph_features.h

Revert "osd/: s/is_degraded_object/is_degraded_or_backfilling_object"

This reverts commit 25f9363b90a66b8bf3c14cf99ee417823fc84fbf.

rgw: update makefile to enable civetweb config

Fixes: #10965
Backport: hammer, firefly

Civetweb compilation now includes conf header to enable ipv6.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit caa90225bad9fe5e9c275e6189b3396b4d396e3f)

civetweb: update submodule

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 0f8be6f086faaea4b20a36dfdd436d2300f56d08)

rgw: pass civetweb configurables to civetweb

Fixes: #10907
Backport: firefly

Pass any configurables in the rgw frontends config line to civetweb.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 986d7554426764a149621ba733c5c075b94e0431)

rgw: don't overwrite bucket / object owner when setting acls

Fixes: #10978
Backport: hammer, firefly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit eb13f2d4b60c031f16139f7cc4237c012644dd78)

Revert "ReplicatedPG: only allow a degraded write if we have at least min_size copies"

This reverts commit 48466f8d7ce2b6762077f3fc686ada5648021517.

Revert "Merge pull request #3911 from athanatos/wip-11057"

This reverts commit 05bb9c9870781941b2e362e6cb9b5ec7fb72e97e, reversing
changes made to b780048c48784a59348d030b5be720d270649a36.

Merge pull request #4134 from dzafman/wip-dzman-hammer

Minor man page improvement

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Regenerate man/ceph.8 based on ceph.rst changes

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Break ceph osd pool get into sections based on pool type

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Output only unsupported features when incomatible

Fixes: #11176
Backport: firefly, giant

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4040 from dzafman/wip-11092-11093

Fix ceph pg dump_stuck with multiple options

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #4107 from ceph/wip-11102

osd: only complain about stored vs actual digest if all peers support it

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: only complain about stored vs actual digest if all peers support it

If we have a mixed cluster of hammer and pre-hammer OSDs, we will fall back
to using 0 as the initial crc32c value.  However, if the primary has a
stored digest, it currently compares its value to the reported value (w/
the wrong initial value) and complains.

There are two possible fixes:
 - avoid storing a digest if all peers don't support it, or
 - avoid complaining on scrub if all peers don't support it.

The latter is easier, and this fix also has the benefit of fixing the bug
even for clusters where this has already happened.

Fixes: #11102
Signed-off-by: Sage Weil <sage@redhat.com>

PG::find_best_info: reject infos with old last_epoch_started

See doc/dev/osd_internals/last_epoch_started.rst

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3988 from ceph/wip-refine-build-configuration-hammer

Refine build configuration for hammer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4105 from athanatos/wip-11110

Wip 11110

Reviewed-by: Sage Weil <sage@redhat.com>

init-radosgw*: don't require rgw_socket_path to be defined

Fixes: #11159
Backport: hammer, firefly

Scripts required rgw_socket_path to exist in order to start radosgw.
This is not needed.

Reported-by: Dan Mick <dmick@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6823bcdcd0ce72cd223e809291f46d82da76115c)

Merge pull request #4064 from dachary/wip-10488-jerasure-idempotent

mon: informative message when erasure-code-profile set fails

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4066 from dachary/wip-11144-erasure-code-profile-rm

osd: erasure-code-profile incremental rm before set

Reviewed-by: Sage Weil <sage@redhat.com>

PG: ensure that info.last_epoch_started only increases

See doc/dev/osd_internals/last_epoch_started.rst

Fixes: #11110
Signed-off-by: Samuel Just <sjust@redhat.com>

doc: add last_epoch_started.rst

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3898 from jdurgin/wip-rbd-merge-diff

qa: make rbd-fuse exit cleanly with lttng

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4071 from athanatos/wip-10718

Wip 10718

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4016 from ceph/wip-11123

osd: fix whiteout handling for delete+create compound ops

Reviewed-by: Samuel Just <sjust@redhat.com>

PG: make sure to update history.last_epoch_started with first write

Otherwise, we might remember the write, but not that the activation
interval was history.last_epoch_started.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd: erasure-code-profile incremental rm before set

It is possible for an incremental change to have both a rm and a set for
a given erasure code profile. It only happens when a rm is followed by a
set. When a set is followed by a rm, the rm will remove the pending set
in the incremental change.

The logic is the same for pool create and pool delete.

We must apply the incremental erasure-code-profile removal before the
creation otherwise rm and set in the same proposal will ignore the set.

This fix is minimal. A better change would be that erasure-code-profile
set checks if there is a pending removal and wait_for_finished_proposal
before creating.

http://tracker.ceph.com/issues/11144 Fixes: #11144

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon: informative message when erasure-code-profile set fails

When erasure-code-profile set refuses to override an existing profile,
it may be non trivial to figure out why. For instance:

   ceph osd set default ruleset-failure-domain=host

fails with:

   Error EPERM: will not override erasure code profile default

although ruleset-failure-domain=host is documented to be the
default. The error message now includes the two profiles that have been
compared to not be equal so that the user can verify the difference.

   Error EPERM: will not override erasure code profile default
   because the existing profile
   {directory=.libs,k=2,m=1,plugin=jerasure,technique=reed_sol_van}
   is different from the proposed profile
   {directory=.libs,k=2,m=1,plugin=jerasure,ruleset-failure-domain=host,technique=reed_sol_van}

http://tracker.ceph.com/issues/10488 Fixes: #10488

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4057 from ceph/wip-rgw-bootstrap

mon: add bootstrap-rgw profile

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

ceph-create-keys: create bootstrap-rgw key

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add 'bootstrap-rgw' profile

This profile is allowed to create client.rgw.* librados users.  They have
access to any pool, which is pretty coarse.

Signed-off-by: Sage Weil <sage@redhat.com>

Compile test_build_librgw only if WITH_BUILD_TESTS

Merge pull request #3910 from athanatos/wip-11027

Wip 11027

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3918 from tchaikov/hammer-fix-leaks-in-C_TwoContexts

osdc: fix a memory leak in C_TwoContexts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3972 from tchaikov/hammer-fix-msg-leak

osd: fix a msg leak

Reviewed-by: Sage Weil <sage@redhat.com>

osd: use (robust) helper for setting exists or clearing whiteout

The current blanket check in prepare_transaction() will trigger only when
there is a net obs.exists change from the commited obs to new_obs.
However, this misses the case where the first osd_op is a delete and then a
subsequent osd_op recreates the object.  Changing the whiteout check to
look only at new_obs does not work because it fails to understand when
_delete_oid sets the whiteout and will simply clear it again.

In order to support sequences of delete + create in general, we need to do
the whiteout flag clearing when the actual create happens (to match the
fact that we set it when we process the delete osd_op).  Use a helper to
do this and consolidate most other obs.exists = true code to use it.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3994 from athanatos/wip-11069

PGLog: split divergent priors as well

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

PGLog::merge_log: in tail extend case, log.log might be empty

Even if the tail != last_update, the log might be empty due to split
moving all entries into other logs.

Fixes: 10718
Signed-off-by: Samuel Just <sjust@redhat.com>

mon: Support multiple args to ceph pg dump_stuck as in usage and man pages

Fixes: #11092
Signed-off-by: David Zafman <dzafman@redhat.com>

Improve "ceph_argparse.py: add stderr note if nonrequired param is invalid"

When processing arguments stash an exception which is seen when looking
for another argument type.  If we have extra args at the end, output
information about the exception which probably caused the problem.

Fixes: #11093
Original fix: 71ff794eca7dd57a0b0473530f685069ac9987f0

Signed-off-by: David Zafman <dzafman@redhat.com>

PGLog: split divergent priors as well

Signed-off-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: warn about lttng LD_PRELOAD for daemons

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3945 from tchaikov/hammer-fix-xio-requeue

XIO: Handle requeue case of XIO messages

Merge pull request #3977 from tchaikov/fix-xio-loop-conn-cleanup

Fix XioLoopbackConnection Lifecycle.

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados_tier: add test case for delete+create compound ops

This triggers #11123, originally observed when rgw does a delete + create
sequence in a single op and the whiteout gets left behind.

Signed-off-by: Sage Weil <sage@redhat.com>

PG: add config to ignore history les in find_best_info

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4004 from rohanmars/wip-refine-build-configuration-hammer

fixes to compile librados on rhel 5.9

rhel 5.9 port fixes to compile librados only
Signed-off-by: Rohan Mars <code@rohanmars.com>

Merge pull request #3982 from dzafman/wip-10678-2

Update ceph.8 man page to match ceph -h output

Reviewed-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Fix ceph command manpage to match ceph -h (hammer)

Include rebuilt man/ceph.8

Fixes: #10678
Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Fix ceph command manpage to match ceph -h (firefly)

Improve synopsis section
Fixes: #10676
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9ac488c1eb0e30511079ba05aaf11c79615b3940)

Conflicts:

doc/man/8/ceph.rst
man/ceph.8

PGBackend: do not rewrite ec object oi checksums

Deep scrub does not actually give us the whole-object checksum for an ec
object, only the checksum for the first shard.  We ignore it in scrub
for ec pools anyway in be_select_auth_object.

Signed-off-by: Samuel Just <sjust@redhat.com>

PGBackend: add debug option to rewrite digest even if present

Related: #11027
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: finish_ctx: do not assume that the oi is for the head object

If we are setting the oi on the clone object, we want to update the
mod_desc.

Fixes: 11027
Signed-off-by: Samuel Just <sjust@redhat.com>

Build ceph-dencoder if server and mds for now

rbd: regenerate rbd(8) man page

merge-diff and status commands, --image-features and --image-shared
options, other stuff.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 356a749f63181d401d16371446bb8dc4f196c2a6)

doc: Fix typo in ceph pg dump_stuck usage in man page

Cause by 9db596974c82207dc0769ca6bc9810a9d6bcb5cc

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3981 from ceph/wip-librbd-invalidate-object-map

librbd: invalid snapshot object maps should only affect the snapshot

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3978 from tchaikov/hammer-sca-fixes

sca fixes ported from master

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: invalidating snapshot object maps should set snapshot flag

Instead of setting the HEAD image flag to indicate an invalid
object map, set the flag for the specific snapshot that is
invalid.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test_cls_rbd: verify set_flags updates snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd_client: add snap_id param to set_flags

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: set_flags can now update snapshots

It's possible for an object map to be invalid only for
a snapshot, so allow snapshot flags to be updated. This
will also be required when rebuilding the object map and
clearing the invalid flag.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw_main.cc: fix null-deref in case get_storage() fails

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit 921c08fce503e171199f8f8b9208dcdf75ac41c4)

rgw/rgw_op.cc: fix potential null-deref in strcmp()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit de7bcfd8d0e578207cbd9237bc166937fdadcd62)

Fix XioLoopbackConnection Lifecycle.

Since XioLoopbackConnection is a RefCountedObject, it can't be
an expanded member of XioMessenger.

Fixes cleanup/shutdown errors.

Signed-off-by: Matt Benjamin <matt@cohortfs.com>
(cherry picked from commit f5735b28d154612b339cf5d3b558e4dc5eaa2884)

osd: refactor RepScrubWQ::_process()

use Mutex::Locker instead of releasing the lock manually.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fix a msg leak when OSD is stopping

Signed-off-by: Kefu Chai <kchai@redhat.com>

XIO: Handle requeue case of XIO messages

If we are not able to send the XIO message using xio_send_msg(),
remove the XIO message from the send Q, before queuing it to the resend
Q. Otherwise, boost will generate a assert.

Signed-off-by: Raju Kurunkad <raju.kurunkad@sandisk.com>
(cherry picked from commit dbb8c93d41d92ffe4a6c181ea199d2fd211a1b37)

Merge branch 'wip-add-cache' into hammer

Reviewed-by: Sage Weil <sage@redhat.com>

qa,cephtool: add test case for 'ceph osd tier add-cache'

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon: Don't forget set read_tier/write_tier for 'osd tier add-cache'.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge branch 'wip-11067' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: add unit test for multiple concurrent resize ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: retrieve image size at start of resize op

For the case where multiple resize requests were pending,
queued resizes should use the most recent value for the
current image size -- not the image size when the resize
was queued.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not invalidate oversized object map

If the object map is too large for the current image, it implies
that a resize operation was interrupted.  It should only be
considered invalid if the object map is smaller than the image,
which shouldn't be possible.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid canceling object map / header updates

During a resize, reduce the possibility that the object map
and the header will get out-of-sync during a resize operation
that is canceled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update size / parent overlap after resize completes

Update the in-memory image metadata for size and parent
overlap after updating the on-disk image metadata. Also
schedule an image refresh in case multiple resizes are
enqueued -- since ictx_refresh isn't async.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use image size and parent overlap from active shrink op

If a shrink operation is in progress, all operations should use
the new size and new parent overlap for IO operations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: queue multiple resize requests

Only allow a single resize request to be executed concurrently.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

xlist: add const versions of front / back

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use ImageCtx::get_parent_overlap helper function

Replace all direct references to ImageCtx::parent_md.overlap
with the helper method.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: resize/flatten should validate image is R/W

When proxying resize/flatten requests, the current image
R/W state is not validated.  Also ensure the proper locks
are held when reading image metadata.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: hold write snap_lock when refreshing object map

If the object map needs to be invalidated, a write lock on
snap_lock is required.

Fixes: #11067
Signed-off-by: Jason Dillaman <dillaman@redhat.com>