osd/PG: debug scrub range

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgarde/jewel-x: add final scrub and legacy snapset check

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: allow creation of obc for new head object

We may be creating the head object to migrate the SnapSet to.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: at end, scrub pgs, verify no legacy snapsets

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: switch require-luminous facet to use full_sequential_finally

This lets us run multiple cleanup steps right before ceph
teardown.

Note that we drop the facet from multimon/ because it
doesn't factor out cluster creation before this step
properly.  That's fine because the require_luminous
cleanup shouldn't be related to the multimon tests.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: refactor thrash and cluster create

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash: simplify

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code: factor out cluster create

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic: factor out cluster start

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash: extra cluster create

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: do not warn about pgp_num for fake "deleted" pools

This breaks the upgrade test from jewel.  We can probably revert it later.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_list: make LibRadosListNP.ListObjectsError delete pool even if faking

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add mon_fake_pool_delete option

Instead of deleting a pool, add a .NNN.DELETED suffix to the end.  This
keeps the data around long enough for it to be scrubbed later (in the
case of a teuthology job cleanup).

If you really want to delete the pool, then instead of the usual force
flag option you can pass --yes-i-really-really-mean-it-not-faking.  :)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_*: do not clean up objects (usually)

First, this is pointless--each test runs in a namespace so they don't
step on each other.  Second, leaving objects in place is an opportunity
for scrub to notice any issues we created.  Third, the cleanup asserts
that delete succeeds but if clones exist pgls will show whiteouts and then
delete will return ENOENT.  We could disable the assert, but why bother
even attempting a sloppy cleanup?

We need to preserve cleanup behavior for a few tests (notably the object
listing ones).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_list: update check_list

We no longer see dup entries on split.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_cls_refcount: fix cls_rgw.test_put_snap

Do not touch the object before the put if we expect to see ENOENT.
We would get it anyway with older code, but with the snapset
changes we are on a whiteout and get ENODATA instead.

And really we shouldn't have been getting ENOENT after a touch
anyway!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: handle both new and legacy SnapSet format

If we encounter legacy snaps, add to snapmapper then; otherwise, use the
clone_snaps field in SnapSet to add all clones to snapmapper when we
process the head object.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-scrub-snaps.sh: fix test

- objects come in a different order, meh
- ss is on head, always, not snapdir.
- error messages on head, not snapdir

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: only clear SnapSet::head_exists for legacy snapsets

In the new world, head_exists is always true.  Stop clearing it in
_delete_oid (except for legacy compatibility), and fix the various callers
and assertions to compensate.  Note that we now use head_exists as a flag
to guide us into the snapdir branch of finish_ctx() (which will go away
post-luminous).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pass legacy flag to from_snap_set

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: document snaps vectors are descending

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedBackend: set recovry_info.ss on non-head push

Replicas need this in order to store the clones in SnapMapper.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: populate recovery_info

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: clear whiteout on exists=false objects in obc cache

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: remove head if snapset is empty and head is whiteout

We include a check to make sure we do not delete a dirty whiteout if this
is a tier pool and the object is dirty.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: maintain pessimistic estimate of num_legacy_snapsets

- Assume that any snapset we update, if not require_luminous, is a net-new
legacy SnapSet.  (It might be an existing one, which would be net-0, but
that is harder to tell.)

Then, during scrub,

- Any unreadable oi is assumed to include a legacy snapset
- Any snapset we encounter if !require_luminous is legacy
- Any object that should have a snapset but doesn't (corrupt or missing)
is assumed to be legacy.
- If were trying to update a legacy Snapset but have to abort, then it is
still legacy.

We could assume that a missing/broken snapset is not legacy since it has
to be repaired anyway (and therefore shouldn't block upgrade), but I'm
not sure.  For now, we'll take the conservative approach of blocking the
upgrade if the snapset metadata is missing/corrupt.

Signed-off-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/osd/PrimaryLogPG.cc

osd/osd_types: add num_legacy_snapsets to stats struct

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: convert snapdir during scrub

During scrub, we assemble the clone legacy_snaps values and put them
in the SnapSet.

Note that we do not bother clearing legacy_snaps in the clones; this is
more complexity to do correctly and not worth the effort; we will always
have the SnapSet to indicate whether we need to look at the oldf ield
or not.

If the rebuild clone_snaps is not the correct size we bail out; this can
happen if some clones are missing oi or missing entirely, or if there are
extra clones.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: store clone list in SnapSet

Store the per-clone snaps list in SnapSet if (1) REQUIRE_LUMINOUS is set
in the OSDMap and (2) the SnapSet isn't a 'legacy' SnapSet that hasn't
been converted yet by scrub.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: rename object_info_t::snaps -> legacy_snaps

This will make it easier to identify users of this field that need to
conditionally use either the old legacy_snaps or the new
SnapSet::clone_snaps vector.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add clone_snaps to SnapSet

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: delete + ignore_cache is a soft hint

We may still need to create a whiteout because clones still exist.

Arguably delete+ignore_cache is not the right way to remove whiteouts and
we should have a separate RADOS operation for this.  But we don't.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: keep snapset on whiteout head if luminous

If REQUIRE_LUMINOUS is set on the OSDMap, put the SnapSet on the head
and make it a whiteout.  This is simpler and will eventually (once all
the old snapdir objects have been fixed up) let us remove a ton of
snapdir-related headaches.

This is surprisingly simple.  Everywhere else we work with snapdir is
already set up to handle the snapset on the head.  The only difference is
that we are preventing outselves from moving from the snapset-on-head
state to the snapset-on-snapdir-with-no-head state.  The only time this
happens is when the object is logically deleted in _delete_oid.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: filter snapc.snaps up front

No reason to wait for make_writeable(); ensure we have a valid snapc
from the start.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: remove unused snapcolls from ScrubMap::object

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: drop ScrubMap::object::nlink; drop some decode compat cruft

We moved to v7 encoding in ca81563d8d71052a2040eda74e7ac418f1616fc7, which
was pre-jewel, so that's the oldest version we need to decode.

Drop the nlink field, which has not been used in a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: move SnapSetContext to osd_internal_types

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: add some debug output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: debug obs.exists from get_object_context

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: improve make_writeable debug output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14834 from runsisi/wip-fix-double-free

librbd: minor fixes for image trash move

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14830 from runsisi/wip-open-cleanup

librbd: remove redundant check for image id emptiness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14927 from tchaikov/wip-19427

logclient: use the seq id of the 1st log entry when resetting session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14920 from tchaikov/wip-14579

mgr: add machinery for python modules to send MCommands to daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13681 from liewegas/wip-epochs

osd: tag fast dispatch messages with min_epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14273 from cbodley/wip-rgw-beast-new-parser

rgw: update Beast for streaming reads in asio frontend

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rename frontend from asio to beast

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update Beast for streaming reads in asio frontend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update Beast submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14931 from tchaikov/wip-19771

qa/tasks/ceph_manager: always fix pgp_num when done with thrashosd task

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14973 from ifed01/wip-bluestore-kvsync-cond

os/bluestore: remove unused condition variable

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove unused condition variable

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14963 from dillaman/wip-19858

rbd-mirror: failover and failback of unmodified image results in split-brain

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14965 from dmick/wip-19865

test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14465 from dillaman/wip-librbd-devstack-tempest

qa/workunits: switch to OpenStack Ocata release for RBD testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

If RBD_FEATURES is not in the environment, set *features to 0 in
get_features(); callers rely on a valid return value.  (This was
breaking on arm64.)

Fixes: http://tracker.ceph.com/issues/19865
Signed-off-by: Dan Mick <dan.mick@redhat.com>

librbd: add no-op event when promoting an image

The rbd-mirror process needs an event in the journal
to properly detect the transition between primary and
non-primary state between peers.

Fixes: http://tracker.ceph.com/issues/19858
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not delay propagation of demote events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: prevent infinite loop when computing replay status

If the image had a non-primary predecessor epoch whose tag tid
duplicates an epoch within its own journal, an infinite loop
would result.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14373 from rzarzynski/wip-rgw-19520

rgw: fix broken /crossdomain.xml, /info and /healthcheck of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9950 from weiqiaomiao/wqm-wip-fetchobj-req

rgw: should delete in_stream_req  if conn->get_obj(...) return not zero value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14601 from Jing-Scott/fix-bulkupload-multisite

rgw: fix forward request for bulkupload to be applied in multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14945 from dillaman/wip-19811

rbd-mirror: ensure missing images are re-synced when detected

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14935 from dillaman/wip-rbd-cli-mirror-pool

rbd: removed spurious error message from mirror pool commands

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14400 from ceph/wip-cd-1node

qa/tasks: few fixes to get ceph-deploy 1node to working state

Merge pull request #14936 from cbodley/wip-19834

rgw: RGWPeriodPusher spawns http thread before cr thread

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14908 from guihecheng/rgw_file-cleanup

rgw_file: cleanup virtual keyword on derived functions

Merge pull request #14891 from tchaikov/wip-librbd-test-odr

librbd: do not instantiate templates while building tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not instantiate templates while building tests

to avoid violation of ODR

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14844 from liewegas/wip-bluestore-bitalloc

unittest_alloc: add test_alloc_big

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

unittest_alloc: add test_alloc_big

Ensure that big allocations work.

Verifies http://tracker.ceph.com/issues/16662

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14953 from tchaikov/wip-librbd-silence-Wunused-var

librbd: silence -Wunused-variable warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14407 from cbodley/wip-rgw-admin-remote

rgw: fix handling of --remote in radosgw-admin period commands
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

librbd: silence -Wunused-variable warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11179 from rzarzynski/wip-rgw-swift-formpost

rgw: add support for FormPost of Swift API
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14949 from zhsj/fix-cryptopp-build

common: fix building against libcryptopp

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: fix building against libcryptopp

libnspr is only needed for libnss, move it to if defined(USE_NSS)
section.

ceph::crypto::shutdown's params should be same with declaration.

Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>

Merge pull request #14923 from joscollin/wip-dmclocktest-function-error

dmclock: include missing <functional> header.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14948 from Liuchang0812/wip-kill-duplicated-cmake-commands

cmake: kill duplicated cmake commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14951 from smithfarm/wip-19853

cmake: align cmake names of library packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Update rgw_rest.cc

fix indentation.
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: align cmake names of library packages

The cmake package names of libraries are case-sensitive. The name used in
FIND_PACKAGE_HANDLE_STANDARD_ARGS - e.g. "cryptopp" - must match the name used
in the module name - "Findcryptopp.cmake" - as well as the name used in
find_package() call - "find_package(cryptopp REQUIRED)"

Note that the term "package" here refers to a cmake abstraction, *not* an
actual RPM or Debian package.

Fixes: http://tracker.ceph.com/issues/19853
Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: kill duplicated cmake commands

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #14947 from dmick/wip-arm-compressor

compressor/zlib: fix plugin for non-Intel arches

Reviewed-by: Kefu Chai <kchai@redhat.com>

compressor/zlib: fix plugin for non-Intel arches

unittest_compressor was failing on arm64 because the zlib
compressor was never initialized, even though it works fine in
non-isal mode

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14363 from majianpeng/finisher-optimize

common/Finisher: batch handle perfcounter && only send signal when waiter existed.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14627 from badone/wip-peering-state-timing

osd: Implement peering state timing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14709 from majianpeng/luestore-txc-finalize-kv

os/blueStore: In osd_tp_thread, call _txc_finalize_kv.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14932 from markhpc/wip-rocksdb-compaction-readahead

common/config_opts.h: compaction readahead for bluestore/rocksdb

Reviewed-by: Sage Weil <sage@redhat.com>

test/librbd: remove duplicated explicit template specializations

to avoid violation of ODR

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: ensure missing images are re-synced when detected

Fixes: http://tracker.ceph.com/issues/19811
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

default vpm is trusty and for master we need latest which is xenial

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

install mgr on the node

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #14879 from wjwithagen/wip-wjw-reduce-noise-3

cls/log/cls_log.cc: reduce logging noise

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14928 from drunkard/master

doc: remove deprecated subcommand in man/8/ceph.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14723 from neha-ojha/objclass-sdk

osd: objclass sdk

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14917 from yuriw/wip-add-overrite-cls

qa/added overrides

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls/log/cls_log.cc: reduce logging noise

 - The other reference in the source as already at 20.
      ./src/cls/timeindex/cls_timeindex.cc:85:
CLS_LOG(20, "storing entry at %s", index.c_str());

   And we need not always know where in the log items are stored.
   So it looks like a leftover debug feature.

Fixes: http://tracker.ceph.com/issues/19835
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14913 from liewegas/wip-19826

mon: do not prime_pg_temp creating pgs; clean up pg create conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>