osd: If ending on a head object get all of meta map

When ending on a head object, the head and snapshots would stay in
cleaned_meta_map until more maps arrive.  The problem as that
during a scrub an eviction could occur because scrubber.start
is already past the stray object(s) so range_intersects_scrub() is false.

Fixes: http://tracker.ceph.com/issues/23909
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add test cases for multiple copy pool and snapshot errors

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Prepare for second test and minor improvements

Check list-inconsistent-obj output
Check how many _scan_snap groupings
Use more general check for crashed osd(s)

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: process _scan_snaps() with all snapshots with head

Fixes: http://tracker.ceph.com/issues/22881
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix comment at end of scrub test scripts

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #21583 from yanghonggang/disable-cache

kv/RocksDBStore: fix rocksdb error when block cache is disabled

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21670 from xiexingguo/wip-23878

osd/OSDMap: more pg upmap fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21709 from liewegas/wip-23917

ceph_test_rados_api_aio: fix race with full pool and osdmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21641 from zy751713126/total_size_format

tools: correct total size formatting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21710 from neha-ojha/wip-cosbench-workloads

qa: add cosbench workloads and override teuthology default settings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/perf-basic: add cosbench write workload

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/perf-basic: override default teuthology settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/perf: add cosbench workloads

Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph_test_rados_api_aio: fix race with full pool and osdmap

We send ops until we get a EDQUOT, and then assert our next op to
a different object also gets EDQUOT.  However, if the second osd
doesn't have as new a map it may succeed.  Make sure the client has
the latest (mon) map, and thus the one marking the pool full, before
we send the second op.  That ensures the second OSD also has that
newer map and also returns EDQUOT.

Fixes: http://tracker.ceph.com/issues/23917
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21704 from liewegas/wip-rados-old-clients

qa/suites/rados/thrash-old-clients: add rbd tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21339 from guzhongyan/tp-cleanup

osd: drop unused osd_disk_tp related options

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21630 from neha-ojha/wip-rgw-pools

qa: modify rgw default pool names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-old-clients: add cls_rbd, rbd api tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/perf: override default teuthology settings

Signed-off-by: Neha Ojha <nojha@redhat.com>

kv/RocksDBStore: fix rocksdb error when block cache is disabled

Fixes: http://tracker.ceph.com/issues/23816
Signed-off-by: Yang Honggang <yanghonggang@umcloud.com>

Merge pull request #21703 from liewegas/wip-rados-old-clients

qa/suites/rados/thrash-old-clients: add hammer clients

qa/suites/rados/thrash-old-clients: add hammer

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21679 from liewegas/wip-rados-old-clients

qa/suites/rados/thrash-old-clients: do some thrashing with jewel and luminous clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21605 from liewegas/wip-osdmap-encode

mon: cache reencoded osdmaps

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

messages/MOSDMap: fix compat_version for MOSDMap

Jewel clients see
 will not decode message of type 41 version 4 because compat_version 4 > supported version 3

The problem is when compat_version == 0 and head_version != 0, we
interpret compat_version == head_version.  From Message::encode,

    // if the encoder didn't specify past compatibility, we assume it
    // is incompatible.
    if (header.compat_version == 0)
      header.compat_version = header.version;

Broken by 49833c3bb264949b8126796997a95a95b50af411 for the
head_version=4 case (compat_version needs to be 3).

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: clean up member init

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDMap: significant feature bits.

1. MOSDMap go with features indicating the features
this map encoded for.

2. Only reencode if significant bits mismatch between
target features and my features.

also update mon/OSDMonitor and osd/OSDService to adopt this change.

Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

mon/OSDMonitor: add feature into osdmap cache key.

Change the cache from
SimpleLRU<version_t, bufferlist> inc_osd_cache;
to
SimpleLRU<pair<version_t, uint64_t>, bufferlist> inc_osd_cache;

By doing this , we can cache several encoded osdmap for different
features, not necessary to re-encoding for each client.

Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

Merge pull request #21693 from LenzGr/mgr-dashboard-documentation

doc: Updated dashboard feature list.

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21586 from trociny/wip-deep-copy-fixes

 librbd: skip head object map update when deep copying object beyond image size

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Updated dashboard feature list.

Updated the dashboard documentation and added recently merged
RBD management features (flatten, clone, copy)

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21681 from Devp00l/wip-rbd-form-type-error-fix

mgr/dashboard: Fixes type error in RBD form

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21682 from dillaman/wip-23888

rbd-mirror: rename asok hook to match image name when not replaying

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21657 from dillaman/wip-23876

rbd-mirror: properly translate remote tag mirror uuid for local mirror

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21662 from tchaikov/wip-cmake

cmake: enable new policies to silence cmake warnings

Review-by: Casey Bodley <cbodley@redhat.com>

osd/OSDMap: check pending pg upmap changes too

Signed-off-by: xiexingguo <xie.xingguo@gmail.com>

osd/OSDMap: check against crush-tree changing

pg_upmap/pg_upmap_items should not bring osds which
no longer belong to a specific crush-tree in.

Signed-off-by: xiexingguo <xie.xingguo@gmail.com>

osd/OSDMap: more sanity checks before applying pg_upmap[_items]

Fixes: http://tracker.ceph.com/issues/23878
Signed-off-by: xiexingguo <xie.xingguo@gmail.com>

osd/OSDMap: refactor maybe_remove_pg_upmaps()

There is too much code redundancy..

Signed-off-by: xiexingguo <xie.xingguo@gmail.com>

Merge pull request #21649 from dillaman/wip-23872

osdc/Objecter: prevent double-invocation of linger op callback

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21656 from neha-ojha/wip-integrate-cosbench

qa/tasks: run cosbench using the CBT task

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks: run cosbench using the CBT task

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #21676 from liewegas/wip-23871

osd/PrimaryLogPG: clear data digest on WRITEFULL if skip_data_digest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21686 from jdurgin/wip-ec-backend-comment

osd/ECBackend: update misleading comment about EIO handling

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21653 from liewegas/wip-23364

osd/PrimaryLogPG: avoid infinite loop when flush collides with write lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21222 from bmflynn/pybind-alignment

pybind/rados: add alignment getter to IoCtx

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ECBackend: update misleading comment about EIO handling

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21061 from jan--f/jan-prometheus-file-sd-command

pybind/mgr/prometheus: add file_sd_config command

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21646 from liewegas/wip-rados-api-tests

qa/suites/rados/verify/tasks/rados_api_tests: whitelist OBJECT_MISPLACED

rbd-mirror: rename asok hook to match image name when not replaying

Fixes: http://tracker.ceph.com/issues/23888
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fixes type error in RBD form

The form validation function wasn't a 'ValidatorFn' because it didn't
return something, now it's returning null in order to be valid for
type checking engines that also type check functions that are returned
by other functions.

Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #19509 from cooboos/wip-cephdencoder-additem

ceph-dencoder: add missing begin_iter & end_iter item for RGWObjManifest

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20835 from croit/fix-mon-crash-on-bad-snap-rm

mon: handle bad snapshot removal reqs gracefully

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21093 from guzhongyan/mgr-ov

mgr/DaemonServer: add overrides value to 'config show'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21585 from xxhdx1985126/wip-23807

msgr/simple: set Pipe::out_seq to in_seq of the connecting side

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: clear data digest on WRITEFULL if skip_data_digest

If we detect we are on bluestore and should skip the data digest, we
need to clear any old digest on WRITEFULL.  This aligns us with the
WRITE behavior, which is also either an update or a clear (but never
neither!).

Fixes: https://tracker.ceph.com/issues/23871
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21667 from ricardoasmarques/wip-fix-delete-modal

mgr/dashboard: Fix missing $event on deletion modal

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21584 from oritwas/wip-rgw-resharding-versioning

rgw: Fix infinite call for bi list when resharding a bucket

mgr/dashboard: Fix missing $event on deletion modal

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21658 from tchaikov/wip-msg-many-failure

qa: reduce "mon client hunt interval max multiple" to 2 for all clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19491 from yaarith/wip-mgr-module-hello

pybind/mgr: new 'hello world' mgr module skeleton

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21529 from ricardoasmarques/wip-fix-rbd-form

mgr/dashboard: Display RBD form errors on submission

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21591 from zhouyuan/fix_wo_mgr_build

cmake: fix building without mgr module

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: enable CMP0051

the new behavior of get_target_property(<var> <target> SOURCES) will be
enforced in future versions of cmake, so let ready for this change now
by removing the generator expressions from the returned source file
list.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: enable CMP0046

so cmake will error on "non-existent dependency in add_dependencies", we
add dependency on CONFIGURE_FILE() output, which is wrong, as its output
is not a target, and is generated when cmake runs. so remove them from
dependencies.

regarding to ceph_ver.h, its path is also wrong. it is created under
${CMAKE_BINARY_DIR}/src/include. so this is another reason to remove it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21526 from ricardoasmarques/wip-rbd-snapshot-clone

mgr/dashboard: RBD copy, RBD flatten and snapshot clone (frontend)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21558 from votdev/cd_error_panel

mgr/dashboard: Add 'cd-error-panel' component to display error messages

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #21612 from Songweibin/wip-pglog-cleanup

osd/PGLog: fix sanity check against **complete-to** iter

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21608 from cbodley/wip-dpp

dout: DoutPrefixProvider operates directly on stream

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20480 from smithfarm/wip-23035

build/ops: run-make-check.sh: handle Python 2 absence

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21629 from liewegas/wip-23840

os/bluestore/bluefs_types: make block_mask 64-bit

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

rbd-mirror: edge-condition for promote/demote during daemon failover

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: reduce "mon client hunt interval max multiple" to 2 for all clients

because with high failure rate, we need to connect to mon more
frequently if the connection fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: drop unused osd_disk_tp related options

disk_tp has been killed. So those are unused.

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

Merge pull request #21639 from guzhongyan/pglog_clean

osd/PrimaryLogPG: Fix log messages

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd-mirror: properly translate remote tag mirror uuid for local mirror

Fixes: http://tracker.ceph.com/issues/23876
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: avoid infinite loop when flush collides with write lock

We try to take a write lock with fop->op.  If we fail, fop->op is put
on the lock's waiting list.  Requeuing it again will simply kick off
processing of another instance of the same op, which will again fail
to take the lock, leading to an infinite loop that can't terminate
because requeue_op is doing a push_front and preventing other PG
messages that might release the lock.

Do the same write lock attempt on any dup_ops so that they too will
end up on the wait list.

It looks like this broke waaay back in commit d700d99f76e0a29bfb419bc85d19ef1950b62a9a,
a 2014 refactor of the OpContext behavior.

Fixes: https://tracker.ceph.com/issues/23664
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21645 from alfredodeza/wip-rm23451

ceph-volume include physical devices associated with an LV when listing

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #21650 from ricardoasmarques/wip-install-deps-tumbleweed

install-deps: Add support for 'opensuse-tumbleweed'

Reviewed-by: Nathan Cutler <ncutler@suse.com>

install-deps: Add support for 'opensuse-tumbleweed'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

osdc/Objecter: prevent double-invocation of linger op callback

Fixes: http://tracker.ceph.com/issues/23872
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21619 from liewegas/wip-merge-bits

mon/OSDMonitor: require --yes-i-really-mean-it for force-create-pg

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21628 from liewegas/wip-23646

osd/PrimaryLogPG: defer evict if head *or* object intersect scrub interval

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21338 from mogeb/iostat-output

mgr/iostat: print output as a table

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #21613 from xiexingguo/wip-fix-delete-priority

osd/PG: fix out of order priority for PG deletion

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21623 from liewegas/wip-21977

osd: publish osdmap to OSDService before starting wq threads

Reviewed-by: David Zafman <dzafman@redhat.com>

qa/suites/rados/verify/tasks/rados_api_tests: whitelist OBJECT_MISPLACED

The api tests do some splits, which can move data.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/ceph-volume include information for devices in listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

pybind/mgr: add 'hello world' mgr module skeleton

This simple 'hello' mgr module prints 'hello world' to stdout and to out/mgr.x.log; for documentation purposes. Also added /doc/mgr/hello.rst.

Signed-off-by: Yaarit Hatuka <yaarithatuka@gmail.com>

ceph-volume tests verify associated pvs reporting on listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update to include devices on listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing include associated physical devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21626 from cbodley/wip-23846

mgrc: free MMgrClose in handle_mgr_close

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21541 from dillaman/wip-18753

qa: introduce rbd-mirror thrasher to existing tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21632 from tchaikov/wip-rocksdb-no-aligned-alloc

rocksdb: do not use aligned_alloc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21479 from cbodley/wip-23547

rgw: add buffering filter to compression for fetch_remote_obj

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

pybind/mgr/prometheus: add file_sd_config command

This command returns a valid list of static targets pointing to the mgr
prometheus modules for a cluster. The output can be stored in a
file and the file be listed under the file_sd_config stanza in the
prometheus configuration.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21576 from cbodley/wip-23506

rgw: fix error handling for GET with ?torrent

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #21569 from cbodley/wip-23803

rgw: aws4 auth supports PutBucketRequestPayment

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #20212 from tianshan/fix_period_pull

rgw: do not reflect period if not current

Reviewed-by: Casey Bodley <cbodley@redhat.com>