osd: Add "trigger_scrub" admin socket command

This allows testing to fake a schedule scrub (must_scrub not set)

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit c8dc945260ee2ee841aca00fbc802d41036033d6)

test: Add test for keeping deep-scrub information

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 64a7012e986ec88994c073b738fd08e8958395c3)

osd: When deep-scrub errors present upgrade regular scrubs

Previously, if a weekly deep-scrub found errors the next daily scrub
would cause the deep-scrub information to be removed.

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 532a759dca466181f37a329f86045c34f1a2506f)

tasks/scrub_test.py: Make test deterministic by updating digests

ceph_test_rados: Get coverage of digest updates in deep-scrub/repair

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e000ab2f46b7b98a223176a2a25bb7195e601af6)

repair_test, scrub_test: Fix whitelists for scrub changes

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5fe8522df47f56842d227e08b2912623504afe24)

scrub_test: Fix for list-inconsistent-obj output changes

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5ae0f5c75a8236d4a026f9ddcb5ff59964d90579)

doc, test: Add schemas for list-inconsistent-* rados command output

If jsonschema cmd is available use it to test output against schema

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit eb73dd473600fbbb45fad00194f7c46b565d6b81)

test: Update testing for new list-inconsistent-obj output

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b7bacd219b000869b9c93e21edba4f8a3ace60d9)

Adapted for difference in Jewel object_info

rados, osd: Improve attrs output of list-inconsistent-obj

Persist the user_version and shard id of scrubbed obj
Rados command dump inconsistent obj's version and shard-id
    so they can be passed to repair command
Rados list-inconsistent-obj output of attrs
    Make attrs an array since there are more than one
    Use base64 encode for values with non-printable chars
    Add indication if base64 encoding used
Add checking for ss_attr_missing and ss_attr_corrupted
    Rename attr errors to attr_key_mismatch and attr_value_mismatch
Add missing size_mismatch_oi scrub checking
For erasure coded pools add ec_size_error and ec_hash_error not just read_error
Use oi_attr_missing and oi_attr_corrupted just like list-inconsistent-snap does
Pick an object info based on version and use that to find specific shards in error
    Check for object info inconsistency which should be rare
Make all errors based on comparing shards to each other object errors
    We don't want give the impression that we've picked the correct one

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit df3ff6dafeadb3822b35c424a890db9a14d7f60f)

osd: Fix logging to help with diagnostics

These messages were not outputing the intended information

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b39001ec6ff0996860bcb5a3578bc2c66355c781)

test: Fix use of wait_for_clean()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d0503a52d0fe5505bdb38dfd613c03a20500c05d)

common: Change cleanbin() to use base64 encoding, update ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 92e982c71995b863466d83671468f84761cb1793)

common: Move cleanbin() function to common/util.cc

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5c79074ffaee34b2956d9dfc67b1eff9f39b47f3)

Conflicts:
src/tools/CMakeLists.txt (changes goes in src/CMakeLists.txt)

test: Add test support for deep-scrub

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 907e79e2b77835d1aca9e8ba2fae5fa2fd437e5a)

common: Fix indentation

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a74418a29e4e469117fc0c4edd80f78b62944c98)

osd: Handle corrupt attributes in get_object_context()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 30f8b0d6593718dd10bcf6ff459c95b4bb68c05c)

ReplicatedPG::failed_push: release read lock on failure

and requeue the blocked ops.

Fixes: http://tracker.ceph.com/issues/17857
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b3224a18f6acc7ed54c2162b140a33b6146a16be)

test.sh: Make check for flags more robust

Low space broke test, saw "flags nearfull,pauserd,pausewr...."

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b4c080b1b4378d659c1ca8a17811cd6f84595166)

test: Remove extra objectstore_tool call which causes a recovery

Caused by: 70e000a9a42c50eda48f5d3b9e432ffc3a70f75b

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 6904529d1b53993304de2927500937ba0d493e9e)

test: Handle object removals in a non-racey way

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit bfe3ebd94fdc1ef36ebe6e8f9d38acea322eca22)

osd: Fix hang on unfound object after mark_unfound_lost is done

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 73a27533beba3587f8447b4d41d200427c45042b)

Conflicts:
src/osd/ReplicatedPG.cc (trivial)

osd: Handle recovery read errors

Fixes: http://tracker.ceph.com/issues/13937
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit c51d70e1e837c972e42ddd5fa66f7ca4477b95cc)

Conflicts:
src/osd/ReplicatedPG.h (trivial)

osd: Fix log messages

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b40ec3fe890679b4c73778815e1bedd492cb264b)

osd: CLEANUP: Remove unused pending_read member

Remove unused struct

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 36fd68c96653e83f87767feb08530a9fc6e841b0)

test/osd-scrub-repair.sh: Use test case specific object names to help with diagnostics

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0bf4da589155ee50969812492cfbc66368efb54c)

Merge pull request #13299 from dachary/wip-18848-jewel

jewel: tests: remove qa/suites/buildpackages

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

buildpackages: remove because it does not belong

It should live in teuthology, not in Ceph. And it is currently broken:
there is no need to keep it around.

Fixes: http://tracker.ceph.com/issues/18846
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 5a43f8d57925da227c95480501ceec10a29395d8)

Conflicts:
qa/tasks/buildpackages/common.sh (remove)
qa/tasks/buildpackages/make-rpm.sh (remove)

Merge pull request #13280 from athanatos/wip-revert-jewel-18581

Revert "Merge pull request #12978 from asheplyakov/jewel-18581"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "Merge pull request #12978 from asheplyakov/jewel-18581"

See: http://tracker.ceph.com/issues/18809

This reverts commit 8e69580c97622abfcbda73f92d9b6b6780be031f, reversing
changes made to c05730ceb3387fb43c35937f0506297a34a44452.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #11872 from dachary/wip-17838-jewel

jewel: leak in RGWFetchAllMetaCR

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13182 from smithfarm/wip-18498-jewel

jewel: rgw: Realm set does not create a new period

Merge pull request #13180 from smithfarm/wip-18547-jewel

jewel: rgw: multisite: segfault after changing value of rgw_data_log_num_shards

Merge pull request #13177 from smithfarm/wip-18676-jewel

jewel: rgw: librgw: objects created from s3 apis are not visible from nfs mount point

Merge pull request #13175 from smithfarm/wip-18684-jewel

jewel: rgw: multisite: sync status reports master is on a different period

Merge pull request #13173 from smithfarm/wip-18710-jewel

jewel: rgw: slave zonegroup cannot enable the bucket versioning

Merge pull request #13171 from smithfarm/wip-18712-jewel

jewel: rgw: radosgw-admin period update reverts deleted zonegroup

Merge pull request #13001 from asheplyakov/jewel-bp/18559

jewel: rgw multisite: fix ref counting of completions

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12997 from asheplyakov/jewel-bp-18569

jewel: radosgw valgrind "invalid read size 4" RGWGetObj

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12764 from SUSE/wip-17342-jewel

jewel: tests: assertion failure in a radosgw-admin related task

Merge pull request #12738 from SUSE/wip-18286-jewel

jewel: rgw: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors

Merge pull request #12678 from Abhishekvrshny/wip-18348-jewel

jewel: rgw ldap: enforce simple_bind w/LDAPv3 redux

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12622 from wido/jewel-15776

jewel: rgw: log name instead of id for SystemMetaObj on failure

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12542 from dreamhost/wip-17076-jewel

jewel: rgw: Replacing '+' with "%20" in canonical uri for s3 v4 auth.

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12428 from SUSE/wip-18216-jewel

jewel: rgw-admin: missing command to modify placement targets

Merge pull request #12426 from SUSE/wip-18214-jewel

jewel: add max_part and nbds_max options in rbd nbd map, in order to keep consistent with

http://tracker.ceph.com/issues/17851#note-59

Merge pull request #12419 from SUSE/wip-18217-jewel

jewel: rgw sends omap_getvals with (u64)-1 limit

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12405 from jan--f/wip-18199-jewel

jewel: build/ops: install-deps.sh based on /etc/os-release

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12397 from linuxbox2/jewel-next

jewel: rgw: do not abort when accept a CORS request with short origin

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12316 from dachary/wip-18101-jewel

jewel: Add workaround for upgrade issues for older jewel versions

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12890 from dillaman/wip-18453-jewel

jewel: librbd: new API method to force break a peer's exclusive lock

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12302 from SUSE/wip-18135-jewel

jewel: build/ops: add hostname sanity check to run-{c}make-check.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12315 from dachary/wip-18102-jewel

jewel: rgw: Unable to commit period zonegroup change

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12314 from dachary/wip-18107-jewel

jewel: multisite: failed assertion in 'radosgw-admin bucket sync status'

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12313 from dachary/wip-18112-jewel

jewel: rgw: multisite requests failing with '400 Bad Request' with civetweb 1.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12258 from rzarzynski/wip-rgw-17931-jewel

jewel: rgw: add support for the prefix parameter in account listing of Swift API

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12156 from dachary/wip-17969-jewel

jewel: rgw: multisite upgrade from hammer -> jewel ignores rgw_region_root_pool

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11990 from asheplyakov/jewel-17908

jewel: rgw: for the create_bucket api, if the input creation_time is zero, we …

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11876 from dachary/wip-17839-jewel

jewel: rgw: the value of total_time is wrong in the result of 'radosgw-admin log show' opt

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11868 from dachary/wip-17512-jewel

jewel: multisite: metadata master can get the wrong value for 'oldest_log_period'

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11866 from dachary/wip-17709-jewel

jewel: rgw: multisite: coroutine deadlock assertion on error in FetchAllMetaCR

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11497 from rzarzynski/wip-rgw-slashinfo-jewel

jewel: rgw: add suport for Swift-at-root dependent features of Swift API

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11477 from dachary/wip-17119-jewel

jewel: rgw: multisite: assert(next) failed in RGWMetaSyncCR

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11476 from dachary/wip-17162-jewel

jewel: rgw: multisite doesn't retry RGWFetchAllMetaCR on failed lease

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11470 from dachary/wip-17514-jewel

jewel: rgw：bucket check remove _multipart_ prefix

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10661 from dachary/wip-16871-jewel

jewel: rgw: Have a flavor of bucket deletion in radosgw-admin to bypass garbage collection

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13187 from asheplyakov/jewel-bp-18729

jewel: cli: ceph-disk: convert none str to str before printing it

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13153 from smithfarm/wip-no-firefly-on-centos

jewel: tests: upgrade:hammer-x: install firefly only on Ubuntu 14.04

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13118 from smithfarm/wip-18274-jewel

jewel: Memory leaks in object_list_begin and object_list_end

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13025 from SUSE/wip-18605-jewel

jewel: ceph-disk prepare writes osd log 0 with root owner

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13106 from ceph/wip-cd-dev-option

qa/tasks: jewel backport -  ceph-deploy use of dev option

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #13062 from asheplyakov/jewel-bp-18379

jewel: msg/simple: clear_pipe when wait() is mopping up pipes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12978 from asheplyakov/jewel-18581

jewel: ReplicatedBackend: take read locks for clone sources during recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12755 from Abhishekvrshny/wip-18284-jewel

jewel: Need CLI ability to add, edit and remove omap values with binary keys

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12677 from SUSE/wip-18104-jewel

jewel: mon: ceph osd down detection behaviour

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12291 from asheplyakov/jewel-18108

jewel: msg/simple/Pipe: error decoding addr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11998 from Abhishekvrshny/wip-17877-jewel

jewel: FileStore: fiemap cannot be totally retrieved in xfs when the number of extents > 1364

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11991 from asheplyakov/jewel-17909

jewel: osd: limit omap data in push op

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12045 from Abhishekvrshny/wip-17875-jewel

jewel: rgw: file: remove spurious mount entries for RGW buckets

Merge pull request #13161 from smithfarm/wip-lfn-upgrade-hammer

jewel: tests: add require_jewel_osds before upgrading last hammer node

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12425 from SUSE/wip-18190-jewel

jewel: rbd-mirror: gmock warnings in bootstrap request unit tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13129 from smithfarm/wip-18558-jewel

jewel: rbd: bench-write will crash if --io-size is 4G

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13155 from smithfarm/wip-18494-jewel

jewel: rbd: [rbd-mirror] sporadic image replayer shut down failure

Merge pull request #13128 from smithfarm/wip-18633-jewel

jewel: rbd: [qa] crash in journal-enabled fsx run

http://tracker.ceph.com/issues/13512

Merge pull request #13104 from idryomov/wip-fio-unmap-devices-jewel

qa/tasks: backport rbd_fio fixes to jewel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13156 from smithfarm/wip-18455-jewel

jewel: rbd: Attempting to remove an image w/ incompatible features results in partial removal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13157 from smithfarm/wip-18434-jewel

jewel: rbd: Improve error reporting from rbd feature enable/disable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13168 from trociny/wip-18550-jewel

jewel: librbd: metadata_set API operation should not change global config setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12739 from SUSE/wip-18323-jewel

jewel: JournalMetadata flooding with errors when being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12416 from SUSE/wip-18219-jewel

jewel: msg: don't truncate message sequence to 32-bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13115 from smithfarm/wip-18404-jewel

jewel: mon: cache tiering: base pool last_force_resend not respected (racing read got wrong version)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13045 from SUSE/wip-18553-jewel

jewel: mon: peon wrongly delete routed pg stats op before receive pg stats ack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13117 from smithfarm/wip-18280-jewel

jewel: mon: osd flag health message is misleading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12320 from dachary/wip-18061-jewel

jewel: rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of js sdk.

Merge pull request #11759 from Abhishekvrshny/wip-17783-jewel

jewel: rgw: json encode/decode of RGWBucketInfo missing index_type field

Merge pull request #12153 from dachary/wip-18010-jewel

jewel: Cleanly reject session evict command when in replay

Merge pull request #12324 from dachary/wip-18103-jewel

jewel: truncate can cause unflushed snapshot data lose

Merge pull request #12783 from SUSE/wip-18413-jewel

jewel: cephfs: lookup of /.. in jewel returns -ENOENT

Merge pull request #12921 from xiaoxichen/wip-18520-jewel

Jewel: speed up readdir by skipping unwanted dn

Merge pull request #13119 from smithfarm/wip-18565-jewel

jewel: MDS crashes on missing metadata object

Merge pull request #13120 from smithfarm/wip-18551-jewel

jewel: ceph-fuse crash during snapshot tests