os/bluestore: verify we can read objects on deep fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os: add fsck deep/shallow option

deep==false checks just metadata, and deep==true will (eventually) check
all data (and checksums) too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix synthetic fsck

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11537 from liewegas/wip-bluestore-parallel

os/bluestore: refactor bluestore_sync_submit_transaction

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #11661 from ceph/wip-git-ver

cmake: fix git version string, cleanup

os/bluestore: default to bluestore_sync_submit_transaction=true

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: randomly inject serialize transaction submission

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: move main() to bottom

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix {nid,blobid}_max

- single case to cover increases ahead of schedule or just in time
- update global max only after txn commits, eliminating race
- drop unneeded id_lock
- improve t naming

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prevent sync transaction submit with extent freelist

It doesn't support unordered transactions.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit txn via kv_sync_thread if ids exceed max

This ensures the txn will not commit before an update to the
global max.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: refactor bluestore_sync_submit_transaction

And drop bluestore_sync_transaction, at least for now.

The key change here is to make a per-txc flag indicating
whether the txn was already submitted.  This allows us
to make a choice between sync and not-sync on a per-txn
basis.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11721 from dachary/wip-17561-log

tests: TestErasureCodePluginJerasure must stop the log thread

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11718 from xiexingguo/xxg-wip-bluestore-1101

os/bluestore: add counter to trace blob splitting

Reviewed-by: Sage Weil <sage@redhat.com>

tests: TestErasureCodePluginJerasure must stop the log thread

When a log entry is created in dout.h, it holds a pointer to the

   static size_t _log_exp_length

variable. When OnExitManager::~OnExitManager from Log.cc is called
during global destruction, it may happen after the static variables
referenced by some entries have been deallocated. The flush() function
will try to access these deallocated variables via the hint_size()
method and core dump.

To prevent this race, the Log::stop() method must be called explicitly
to ensure no entries are left when the global destructors are called.

Fixes: http://tracker.ceph.com/issues/17561
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10248 from ktdreyer/wip-doc-no-rpm-md

doc: rm "type=rpm-md" from yum repositories

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11690 from yonghengdexin735/wip-zzz-fixword

src/common/perf_counters.h: fix wrong word

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check when compress_extent_map()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting changes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add counter to trace blob splitting

As we want to know if it really happens and how often does it happen.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11568 from dillaman/wip-17588

librbd: cannot access non-primary image when mirroring force disabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11280 from ceph/wip-17440

rgw: fixes for virtual hosting of buckets
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11559 from dillaman/wip-17618

librbd: exclusive lock incorrectly initialized when switching to head revision

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11623 from dillaman/wip-cls-rbd-group

librbd: ensure consistency groups will gracefully fail on older OSDs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11703 from dillaman/wip-17750

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11714 from dachary/wip-17757-eio

tests: disable failing tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

tests: disable failing unittest_erasure_code_plugin_jerasure

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17561

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: disable failing test-erasure-eio.sh

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17757

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11382 from yuyuyu101/wip-fix-shadow

osd/PGBackend: fix collection_list shadow return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
tchaikov

Merge pull request #11547 from javeme/check-pgid

osd/command tell: check pgid at the right time

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11203 from stiopaa1/osd_pg_replaceCountPart1

osd/PG.cc: prevent repeated searching of map/set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11242 from stiopaa1/osd_replicatedpg_removeCount1

osd/ReplicatedPG: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11365 from badone/wip-lsb_release-remove

common: Remove the runtime dependency on lsb_release

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Fixes: http://tracker.ceph.com/issues/17750
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: rm "type=rpm-md" from yum repositories

As far as I can tell, this is only relevant to SUSE's Zypper, and yum
does not need it.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

librbd: unit test cases and minor fixes for mirror::DisableRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/journal: use librados::IoCtx reference for helper state machines

This facilitates being able to properly mock test cases around journal
cls method calls.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: restore journal access when force disabling mirroring

If mirroring is force disabled on a demoted image, the journal was
being left in an inconsistent ownership state.

Fixes: http://tracker.ceph.com/issues/17588
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new async journal promote state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new async journal open state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move journal tag decode helpers to shared location

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: re-add support for disabling a corrupt journal

Fixes: http://tracker.ceph.com/issues/16740
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

make-dist: do not use make_version (and remove it)

Signed-off-by: Sage Weil <sage@redhat.com>

make-dist: do not generate ceph_ver.h

cmake does this on its own, and puts it in a different location (the build
dir).

Signed-off-by: Sage Weil <sage@redhat.com>

src/CMakeLists.txt: remove leading 'v' from version string

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11249 from wido/osd-flush-journal-shutdown

osd: Flush Journal on shutdown

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11686 from ceph/wip-ceph-rest-api-new-addr

ceph-rest-api: understand the new style entity_addr_t representation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11530 from markhpc/wip-rocksdb_tuning

common/config_opts.h: Optimized RocksDB WAL settings.

Merge pull request #11688 from stiopaa1/mds_snapserver_addConstToMemFun

mds/SnapServer: add const to member function

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds/SnapServer: add const to member function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

ceph-rest-api: understand the new style entity_addr_t representation

the new entity_addr_t representation was introduced by 3561eb3, we
should update ceph_rest_api.py accordingly.

Fixes: http://tracker.ceph.com/issues/17742
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10993 from Vicente-Cheng/fix-alloc-choose_tries

crush: make counting of choose_tries consistent

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11246 from badone/wip-set_xattr_limits_via_conf-fix

os/filestore: Fix erroneous WARNING: max attr too small

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11614 from runsisi/wip-fix-PK11_FreeSymKey-segfault

auth: fix NULL pointer access when trying to delete CryptoAESKeyHandler instance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11654 from hjwsm1989/fix

mon: print the num_pools and num_objects in 'ceph -s -f json/json-p…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11639 from atheism/mgr-PyModules-remove-duplicated-else-if

mgr: PyModules.cc: remove duplicated if condition for fs_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11659 from tchaikov/wip-delete-cct-later

librados: postpone cct deletion

Reviewed-by: Sage Weil <sage@redhat.com>

src/common/perf_counters.h:fix wrong word

Signed-off-by: zhang.zezhu <zhang.zezhu@zte.com.cn>

Merge pull request #11314 from jcsp/wip-mgr-flake8

pybind: clean up mgr stuff for flake8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11650 from wjwithagen/wip-wjw-log-loging

Log: Replace namespace log with logging

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11680 from tchaikov/wip-17728

test/ceph_test_msgr: do not use Message::middle for holding transient…

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11397 from dzafman/wip-scrub-misc

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/ceph_test_msgr: do not use Message::middle for holding transient data

Message::middle is used for holding encoded data, so we we can not stuff
it with payload and leave the "payload" field empty. this change
refactors the ceph_test_msgr by introducing a Payload class which
encodes all test data in it.

Fixes: http://tracker.ceph.com/issues/17728
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11677 from ceph/wip-rocksdb-no-march-native

rocksdb: do not build with --march=native

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11678 from tchaikov/wip-silence-rbd-warning

librbd: silence the unused variable warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: silence the unused variable warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: do not build with --march=native

we can not assume that building host has the same arch (or the same cpu
capability) with the target machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11663 from stiopaa1/mds_mdslog_addConstToMemFunc

mds/MDLog: add const to member functions

Merge pull request #11668 from Enproduktion/master

doc: Fixed mapping error in legacy mds command

Merge pull request #11446 from ukernel/wip-cephfs-scan-links

tools/cephfs: add scan_links command which fixes linkages errors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11609 from stiopaa1/mds_snaprealm_unneededuseofcount

mds/SnapRealm: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11625 from stiopaa1/mds_damageTable_removeUnneededCount

mds/DamageTable.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9304 from dzafman/wip-13937

osd: handle EC recovery read errors

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11645 from dillaman/wip-librbd-discard-after-write

test: new librbd discard after write test case

Merge pull request #11670 from stiopaa1/mds_mutation_addConstToMemFun

mds/Mutation.h: add const to member functions

Merge pull request #11608 from atheism/mds-duplicated-check

client: Client.cc: remove duplicated op type checking against CEPH_MD…

client: Client.cc: correct the op type checking for CEPH_MDS_OP_CREATE

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

global: Always log loading erasure code for tests

Fix for broken test-erasure-code.sh and test-erasure-eio.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test.sh: Make check for flags more robust

Low space broke test, saw "flags nearfull,pauserd,pausewr...."

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Remove extra objectstore_tool call which causes a recovery

Caused by: 70e000a9a42c50eda48f5d3b9e432ffc3a70f75b

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Handle object removals in a non-racey way

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix hang on unfound object after mark_unfound_lost is done

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Handle recovery read errors

Fixes: http://tracker.ceph.com/issues/13937
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log messages

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: CLEANUP: Remove unused pending_read member

Remove unused struct

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "test: Disable tests due to recovery race"

This reverts commit 5bc55338f5a1645bc651811fae2f89ad855ff86e.

Conflicts:
src/test/Makefile.am (no longer exists)
src/test/erasure-code/Makefile.am (no longer exists)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11582 from batrick/ceph-dumpable

core: set dumpable flag after setuid

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds/Mutation.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

doc: Fixed mapping error in legacy mds command

Signed-off-by: Malte Fiala <maltefiala@produktion.io>

Merge pull request #11666 from tchaikov/wip-cmake

cmake: remove include/Makefile.am

cmake: remove include/Makefile.am

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: Fix erroneous WARNING: max attr too small

When we read the conf values from the conf file we call handle_conf_change which
calls set_xattr_limits_via_conf but this is before m_fs_type is initialised so
we fall through to the default comaprison with
filestore_max_xattr_value_size_other (1024) and erroneously issue the warning.
This happens at every startup but subsequent calls once the filestore is mounted
proceed as expected. Don't call set_xattr_limits_via_conf unless the filestore
is mounted.

Fixes: http://tracker.ceph.com/issues/17420
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11638 from tchaikov/wip-clang

include/enc: make clang happy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11658 from hjwsm1989/doc

doc: change the osd_max_backfills default to 1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds/MDLog: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10887 from michaelsevilla/mantle

fs: Mantle: A Programmable Metadata Load Balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11641 from vshankar/rbd-data-pool-assert

librbd: possible assert failure creating image when using data pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/cmdparse,msg/async: remove unnecessary std::move()

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: postpone cct deletion

RefCountedObject::put() prints log using cct, but RadosClient
relinquishes the reference to cct too early, so when its
`RefCountedObject` member variables print log when they are
destructed, the cct is already freed. hence we have a segfault.

to address this problem, we need postpone destructure of it after all
other RefCountedObject member variables are destroyed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11570 from tchaikov/wip-revise-SubmittingPatches

doc: reformat SubmittingPatches with more rst syntax

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>