os/bluestore: fix _split_collections race with osr_reap

The SharedBlobSet may not yet be empty at split time because previous
transactions may not have been reaped, leaving some Blobs still alive
even after the cache refs are cleared.  Drop them explicitly here (they
will go away shortly).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11726 from liewegas/wip-bluestore-fsck-omap

os/bluestore: fsck: fix omap_head check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck: fix omap_head check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11740 from xiexingguo/xxg-wip-bluestore-1102

os/bluestore: less code redundancy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11725 from liewegas/wip-bluestore-shard-span

os/bluestore: fix extent shard span check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11738 from ceph/wip-cmake

cmake: do not link erasure tests again libosd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11384 from wjwithagen/patch-14

test_subman.sh: Don't use --tmpdir

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test_subman.sh: Don't use --tmpdir

test_subman.sh: Don't use --tmpdir since on most Linux-is it is consideren default
when creating a directory:
`If TEMPLATE is not specified, use tmp.XXXXXXXXXX, and --tmpdir is implied.`

And on FreeBSD --tmpdir is not available anyways.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: use string for most cases in _open_db()

For a better readability.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: do not link erasure tests again libosd

they are not referencing any symbols from it. and also the tests testing
plugins are not linked against the ec plugins, they *depend* on them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11730 from ceph/wip-17654

librados-dev: install inline_memory.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: drop unused parameter "deep" from _fsck_check_extents()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/ceph_objectstore_tool: handle fsck and fsck-deep op in unified path

https://github.com/ceph/ceph/pull/11724/commits/181e28ba999c4134957f04183bad5c4dfc0e7062
introduces too much code redundancy, this patch simplifies it a little.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librados-dev: install inline_memory.h

This is needed to compile against the c++ api since
c38869232c3c852d98fa1826632db360c5a6afd4

Fixes: http://tracker.ceph.com/issues/17654
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11697 from weiqiaomiao/wqm-wip-rgw-bilist

rgw: fix osd crashes when execute "radosgw-admin bi list --max-entries=1" command

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11695 from tanghaodong25/cache-logger-wip

os/bluestore: update cache logger after 'trim_cache' operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11506 from cbodley/wip-17569

rgw: clear data_sync_cr if RGWDataSyncControlCR fails

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11707 from yehudasa/wip-17755

rgw: json encode/decode index_type, allow modification

os/bluestore: rename variable in set_lextent

Defensive naming

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix shard span check in set_lextent

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: check for extents spanning shards

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: udpate cache logger after trim_cache

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #11708 from nitinkamble/master

install-deps.sh: allow building on SLES systems

Merge pull request #10767 from rzarzynski/wip-rgw-frontend-rework

rgw: frontend subsystem rework

all requested changes have been made by the author
tested by-hand w/sigv2 and sigv4, and already passed teuthology rgw suite

Merge pull request #11724 from liewegas/wip-bluestore-fsck

os/bluestore: deep fsck

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: verify we can read objects on deep fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os: add fsck deep/shallow option

deep==false checks just metadata, and deep==true will (eventually) check
all data (and checksums) too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix synthetic fsck

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11537 from liewegas/wip-bluestore-parallel

os/bluestore: refactor bluestore_sync_submit_transaction

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #11661 from ceph/wip-git-ver

cmake: fix git version string, cleanup

os/bluestore: default to bluestore_sync_submit_transaction=true

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: randomly inject serialize transaction submission

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: move main() to bottom

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix {nid,blobid}_max

- single case to cover increases ahead of schedule or just in time
- update global max only after txn commits, eliminating race
- drop unneeded id_lock
- improve t naming

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prevent sync transaction submit with extent freelist

It doesn't support unordered transactions.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit txn via kv_sync_thread if ids exceed max

This ensures the txn will not commit before an update to the
global max.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: refactor bluestore_sync_submit_transaction

And drop bluestore_sync_transaction, at least for now.

The key change here is to make a per-txc flag indicating
whether the txn was already submitted.  This allows us
to make a choice between sync and not-sync on a per-txn
basis.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11721 from dachary/wip-17561-log

tests: TestErasureCodePluginJerasure must stop the log thread

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11718 from xiexingguo/xxg-wip-bluestore-1101

os/bluestore: add counter to trace blob splitting

Reviewed-by: Sage Weil <sage@redhat.com>

tests: TestErasureCodePluginJerasure must stop the log thread

When a log entry is created in dout.h, it holds a pointer to the

   static size_t _log_exp_length

variable. When OnExitManager::~OnExitManager from Log.cc is called
during global destruction, it may happen after the static variables
referenced by some entries have been deallocated. The flush() function
will try to access these deallocated variables via the hint_size()
method and core dump.

To prevent this race, the Log::stop() method must be called explicitly
to ensure no entries are left when the global destructors are called.

Fixes: http://tracker.ceph.com/issues/17561
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10248 from ktdreyer/wip-doc-no-rpm-md

doc: rm "type=rpm-md" from yum repositories

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11690 from yonghengdexin735/wip-zzz-fixword

src/common/perf_counters.h: fix wrong word

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check when compress_extent_map()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting changes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add counter to trace blob splitting

As we want to know if it really happens and how often does it happen.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11568 from dillaman/wip-17588

librbd: cannot access non-primary image when mirroring force disabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11280 from ceph/wip-17440

rgw: fixes for virtual hosting of buckets
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11559 from dillaman/wip-17618

librbd: exclusive lock incorrectly initialized when switching to head revision

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11623 from dillaman/wip-cls-rbd-group

librbd: ensure consistency groups will gracefully fail on older OSDs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11703 from dillaman/wip-17750

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11714 from dachary/wip-17757-eio

tests: disable failing tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

tests: disable failing unittest_erasure_code_plugin_jerasure

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17561

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: disable failing test-erasure-eio.sh

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17757

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11382 from yuyuyu101/wip-fix-shadow

osd/PGBackend: fix collection_list shadow return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
tchaikov

Merge pull request #11547 from javeme/check-pgid

osd/command tell: check pgid at the right time

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11203 from stiopaa1/osd_pg_replaceCountPart1

osd/PG.cc: prevent repeated searching of map/set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11242 from stiopaa1/osd_replicatedpg_removeCount1

osd/ReplicatedPG: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11365 from badone/wip-lsb_release-remove

common: Remove the runtime dependency on lsb_release

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Fixes: http://tracker.ceph.com/issues/17750
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: allow building on SLES systems

Avoids this error on SLES systems:

> ./install-deps.sh
sles is unknown, dependencies will have to be installed manually.

Signed-off-by: Nitin A Kamble <Nitin.Kamble@Teradata.com>

rgw: json encode/decode index_type, allow modification

Fixes: http://tracker.ceph.com/issues/17755
We didn't dump the index_type field of RGWBucketInfo, also didn't
allow modifying it via metadata api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: rm "type=rpm-md" from yum repositories

As far as I can tell, this is only relevant to SUSE's Zypper, and yum
does not need it.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

librbd: unit test cases and minor fixes for mirror::DisableRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/journal: use librados::IoCtx reference for helper state machines

This facilitates being able to properly mock test cases around journal
cls method calls.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: restore journal access when force disabling mirroring

If mirroring is force disabled on a demoted image, the journal was
being left in an inconsistent ownership state.

Fixes: http://tracker.ceph.com/issues/17588
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new async journal promote state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new async journal open state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move journal tag decode helpers to shared location

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: re-add support for disabling a corrupt journal

Fixes: http://tracker.ceph.com/issues/16740
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

make-dist: do not use make_version (and remove it)

Signed-off-by: Sage Weil <sage@redhat.com>

make-dist: do not generate ceph_ver.h

cmake does this on its own, and puts it in a different location (the build
dir).

Signed-off-by: Sage Weil <sage@redhat.com>

src/CMakeLists.txt: remove leading 'v' from version string

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11249 from wido/osd-flush-journal-shutdown

osd: Flush Journal on shutdown

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11686 from ceph/wip-ceph-rest-api-new-addr

ceph-rest-api: understand the new style entity_addr_t representation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11530 from markhpc/wip-rocksdb_tuning

common/config_opts.h: Optimized RocksDB WAL settings.

rgw: fix osd crashes when execute "radosgw-admin bi list --max-entries=1" command

Fixes: http://tracker.ceph.com/issues/17745
Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

Merge pull request #11688 from stiopaa1/mds_snapserver_addConstToMemFun

mds/SnapServer: add const to member function

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds/SnapServer: add const to member function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

ceph-rest-api: understand the new style entity_addr_t representation

the new entity_addr_t representation was introduced by 3561eb3, we
should update ceph_rest_api.py accordingly.

Fixes: http://tracker.ceph.com/issues/17742
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10993 from Vicente-Cheng/fix-alloc-choose_tries

crush: make counting of choose_tries consistent

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11246 from badone/wip-set_xattr_limits_via_conf-fix

os/filestore: Fix erroneous WARNING: max attr too small

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11614 from runsisi/wip-fix-PK11_FreeSymKey-segfault

auth: fix NULL pointer access when trying to delete CryptoAESKeyHandler instance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11654 from hjwsm1989/fix

mon: print the num_pools and num_objects in 'ceph -s -f json/json-p…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11639 from atheism/mgr-PyModules-remove-duplicated-else-if

mgr: PyModules.cc: remove duplicated if condition for fs_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11659 from tchaikov/wip-delete-cct-later

librados: postpone cct deletion

Reviewed-by: Sage Weil <sage@redhat.com>

src/common/perf_counters.h:fix wrong word

Signed-off-by: zhang.zezhu <zhang.zezhu@zte.com.cn>

Merge pull request #11314 from jcsp/wip-mgr-flake8

pybind: clean up mgr stuff for flake8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11650 from wjwithagen/wip-wjw-log-loging

Log: Replace namespace log with logging

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11680 from tchaikov/wip-17728

test/ceph_test_msgr: do not use Message::middle for holding transient…

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11397 from dzafman/wip-scrub-misc

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/ceph_test_msgr: do not use Message::middle for holding transient data

Message::middle is used for holding encoded data, so we we can not stuff
it with payload and leave the "payload" field empty. this change
refactors the ceph_test_msgr by introducing a Payload class which
encodes all test data in it.

Fixes: http://tracker.ceph.com/issues/17728
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11677 from ceph/wip-rocksdb-no-march-native

rocksdb: do not build with --march=native

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11678 from tchaikov/wip-silence-rbd-warning

librbd: silence the unused variable warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: silence the unused variable warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

rocksdb: do not build with --march=native

we can not assume that building host has the same arch (or the same cpu
capability) with the target machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11663 from stiopaa1/mds_mdslog_addConstToMemFunc

mds/MDLog: add const to member functions

Merge pull request #11668 from Enproduktion/master

doc: Fixed mapping error in legacy mds command

Merge pull request #11446 from ukernel/wip-cephfs-scan-links

tools/cephfs: add scan_links command which fixes linkages errors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11609 from stiopaa1/mds_snaprealm_unneededuseofcount

mds/SnapRealm: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11625 from stiopaa1/mds_damageTable_removeUnneededCount

mds/DamageTable.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>