os/bluestore: avoid comparing object key and name twice

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use id of shard_id_t for encoding

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill struct DeferredCsum

Which is dead, and kill some other dead lines too.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11267 from liewegas/wip-bluestore-ref-map

os/bluestore/bluestore_types: uint64_t for ref_map

ceph_test_objecstore: use 1TB block device

Big enough to get into 32-bit LBAs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: uint64_t for ref_map

We use this to track raw extents on disk for SharedBlob.  We
also use it for the in-memory Blob with the blob namespace, but
using uint64_t intead of uint32_t doesn't hurt us there.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11266 from stiopaa1/osd_passSharedPtrByConstRef

osd: pass shared_ptr by const reference

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11137 from jcsp/wip-17173

mds: catch duplicates in DamageTable

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #11244 from xiexingguo/xxg-wip-bluestore-0928

os/bluestore: upgrade compression settings to atomics

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: upgrade compression settings to atomics

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: pass shared_ptr by const reference

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11185 from dillaman/wip-17355

rbd-mirror: quiesce in-flight event commits before shut down

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11250 from liewegas/wip-bluestore-leaks

os/bluestore: fix leaks in our use of rocksdb

os/bluestore: do not leak when using EnvMirror

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: include EnvMirror leak fixes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11180 from liewegas/wip-mon-forward

messages/MForward: fix encoding features

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

messages/MForward: fix encoding features

We were encoding the message with the sending client's
features, which makes no sense: we need to encode with
the recipient's features so that it can decode the
message.

The simplest way to fix this is to rip out the bizarre
msg_bl handling code and simply keep a decoded Message
reference, and encode it when we send.

We encode the encapsulated message with the intersection
of the target mon's features and the sending client's
features.  This probably doesn't matter, but it's
conceivable that there is some feature-dependent
behavior in the message encode/decode that is important.

Fixes: http://tracker.ceph.com/issues/17365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11243 from liewegas/wip-bluestore-deferred-csum

os/bluestore: remove deferred_csum machinery

Merge pull request #11113 from yuyuyu101/wip-fix-filestore-clone

os/filestore: disable use of splice by default

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9652 from stiopaa1/osdc_ObjectCacher_constMember

osdc/ObjectCacher.h: add const to member functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11248 from mikulely/cleanup-vstart-doc

doc: cleanup outdated radosgw description

Merge pull request #11217 from ceph/wip-17395-rbdmap-permissions

rpm: fix permissions for /etc/ceph/rbdmap

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11240 from liewegas/wip-bluestore-assert

os/bluestore: add assert to compress_extent_map

Merge pull request #11245 from xiexingguo/xxg-wip-bluestore-bsearch-shard

os/bluestore: binary search specified shard

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11235 from xiexingguo/xxg-wip-bluestore-0926

os/bluestore: more cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11247 from majianpeng/msg-async-cleanup

msg/async: remove unused code.

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: remove unused code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #10980 from trociny/wip-16213

rbd-mirror: replicate dynamic feature updates

Reviewed-by: Jason Dillaman <dillamana@redhat.com>

librbd: potential null pointer dereference when requesting exclusive lock

m_require_lock_on_read should be cleared when holding owner_lock.

For safety, also check that exclusive_lock is not null.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: interlock image state machine and update features operations

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: use counter to track exclusive lock block/unblock requests

It's possible the watch/notify message is duplicated resulting in two
concurrent block_requests() call.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: replicate dynamic feature updates

Fixes: http://tracker.ceph.com/issues/16213
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async state machine to enable/disable image features

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: templetize journal StandardPolicy

It will be needed for unit tests with a mocked ImageCtx.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: allow to call append_disabled for DisabledPolicy

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async methods to enable/disable mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async method to check journal tag owner

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async image set flags method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async methods to create/remove object map

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/rbd: alow to call set_features asynchronously

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/journal: async client_list method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11120 from renhwztetecs/renhw-wip-mds-server

mds/server: clean up handle_client_open()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11078 from ukernel/wip-17271

mds: remove fail-safe queueing replay request

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11225 from ukernel/wip-17392

client: protect InodeRef with client_lock

Reviewed-by: John Spray <john.spray@redhat.com>

doc: cleanup outdated radosgw description

radosgw support is on by default, as built-in Civetweb, set up Apache
is no necessary any more.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

os/bluestore: add a counter to trace rejected compress ops

This is useful for choosing a more proper bluestore_compression_required_ratio.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make allocated space calc a little faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: binary search specified shard

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use std namespace for hex/dec output formatting

Because we use std::hex/dec everywhere in bluestore except these two places.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove fadvise_flags from WriteContext

Currently fadvise_flags is only used to check whether
buffered write is necessary, so there is no need
to keep it in the WriteContext as we have already
pre-calculated and kept the buffered field instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove dead local variables from BlueStore::_wctx_finish()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move nid assignment into BlueStore::_zero()

So other callers, such as BlueStore::_clone_range(),
won't do it twice.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove deferred_csum machinery

When we added this way back in d4f4fa0312d943dd0ce3c27f5fc56c7a753bb471,
we did not have our own buffer cache, and were relying
on the cache at the BlockDevice layer.  In that case,
we would have the problem of a partial wal overwrite
followed by another partial write that needed to read
the rest of the chunk.

However, now we have our own cache, and any data we write
in the _do_write_small() wal path will go into the cache,
which means we will never read the old data off of
disk and need the old csum values.

Remove this now-unnecessary kludge!

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: init g_ceph_context

The methods we're testing use it for dout.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add assert to compress_extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11238 from theanalyst/changelog-v10.2.3

doc/changelog: add v10.2.3

Merge pull request #10814 from liewegas/wip-bluestore-precondition

os/bluestore: precondition rocksdb/bluefs during mkfs

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

doc/changelog: add v10.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

os/bluestore: precondition rocksdb on mkfs

Write N bytes of garbage to the kv store on startup.  With rocksdb,
this ensures that our log files are preallocated.

This option needs to match up with the rocksdb tunables so that it
is enough data to start recycling log files.  For now, start with
128MB.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11234 from theanalyst/jewel-release-notes

doc/release-notes: v10.2.3 jewel

Merge pull request #11216 from liewegas/wip-bluestore-compress-map

os/bluestore: prevent extent merging across shard boundaries

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

doc/release-notes: more fixes to 10.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10445 from rjfd/wip-15259

journal: increase concurrency/parallelism of journal recorder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/release-notes: v10.2.3 jewel

Adding release notes for the next jewel release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add v10.2.3 to releases table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11226 from zealoussnow/wip-0926

doc: missing "make vstart" in quick_guide.rst

Merge pull request #11228 from stiopaa1/osd_pg_removeUnneededCount1

osd/PG.cc: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11229 from liewegas/wip-compressor

cmake: remove more autotools hacks

Reviewed-by: Ali Maredia <amaredia@redhat.com>

ceph-rest-api: drop autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: drop autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

detect-build-env-vars.sh: drop autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: remove autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

doc: remove old .libs reference

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression_*: remove lib dir hackery

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10143 from majianpeng/bluefs-fix

os/bluestore/BlueFS: don't inc l_bluefs_files_written_wal if overwrite.

Merge pull request #11138 from jcsp/wip-17236

mds: handle blacklisting during journal recovery

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: missing "make vstart" in quick_guide.rst

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #11220 from stiopaa1/mds_mdsdaemon_CMDSTickclass

mds/MDSDaemon: move C_MDS_Tick class to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11224 from wjwithagen/patch-13

osd-fast-mark-down.sh: fix typo in variable assignments

Merge pull request #11212 from liewegas/wip-bluestore-spanning-leak

os/bluestore: drop unreferenced spanning blobs

Merge pull request #11223 from somnathr/wip-bluestore-leak-fix

os/bluestore: fix spanning blob leak from ~ExtentMap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11219 from xiexingguo/xxg-wip-fsck-0924

os/bluestore: fix fsck() won't catch stray shard sometimes

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: test: Fixed timeout problem in rbd_mirror_stress.sh

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: Fix race between journal flush and append events

Signed-off-by: Ricardo Dias <rdias@suse.com>

journal: make librados call async in ObjectRecorder

Signed-off-by: Ricardo Dias <rdias@suse.com>

client: protect InodeRef with client_lock

Fixes: http://tracker.ceph.com/issues/17392
Signed-off-by: Yan, Zheng <zyan@redhat.com>

osd-fast-mark-down.sh: fix typo in variable assignments

Assignments to variables do not want a '$' in front of them.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

journal: increase concurrency of journal recorder

Signed-off-by: Ricardo Dias <rdias@suse.com>

BlueStore: Fixed a memory leak

If a Blob is spanning is not getting deleted as spanning_blob_map
is holding one ref.Explicitly decreasing Blob ref from ExtentMap
destructor and erasing from spanning_blob_map is fixing the leak.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #11196 from tchaikov/wip-lttng-the-right-way

lttng: build the tracepoint provider lib from .c files in repo

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/PG.cc: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

rbd-mirror: wait for in-flight event commit notifications

Fixes: http://tracker.ceph.com/issues/17355
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: move AsyncOpTracker to common library

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds/MDSDaemon: move C_MDS_Tick class to .cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11090 from dillaman/wip-16974

rbd-mirror: force-promoted image will remain R/O until rbd-mirror daemon restarted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: fix dout of BlueStore::fiemap()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add a counter to trace checksum latency

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove a never reachable assert

As this assert is deprecated by the
  "else if (it->key() >= tail)"
checking lying ahead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: promote output to derr on catching bad/stray objects

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix fsck() won't catch stray shard sometimes

which is introduced by https://github.com/ceph/ceph/pull/11210/commits/dcfbc72fcbb48f4ced0c1a2d46a77f43bff38c9f

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>