os/bluestore/BlueFS: fix overwrite

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix writes spanning extents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reenable rocksdb recycling

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: lock device while open

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: debug read result

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix alignment check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: check aio return values

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: avoid lock during reads

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: prevent read+write sharing

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug bluefs and rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: periodically compact log

Rewrite only the current metadata in a fresh log
periodically to free log space.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: simplify extent list

Merge contiguous extents.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix read

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: trivial init fix

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: rocksdb_separate_wal_dir option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: ref count BlueFS::File *

There are FileWriters that exist when the file is
deleted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: readdir list dirs, too

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluefs-tool: simple tool to export bluefs content

Currently we just do a dump.  We'll add more
functionality later.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: many fixes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: share space with BlueFS

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: move to simple mutex model

Just for now, while we get the rest of this working.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: simple file system to back rocksdb

BlueFS is a simple file system that will back rocksdb.
BlueRocksEnv is the rocksdb::Env implementation that
glues them together.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: less verbose

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: less verbose on hash collision test

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: fix locking

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: fix misc bugs

Can't use invalid iterator; fix init_rm_free.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/Allocator: init_rm_free

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: take custom Env

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read return value

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix read return value

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: separate Allocator from freelist storage

FreelistManager perists our freelist.  Allocator is a policy that
allocates it.

Signed-off-by: Sage Weil <sage@redhat.com>

newstore -> bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: always create db.wal

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: create db dir

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: consume a raw block device

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: make collection_list tolerate sloppy start position

Because of this change (#6076), the hobject_t will contain pool id, hence
the ghobject_t having this hobject_t will be not equal to ghobject_t().

In newstore, this will cause assertion failure:
FAILED assert(k >= start_key && k < end_key)

The fix is to make compatible with previous change to create a
ghobject_t object with pool id and shard id in newstore.

Fixes: #13801
Reported-by: Zhi Zhang <zhangz.david@outlook.com>
Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: make key names more efficient

- pack u32 and u64 in binary (instead of in hex)
- avoid duplicating the object name while making things still
  sort by (key,name).  Use < when key < name, = when key == name,
  > when key > name) as a prefix.  And in the = case (which is
  basically always) include the name just once.

Note that this breaks on-disk compatibility.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix collection_list vs max entries

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: do not set/change frag_size if there are overlays

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: define a fid_backpointer_t type

Signed-off-by: Sage Weil <sage@redhat.com>
fix wal_oP_t

os/newstoer: add newstore types to ceph-dencoder

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: set alloc hint on new frags

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: dump onode contents

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fixed fragment size

Instead of a single, variable-length fragment for each object,
set a fixed size (newstore_min_frag_size = 1 MB) and stripe the
object over these.  The last fragment will be smaller
than 1 MB if the object is not a multiple of 1 MB.

On write, this is basically free: we can just as cheaply write
4 inodes created together and fsync them than we can one.  On
overwrite, it allows us to replace individual fragments and avoid
write-ahead many cases.

On read it is a bit slower because of inode lookups and disk
seeks.  In the common case (big object written sequentially) we
hope that fs prefetching will hide most of it (e.g., all inodes
will be loaded together in the same metadata btree node, and the
files' data is written sequentially on disk).

Allowing for a singe large fragment in the case of a sequentially
written large object may save us something, but it complicates
the code significantly.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: recycle rocksdb log files

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: latest master

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6649 from majianpeng/filesstore-lfnunlink

osd: FileStore:: optimize lfn_unlink

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7017 from efirs/ef_atomic_ceph_tid

osd: use atomic to generate ceph_tid

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7077 from XinzeChi/wip-fix-wip-perf

osd: fix wip (l_osd_op_wip) perf counter and remove repop_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6893 from kylinstorage/wip-osd_command

librados: add c++ style osd/pg command interface

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5630 from wonzhq/evict-after-flush

osd: try evicting after flushing is done

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6639 from xiexingguo/xxg-wip-13822

librados: potential null pointer access in list_(n)objects

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6702 from liewegas/wip-fix-recency

osd/ReplicatedPG: fix promotion recency logic

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6824 from Sandy4999/wip-crushtool-build

crushtool: set type 0 name "device" for --build option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6962 from liewegas/wip-buffer-lastp

buffer: fix internal iterator invalidation on rebuild, get_contiguous

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6970 from aiicore/drop_removal_pg_type

osd: drop deprecated removal pg type

Reviewed-by: Sage Weil <sage@redhat.com>

osd: remove repop_map in osd

If I do not misread, repop_map is useless.

Signed-off-by: Xinze Chi <xinze@xsky.com>

osd: fix wip (l_osd_op_wip) perf counter

The l_osd_op_wip is for osd, so it should be the sum of all pgs in osd

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6987 from H3C/wip-addr-bugfix

common/address_help.cc: fix the leak in entity_addr_from_url()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'origin/jewel'

common/address_help.cc: fix the leak in entity_addr_from_url()

Fixes: #14132
Signed-off-by: Qiankun Zheng <zheng.qiankun@h3c.com>

Merge pull request #7026 from xdonghai/master

rbd: must specify both of stripe-unit and stripe-count when specifying stripingv2 feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6998 from xiexingguo/xxg-wip-clsrbd

stringify outputted error code and fix unmatched parentheses.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6983 from xiexingguo/xxg-wip-14126

librbd: small fixes for error messages and readahead counter

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6298 from guangyy/wip-13441

osd: make list_missing query missing_loc.needs_recovery_map

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6572 from liewegas/wip-crush-chooseleaf-stable

crush: add chooseleaf_stable tunable

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7043 from dillaman/wip-14170-jewel

librbd: do not ignore self-managed snapshot release result

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: do not ignore self-managed snapshot release result

Fixes: #14170
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7042 from dillaman/wip-14164-jewel

librbd: properly handle replay of snap remove RPC message

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7041 from dillaman/wip-14165

qa/workunits: merge_diff shouldn't attempt to use striping

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7040 from dillaman/wip-14092-jewel

librbd: ensure librados callbacks are flushed prior to destroying

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7036 from xiexingguo/xxg-wip-kvstore

KeyValueStore: fix return code of mkfs

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #7020 from chenyehua11692/master

doc:adding "--allow-shrink" in decreasing the size of the rbd block to distinguish from the increasing option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7035 from dillaman/wip-14122-jewel

librbd: clear error when older OSD doesn't support image flags

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: properly handle replay of snap remove RPC message

Fixes: #14164
Backport: infernalis
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits: merge_diff shouldn't attempt to use striping v2

Fixes: #14165
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure librados callbacks are flushed prior to destroying image

Fixes: #14092
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

KeyValueStore: fix return code of mkfs

It shall return a negative result code instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librbd: clear error when older OSD doesn't support image flags

Fixes: #14122
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6996 from H3C/wip-mds-f11322

mds: we should wait messenger when MDSDaemon suicide

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7015 from chipitsine/master

ceph-fuse: fix double free of args

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7004 from jazeltq/master

doc: fix typo

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6870 from mslovy/wip-pgmeta-object

osd: os: skip checking pg_meta object existance in FileStore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6744 from majianpeng/scrub-fix

osd: release related sources when scrub is interrupted

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6961 from liewegas/wip-lockdep-max

common/lockdep: increase max lock names

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: we should wait messenger when MDSDaemon suicide

Signed-off-by: Wei Feng <feng.wei@h3c.com>

Merge pull request #7025 from tchaikov/wip-ceph-detect-init-py3

ceph-detect-init: fix py3 test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-detect-init: fix py3 test

* in python3, None can not be compared with a str. and
  we should always set codename with a non empty str, so update
  TestCephDetectInit.test_debian so it always set a code name.
  and assert(codename and distroname) in choose_init().

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7023 from tchaikov/wip-doc-cachemode

doc: document "readforward" and "readproxy" cache mode

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd:must specify both of stripe-unit and stripe-count when specify striping features

when create a rbd image with striping features, Be sure specify both of stirpe-unit and stripe-count too, If not, refuse to excute and give a message

Signed-off-by:Donghai Xu <xu.donghai@h3c.com>

doc: document "readforward" and "readproxy" cache mode

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc:adding "--allow-shrink" in decreasing the size of the rbd block to distinguish from the increasing option

In the original file, the increasing and decreaing of the size of the rbd block shares the same option:
    "rbd resize --size 2048 foo".
However, it is not proper, as the "--allow-shrink" needs to be added while decreasing the size of the rbd block.
As a result, it is necessary to make a distinguish between these two options as follows:
    "rbd resize --size 2048 foo (to increase)"
    "rbd resize --size 2048 foo --allow-shrink (to decrease)"

Signed-off-by: Yehua <chen.yehua@h3c.com>

ceph doc fix slip of pen

mon client hung interval -> mon client hunt interval

Signed-off-by: tianqing <tianqing@unitedstack.com>

ceph-fuse: fix double free of args

Checking src/ceph_fuse.cc...
[src/ceph_fuse.cc:55]: (error) Memory pointed to by 'argv' is freed twice.
[src/ceph_fuse.cc:55]: (error) Deallocating a deallocated pointer: argv

Signed-off-by: Ilya Shipitsin <chipitsine@gmail.com>

Merge pull request #6988 from xiexingguo/xxg-wip-14134

FileJournal: fix return code of create method

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #7005 from YankunLi/patch-4

delete default zone

Merge pull request #7006 from YankunLi/patch-5

correct radosgw-admin command

Merge pull request #6997 from zhouyuan/evict_check_range

osd: cache tier: add config option for eviction check list size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6990 from linuxbox2/master-fixes

Fixes some small issues

Reviewed-by: Kefu Chai <kchai@redhat.com>