Merge pull request #11641 from vshankar/rbd-data-pool-assert

librbd: possible assert failure creating image when using data pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11570 from tchaikov/wip-revise-SubmittingPatches

doc: reformat SubmittingPatches with more rst syntax

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11630 from cbodley/wip-rgw-curl-upgrade

rgw: remove suggestion to upgrade libcurl
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11385 from bassamtabbara/wip-rocksdb-external-project

cmake: use external project for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: remove ending period in section titles

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11653 from ktdreyer/doc-epel-instructions

doc: standardize EPEL instructions

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: use external project for rocksdb

This commit makes rocksdb an external cmake project. This ensures
that rocksdb is no longer built from src/rocksdb, and that make clean
works correctly.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

pickup new rocksdb with support for linux cmake

This picksup a new version of rocksdb with support for
linux cmake.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11619 from dachary/wip-17662-ceph-disk

ceph-disk: allow using a regular file as a journal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11652 from ktdreyer/doc-mgr-syntax

doc: fix mgr literal block rST syntax

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: assert failure when using data pool

This fixes a silly assert that's hit during image creation
(cli/api) when the data pool specified is same as the pool
specified by -p/--pool option (or the default).

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: standardize EPEL instructions

Prior to this change, the documentation pages contained different ways
to enable EPEL. Pick a simple, secure (https) way and standardize on
that.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: fix mgr literal block rST syntax

Prior to this change, there were double-colon ("::") characters in the
HTML docs to indicate literal blocks.

The proceeding paragraphs were not pre-formatted because there was no
newline between the double-colon and the to-be-pre-formatted paragraph.

Move the "::" characters inline to simplify the syntax and make Sphinx
properly interpret these texts as literal blocks.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11594 from cbodley/wip-rgw-init-bucket-sync

rgw: fix for passing temporary in InitBucketSyncStatus

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11505 from cbodley/wip-17571

rgw: clean up RGWShardedOmapCRManager on early return

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11504 from cbodley/wip-17568

rgw: RGWSimpleRadosReadCR tolerates empty reads

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11352 from dachary/wip-17526-rados-bench

doc: document hostname constraints for rados bench

reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #11602 from dillaman/wip-17665

doc: clarify QEMU rbd cache settings

reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #9825 from liewegas/wip-addrvec

msg: add entity_addr_t types; add new entity_addrvec_t type

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11517 from ukernel/wip-17562

Fix #17562 (backtrace check fails when scrubbing directory created by fsstress)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11593 from ukernel/wip-17275

osdc/ObjectCacher: wake up dirty stat waiters after removing buffers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11604 from stiopaa1/mds_inotable_addOverride

mds/InoTable.h: add override to virtual functs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11613 from stiopaa1/mds_cdir_removeUnneededUseofCount

mds/CDir.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11611 from batrick/i17670

mds: check if down mds is known

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11191 from renhwztetecs/renhw-wip-add-libcephfs

libcephfs: add ceph_fsetattr&&ceph_lchmod&&ceph_lutime

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11560 from ukernel/wip-mds-context-wrapping

MDS: reduce usage of context wrapper

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11345 from jcsp/wip-mgr-assert

messages: fix out of range assertion

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11632 from VictorDenisov/group_pool_fix

rbd: fix parsing of group and image specific pools

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: fix parsing of group and image specific pools

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

Merge pull request #11036 from tomy-ac/wip-tc-rados_append

Added append functionality to rados tool.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11221 from stiopaa1/osd_osdmap_removeUseOfCount

osd/OSDMap.cc: remove unneeded use of count

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11388 from ukernel/wip-17177

os/ObjectStore: properly clear object map when replaying OP_REMOVE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove suggestion to upgrade libcurl

Reported-by: Ken Dreyer <kdreyer@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11143 from kylinstorage/wip-fix-write-hang

osd: fix rados write op hang

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11532 from javeme/run_in_thread-return

pybind: avoid "exception 'int' object is not iterable"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11610 from liewegas/wip-mon-forward-features

messages/MForward: reencode forwarded message if target has differing features

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/Message: fix set_middle vs throttler

Signed-off-by: Sage Weil <sage@redhat.com>

msg: adjust byte_throttler from Message::encode

Normally we never call encode on a message that has a byte_throttler set
because we only use it for messages we received.  However, for forwarded
messages that we clear_payload() before resending, we *do* reencode, and in
that case we need to retake the appropriate number of bytes from the
throttler--just like we release them in clear_payload().

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MForward: reencode forwarded message if target has differing features

This ensures we reencode the payload with the
appropriate set of features if the client, us, or the
target do not have identical features.  Otherwise we
may forward an encoding with more features than the
target can handle.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11595 from liewegas/wip-move-ranges

os: fix offsets for move_ranges operation

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11160 from VictorDenisov/consistency_groups_namespaces

librbd: add support for snapshot namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-disk: allow using a regular file as a journal

Because of a missing return, ceph-disk prepare would fail if given a
regular file as a journal. If the journal file does not exist, ceph-disk
will create it but fail to ensure that the ceph user owns it. The
symlink to the journal file is not set when the journal file is
specified on the command line and the journal file does not exist at
all. The ceph-osd daemon will silently create it as a file but it will
not be the file given in argument.

Add a test case to verify using a regular file as a journal works as
expected.

Fixes: http://tracker.ceph.com/issues/17662
Signed-off-by: Jayashree Candadai <jayaajay@indiana.edu>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

mds: fix compile error from merges

Two patches both added these definitions
in slightly different places, so git let them
merge without a conflict.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'ukernel/wip-mdstypes-split'

Merge pull request #11576 from ukernel/wip-17636-debug

mds: add debug assertion for issue #17636

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11578 from jcsp/wip-backtrace-repair

mds: repair backtraces during scrub

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11596 from liewegas/wip-osdmap-features

mon/OSDMonitor: encode OSDMap::Incremental with same features as OSDMap

Reviewed-by: Kefu Chai <kchai@redhat.com>

unittest_bluestore_types: fix csum const name

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-bluestore-per-pool-compress2' of https://github.com/ifed01/ceph

Merge pull request #8751 from flyd1005/wip-fix-typo

doc: fix typos

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10955 from objoo/YD-mailmap-updates-Sept

mailmap updates sept

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds/CDir.cc: remove unneeded use of count

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

mds: check if down mds is known

This avoids an assertion failure where an MDS receives an mdsmap that
causes it to enter up:replay and also see another MDS go down.

Fixes: http://tracker.ceph.com/issues/17670
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11607 from xiexingguo/xxg-wip-bluestore-1022

os/bluestore: simplify can_split_at()

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: literal changes
1. drop redundant semicolon
2. fix wrong output message

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/libcephfs: add ceph_fsetattr() test

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

cls_rbd: Allow snapshots with the same name in different namespaces

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Use user snapshot namespace in rbd_mirror for now

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: By default create snapshots in UserNamespace

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Add snapshot namespaces data structures

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

os/bluestore: simplify can_split_at()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11026 from jcsp/wip-pg-files

tools/cephfs: add pg_files command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11456 from tchaikov/wip-17558

mon: send updated monmap to its subscribers

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds/InoTable.h: add override to virtual functs

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11599 from tanghaodong25/wip-2q-cache

os/bluestore: make 2q cache kin/kout size tunable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11603 from liewegas/wip-bluestore-obj-remove

os/bluestore: clear extent map on object removal

Merge pull request #11597 from ifed01/wip-bluestore-reshard-fix

os/bluestore: fix lack of resharding

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: clear extent map on object removal

Clear ExtentMap (esp shards, etc.) when an object is removed.  Otherwise
if we recreate it we will have stale state (like the shards vector or
inline_bl) that are bogus.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: clarify QEMU rbd cache settings

http://tracker.ceph.com/issues/17665
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11230 from yehudasa/wip-rgw-resharding

rgw: bucket resharding
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: make 2q cache kin/kout size tunable

default kin/hot queue size is 50% of max buffer size.
50% is a better choice for the number of page slot in kout.
we make it tunable.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

os/bluestore: fix black of resharding

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

mon/OSDMonitor: encode OSDMap::Incremental with same features as OSDMap

The Incremental encode stashes encode_features, which is
what we use later to reencode the updated OSDMap.  Use
the same features so that the encoding will match!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not encode osdmap with MSG_ADDR2 until kraken flag is set

Signed-off-by: Sage Weil <sage@redhat.com>

os: fix offsets in move_ranges

This simplifies the interface, and avoids problems in
bluestore with alignment.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11566 from oritwas/wip-rgw-put-acl-underscore

rgw: fix put_acls for objects starting and ending with underscore

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11591 from liewegas/wip-bluestore-test-dev

ceph_test_objectstore: smaller device

Merge pull request #11444 from cbodley/wip-rgw-get-system-obj

rgw: get_system_obj does not use result of get_system_obj_state
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11588 from xiexingguo/wip-fix-fastinfo-id

osd: fix duplicated id of incompat feature "fastinfo"

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix for passing temporary in InitBucketSyncStatus

Fixes: http://tracker.ceph.com/issues/17661
Signed-off-by: Casey Bodley <cbodley@redhat.com>

osdc/ObjectCacher: wake up dirty stat waiters after removing buffers

Fixes: http://tracker.ceph.com/issues/17275
Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph_test_objectstore: smaller device

This is faster.  The bitmap has to be scanned during
startup and fsck and that's slow.

Signed-off-by: Sage Weil <sage@redhat.com>

messages: fix out of range assertion

When clang uses an 8 bit type for the enum, it
complains (out of range) if comparing <256,
and complains (tautological) if comparing <=256.

Avoid this by explicitly making the enum an
uint8_t, and just asserting that that it has
that size at the point that we assume so for
the encoding (in case someone modified the
type definition without checking how it was used).

Signed-off-by: John Spray <john.spray@redhat.com>

mds: repair backtraces during scrub

Fixes: http://tracker.ceph.com/issues/17639
Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix CDir::log_mark_dirty()

CDir::log_mark_dirty() moves dirfrag to current log segment's dirty
dirfrag list, but it does not submit any log event. Old log segments
(that include events which dirty the dirfrag) may get expired before
the dirfrag gets committed. If MDS crashes, the changes in expired
log segments get lost.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid wrapping contexts during logging

For each log event, mds allocate two extra contexts, one for
marking op tracker event, one for updating log's safe_pos after
executing the finish context. This is sub-optimization.

This patch defines MDSLogContextBase for log event context.
MDSLogContextBase::complete() function can do the extra jobs.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid wrapping C_IO_Wrapper with C_OnFinisher

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #11575 from ceph/wip-17260

cmake: link tests against static librados

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10490 from Yan-waller/yj-wip-msgtest-0729

msg/test: fix the guided compile-command to ceph_test_msgr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix duplicated id of incompat feature "fastinfo"

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11436 from chhabaramesh/master

kv/MemDB: making memdb code adapt to generic maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11455 from stiopaa1/mds_mutation_simplifyConstructors

mds/Mutation.h: simplify constructors

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11509 from stiopaa1/mds_fsmapuser_makeAssignCopyDefault

mds/FSMapUser.h: remove copy ctr and assign op

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10359 from songbaisen/b11

mon:  paxos add the timeout function when peon recovery

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11165 from xiexingguo/xxg-wip-throttle-0921

common/throttle: simplify Throttle::_wait()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11583 from stiopaa1/mds_snapserver_addOverride

mds/SnapServer.h: add override to virtual functs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11523 from stiopaa1/mds_migrator_removeUnneededCount

mds/Migrator.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11543 from batrick/cdir-unique-ptr

mds: use unique_ptr to simplify resource mgmt

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11581 from yehudasa/wip-17635

rgw: handle empty POST condition

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11565 from stiopaa1/mds_sessionmap_removeUnneededFunction

mds/SessionMap.h: remove unneeded function

Reviewed-by: Greg Farnum <gfarnum@redhat.com>