Merge pull request #12604 from ommoreno/wip-16966

os/bluestore: allow multiple SPDK BlueStore OSD instances

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12618 from Liuchang0812/fix-blue-store-unused-var

os/bluestore: remove a never read value

os/bluestore: remove a never read value

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12556 from Adirl/support_roceV2

msg/async/rdma: Support for RoCE v2 and SL

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge remote-tracking branch 'gh/kraken'

Merge pull request #12603 from liewegas/wip-fix-rwordered

osd: fix CEPH_OSD_FLAG_RWORDERED

Reviewed-by: Samuel Just <sjust@redhat.com>

osd: fix CEPH_OSD_FLAG_RWORDERED

Broke this in 3a2ed3b136de68192f661b6992cbb96ce6aafd20.

Signed-off-by: Sage Weil <sage@redhat.com>

Added "bluestore" prefix to new config options

Signed-off-by: Orlando Moreno <orlando.moreno@intel.com>

msg/async/rdma: Support for RoCE v2 and SL

Adding GID support to ceph.conf
Search configured GID in GIDs table with specific RoCE version.
Use it instead of index 0 (default)
Add service level support to supprt PFC

Change-Id: I5369bdeb6797e1c440110bdb4ad8e54ee6db9857
Signed-off-by: Oren Duer <oren@mellanox.com> Adir Lev <adirl@mellanox.com>

Merge pull request #12598 from ly798/typo-fix-in-docs

doc: typo fix in s3_compliance

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12601 from rishabhkumar296/ceph12684

msg/simple: Remove dead code in pipe.cc

Reviewed-by: Sage Weil <sage@redhat.com>

Remove dead code in pipe.cc

Fixes: http://tracker.ceph.com/issues/12684
Signed-off-by: Rishabh Kumar <kris.kr296@gmail.com>

doc: typo fix in s3_compliance

Signed-off-by: LiuYang <yippeetry@gmail.com>

Merge pull request #12587 from yuriw/wip-fix-subset-script-master

qa: fixed script to schedule rados and other suites with --subset option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

changed script name
added kraken
parameterized suite name to be able to use for rados, rbd etc.

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #12590 from athanatos/wip-ping-write-ordered

osd: force watch PING to be write ordered

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12593 from wjwithagen/wip-wjw-freebssd-SO_reuseaddr

msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12584 from mkoderer/openstack_nova_doc

doc: mention corresponding libvirt section in nova.conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

osd/: treat PINGs as RWORDERED

89fd030bf9436dc4e37cc3a0f935ec077455d9d5 switched them to show up
as reads to avoid logging them, but we still pipeline them with
reconnects.  Thus, also force them to be rwordered.

Fixes: http://tracker.ceph.com/issues/18310
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12540 from dmick/wip-18297

ceph.in: allow 'flags' to not be present in cmddescs

Merge pull request #12577 from liewegas/wip-osd-backfill

qa/tasks/osd_backfill.py: wait for osd.[12] to start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12574 from liewegas/wip-bluestore-split

os/bluestore: preserve source collection cache during split

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

DOC: Mention corresponding section in nova.conf

rbd_user and rbd_secret_uuid are part of the libvirt configuration.

Signed-off-by: Marc Koderer <marc@koderer.com>

Merge pull request #12472 from dillaman/wip-18244

librbd: remove image header lock assertions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12532 from ifed01/wip-bluestore-large-onode-test

os/bluestore: add UT for an estimation of Onode in-memory size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12505 from ceph/wip-jlayton-submount

client: set metadata["root"] from mount method when it's called with …

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12570 from wjwithagen/patch-9

common/str_list.h: fix clang warning about std::move

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12541 from liewegas/wip-bluestore-omap-flush

os/bluestore: include modified objects in flush list even if onode unchanged

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/tasks/osd_backfill.py: wait for osd.[12] to start

...before sending a tell command.  Otherwise osd.2 might
start without 1, the io unblocks, and the tell fails
because osd.1 is still down.

Fixes: http://tracker.ceph.com/issues/18303
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow multiple SPDK BlueStore OSD instances

Fixes: http://tracker.ceph.com/issues/16966
Signed-off-by: Orlando Moreno <orlando.moreno@intel.com>

os/bluestore: preserve source collection cache during split

OSD split transactions look something like

 mkcoll new
 split old
 ...
 omap_rmkey_range old
 omap_setkeys old
 omap_setkeys new

The last part splits the log into two pieces.  The
problem is that the rmkey_range needs to wait on old
omap transactions to flush, and those are linked to the
old onode, and split clears the cache.  The result is
that we don't wait, rmkeyrange leaves some recent pg log
keys behind, and on OSD restart we get an error because
the object doesn't belong to the (old) collection.

Fix this by preserving objects in the old collection and
only clear out objects that are moving to the newly
split collections.  This will include the pgmeta object
that we care about.

(Note that we are one step closer to preserving the
cache contents across the split, but not quite there
yet: at this point we don't have all of the destination
collections.  A change in the ObjectStore interface is
probably needed to make that not be extremely awkward.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12571 from athanatos/wip-15943

osd: Fix map gaps again (bug 15943)

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12572 from liewegas/wip-python-rados-test

qa/workunits: include extension for nose tests

Merge pull request #12568 from ifed01/wip-bluestore-onode-diet2

os/bluestore: reduce Onode in-memory footprint

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12555 from socketpair/usr_bin

systemd: Fix startup of ceph-mgr on Debian 8

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12355 from chhabaramesh/bit_alloc_mempool

os/bluestore: simplified allocator interfaces to single apis

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits: include extension for nose tests

When you have a relative path you have to include the extension.
Weird.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12412 from ceph/wip-jlayton-suid

client/mds: Clear setuid bits when writing or truncating

Clang/str_list.h: fix clang warning about std::move

/home/jenkins/workspace/ceph-master/src/include/str_list.h:99:10: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
  return std::move(str_vec);
         ^
/home/jenkins/workspace/ceph-master/src/include/str_list.h:99:10: note: remove std::move call here
  return std::move(str_vec);
         ^~~~~~~~~~       ~
1 warning generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12565 from trociny/wip-18247-fixup

common: fix clang compilation error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: remove duplicate sbid member for Blob, use one from shared_blob member
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: declare a member for Blob's cached bufferlist if explicitly enabled by CACHE_BLOG_BL macro only

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: recalculate shared blob key instead of keeping it in memory
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: include modified objects in flush list even if onode unchanged

We use the onode flush list so that we can ->flush() as
a barrier before doing any read/modify/write.  For
example, omap_rmkeyrange will flush before reading to
see what keys to erase in order to ensure that any
previous inserts are applied to the db and we see them
and remove them.

However, some omap operations don't update the onode
itself, which means write_onode() doesn't get called and
we aren't put on this list.

Add a note_modified_object() helper that can be called
instead of write_onode() for those cases.  That way we
get on the list and flush() works as expected.

We could have resolved this by just putting ourselves on
the dirty onode list, but in practice every OSD op is
writing omap keys to the pgmeta object and there is no
need to touch the onode key in this case, so doing so
would be a big regression.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12548 from Liuchang0812/add-document-rgw

doc: explain rgw_fcgi_socket_backlog in rgw/config-ref.rst

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12561 from kylinstorage/wip-fix-osdmaptool-zero

osdmaptool: fix divide by zero error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12497 from dillaman/wip-18243

journal: prevent repetitive error messages after being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

remove unused wait argument and related function versions

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

common: fix clang compilation

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osdmaptool: fix divide by zero error

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

Merge pull request #12558 from Adirl/fix_bad_len_msg

msg/async/rdma: fix bad message that went to the user

Reviewed-by: Haomai Wang <haomai@xsky.com>

zone block offset bug fix and related unit test case

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>
zone block offset related test case

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

allocat_extent api rename and tests for new api

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

remove cont alloc alloc_blocks calls from allocator code and test

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

consolidate allocator calls to single alloc_extent api

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

allocator min_alloc arguments in call stack and tests

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

allocator alloc_extents min_alloc

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore:Bluefs balance should use alloc_extent api with bluefs_alloc_size

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

msg/async/rdma: fix bad message that went to the user

1. the printed value of "bad length" was incorrect, because 'r' was changed before the log line.
2. log appeared to user as error even though everything was calm; hence, reducing its severity

issue: 914840

Change-Id: I41520d27fc53fbbe07b61dd915e16448b87769d5
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Fix startup of Ceph cluster manager daemon on Debian 8

Signed-off-by: Mark Korenberg <socketpair@gmail.com>

Merge pull request #12531 from dachary/wip-18291-bluestore

ceph-disk: do not create bluestore wal/db partitions by default

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-disk: do not create bluestore wal/db partitions by default

Fixes: http://tracker.ceph.com/issues/18291
Signed-off-by: Loic Dachary <loic@dachary.org>

doc: explain rgw_fcgi_socket_backlog in rgw/config-ref.rst

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12458 from Liuchang0812/add-more-log-for-network-error

osd: enhance logging for osd network error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

ods: enhance logging for osd network error

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12343 from liewegas/wip-bluestore-release

os/bluestore: simplify allocator release flow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12351 from xiexingguo/xxg-wip-bluestore-1207

os/bluestore: clear omap flag if parent has none

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12353 from xiexingguo/xxg-wip-bluestore-1207-01

os/bluestore: don't implicitly create the source object for clone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12466 from kylinstorage/wip-remove-unused-class-declaration

remove unused declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' into wip-remove-unused-class-declaration

ceph.in: allow 'flags' to not be present in cmddescs

the 'flags' key in the command description dict may not be
present on older monitors.  Check for existence before
using it.

Introduced in 61d63433dfdcc08ecc21ba4f645794da5e01823c

Fixes: http://tracker.ceph.com/issues/18297
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12538 from liewegas/wip-workunits

qa/tasks/workunit: retry on ceph.git if checkout fails

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12534 from liewegas/wip-truncate-fix

osd/PrimaryLogPG: don't truncate if we don't have to for WRITEFULL

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/workunit: retry on ceph.git if checkout fails

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12486 from dillaman/wip-18247

librbd: keep rbd_default_features setting as bitmask

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12514 from vasukulkarni/wip-cd-shaman

qa/tasks/ceph_deploy: use dev option instead of dev-commit

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: don't truncate if we don't have to for WRITEFULL

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #10160 from jmunhoz/bugfix-16463-presigned-url

rgw: aws4: add presigned url bugfix in runtime

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12530 from liewegas/wip-bluestore-ceph-disk

ceph-disk: adjust bluestore default device sizes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: add mempool dump to Onode size tracking test case

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add an estimation for Onode in-memory size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12516 from hjwsm1989/fix-rpg

osd/PG: fix the unreadable log in choose_acting

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12521 from kylinstorage/wip-for-coverity

osd/osdmap: fix divide by zero error

Reviewed-by: Sage Weil <sage@redhat.com>

client: set metadata["root"] from mount method when it's called with a pathname

Currently, we only set the root properly config file or the
--client_metadata command line option. If a userland client program
tries to call ceph_mount with a pathname, it's not being properly
set.

Since we already hold the mutex, we can just update it directly.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #12507 from ifed01/wip-bluestore-fix-cache

os/bluestore: fix target_buffer value overflow in Cache::trim()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12525 from trociny/wip-18261

rbd: fix json formatting for image and journal status output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-disk: default DB size of 20GB

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: default WAL size of 576 MB

Enough for 2 256MB write buffers, and then some.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: keep rbd_default_features setting as bitmask

Support both human readable, comma delimited list of feature
names and also integer bitmask value. Attempting to read the
setting will always result in the feature bitmask integer
value.

This is required to avoid breaking backwards compatibility with
librbd clients that are dependent on the older behavior (e.g.
OpenStack).

Fixes: http://tracker.ceph.com/issues/18247
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12407 from majianpeng/rbd-stripe-unit

rbd : make option --stripe-unit w/ B/K/M work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12484 from runsisi/wip-fix-rollback-warn

librbd: ignore error when object map is already locked by current client

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: fix json formatting for image and journal status output

Fixes: http://tracker.ceph.com/issues/18261
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: make option --stripe-unit w/ B/K/M work.

As 'man rbd' said:  --stripe-unit size-in-B/K/M. But w/ B/K/M, the
command failed and said invalid options. So fix it.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdmap: fix divide by zero error

*** CID 1397255:  Incorrect expression  (DIVIDE_BY_ZERO)
/home/brad/working/src/ceph/src/osd/OSDMap.cc: 2995 in
OSDMap::summarize_mapping_stats(OSDMap*, const std::set<long,
std::less<long>, std::allocator<long>> *,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char>> *, ceph::Formatter *) const()
2989         f->open_object_section("utilization");
2990       if (newmap) {
2991         if (f) {
2992           f->dump_unsigned("moved_pgs", moved_pg);
2993           f->dump_unsigned("total_pgs", total_pg);
2994         } else {
>>>     CID 1397255:  Incorrect expression  (DIVIDE_BY_ZERO)
>>>     In expression "(float)moved_pg * 100. / (float)total_pg",
division by expression "total_pg" which may be zero has undefined
behavior.
2995           ss << "moved " << moved_pg << " / " << total_pg
2996       << " (" << ((float)moved_pg * 100.0 / (float)total_pg) <<
"%)\n";
2997         }
2998       }
2999       if (f) {
3000         f->dump_float("avg_pgs", avg_pg);

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

  osd/PG: fix the unreadable log in choose_acting
          To avoid the log like: failed to fill position ^B
Signed-off-by: huangjun <hjwsm1989@gmail.com>

librbd: ignore error when object map is already locked by current client

otherwise when using rbd cli to rollback image with object-map feature
enabled, the following error message will be printed out on the screen,
which is confusing to users:
  librbd::object_map::LockRequest: failed to lock object map: (17) File exists

Fixes: http://tracker.ceph.com/issues/16179
Signed-off-by: runsisi <runsisi@zte.com.cn>

rgw: aws4: add rgw_s3_auth_aws4_force_boto2_compat conf option

Runtime bugfix to handle presigned urls computed with canonical requests using
the port number once.

Boto2 computes canonical requests using the port number twice although it
should be used once only. This behaviour is a bug supported by AWS S3. Boto2 is
used in RGW S3 as reference implementation.

The client-side tools not supporting this boto2 bug will fail although they
should work too.

In order to support both presigned url implementations this patch adds a config
option to compute a second signature. With this option disabled, the code will
compute two signatures when the first signature is not valid. The aws4 auth
succeed if some of the two signatures is valid.

The config option rgw_s3_auth_aws4_force_boto2_compat, is enabled by default so
one signature, working with boto2, is computed only.

Fixes: http://tracker.ceph.com/issues/16463
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #12463 from wjwithagen/wip-wjw-freebsd-osd-avoidports

FreeBSD/OSD.cc: add client_messenger to the avoid_ports set.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

use dev option instead of dev-commit

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>