os/bluestore: include modified objects in flush list even if onode unchanged

We use the onode flush list so that we can ->flush() as
a barrier before doing any read/modify/write.  For
example, omap_rmkeyrange will flush before reading to
see what keys to erase in order to ensure that any
previous inserts are applied to the db and we see them
and remove them.

However, some omap operations don't update the onode
itself, which means write_onode() doesn't get called and
we aren't put on this list.

Add a note_modified_object() helper that can be called
instead of write_onode() for those cases.  That way we
get on the list and flush() works as expected.

We could have resolved this by just putting ourselves on
the dirty onode list, but in practice every OSD op is
writing omap keys to the pgmeta object and there is no
need to touch the onode key in this case, so doing so
would be a big regression.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12497 from dillaman/wip-18243

journal: prevent repetitive error messages after being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12558 from Adirl/fix_bad_len_msg

msg/async/rdma: fix bad message that went to the user

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: fix bad message that went to the user

1. the printed value of "bad length" was incorrect, because 'r' was changed before the log line.
2. log appeared to user as error even though everything was calm; hence, reducing its severity

issue: 914840

Change-Id: I41520d27fc53fbbe07b61dd915e16448b87769d5
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #12531 from dachary/wip-18291-bluestore

ceph-disk: do not create bluestore wal/db partitions by default

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-disk: do not create bluestore wal/db partitions by default

Fixes: http://tracker.ceph.com/issues/18291
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12458 from Liuchang0812/add-more-log-for-network-error

osd: enhance logging for osd network error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

ods: enhance logging for osd network error

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12343 from liewegas/wip-bluestore-release

os/bluestore: simplify allocator release flow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12351 from xiexingguo/xxg-wip-bluestore-1207

os/bluestore: clear omap flag if parent has none

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12353 from xiexingguo/xxg-wip-bluestore-1207-01

os/bluestore: don't implicitly create the source object for clone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12466 from kylinstorage/wip-remove-unused-class-declaration

remove unused declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' into wip-remove-unused-class-declaration

Merge pull request #12538 from liewegas/wip-workunits

qa/tasks/workunit: retry on ceph.git if checkout fails

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12534 from liewegas/wip-truncate-fix

osd/PrimaryLogPG: don't truncate if we don't have to for WRITEFULL

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/workunit: retry on ceph.git if checkout fails

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12486 from dillaman/wip-18247

librbd: keep rbd_default_features setting as bitmask

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12514 from vasukulkarni/wip-cd-shaman

qa/tasks/ceph_deploy: use dev option instead of dev-commit

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: don't truncate if we don't have to for WRITEFULL

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #10160 from jmunhoz/bugfix-16463-presigned-url

rgw: aws4: add presigned url bugfix in runtime

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12530 from liewegas/wip-bluestore-ceph-disk

ceph-disk: adjust bluestore default device sizes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12516 from hjwsm1989/fix-rpg

osd/PG: fix the unreadable log in choose_acting

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12521 from kylinstorage/wip-for-coverity

osd/osdmap: fix divide by zero error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12507 from ifed01/wip-bluestore-fix-cache

os/bluestore: fix target_buffer value overflow in Cache::trim()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12525 from trociny/wip-18261

rbd: fix json formatting for image and journal status output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-disk: default DB size of 20GB

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: default WAL size of 576 MB

Enough for 2 256MB write buffers, and then some.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: keep rbd_default_features setting as bitmask

Support both human readable, comma delimited list of feature
names and also integer bitmask value. Attempting to read the
setting will always result in the feature bitmask integer
value.

This is required to avoid breaking backwards compatibility with
librbd clients that are dependent on the older behavior (e.g.
OpenStack).

Fixes: http://tracker.ceph.com/issues/18247
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12407 from majianpeng/rbd-stripe-unit

rbd : make option --stripe-unit w/ B/K/M work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12484 from runsisi/wip-fix-rollback-warn

librbd: ignore error when object map is already locked by current client

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: fix json formatting for image and journal status output

Fixes: http://tracker.ceph.com/issues/18261
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: make option --stripe-unit w/ B/K/M work.

As 'man rbd' said:  --stripe-unit size-in-B/K/M. But w/ B/K/M, the
command failed and said invalid options. So fix it.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdmap: fix divide by zero error

*** CID 1397255:  Incorrect expression  (DIVIDE_BY_ZERO)
/home/brad/working/src/ceph/src/osd/OSDMap.cc: 2995 in
OSDMap::summarize_mapping_stats(OSDMap*, const std::set<long,
std::less<long>, std::allocator<long>> *,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char>> *, ceph::Formatter *) const()
2989         f->open_object_section("utilization");
2990       if (newmap) {
2991         if (f) {
2992           f->dump_unsigned("moved_pgs", moved_pg);
2993           f->dump_unsigned("total_pgs", total_pg);
2994         } else {
>>>     CID 1397255:  Incorrect expression  (DIVIDE_BY_ZERO)
>>>     In expression "(float)moved_pg * 100. / (float)total_pg",
division by expression "total_pg" which may be zero has undefined
behavior.
2995           ss << "moved " << moved_pg << " / " << total_pg
2996       << " (" << ((float)moved_pg * 100.0 / (float)total_pg) <<
"%)\n";
2997         }
2998       }
2999       if (f) {
3000         f->dump_float("avg_pgs", avg_pg);

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

  osd/PG: fix the unreadable log in choose_acting
          To avoid the log like: failed to fill position ^B
Signed-off-by: huangjun <hjwsm1989@gmail.com>

librbd: ignore error when object map is already locked by current client

otherwise when using rbd cli to rollback image with object-map feature
enabled, the following error message will be printed out on the screen,
which is confusing to users:
  librbd::object_map::LockRequest: failed to lock object map: (17) File exists

Fixes: http://tracker.ceph.com/issues/16179
Signed-off-by: runsisi <runsisi@zte.com.cn>

rgw: aws4: add rgw_s3_auth_aws4_force_boto2_compat conf option

Runtime bugfix to handle presigned urls computed with canonical requests using
the port number once.

Boto2 computes canonical requests using the port number twice although it
should be used once only. This behaviour is a bug supported by AWS S3. Boto2 is
used in RGW S3 as reference implementation.

The client-side tools not supporting this boto2 bug will fail although they
should work too.

In order to support both presigned url implementations this patch adds a config
option to compute a second signature. With this option disabled, the code will
compute two signatures when the first signature is not valid. The aws4 auth
succeed if some of the two signatures is valid.

The config option rgw_s3_auth_aws4_force_boto2_compat, is enabled by default so
one signature, working with boto2, is computed only.

Fixes: http://tracker.ceph.com/issues/16463
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #12463 from wjwithagen/wip-wjw-freebsd-osd-avoidports

FreeBSD/OSD.cc: add client_messenger to the avoid_ports set.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

use dev option instead of dev-commit

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #12500 from badone/wip-doc-PGPool

doc/dev/osd_internals: add pgpool.rst

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12511 from liewegas/wip-workunits

qa/workunits/rbd: fix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits: use relative path instead of wget from git

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12508 from liewegas/wip-qa-admin-socket

qa/tasks/admin_socket: subst in repo name

doc/dev/osd_internals: add pgpool.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: clone_range of len 0 creates but does not size object

This mimics, roughly,

 open(..., O_WRONLY|O_CREAT, ...)
 seek(...)
 if (len > 0)
   write(...)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove commit_{start,finish} from Allocator

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not call commit_{start,finish}

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: defer releasing extents explicitly

Don't rely on the Allocator to do this for us (that is
about to go away!).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not call commit_{start,finish}

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: defer bluefs reclaimed extent release

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release alloc directly into usable pool

Wait until after the kv transaction commits, and then
release extents directly into the usable pool.

This will let us remove the commit_{start,end} business.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/workunit.py: add CEPH_BASE env var

Root of git checkout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/workunit: leave workunits inside git checkout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/admin_socket: subst in repo name

It is either ceph.git or ceph-ci.git.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix target_buffer value overflow in Cache::trim() when metadata_ratio is set to 1

Signed-off-by: Igor Fedotov<ifedotov@mirantis.com>

Merge pull request #12396 from xiaoxichen/fix_diff

librbd: use proper snapshot when computing diff parent overlap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12482 from badone/wip-librados-objectcursor-leak

librados: Memory leaks in object_list_begin and object_list_end

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12286 from dillaman/wip-18123

rados: optionally support reading omap key from file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12462 from XinzeChi/wip-blk-link

common/blkdev: use realpath instead of readlink to resolve the recurs…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12488 from liewegas/wip-bluestore-enc

os/bluestore: fix ondisk encoding for blobs

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

os/bluestore/bluestore_types: bluestore_onode_t: varint encoding

Changing the uint32_t encoding to varint encoding saves around 120 bytes
per onode.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #12502 from kylinstorage/wip-ctypos2

fix typos

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rc/rgw/rgw_rados.cc : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/rgw/rgw_coroutine.cc : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/os/memstore/MemStore.h : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/os/filestore/FileStore.cc : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/os/bluestore/NVMEDevice.cc :fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/mon/DumplingMonCommands.h :fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

Merge pull request #12495 from kylinstorage/wip-ctypos

Wip ctypos

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12479 from badone/wip-ceph_test_rados_api_list-leak

test/rados/list.cc: Memory leak in ceph_test_rados_api_list

Reviewed-by: Sage Weil <sage@redhat.com>

journal: prevent repetitive error messages after being blacklisted

Fixes: http://tracker.ceph.com/issues/18243
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

src/os/bluestore/BlueStore.cc:fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

Merge pull request #12493 from athanatos/wip-18113-qa

Wip 18113 qa

Merge remote-tracking branch 'ceph-qa-suite/master' into wip-18113-qa

Merge pull request #1310 from ceph/wip-18113

test: Update for new error message when doing scrub with deep-scrub e…

Reviewed-by: Samuel Just <sjust@redhat.com>

test/rados/list.cc: Memory leak in ceph_test_rados_api_list

Valgrind flags a memory leak when we create a temporary from
rados_object_list_end()

Fixes: http://tracker.ceph.com/issues/18250
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12438 from songbaisen/s6

 mon: make it more clearly to debug for paxos state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12461 from liupan1111/wip-replace-atomic

osd: remove the lock heartbeat_update_lock, and change heatbeat_need_…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12452 from ceph/wip-rgw-fix-cr-deadlock

rgw: use explicit flag to cancel RGWCoroutinesManager::run()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12487 from athanatos/wip-rpg-rename

src: rename ReplicatedPG to PrimaryLogPG

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore/bluestore_types: bluestore_blob_t: varint encode csum_data length

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: version shard and spanning blob buffers, not each blob

Instead of versioning every blob encoding, and adding a full byte
per blob, instead version the entire shard or spanning blob
chunk, since they are always encoded together.

We overload the 'features' argument here to pass through a
struct_v.  This is slightly abusing an argument that is normally
used for feature bits, but only slightly.

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: pass features through to decode() on featured objects

This allows featured objects the *option* to do something
meaningful with the features.  Generally the decoding is feature
independent, but in some cases decoding might be dependent on the
struct_v or features from a containing context and we need to
pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: single byte for blob struct_v

We forego decoding compatibility by older code by dropping
compat_v and len.  We have to keep struct_v per blob here because
decode doesn't have a feature bit argument (yet?).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7639 from linuxbox2/wip-rgw-header

rgw: add option to log custom HTTP headers (rgw_log_http_headers)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12268 from dzafman/wip-18113

osd: When deep-scrub errors present upgrade regular scrubs

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore/bluestore_types: fix bluestore_pextent_t encoding

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

src: rename ReplicatedPG to PrimaryLogPG

It's been confusing for a long time that EC pools are implemented by
ReplicatedPG.  What PG/ReplicatedPG really implement is the concept
of a PG where consistency is managed by the primary via a log.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12453 from liewegas/qa-suite-master

master: merge ceph-qa-suite

merge ceph-qa-suite

move ceph-qa-suite dirs into qa/

Revert "tasks/workunit.py: depth 1 clone"

This reverts commit 4faf77a649cb3f8ddf497ca81937b3dbf63a18dc.

Merge pull request #12420 from dillaman/wip-16176

librbd: properly order concurrent updates to the object map

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #1307 from ceph/wip-workunit-repo

tasks/workunit: use correct git mirror url

tasks/workunit.py: depth 1 clone

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/workunit: remove kludge to use git.ceph.com

This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo.  Remove it entirely.  We'll see if
github.com is better at handling our load than it used to
be!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12483 from trociny/wip-18242

rbd-nbd: invalid error code for "failed to read nbd request" messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1309 from ceph/wip-16800-kefu

tasks/ceph: restore context of osd mount path before mkfs

Merge pull request #12476 from liewegas/wip-vstart-rados

CMakeLists: add vstart-base target

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add 'rgw log http headers' gloss to config-ref.rst

Explain the option for upstream doc.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12393 from dachary/wip-18036-throttle

tests: unittest_throttle avoid ASSERT_DEATH

Reviewed-by: Sage Weil <sage@redhat.com>