test: add mempool dump to Onode size tracking test case

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add an estimation for Onode in-memory size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12484 from runsisi/wip-fix-rollback-warn

librbd: ignore error when object map is already locked by current client

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore error when object map is already locked by current client

otherwise when using rbd cli to rollback image with object-map feature
enabled, the following error message will be printed out on the screen,
which is confusing to users:
  librbd::object_map::LockRequest: failed to lock object map: (17) File exists

Fixes: http://tracker.ceph.com/issues/16179
Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #12463 from wjwithagen/wip-wjw-freebsd-osd-avoidports

FreeBSD/OSD.cc: add client_messenger to the avoid_ports set.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12500 from badone/wip-doc-PGPool

doc/dev/osd_internals: add pgpool.rst

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12511 from liewegas/wip-workunits

qa/workunits/rbd: fix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits: use relative path instead of wget from git

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12508 from liewegas/wip-qa-admin-socket

qa/tasks/admin_socket: subst in repo name

doc/dev/osd_internals: add pgpool.rst

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/workunit.py: add CEPH_BASE env var

Root of git checkout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/workunit: leave workunits inside git checkout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/admin_socket: subst in repo name

It is either ceph.git or ceph-ci.git.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12396 from xiaoxichen/fix_diff

librbd: use proper snapshot when computing diff parent overlap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12482 from badone/wip-librados-objectcursor-leak

librados: Memory leaks in object_list_begin and object_list_end

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12286 from dillaman/wip-18123

rados: optionally support reading omap key from file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12462 from XinzeChi/wip-blk-link

common/blkdev: use realpath instead of readlink to resolve the recurs…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12488 from liewegas/wip-bluestore-enc

os/bluestore: fix ondisk encoding for blobs

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

os/bluestore/bluestore_types: bluestore_onode_t: varint encoding

Changing the uint32_t encoding to varint encoding saves around 120 bytes
per onode.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #12502 from kylinstorage/wip-ctypos2

fix typos

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rc/rgw/rgw_rados.cc : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/rgw/rgw_coroutine.cc : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/os/memstore/MemStore.h : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/os/filestore/FileStore.cc : fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/os/bluestore/NVMEDevice.cc :fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

rc/mon/DumplingMonCommands.h :fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

Merge pull request #12495 from kylinstorage/wip-ctypos

Wip ctypos

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12479 from badone/wip-ceph_test_rados_api_list-leak

test/rados/list.cc: Memory leak in ceph_test_rados_api_list

Reviewed-by: Sage Weil <sage@redhat.com>

src/os/bluestore/BlueStore.cc:fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

Merge pull request #12493 from athanatos/wip-18113-qa

Wip 18113 qa

Merge remote-tracking branch 'ceph-qa-suite/master' into wip-18113-qa

Merge pull request #1310 from ceph/wip-18113

test: Update for new error message when doing scrub with deep-scrub e…

Reviewed-by: Samuel Just <sjust@redhat.com>

test/rados/list.cc: Memory leak in ceph_test_rados_api_list

Valgrind flags a memory leak when we create a temporary from
rados_object_list_end()

Fixes: http://tracker.ceph.com/issues/18250
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12438 from songbaisen/s6

 mon: make it more clearly to debug for paxos state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12461 from liupan1111/wip-replace-atomic

osd: remove the lock heartbeat_update_lock, and change heatbeat_need_…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12452 from ceph/wip-rgw-fix-cr-deadlock

rgw: use explicit flag to cancel RGWCoroutinesManager::run()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12487 from athanatos/wip-rpg-rename

src: rename ReplicatedPG to PrimaryLogPG

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore/bluestore_types: bluestore_blob_t: varint encode csum_data length

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: version shard and spanning blob buffers, not each blob

Instead of versioning every blob encoding, and adding a full byte
per blob, instead version the entire shard or spanning blob
chunk, since they are always encoded together.

We overload the 'features' argument here to pass through a
struct_v.  This is slightly abusing an argument that is normally
used for feature bits, but only slightly.

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: pass features through to decode() on featured objects

This allows featured objects the *option* to do something
meaningful with the features.  Generally the decoding is feature
independent, but in some cases decoding might be dependent on the
struct_v or features from a containing context and we need to
pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: single byte for blob struct_v

We forego decoding compatibility by older code by dropping
compat_v and len.  We have to keep struct_v per blob here because
decode doesn't have a feature bit argument (yet?).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7639 from linuxbox2/wip-rgw-header

rgw: add option to log custom HTTP headers (rgw_log_http_headers)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12268 from dzafman/wip-18113

osd: When deep-scrub errors present upgrade regular scrubs

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore/bluestore_types: fix bluestore_pextent_t encoding

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

src: rename ReplicatedPG to PrimaryLogPG

It's been confusing for a long time that EC pools are implemented by
ReplicatedPG.  What PG/ReplicatedPG really implement is the concept
of a PG where consistency is managed by the primary via a log.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12453 from liewegas/qa-suite-master

master: merge ceph-qa-suite

merge ceph-qa-suite

move ceph-qa-suite dirs into qa/

Revert "tasks/workunit.py: depth 1 clone"

This reverts commit 4faf77a649cb3f8ddf497ca81937b3dbf63a18dc.

Merge pull request #12420 from dillaman/wip-16176

librbd: properly order concurrent updates to the object map

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #1307 from ceph/wip-workunit-repo

tasks/workunit: use correct git mirror url

tasks/workunit.py: depth 1 clone

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/workunit: remove kludge to use git.ceph.com

This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo.  Remove it entirely.  We'll see if
github.com is better at handling our load than it used to
be!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12483 from trociny/wip-18242

rbd-nbd: invalid error code for "failed to read nbd request" messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1309 from ceph/wip-16800-kefu

tasks/ceph: restore context of osd mount path before mkfs

Merge pull request #12476 from liewegas/wip-vstart-rados

CMakeLists: add vstart-base target

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add 'rgw log http headers' gloss to config-ref.rst

Explain the option for upstream doc.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12393 from dachary/wip-18036-throttle

tests: unittest_throttle avoid ASSERT_DEATH

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12473 from dillaman/wip-18243

journal: avoid logging an error when a watch is blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #1156 from ceph/wip-kclient-python

Enable cephfs python tests on kclient

Merge pull request #11542 from batrick/systemd-ceph-fuse

systemd: add ceph-fuse service file

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12337 from vishalkanaujia/wip_12132

tools: cleanup phase of cephfs-data-scan

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12358 from tchaikov/wip-18157

ceph-fuse: start up log on parent process before shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12384 from renhwztetecs/renhw-wip-libcephfs-addreadlink

libcephfs: add readlink function in cephfs.pyx

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12395 from jcsp/wip-18166

mon/MDSMonitor: fix iterating over mutated map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12401 from kylinstorage/wip-typos

client: fix mutex name typos

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12443 from jcsp/wip-18225

mds: release pool allocator memory after exceeding size limit

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph: restore context of osd mount path before mkfs

all newly created files and directories under the mount dir inherit the
SELinux type of their parent directory. so we need to set it before
mkfs.

Fixes: http://tracker.ceph.com/issues/16800
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: invalid error code for "failed to read nbd request" messages

Fixes: http://tracker.ceph.com/issues/18242
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

src/kv/RocksDBStore.cc:fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

src/erasure-code/shec/ErasureCodeShec.cc:fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

src/client/SyntheticClient.cc: fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

src/rgw/rgw_cache.cc:fix typos

Signed-off-by: xianxiaxiao <xianxia.xiao@kylin-cloud.com>

Merge pull request #12333 from xiexingguo/xxg-wip-bluestore-1206

os/bluestore: fix potential csum_order overflow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12382 from dachary/wip-18092-gf-complete

erasure-code: synchronize with upstream gf-complete

CMakeLists: add vstart-base target

No mds or radosgw.  Add rados CLI.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: Memory leaks in object_list_begin and object_list_end

We allocate a cursor in the constructor but simply reassign it in these
functions without cleaning up the original. We have a utility setter that
handles this exact case so we should use it.

Fixes: http://tracker.ceph.com/issues/18252
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12459 from runsisi/wip-ec-minor-cleanup

osd/ECBackend: minor cleanup to remove duplicate code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/kraken'

rgw: custom headers: use boost::container::flat_map

Use this occasion of serialized map to implement encoders
for boost::container::flat_map, which is optimized for
in-order insertion.

After some discussion, I'm proposing to just add the new template
forms, rather than (e.g.) adding either a large amount of
specialization machinery, or alternatively using C-style
macros.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: add rgw_log_http_headers option

Tracks headers that should be handled conditionally (currently,
can only log, so using minimal structure to represent the
mapping).

Adds map of custom headers to rgw_log_entry, and populate it with
headers pre-selected for custom logging in RGWREST.  Added to encoder
and Formatter output.

Some additional optimization possible.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: block concurrent in-flight object map updates for the same object

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new block guard helper to prevent concurrent IO to blocks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert ObjectMap to template for unit testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clean up object map update interface

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update in-memory object map after on-disk update committed

Concurrent IO to the same object would previously result in the first
IO pausing to update the object map while the other IO would proceed
to directly update the object before the object map state was properly
updated.

Fixes: http://tracker.ceph.com/issues/16176
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: avoid logging an error when a watch is blacklisted

Fixes: http://tracker.ceph.com/issues/18243
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12447 from ifed01/wip-bluestore-fix-reshard2

os/bluestore: avoid resharding if the last shard size fall below shar…

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12289 from liewegas/wip-pg-down

osd/PG: add "down" pg state (distinct from down+peering)

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12389 from ovh/wip-rework-recovery-priorities

osd: Increase priority for inactive PGs backfill

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12352 from kylinstorage/wip-cleanup-mark-queued

osd: mark queued flag for op

Reviewed-by: Sage Weil <sage@redhat.com>

common/blkdev: use realpath instead of readlink to resolve the recursion symbol link

Signed-off-by: Xinze Chi <xinze@xsky.com>

os/bluestore: avoid resharding if the last shard size fall below shard min size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

FreeBSD/OSD.cc: add client_messenger to the avoid_ports set.

Observed "feature":
  During rebind due to a "wrongly marked down" log message, FreeBSD is
  able to bind to the port used by client_messenger.
  The Linux variant avoids that port because it is already in use.

Result:
  In FreeBSD there would be 2 listeners on the port, and due to the rebind
  they have different nonces. (This is written in the logfile)
  But they also will expect different protocols on that same port.

This is likely due to an interpretation difference in the SO_REUSEADDR
socket option:

Linux:
  SO_REUSEADDR
        Indicates that the rules used in validating addresses supplied
        in a bind(2)  call  should  allow  reuse  of  local addresses.
        For AF_INET sockets this means that a socket may bind,
        except when there is an active listening socket bound to the address.
        When the listening socket is bound to INADDR_ANY with a specific port
        then it is not possible to bind to this port for any local address.
        Argument is an integer boolean flag.

FreeBSD:
  SO_REUSEADDR
        Enables local address reuse
        indicates that the rules used in validating addresses supplied in a
        bind(2) system call should allow reuse of local addresses.

So FreeBSD doesn't guarantee that the connection is refused when there is already
a connection. So it is best avoided during rebinding otherwise any of the
cluster_messengers will attach to the port.

FreeBSD log with wrong connect:
bb98d80  0 log_channel(cluster) log [WRN] : map e18 wrongly marked me down
bb98d80  1 -- 127.0.0.1:6801/17881 rebind rebind avoid 6801,6802,6803
bb98d80  1 -- 127.0.0.1:6801/17881 shutdown_connections
bb98d80  1 -- 127.0.0.1:6800/1017881 _finish_bind bind my_inst.addr is 127.0.0.1:6800/1017881
bb98d80  1  Processor -- start
bb98d80  1 -- 127.0.0.1:6802/17881 rebind rebind avoid 6801,6802,6803
bb98d80  1 -- 127.0.0.1:6802/17881 shutdown_connections
bb98d80  1 -- 127.0.0.1:0/17881 learned_addr learned my addr 127.0.0.1:0/17881
bb98d80  1 -- 127.0.0.1:6804/1017881 _finish_bind bind my_inst.addr is 127.0.0.1:6804/1017881

FreeBSD with the correct behaviour:
bb98d80  0 log_channel(cluster) log [WRN] : map e17 wrongly marked me down
bb98d80  1 -- 127.0.0.1:6802/15296 rebind rebind avoid 6801,6802,6803,6812
bb98d80  1 -- 127.0.0.1:6802/15296 shutdown_connections
bb98d80  1 -- 127.0.0.1:6806/1015296 _finish_bind bind my_inst.addr is 127.0.0.1:6806/1015296
bb98d80  1  Processor -- start
bb98d80  1 -- 127.0.0.1:6803/15296 rebind rebind avoid 6801,6802,6803,6812
bb98d80  1 -- 127.0.0.1:6803/15296 shutdown_connections
bb98d80  1 -- 127.0.0.1:0/15296 learned_addr learned my addr 127.0.0.1:0/15296
bb98d80  1 -- 127.0.0.1:6807/1015296 _finish_bind bind my_inst.addr is 127.0.0.1:6807/1015296

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

ECBackend: minor cleanup to remove duplicate code

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd: remove the lock heartbeat_update_lock, and change heatbeat_need_update to atomic_bool

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #12294 from dx9/test-ecp-uname

test: use uname -m vs. uname -p in test-erasure-code-plugins.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd/diff_iterator: use proper snap to query parent overlap

If no snapshot has been made, we will use CEPH_NOSNAP instead of 0,
to query the parent overlap.

Fixes bug http://tracker.ceph.com/issues/18200

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

tasks/cephfs: set norequire_active_mds on kclient

The kernel client's cluster availability test is
more primitive than the fuse client, so we need
to switch it off to avoid client mounts failing
while MDSs are still coming up.

Fixes: http://tracker.ceph.com/issues/18161
Signed-off-by: John Spray <john.spray@redhat.com>

test: Update for new error message when doing scrub with deep-scrub errors

Signed-off-by: David Zafman <dzafman@redhat.com>