os/bluestore/bluestore_types: bluestore_onode_t: varint encoding

Changing the uint32_t encoding to varint encoding saves around 120 bytes
per onode.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

os/bluestore/bluestore_types: bluestore_blob_t: varint encode csum_data length

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: version shard and spanning blob buffers, not each blob

Instead of versioning every blob encoding, and adding a full byte
per blob, instead version the entire shard or spanning blob
chunk, since they are always encoded together.

We overload the 'features' argument here to pass through a
struct_v.  This is slightly abusing an argument that is normally
used for feature bits, but only slightly.

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: pass features through to decode() on featured objects

This allows featured objects the *option* to do something
meaningful with the features.  Generally the decoding is feature
independent, but in some cases decoding might be dependent on the
struct_v or features from a containing context and we need to
pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: single byte for blob struct_v

We forego decoding compatibility by older code by dropping
compat_v and len.  We have to keep struct_v per blob here because
decode doesn't have a feature bit argument (yet?).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: fix bluestore_pextent_t encoding

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #12453 from liewegas/qa-suite-master

master: merge ceph-qa-suite

merge ceph-qa-suite

move ceph-qa-suite dirs into qa/

Revert "tasks/workunit.py: depth 1 clone"

This reverts commit 4faf77a649cb3f8ddf497ca81937b3dbf63a18dc.

Merge pull request #12420 from dillaman/wip-16176

librbd: properly order concurrent updates to the object map

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #1307 from ceph/wip-workunit-repo

tasks/workunit: use correct git mirror url

tasks/workunit.py: depth 1 clone

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/workunit: remove kludge to use git.ceph.com

This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo.  Remove it entirely.  We'll see if
github.com is better at handling our load than it used to
be!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12483 from trociny/wip-18242

rbd-nbd: invalid error code for "failed to read nbd request" messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1309 from ceph/wip-16800-kefu

tasks/ceph: restore context of osd mount path before mkfs

Merge pull request #12476 from liewegas/wip-vstart-rados

CMakeLists: add vstart-base target

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12393 from dachary/wip-18036-throttle

tests: unittest_throttle avoid ASSERT_DEATH

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12473 from dillaman/wip-18243

journal: avoid logging an error when a watch is blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #1156 from ceph/wip-kclient-python

Enable cephfs python tests on kclient

Merge pull request #11542 from batrick/systemd-ceph-fuse

systemd: add ceph-fuse service file

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12337 from vishalkanaujia/wip_12132

tools: cleanup phase of cephfs-data-scan

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12358 from tchaikov/wip-18157

ceph-fuse: start up log on parent process before shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12384 from renhwztetecs/renhw-wip-libcephfs-addreadlink

libcephfs: add readlink function in cephfs.pyx

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12395 from jcsp/wip-18166

mon/MDSMonitor: fix iterating over mutated map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12401 from kylinstorage/wip-typos

client: fix mutex name typos

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12443 from jcsp/wip-18225

mds: release pool allocator memory after exceeding size limit

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph: restore context of osd mount path before mkfs

all newly created files and directories under the mount dir inherit the
SELinux type of their parent directory. so we need to set it before
mkfs.

Fixes: http://tracker.ceph.com/issues/16800
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: invalid error code for "failed to read nbd request" messages

Fixes: http://tracker.ceph.com/issues/18242
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12333 from xiexingguo/xxg-wip-bluestore-1206

os/bluestore: fix potential csum_order overflow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12382 from dachary/wip-18092-gf-complete

erasure-code: synchronize with upstream gf-complete

CMakeLists: add vstart-base target

No mds or radosgw.  Add rados CLI.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12459 from runsisi/wip-ec-minor-cleanup

osd/ECBackend: minor cleanup to remove duplicate code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/kraken'

librbd: block concurrent in-flight object map updates for the same object

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new block guard helper to prevent concurrent IO to blocks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert ObjectMap to template for unit testing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clean up object map update interface

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update in-memory object map after on-disk update committed

Concurrent IO to the same object would previously result in the first
IO pausing to update the object map while the other IO would proceed
to directly update the object before the object map state was properly
updated.

Fixes: http://tracker.ceph.com/issues/16176
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: avoid logging an error when a watch is blacklisted

Fixes: http://tracker.ceph.com/issues/18243
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12447 from ifed01/wip-bluestore-fix-reshard2

os/bluestore: avoid resharding if the last shard size fall below shar…

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12289 from liewegas/wip-pg-down

osd/PG: add "down" pg state (distinct from down+peering)

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12389 from ovh/wip-rework-recovery-priorities

osd: Increase priority for inactive PGs backfill

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12352 from kylinstorage/wip-cleanup-mark-queued

osd: mark queued flag for op

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid resharding if the last shard size fall below shard min size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ECBackend: minor cleanup to remove duplicate code

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #12294 from dx9/test-ecp-uname

test: use uname -m vs. uname -p in test-erasure-code-plugins.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks/cephfs: set norequire_active_mds on kclient

The kernel client's cluster availability test is
more primitive than the fuse client, so we need
to switch it off to avoid client mounts failing
while MDSs are still coming up.

Fixes: http://tracker.ceph.com/issues/18161
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1304 from vshankar/wip-15028

cram: support fetching from sha1 branch, tag, commit hash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12450 from athanatos/wip-18180

osd/ReplicatedPG::record_write_error: don't leak orig_reply on cancel

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12402 from liewegas/wip-scrub-recovery

osd: default osd_scrub_during_recovery=false

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12448 from theanalyst/doc/11.1.0-rc

doc: final additions to 11.1.0-rc release notes

Reviewed-by: Sage Weil <sage@redhat.com>

doc: final additions to 11.1.0-rc release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

11.1.0

Merge pull request #1305 from ceph/wip-h-j-x

upgrade/hammer-jewel-x: whitelist quota

upgrade/hammer-jewel-x: whitelist quota

This is from teh librados api test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10209 from runsisi/sisi-wip-remove-redundant-check

osd/ReplicatedPG: remove redundant check for balance/localize read

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12190 from liupan1111/wip-speed-up-acting-ods

osd/OSDMap: improve the performance of pg_to_acting_osds

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12279 from wjwithagen/wip-wjw-freebsd-ENODATA-2

os/filestore/chain_xattr.h:uses ENODATA, so include compat.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12365 from liewegas/wip-mon-flag-warning

mon/OSDMonitor: only show interesting flags in health warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12439 from tchaikov/wip-pybind-requires

pybind/rados: add missing "length" requires for aio_execute()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12390 from tchaikov/wip-start-after-setting-nonce

msg/async: set nonce before starting the workers

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12309 from wjwithagen/wip-wjw-freebsd-CLOCK_MONOTONE

rgw/rgw_file.cc: Add compat.h to allow CLOCK_MONOTONE

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12257 from kylinstorage/wip-remove-unneeded-forward-declaration

remove unneeded forward declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12301 from wjwithagen/wip-wjw-freebsd-blkid

src/test/CMakeLists.txt: Exclude test on HAVE_BLKID

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12378 from dzafman/wip-18139

test: Don't write to a poolid that this test might not have created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12371 from ceph/wip-18149

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mds: make cache size health warning factor configurable

...now that it's used more than once place, copying around
the fraction was bad.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: release pool allocator memory after exceeding size limit

Usually we like keeping the memory in our pools, but in circumstances
where we have violated our self-imposed inode count cap, we should
shrink our usage of system memory afterwards.

Fixes: http://tracker.ceph.com/issues/18225
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #12436 from badone/wip-cov1395747

crush: Silence coverity warnings for test/crush/crush.cc

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #12409 from tchaikov/wip-doc-releases

doc/releases: add links to kraken and v10.2.4

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

pybind/rados: add missing "length" requires for aio_execute()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: Silence coverity warnings for test/crush/crush.cc

Coverity 1395747 reports that "c" leaks. Switching to smart pointers should
resolve that.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

cram: support fetching from sha1 branch, tag, commit hash

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Fixes: http://tracker.ceph.com/issues/18149
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12410 from dachary/wip-10-2-5

doc: release notes for 10.2.5

Reviewed-by: Sage Weil <sage@redhat.com>

doc: release notes for 10.2.5

Fixes: http://tracker.ceph.com/issues/18207
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12366 from dillaman/wip-10773

qa/workunits/rbd: removed qemu-iotest case 077

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12253 from aclamk/fix-copy-object-compression

Fixed problem with PUT with x-amz-copy-source when source object is compressed.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12283 from ceph/wip-rgw-18106

rgw: make RGWLocalAuthApplier::is_admin_of() aware about system users.

Reviewed-by: Orit Wasserman <owassrm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge remote-tracking branch 'upstream/pull/1302/head'

* upstream/pull/1302/head:
  fsx-mpi: fsx-mpi expects a file name not a dir
  fsx-mpi: remove chef task
  Fix mpi calls

Merge pull request #12414 from ceph/wip-16125

ceph-disk: prevent unnecessary tracebacks from subprocess.check_call

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12368 from trociny/wip-18146

rbd-mirror: fix sparse read optimization in image sync

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-disk: update tests to use the new exit flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: remove try/except for CalledProcessError to use new exit flag handler

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: allow to exit nicely when using command_check_call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: Increase priority for inactive PGs backfill

This change does prioritize backfill of PGs which don't
have min_size active copies. Such PGs would cause IO stalls
for clients and would increase throttlers usage.

This change also fixes few subtlle out-of-bounds bugs.

Signed-off-by: Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Merge pull request #12408 from badone/wip-mon_osd_down_out_interval-docfix

doc: Modify documentation for mon_osd_down_out_interval

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

doc/releases: add links to kraken and v10.2.4

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Modify documentation for mon_osd_down_out_interval

Update docs to reflect the new default of 600 set by 817d65bd4

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12379 from liupan1111/wip-fix-rbd-params

rbd: --max_part and --nbds_max options for nbd map

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12400 from gregsfortytwo/wip-fix-pg-unsigned-warning

osd: fix signed/unsigned comparison warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12211 from leseb/doc-openstack

doc: update cinder key permissions for mitaka

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: default osd_scrub_during_recovery=false

This is a more friendly behavior.  Tell users in the release notes how to
get the old behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

client: fix typos

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

osd: fix signed/unsigned comparison warning

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12367 from liewegas/wip-bluestore-sync-submit

os/bluestore: bluestore_sync_submit_transaction = false

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12356 from xiexingguo/xxg-wip-bluestore-commit-latency

os/bluestore: fix global commit latency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12381 from ly798/fix-rgw-crush-due-to-bad-cors

rgw: do not abort when accept a CORS request with short origin

msg/async: set nonce before starting the workers

otherwise workers will respond with difference nonces to peers.
and remove nonce from Processor. as there is only one nonce for each
Messenger at a given time.

Signed-off-by: Kefu Chai <kchai@redhat.com>