client: clamp I/O sizes to INT_MAX when we can't return larger values

We have several API functions that allow the caller to request I/Os
larger than INT_MAX bytes, but that return an int. Ensure that we don't
try to do more I/O than we can represent in the return value.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: new testcase for ceph_ll_readv and ceph_ll_writev

Just copy the PreadvPwritev test and alter accordingly.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: hook up ceph_ll_readv and ceph_ll_writev

Break the core of _preadv_pwritev out into a function that takes a Fh.
Make _preadv_pwritev into a wrapper around that.

Then add in plumbing for ceph_ll_readv and ceph_ll_writev.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: type safety cleanup for _read and _write codepaths

In the read codepath, bl->length() returns an unsigned value, and that
could end up looking negative when cast to int. On the write side,
totalwritten is a uint64_t, which could look negative when cast to int.

Have the underlying layers pass back an int64_t and convert them to
int at a higher level. This prepares the underlying infrastructure for
ceph_ll_readv and ceph_ll_writev support.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #20264 from ceph/wip-rm22620

ceph-volume dmcrypt support for simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20307 from tchaikov/wip-std-filesystem

test/admin_socket_output: switch to std::experimental::filesystem

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20324 from tchaikov/wip-cmake-build-boost

cmake: fix the include dir for building boost::python

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #20347 from Songweibin/wip-osd-cleanup

osd: drop redundant comment

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: drop redundant comment

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge PR #20288 into master

* refs/pull/20288/head:
ptl-tool: open githubmap with utf-8 code

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20313 from guzhongyan/fix-22350

osd/OSDMap: misleading message in print_oneline_summary()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20317 from liewegas/wip-pg-scrub-priority

osd/PG: pass scrub priority to replica

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20319 from tchaikov/wip-ops_blocked_by_scrub-cleanup

osd: pass ops_blocked_by_scrub() to requeue_scrub()

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: do not test custom cluster names with simple

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20279 from liewegas/wip-idempotent-sequence

ceph_test_filestore_idempotent_sequence: many fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: sleep 2 minutes after reboot in simple tests

This will allow for some time for OSD services to start back up after
reboot. If not we see race conditions where tests fail because they
were run before the OSD finished startup.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20267 from liewegas/wip-22847

mon/OSDMonitor: filter out pgs that shouldn't exist from force-create-pg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20259 from guzhongyan/rados-cleanup

rados: clean up rados_getxattrs() and rados_striper_getxattrs()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20260 from guzhongyan/fix-22042

librados: invalid free() in rados_getxattrs_next()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20255 from david-z/wip-fix-kv-max-option

config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20301 from bigclouds/wip-fix-declaration-of-rados-striper-trunc

libradosstriper: fix the function declaration of rados_striper_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20233 from cbodley/wip-test-log-nullptr

test/log: fix for crash with libc++

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20223 from Yan-waller/wip-walle-180124cephosd

osd: use existing osd_required variable for messenger policy

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20201 from majianpeng/mosdrepop-encode-log

osd: avoid encoding the same log entries repeatedly for different peers

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/filestore: add replay guards for omap operations

These were not guarded, meaning they could get replayed onto an object
that was cloned or renamed.  Eek!

Fixes: http://tracker.ceph.com/issues/22920
Signed-off-by: Sage Weil <sage@redhat.com>

test/objectstore/run_seed_to_range: stop on first error

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_idempotent_sequence: use coll_entry_t

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_filestore_idempotent_sequence: fix up clone, move ops; cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/filestore_idempotent: run failure tests later in sequence

We want to get past the initial colleciton and object creation steps,
which don't include interesting ops.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20016 from cbodley/wip-18885

rgw: log unlink_instance mtime as object's mtime

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume: adds the simple dmcrypt_plain and dmcrypt_luks tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix ceph-volume simple scan help menu test

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20145 from linuxbox2/rgwf-deref-readdir-off

rgw_file: avoid evaluating nullptr for readdir offset

Merge pull request #20188 from cbodley/wip-rgw-frontend-multimap

rgw: beast frontend can listen on multiple endpoints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20013 from cbodley/wip-rgw-auth-skew

rgw: share time skew check between v2 and v4 auth

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20017 from cbodley/wip-22248

rgw: unlink deleted bucket from bucket's owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20120 from cbodley/wip-22797

rgw: fix for empty query string in beast frontend

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20038 from oritwas/wip-rgw-22742

rgw: bucket resharding should not update bucket ACL or user stats
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20295 from trociny/wip-test_deep_copy_clone

test/pybind/rbd: skip test_deep_copy_clone if layering not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: fix the include dir for building boost::python

the "includes" directories are separated by spaces not semicolons.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20309 from pritha-srivastava/wip-rgw-roles-error-codes

rgw: Reinstating error codes mapping for Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20316 from smithfarm/wip-doc-log

doc: log-and-debug: fix default value of "log max recent"

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: pass ops_blocked_by_scrub() to requeue_scrub()

less nesting levels, and less repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: pass scrub priority to replica

If we are scrubbing with high priority on the primary, pass that along
to the replica so that it can schedule its scrub work accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: log-and-debug: fix default value of "log max recent"

Doc says 1000000, but the real figure is two orders of magnitude less:

    Option("log_max_recent", Option::TYPE_INT, Option::LEVEL_ADVANCED)
      .set_default(500)
      .set_daemon_default(10000)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20229 from zhsj/fix-dashboard-prefix

pybind/mgr/dashboard: fix duplicated slash in html href

Reviewed-by: John Spray <john.spray@redhat.com>

osd/OSDMap: misleading message in print_oneline_summary()
The output scared people as showing so many osds were full/nearfull

Improve the output to make the message clearer

Fixes: http://tracker.ceph.com/issues/22350
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

rgw: Reinstating error codes mapping for Roles.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

test/admin_socket_output: switch to std::experimental::filesystem

so we don't need to compile boost::filesystem anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20114 from cethikdata/ceph-disk-lock

ceph-disk: reduce the scope of activate_lock.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/pybind/test_rados.py: expand test_xattrs/test_obj_xattrs
to cover attribute with empty value case.

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

librados: invalid free() in rados_getxattrs_next()

Invalid free() can cause corruption when getting an object
attribute with empty value.

Check the validity of the pointer before free(). Also move
the free() call at the start of rados_getxattrs_next() to
avoid memory leak.

Fixes: http://tracker.ceph.com/issues/22042
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

libradosstriper: fix the function declaration of rados_striper_trunc
change the type of first parameter from rados_ioctx_t to rados_striper_t

Signed-off-by: yuelongguang <yuelongguang@gmail.com>

Merge pull request #20270 from ivancich/wip-fix-bad-dmclock-sim-change

osd,dmclock: fix dmclock test simulator change

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20258 from tchaikov/wip-balancer-cleanup

 mgr/balancer: replace magic value of -1 for DEFAULT_CHOOSE_ARGS

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20230 from ifed01/wip-ifed-store-test-improve

test/store_test: update Many4KWritesTest* test cases to finalize with…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20221 from fangyuxiangGL/max

osd/ReplicatedBackend: 'osd_deep_scrub_keys' doesn't work

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20217 from rzarzynski/wip-osd-legacy-conf-for-perf

osd: avoid the config's get_val() overhead on the read path.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20169 from liewegas/wip-22668-revert

os/bluestore: disable on_applied sync_complete

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd: use defined osd_required features for messenger policy

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

rgw: bucket resharding should not update bucket ACL or user stats

bucket link op resets bucket ACL using rgw_link_bucket instead

Fixes: http://tracker.ceph.com/issues/22124
Fixes: http://tracker.ceph.com/issues/22742
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

pybind/mgr/dashboard: fix duplicated slash in html href

The url_prefix val in html templates is expected not containing '/'
in the end. Otherwise the href will be rendered as '//static/', which
causes browser treating it as http://static/.

Fixes: http://tracker.ceph.com/issues/22851
Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>

test/pybind/rbd: skip test_deep_copy_clone if layering not enabled

(otherwise it will fail on protect snap)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20184 from shun-s/wip-add-osdmap-epoch-show

osd/OSDMap: add osdmap epoch info when printing info summary

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20225 from xiexingguo/wip-balancer-pool-options

mgr/balancer: more pool-specific enhancements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20224 from xiexingguo/wip-still-mgr-fixes

pybind/mgr/balancer: remove optimization plan properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20278 from xiexingguo/wip-null-ws-check

pybind/mgr/balancer: fix sanity check against empty weight-set

Reviewed-by: Sage Weil <sage@redhat.com>

ptl-tool: open githubmap with utf-8 code

This resolves a decode error:

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 8: ordinal not in range(128)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20261 into master

* refs/pull/20261/head:
doc: Fix path in ceph-fuse.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/mgr_module: do not offer empty default value for choose_args

as it should be always available. this also silence a pep8 warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/balancer: introduce CRUSHMap.DEFAULT_CHOOSE_ARGS and use it

avoid using '-1' as a magic variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/objectstore/run_seed_to_range.sh: objects

We need some initial objects for this to work.

Signed-off-by: Sage Weil <sage@redhat.com>

test/objectstore/run_seed_to.sh: make output readable

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_filestore_idempotent_sequence: fix clone check

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_filestore_idempotent_sequence: fix do_coll_create

The old code made no sense whatsoever!

- create one colletion, not a zillion
- inform the model in the parent class

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_filestore_idempotent_sequence: fix gap in ops

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_filestore_idempotent_sequence: start poolid at 2

Signed-off-by: Sage Weil <sage@redhat.com>

test/objectstore/FilestoreDiff: write to stdout

This keeps our output separate from the filestore log, which we send to
a file so that the terminal output is readable.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/balancer: fix sanity check against empty weight-set

There are paths get_compat_weight_set_weights() will return a
empty dict, not None.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert change from commit 8c5fea68cb9e0c2e6cc2352e64c4f3f46e29c58f
that introduced a compilation error that was not noticed since the
dmclock support/testing simulators are not built by ceph.

Also, changes below the src/dmclock tree should be made in the dmclock
library and then pulled into ceph.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #19711 from trociny/wip-20762

rbd: unified way to map images using different drivers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mon/OSDMonitor: ignore and/or remove pgs that shouldn't exist

The update_creating_pgs() change is there mainly for clusters that already
have an invalid pg in the map; going forward they hopefully shouldn't
appear!

Fixes: https://tracker.ceph.com/issues/22847
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: reject force-create-pg on pgids that shouldn't exist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19902 from linuxbox2/wip-rgw-mploc

rgw: return valid Location element, CompleteMultipartUpload

osd/ReplicatedBackend: only do once encode log_entries  for multi MOSDRepOp

Encode log_entries is the same work for multi MOSDRepOp and encode is
heavy load.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #20199 from Songweibin/wip-group-cleanup

rbd: group misc cleanup and update rbd man page

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20002 from malc0lm/fix-copypart

rgw: fix 'copy part' without 'x-amz-copy-source-range'

ceph-volume tests.functional add simple xenial filestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple xenial filestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple xenial bluestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple centos7 filestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple centos7 filestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple centos7 bluestore dmcrypt plain support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple centos7 bluestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add simple bluestore dmcrypt luks support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm prepare fully supports encryption now

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume scan update for encryption support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20222 from trociny/wip-deep-copy-default-format

librbd: by default use new format for deep copy destination

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20170 from trociny/wip-22803

qa/workunits/rbd: remove "trash purge --threshold" test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20099 from Songweibin/wip-fix-deep-cp

librbd: fix deep copy a child-image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>