osd: remove the lock heartbeat_update_lock, and change heatbeat_need_update to atomic_bool

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #12294 from dx9/test-ecp-uname

test: use uname -m vs. uname -p in test-erasure-code-plugins.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12450 from athanatos/wip-18180

osd/ReplicatedPG::record_write_error: don't leak orig_reply on cancel

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12402 from liewegas/wip-scrub-recovery

osd: default osd_scrub_during_recovery=false

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12448 from theanalyst/doc/11.1.0-rc

doc: final additions to 11.1.0-rc release notes

Reviewed-by: Sage Weil <sage@redhat.com>

doc: final additions to 11.1.0-rc release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #10209 from runsisi/sisi-wip-remove-redundant-check

osd/ReplicatedPG: remove redundant check for balance/localize read

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12190 from liupan1111/wip-speed-up-acting-ods

osd/OSDMap: improve the performance of pg_to_acting_osds

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12279 from wjwithagen/wip-wjw-freebsd-ENODATA-2

os/filestore/chain_xattr.h:uses ENODATA, so include compat.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12365 from liewegas/wip-mon-flag-warning

mon/OSDMonitor: only show interesting flags in health warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12439 from tchaikov/wip-pybind-requires

pybind/rados: add missing "length" requires for aio_execute()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12390 from tchaikov/wip-start-after-setting-nonce

msg/async: set nonce before starting the workers

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12309 from wjwithagen/wip-wjw-freebsd-CLOCK_MONOTONE

rgw/rgw_file.cc: Add compat.h to allow CLOCK_MONOTONE

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12257 from kylinstorage/wip-remove-unneeded-forward-declaration

remove unneeded forward declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12301 from wjwithagen/wip-wjw-freebsd-blkid

src/test/CMakeLists.txt: Exclude test on HAVE_BLKID

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12378 from dzafman/wip-18139

test: Don't write to a poolid that this test might not have created

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12371 from ceph/wip-18149

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12436 from badone/wip-cov1395747

crush: Silence coverity warnings for test/crush/crush.cc

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #12409 from tchaikov/wip-doc-releases

doc/releases: add links to kraken and v10.2.4

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

pybind/rados: add missing "length" requires for aio_execute()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: Silence coverity warnings for test/crush/crush.cc

Coverity 1395747 reports that "c" leaks. Switching to smart pointers should
resolve that.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Fixes: http://tracker.ceph.com/issues/18149
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12410 from dachary/wip-10-2-5

doc: release notes for 10.2.5

Reviewed-by: Sage Weil <sage@redhat.com>

doc: release notes for 10.2.5

Fixes: http://tracker.ceph.com/issues/18207
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12366 from dillaman/wip-10773

qa/workunits/rbd: removed qemu-iotest case 077

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12253 from aclamk/fix-copy-object-compression

Fixed problem with PUT with x-amz-copy-source when source object is compressed.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12283 from ceph/wip-rgw-18106

rgw: make RGWLocalAuthApplier::is_admin_of() aware about system users.

Reviewed-by: Orit Wasserman <owassrm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12414 from ceph/wip-16125

ceph-disk: prevent unnecessary tracebacks from subprocess.check_call

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12368 from trociny/wip-18146

rbd-mirror: fix sparse read optimization in image sync

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-disk: update tests to use the new exit flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: remove try/except for CalledProcessError to use new exit flag handler

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: allow to exit nicely when using command_check_call

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #12408 from badone/wip-mon_osd_down_out_interval-docfix

doc: Modify documentation for mon_osd_down_out_interval

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

doc/releases: add links to kraken and v10.2.4

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Modify documentation for mon_osd_down_out_interval

Update docs to reflect the new default of 600 set by 817d65bd4

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12379 from liupan1111/wip-fix-rbd-params

rbd: --max_part and --nbds_max options for nbd map

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12400 from gregsfortytwo/wip-fix-pg-unsigned-warning

osd: fix signed/unsigned comparison warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12211 from leseb/doc-openstack

doc: update cinder key permissions for mitaka

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: default osd_scrub_during_recovery=false

This is a more friendly behavior.  Tell users in the release notes how to
get the old behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix signed/unsigned comparison warning

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12367 from liewegas/wip-bluestore-sync-submit

os/bluestore: bluestore_sync_submit_transaction = false

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12356 from xiexingguo/xxg-wip-bluestore-commit-latency

os/bluestore: fix global commit latency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12381 from ly798/fix-rgw-crush-due-to-bad-cors

rgw: do not abort when accept a CORS request with short origin

msg/async: set nonce before starting the workers

otherwise workers will respond with difference nonces to peers.
and remove nonce from Processor. as there is only one nonce for each
Messenger at a given time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12369 from tchaikov/wip-c99

cmake: compile C code with c99

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12375 from liewegas/wip-msgr-eagain-loop

msg/simple/Pipe: avoid returning 0 on poll timeout

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12331 from ceph/wip-jlayton-suid

Have ceph clear setuid/setgid bits on chown

doc: update cinder key permissions for mitaka

OpenStack Mitaka introduced the support of RBD snapshots while taking a
snapshot of a Nova instance. For this to work we need to grant write
access to the Glance pool to the Cinder key.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #12388 from kylinstorage/wip-remove-librbd-unused-variable

librbd: remove unused local variable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12383 from dachary/wip-18188-k-2-m-2

tests: erasure-code: add k=2, m=2 for isa & jerasure

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: --max_part and --nbds_max options for nbd map

Fixes: http://tracker.ceph.com/issues/18186
Signed-off-by: Pan Liu <pan.liu@istuary.com>

tests: erasure-code: add k=2, m=2 for isa & jerasure

Fixes: http://tracker.ceph.com/issues/18188
Signed-off-by: Loic Dachary <loic@dachary.org>

tests: fix erasure-code premature deallocation of cct

The setup function returns before the run function, the cct variable
must be a data member, not a local variable that gets de-allocated
before run() starts.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12387 from tchaikov/wip-remove-unused-var

msg/async: remove unused member variable

Reviewed-by: Haomai Wang <haomai@xsky.com>

rbd-mirror: fix sparse read optimization in image sync

Issue truncate or zero ops for the subtracted extents between the
diff and the sparse read.

Fixes: http://tracker.ceph.com/issues/18146
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: remove unused local variable

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

msg/async: remove unused member variable

PosixServerSocketImpl::sa is not referenced anywhere, so remove it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: do not abort when accept a CORS request with short origin

Fixed: #18187

when accept a CROS request, the request http origin shorter than the bucket's corsrule
(eg. origin: http://s.com corsrule: <AllowedOrigin>*.verylongdomain.com</AllowedOrigin>).
the rgw_cors.cc::is_string_in_set() will have a wrong index, the radosrgw server will
abort.

$ curl http://test.localhost:8000/app.data -H "Origin:http://s.com"

 0> 2016-12-05 03:22:29.548138 7f6add05d700 -1 *** Caught signal (Aborted) **
 in thread 7f6add05d700 thread_name:civetweb-worker

 ceph version 11.0.2-2168-gd2f8fb4 (d2f8fb4a6ba75af7e6da0f5a7f1b49ec998b1631)
 1: (()+0x50720a) [0x7f6b147c420a]
 2: (()+0xf370) [0x7f6b09a33370]
 3: (gsignal()+0x37) [0x7f6b081ca1d7]
 4: (abort()+0x148) [0x7f6b081cb8c8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f6b08ace9d5]
 6: (()+0x5e946) [0x7f6b08acc946]
 7: (()+0x5e973) [0x7f6b08acc973]
 8: (()+0x5eb93) [0x7f6b08accb93]
 9: (std::__throw_out_of_range(char const*)+0x77) 0x7f6b08b21a17]
 10: (()+0xbd97a) [0x7f6b08b2b97a]
 11: (()+0x449c1e) [0x7f6b14706c1e]
 12: (RGWCORSRule::is_origin_present(char const*)+0x48) [0x7f6b147073b8]
 13: (RGWCORSConfiguration::host_name_rule(char const*)+0x37) [0x7f6b147074e7]
 14: (RGWOp::generate_cors_headers(std::string&, std::string&, std::string&, std::string&, unsigned int*)+0xa3) [0x7f6b14593e63]
 15: (dump_access_control(req_state*, RGWOp*)+0x61) [0x7f6b14653f91]

Signed-off-by: LiuYang <yippeetry@gmail.com>

cmake: compile C code with c99

so the functions like snprintf() available in C99 are available without
defining _XOPEN_SOURCE. and "asm()" is used to inline assembly, so use
the GNU's extension instead of C99 standard.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should detect c++11 before enabling it

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Don't write to a poolid that this test might not have created

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12354 from umullan/correction

doc: correcting the object name

msg/simple/Pipe: avoid returning 0 on poll timeout

If poll times out it will return 0 (no data to read on socket).  In
165e5abdbf6311974d4001e43982b83d06f9e0cc we changed tcp_read_wait from
returning -1 to returning -errno, which means we return 0 instead of -1
in this case.

This makes tcp_read() get into an infinite loop by repeatedly trying to
read from the socket and getting EAGAIN.

Fix by explicitly checking for a 0 return from poll(2) and returning
EAGAIN in that case.

Fixes: http://tracker.ceph.com/issues/18184
Signed-off-by: Sage Weil <sage@redhat.com>

ReplicatedPG::record_write_error: don't leak orig_reply on cancel

The submit_log_entries machinery depends on the destructor for the
functor cleaning up after itself to handle cancelation.  I could have
introduced a local intrusive_ptr and captured that instead, but this is
slightly less magic.

Fixes: http://tracker.ceph.com/issues/18180
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12347 from gregsfortytwo/wip-18157-fix

ceph-fuse: start up log on parent process before shutdown

Reviewed-by: Sage Weil <sage@redhat.com>

client: drop setuid/setgid bits on ownership change

When we hold exclusive auth caps, then the client is responsible for
handling changes to the mode. Make sure we remove any setuid/setgid
bits on an ownership change.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: clear setuid/setgid bits on ownership changes

If we get a ownership change, POSIX mandates that you clear the
setuid and setgid bits unless you are "appropriately privileged", in
which case the OS is allowed to leave them intact.

Linux however always clears those bits, regardless of the process
privileges, as that makes it simpler to close some potential races.
Have ceph do the same.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: add new testcase for clearing setuid/setgid bits on chown/chgrp

Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: remove improper casts from SetSize test

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #12259 from liupan1111/wip-fix-nbd-max-part

rbd-nbd: support partition for rbd-nbd mapped raw block device

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: bluestore_sync_submit_transaction = false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12359 from tchaikov/wip-cmake

cmake: detect keyutils if WITH_LIBCEPHFS OR WITH_RBD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: removed qemu-iotest case 077

The test case is not stable due to racing console output. This
results in spurious failures.

Fixes: http://tracker.ceph.com/issues/10773
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: only show interesting flags in health warning

Also add PAUSEREC flag into the list.

Fixes: http://tracker.ceph.com/issues/18175
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12361 from dachary/wip-18030-pglog

tests: remove TestPGLog EXPECT_DEATH tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12208 from ukernel/wip-18047

mds: don't access mdsmap from log submit thread

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12077 from jcsp/wip-17954

mds: fix dropping events in standby replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12133 from jcsp/wip-16924

mds: don't maintain bloom filters in standby replay

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12360 from wido/docs-cephfs-datascan-workers

doc: Fix for worker arguments to cephfs-data-scan tool

Reviewed-by: John Spray <john.spray@redhat.com>

tests: remove TestPGLog EXPECT_DEATH tests

As per Sam Just advice, remove the EXPECT_DEATH tests to avoid
intermittent hang because they do not play well with threads.

Fixes: http://tracker.ceph.com/issues/18030
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12327 from iain-buclaw-sociomantic/pybind-validate-args

pybind/rados: Add @requires for all aio methods

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix for worker arguments to cephfs-data-scan tool

The docs were incorrect and these arguments should work.

Signed-off-by: Wido den Hollander <wido@42on.com>

cmake: turn krbd into a static lib

* simplify the link dependencies.
* s/keyutils/${KEYUTILS_LIBRARIES}/

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: find keyutils if rbd or libcephfs is enabled

* find_package(keyutils REQUIRED) if (WITH_LIBCEPHFS OR WITH_RBD)
  prior to this change, we detect keyutils if the building platform is not
  FreeBSD, we should instead check the WITH_* options, and let the
  maintainer to decided what is the best for his/her platform, and error
  out if the building host cannot fulfill the requirement to build the
  asserts.
* build krbd.cc if (WITH_RBD)

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix global commit latency

"start" is used to calculate the global bluestore commit latency
and hence shall not be updated at each internal state enter/exit.

Otherwise the l_bluestore_commit_lat counter won't reflect the
real commit latency precisely.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: correcting the object name
Signed-off-by: umullangi@walmartlabs.com

Merge pull request #12350 from chanyoung/struct-OSDOp-typos

osd: fix typos in "struct OSDOp" comments

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix typos in "struct OSDOp" comments

"nembers" -> "members"

Signed-off-by: Chanyoung Park <park910113@gmail.com>

Merge pull request #12344 from trociny/wip-resync-fixup

rbd-mirror: fix gmock warnings in bootstrap request unit tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11369 from theanalyst/kraken-release-notes

doc: release notes for 11.0.2

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add salient features of kraken 11.1.0 rc

Reordering to add Sage's release note comments on top

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add release notes for 11.1.0

Release notes as of 7b1293e70db51f620210332bde1f83ade15bd9d3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: bump release notes to 11.0.2

bumping release notes to 11.0.2 kraken and adding the single commit that
differs between the changes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes for 11.0.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #12346 from theanalyst/doc/changelog/v10.2.4

doc: adding changelog for v10.2.4

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-fuse: start up log on parent process before shutdown

Otherwise, we hit an assert in the Ceph context and logging teardown.

Fixes: http://tracker.ceph.com/issues/18157
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc: adding changelog for v10.2.4

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12328 from liewegas/wip-bluestore-flush

os/bluestore: flush before enumerating omap values

Merge pull request #12265 from liewegas/wip-bluestore-precondition

os/bluestore: drop old bluestore preconditioning; replace with wal preextension of file size

Merge pull request #12281 from dachary/wip-17743-memstore

tests: do not use memstore.test_temp_dir in two tests

Reviewed-by: David Zafman <dzafman@redhat.com>