Merge pull request #16756 from liewegas/wip-20113

ceph: wait for maps before doing 'ceph tell ... help'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16760 from cbodley/wip-rgw-disable-lifecycle-s3tests

qa/rgw: disable lifecycle tests because of expiration failures

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #16491 from cbodley/wip-20308

rgw: match wildcards in StringLike policy conditions

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #16738 from liewegas/wip-20810

os/bluestore: fix dirty_range on _do_clone_range

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16789 from liewegas/wip-ec-m-2

qa: avoid map-gap tests for k=2 m=1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16795 from liewegas/wip-mgr-whitelist

qa/suites/rados/mgr/tasks/failover: whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-erasure-code: do not test map gap with m=1

We test EC profiles with m=1 here, and mapgap can lead to incomplete pgs
because it takes an osd down and waits for healthy.

Fixes: http://tracker.ceph.com/issues/20844
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-coe-big/clsuter: 12 osds on 3 nodes not 4

smithi have 4 nvme partitions available, not 3.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16797 from jdurgin/wip-upgrade-jewel-x

qa: timeout when waiting for mgr to be available in healthy()

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash/worklaods/rados_api_tests: whitelist SMALLER_PGP_NUM

The rados/test.sh fiddles with pg_num.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/mgr/tasks/failover: whitelist

remote/smithi025/log/ceph.log.gz:2017-08-03 07:02:15.049074 mon.b mon.0 172.21.15.25:6789/0 197 : cluster [INF] Manager daemon x is unresponsive, replacing it with standby daemon y
remote/smithi025/log/ceph.log.gz:2017-08-03 07:03:10.078032 mon.b mon.0 172.21.15.25:6789/0 226 : cluster [WRN] Manager daemon x is unresponsive.  No standby daemons available.

x and y may be swapped, so whitelist the rest of the string.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16764 from liewegas/wip-release-notes

doc/release-notes: sort release note changes into the right section

Merge pull request #16765 from liewegas/wip-bluestore-docs

doc/rados/configuration: document bluestore

doc/rados/configuration/bluestore-config-ref: devices, checksumming, cache

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16773 from dmick/wip-make-check-cleanup

Clean up make check for persistent test nodes (like arm64)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16683 from Liuchang0812/clear-result-in-getchildren

os/bluestore: clear result in BlueRocksEnv::getChildren

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16717 from gmayyyha/osd-objectstore-20865

ceph-osd: fix auto detect which objectstore is currently running

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #16777 from SUSE/wip-20894

doc: document "rados cleanup" in rados manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16775 from ovh/bp-fr-pendingnotes-update

PendingReleaseNotes: mention forced recovery

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: document "rados cleanup" in rados manpage

Fixes: http://tracker.ceph.com/issues/20894
Signed-off-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: mention forced recovery

Mention forced recovery in release notes.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

test/encoding/readable.sh: clean up all tmpfiles/dirs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

run-make-check.sh: set ulimit -n (because no one else does)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #16678 from mychoxin/yuanxin_make_print_informative

print more information when run ceph-osd cmd with 'check options'

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #16713 into master

* refs/remotes/upstream/pull/16713/head:
qa: ignore failed MDS message during upgrade

Merge PR #16691 into master

* refs/remotes/upstream/pull/16691/head:
doc: update client.admin key refs to "mds 'allow'"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16714 into master

* refs/remotes/upstream/pull/16714/head:
qa: test export_pin is correct in dumped subtree
mds: print export_pin for dumped subtree

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #16732 from Abhishekvrshny/fix-rgw-mp-races

rgw : fix race in RGWCompleteMultipart

doc/rados/configuration: document bluestore

Initial pass here.  Not yet complete.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

rgw : fix race in RGWCompleteMultipart

Fixes : http://tracker.ceph.com/issues/20861

Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #16377 from theanalyst/doc/release-notes/12.1.1

doc: 12.1.1 & 12.1.2 release notes

Merge pull request #16308 from joscollin/wip-rbd-nbd-return-values

tools/rbd, rgw: Removed unreachable returns

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16370 from rzarzynski/wip-rgw-swift-prefixed-tempurls

rgw: add support for Swift's TempURLs with prefix-based scope

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16442 from theanalyst/rgw-admin-tenant-validate

rgw: validate tenant names during user create.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #16485 from rzarzynski/wip-rgw-s3anonauthengine

rgw: move the S3 anonymous auth handling to a dedicated engine.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16690 from scienceluo/wip-luo-unused-var-branch

common/rgw/rbd: remove some unused variables

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16692 from oritwas/wip-rgw-dir-suggest

rgw: Fix up to 1000 entries at a time in check_bad_index_multipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: release notes for v12.1.2

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.1.1 release notes add major changes

Also cleanup PendingReleasenotes to an empty file so that only newer
changes are tracked, adding the relevant section back to
RC1 where relevant. Moving all the RC1 announcements back to RC2, when
we go to 12.2.0 we'll collapse all of these back to the release
announcments

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.1.1 release notes

Adding the changelog from 12.1.0 -> 12.1.1

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16759 from liewegas/wip-powercycle-health

qa/suites/powercycle: whitelist health for thrashing

qa/suites/powercycle: whitelist health for thrashing

Signed-off-by: Sage Weil <sage@redhat.com>

qa/rgw: disable lifecycle tests because of expiration failures

lifecycle expiration tests are too reliant on timing, and have been
failing consistently for a long time

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16758 from ceph/luminous

sync luminous tag back to master

Merge pull request #16535 from liewegas/wip-osd-shard-init

osd: initialize waiting_for_pg_osdmap on startup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16684 from tchaikov/wip-pybind-replace-deprecated-apis

pybind/rados: use new APIs instead of deprecated ones

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16735 from dillaman/wip-20869

common: avoid statically allocating configuration options

Reviewed-by: John Spray <john.spray@redhat.com>

ceph: wait for maps before doing 'ceph tell ... help'

If we don't have a mgrmap yet (usually we do, but it's racy) MgrClient
will assume EACCES if we try to send a command.  This can lead to
EACCES from 'ceph tell mgr help'.

The mon sends maps in alphanumeric order, so waiting for osdmap will
include mgrmap and fsmap.  That's everything (current) except servicemap,
but we can't 'tell' those anyway.

Fixes: http://tracker.ceph.com/issues/20113
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16747 from tchaikov/wip-doc-fixes

doc: document bluestore compression settings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: document bluestore compression settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/8/radosgw-admin: document --compression option

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: timeout when waiting for mgr to be available

Otherwise during upgrades we wait forever.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16741 from liewegas/wip-more-doc-links

doc/release-notes: fix links, formatting; add crush device class docs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16722 from tchaikov/wip-qa-fixes

qa/suites: escape the parenthesis of the whitelist text

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16474 from shun-s/wip-fix-deactive

ceph-disk: do not remove mount point if deactive --once

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites: more whitelisting

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/release-notes: clean up links

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/crush-map: more device class docs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix dirty_range on _do_clone_range

If we are cloning several extents that start from logical offset 0, we may
need to dirty those extents on the source object if they are being made
shared.  However, if there are two or more such extents, and the first
starts at 0, the current code will adjust dirty_range_start to the
second extent because the second time around the loop dirty_range_begin
is still 0.

Fix by adjusting the condition to be dirty_range_begin and dirty_range_end
== 0 so that it only triggers on the first pass.

Fixes: http://tracker.ceph.com/issues/20810
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: policy uses ceph::not_fn instead of std::not2

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add std::not_fn to backport14.h and unit tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add missing object tagging actions to policy tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add tests for match_wildcards, match_policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: rename ceph_test_rgw_iam_policy to unittest_

tagging this as a unittest will make it run during 'make check'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: match wildcards in StringLike policy conditions

Fixes: http://tracker.ceph.com/issues/20308
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split match into match_policy, match_wildcards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16700 from jcsp/wip-20803

osd/mon/mds: fix `config set` tell command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16701 from jcsp/wip-health-format

mon: extensible output format for health checks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16706 from jdurgin/wip-20850

osd: do not try to set device class before luminous

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16712 from gregsfortytwo/wip-20808

osd: pg: be more careful with locking around forced pg recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16688 from amitkumar50/cov-1395347-1

os: Argument cannot be negative

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

12.1.2

Merge pull request #15801 from zhangsw/feature-bucket-sync

rgw multisite: feature of bucket sync enable/disable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common: avoid statically allocating configuration options

This was resulting in seg faults within rbd-fuse during shutdown. This also
fixes the "variable tracking size limit exceeded with -fvar-tracking-assignments,
retrying without" compiler warning".

Fixes: http://tracker.ceph.com/issues/20869
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16616 from tchaikov/wip-doc-cleanup

doc: remove docs on non-existant command

rbd-fuse: remove dependency on ceph-common

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16727 from jcsp/wip-doc-config-hel

doc/qa: cover `config help` command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16707 from ivancich/wip-more-dmclock-docs

doc: Describe mClock's use within Ceph in great detail.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16730 from liewegas/wip-more-doc-links

doc/release-notes: fix upmap and osd replacement links; add fixme

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/release-notes: fix a few doc links

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16653 from amitkumar50/Coverity-1409697

rgw: Use comparison instead of assignment

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

os: Argument cannot be negative
Fixed:

** CID 1395347 (#1 of 1): Argument cannot be negative (NEGATIVE_RETURNS)
3. negative_returns: fd is passed to a parameter that cannot be negative

Signed-off-by: Amit Kumar amitkuma@redhat.com

Merge pull request #16656 from idryomov/wip-qa-newer-fio

qa/tasks/rbd_fio: bump default fio version to 2.21

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: initialize shardedwq osdmap references on startup

The _process methods assumes the shard osdmap refs are
non-null, but that leaves a window between startup and
the first call to prune_pg_waiters (via consume_map()),
which can lead to a seg fault.

Calling prune during startup, before we have any work,
will initialize those references.

Fixes: http://tracker.ceph.com/issues/20748
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: escape the parenthesis of the whitelist text

so we can avoid the warnings like

grep: Unmatched ( or \(

because we pass the whitelisted string to `egrep -v "$1"` directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-disk: fix the path to ceph-helpers-root.sh

partially reverts 841f3bd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16652 from shangzhong/wip-update-quick-guide

doc: update the pool names created by vstart.sh by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16637 from ceph/wip-doc-links

doc: upmap docs; various missing links for release notes

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/release-notes: add various links to docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/upmap: document new upmap feature

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15838 from theanalyst/doc-pool-ns-update

doc: update the rados namespace docs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16623 from xiexingguo/wip-weightset

crush: a couple of weight-set fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16611 from liewegas/wip-doc-health

doc/rados/operations/health-checks: osd section

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/rados/operations/health-checks: add PG health check commentary

Include a link to pg-repair.rst, although there is no
content there yet.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: include which pgs have unfound objects in detail

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: put the _toofull states under DEGRADED_FULL

I think this was an oversight?

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: 'incomplete' means data is unavailable

Well, data is certainly unavailable, and may also be
degraded in the sense that we can't peer.  I think
unavailable is the more severe of the two, though, so
let's put it there!

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/health-checks: osd section

First paragraph: explain what the error means.

Second or later paragraph: describe steps to fix or mitigate.

Signed-off-by: Sage Weil <sage@redhat.com>