rgw: break sending data-log list infinitely

radosgw send data-log list infinitely when opposite end trimmed the data-log and in quiescence.
Fixes: http://tracker.ceph.com/issues/20951
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #16802 into master

* refs/remotes/upstream/pull/16802/head:
qa: update wait_for_health for new health json syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update wait_for_health for new health json syntax

Fixes: http://tracker.ceph.com/issues/20890
Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #16766 into master

* refs/remotes/upstream/pull/16766/head:
cephfs: Clarify health warning
cephfs: Always print filesystem name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: Clarify health warning

Add text to MDS_INSUFFICIENT STANDBY to clarify it refers
to MDS daemons.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Always print filesystem name

Previously, the FSMap summary omitted the filesystem
name in clusters with only one filesystem. Print the
file system name no matter what.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #16740 from liewegas/wip-crush-tree

mon: show class in 'osd crush tree' output; sort output

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #16772 into master

* refs/remotes/upstream/pull/16772/head:
qa: add health whitelist for all fs sub-suites

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add health whitelist for all fs sub-suites

Fixes: http://tracker.ceph.com/issues/20892
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16768 into master

* refs/remotes/upstream/pull/16768/head:
qa: fix log whitelist string

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #16743 into master

* refs/remotes/upstream/pull/16743/head:
mds: add inter-mds operations to PerfCounter

Merge PR #16654 into master

* refs/remotes/upstream/pull/16654/head:
mds: fix race in "open inode by inode number"
mds: cleanup {do_open,_do_find}_ino_peer()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16631 into master

* refs/remotes/upstream/pull/16631/head:
cephfs: Resend discover when joining from starting state
cephfs: Wait for MDCache to be ready before starting log segment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16363 into master

* refs/remotes/upstream/pull/16363/head:
mds: fix health checks for multiple filesystems
mon: fix log message on mds rank stop
mon: uniform way of referring to MDS daemons
mds: revise health checks for _FAILED/_DEGRADED
mon/MDSMonitor: log messages on MDS rank changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16292 into master

* refs/remotes/upstream/pull/16292/head:
qa: use new hex rep of inode
qa: fix whitelist error message
mds: refine "Scrub error" cluster log message
mds: polish clog messages
doc: developer logging guidance

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16749 from tchaikov/wip-restful-delete-key

mgr: handle "module.set_config(.., None)" correctly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16756 from liewegas/wip-20113

ceph: wait for maps before doing 'ceph tell ... help'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16760 from cbodley/wip-rgw-disable-lifecycle-s3tests

qa/rgw: disable lifecycle tests because of expiration failures

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #16491 from cbodley/wip-20308

rgw: match wildcards in StringLike policy conditions

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #16738 from liewegas/wip-20810

os/bluestore: fix dirty_range on _do_clone_range

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16789 from liewegas/wip-ec-m-2

qa: avoid map-gap tests for k=2 m=1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16795 from liewegas/wip-mgr-whitelist

qa/suites/rados/mgr/tasks/failover: whitelist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-erasure-code: do not test map gap with m=1

We test EC profiles with m=1 here, and mapgap can lead to incomplete pgs
because it takes an osd down and waits for healthy.

Fixes: http://tracker.ceph.com/issues/20844
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-coe-big/clsuter: 12 osds on 3 nodes not 4

smithi have 4 nvme partitions available, not 3.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16797 from jdurgin/wip-upgrade-jewel-x

qa: timeout when waiting for mgr to be available in healthy()

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code-big: add k=4 m=2

Get better coverage for larger codes.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/monthrash/worklaods/rados_api_tests: whitelist SMALLER_PGP_NUM

The rados/test.sh fiddles with pg_num.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/mgr/tasks/failover: whitelist

remote/smithi025/log/ceph.log.gz:2017-08-03 07:02:15.049074 mon.b mon.0 172.21.15.25:6789/0 197 : cluster [INF] Manager daemon x is unresponsive, replacing it with standby daemon y
remote/smithi025/log/ceph.log.gz:2017-08-03 07:03:10.078032 mon.b mon.0 172.21.15.25:6789/0 226 : cluster [WRN] Manager daemon x is unresponsive.  No standby daemons available.

x and y may be swapped, so whitelist the rest of the string.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16764 from liewegas/wip-release-notes

doc/release-notes: sort release note changes into the right section

Merge pull request #16765 from liewegas/wip-bluestore-docs

doc/rados/configuration: document bluestore

doc/rados/configuration/bluestore-config-ref: devices, checksumming, cache

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16773 from dmick/wip-make-check-cleanup

Clean up make check for persistent test nodes (like arm64)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16683 from Liuchang0812/clear-result-in-getchildren

os/bluestore: clear result in BlueRocksEnv::getChildren

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16717 from gmayyyha/osd-objectstore-20865

ceph-osd: fix auto detect which objectstore is currently running

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #16777 from SUSE/wip-20894

doc: document "rados cleanup" in rados manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16775 from ovh/bp-fr-pendingnotes-update

PendingReleaseNotes: mention forced recovery

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: document "rados cleanup" in rados manpage

Fixes: http://tracker.ceph.com/issues/20894
Signed-off-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: mention forced recovery

Mention forced recovery in release notes.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

qa: use new hex rep of inode

Resolves a failure from QA:

    2017-08-02T19:23:27.489 INFO:tasks.cephfs_test_runner:======================================================================
    2017-08-02T19:23:27.489 INFO:tasks.cephfs_test_runner:FAIL: test_oversize (tasks.cephfs.test_fragment.TestFragmentation)
    2017-08-02T19:23:27.489 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20170802/qa/tasks/cephfs/test_fragment.py", line 71, in test_oversize
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:    self.assertEqual(frags[0]['dirfrag'], "10000000000.0*")
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:AssertionError: u'0x10000000000.0*' != '10000000000.0*'
    2017-08-02T19:23:27.490 INFO:tasks.cephfs_test_runner:

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/encoding/readable.sh: clean up all tmpfiles/dirs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

run-make-check.sh: set ulimit -n (because no one else does)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #16678 from mychoxin/yuanxin_make_print_informative

print more information when run ceph-osd cmd with 'check options'

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #16713 into master

* refs/remotes/upstream/pull/16713/head:
qa: ignore failed MDS message during upgrade

Merge PR #16691 into master

* refs/remotes/upstream/pull/16691/head:
doc: update client.admin key refs to "mds 'allow'"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16714 into master

* refs/remotes/upstream/pull/16714/head:
qa: test export_pin is correct in dumped subtree
mds: print export_pin for dumped subtree

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

qa: fix whitelist error message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix log whitelist string

Fixes: http://tracker.ceph.com/issues/20889
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16732 from Abhishekvrshny/fix-rgw-mp-races

rgw : fix race in RGWCompleteMultipart

doc/rados/configuration: document bluestore

Initial pass here.  Not yet complete.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

rgw : fix race in RGWCompleteMultipart

Fixes : http://tracker.ceph.com/issues/20861

Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

Merge pull request #16377 from theanalyst/doc/release-notes/12.1.1

doc: 12.1.1 & 12.1.2 release notes

Merge pull request #16308 from joscollin/wip-rbd-nbd-return-values

tools/rbd, rgw: Removed unreachable returns

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16370 from rzarzynski/wip-rgw-swift-prefixed-tempurls

rgw: add support for Swift's TempURLs with prefix-based scope

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16442 from theanalyst/rgw-admin-tenant-validate

rgw: validate tenant names during user create.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #16485 from rzarzynski/wip-rgw-s3anonauthengine

rgw: move the S3 anonymous auth handling to a dedicated engine.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16690 from scienceluo/wip-luo-unused-var-branch

common/rgw/rbd: remove some unused variables

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16692 from oritwas/wip-rgw-dir-suggest

rgw: Fix up to 1000 entries at a time in check_bad_index_multipart

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: release notes for v12.1.2

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.1.1 release notes add major changes

Also cleanup PendingReleasenotes to an empty file so that only newer
changes are tracked, adding the relevant section back to
RC1 where relevant. Moving all the RC1 announcements back to RC2, when
we go to 12.2.0 we'll collapse all of these back to the release
announcments

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.1.1 release notes

Adding the changelog from 12.1.0 -> 12.1.1

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16759 from liewegas/wip-powercycle-health

qa/suites/powercycle: whitelist health for thrashing

qa/suites/powercycle: whitelist health for thrashing

Signed-off-by: Sage Weil <sage@redhat.com>

qa/rgw: disable lifecycle tests because of expiration failures

lifecycle expiration tests are too reliant on timing, and have been
failing consistently for a long time

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mds: add inter-mds operations to PerfCounter

Fixes: http://tracker.ceph.com/issues/20760
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16758 from ceph/luminous

sync luminous tag back to master

Merge pull request #16535 from liewegas/wip-osd-shard-init

osd: initialize waiting_for_pg_osdmap on startup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16684 from tchaikov/wip-pybind-replace-deprecated-apis

pybind/rados: use new APIs instead of deprecated ones

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16735 from dillaman/wip-20869

common: avoid statically allocating configuration options

Reviewed-by: John Spray <john.spray@redhat.com>

ceph: wait for maps before doing 'ceph tell ... help'

If we don't have a mgrmap yet (usually we do, but it's racy) MgrClient
will assume EACCES if we try to send a command.  This can lead to
EACCES from 'ceph tell mgr help'.

The mon sends maps in alphanumeric order, so waiting for osdmap will
include mgrmap and fsmap.  That's everything (current) except servicemap,
but we can't 'tell' those anyway.

Fixes: http://tracker.ceph.com/issues/20113
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16747 from tchaikov/wip-doc-fixes

doc: document bluestore compression settings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

qa: refactor suites/rados/rest/mgr-restful

- use "ceph restful restart" to restart the restful API server instead
of restarting the ceph-mgr
- test "ceph restful delete-key"
- test "ceph restful list-keys"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: handle "module.set_config(.., None)" correctly

otherwise "ceph restful delete-key foo" would fail:

TypeError: ceph_config_set() argument 3 must be string, not None

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: document bluestore compression settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/8/radosgw-admin: document --compression option

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: timeout when waiting for mgr to be available

Otherwise during upgrades we wait forever.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16741 from liewegas/wip-more-doc-links

doc/release-notes: fix links, formatting; add crush device class docs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16722 from tchaikov/wip-qa-fixes

qa/suites: escape the parenthesis of the whitelist text

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16474 from shun-s/wip-fix-deactive

ceph-disk: do not remove mount point if deactive --once

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites: more whitelisting

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/release-notes: clean up links

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/operations/crush-map: more device class docs

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushTreeDumper: sort tree items by (class, name)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/crush/CrushWrapper: include CLASS column in 'osd crush tree'

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix dirty_range on _do_clone_range

If we are cloning several extents that start from logical offset 0, we may
need to dirty those extents on the source object if they are being made
shared.  However, if there are two or more such extents, and the first
starts at 0, the current code will adjust dirty_range_start to the
second extent because the second time around the loop dirty_range_begin
is still 0.

Fix by adjusting the condition to be dirty_range_begin and dirty_range_end
== 0 so that it only triggers on the first pass.

Fixes: http://tracker.ceph.com/issues/20810
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: policy uses ceph::not_fn instead of std::not2

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add std::not_fn to backport14.h and unit tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add missing object tagging actions to policy tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add tests for match_wildcards, match_policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: rename ceph_test_rgw_iam_policy to unittest_

tagging this as a unittest will make it run during 'make check'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: match wildcards in StringLike policy conditions

Fixes: http://tracker.ceph.com/issues/20308
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split match into match_policy, match_wildcards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16700 from jcsp/wip-20803

osd/mon/mds: fix `config set` tell command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16701 from jcsp/wip-health-format

mon: extensible output format for health checks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16706 from jdurgin/wip-20850

osd: do not try to set device class before luminous

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16712 from gregsfortytwo/wip-20808

osd: pg: be more careful with locking around forced pg recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16688 from amitkumar50/cov-1395347-1

os: Argument cannot be negative

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

12.1.2

Merge pull request #15801 from zhangsw/feature-bucket-sync

rgw multisite: feature of bucket sync enable/disable

Reviewed-by: Casey Bodley <cbodley@redhat.com>