rgw: add radosgw-admin sync error trim to trim sync error log

Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

Merge pull request #19691 from MrStupnikov/fix_logic

doc: Improved hitset parameters description

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Improve parameters description in cache_tiering article

hit_set parameters aren't described in a sequence they introduced.
This patch effecrively changes the sequence to improve
description's logic.

Signed-off-by: Alexey Stupnikov <aleksey.stupnikov@gmail.com>

Merge pull request #19690 from Songweibin/wip-doc-librbdpy

doc/rbd: correct the path of librbd python APIs

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: correct the path of librbd python APIs

* update the API Documentation page for librbdpy with the
  correct path.
* fix a typo.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #19255 from JianyuLi/joa-mdbalancer

MDS: make rebalancer evaluate the overload state of each mds with the same criterion

Merge pull request #19591 from qrGitHub/wip-rgw-unused-variable3

rgw: drop marker in RGWLC::process()

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19675 from qrGitHub/wip-rgw-typo-complete

rgw: fix typo compete to complete

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: fix typo compete to completes

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #19583 from qrGitHub/wip-rgw-unused-variable

rgw: drop iter in rgw_op.cc

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19667 from trociny/wip-affiliation

mailmap: update affiliation for Mykola Golub

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: remove unused variable iter in file rgw_op.cc

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #19618 from dillaman/wip-18435

librbd: filter out potential race with image rename

Reviewed-by: Mykola Golub <mgolub@suse.com>

mailmap: update affiliation for Mykola Golub

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19536 from dillaman/wip-rbd-mirror-trash

rbd-mirror: support deferred deletions of mirrored images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19623 from chnmagnus/wip-a-tiny-error

msg: drop duplicate include

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19601 from adamemerson/wip-CORS-of-course

rgw: retry CORS put/delete operations on ECANCELLED

Merge PR #19369 into master

* refs/pull/19369/head:
qa: update handling of fs status format
PendingReleaseNotes: add note for format change
mds/MDSMap : use arrary_section for mds stat

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19378 into master

* refs/pull/19378/head:
mds,rgw: Initializing standby_reply, sync_tracer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19556 into master

* refs/pull/19556/head:
mds: reduce debugging level for balancer messages

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19574 into master

* refs/pull/19574/head:
mds: track dirty dentries in separate list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19587 into master

* refs/pull/19587/head:
mds: add "client." before cap->get_client()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19609 into master

* refs/pull/19609/head:
mds: respect mds_client_writeable_range_max_inc_objs config

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19633 into master

* refs/pull/19633/head:
ceph_volume_client: fix usage of get_mds_map()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19580 from cbodley/wip-22473

rgw: dont log EBUSY errors in 'sync error list'

Merge pull request #19455 from mkogan1/wip-rgw-swift_container_create_mem_inc

rgw: fix chained cache invalidation to prevent cache size growth

qa: update handling of fs status format

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph_volume_client: fix usage of get_mds_map()

get_mds_map() is an instance method, and not a global
function. Use it as such.

Fixes: http://tracker.ceph.com/issues/22524
Signed-off-by: Ramana Raja <rraja@redhat.com>

mds: track dirty dentries in separate list

this should improve performance of large directory

Fixes: http://tracker.ceph.com/issues/19578
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

msg: the #include "include/assert.h" statement is repeated in src/msg/Dispatcher.h

Signed-off-by: chnmagnus chnmagnus@qq.com

PendingReleaseNotes: add note for format change

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: retry CORS put/delete operations on ECANCELLED

Fixes: http://tracker.ceph.com/issues/22517
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librbd: filter out potential race with image rename

Fixes: http://tracker.ceph.com/issues/18435
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge remote-tracking branch 'gh/mimic-dev1'

Merge pull request #19600 from liewegas/wip-mimic-trusty-backports

qa: mimic-dev1 backports to avoid trusty nodes

Merge pull request #19544 from tchaikov/mimic-qa-pass-options-the-right-way

qa/stanalone: pass options using --<option-name>=<value>

Revewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19617 from liewegas/wip-rbd-socket-failures

qa/suites/rbd: fewer socket failures

qa/suites/rbd: fewer socket failures

500 makes the mgr fail.  Fewer failures, and whitelist them if they
happen.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19508 from cfanz/wip-mulizone-correct-comment

rgw: correct log output for metadata section name in RGWListBucketIndexesCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19604 from dillaman/wip-22485

qa/workunits/rbd: simplify split-brain test to avoid potential race

Reviewed-by: Mykola Golub <mgolub@suse.com>

mds: add "client." before cap->get_client()

 unify the log and make it easy to search
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

simplify mds overload judgement logic

Signed-off-by: Jianyu Li <joannyli@tencent.com>

Make MDS evaluates the overload situation with the same criterion

Currently, the prep_rebalancer requires the mds_load of itself
being greater than average level mds_bal_min_rebalance percent
to be counted as exporter, however, for the other mds, as long
as their mdsload is greater than average level, they could be
consider as exporter. This inconsistent standard would cause
gray area in the rebalance activity: one may expect the other
do some work, but it didn't. With the increasing amount of mds,
this inconsistent decision would get worse and affects the
optimization effect of rebalancer.

Signed-off-by: Jianyu Li <joannyli@tencent.com>

Merge pull request #19608 from luomuyao/dev

rgw: fix a typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: respect mds_client_writeable_range_max_inc_objs config

get_layout_size_increment() return layout.get_period(). A period
contain layout.stripe_count objects.

The config is for limiting number of objects need to probe when
recovering a file.

Fixes: http://tracker.ceph.com/issues/22492
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

cleanup: rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge PR #19424 into master

* refs/pull/19424/head:
osdc/Journaler: introduce STATE_STOPPING state
osdc/Journaler: add 'stopping' check to various finish callbacks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18898 from Leeshine/wip-rgw-bug-22080

rgw: create sync-module instance when execute radosgw-admin data sync run

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/workunits/rbd: simplify split-brain test to avoid potential race

Fixes: http://tracker.ceph.com/issues/22485
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19456 from liewegas/wip-22373

qa/standalone/ceph-helpers: pass --verbose to ceph-disk

Merge pull request #19584 from qrGitHub/wip-rgw-unused-variable2

rgw: drop variable bl in rgw_op.cc

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17967 from tone-zhang/bug_21554

rgw: add "Accept-Ranges" to response header of Swift API

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #19274 from badone/wip-rgw_sync_log_trim-uninit-errors

rgw: Silence maybe-uninitialized false positives

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19310 from qrGitHub/wip-rgw-auth-retVal

rgw: fix return value of auth v2/v4

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19401 from yaozongyou/rgw-admin-improve-bucket-stats

rgw-admin: properly filtering bucket stats by user_id or bucket_name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19474 from qrGitHub/wip-rgw-redundant-S3AnonymousEngine

rgw: remove redundant S3AnonymousEngine

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19511 from qrGitHub/wip-rgw-optimize-timeSkewCheck

rgw: optimize time skew check

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/docker-test: drop support for ubuntu 14.04 and centos 6

* we cannot build on centos 6. and we are dropping the support for ubuntu
  14.04 also.
* add ubuntu 17.04: this release is not a LTS release, and added merely for
  demostration purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1acfa03ef922044ae68d37e580a296c7e9c37913)

qa/suites/hadoop: switch from trusty to ubuntu latest

use the ubuntu_face.yaml facet for the distro used for testing. since we
are dropping the support for trusty.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit dbe33502cbc0f69f54c2a467fa0bade585214389)

qa/suites/fs: use ubuntu_latest for multiclient tasks

instead of using ubuntu 14.04, since we want to drop the support of this
release.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 88311be4393586ae7f92862edebad907ee3a133f)

qa/suites/fs: use ubuntu_latest for libcephfs_java

* instead of using ubuntu 14.04, use ubuntu_latest.since we want
  to drop the support of this release.
* refactor this test to use the facet of ubuntu_latest.yaml.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit aa89bb2f93a0ee7b26dff3972f09c64529054744)

qa/tasks: remove test_buildpackages

the buildpackages feature is not part of ceph. and its test was moved to
teuthology in 5a43f8d5. let's remove the leftovers from ceph/ceph. BTW,
we have not run this test suite since then. and its dependency cannot be
fulfilled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 582f04508faa89b5d53cac9fe41e47f2b3d2e6aa)

Merge pull request #19576 from qrGitHub/wip-rgw-log-adjustment

rgw: adjust log format for lifecycle

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #19043 from liewegas/wip-22161

os/bluestore: prevent mount if osd_max_object_size >= 4G

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19273 from qrGitHub/wip-rgw-typo-UNKNOWN

rgw: correct typo UNKOWN to UNKNOWN

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18196 from liewegas/wip-bluestore-sync-onreadable

os/bluestore: synchronous on_applied completions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror: propagate deletion only if remote image doesn't exist

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: only the leader should initialize the image deleter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image deleter now only removes images from the trash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: move images to trash when propagating deletions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: integrate trash watcher within image deleter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: new 'rbd_mirroring_delete_delay' configuration option

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: async mirroring trash move state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: watch notifications for images added to trash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: use one image deleter per pool

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: avoid logging an error for expected bootstrap failures

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: free memory after image deleter fails to open image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async journal reset state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: incorporate new trash notification messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: watch/notify types for trash add/remove events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async trash move state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: add mock version of `create`

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19578 from cernceph/dvanders_brm

mgr/balancer: fix KeyError in balancer rm

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: remove unused varialbe "marker" in function RGWLC::process

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge PR #19534 into mimic-dev1

* refs/pull/19534/head:
qa: don't configure ec data pool with memstore

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: don't configure ec data pool with memstore

Fixes: http://tracker.ceph.com/issues/22436
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19586 from liewegas/wip-22231

Revert "msg/async/AsyncConnection: unregister connection when racing happened"

Revert "msg/async/AsyncConnection: unregister connection when racing happened"

This reverts commit 5216309c25522e9e4a3c3a03ceb927079de91e9b.  Seeing many
many instances of

     0> 2017-12-19 00:04:34.146 7fed2ebc1700 -1 /build/ceph-13.0.0-4128-gc27b472/src/msg/async/AsyncConnection.cc: In function 'void AsyncConnection::process()' thread 7fed2ebc1700 time 2017-12-19 00:04:34.147015
/build/ceph-13.0.0-4128-gc27b472/src/msg/async/AsyncConnection.cc: 726: FAILED assert(0 == "old msgs despite reconnect_seq feature")

 ceph version 13.0.0-4128-gc27b472 (c27b4726c18322825fbc2a58de2759766e332fa6) mimic (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55ce15c5bab2]
 2: (AsyncConnection::process()+0x25f6) [0x55ce15f347e6]
 3: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x6e8) [0x55ce15cf5988]
 4: (()+0xb7b965) [0x55ce15cf8965]
 5: (()+0xb8c80) [0x7fed30e02c80]
 6: (()+0x76ba) [0x7fed314f16ba]
 7: (clone()+0x6d) [0x7fed305683dd]

See http://tracker.ceph.com/issues/22231
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: create sync-module instance when execute radosgw-admin data sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: remove unused variable bl in file rgw_op.cc

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

mds: reduce debugging level for balancer messages

Fixes: http://tracker.ceph.com/issues/21853
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19129 from qrGitHub/wip-rgw-unused-param

rgw: remove unused param in get_bucket_instance_policy_from_attr

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18662 from ZVampirEM77/wip-em-rewriteobj

rgw: fix rewrite a versioning object create a new object bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19131 from theanalyst/usage-trim

cls/rgw: trim all usage entries in cls_rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19272 from cfanz/wip2-optimize-get_obj_vals

cls_rgw: use more effective container operations in get_obj_vals

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19438 from cbodley/wip-qa-rgw-multisite-trim

qa/rgw: disable log trim in multisite suite

Merge pull request #17482 from yaozongyou/rgw-admin-process-all-gc-objects

rgw-admin: support for processing all gc objects including unexpired.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: dont log EBUSY errors in 'sync error list'

these temporary errors get retried automatically, so no admin
intervention is required. logging them only serves to waste space in
omap and obscure the more serious sync errors

Fixes: http://tracker.ceph.com/issues/22473
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #19234 into master

* refs/pull/19234/head:
mds: handle client session messages when mds is stopping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>