rgw: Have ChainedCacheImpl::find return an optional

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19580 from cbodley/wip-22473

rgw: dont log EBUSY errors in 'sync error list'

Merge pull request #19455 from mkogan1/wip-rgw-swift_container_create_mem_inc

rgw: fix chained cache invalidation to prevent cache size growth

Merge remote-tracking branch 'gh/mimic-dev1'

Merge pull request #19600 from liewegas/wip-mimic-trusty-backports

qa: mimic-dev1 backports to avoid trusty nodes

Merge pull request #19544 from tchaikov/mimic-qa-pass-options-the-right-way

qa/stanalone: pass options using --<option-name>=<value>

Revewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19617 from liewegas/wip-rbd-socket-failures

qa/suites/rbd: fewer socket failures

qa/suites/rbd: fewer socket failures

500 makes the mgr fail.  Fewer failures, and whitelist them if they
happen.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19508 from cfanz/wip-mulizone-correct-comment

rgw: correct log output for metadata section name in RGWListBucketIndexesCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19604 from dillaman/wip-22485

qa/workunits/rbd: simplify split-brain test to avoid potential race

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19608 from luomuyao/dev

rgw: fix a typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

cleanup: rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge PR #19424 into master

* refs/pull/19424/head:
osdc/Journaler: introduce STATE_STOPPING state
osdc/Journaler: add 'stopping' check to various finish callbacks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18898 from Leeshine/wip-rgw-bug-22080

rgw: create sync-module instance when execute radosgw-admin data sync run

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/workunits/rbd: simplify split-brain test to avoid potential race

Fixes: http://tracker.ceph.com/issues/22485
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19456 from liewegas/wip-22373

qa/standalone/ceph-helpers: pass --verbose to ceph-disk

Merge pull request #19584 from qrGitHub/wip-rgw-unused-variable2

rgw: drop variable bl in rgw_op.cc

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17967 from tone-zhang/bug_21554

rgw: add "Accept-Ranges" to response header of Swift API

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #19274 from badone/wip-rgw_sync_log_trim-uninit-errors

rgw: Silence maybe-uninitialized false positives

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19310 from qrGitHub/wip-rgw-auth-retVal

rgw: fix return value of auth v2/v4

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19401 from yaozongyou/rgw-admin-improve-bucket-stats

rgw-admin: properly filtering bucket stats by user_id or bucket_name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19474 from qrGitHub/wip-rgw-redundant-S3AnonymousEngine

rgw: remove redundant S3AnonymousEngine

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19511 from qrGitHub/wip-rgw-optimize-timeSkewCheck

rgw: optimize time skew check

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/docker-test: drop support for ubuntu 14.04 and centos 6

* we cannot build on centos 6. and we are dropping the support for ubuntu
  14.04 also.
* add ubuntu 17.04: this release is not a LTS release, and added merely for
  demostration purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1acfa03ef922044ae68d37e580a296c7e9c37913)

qa/suites/hadoop: switch from trusty to ubuntu latest

use the ubuntu_face.yaml facet for the distro used for testing. since we
are dropping the support for trusty.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit dbe33502cbc0f69f54c2a467fa0bade585214389)

qa/suites/fs: use ubuntu_latest for multiclient tasks

instead of using ubuntu 14.04, since we want to drop the support of this
release.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 88311be4393586ae7f92862edebad907ee3a133f)

qa/suites/fs: use ubuntu_latest for libcephfs_java

* instead of using ubuntu 14.04, use ubuntu_latest.since we want
  to drop the support of this release.
* refactor this test to use the facet of ubuntu_latest.yaml.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit aa89bb2f93a0ee7b26dff3972f09c64529054744)

qa/tasks: remove test_buildpackages

the buildpackages feature is not part of ceph. and its test was moved to
teuthology in 5a43f8d5. let's remove the leftovers from ceph/ceph. BTW,
we have not run this test suite since then. and its dependency cannot be
fulfilled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 582f04508faa89b5d53cac9fe41e47f2b3d2e6aa)

Merge pull request #19576 from qrGitHub/wip-rgw-log-adjustment

rgw: adjust log format for lifecycle

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #19043 from liewegas/wip-22161

os/bluestore: prevent mount if osd_max_object_size >= 4G

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19273 from qrGitHub/wip-rgw-typo-UNKNOWN

rgw: correct typo UNKOWN to UNKNOWN

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18196 from liewegas/wip-bluestore-sync-onreadable

os/bluestore: synchronous on_applied completions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19578 from cernceph/dvanders_brm

mgr/balancer: fix KeyError in balancer rm

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #19534 into mimic-dev1

* refs/pull/19534/head:
qa: don't configure ec data pool with memstore

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: don't configure ec data pool with memstore

Fixes: http://tracker.ceph.com/issues/22436
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19586 from liewegas/wip-22231

Revert "msg/async/AsyncConnection: unregister connection when racing happened"

Revert "msg/async/AsyncConnection: unregister connection when racing happened"

This reverts commit 5216309c25522e9e4a3c3a03ceb927079de91e9b.  Seeing many
many instances of

     0> 2017-12-19 00:04:34.146 7fed2ebc1700 -1 /build/ceph-13.0.0-4128-gc27b472/src/msg/async/AsyncConnection.cc: In function 'void AsyncConnection::process()' thread 7fed2ebc1700 time 2017-12-19 00:04:34.147015
/build/ceph-13.0.0-4128-gc27b472/src/msg/async/AsyncConnection.cc: 726: FAILED assert(0 == "old msgs despite reconnect_seq feature")

 ceph version 13.0.0-4128-gc27b472 (c27b4726c18322825fbc2a58de2759766e332fa6) mimic (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55ce15c5bab2]
 2: (AsyncConnection::process()+0x25f6) [0x55ce15f347e6]
 3: (EventCenter::process_events(int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x6e8) [0x55ce15cf5988]
 4: (()+0xb7b965) [0x55ce15cf8965]
 5: (()+0xb8c80) [0x7fed30e02c80]
 6: (()+0x76ba) [0x7fed314f16ba]
 7: (clone()+0x6d) [0x7fed305683dd]

See http://tracker.ceph.com/issues/22231
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: create sync-module instance when execute radosgw-admin data sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: remove unused variable bl in file rgw_op.cc

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #19129 from qrGitHub/wip-rgw-unused-param

rgw: remove unused param in get_bucket_instance_policy_from_attr

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18662 from ZVampirEM77/wip-em-rewriteobj

rgw: fix rewrite a versioning object create a new object bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19131 from theanalyst/usage-trim

cls/rgw: trim all usage entries in cls_rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19272 from cfanz/wip2-optimize-get_obj_vals

cls_rgw: use more effective container operations in get_obj_vals

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19438 from cbodley/wip-qa-rgw-multisite-trim

qa/rgw: disable log trim in multisite suite

Merge pull request #17482 from yaozongyou/rgw-admin-process-all-gc-objects

rgw-admin: support for processing all gc objects including unexpired.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: dont log EBUSY errors in 'sync error list'

these temporary errors get retried automatically, so no admin
intervention is required. logging them only serves to waste space in
omap and obscure the more serious sync errors

Fixes: http://tracker.ceph.com/issues/22473
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #19234 into master

* refs/pull/19234/head:
mds: handle client session messages when mds is stopping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19533 from batrick/vstart-runner-fixes

vstart_runner: fixes for recent cephfs changes

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/balancer: fix KeyError in balancer rm

Fix the typo in the plan name which leads to a KeyError in balancer
rm.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Fixes: http://tracker.ceph.com/issues/22470

rgw: adjust log format for lifecycle

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

mds: handle client session messages when mds is stopping

handle session messages except CEPH_SESSION_REQUEST_OPEN. The problem
I found is that mds ignores CEPH_SESSION_REQUEST_RENEWCAPS, which
causes client sessions to become stale. Locker::revoke_stale_caps()
increases client caps' sequence number. This causes clients to warn
about caps' sequence number mismatch when handle caps import/export
message.

mds should handle CEPH_SESSION_FLUSHMSG_ACK message too. Because
one step of exporting subtree is flushing session messages.

Fixes: http://tracker.ceph.com/issues/22460
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

osdc/Journaler: introduce STATE_STOPPING state

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #19421 into master

* refs/pull/19421/head:
cephfs-journal-tool: tool would miss to report some invalid range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19277 into master

* refs/pull/19277/head:
cephfs: potential adjust failure in lru_expire

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #19548 from tchaikov/mimic-gcc-7

build mimic-dev1 with gcc 7

Merge pull request #19550 from dillaman/wip-22461

rbd-mirror: cluster watcher should ensure it has latest OSD map

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19458 from wido/ceph-volume-eol

ceph-volume: Add linesep/newline at end of JSON file when writing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19518 from jcsp/wip-mgr-access-denied

mgr: fix "access denied" message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19515 from tchaikov/wip-cmake-lttng-leak

 cmake: link libcommon with libstdc++ statically if WITH_STATIC_LIBSTDCXX

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19490 from adamemerson/wip-hypomodern

Update C++ standard to 14 and clean up

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19486 from liewegas/wip-22419

osd: pass pool options to ObjectStore on pg create

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #19385 from liewegas/wip-stdout-logs

mon: allow cluter and debug logs to go to stderr, with appropriate prefix

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19542 from tchaikov/mimic-22093

qa: decrease the msg_inject_socket_failures from 1/500 to 1/1000

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

cephfs-journal-tool: tool would miss to report some invalid range

Fixes: http://tracker.ceph.com/issues/22459
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

cephfs: potential adjust failure in lru_expire

Fix: the first adjust is no needed,it will never take real effect.
     the second 'adjust' may never get the chance to be executed
     suppose we can reach the second 'adjust', it will crash because the bottom list is empty now.

Fixes: http://tracker.ceph.com/issues/22458
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #19464 from myoungwon/wip-bug-fix-22369

osd: fix unordered read bug (for chunked object)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19535 into master

* refs/pull/19535/head:
doc: clarify type of client in config ref

Reviewed-by: Jos Collin <jcollin@redhat.com>

rbd-mirror: cluster watcher should ensure it has latest OSD map

Fixes: http://tracker.ceph.com/issues/22461
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #18512 into master

* refs/pull/18512/head:
mon,mgr,doc,test: standardize argument as role

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18899 into master

* refs/pull/18899/head:
mds: add asok command that dumps metadata popularity

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19170 into master

* refs/pull/19170/head:
mds: Improve the log info for read pointers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19271 into master

* refs/pull/19271/head:
client: update Dir::num_null_dentries in Dentry::{link,unlink}
client: drop null child dentries before try pruning inode's alias

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19414 into master

* refs/pull/19414/head:
mds: properly eval locks after importing inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19440 into master

* refs/pull/19440/head:
mds: update mds option descriptions
mds: obsolete MDSMap option configs
mds: organize Filesystem class def

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19540 from dillaman/wip-rbd-trash-list-enoent

rbd: 'trash list --long' will return a failure on non-cloned images

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rbd: 'trash list --long' will return a failure on non-cloned images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: link libcommon with libstdc++ statically if WITH_STATIC_LIBSTDCXX

this matches the linkage of libceph-common, because valgrind reports
Leak_StillReachable if daemons are linked against libceph-common, will
link daemons against common instead, and let common link libstdc++
statically, if WITH_STATIC_LIBSTDCXX=ON.

this change also reverts e6695bb6

Fixes: http://tracker.ceph.com/issues/22438
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit df9a59894291fe81e727babb631fb0e23cd1bbb0)

cmake: link global-static against common

* as global-static references symbols offered by common.
* and remove DPDK_LIBRARIES from global, as it is libcommon which
  is using DPDK.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a2cad3bb1cad57cab5f0acc11310613077866a87)

cmake: link executables against ceph-common

so they can have access to libstdc++ if it is linked statically.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit e6695bb63ace3c5ccb3398915e5af73decb4d9e7)

cmake: add WITH_STATIC_LIBSTDCXX option

to link libceph-common with libstdc++ if it is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4c2216de6a08f048374292a567dbabd083d64e87)

install-deps.sh: use tee for writing a file

because the stdout redirect is performed by current shell not `$SUDO
cat`, so $SUDO does not help here. use `$SUDO tee` instead, which is
able to read the stdin and write to the given file just as expected.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0beb846e57b94786e84c35927fe5ef3a67857fa3)

install-deps.sh: avoid re-installing g++-7

* add ubuntu-toolchain-r mirrors in case the ppa.launchpad.net is not
  accessible
* add ppa repo manually, it's faster than installing
  `software-properties-common` and then launch `add-apt-repository`
* hardwire $old to the gcc version shipped by the distro, simpler this
  way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b69e3efe64159417b2c80894ba5c249eb470d80a)

install-deps.sh: use GCC-7 on xenial also

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8a0935988fb845d0b88d97470b08e93a8709a03d)

install-deps.sh: install new gcc as the default the right way

* should install software-properties-common beforehand, otherwise
  the `add-apt-repository` command will not be available.
* the update-alternative commandline were copied from ceph-build,
  should remove the escape characters.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3882deae8c94802cfd49153ad2cf33f0a3dba60d)

install-deps.sh: enable testing repo on centos on aarch64

the DTS-6 is only available in the testing repo

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4cedada14e12e677a180cfa1fce6442ecc763204)

install-deps.sh: use DTS on centos if GCC is too old

please note, run-make-check.sh sources install-deps.sh here to import
the $PATH and other environmental variables, which could be changed by
the the DTS "enable" script.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3f8647a7e80b20079bec97d5be12372909083bb4)

install-deps.sh: install centos-release-scl-rh on aarch64

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 84660c977cc7511290a3c5497468052b3eaccf43)

install-deps.sh: use gcc-7 on trusty

* always install gcc-7 on trusty
* point g++ to g++-7 if not yet

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 24d0f935dc661804a5b1fd619120026950f727d2)

install-deps.sh: always use yum-config-manager for consistency

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit eccc961c728153c928baa4478e9c8d5a1139a834)

install-deps.sh: install devtoolset-7 for centos/rhel

ready them for gcc7 and libstdc++-7 for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f9aa49cee15cd571a8dc2efcd46a54b0a1f68f89)

cmake: disable FAIL_ON_WARNINGS for rocksdb

otherwise -Werror=implicit-fallthrough will fail the build with GCC-7

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6559a850512ddefef384d2733823ac2cb8027e3a)

qa/stanalone: pass options using --<option-name>=<value>

not "--<option-name> <value>', otherwise `ceph-authtool` would error
out:

$ CEPH_ARGS='--osd-map-max-advance 1000' bin/ceph-authtool --gen-print-key
bin/ceph-authtool: unexpected '1000'
usage: ceph-authtool keyringfile [OPTIONS]...
....

but using the syntax of `--<option-name>=<value>', it works:

$ CEPH_ARGS='--osd-map-max-advance=1000' bin/ceph-authtool --gen-print-key
AQBAhTNamf5+ABAASkAp/6IGq7LkUTEOMp/fgw==

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19493 from cernceph/dvanders_balancer

mgr/balancer: cast config vals to int or float

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Silence maybe-uninitialized false positives

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19482 from jcsp/wip-22361

mgr/balancer: don't use 'foo' tags on commands

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19465 from shinobu-x/no_copy_ctor_assignment

common,mds,osd: Explicitly delete copy ctor if noncopyable

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19211 from ZVampirEM77/wip-em-pg-cleanup

osd: fix typos and some cleanups

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19013 from yuyuyu101/wip-fix-racing

msg/async/AsyncConnection: unregister connection when racing happened

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: decrease the msg_inject_socket_failures from 1/500 to 1/1000

Fixes: http://tracker.ceph.com/issues/22093
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19361 from Songweibin/wip-init-image-id

pybind/rbd: support open the image by image_id

Reviewed-by: Jason Dillaman <dillaman@redhat.com>