ceph-volume tests verify proper formatting of RuntimeError on vg/lv error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare correctly format argument for vg/lv error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19264 from theanalyst/doc/releases/12.2.2

doc: add 12.2.2 release notes

Reviewed-By: Sage Weil <sage@redhat.com>
Reviewed-By: Nathan Cutler <ncutler@suse.com>

Merge pull request #19284 from theanalyst/changelog/12.2.2

doc: add changelog for v12.2.2

doc: add changelog for v12.2.2

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #19278 from badone/wip-librgw_file_marker-unused-function-warnings

rgw_file: Silence unused-function warnings

Merge pull request #19281 from qrGitHub/wip-rgw-typo-anynoymous

rgw: fix typo anynoymous to anonymous

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add 12.2.2 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: correct typo anynoymous to anonymous

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

rgw_file: Silence unused-function warnings

Comment out unused insertion operators.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19233 from tchaikov/wip-no-mon-health-to-clog-when-upgrading

qa: disable mon-health-to-clog in upgrade test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19225 from tchaikov/wip-more-whitelist

qa/suites/rados/singleton: more whitelist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19202 from tchaikov/wip-check-pthread-ext

cmake,common/RWLock: check for libpthread extensions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19160 from shinobu-x/bluestore_tool_remove_redundant_code

os/bluestore/bluestore_tool: Move redundant code into one method

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #19100 from tchaikov/wip-log-reuse-streambuf

common/log: Speed improvement for log

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18124 from amitkumar50/cleanup-Min-Max

cleanup: Replacing MIN,MAX with std::min,std::max

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18991 from dmick/wip-ceph-disk-fsid

ceph_disk: allow "no fsid" on activate

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19262 from ceph/wip-fix-distro-master

tests - Replaced requests for "centos 7.3" to centos_latest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19164 from hrchu/adminopsBindings

doc: rgw adminops binding libraries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: add 12.2.2 to releases table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: cleanup PendingReleaseNotes

drop sections already in previous releases, keeping only Mimic sections
and a new section header for items going post 12.2.2

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa: disable mon-health-to-clog in upgrade test

otherwise the cluster will complain that the quorum size is smaller than
that of monmap.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests - Replaced requests for "centos 7.3" to centos_latest

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #19247 from shinobu-x/rados_op_crush-map-edits_straw2

doc/rados: Add explanation of straw2

Merge pull request #19254 from tchaikov/wip-ceph-disk-mock-get-fsid

ceph-disk: mock get fsid

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-disk: silence deprecate warnings while testing

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk/tests: mock get_fsid

should offer a valid fsid for the ceph-disk under testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #18995 into master

* refs/pull/18995/head:
client: emplace Cap in Inode caps map

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19009 into master

* refs/pull/19009/head:
client: Switch to new style config options and some cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19069 into master

* refs/pull/19069/head:
qa/tasks/ceph_fuse: fix ceph-fuse mount failed

Reviewed-by: Yunfei Guan <yunfei.guan@xtaotech.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19078 into master

* refs/pull/19078/head:
pybind: return error message when ceph_mds_command() returns error
mds: Fix error message when mds not active

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19095 into master

* refs/pull/19095/head:
test: pass correct buffer length to ceph_getdents

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19181 into master

* refs/pull/19181/head:
client: give full error message on failed remount
client: use PATH_MAX for mount path size

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19207 into master

* refs/pull/19207/head:
mds: handle client reconnect gather race

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: Add explanation of straw2

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #15482 from myoungwon/wip-chunked-manifest

osd,librados: add manifest, operations for chunked object

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_disk: allow "no fsid" on activate

The intent was to allow "no fsid" configurations when only one
conf file named 'ceph.conf' was present, but the code has a bug
in that ceph-osd --show-config-value will return a default all-0
uuid.  Treat 'all-0' as we were treating 'None'.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #18974 from liewegas/wip-22145

osd/PG: restart recovery if NotRecovering and unfound found

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: restart recovery if NotRecovering and unfound found

If we are in recovery_unfound state waiting for unfound objects, and we
find them, we need to restart the recovery reservation process so that we
can recover.  Do this by queueing DoRecover() event instead of calling
queue_recovery() (which won't do anything since we're not in
recoverying|backfilling pg states).

Make the parent Active state ignore DoRecovery so that if we are already
in some phase of recovery/backfill the event gets ignored.  It is already
handled by the other important substates that care, like Clean (for
repair's benefit).

I'm not sure why states like Activating are paying attention tot his vevent...

Fixes: http://tracker.ceph.com/issues/22145
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18954 from adamemerson/wip-hole-in-the-bucket-dear-liza

rgw: Add try_refresh_bucket_info function
rgw: Add retry_raced_bucket_write
rgw: Handle stale bucket info in RGWPutMetadataBucket
rgw: Handle stale bucket info in RGWSetBucketVersioning
rgw: Handle stale bucket info in RGWSetBucketWebsite
rgw: Handle stale bucket info in RGWDeleteBucketWebsite
rgw: Handle stale bucket info in RGWPutBucketPolicy
rgw: Handle stale bucket info in RGWDeleteBucketPolicy
rgw: Expire entries in bucket info cache

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19215 from joscollin/wip-mount-fuse-ceph-typo

doc: Fix typo in mount.fuse.ceph

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19209 from joscollin/wip-githubman-ktdreyer

githubmap: Add ktdreyer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19208 from tchaikov/wip-spdk-17.10

 spdk: update SPDK to v17.10

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19238 from tchaikov/wip-22266

tools/ceph_monstore_tool: rebuild initial mgrmap also

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #19079 from adamemerson/wip-static-ptr

Static Pointer

doc: Fix typo in mount.fuse.ceph

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #19212 into master

* refs/pull/19212/head:
doc: Add section about MDS cache limits

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19213 from ceph/wip-bz1518264

ceph-volume: correctly fallback to bluestore when no objectstore is specified

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18267 from joscollin/wip-update-radosgw-admin-doc

doc: Update commands and options in radosgw-admin
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

tools/ceph_monstore_tool: rebuild initial mgrmap also

Fixes: http://tracker.ceph.com/issues/22266
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18649 from b-ranto/wip-restful-fixes

mgr/restful: A couple of restful fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa/tasks/ceph: add mon-health-to-clog option to "restart"

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Add section about MDS cache limits

Signed-off-by: Bara Ancincova (bara@redhat.com)

doc: Update all OPTIONS sections in radosgw-admin

Updated all OPTIONS sections in RADOSGW-ADMIN doc page, using the latest options from rgw_admin.cc::usage()

Signed-off-by: Jos Collin <jcollin@redhat.com>

qa/tasks/ceph_manager: add inject_args() method

* move Thrasher._set_config() to CephManager, and make it a public
  method, and rename it to inject_args(),
* use this method instead of using 'tell ... injectargs ...' directly

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,rpm,deb: update to accommodate SPDK v17.10

* cmake/modules/BuildSPDK.cmake: add lvol
* cmake/modules/BuildDPDK.cmake: add pci and bus_pci
* ceph.spec.in, cmake/modules/BuildSPDK.cmake, debian/control:
  re-introduce libuuid dependency, as 17.07 added lvol. and the latter
  depends on uuid.
* cmake/modules/BuildSPDK.cmake: avoid introducing local variable of
  `iface_libs`.
* cmake/modules/patch-dpdk-conf.sh: disable
  CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES, this option introduces the
  balanced allocation of memory. but it also requires libnuma-dev.
  let's disable it for now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9571 from yehudasa/wip-rgw-redirect-zone

rgw: add a default redirect field for zones

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19159 from shinobu-x/drop_using_namespace_std

common: drop namespace using directives for std

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19165 from xiaoxichen/fix_used_pct

mon/PGMap: Fix %USED calculation bug.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19156 from cyx1231st/fix_blkin_tracing

blkin: Fix unconditional tracing in OSD

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19149 from rzarzynski/wip-common-perfguards

common:  RAII-styled mechanism for updating PerfCounters

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19112 from xiaoxichen/osd_mapped_pool

OSD/admin_socket: add get_mapped_pools command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18959 from cooboos/wip-cephdencoder-LCtype

ceph-dencoder: add support for RGWLifecycleConfiguration

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19174 from smithfarm/wip-cmake-exclude-gitignore

cmake: mgr: exclude .gitignore

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18938 from tchaikov/wip-cmake

cmake: check gcc version not release date for libstdc++ saneness

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rados/singleton: more whitelist

* SLOW_OPS is normal in a cluster with flattering OSDs
* so is OBJECT_MISPLACED.

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: give full error message on failed remount

Fixes: http://tracker.ceph.com/issues/22254
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19206 into master

* refs/pull/19206/head:
doc: add pre-requisite to enable CephFS snapshot

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

common: Add static_ptr

Add static_ptr, a pointer-like class that contains its own storage,
avoiding use of the heap.

The full range of *_pointer_cast functions are included.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cmake,common/RWLock: check for libpthread extensions

pthread_rwlockattr_setkind_np() is a GNU extension of libpthread. and
Tianshan Qu pointed out, we cannot use
ifdef(PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP) to detect the
availability of this function, because it's an enum not a macro. so,
like other *_np() extensions, we check this one also using cmake at
the configure phase.

Reported-by: Tianshan Qu <tianshan@xsky.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19199 from tchaikov/wip-22136

 qa/ceph-disk: enlarge the simulated SCSI disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate correct detection and fallback of objectstore flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.lvm verify objectstore flags in activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19163 from vshankar/rbd-image-map-mem-leak

rbd-mirror: ImageMap memory leak fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

.githubmap: Add ktdreyer

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19172 from joscollin/wip-mount-fuse-ceph-doc

doc: doc for mount.fuse.ceph

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: doc for mount.fuse.ceph

Created doc for mount.fuse.ceph.

Fixes: http://tracker.ceph.com/issues/21539
Signed-off-by: Jos Collin <jcollin@redhat.com>

spdk: update SPDK to v17.10

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_multi: add a test for zone redirect

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add option to configure redirect_zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mds: handle client reconnect gather race

Fixes: http://tracker.ceph.com/issues/22263
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

doc: add pre-requisite to enable CephFS snapshot
Signed-off-by: Jian Su <sujian@umcloud.com>

common/log: Added new version of PrebufferedStreambuf, tuned for log usage

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/ceph-disk: enlarge the simulated SCSI disk

100MB will be allocated for journal, and the remaining 100MB is for data
device. taking the inode into consideration, there will be approximately
87988 kB available for the activated OSD. and it will complain with a
"nearfull" state.

Fixes: http://tracker.ceph.com/issues/22136
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: should throw if fail to read key-management-mode

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: fix the logging messages

logging.Logger.warn(format, *args). so no need to prepar a string
for it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: path_set_context() after rename()

it does not matter if we chown/restorecon before or after the rename,
but the logging message looks better this way: instead of fixing the
.tmp files, we are updating the attributes of the dest files w/o
.${pid}.tmp extension.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-disk: do not redirect stderr to stdout

normally, if we care about the output of ceph-disk, we expect a json
string, and ceph-disk sends the output to stdout, and errors/warnings
to stderr. so everything works as expected. and the test should also
follow this tradition. for example, if deprecated warnings are printed,
the warning message should not be collected along with the json string.

see also: d44334f3

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: Fix %USED calculation bug.

Previous code forgot to multiple raw_used_ratio to calculate
used byte.

Fixes: http://tracker.ceph.com/issues/22247
Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

common/log: Performance test for logging.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

rgw: Expire entries in bucket info cache

To bound the degree to which an RGW instance can go out to lunch if
the watch/notify breaks down, force refresh of any cache entry over a
certain age.

Fifteen minutes by default, and expiration can be turned off entirely.

This is separate from the LRU. The LRU removes entries based on the
last time of access. This expiration patch forces refresh based on the
last time they were updated.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWDeleteBucketPolicy

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWPutBucketPolicy

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWDeleteBucketWebsite

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWSetBucketWebsite

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWSetBucketVersioning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWPutMetadataBucket

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add retry_raced_bucket_write

If the OSD informs us that our bucket info is out of date when we need
to write, we should have a way to update it.

This template function allows us to wrap relevant sections of code so
they'll be retried against new bucket info on -ECANCELED.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add try_refresh_bucket_info function

Sometimes operations fail with -ECANCELED. This means we got raced. If
this happens we should update our bucket info from cache and try again.

Some user reports suggest that our cache may be getting and staying
out of sync. This is a bug and should be fixed, but it would also be
nice if we were robust enough to notice the problem and refresh.

So in that case, we invalidate the cache and fetch direct from the
OSD, putting a warning in the log.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>