doc/rados: Add explanation of straw2

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #18974 from liewegas/wip-22145

osd/PG: restart recovery if NotRecovering and unfound found

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: restart recovery if NotRecovering and unfound found

If we are in recovery_unfound state waiting for unfound objects, and we
find them, we need to restart the recovery reservation process so that we
can recover.  Do this by queueing DoRecover() event instead of calling
queue_recovery() (which won't do anything since we're not in
recoverying|backfilling pg states).

Make the parent Active state ignore DoRecovery so that if we are already
in some phase of recovery/backfill the event gets ignored.  It is already
handled by the other important substates that care, like Clean (for
repair's benefit).

I'm not sure why states like Activating are paying attention tot his vevent...

Fixes: http://tracker.ceph.com/issues/22145
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18954 from adamemerson/wip-hole-in-the-bucket-dear-liza

rgw: Add try_refresh_bucket_info function
rgw: Add retry_raced_bucket_write
rgw: Handle stale bucket info in RGWPutMetadataBucket
rgw: Handle stale bucket info in RGWSetBucketVersioning
rgw: Handle stale bucket info in RGWSetBucketWebsite
rgw: Handle stale bucket info in RGWDeleteBucketWebsite
rgw: Handle stale bucket info in RGWPutBucketPolicy
rgw: Handle stale bucket info in RGWDeleteBucketPolicy
rgw: Expire entries in bucket info cache

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19215 from joscollin/wip-mount-fuse-ceph-typo

doc: Fix typo in mount.fuse.ceph

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19209 from joscollin/wip-githubman-ktdreyer

githubmap: Add ktdreyer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19208 from tchaikov/wip-spdk-17.10

 spdk: update SPDK to v17.10

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19238 from tchaikov/wip-22266

tools/ceph_monstore_tool: rebuild initial mgrmap also

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #19079 from adamemerson/wip-static-ptr

Static Pointer

doc: Fix typo in mount.fuse.ceph

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #19212 into master

* refs/pull/19212/head:
doc: Add section about MDS cache limits

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19213 from ceph/wip-bz1518264

ceph-volume: correctly fallback to bluestore when no objectstore is specified

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18267 from joscollin/wip-update-radosgw-admin-doc

doc: Update commands and options in radosgw-admin
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

tools/ceph_monstore_tool: rebuild initial mgrmap also

Fixes: http://tracker.ceph.com/issues/22266
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18649 from b-ranto/wip-restful-fixes

mgr/restful: A couple of restful fixes

Reviewed-by: John Spray <john.spray@redhat.com>

doc: Add section about MDS cache limits

Signed-off-by: Bara Ancincova (bara@redhat.com)

doc: Update all OPTIONS sections in radosgw-admin

Updated all OPTIONS sections in RADOSGW-ADMIN doc page, using the latest options from rgw_admin.cc::usage()

Signed-off-by: Jos Collin <jcollin@redhat.com>

cmake,rpm,deb: update to accommodate SPDK v17.10

* cmake/modules/BuildSPDK.cmake: add lvol
* cmake/modules/BuildDPDK.cmake: add pci and bus_pci
* ceph.spec.in, cmake/modules/BuildSPDK.cmake, debian/control:
  re-introduce libuuid dependency, as 17.07 added lvol. and the latter
  depends on uuid.
* cmake/modules/BuildSPDK.cmake: avoid introducing local variable of
  `iface_libs`.
* cmake/modules/patch-dpdk-conf.sh: disable
  CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES, this option introduces the
  balanced allocation of memory. but it also requires libnuma-dev.
  let's disable it for now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9571 from yehudasa/wip-rgw-redirect-zone

rgw: add a default redirect field for zones

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19159 from shinobu-x/drop_using_namespace_std

common: drop namespace using directives for std

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19165 from xiaoxichen/fix_used_pct

mon/PGMap: Fix %USED calculation bug.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19156 from cyx1231st/fix_blkin_tracing

blkin: Fix unconditional tracing in OSD

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19149 from rzarzynski/wip-common-perfguards

common:  RAII-styled mechanism for updating PerfCounters

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19112 from xiaoxichen/osd_mapped_pool

OSD/admin_socket: add get_mapped_pools command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18959 from cooboos/wip-cephdencoder-LCtype

ceph-dencoder: add support for RGWLifecycleConfiguration

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19174 from smithfarm/wip-cmake-exclude-gitignore

cmake: mgr: exclude .gitignore

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18938 from tchaikov/wip-cmake

cmake: check gcc version not release date for libstdc++ saneness

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #19206 into master

* refs/pull/19206/head:
doc: add pre-requisite to enable CephFS snapshot

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

common: Add static_ptr

Add static_ptr, a pointer-like class that contains its own storage,
avoiding use of the heap.

The full range of *_pointer_cast functions are included.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19199 from tchaikov/wip-22136

 qa/ceph-disk: enlarge the simulated SCSI disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate correct detection and fallback of objectstore flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.lvm verify objectstore flags in activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19163 from vshankar/rbd-image-map-mem-leak

rbd-mirror: ImageMap memory leak fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

.githubmap: Add ktdreyer

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19172 from joscollin/wip-mount-fuse-ceph-doc

doc: doc for mount.fuse.ceph

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: doc for mount.fuse.ceph

Created doc for mount.fuse.ceph.

Fixes: http://tracker.ceph.com/issues/21539
Signed-off-by: Jos Collin <jcollin@redhat.com>

spdk: update SPDK to v17.10

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_multi: add a test for zone redirect

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add option to configure redirect_zone

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: add pre-requisite to enable CephFS snapshot
Signed-off-by: Jian Su <sujian@umcloud.com>

qa/ceph-disk: enlarge the simulated SCSI disk

100MB will be allocated for journal, and the remaining 100MB is for data
device. taking the inode into consideration, there will be approximately
87988 kB available for the activated OSD. and it will complain with a
"nearfull" state.

Fixes: http://tracker.ceph.com/issues/22136
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: should throw if fail to read key-management-mode

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: fix the logging messages

logging.Logger.warn(format, *args). so no need to prepar a string
for it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: path_set_context() after rename()

it does not matter if we chown/restorecon before or after the rename,
but the logging message looks better this way: instead of fixing the
.tmp files, we are updating the attributes of the dest files w/o
.${pid}.tmp extension.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-disk: do not redirect stderr to stdout

normally, if we care about the output of ceph-disk, we expect a json
string, and ceph-disk sends the output to stdout, and errors/warnings
to stderr. so everything works as expected. and the test should also
follow this tradition. for example, if deprecated warnings are printed,
the warning message should not be collected along with the json string.

see also: d44334f3

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: Fix %USED calculation bug.

Previous code forgot to multiple raw_used_ratio to calculate
used byte.

Fixes: http://tracker.ceph.com/issues/22247
Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

rgw: Expire entries in bucket info cache

To bound the degree to which an RGW instance can go out to lunch if
the watch/notify breaks down, force refresh of any cache entry over a
certain age.

Fifteen minutes by default, and expiration can be turned off entirely.

This is separate from the LRU. The LRU removes entries based on the
last time of access. This expiration patch forces refresh based on the
last time they were updated.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWDeleteBucketPolicy

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWPutBucketPolicy

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWDeleteBucketWebsite

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWSetBucketWebsite

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWSetBucketVersioning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Handle stale bucket info in RGWPutMetadataBucket

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add retry_raced_bucket_write

If the OSD informs us that our bucket info is out of date when we need
to write, we should have a way to update it.

This template function allows us to wrap relevant sections of code so
they'll be retried against new bucket info on -ECANCELED.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add try_refresh_bucket_info function

Sometimes operations fail with -ECANCELED. This means we got raced. If
this happens we should update our bucket info from cache and try again.

Some user reports suggest that our cache may be getting and staying
out of sync. This is a bug and should be fixed, but it would also be
nice if we were robust enough to notice the problem and refresh.

So in that case, we invalidate the cache and fetch direct from the
OSD, putting a warning in the log.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19182 from cbodley/wip-beast-v124

submodule: update Beast to ceph/ceph-master branch

submodule: update Beast to ceph/ceph-master branch

pulls beast submodule up from v116 tag to v124, with an additional bug
fix for async_read_some()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19175 from liewegas/wip-rest-test

qa/suites/rados/rest: move rest_test from qa/suites/rest/

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/rados/rest: move rest_test from qa/suites/rest/

...and add more health whitelists.

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: mgr: exclude .gitignore

Fixes RPMLINT warning:

ceph-mgr.x86_64: W: version-control-internal-file /usr/lib64/ceph/mgr/.gitignore
ceph-mgr.x86_64: W: version-control-internal-file /usr/lib64/ceph/mgr/dashboard/static/AdminLTE-2.3.7/.gitignore
You have included file(s) internally used by a version control system in the
package. Move these files out of the package and rebuild it.

Note: the backslash has to be doubled up for the regex to make
it through CMake.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19109 from mikulely/main-cleanup

rgw: drop useless includes in rgw_{main.cc, common.h}

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

common: introduce PerfGuard to PerfCounters.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #18935 from cfanz/cfanz-version-id-wip

rgw:send x-amz-version-id header when upload files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18981 from yaozongyou/fix-rgw-bi-decode-bug

cls/rgw: mtime in rgw_bucket_dir_entry_meta not really decoded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18978 from yaozongyou/wip-radosgw-admin-add-check

rgw-admin: check the data extra pool supports omap

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18962 from ZVampirEM77/wip-em-readlen

rgw: check read_op.read return value in RGWRados::copy_obj_data

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19130 from trociny/wip-mirror-enable-cleanup

librbd: update mirror::EnableRequest diagram according to code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/restful: Test pg_num/pgp_num modifications

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #19128 from joscollin/wip-mountfuseceph-typo

mount.fuse.ceph: Fix typo

Reviewed-by: Zheng Yan <zyan@redhat.com>

blkin: fix unconditional tracing

Blkin trace will be triggered unconditionally at OSD `issue_op`, even if
op->pg_trace is not initialized. This issue introduces unnecessary
overhead and confusing tracing records when blkin tracing is ON.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #19169 from wwba/fix-ceph-bluestore-tool-man-page

doc: drop duplicate line in ceph-bluestore-tool man page

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: remove duplicate line from ceph-authtool man page

Signed-off-by: Xiaojun Liao <xiaojunliao85@gmail.com>

ceph-dencoder: add support for RGWLifecycleConfiguration

Signed-off-by: Songbo Wang wangsongbo@cloudin.cn

rbd-mirror: ImageMap memory leak fixes

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cmake: silence CMP0054 warning

see https://gitlab.kitware.com/cmake/cmake/issues/17381 and
https://gitlab.kitware.com/cmake/cmake/commit/a8be8b1b54fe1922a1d1fc0365c3ae5c918b6654,
so before the updated cmake is released and packaged. we should
add this setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>

[cleanup] Remove namespace using directives for std

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #19154 from Fusl/master

doc: correcting typos in bluestore-config-ref and bluestore-migration

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: correcting typos in bluestore-config-ref and bluestore-migration

Signed-off-by: Katie Holly <git@meo.ws>

Merge pull request #18961 from tchaikov/wip-rgw-print-iam-policy

rgw: iam policy printing cleanups

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19124 from tchaikov/wip-make-dist-boost

make-dist,cmake: avoid re-downloading boost

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19081 from smithfarm/wip-ceph-volume-spec

build/ops: rpm: fix systemd macros for ceph-volume@.service

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #19097 from tchaikov/wip-osd-prefix-inc

osd: use prefix increment for non trivial iterator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #19094 from tchaikov/wip-coverity-cid-1395794

msg/simple: pass a char for reading from shutdown_rd_fd

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18997 from tchaikov/wip-rpm-cunit-for-suse

rpm: build-depends on "cunit-devel" for suse

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

librbd: update mirror::EnableRequest diagram according to code

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

mount.fuse.ceph: Fix typo

Fixed typos in the help of mount.fuse.ceph.

Signed-off-by: Jos Collin <jcollin@redhat.com>

make-dist: exclude unused bits in boost

the docs, examples and tests are not used. so drop them. we could go
further by removing unused components in boost. but that'd be an issue
if somebody added a component in CMakeLists but forgets to update this
script. also, we need to remove boost/$component and lib/$component to
achieve this goal. this also introduces extra complicity. so leave it
for another change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19127 from tchaikov/wip-mon-better-logging

mon/OSDMonitor: add space after __func__ in log msg

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: add space after __func__ in log msg

Signed-off-by: Kefu Chai <kchai@redhat.com>

OSD/admin_socket: add get_mapped_pools

get_mapped_pools will return all pool_id that
mapped to the paticular OSD.

It is super useful for monitoring system that want to
**group** OSDs based on application. PoolID is a nature
grouping tag to be attached to OSDs.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

cmake: check bootstrap.sh instead before downloading boost

config.hpp does not exist if boost is not configured. so a boost source
directory unpacked from dist tarball does not have this file. the
bootstrap.sh instead, it is used to create the necessary config files
for jam.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: repackage boost in the correct path

before this change, boost is put into ./src. after this change, it is
put into $outfile/src. i.e. something like
ceph-12.1.2-4592-gf5f2ced624/src .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19026 from badone/wip-selinux-policy-doc-rawhide-build

build/ops: rpm: selinux-policy fixes

Reviewed - by: branto@redhat.com

Merge pull request #18614 from tchaikov/wip-osd-mgrreport

osd,mgr: report slow requests and pending creating pgs to mgr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18890 from pjjw/pjjw/prom-deep-scrub-state-missing

mgr/prometheus: add missing 'deep' state to PG_STATES in ceph-mgr prometheus plugin

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19030 from tangwenjun3/wip-fix-bluefs-allocate

os/bluestore: fix the allocate in bluefs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18785 from majianpeng/bluestore-remove-collection

os/bluestore: avoid omit cache for remove-collection.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18760 from david-z/wip-mon-ceph-report

mon: dump servicemap along with MgrStatMonitor dump info

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17596 from amitkumar50/cov-message-10

messages: Silence uninitialized member warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa: silence SLOW_OPS,PENDING_CREATING_PGS warnings

this is an intermediate step to deprecate REQUEST_SLOW warnings.

Signed-off-by: Kefu Chai <kchai@redhat.com>