common: Add static_ptr

Add static_ptr, a pointer-like class that contains its own storage,
avoiding use of the heap.

The full range of *_pointer_cast functions are included.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19163 from vshankar/rbd-image-map-mem-leak

rbd-mirror: ImageMap memory leak fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19172 from joscollin/wip-mount-fuse-ceph-doc

doc: doc for mount.fuse.ceph

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: doc for mount.fuse.ceph

Created doc for mount.fuse.ceph.

Fixes: http://tracker.ceph.com/issues/21539
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19182 from cbodley/wip-beast-v124

submodule: update Beast to ceph/ceph-master branch

submodule: update Beast to ceph/ceph-master branch

pulls beast submodule up from v116 tag to v124, with an additional bug
fix for async_read_some()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19175 from liewegas/wip-rest-test

qa/suites/rados/rest: move rest_test from qa/suites/rest/

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/rados/rest: move rest_test from qa/suites/rest/

...and add more health whitelists.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19109 from mikulely/main-cleanup

rgw: drop useless includes in rgw_{main.cc, common.h}

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18935 from cfanz/cfanz-version-id-wip

rgw:send x-amz-version-id header when upload files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18981 from yaozongyou/fix-rgw-bi-decode-bug

cls/rgw: mtime in rgw_bucket_dir_entry_meta not really decoded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18978 from yaozongyou/wip-radosgw-admin-add-check

rgw-admin: check the data extra pool supports omap

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18962 from ZVampirEM77/wip-em-readlen

rgw: check read_op.read return value in RGWRados::copy_obj_data

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19130 from trociny/wip-mirror-enable-cleanup

librbd: update mirror::EnableRequest diagram according to code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19128 from joscollin/wip-mountfuseceph-typo

mount.fuse.ceph: Fix typo

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #19169 from wwba/fix-ceph-bluestore-tool-man-page

doc: drop duplicate line in ceph-bluestore-tool man page

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: remove duplicate line from ceph-authtool man page

Signed-off-by: Xiaojun Liao <xiaojunliao85@gmail.com>

rbd-mirror: ImageMap memory leak fixes

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #19154 from Fusl/master

doc: correcting typos in bluestore-config-ref and bluestore-migration

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: correcting typos in bluestore-config-ref and bluestore-migration

Signed-off-by: Katie Holly <git@meo.ws>

Merge pull request #18961 from tchaikov/wip-rgw-print-iam-policy

rgw: iam policy printing cleanups

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19124 from tchaikov/wip-make-dist-boost

make-dist,cmake: avoid re-downloading boost

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19081 from smithfarm/wip-ceph-volume-spec

build/ops: rpm: fix systemd macros for ceph-volume@.service

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #19097 from tchaikov/wip-osd-prefix-inc

osd: use prefix increment for non trivial iterator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #19094 from tchaikov/wip-coverity-cid-1395794

msg/simple: pass a char for reading from shutdown_rd_fd

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18997 from tchaikov/wip-rpm-cunit-for-suse

rpm: build-depends on "cunit-devel" for suse

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

librbd: update mirror::EnableRequest diagram according to code

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

mount.fuse.ceph: Fix typo

Fixed typos in the help of mount.fuse.ceph.

Signed-off-by: Jos Collin <jcollin@redhat.com>

make-dist: exclude unused bits in boost

the docs, examples and tests are not used. so drop them. we could go
further by removing unused components in boost. but that'd be an issue
if somebody added a component in CMakeLists but forgets to update this
script. also, we need to remove boost/$component and lib/$component to
achieve this goal. this also introduces extra complicity. so leave it
for another change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19127 from tchaikov/wip-mon-better-logging

mon/OSDMonitor: add space after __func__ in log msg

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: add space after __func__ in log msg

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check bootstrap.sh instead before downloading boost

config.hpp does not exist if boost is not configured. so a boost source
directory unpacked from dist tarball does not have this file. the
bootstrap.sh instead, it is used to create the necessary config files
for jam.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: repackage boost in the correct path

before this change, boost is put into ./src. after this change, it is
put into $outfile/src. i.e. something like
ceph-12.1.2-4592-gf5f2ced624/src .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19026 from badone/wip-selinux-policy-doc-rawhide-build

build/ops: rpm: selinux-policy fixes

Reviewed - by: branto@redhat.com

Merge pull request #18614 from tchaikov/wip-osd-mgrreport

osd,mgr: report slow requests and pending creating pgs to mgr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18890 from pjjw/pjjw/prom-deep-scrub-state-missing

mgr/prometheus: add missing 'deep' state to PG_STATES in ceph-mgr prometheus plugin

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19030 from tangwenjun3/wip-fix-bluefs-allocate

os/bluestore: fix the allocate in bluefs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18785 from majianpeng/bluestore-remove-collection

os/bluestore: avoid omit cache for remove-collection.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18760 from david-z/wip-mon-ceph-report

mon: dump servicemap along with MgrStatMonitor dump info

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17596 from amitkumar50/cov-message-10

messages: Silence uninitialized member warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa: silence SLOW_OPS,PENDING_CREATING_PGS warnings

this is an intermediate step to deprecate REQUEST_SLOW warnings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: summarize osd metrics in MMgrReport and sent it to mon

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: drop useless includes in rgw_{main.cc, common.h}

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge PR #18697 into master

* refs/pull/18697/head:
mds: limit size of MExportDir message
mds: optimize MDCache::try_subtree_merge
mds: optimize import/export state access

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18748 into master

* refs/pull/18748/head:
client/Client.cc: trivial [Cleanup] Stop different types comparison

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18787 into master

* refs/pull/18787/head:
mds: prevent filelock from being stuck at XSYN state
mds: implement LOCK_XSYN_LOCK and LOCK_XSYN_MIX
mds: cleanup set/drop loner code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18818 into master

* refs/pull/18818/head:
libceph: Avoiding divide by zero

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18942 into master

* refs/pull/18942/head:
mds: lookup snapped inodes by vinodeno_t

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19007 into master

* refs/pull/19007/head:
src/libcephfs.cc: Cleanup nullptr initializations against deleted objects, etc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19014 into master

* refs/pull/19014/head:
mds: properly execute scrub finish context

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19018 into master

* refs/pull/19018/head:
mds: don't delay processing completed requests in replay queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19060 into master

* refs/pull/19060/head:
client: anchor Inode while trimming caps

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #19102 from jecluis/wip-qa-find-used-ports

qa: add simple and dirty script to find ports being used

Merge pull request #19044 from idryomov/wip-resurrect-msg-auth

msg: resurrect support for !CEPH_FEATURE_MSG_AUTH

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa: add simple and dirty script to find ports being used

Seriously, having this is way better than realizing that one needs to
find an unused port each time a new test is created.

It's far from a perfect script. It's downright not elegant.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #19082 from trociny/wip-qa-demote-fixup

qa/workunits/rbd: wait for status propagated only if daemon started

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: use prefix increment for non trivial iterator

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: limit size of MExportDir message

Only export port of subtree if whole subtree tree is too large for
sinlge MExportDir message. The un-exported portion are added to
export queue.

Fixes: http://tracker.ceph.com/issues/21892
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: optimize MDCache::try_subtree_merge

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #19067 from tchaikov/wip-nvme-mgr-fixes

bluestore/NVMEDevice: do not deference a dangling pointer

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

build/ops: rpm: selinux-policy fixes

Requiring the file /usr/share/selinux/devel/policyhelp breaks fc27 and
Rawhide builds as they do not have that file. This exposed the fact this
code had not worked in some time due to changes in selinux policy
modules packaging. See
https://bugzilla.redhat.com/show_bug.cgi?id=999584

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19035 from ifed01/wip-ifed-rep-backend-speedup

osd: make operations on ReplicatedBackend::in_progress_ops more effective

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17799 from amitkumar50/cov-misc-7

osd,messages: Initialize read_length,options,send_reply

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17792 from amitkumar50/cov-misc-5

messages,journal: Initialization of stats_period,m_active_set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17772 from amitkumar50/cov-misc-3

client,osd,test: Initialize fuse_req_key,snap,who,seq

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17767 from amitkumar50/cov-misc-2

messages,test,msg: initialize h,reply_type,owner

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17766 from amitkumar50/cov-misc-1

common,os: initialize commit_data,cmount,iocb

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17443 from JiYou/restapi-default-port

ceph-rest-api: when port=0 use the DEFAULT_PORT instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17085 from yaozongyou/remove_unused_using

librgw_file: remove unused `using` statement.

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/simple: pass a char for reading from shutdown_rd_fd

Fixes the coverity issue:

> CID 1395794 (#1 of 1): Wrong size argument (SIZEOF_MISMATCH)
> suspicious_sizeof: Passing argument &ch of type int * and
> argument 1UL to function read is suspicious because sizeof
> (int) /*4*/ is expected.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19017 from tchaikov/wip-bluestore-without-aio

bluestore,cmake: enable building bluestore without aio

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19029 from tchaikov/wip-cmake-sysctl-cleanup

cmake: only create sysctl file on linux

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19072 from smithfarm/wip-22207

tests: ceph-disk: ignore E722 in flake8 test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #17678 into master

* refs/pull/17678/head:
mon/AuthMonitor: improve error message
mon/OSDMonitor: disallow "all" as a key or value name
cephfs, mon/AuthMonitor, OSD/osdcap: make 'all' a synonym for '*'
vstart.sh: Create an admin user for each CephFS
mon/AuthMonitor: Allow * wildcard for filesystem name
OSD/OSDCap: Allow namespace and pool tag to be combined
OSD/OSDCap: Namespace globbing
mon/AuthMonitor: Use new osd auth caps for ceph fs authorize
OSD/auth caps: Add OSD auth caps based on pool tag
mon/FSCommands: Tag pools used for cephfs by default
mon/OSDMonitor: Add key/value arguments for pool tagging

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

build/ops: rpm: fix systemd macros for ceph-volume@.service

Fixes: http://tracker.ceph.com/issues/22217
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/rbd: wait for status propagated only if daemon started

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19000 from dillaman/wip-15322

rbd-mirror: removed dedicated thread from image deleter

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17072 from rzarzynski/wip-rgw-21015

rgw: incorporate the Transfer-Encoding fix for CivetWeb.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19073 from trociny/wip-qa-demote

qa/workunits/rbd: wait for demote status is propagated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19075 from tchaikov/wip-22154

qa/workunits: silence py warnings for ceph-disk tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rbd-mirror: fix possible race condition in error path tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19064 from qrGitHub/wip-rgw-typo-refity

rgw: correct typo refity to refit

Reviewed-by: Amit Kumar <amitkuma@redhat.com>

rbd-mirror: removed dedicated thread from image deleter

Fixes: http://tracker.ceph.com/issues/15322
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image deleter now takes a librados::IoCtx shared pointer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: move image deletion to standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: move snap purge to standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure deep-copy can remove an unprotected snapshot

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: added async version of get_snapcontext

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18790 from amitkumar50/cov-rbd-6

rbd: null check before pool_name use

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: null check before pool_name use

Fixes the coverity issue:

CID 1394846 (#1 of 1): Dereference after null check (FORWARD_NULL)
15. var_deref_model: Passing null pointer pool_name to regex_match,
which dereferences it.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

qa/workunits: silence py warnings for ceph-disk tests

ceph-disk now prints "depreacted" warning message when it starts. but
the tests parses its stdout and stderr for a json string. so we need to
silence the warnings for the tests.

Fixes: http://tracker.ceph.com/issues/22154
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix the allocate in bluefs

when bluefs succeed to reserve but failed to alloc in db space,

it will cause a assert, just because of the space fragmentation.

in this situation, it could not use slow device space,

and it would happen in stupid or avl allocator.

Signed-off-by: tangwenjun <tang.wenjun3@zte.com.cn>

qa/workunits/rbd: wait for demote status is propagated

(before promoting the image on another cluster)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19045 from dillaman/wip-22200

rbd: disk usage on empty pool no longer returns an error message

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

tests: ceph-disk: ignore E722 in flake8 test

Very old, and very new, versions of flake8 treat E722 as an error:

flake8 runtests: commands[0] | flake8 --ignore=H105,H405,E127 ceph_disk tests
ceph_disk/main.py:1575:9: E722 do not use bare except'
ceph_disk/main.py:1582:9: E722 do not use bare except'
ceph_disk/main.py:3252:5: E722 do not use bare except'
ceph_disk/main.py:3288:21: E722 do not use bare except'
ceph_disk/main.py:3296:17: E722 do not use bare except'
ceph_disk/main.py:4358:5: E722 do not use bare except'
tests/test_main.py:26:1: E722 do not use bare except'
ERROR: InvocationError: '/opt/j/ws/mkck/src/ceph-disk/.tox/flake8/bin/flake8 --ignore=H105,H405,E127 ceph_disk tests'

Fixes: http://tracker.ceph.com/issues/22207
References: https://gitlab.com/pycqa/flake8/issues/361

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: BuildDPDK: use ${CMAKE_MAKE_PROGRAM} when appropriate

DPDK requires GNU Make. see
spdk/dpdk/doc/guides/freebsd_gsg/build_dpdk.rst. so, we should use the
configured make program to do the job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/health_check: s/std::list/list/

so it's self-contained, and .cc files including it are not obliged to `using namespace std`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: send health-checks to mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: BuildDPDK: use POSIX compliant syntax

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: remember is_primary when adding pending creating pgs

so mgr can avoid calculating the same PG from different OSDs.

Signed-off-by: Kefu Chai <kchai@redhat.com>