qa: silence SLOW_OPS,PENDING_CREATING_PGS warnings

this is an intermediate step to deprecate REQUEST_SLOW warnings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: summarize osd metrics in MMgrReport and sent it to mon

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/health_check: s/std::list/list/

so it's self-contained, and .cc files including it are not obliged to `using namespace std`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: send health-checks to mgr

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: remember is_primary when adding pending creating pgs

so mgr can avoid calculating the same PG from different OSDs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/OpTracker: refactor check_ops_in_flight()

* add a function visit_ops_in_flight() so it accepts a std::function,
  which can be applied to all ops in flight. and is used to count
  slow ops. the # of slow ops will be sent to mgr.
* add a function with_slow_ops_in_flight(), so we can use it to
  implement command like "dump_slow_ops_in_flight". unlike the
  existing feature which dumps slow ops to clog periodically, it
  will dump the slow ops on demand. and backoff the warning
  interval only when the slow ops are dumped.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/MgrClient: send health_checks using MMgrReport

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18980 from shinobu-x/os_filestore_filestore_nit_cleanup

os/filestore/FileStore: Initialized by nullptr, NULL or 0 instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18982 from neha-ojha/wip-const-cleanup

osd/PG: const cleanup for recoverable/readable predicates

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18934 from tchaikov/wip-option-return-by-ref

Revert "common/config: return const reference instead of a copy"

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18916 from tchaikov/wip-22113

osd: subscribe osdmaps if any pending pgs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18816 from amitkumar50/cov-common-2

common: Check this->data.op_size before use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18810 from majianpeng/fix-21982

osdc/Objecter: record correctly value for l_osdc_op_send_bytes.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18777 from majianpeng/bluestore-collection-list

os/bluestore: optimize _collection_list.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18544 from ddiss/pid_max

sysctl.d: set kernel.pid_max=4194304

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18718 from shinobu-x/sk-ceph_osd

ceph_osd.cc: Drop legacy or redundant code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18776 from JianyuLi/joa-mul-mds

MDS: make popular counter decay at proper rate

Merge pull request #18784 from taodd/client-bugfix

Client:Fix readdir bug

Merge pull request #18786 from taodd/master

Client: setattr should drop "Fs" rather than "As" for mtime and size

Merge pull request #18836 from taodd/cephfs-hadoop

MDS: correct the error msg when init mon client

Merge pull request #18892 from taodd/client-bugfix2

MDS: add null check before we push_back "onfinish"

Merge pull request #18893 from taodd/mds-inotable-cleanup

mds: simplify the code logic in replay_alloc_ids

Merge pull request #18904 from taodd/mds-sesseionmap-cleanup

mds: get rid of the "if" check which is unnecessary inside a loop

Merge pull request #19003 from dillaman/wip-22158

librbd: set deleted parent pointer to null

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: set deleted parent pointer to null

Fixes: http://tracker.ceph.com/issues/22158
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18963 from amitkumar50/test-cov-test-9

test: Incorrect conversion to double

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18901 from jan--f/jan-crushtool-decompile-missing-ids

tools/crushtool: skip device id if no name exists

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18939 from majianpeng/bluestore-reap-collection

os/bluestore: fix the wrong usage for map_any

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18913 from yaozongyou/wip-rados-add-touch-command

tools/rados: add touch command to change object modification time

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18848 from jcsp/wip-22098

mgr/dashboard: fix audit log loading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18847 from jcsp/wip-mgr-mon-upness

mgr/dashboard: show warnings if data is out of date or mons are down

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #18484 from Liuchang0812/wip-mgr-request-osd-metadata

mgr: request daemon's metadata when receiving a report from an unknown server

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18976 from yahooguntu/patch-1

udev: Fix typo in udev OSD rules file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18944 from tchaikov/wip-dpdk-spdk

cmake: WITH_SPDK=ON by default

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

msg/async: rebase msg/async/dpdk to spdk/dpdk

Signed-off-by: Liu-Chunmei <chunmei.liu@intel.com>

cmake: use BuildDPDK if libdpdk not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add rte_mempool_{ring,stack} to Finddpdk.cmake

* also refactor it a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: BuildDPDK: do not build kernel modules

see also http://pkgs.fedoraproject.org/cgit/rpms/dpdk.git/tree/dpdk.spec

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated spdk linkage

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: refactor dpdk related code

* move the check of `USE_CRYPTOPP` to $top_srcdir/CMakeLists.txt
* remove reference of DPDK_LIBRARY, it's defined nowhere
* move the dpdk code to a single place

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18988 from ceph/wip-rm22154

ceph-disk: create deprecation warnings

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc/ceph-volume create a migration section to help users coming from ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk add deprecation warnings in favor of ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Fixes: http://tracker.ceph.com/issues/22154

Merge pull request #18917 from ZVampirEM77/wip-em-copyobjdata-cleanup

rgw: cleanup unused parameters in RGWRados::copy_obj_data

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

udev: Fix typo in udev OSD rules file

The rule for lockbox partitions had an invalid UUID for the "change" action.

Signed-off-by: Mitch Birti <yahooguntu@gmail.com>

Merge pull request #18922 from Synesis-LLC/fix-22122

rgw: fix radosgw-admin bucket rm with --purge-objects and --bypass-gc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18956 from Jeegn-Chen/wip-issue-22129

rgw: AWS v4 authorization work when INIT_MULTIPART is chunked

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #18984 from ceph/wip-cv-unittest-fix

ceph-volume tests.devices.lvm prepare isn't bluestore specific anymore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18968 from joscollin/wip-rgwadmin-rewrite

rgw: fix rewrite options usage text

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17797 from amitkumar50/cov-msg-3

msg: Initialize lkey,bound,port_cnt,num_chunk,gid_idx

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17874 from ShowingCloud/master

ceph-bluestore-tool: check if bdev is empty on 'bluefs-bdev-expand'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18090 from amitkumar50/test-cov-8

test: checking negative returns from creat()

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume tests.devices.lvm prepare isn't bluestore specific anymore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

sysctl.d: set kernel.pid_max=4194304 on 64-bit systems

For CONFIG_BASE_FULL Linux kernels, the maximum number of proc/thread
IDs is set to 32768 by default. This default limit can be quite easily
hit during recovery on nodes with high OSD counts.
To avoid hitting the pid_max default limit, attempt to configure it to
4194304, which corresponds to the maximum limit possible on 64-bit
CONFIG_BASE_FULL kernels.

Fixes: http://tracker.ceph.com/issues/21929
Signed-off-by: David Disseldorp <ddiss@suse.de>

sysctl: relocate 90-ceph-osd.conf under etc/sysctl/

Signed-off-by: David Disseldorp <ddiss@suse.de>

os/filestore: remove redudant check.

If next is null, we already set a non-null to it. So don't check again.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: using ghobject_t::is_max to judge.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/PG: const cleanup for recoverable/readable predicates

Signed-off-by: Neha Ojha <nojha@redhat.com>

os/bluestore: bypass check if collection has more objects than nonexistent_count.

If next != ghobject_t::get_max(), this mean collection has more data
than nonexistent_count. For this we should bypass check.
And fix wrong comments because _collection_list don't return object num.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: WITH_SPDK=ON by default

Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: add cunit1 as build dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build spdk/dpdk in cmake modules

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

os/filestore/FileStore: [Cleanup (trivial)] Initialized by nullptr, NULL or 0 instead

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

rgw: fix rewrite options usage text

Fixed rewrite options usage text properly.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18950 from sknorr/fix-rn-typos

doc: Fix typos in release notes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17977 from cbodley/wip-rgw-beast-qa

qa/rgw: add beast frontend to some rgw suites

Merge PR #18600 into master

* refs/pull/18600/head:
MDSMonitor: only clog changes to active

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18774 into master

* refs/pull/18774/head:
mds: fix CDir::log_mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18887 from dillaman/wip-20789

librbd: compare and write against a clone can result in failure

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18924 from ceph/wip-cv-filestore-devices

ceph-volume: allow using a device or partition for `lvm --data`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18700 from tianshan/fix_cls_bucket_head

rgw: fix cls_bucket_head result order consistency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18926 from theanalyst/wip-mp-jewel-fix

rgw: set sync_from_all as true when no value is seen

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.functional create a common playbook directory for setups

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm prepare can accept devices for --data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add data device and journal to xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional tox.ini addition of setup playbook for partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw device to centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common update --data flag to reflect device acceptance

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18969 from vshankar/rbd-mirror-image-map-state-transition

rbd-mirror: use next transition state to check transition completeness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18960 from Liuchang0812/mailmap-update

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's…

test/librbd: disable compare and write if skipping partial discard

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor object write request state machines

Fixes: http://tracker.ceph.com/issues/20789
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consolidate all object discard-related logic into single state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify interface between object IO and copyup state machines

The initial copyup was not receiving a write hint and the
code for hints was duplicated multiple times. Additionally,
the object map state should match the last executed write op.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copyup state machine needs to handle empty write ops

The compare-and-write object operation cannot be executed
concurrently within a copyup operation since the object might not
exist yet (if not performing a deep-copy).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: object cacher should re-use read state machine

This adds support for sparse-reads and ensures all object reads
utilize a single, tested code path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor io::ObjectReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: default template parameter for C_SparseReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce lock scope when assembling read results

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ObjectReadRequest does not require buffer extents

The ReadResult wrapper requires the buffer extents to know
how to properly reconstruct the out buffer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: templatize IO object request state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: mocked remainder of IO ops used by librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: simulate set_alloc_hint creating a non-existent file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>