mds: don't delay processing completed requests in replay queue

Completed requests can also be in replay queue, they need to be processed
(treated as lookup request) in clientreplay stage. MDS will be stuck at
clientreplay stage if it delays processing requests of this type.

the bug was introduced by commit 0afbc033
(mds: don't rdlock locks in replica object while auth mds is recovering)

Fixes: http://tracker.ceph.com/issues/22163
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #18776 from JianyuLi/joa-mul-mds

MDS: make popular counter decay at proper rate

Merge pull request #18784 from taodd/client-bugfix

Client:Fix readdir bug

Merge pull request #18786 from taodd/master

Client: setattr should drop "Fs" rather than "As" for mtime and size

Merge pull request #18836 from taodd/cephfs-hadoop

MDS: correct the error msg when init mon client

Merge pull request #18892 from taodd/client-bugfix2

MDS: add null check before we push_back "onfinish"

Merge pull request #18893 from taodd/mds-inotable-cleanup

mds: simplify the code logic in replay_alloc_ids

Merge pull request #18904 from taodd/mds-sesseionmap-cleanup

mds: get rid of the "if" check which is unnecessary inside a loop

Merge pull request #19003 from dillaman/wip-22158

librbd: set deleted parent pointer to null

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: set deleted parent pointer to null

Fixes: http://tracker.ceph.com/issues/22158
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18963 from amitkumar50/test-cov-test-9

test: Incorrect conversion to double

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18901 from jan--f/jan-crushtool-decompile-missing-ids

tools/crushtool: skip device id if no name exists

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18939 from majianpeng/bluestore-reap-collection

os/bluestore: fix the wrong usage for map_any

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18913 from yaozongyou/wip-rados-add-touch-command

tools/rados: add touch command to change object modification time

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18848 from jcsp/wip-22098

mgr/dashboard: fix audit log loading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18847 from jcsp/wip-mgr-mon-upness

mgr/dashboard: show warnings if data is out of date or mons are down

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #18484 from Liuchang0812/wip-mgr-request-osd-metadata

mgr: request daemon's metadata when receiving a report from an unknown server

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18976 from yahooguntu/patch-1

udev: Fix typo in udev OSD rules file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18944 from tchaikov/wip-dpdk-spdk

cmake: WITH_SPDK=ON by default

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

msg/async: rebase msg/async/dpdk to spdk/dpdk

Signed-off-by: Liu-Chunmei <chunmei.liu@intel.com>

cmake: use BuildDPDK if libdpdk not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add rte_mempool_{ring,stack} to Finddpdk.cmake

* also refactor it a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: BuildDPDK: do not build kernel modules

see also http://pkgs.fedoraproject.org/cgit/rpms/dpdk.git/tree/dpdk.spec

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated spdk linkage

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: refactor dpdk related code

* move the check of `USE_CRYPTOPP` to $top_srcdir/CMakeLists.txt
* remove reference of DPDK_LIBRARY, it's defined nowhere
* move the dpdk code to a single place

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18988 from ceph/wip-rm22154

ceph-disk: create deprecation warnings

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc/ceph-volume create a migration section to help users coming from ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-disk add deprecation warnings in favor of ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Fixes: http://tracker.ceph.com/issues/22154

Merge pull request #18917 from ZVampirEM77/wip-em-copyobjdata-cleanup

rgw: cleanup unused parameters in RGWRados::copy_obj_data

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

udev: Fix typo in udev OSD rules file

The rule for lockbox partitions had an invalid UUID for the "change" action.

Signed-off-by: Mitch Birti <yahooguntu@gmail.com>

Merge pull request #18922 from Synesis-LLC/fix-22122

rgw: fix radosgw-admin bucket rm with --purge-objects and --bypass-gc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18956 from Jeegn-Chen/wip-issue-22129

rgw: AWS v4 authorization work when INIT_MULTIPART is chunked

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #18984 from ceph/wip-cv-unittest-fix

ceph-volume tests.devices.lvm prepare isn't bluestore specific anymore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18968 from joscollin/wip-rgwadmin-rewrite

rgw: fix rewrite options usage text

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17797 from amitkumar50/cov-msg-3

msg: Initialize lkey,bound,port_cnt,num_chunk,gid_idx

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17874 from ShowingCloud/master

ceph-bluestore-tool: check if bdev is empty on 'bluefs-bdev-expand'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18090 from amitkumar50/test-cov-8

test: checking negative returns from creat()

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume tests.devices.lvm prepare isn't bluestore specific anymore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cmake: WITH_SPDK=ON by default

Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: add cunit1 as build dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build spdk/dpdk in cmake modules

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

rgw: fix rewrite options usage text

Fixed rewrite options usage text properly.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18950 from sknorr/fix-rn-typos

doc: Fix typos in release notes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17977 from cbodley/wip-rgw-beast-qa

qa/rgw: add beast frontend to some rgw suites

Merge PR #18600 into master

* refs/pull/18600/head:
MDSMonitor: only clog changes to active

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18774 into master

* refs/pull/18774/head:
mds: fix CDir::log_mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18887 from dillaman/wip-20789

librbd: compare and write against a clone can result in failure

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18924 from ceph/wip-cv-filestore-devices

ceph-volume: allow using a device or partition for `lvm --data`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18700 from tianshan/fix_cls_bucket_head

rgw: fix cls_bucket_head result order consistency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18926 from theanalyst/wip-mp-jewel-fix

rgw: set sync_from_all as true when no value is seen

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.functional create a common playbook directory for setups

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm prepare can accept devices for --data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add data device and journal to xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional tox.ini addition of setup playbook for partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw device to centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common update --data flag to reflect device acceptance

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18969 from vshankar/rbd-mirror-image-map-state-transition

rbd-mirror: use next transition state to check transition completeness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18960 from Liuchang0812/mailmap-update

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's…

test/librbd: disable compare and write if skipping partial discard

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor object write request state machines

Fixes: http://tracker.ceph.com/issues/20789
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consolidate all object discard-related logic into single state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify interface between object IO and copyup state machines

The initial copyup was not receiving a write hint and the
code for hints was duplicated multiple times. Additionally,
the object map state should match the last executed write op.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copyup state machine needs to handle empty write ops

The compare-and-write object operation cannot be executed
concurrently within a copyup operation since the object might not
exist yet (if not performing a deep-copy).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: object cacher should re-use read state machine

This adds support for sparse-reads and ensures all object reads
utilize a single, tested code path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor io::ObjectReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: default template parameter for C_SparseReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce lock scope when assembling read results

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ObjectReadRequest does not require buffer extents

The ReadResult wrapper requires the buffer extents to know
how to properly reconstruct the out buffer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: templatize IO object request state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: mocked remainder of IO ops used by librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: simulate set_alloc_hint creating a non-existent file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Fix typos in release notes

Signed-off-by: Stefan Knorr <sknorr@suse.de>

rbd-mirror: use next transition state to check transition completeness

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: Incorrect conversion to double

Fixes the coverity issue:

CID 1417062 (#1 of 1): Result is not floating-point
(UNINTENDED_INTEGER_DIVISION)
integer_division: Dividing integer expressions 100 and temp
, and then converting the integer quotient to type double.
Any remainder, or fractional part of the quotient, is ignored.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #18952 from dillaman/wip-test-librbd

test/librbd: clean up for several mock function tests

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18947 from dillaman/wip-22131

rbd-nbd: certain kernels may not discover resized block devices

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18909 from dillaman/wip-22120

librbd: possible deadlock with synchronous maintenance operations

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18923 from dillaman/wip-21966

librbd: prevent overflow of discard API result code

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rgw: fix radosgw-admin bucket rm with --purge-objects and --bypass-gc

Call RGWRados::delete_bucket() from rgw_remove_bucket_bypass_gc()
instead of partial copy of code of RGWRados::delete_bucket().

Fix updating user stats after radosgw-admin bucket rm --purge-objects and --bypass-gc
Due to rgw_user(const std::string& s) was called incorrect version of rgw_bucket_sync_user_stats().

Fixes: http://tracker.ceph.com/issues/22122
Fixes: http://tracker.ceph.com/issues/19959
Signed-off-by: Aleksei Gutikov <aleksey.gutikov@synesis.ru>

Merge pull request #18918 from ZVampirEM77/wip-em-rewrite-cmd

rgw: add rewrite cmd and options into radosgw-admin usage and doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's mailmap and affiliation

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: cleanup unused parameters in RGWRados::copy_obj_data

src_obj, max_chunk_size, category and ptag are out
of use in RGWRados::copy_obj_data.

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

mgr: request daemon's metadata when receiving a report message from an unknown server

Fixes: http://tracker.ceph.com/issues/21687
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: add rewrite related cmd and parameter into radosgw-admin help info

+ radosgw-admin object rewrite
+ radosgw-admin bucket rewrite
+ --min-rewrite-size
+ --max-rewrite-size
+ --min-rewrite-stripe-size

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #18808 from tchaikov/wip-qa-max-pg-per-osd

qa: do not wait for down/out osd for pg convergence

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15788 from vshankar/mirror-ha-poolwatcher-policy

rbd-mirror: track images in policy map in support of A/A

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: AWS v4 authorization work when INIT_MULTIPART is chunked

Add RGW_OP_INIT_MULTIPART as a the single chunk special case
like RGW_OP_COMPLETE_MULTIPART.

Fixes: http://tracker.ceph.com/issues/22129
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18953 from cbodley/wip-22139

common: fix BoundedKeyCounter const_pointer_iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

common: fix BoundedKeyCounter const_pointer_iterator

with libc++, clang fails to compile a call to:

  vector::assign(const_pointer_iterator, const_pointer_iterator)

because const_pointer_iterator does not satisfy the InputIterator
concept. added the necessary typedefs for value_type and reference
to reflect the pointer type

Fixes: http://tracker.ceph.com/issues/22139
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/librbd: fixed gmock uninteresting function call warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: eliminate sleeps within deep-copy mock tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18883 from cbodley/wip-22083+

rgw: set num_shards on 'radosgw-admin data sync init'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: set num_shards on 'radosgw-admin data sync init'

Fixes: http://tracker.ceph.com/issues/22083
Signed-off-by: Casey Bodley <cbodley@redhat.com>