sysctl.d: set kernel.pid_max=4194304 on 64-bit systems

For CONFIG_BASE_FULL Linux kernels, the maximum number of proc/thread
IDs is set to 32768 by default. This default limit can be quite easily
hit during recovery on nodes with high OSD counts.
To avoid hitting the pid_max default limit, attempt to configure it to
4194304, which corresponds to the maximum limit possible on 64-bit
CONFIG_BASE_FULL kernels.

Fixes: http://tracker.ceph.com/issues/21929
Signed-off-by: David Disseldorp <ddiss@suse.de>

sysctl: relocate 90-ceph-osd.conf under etc/sysctl/

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #18950 from sknorr/fix-rn-typos

doc: Fix typos in release notes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17977 from cbodley/wip-rgw-beast-qa

qa/rgw: add beast frontend to some rgw suites

Merge PR #18600 into master

* refs/pull/18600/head:
MDSMonitor: only clog changes to active

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18774 into master

* refs/pull/18774/head:
mds: fix CDir::log_mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18887 from dillaman/wip-20789

librbd: compare and write against a clone can result in failure

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18924 from ceph/wip-cv-filestore-devices

ceph-volume: allow using a device or partition for `lvm --data`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18700 from tianshan/fix_cls_bucket_head

rgw: fix cls_bucket_head result order consistency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18926 from theanalyst/wip-mp-jewel-fix

rgw: set sync_from_all as true when no value is seen

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.functional create a common playbook directory for setups

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm prepare can accept devices for --data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add data device and journal to xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional tox.ini addition of setup playbook for partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw device to centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common update --data flag to reflect device acceptance

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18969 from vshankar/rbd-mirror-image-map-state-transition

rbd-mirror: use next transition state to check transition completeness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18960 from Liuchang0812/mailmap-update

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's…

test/librbd: disable compare and write if skipping partial discard

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor object write request state machines

Fixes: http://tracker.ceph.com/issues/20789
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consolidate all object discard-related logic into single state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify interface between object IO and copyup state machines

The initial copyup was not receiving a write hint and the
code for hints was duplicated multiple times. Additionally,
the object map state should match the last executed write op.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copyup state machine needs to handle empty write ops

The compare-and-write object operation cannot be executed
concurrently within a copyup operation since the object might not
exist yet (if not performing a deep-copy).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: object cacher should re-use read state machine

This adds support for sparse-reads and ensures all object reads
utilize a single, tested code path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor io::ObjectReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: default template parameter for C_SparseReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce lock scope when assembling read results

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ObjectReadRequest does not require buffer extents

The ReadResult wrapper requires the buffer extents to know
how to properly reconstruct the out buffer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: templatize IO object request state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: mocked remainder of IO ops used by librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: simulate set_alloc_hint creating a non-existent file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Fix typos in release notes

Signed-off-by: Stefan Knorr <sknorr@suse.de>

rbd-mirror: use next transition state to check transition completeness

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #18952 from dillaman/wip-test-librbd

test/librbd: clean up for several mock function tests

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18947 from dillaman/wip-22131

rbd-nbd: certain kernels may not discover resized block devices

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18909 from dillaman/wip-22120

librbd: possible deadlock with synchronous maintenance operations

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18923 from dillaman/wip-21966

librbd: prevent overflow of discard API result code

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18918 from ZVampirEM77/wip-em-rewrite-cmd

rgw: add rewrite cmd and options into radosgw-admin usage and doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's mailmap and affiliation

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: add rewrite related cmd and parameter into radosgw-admin help info

+ radosgw-admin object rewrite
+ radosgw-admin bucket rewrite
+ --min-rewrite-size
+ --max-rewrite-size
+ --min-rewrite-stripe-size

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #18808 from tchaikov/wip-qa-max-pg-per-osd

qa: do not wait for down/out osd for pg convergence

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15788 from vshankar/mirror-ha-poolwatcher-policy

rbd-mirror: track images in policy map in support of A/A

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18953 from cbodley/wip-22139

common: fix BoundedKeyCounter const_pointer_iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

common: fix BoundedKeyCounter const_pointer_iterator

with libc++, clang fails to compile a call to:

  vector::assign(const_pointer_iterator, const_pointer_iterator)

because const_pointer_iterator does not satisfy the InputIterator
concept. added the necessary typedefs for value_type and reference
to reflect the pointer type

Fixes: http://tracker.ceph.com/issues/22139
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/librbd: fixed gmock uninteresting function call warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: eliminate sleeps within deep-copy mock tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18883 from cbodley/wip-22083+

rgw: set num_shards on 'radosgw-admin data sync init'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: set num_shards on 'radosgw-admin data sync init'

Fixes: http://tracker.ceph.com/issues/22083
Signed-off-by: Casey Bodley <cbodley@redhat.com>

dencoder/rgw: expose rgw sync status types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: added missing locks for snap rollback and rename

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18940 from trociny/wip-valgrind-DeadLockOwner

test/librbd: valgrind warning in TestMockManagedLockBreakRequest.DeadLockOwner

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17761 from cbodley/wip-rgw-bilog-trim

rgw multisite: automated trimming for bucket index logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #18811 from oritwas/wip-rgw-22046

tenant fixes for dynamic resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18920 from yaozongyou/wip-ceph-dencoder-add-types

ceph-dencoder: add support for cls_rgw_lc_obj_head

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18946 from joscollin/wip-add-cbodley

.githubmap: Add cbodley

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18900 from qrGitHub/master

rgw.iam: change '1' to '1ULL' in function print_actions

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

librbd: invalidating the cache shouldn't hold write lock

This can cause deadlock when readahead is in-progress since neither
can make forward progress.

Fixes: http://tracker.ceph.com/issues/22131
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: rescan partition table after image resize event

Fixes: http://tracker.ceph.com/issues/22131
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

.githubmap: Add cbodley

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18921 from Leeshine/wip-data-sync-cmd

rgw: add metadata and data sync related cmd into radosgw-admin usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: add data sync related cmd into radosgw-admin help info

+ radosgw-admin data sync status
+ radosgw-admin data sync init
+ radosgw-admin data sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add metadata sync related cmd into radosgw-admin help info

+ radosgw-admin metadata sync status
+ radosgw-admin metadata sync init
+ radosgw-admin metadata sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

librbd: prevent overflow of discard API result code

Prevent discard/writesame lengths larger than 2GB.

Fixes: http://tracker.ceph.com/issues/21966
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: valgrind warning in TestMockManagedLockBreakRequest.DeadLockOwner

When running with 'debug_rbd >= 20':

  Conditional jump or move depends on uninitialised value(s)
    by 0x94277B: operator<< (ostream:167)
    by 0x94277B: librbd::managed_lock::BreakRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_watchers(int) (BreakRequest.cc:101)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

rgw: fix cls_bucket_head result order consistency

headers's key is string but bucket_instance_ids's key is int
looked around and found no one use header's key, so change
header from map to vector

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #18905 from ifed01/wip-bluestore-recalc-alloc

os/bluestore: remove ineffective BlueFS fnode extent calculation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18903 from jan--f/jan-mgr-prometheus-unkown-pgstate

pybind/mgr/prometheus: don't crash when encountering an unknown PG state

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18885 from dzafman/wip-22112

ceph-objectstore-tool: Fix set-size to clear data_digest if changing …

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18622 from majianpeng/bluestore-opseq

os/bluestore: OpSequencer: reduce kv_submitted_waiters if _is_all_kv_submitted() return true.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #17032 from yangdongsheng/rbd_qos

rbd: implement image qos in tokenbucket algorithm

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18483 from Songweibin/wip-list-children2

 rbd: fix rbd children listing when child is in trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18007 from yehudasa/wip-rgw-config-desc

options.cc: document rgw config options

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: set sync_from_all as true when no value is seen

In order to support jewel-luminous multisite scenarios where when a
jewel cluster is the master, the sync-from-all json field wouldn't be
set leading to the secondary not syncing data as this value can't be
overriden to true, so defaulting the value of sync_from_all to true when
we don't encounter it in the json

Fixes: http://tracker.ceph.com/issues/22062
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18894 from liewegas/wip-fix-balancer

mgr/balancer: skip CRUSH_ITEM_NONE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18763 from cfanz/master

rgw: don't write bucket_header when it is not changed in rgw_bucket_prepare_op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18852 from theanalyst/wip-22083

rgw: data sync: set num_shards when building full maps

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18866 from cbodley/wip-22101

rgw: RGWEnv::set() takes std::string

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18875 from yaozongyou/rgw-cleanup

rgw: remove unused disable_signal_fd

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18895 from cooboos/wip-opt-maperase

rgw: optimize the rgw-attr del code logic

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18845 from yaozongyou/wip-lifecycle-log-level

rgw: update life cycle related log level

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: possible deadlock with synchronous maintenance operations

Fixes: http://tracker.ceph.com/issues/22120
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dencoder: adding support for cls_rgw_lc_obj_head

After adding cls_rgw_lc_obj_head, we can using the following command
to look into lc processing related information:
rados -p default.rgw.log getomapheader lc.0 ./header.bin --namespace=lc
ceph-dencoder type cls_rgw_lc_obj_head import ./header.bin decode dump_json

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18873 from tchaikov/wip-bluestore-aio-log

os/bluestore: print aio in batch

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18888 from tchaikov/wip-osx

common,test,cmake: various changes to re-enable build on osx

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16369 from bspark8/wip_dmc2_client_tracker

dmclock: Delivery of the dmclock delta, rho and phase parameter + Enabling the client service tracker

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18798 from cbodley/wip-str-list-view

 common: add for_each_substr() for cheap string split

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: add new API methods and expand the rbd CLI to list trashed images

implement librbd::RBD::list_children2 and rbd_list_children2
methods and expand the rbd CLI to list trashed images

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: limit IO per second by TokenBucketThrottle

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18663 from dragonylffly/wip-rbd-nbd-fix

rbd-nbd: bug fix when running in container

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.prepare add example of raw device or partition in help menu

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18713 from liewegas/wip-snapset-snaps

osd/PrimaryLogPG: do not pull-up snapc to snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.prepare allow partition or raw device as input for --data in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18882 from ceph/wip-cv-simple-testing

ceph-volume add functional tests for simple, rearrange lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>