luminous: filestore: add pgid in filestore pg dir split log message

Fixes: http://tracker.ceph.com/issues/25223
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
(cherry picked from commit d62ea5cc05857bc1a1f9cf4a0e3d4df315bef8af)

Merge pull request #23418 from dzafman/wip-24845

luminous: tools/ceph-objectstore-tool: split filestore directories offline to target hash level

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23130 from ivancich/wip-fix-dir-suggest-lumious

luminous: rgw: fail to recover index from crash luminous backport

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #22932 from smithfarm/wip-24814-luminous

luminous: rgw: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #22930 from smithfarm/wip-24632-luminous

luminous: rgw performance regression for luminous 12.2.4

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22934 from smithfarm/wip-24810-luminous

luminous: Invalid Access-Control-Request-Request may bypass validate_cors_rule_method

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22984 from pdvian/wip-24808-luminous

luminous: rgw: fix gc may cause a large number of read traffic

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23000 from pdvian/wip-24830-luminous

luminous: rgw: "radosgw-admin objects expire" always returns ok even if the pro…

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23019 from pdvian/wip-24833-luminous

luminous: rgw: radogw-admin reshard status command should print text for reshar…

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23144 from cbodley/wip-25032

luminous: rgw: add curl_low_speed_limit and curl_low_speed_time config to avoid the thread hangs in data sync.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #23207 from linuxbox2/luminous-rgw-22790

luminous: rgw: do not ignore EEXIST in RGWPutObj::execute

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22729 from pdvian/wip-24495-luminous

luminous : osd/Session: fix invalid iterator dereference in Sessoin::have_backoff()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22918 from smithfarm/wip-24804-luminous

luminous: Python bindings use iteritems method which is not Python 3 compatible

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23054 from liewegas/wip-osdmap-warning-luminous

luminous: messages,mon,osd: silence gcc-8 warnings related to memset()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23113 from smithfarm/wip-24824-luminous

luminous: tests: test_ceph_argparse.py broken on py3-only system

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23155 from smithfarm/wip-25033-luminous

luminous: tests: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) in powercycle

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23202 from smithfarm/wip-25074-luminous

luminous: Boost system library is no longer required to compile and link example librados program

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23313 from cernceph/dvanders_chown

luminous: rpm: silence osd block chown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22546 from smithfarm/wip-24501-luminous

luminous: osd: eternal stuck PG in 'unfound_recovery'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22708 from pdvian/wip-24474-luminous

luminous: librados: fix buffer overflow for aio_exec python binding

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22723 from liewegas/wip-hobject-hash-luminous

luminous: common/hobject: improve hobject hash

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #22773 from smithfarm/wip-24714-luminous

luminous: mon: Add option to view IP addresses of clients in output of 'ceph features'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23014 from batrick/i24190

luminous: client: increase verbosity level for log messages in helper methods

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23169 from ukernel/luminous-24856

luminous: mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22885 from smithfarm/wip-24696-luminous

luminous: mds: low wrlock efficiency due to dirfrags traversal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22895 from smithfarm/wip-22386-luminous

luminous: ceph-fuse mount failed because no mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23013 from batrick/i24136

luminous: MDSMonitor: uncommitted state exposed to clients/mdss

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23015 from batrick/i23790

luminous: mds: unset deleted vars in shutdown_pass

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23212 from batrick/i25036

luminous: mds: print mdsmap processed at low debug level

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23213 from batrick/i25039

luminous: mds: dump recent events on respawn

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23214 from batrick/i25041

luminous: mds: increase debug level for dropped client cap msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22680 from VictorDenisov/backport_23772

luminous: let "ceph status" use base 10 when printing numbers not sizes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #22686 from pdvian/wip-24471-luminous

luminous: spdk: fix ceph-osd crash when activate SPDK

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22774 from theanalyst/wip-luminous-24717

luminous: qa/workunits/suites/blogbench.sh: use correct dir name

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #22848 from smithfarm/wip-24739-luminous

luminous: Bring back diff -y for non-FreeBSD

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #22860 from smithfarm/wip-24770-luminous

luminous: bluestore: set correctly shard for existed Collection

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22862 from smithfarm/wip-24772-luminous

luminous: osd: may get empty info at recovery

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #22864 from smithfarm/wip-24774-luminous

luminous: Mimic build fails with -DWITH_RADOSGW=0

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #22904 from ifed01/wip-ifed-backport-24260

luminous: os/bluestore: fix flush_commit locking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22909 from ifed01/wip-ifed-fix-23540-luminous

luminous: os/bluestore: fix incomplete faulty range marking when doing compression

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23025 from smithfarm/wip-24864-luminous

luminous: common: Abort in OSDMap::decode() during qa/standalone/erasure-code/test-erasure-eio.sh

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23112 from smithfarm/wip-24979-luminous

luminous: tests: ceph-helpers.sh tries to use dirname without mandatory parameter

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23374 from ceph/backport-luminous-22764

luminous: ceph-volume: refuse to zap mapper devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23395 from alfredodeza/backport-luminous-rm24020

luminous: ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mds: update MDSRank::cluster_degraded before handling mds failure

Migrator checks MDSRank::is_cluster_degraded() to decide if it needs to
send message to other mds. MDSRank::cluster_degraded should get updated
before calling Migrator::handle_mds_failure_or_stop().

Introduced by commit 7de9da4a "mds: handle discontinuous mdsmap"

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 93458c7306c3822175ed3c048f0937af8f02c341)

Merge pull request #23394 from alfredodeza/backport-luminous-rm24152

luminous: ceph-volume enable the ceph-osd during lvm activation

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23397 from alfredodeza/backport-luminous-rm25216

luminous: ceph-volume lvm.activate conditional mon-config on prime-osd-dir

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

client: increase verbosity level for log messages in helper methods

Fixes: http://tracker.ceph.com/issues/21014
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit e276fdaaa8b737ea845e19388796796ca30d2162)

Conflicts:
src/client/Client.cc

ceph-volume tests patch __release__ to mimic always for stdin keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 99c17ecbaf443824ea8e155e06c11b20a8603d76)

ceph-volume lvm.activate conditional mon-config on prime-osd-dir

The --no-mon-config should only be done in Mimic releaes and later

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 020d6b1e5555e4547e8a0a19bd29eea89e1b7a31)

ceph-volume api.lvm fallback to 1 part if none are define when creating lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 552d987233545f6f33b9220ea4ea98753f96962f)

ceph-volume tests verify create_lvs behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8c487124d795778dbbf461ffb95841b1c8025e08)

ceph-volume api.lvm default to null tags on lvs created

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 490e73eff1b5b7cc98db5d3df292759e1773c0ab)

ceph-deploy api.lvm allow multiple lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e073c56d80d8f37e0f96093f8dd4a95aef81ea02)

ceph-volume api.lvm add a sizing method for VGs to calculate LV creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61b9e5c771e167990caad3d85c0bfd27682d643)

ceph-volume exceptions create an error for size allocations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b035bb0f54d4a7777ce242798bdeb18ad0dc28e0)

ceph-volume tests verify vg sizing behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 08832f21e52f28d65a16aadae54a10c8fd16259f)

ceph-volume api.lvm force vgs to display units in 'g'

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit da9f90fcb40e1e82c322f6020253b63da69238da)

ceph-volume: optional systemd enable --runtime

Allow units to be enabled but not persisted across a reboot,
and use this when enabling osds.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
(cherry picked from commit 3e6f387be13b209aa9f56f5069331e449615d316)

ceph-volume: enable ceph-osd during lvm activation

Enable the ceph-osd@<id> unit during lvm activate to link these
units to the ceph-osd.target.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Fixes: http://tracker.ceph.com/issues/24152
(cherry picked from commit 261d8ac94d0008f57ac194718e9277c126e096b5)

Merge pull request #23367 from alfredodeza/luminous-rm25173

luminous ceph-volume do not use stdin in luminous

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

lvm zap: refuse to zap mapper devices

Fixes: http://tracker.ceph.com/issues/24504
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8e602d52657d50888a83963148334deb1f8d4e1c)

adds the util.disk.is_mapper_device method

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 852a466fdfa29c91f266ec122dcd5b01b8d1fca5)

Merge pull request #23369 from alfredodeza/luminous-cv-auto

luminous: ceph-volume initial take on auto sub-command

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23236 from dzafman/wip-25085-luminous

luminous: Fix  25085 and 24949

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22806 from smithfarm/wip-24697-luminous

luminous: ceph osd safe-to-destroy crashes the mgr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22753 from pdvian/wip-24498-luminous

luminous: librbd: force 'invalid object map' flag on-disk update

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-volume tests verify get_devices captures device metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8deb088f101894763d27a6c31a3e8a2e96fc8320)

ceph-volume tests.conftest allow custom dirs in tmpfile fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9d820e82ac6bc0f46890df1cdaa104e88cab2a7e)

ceph-volume tests ensure human_readable_size works up to terabytes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4ad42f4244062995de003d8141cddcdc1d2b80fc)

ceph-volume tests verify mapper and /dev devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9cfd37344c69ebf1f3d8440f295d79ae05b545e9)

ceph-volume tests add tests for get_block_devs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 672c4c4d1c31b4631b1238ed0efb98fef34f2a3d)

ceph-volume tests verify _map_dev_paths mappings

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2c4d46955e8d7f52b090cd8b5b5561a2dba443e5)

ceph-volume devices.auto initial take on auto sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e3b04c9677c273f91e0a9de35dcd68c578d124db)

ceph-volume util.disk create a device metadata utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c6559538db28d6b8bc8529e6b2a62e8619c69bd6)

ceph-volume util.disk add dev mapping helpers for path resolution

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c06e63fed8a11e80444e8e7ec2e82373d75aa501)

ceph-volume tests verify get_file_contents utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0177f71caba3fd64de285d6808b5f859c55ed2fe)

ceph-volume util.system capture file contents

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 39792573dabac3b85a41760c965b60746ca1f059)

ceph-volume devices.auto initial take on auto sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5649bd724b5d300c7080052433f9b9d1c6774cff)

ceph-volume main include commented out auto subcommand

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ad1b8cae055ee6c822370d91027613680ccd3dd3)

ceph-volume tests ensure is_lv detection on LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 47060a1ef4562b29771bdfb5a375a225f18c228c)

ceph-volume api.lvm create helpers to detect an lv from a device path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 68ca7a691e6e333209b67eb5cea8594f37fb93c1)

Merge pull request #22844 from smithfarm/wip-24735-luminous

luminous: order rbdmap.service before remote-fs-pre.target

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22865 from pdvian/wip-24584-luminous

luminous: osdc: Fix the wrong BufferHead offset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22767 from pdvian/wip-24514-luminous

luminous: rgw: fix the bug of radowgw-admin zonegroup set requires realm

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21806 from ivancich/wip-rgw-unordered-luminous

luminous: rgw: ability to list bucket contents in unsorted order for efficiency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22817 from smithfarm/wip-24690-luminous

luminous: rgw-multisite: endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22820 from smithfarm/wip-24692-luminous

luminous: rgw: index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22822 from smithfarm/wip-24693-luminous

luminous: rgw: meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22846 from smithfarm/wip-24737-luminous

luminous: add unit test for cls bi list command

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22942 from pdvian/wip-24782-luminous

luminous: rgw: set cr state if aio_read err return in RGWCloneMetaLogCoroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22985 from joscollin/wip-24844-luminous

luminous: rgw: require --yes-i-really-mean-it to run radosgw-admin orphans find

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23345 into luminous

* refs/pull/23345/head:
Revert "luminous: pybind/mgr/mgr_module: make rados handle available to all modules"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-volume tests verify that luminous doesn't use stdin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a97a63540290c93e05857ac175f3c9c45a4602c6)

ceph-volume util do not use stdin for luminous

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e03be24a4ff9b882d4329059fd1f04963d9ba2c5)

ceph-volume tests verify that mkfs filestore uses default flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fa9a8afe491445f622e95d32b1f822df8c5b3055)

ceph-volume lvm.prepare pass the cephx_secret for mkfs in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b25c8f6897bd66a7156c3a59724cd85eb835c5b6)

ceph-volume util.prepare use keyfile from stdin for mkfs in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 70b495ac06b1fa382654759764014d5391d67f9a)

tools/ceph-objectstore-tool: split filestore directories offline to target hash level

Currently ceph-objectstore-tool can only split dirs that already
meet the usual object number criteria. It won't reduce the cost of
online split for those dirs that would meet the criteria online.

This change uses 'arg1' option for apply-layout-settings to specify
the target hash level wants to split to. Then dirs will be split
offline no matters how many objects they have. This also helps
those pools that forgot to do pre-splitting at their creation time.

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit e71659b11a54ea3abdb047d2ec480bd855acee4a)

Merge pull request #23137 from smithfarm/wip-24068-luminous

luminous: tests: osd sends op_reply out of order

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>