common: assert if buffer advance length overflow

In the old ceph version, buffer advance length was defined as int, but
in async msg, the real length of data buffer was defined as unsigned.

Occassionly some MDS message back from OSD was too large, which caused
this length overflow and made MDS crash.

For compatibility reason, add an assertion here if buffer advance length
is overflow.

Fixes: http://tracker.ceph.com/issues/36340
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge PR #24437 into master

* refs/pull/24437/head:
PendingReleaseNotes: add note for mds drop cache

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #24444 from theanalyst/wip-reshard-stats

rgw: copy actual stats from the source shards during reshard

Merge pull request #23504 from pritha-srivastava/wip-sts-authentication

rgw: Code for STS Authentication.

rgw: copy actual stats from the source shards during reshard

Currently we don't copy the actual_stats field during reshard, which makes
resharded buckets show a size_utilized as 0, which further has the problem that
a subsequent object removal would subtract the object size from the 0 size
utilized showing up large uint64_t values. Copy the size_actual from the source
object in both cls and in reshard_process. This will fix the new buckets,
existing buckets will still have to go through a bucket check --fix for their
stats to be corrected.

Fixes: http://tracker.ceph.com/issues/36290
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #24433 from dillaman/wip-rbd-mirror-doc

doc/rbd: explicitly state that mirroring requires connectivity to clusters

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24144 from tchaikov/wip-spdk

bluestore/NVMe: use PCIe selector as the path name

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24168 from tchaikov/wip-17516

cmake: detect armv8 crc and crypto feature using CHECK_C_COMPILER_FLAG

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24322 from votdev/improve_error_interceptor

mgr/dashboard: Improve error message handling

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

PendingReleaseNotes: add note for mds drop cache

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rbd: explicitly state that mirroring requires connectivity to cluster

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24370 from tspmelo/fix-rbd-trash-purge

mgr/dashboard: Add permission validation to the  "Purge Trash" button

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24271 from croit/fix-36041

rgw/beast: drop privileges after binding ports

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Improve error message handling

The current implementation of the HTTP interceptor does not use the error description in all cases.

Fixes: https://tracker.ceph.com/issues/36190
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #24350 from ricardoasmarques/wip-json-decode

mgr/dashboard: Check content-type before decode json response

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #24213 from familyuu/mgr

mgr/dashboard: Display logged in user

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20645 from rishabh-d-dave/fix-doc-add-subcommand-application-manpages

ceph: describe application subcommand in ceph man page

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24117 from trociny/wip-35915

mgr: create shell OSD performance query class

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

mgr: module to test osd_perf_query methods

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: add osd_perf_query methods to ActivePyModules

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: create shell OSD performance query class

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24380 from smithfarm/wip-opensuse-15

qa/distros: add openSUSE Leap 42.3 and 15.0

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #24399 from alfredodeza/wip-rm36246

ceph-volume util.encryption don't push stderr to terminal

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24394 from alfredodeza/wip-rm24795

ceph-volume lvm.prepare update help to indicate partitions are needed, not devices

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rgw/beast: drop privileges after binding ports

Fixes: http://tracker.ceph.com/issues/36041
Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>

ceph: describe application subcommand in ceph man page

Add the summary and syntax for all the subcommands under the subcommand
"ceph osd pool application" to ceph the manual page.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-volume util.encryption don't push stderr to terminal when cryptsetup fails to find device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare update help to indicate partitions are needed, not devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24348 from trociny/wip-config-pool-mirror

 rbd-mirror: use pool level config overrides

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Delete white space beform 'Sign'

Signed-off-by: guodan <guodan1@lenovo.com>

mgr/dashboard: Delete the old logout component

Signed-off-by: guodan1 <guodan1@lenovo.com>

mgr/dashboard: Display logged in user

Signed-off-by: guodan1 <guodan1@lenovo.com>

Merge PR #24367 into master

* refs/pull/24367/head:
doc: Fix typo error on cephfs/fuse/

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24362 into master

* refs/pull/24362/head:
doc: fix broken fstab url in cephfs/fuse

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24276 from dillaman/wip-36183

osdc/Objecter: possible race condition with connection reset

Reviewed-by: Greg Farnum gfarnum@redhat.com

Merge pull request #24262 from cernceph/dvanders_creating

osd: add creating to pg_string_state

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24304 from tchaikov/wip-monclient-remove-unused-label

mon/MonClient: use scoped_guard instead of goto

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24372 from alfredodeza/wip-rm36247

ceph-volume: skip processing devices that don't exist when scanning system disks

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa/distros: add openSUSE Leap 42.3 and 15.0

References: https://tracker.ceph.com/issues/35927
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24180 from colletj/wip-35914-mmgrconfigure-osd-query

mgr: update MMgrConfigure message to include optional OSD perf queries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: add optional OSD perf queries to MMgrConfigure message

Fixes: https://tracker.ceph.com/issues/35914
Signed-off-by: Julien Collet <julien.collet@cern.ch>

ceph-volume util.disk when there are no devices mapped, skip to the next one

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util verify devices that don't exist don't break get_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Add permission validation to the  "Purge Trash" button

Added missing permission validations in the RBD Trash endpoints.

Fixes: http://tracker.ceph.com/issues/36272
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24349 from ricardoasmarques/wip-ex-message-py3

mgr/dashboard: Exception.message doesn't exist on Python 3

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: Fix typo error on cephfs/fuse/

Fixes: http://tracker.ceph.com/issues/36180
Signed-off-by: Karun Josy <kjosy@redhat.com>

Merge pull request #24266 from theanalyst/doc/releases/13.2.2

doc: release notes for 13.2.2 mimic

Reviewed-By: Nathan Cutler <nculter@suse.com>

Merge pull request #24354 from idryomov/wip-krbd-kfsx-fancy-striping

qa/suites/krbd: more fsx tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: fix broken fstab url in cephfs/fuse

Fixes: http://tracker.ceph.com/issues/36286
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #24291 from tchaikov/wip-23964

erasure-code: add clay codes

Reviewed-by: Myna Vajha <mynaramana@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24256 from dzafman/wip-36170

osd: Remove old bft= which has been superceded by backfill=

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24147 from sebastian-philipp/orchestrator-debuggability

mgr/orchestrator: Improve debuggability

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #24346 into master

* refs/pull/24346/head:
doc: misc fixes cephfs-shell

Reviewed-by: Pavani Rajula <rpavani1998@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24340 from tchaikov/wip-36260

qa/workunits/mon/test_mon_config_key.py: bump up the size limit

Merge PR #21566 into master

* refs/pull/21566/head:
test: add test for mds drop cache command
mds: command to trim mds cache and client caps
mds: implement journal flush as asynchronous context execution
mds: cleanup some asok commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24355 from alfredodeza/wip-rm36249

ceph-volume:  activate option --auto-detect-objectstore respects --no-systemd

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24044 into master

* refs/pull/24044/head:
qa: tolerate failed rank while waiting for state

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #23674 from wjwithagen/wjw-wip-ceph-volume-zfs

ceph-volume: zfs, the initial first submit.

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24275 into master

* refs/pull/24275/head:
qa: add timeout to cleaning up workunit sandbox
qa: add timeout to kclient umount
qa: do not cleanup sandbox on error
qa: use default timeout in fs workunits

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-volume lvm.activate pass no_systemd args when detecting objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.lvm verify activate works with --auto-detect-objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: zfs, the inital first submit.

Does nothing other than invoce the ceph-volume-zfs modules when
called with 'ceph-volume zfs

Like:
> ceph-volume zfs list

Use ZFS as the underlying technology for OSDs

--verbose   Increase the verbosity level

['list']
-->  TypeError: main() takes exactly 2 arguments (1 given)
Exit 1

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/suites/krbd: more fsx tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: don't use time() for seeding

When starting multiple concurrent jobs with rbd_fsx task, all jobs get
the same seed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #24073 from AntonOks/master

doc: added some Civetweb configuration options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24343 from jecluis/wip-dashboard-fix-oa

mgr/dashboard: fix reference to oA

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24237 from votdev/ignore_idea_dir

.gitignore: Ignore .idea directory

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Check content-type before decode json response

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Exception.message doesn't exist on Python 3

Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc: misc fixes cephfs-shell

Signed-off-by: Jos Collin <jcollin@redhat.com>

qa: add timeout to cleaning up workunit sandbox

If there is a bug preventing rm from completing, the workunit will get stuck.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add timeout to kclient umount

Otherwise QA sits forever waiting for the kclient to umount when there is a
problem.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not cleanup sandbox on error

Otherwise the command will hang if the mount is broken.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use default timeout in fs workunits

Six hours is unnecessarily long.

Fixes: http://tracker.ceph.com/issues/36184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: fix reference to oA

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

rbd-mirror: use pool level config overrides

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add rbd mirror tunables to pool level config overrides

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/mon/test_mon_config_key.py: bump up the size limit

in b38b8e980cb477ab2b0f320ab51eaa0c0fec7da6, we changed the upper
limit of size of `config key` 's value to 64k, so we need to update
the test accordingly.

Fixes: http://tracker.ceph.com/issues/36260
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24302 from hydro-b/fix-rbd-nbd-docu

doc: add "--timeout" option to rbd-nbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24289 from ukernel/wip-msg-clear-payload

msg: clear message middle when clearing encoded message buffer

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24301 from tchaikov/wip-async-connect-timeout

msg/async: keep connection alive only actually sending

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #24278 from wjwithagen/wjw-fix-inline_variant

common: fix missing include boost/noncopyable.hpp

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24263 from tchaikov/wip-drop-lock-policy

auth,common: use ceph::mutex instead of LockMutex

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24336 from alfredodeza/wip-rm36251

ceph-volume add new ceph-handlers role from ceph-ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: add "--timeout" option to rbd-nbd

Fixes: https://tracker.ceph.com/issues/22333
Signed-off-by: Stefan Kooman <stefan@bit.nl>

ceph-volume tests.functional add new ceph-handlers role from ceph-ansible

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24330 from alfredodeza/wip-rm36098

ceph-volume:  lsblk can fail to find PARTLABEL, must fallback to blkid

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24315 from tchaikov/wip-jenkins-does-not-read-package-desc

install-deps: do not specify unknown options

Reviewed-by: Alfredo Deza <adeza@redhat.com>

test: add test for mds drop cache command

Fixes: http://tracker.ceph.com/issues/23362
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: command to trim mds cache and client caps

With this command, the MDS would request clients to release
caps followed by trimming its own cache and a journal flush.
The command accepts a timeout to wait for clients to respond
to session recall and flush messages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Rishabh Dave <ridave@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: implement journal flush as asynchronous context execution

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: cleanup some asok commands

Signed-off-by: Rishabh Dave <ridave@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

install-deps: do not specify unknown options

per apt.conf(5), and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550564,
"Acquire::Translation" is not documented and will be deprecated. it
would be better to use "Acquire::Languages" solely. and it's consistent
with https://github.com/ceph/ceph-build/pull/1137 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume simple.scan better detection for ceph-disk data devices

Uses the new ``CephDiskDevice`` to look into PARTLABEL from both lsblk
(the default) falling back to blkid, which in some cases has the right
value when lsblk fails.

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util verify CephDiskDevice behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.conftest add blkid patching to device_info fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device add a ceph-disk device object to detect types

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24300 from trociny/wip-36236

test/rbd_mirror: race in WaitingOnLeaderReleaseLeader

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23950 from ceph/wip-rgw-dpp-metasync

rgw: log refactoring for meta sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon/MonClient: use scoped_guard instead of goto

also silences the unused label warning. as "out_shutdown" is not used
anymore after f35e10f4849440f0583d021e23a83563dd3b1607 .

Signed-off-by: Kefu Chai <kchai@redhat.com>