mgr/dashboard: Save button doesn't prevent saving an invalid form

Fixes: http://tracker.ceph.com/issues/36426
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24564 from neha-ojha/wip-21931

osd: do not overestimate the size of the object for reads with trimtrunc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #24204 into master

* refs/pull/24204/head:
qa/suites/rgw/tempest: valgrind on centos only

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge PR #24249 into master

* refs/pull/24249/head:
doc/rados/troubleshooting-mon: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24296 into master

* refs/pull/24296/head:
osd: Handle is_stopping() by discarding item so it doesn't loop
osd: Simplify _process() logic recently added

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24131 into master

* refs/pull/24131/head:
tests, common: introduce bufferlist::apennd benchmark.
test, common: add BufferListIterator.BenchDeref benchmark.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24270 into master

* refs/pull/24270/head:
osd: make 'cache drop' command require 'executable' permission
osd: rename 'drop cache' and 'get cache stats' to group them by component
doc: add documentation for 'drop cache' and 'get cache stats'
osd: don't print osdmap cache stats in 'get cache stats' command
osd: do not clear osdmap cache on 'drop cache' command
osd: offload dumping cache stats to the object store
osd: pass a stream to flush_cache commands for more verbosity
osd: implement flush_cache() method for Filestore
osd: add clear_cache and get_cache_object_count commands

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24312 into master

* refs/pull/24312/head:
osd: kill the std::stringstream in ReplicatedBackend::do_repop_reply.
osd: bump-up the dout level in PGLog::write_log_and_missing.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24520 into master

* refs/pull/24520/head:
install-deps.sh: dashboard frontend needs git

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24521 into master

* refs/pull/24521/head:
ceph.in: reinstate interactive mode output

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PrimaryLogPG.cc: reassign size only when object size > truncate_size

Before setting size equal to op.extent.truncate_size, we need to check
if the size of the object is greater than the truncate_size. We do not
need to set size to op.extent.truncate_size, in the case where the size of
the object is less than op.extent.truncate_size.

Without this change, we were always setting size =
op.extent.truncate_size, when (seq < op.extent.truncate_seq) and
(op.extent.offset + op.extent.length > op.extent.truncate_size), were both
true. This ended up in:

1. overestimating the size of the object
2. not considering the correct size of the object, for
   the later checks, which calculate op.extent.length for the read ops
3. causing crashes when trying to read more data than what was present

Fixes: http://tracker.ceph.com/issues/21931
Fixes: http://tracker.ceph.com/issues/22330
Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: print offset and length to track trimtrunc

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24341 from joke-lee/WIP_FIX_RGWPOSTOBJ_VERSIONING

rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning

Merge pull request #24550 from dillaman/wip-36409

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24517 from jcsp/wip-23970-test

mgr/selftest: fix disabled module selection

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24426 from Miouge1/grafana-fix1

monitoring/grafana: Fix OSD Capacity Utlization Grafana graph

Merge pull request #24511 from a2batic/grafana-doc

mgr/dashboard: Improves documentation for Grafana Setting

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24513 from rhcs-dashboard/27047-landing-page-info-visibility

mgr/dashboard: Landing Page: info visibility

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24222 from votdev/bug_36109

mgr/dashboard: The RGW backend doesn't handle IPv6 properly

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/tasks: qemu task now uses a relative path in suite repo for test

This makes it easier to re-run tests against a suite branch without
requiring a full ceph-ci build and repo.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24402 from cbodley/wip-36302

librgw: initialize curl and http client for multisite

Merge PR #24514 into master

* refs/pull/24514/head:
mgr/restful: do not print warning message at seeing unknown request

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Fixes: http://tracker.ceph.com/issues/36409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24264 into master

* refs/pull/24264/head:
tools/rados/rados.cc: fix rados rm --force-full blocking problem

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24287 into master

* refs/pull/24287/head:
osd/OSD: drop osd_lock only if pg osdmaps are truly laggy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24331 into master

* refs/pull/24331/head:
common: fix memory leaks in WeightedPriorityQueue.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24365 into master

* refs/pull/24365/head:
erasure-code: var-sized array init is done separatly.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24506 into master

* refs/pull/24506/head:
doc/rados/troubleshooting: update notes on rebuilding mon db
ceph-objectstore-tool: update-mon-db: remove pgmap references
ceph-monstore-tool: remove remaining pgmap references, inflate-pgmap command
ceph-monstore-tool: rebuild creating_pgs
ceph-monstore-tool: rebuild: remove obsolete pgmap_meta fields

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24409 from alimaredia/wip-rgw-dpp-auth

rgw: thread DoutPrefixProvider into RGW::Auth_S3::authorize

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rados/troubleshooting-mon: update mondb recovery script

- some cleanup (e.g., use $ms throughput)
- behave if the local host is in the $hosts list (use $ms.remote)
- be clear about updating all mons
- mon.0 -> mon.foo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24428 from trociny/wip-rbd-config-global

rbd: add 'config global' command to get/store overrides in mon config db

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24541 into master

* refs/pull/24541/head:
common/config.h: fix spelling error in this file

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/selftest: fix disabled module selection

This was broken by always on modules -- it was also
unnecessary complication, let's just pick some
hardcoded module to use.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #24305 into master

* refs/pull/24305/head:
msg/async: ProtocolV1: use continuations instead of direct function calls
msg/async: ProtocolV1 state machine ascii diagram

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

install-deps.sh: dashboard frontend needs git

Running `npm install` for the dashboard pulls dependenies with git.
Under some setups, git was not installed when building the frontend.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #24404 from ceph/wip-batch-report-idempotent

ceph-volume: make `lvm batch` idempotent

Reviewed-by: Alfredo Deza <adeza@redhat.com>

common/config.h: fix spelling error in this file

Signed-off-by: xiaomanh <huangxiaoman@cmss.chinamobile.com>

Merge pull request #24425 from b-ranto/wip-ld-hardening

rpm: Use hardened LDFLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Improves documentation for Grafana Setting

Fixes: http://tracker.ceph.com/issues/36371
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge PR #23069 into master

* refs/pull/23069/head:
tests/libcephfs: add simple reclaim test
mds: check auth name before reclaiming session
mds: reclaim session before allowing mds to become active
mds: allow client to specify its session timeout
mds: initial code for client states reclaim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: extracts batch.filter_devices from Batch._get_strategy

This allows us to easily provide tests for that method.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: failing to get block db size from conf logs an exception

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when all devices are filtered exit gracefully

Even if all devices are filtered we want to return a 0 exit code and
make sure the json reporting still works.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: filter devices used by journals/block.db

If after filterering of data/block devices there are only
one device left it can not be used if it is an SSD and
has been used previously as a journal or block.db

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add rotational property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add info about filtered devices to batch pretty reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove the used_by_ceph key in the json output

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix idempotency checks for lvm batch tests

The mixed type tests will change strategy after the idempotency test so
we need to handle that in test playbook.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix bluestore strategy json reporting and type

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: ignore failure to load ceph configuration for block.db size

If we fail to load a ceph configureation file when trying to get the
block.db size then just use defaults instead of throwing an error.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix strategy comparison in 'lvm batch'

This also fixes some small json reporting issues with the
filestore MixedType strategy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: raise a non zero exit code if strategy changes with batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add functional tests to ensure lvm batch is idempotent

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add tests for util.device.Device.used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update tests to account for filtered_devices in batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: consider block and data devices used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add filtered_devices and used_by_ceph to all batch reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: pick strategy for batch with only the unused devices

This will pick a strategy, filter out any devices already been used by
ceph and then pick a strategy again. If the strategy has changed the
call should error, if the strategy is the same proceed. If there are no
unused devices then the command is a noop.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a 'changed' key to lvm batch --report

This will indicate if the command would result in any OSDs being created
or not. Other tooling can use that key for idempotency checks.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds used_by_ceph to filestore singletype batch report

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a used_by_ceph property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a lvs property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add vg_name to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24525 from jtlayton/k8s-no-module-warn

mgr/orchestrator: clarify error message about kubernetes python module

Reviewed-by: John Spray <john.spray@redhat.com>

osd: make 'cache drop' command require 'executable' permission

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: rename 'drop cache' and 'get cache stats' to group them by component

Rename 'drop cache' to 'cache drop' and 'get cache stats' to 'cache
status'.

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

doc: add documentation for 'drop cache' and 'get cache stats'

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: don't print osdmap cache stats in 'get cache stats' command

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: do not clear osdmap cache on 'drop cache' command

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: offload dumping cache stats to the object store

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: pass a stream to flush_cache commands for more verbosity

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: implement flush_cache() method for Filestore

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

osd: add clear_cache and get_cache_object_count commands

Fixes: http://tracker.ceph.com/issues/24176
Signed-off-by: Mohamad Gebai <mgebai@suse.com>

Merge pull request #24476 from dillaman/wip-35944

librbd: add new configuration option to always move deleted items to the trash

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24512 from trociny/wip-qa-namespace-create_args

qa/workunits/rbd: force v2 image format for namespace test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: add 'config global' command to get/store overrides in mon config db

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Landing Page: info group visibility
Do not display info groups not containing any card.
Fixes: https://tracker.ceph.com/issues/27047
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

ceph.in: reinstate interactive mode output

This was broken in the following commit:
"""
commit 474c94b81efad1b954ab789bf8a97d93cd0e74f3
Author: Mohamad Gebai <mgebai@suse.com>
Date:   Wed Feb 28 18:12:19 2018 -0500

ceph.in: handle polling commands

Signed-off-by: Mohamad Gebai <mgebai@suse.com>
"""

Fixes: http://tracker.ceph.com/issues/36358
Signed-off-by: John Spray <john.spray@redhat.com>

doc/rados/troubleshooting: update notes on rebuilding mon db

- those old pg settings are no longer lost as they go in the OSDMap now
- add a note about pools that were mid-create

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: update-mon-db: remove pgmap references

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: remove remaining pgmap references, inflate-pgmap command

These are all also obsolete.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: rebuild creating_pgs

In particular, we need to rebuild the created_pools set or else the mon
will try to recreate every PG in the system.  On an aged cluster, that
is extremely expensive, and will break things.

Fixes: http://tracker.ceph.com/issues/36306
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: rebuild: remove obsolete pgmap_meta fields

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: test new move-to-trash on image remove config override

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optionally move image to trash when removed by user

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move remove API method to api namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move trash API methods to api namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24407 from alfredodeza/wip-rm26953

ceph-volume: allow to specify --cluster-fsid instead of reading from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24515 from tchaikov/wip-cmake-rgw

cmake: do not use plain target_link_libraries(rgw_a ...)

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23622 from renhwztetecs/renhw-wip-25103

mgr: fixup pgs show in unknown state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24448 from votdev/bug_36325

mgr/dashboard: Performance counter progress bar keeps infinitely looping

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23970 from noahdesu/mgr-always-on-mods-v2

mgr: always on modules v2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24239 from votdev/python_ceph_audit_log

mgr: Add ability to trigger a cluster/audit log message from Python

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: do not use plain target_link_libraries(rgw_a ...)

this addresses following error:

CMake Error at src/rgw/CMakeLists.txt:178 (target_link_libraries):
  The plain signature for target_link_libraries has already been used
with
  the target "rgw_a".  All uses of target_link_libraries with a target
must
  be either all-keyword or all-plain.

  The uses of the plain signature are here:

   * src/rgw/CMakeLists.txt:168 (target_link_libraries)

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/restful: do not print warning message at seeing unknown request

mgr broadcast "notify()" to call registered plugins upon finishing a
send_command() request. and restful plugin prints a warning message if
the request is not issued by itself. apparently, there is good chance
that the finished request is sent by other mgr plugins, in that case,
restful plugin always prints the warning messages like:

 "2018-10-09 14:52:07.818 7fbc4d8c5700  1 mgr[restful] Unknown request
''

this is misleading and annoying. so, in this change

* add a prefix of "restful.module" before that tag used for identifying
  a request. this prefix helps to differentiate the requests sent by
  restful plugin from other requests.
* do not print warning message if none running request matches the given tag
* break at seeing the first request matches the given tag, because
  the tags used by restful plugin are unique.
* do not str(self.id) before composing the tag, as it is a `str`
  already, see CommandsRequest.__init__()
* early return to reduce the indent level

Fixes: http://tracker.ceph.com/issues/36374
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24487 from badone/wip-undefined-references-to-dpdk-symbols

cmake/modules/BuildDPDK.cmake: Build required DPDK libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24450 from dillaman/wip-rbd-conf-keys

librbd: simplify config override handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: force v2 image format for namespace test

While here, fix test_locking too.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24509 from badone/wip-releases-add-13.2.2

doc/releases: Add Mimic release 13.2.2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24368 from tchaikov/wip-cmake-cleanup

cmake: use OpenSSL::Crypto instead of OPENSSL_LIBRARIES

Reviewed-by: Casey Bodley <cbodley@redhat.com>