osd,mon: keep last_epoch_started along with last_epoch_clean premerge

We need to populate both the last_epoch_clean and last_epoch_started
accurately when fabricating a pg merge target.  In the case where the
premerge PG peers, recovers (e.g., backfill), then becomes ready to
merge, the last_epoch_clean may be > the last_epoch_started, and using
the later last_epoch_clean will prevent the PG from subsequently
peering (it'll go to incomplete state instead).

Fix by keeping both values, separately.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24341 from joke-lee/WIP_FIX_RGWPOSTOBJ_VERSIONING

rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning

Merge pull request #24550 from dillaman/wip-36409

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24517 from jcsp/wip-23970-test

mgr/selftest: fix disabled module selection

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24426 from Miouge1/grafana-fix1

monitoring/grafana: Fix OSD Capacity Utlization Grafana graph

Merge pull request #24511 from a2batic/grafana-doc

mgr/dashboard: Improves documentation for Grafana Setting

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24513 from rhcs-dashboard/27047-landing-page-info-visibility

mgr/dashboard: Landing Page: info visibility

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24222 from votdev/bug_36109

mgr/dashboard: The RGW backend doesn't handle IPv6 properly

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/tasks: qemu task now uses a relative path in suite repo for test

This makes it easier to re-run tests against a suite branch without
requiring a full ceph-ci build and repo.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24402 from cbodley/wip-36302

librgw: initialize curl and http client for multisite

Merge PR #24514 into master

* refs/pull/24514/head:
mgr/restful: do not print warning message at seeing unknown request

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Fixes: http://tracker.ceph.com/issues/36409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24264 into master

* refs/pull/24264/head:
tools/rados/rados.cc: fix rados rm --force-full blocking problem

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24287 into master

* refs/pull/24287/head:
osd/OSD: drop osd_lock only if pg osdmaps are truly laggy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24331 into master

* refs/pull/24331/head:
common: fix memory leaks in WeightedPriorityQueue.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #24365 into master

* refs/pull/24365/head:
erasure-code: var-sized array init is done separatly.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24506 into master

* refs/pull/24506/head:
doc/rados/troubleshooting: update notes on rebuilding mon db
ceph-objectstore-tool: update-mon-db: remove pgmap references
ceph-monstore-tool: remove remaining pgmap references, inflate-pgmap command
ceph-monstore-tool: rebuild creating_pgs
ceph-monstore-tool: rebuild: remove obsolete pgmap_meta fields

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24409 from alimaredia/wip-rgw-dpp-auth

rgw: thread DoutPrefixProvider into RGW::Auth_S3::authorize

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24428 from trociny/wip-rbd-config-global

rbd: add 'config global' command to get/store overrides in mon config db

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24541 into master

* refs/pull/24541/head:
common/config.h: fix spelling error in this file

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/selftest: fix disabled module selection

This was broken by always on modules -- it was also
unnecessary complication, let's just pick some
hardcoded module to use.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #24305 into master

* refs/pull/24305/head:
msg/async: ProtocolV1: use continuations instead of direct function calls
msg/async: ProtocolV1 state machine ascii diagram

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24404 from ceph/wip-batch-report-idempotent

ceph-volume: make `lvm batch` idempotent

Reviewed-by: Alfredo Deza <adeza@redhat.com>

common/config.h: fix spelling error in this file

Signed-off-by: xiaomanh <huangxiaoman@cmss.chinamobile.com>

Merge pull request #24425 from b-ranto/wip-ld-hardening

rpm: Use hardened LDFLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Improves documentation for Grafana Setting

Fixes: http://tracker.ceph.com/issues/36371
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge PR #23069 into master

* refs/pull/23069/head:
tests/libcephfs: add simple reclaim test
mds: check auth name before reclaiming session
mds: reclaim session before allowing mds to become active
mds: allow client to specify its session timeout
mds: initial code for client states reclaim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: extracts batch.filter_devices from Batch._get_strategy

This allows us to easily provide tests for that method.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: failing to get block db size from conf logs an exception

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when all devices are filtered exit gracefully

Even if all devices are filtered we want to return a 0 exit code and
make sure the json reporting still works.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: filter devices used by journals/block.db

If after filterering of data/block devices there are only
one device left it can not be used if it is an SSD and
has been used previously as a journal or block.db

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add rotational property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add info about filtered devices to batch pretty reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove the used_by_ceph key in the json output

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix idempotency checks for lvm batch tests

The mixed type tests will change strategy after the idempotency test so
we need to handle that in test playbook.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix bluestore strategy json reporting and type

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: ignore failure to load ceph configuration for block.db size

If we fail to load a ceph configureation file when trying to get the
block.db size then just use defaults instead of throwing an error.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix strategy comparison in 'lvm batch'

This also fixes some small json reporting issues with the
filestore MixedType strategy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: raise a non zero exit code if strategy changes with batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add functional tests to ensure lvm batch is idempotent

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add tests for util.device.Device.used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update tests to account for filtered_devices in batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: consider block and data devices used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add filtered_devices and used_by_ceph to all batch reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: pick strategy for batch with only the unused devices

This will pick a strategy, filter out any devices already been used by
ceph and then pick a strategy again. If the strategy has changed the
call should error, if the strategy is the same proceed. If there are no
unused devices then the command is a noop.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a 'changed' key to lvm batch --report

This will indicate if the command would result in any OSDs being created
or not. Other tooling can use that key for idempotency checks.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds used_by_ceph to filestore singletype batch report

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a used_by_ceph property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a lvs property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add vg_name to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24525 from jtlayton/k8s-no-module-warn

mgr/orchestrator: clarify error message about kubernetes python module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24476 from dillaman/wip-35944

librbd: add new configuration option to always move deleted items to the trash

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24512 from trociny/wip-qa-namespace-create_args

qa/workunits/rbd: force v2 image format for namespace test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: add 'config global' command to get/store overrides in mon config db

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Landing Page: info group visibility
Do not display info groups not containing any card.
Fixes: https://tracker.ceph.com/issues/27047
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

doc/rados/troubleshooting: update notes on rebuilding mon db

- those old pg settings are no longer lost as they go in the OSDMap now
- add a note about pools that were mid-create

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: update-mon-db: remove pgmap references

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: remove remaining pgmap references, inflate-pgmap command

These are all also obsolete.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: rebuild creating_pgs

In particular, we need to rebuild the created_pools set or else the mon
will try to recreate every PG in the system.  On an aged cluster, that
is extremely expensive, and will break things.

Fixes: http://tracker.ceph.com/issues/36306
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-monstore-tool: rebuild: remove obsolete pgmap_meta fields

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: test new move-to-trash on image remove config override

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optionally move image to trash when removed by user

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move remove API method to api namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move trash API methods to api namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24407 from alfredodeza/wip-rm26953

ceph-volume: allow to specify --cluster-fsid instead of reading from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24515 from tchaikov/wip-cmake-rgw

cmake: do not use plain target_link_libraries(rgw_a ...)

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23622 from renhwztetecs/renhw-wip-25103

mgr: fixup pgs show in unknown state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24448 from votdev/bug_36325

mgr/dashboard: Performance counter progress bar keeps infinitely looping

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23970 from noahdesu/mgr-always-on-mods-v2

mgr: always on modules v2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24239 from votdev/python_ceph_audit_log

mgr: Add ability to trigger a cluster/audit log message from Python

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: do not use plain target_link_libraries(rgw_a ...)

this addresses following error:

CMake Error at src/rgw/CMakeLists.txt:178 (target_link_libraries):
  The plain signature for target_link_libraries has already been used
with
  the target "rgw_a".  All uses of target_link_libraries with a target
must
  be either all-keyword or all-plain.

  The uses of the plain signature are here:

   * src/rgw/CMakeLists.txt:168 (target_link_libraries)

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/restful: do not print warning message at seeing unknown request

mgr broadcast "notify()" to call registered plugins upon finishing a
send_command() request. and restful plugin prints a warning message if
the request is not issued by itself. apparently, there is good chance
that the finished request is sent by other mgr plugins, in that case,
restful plugin always prints the warning messages like:

 "2018-10-09 14:52:07.818 7fbc4d8c5700  1 mgr[restful] Unknown request
''

this is misleading and annoying. so, in this change

* add a prefix of "restful.module" before that tag used for identifying
  a request. this prefix helps to differentiate the requests sent by
  restful plugin from other requests.
* do not print warning message if none running request matches the given tag
* break at seeing the first request matches the given tag, because
  the tags used by restful plugin are unique.
* do not str(self.id) before composing the tag, as it is a `str`
  already, see CommandsRequest.__init__()
* early return to reduce the indent level

Fixes: http://tracker.ceph.com/issues/36374
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24487 from badone/wip-undefined-references-to-dpdk-symbols

cmake/modules/BuildDPDK.cmake: Build required DPDK libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24450 from dillaman/wip-rbd-conf-keys

librbd: simplify config override handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: force v2 image format for namespace test

While here, fix test_locking too.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24509 from badone/wip-releases-add-13.2.2

doc/releases: Add Mimic release 13.2.2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24368 from tchaikov/wip-cmake-cleanup

cmake: use OpenSSL::Crypto instead of OPENSSL_LIBRARIES

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake/modules/BuildDPDK.cmake: Build required DPDK libraries

Make sure the mbuf and ethdev libraries are created as they export
required symbols.

Fixes: http://tracker.ceph.com/issues/36341
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/releases: Add Mimic release 13.2.2

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rgw: add DoutPrefixProvider to the Auth Identifier classes

This caused DPP to be threaded through the
verify_{objecy/bucket/user}_permissions functions
in rgw_common.cc and change all of the calls to
them in rgw_op.cc and in other places.

The logging in STS engine authorize was
changed too.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rbd: nicer help output for commands without options

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: print long aliases specs shorter in help output

(by substituting the common part with '...')

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #24496 into master

* refs/pull/24496/head:
doc: fix typo in blkin.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: changing logging in auth helper functions

Functions that aren't auth engine related were did
not have their logging changed

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: refactor logging in auth engines P1

pass DPP into authenticate functions for auth
engines and change any of the logging inside of
them.

Next up is passing DPP and changing the douts
in the helper functions called in authenticate.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: thread DoutPrefixProvider into RGW::Auth_S3::authorize

Pass a DPP into RGWOp::verify_request that leads
through RGWHandler -> RGWHandler_REST->
RGW::Auth_S3::authorize().

Change authenticate() in all the Handler classes
to take a DPP

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #24484 from hsiang41/mgr/predictor_model_file

mgr/diskprediction: Replace local predictor model file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21614 from Devp00l/pool-creation-form

mgr/dashboard: Pool management

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24497 from idryomov/wip-clarify-copy-from

include/rados: clarify which flags go where for copy_from

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

cmake: use OpenSSL::Crypto instead of OPENSSL_LIBRARIES

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/rados: clarify which flags go where for copy_from

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: fix typo in blkin.rst

Signed-off-by: Scoots Hamilton <scoots@redhat.com>

Merge pull request #24491 from tchaikov/wip-add-erasure-clay-in-release-note

PendingReleaseNotes: note newly added CLAY code

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: New formatter byte unit tests

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Unset any compression setting for pools in UI

The dashboard API now can delete all compression options if the
compression mode is set to 'unset' during edit. Other dismissed options
will be overwritten with the reset value on submit, which will cause
the deletion of the previously set settings.

Fixes: https://tracker.ceph.com/issues/36355
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #23230 from s0nea/wip-dashboard-config-options-add

mgr/dashboard config options add

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Update PG update test

It is now commented out like it was before,
but I've added a comment what happened during this test with the QA
system. The problem was that even with only a increase of 1 PG the QA
cluster went into a cluster warning state and did not recover in time.
The QA coverage timeout is 2 minutes.

I could not reproduce this behavior with a local cluster, but I've
added a loop to wait until pgp and pg number are equal and the cluster
is in a healthy state again. This can take locally about 5 seconds.
The internal loop has a timeout of 3 minutes.

Fixes: https://tracker.ceph.com/issues/36362
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Unset compression arguments for pools

The dashboard backend can now unset all set compression arguments if the
compression mode is switched to 'unset'. In the case of 'unset' Ceph
itself will only delete the 'compression_mode' argument, not all other
set arguments. The other arguments that should be removed, too, are
added to the update arguments in order to delete all set arguments.

Fixes: https://tracker.ceph.com/issues/36355
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Readd pool notifications

As of the merge of cleaner notifications, the notification component was
refactored and move this caused a loss of previously added
notifications.

Fixes: https://tracker.ceph.com/issues/36355
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use task list service in pool list

Now the pool list uses the task list service to monitor all tasks, in
order to extend each pool item with corresponding tasks if any. If there
is a running task for a pool you will see what action runs on it in the
listing.

Fixes: https://tracker.ceph.com/issues/36355
Signed-off-by: Stephan Müller <smueller@suse.com>