rgw: check lc objs not empty after fetching

During lifecycle processing, the bucket's objects
may be deleted by other clients, so we need to
check objs not empty after each fetch.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #25745 from socketpair/precision

Formatters: improve precision of double numbers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25725 from cfanz/wip-mgr-handle-monmap

mgr: update daemon_state when necessary

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26155 from tchaikov/wip-osd-cleanup

osd/PG: remove unused functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26156 from tchaikov/wip-use-latest-setuptools

install-deps.sh: use the latest setuptools

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26138 from Songweibin/wip-rbd-py-cleanup

pybind/rbd: fix a typo in metadata_get comments

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26105 from neha-ojha/wip-37886

mgr/DaemonServer: log pgmap usage to cluster log

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26086 into master

* refs/pull/26086/head:
include/xlist: prevent item moves
include/xlist: privatize item members

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25616 from s0nea/wip-dashboard-osd-recv-speed-default-vals

mgr/dashboard: consider config option default values

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #24813 from wjwithagen/wjw-fix-alpine-compat

common: Fix Alpine compatability for TEMP_FAILURE_RETRY and ACCESSPERMS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25916 from tchaikov/wip-debian-librados-dev

debian: librados-dev should replace librados2-dev

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26099 from jdurgin/wip-remove-ms-async-affinity

common/options: remove unused ms async affinity options

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26026 from tchaikov/wip-memstore-ceph-mutex

os/memstore: use ceph::mutex and friends

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26159 from dzafman/wip-gitignore

git: Ignore tags anywhere

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26150 from dillaman/wip-rados-lock-info

rados: return legacy address in 'lock info'

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #25599 into master

* refs/pull/25599/head:
mds: do not log internal journal error
mds: clean up log level 2 messages
mds: log session open stats

Reviewed-by: Zheng Yan <zyan@redhat.com>

git: Ignore tags anywhere

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #26096 into master

* refs/pull/26096/head:
mds: remove cache drop asok command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #26087 into master

* refs/pull/26087/head:
mds: fix MDSGather inheritance
include: remove unnecessary inheritance

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26083 into master

* refs/pull/26083/head:
mds: trim cache after journal flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #26082 into master

* refs/pull/26082/head:
client: flush cap release on session flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

install-deps.sh: use latest setuptools

this change reverts "install-deps.sh: workaround setuptools' dependency on six"

This reverts commit 30ce5e55460e333e1240704112a16fae68b3325f.

tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.

long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: remove unused functions

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23904 from xiexingguo/wip-27985

osd/PG: fix misused FORCE_RECOVERY[BACKFILL] flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #25254 into master

* refs/pull/25254/head:
mds: optimize resuming stale caps
client: avoid unnecessary wakeup when handling RENEWCAPS
client: set cap->wanted when adding new cap
client: don't wakeup cap waiters twice when mds recovered
mds: optimize revoking stale caps
mds: put notable caps at the front of session's caps list
mds: track if client has writeable range in Capability
mds: add session pointer to Capability
client: skip updating 'wanted' caps if caps are already issued
client: sync 'retain caps' logical from kernel client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25497 from jmolmo/wip_ansible_orchestrator

mgr/ansible: Create/Remove OSDs

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #25848 from majianpeng/debug-buffer-compiler-error

common/buffer: fix compiler bug when enable DEBUG_BUFFER.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rados: return legacy address in 'lock info'

There is no need to differentiate v1 vs v2 addresses and we should
avoid breaking backwards compatibility in the output.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26022 from theanalyst/common-mclock-prio

common, osd: set mclock priority as 1 by default.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #26043 from tchaikov/wip-37975

osd: unmount store after service.shutdown()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26053 from Aran85/pipeline-fix

osd: ec saves a write access to the memory under most circumstances

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26119 from smithfarm/wip-fix-suse-buildrequires

rpm: fix xmlsec1 build dependency for dashboard make check

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mds: optimize resuming stale caps

If client doesn't want any cap, there is no need to re-issue stale
caps.

A special case is that client wants some caps, but skipped updating
'wanted'. For this case, client needs to update 'wanted' when stale
session get renewed.

Fixes: http://tracker.ceph.com/issues/38043
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

pybind/rbd: fix a typo in metadata_get comments

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #26120 from idryomov/wip-doc-osd-map-config

doc/rados/configuration: refresh osdmap section

Reviewed-by: Neha Ojha <nojha@redhat.com>

mds: trim cache after journal flush

Dropping cache should trim the cache only after journal flush otherwise dirty
inodes will remain pinned in cache.

Fixes: http://tracker.ceph.com/issues/38010
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include/xlist: prevent item moves

This would result in undefined behavior.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include/xlist: privatize item members

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix MDSGather inheritance

It is no longer needlessly a MDSInternalContext.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: remove unnecessary inheritance

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26122 from neha-ojha/wip-fix-crush-tunables

qa/suites/rados/thrash: change crush_tunables to jewel in rados_api_tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #26057 from yaozongyou/remove-unused-class-read_log_and_missing_error

osd: remove unused class read_log_and_missing_error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26101 from neha-ojha/wip-pglog-overrides

qa/overrides/short_pg_log.yaml: reduce osd_{min,max}_pg_log_entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash: change crush_tunables to jewel in rados_api_tests

Fixes: http://tracker.ceph.com/issues/38042
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/rados/configuration: refresh osdmap section

"osd map cache size" and "osd map message max" were reduced in commit
855955e58e63 ("osd: reduce size of osdmap cache, messages").

"osd map cache bl size" and "osd map cache bl inc size" were removed
six years ago.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rpm: fix xmlsec1 build dependency for dashboard make check

Fixes: 46d286c7f71f4c43766d2a3d7597816346e36151
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25796 from linuxbox2/luminous-fix-ldap-secret

rgw: fix ldap secret parsing

Merge pull request #26084 from ceph/wip-rm24871

ceph-volume: mark a device not available if it belongs to ceph-disk

mgr/dashboard: consider config option default values

Consider config option default values as a fallback if no values are
defined by the user when detecting a profile.

Fixes: https://tracker.ceph.com/issues/37683
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25472 from s0nea/wip-dashboard-osd-recv-speed-validation

mgr/dashboard: Validate the OSD recovery priority form input values

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26034 from alfredodeza/wip-rm37963

ceph-volume fix TypeError on dmcrypt when using Python3

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/DaemonServer: log pgmap usage to cluster log

- change default of mon_cluster_log_file_level to debug
- log to clog->debug()

Fixes: https://tracker.ceph.com/issues/37886
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26039 from xiexingguo/wip-37940

osd/OSDMap: be more aggressive when trying to balance

Reviewed-by: Sage Weil <sage@redhat.com>

qa/overrides/short_pg_log.yaml: reduce osd_{min,max}_pg_log_entries

Fixes: http://tracker.ceph.com/issues/38025
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #26017 into master

* refs/pull/26017/head:
ceph: cleanup mds ceph status output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26079 from trociny/wip-38006

librbd: properly filter out trashed non-user images on purge

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/options: remove unused ms async affinity options

These were never implemented. They can be added back if they are
implemented and shown to help performance.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25984 from tspmelo/wip-fix-summary

mgr/dashboard: Fix summary refresh call stack

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mds: remove cache drop asok command

`cache drop` is a long running command that will block the asok interface
(while the tell version does not). Attempting to abort the command with ^C or
equivalents will simply cause the `ceph` command to exit but won't stop the
asok command handler from waiting for the cache drop operation to complete.

Instead, just allow the tell version.

Fixes: http://tracker.ceph.com/issues/38020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fix summary refresh call stack

Currently each time we created a task it would call the summary refresh method.
Since that method was a recursive method, overtime we would get N calls to the
refresh each 5 seconds.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25988 from Exotelis/branding

mgr/dashboard: Add guideline how to brand the UI and update the color scheme

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/ansible: Solve merge problems

Add/fix <forgotten> things after a <git rebase --onto>

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

mgr/dashboard: add validators to osdRecvSpeedForm

Fixes: https://tracker.ceph.com/issues/37436
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/ansible: Adapt code to use DriveGroupSpec and OrchestratorClientMixin

Apart of this two major changes, and due the long time this PR has been opened, other minor modifications coming from the orchestrators world has been merged.

Not solved yet the discussion around <osd add/rm vs osd create/remove>. Once we have a a common code base we can decide what names we should use and what implementation is the more adequate.

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

mgr/ansible: Add create/remove osd functionality

- Create/remove osd functionality
- Addressed typo find by @tchaikov
- Improved code in functional orchestrator methods following @leseb suggestions
- Addressed @sebastian-philipp  suggestions
- Use <drive group> specification for create OSD in all the orchestrators
- fixed errors calling HandleCommandResult

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #25991 from jtlayton/wip-dashboard

mgr/dashboard: add a script to run an API request on a rook cluster

Reviewed-by: Ricardo Dias <rdias@suse.com>

os/memstore: use ceph::mutex and friends

it is another step to ditch RWLock, and paves the road to lockless
memstore which can be used by crimson-osd directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: do a little clean-up of the component

- order component methods alphabetically
- add missing newline
- uncheck checkbox and clear validations if a new profile has been choosen

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: convert ``priorityAttrs`` from array to object

Convert ``priorityAttrs`` from array to object for better handling,
e.g. to get rid of unnecessary for-loops.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: call API only once to get config options infos

Refactor the component in order to call API only once to get the infos
about the relevant config options and (if set) the current values.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: move config option types to their own class

Move the config option types information into their own class to
reuse them easily in other components.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix linting errors

Fixes linting error in two files

Signed-off-by: Sebastian Krah <skrah@suse.com>

ceph-volume: mark a device not available if it belongs to ceph-disk

The `ceph-volume inventory` command will now show if a device is being
used by ceph-disk and mark it not available if so.

Fixes: https://tracker.ceph.com/issues/24871
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

client: flush cap release on session flush

MDS expects the completed cap release prior to responding to the session flush
for cache drop.

Fixes: http://tracker.ceph.com/issues/38009
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25774 from yehudasa/wip-rgw-tiering-3

rgw: rados tiering

librbd: properly filter out trashed non-user images on purge

Fixes: https://tracker.ceph.com/issues/38006
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26073 from rhcs-dashboard/add-tox-ini-command

mgr/dashboard: added command to tox.ini

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26020 from rhcs-dashboard/issue-37854

mgr/dashboard: Fixed performance details context for host list row selection

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #26007 from rhcs-dashboard/upgrade-python-dev-deps

mgr/dashboard: upgraded python dev dependencies

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Formatters: improve precision of double numbers

Default precision is not enough for at least JSON
formatter. So some information loss possible. This
fix makes numbers long enough.

Signed-off-by: Коренберг Марк <socketpair@gmail.com>

mgr/dashboard: added command to tox.ini

* Added 'run' command to run arbitrary commands so
  you don't have to run the whole test/lint suite.
  Use case examples:
  - Run only 1 test:
  tox -e py3-run pytest tests/test_rgw_client.py::RgwClientTest::test_ssl_verify
  - Run pycodestyle for only 1 file:
  tox -e py3-run pycodestyle tests/test_rgw_client.py

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #26032 from jtlayton/wip-rook

mgr/rook: remove dead code and fix bug in url fetching code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25895 from jan--f/c-v-func-test-local

ceph-volume: functional tests, add libvirt customization

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26044 from tchaikov/wip-build-doc-pyyaml

doc: bump up sphinx and pyyaml versions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: bump up sphinx and pyyaml versions

* bump up pyyaml version to address CVE-2017-18342, see
  https://nvd.nist.gov/vuln/detail/CVE-2017-18342
* bump up sphinx to the latest stable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26052 from tchaikov/wip-37976

mgr/zabbix: drop "total_objects" field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26054 from tchaikov/wip-crimson-dispatcher-fix-ftbfs

crimson/osd: remove "force_new" from ms_get_authorizer()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #26045 from hsiang41/Ticket_33736

mgr/diskprediction_cloud: fix divide by zero when total_size is 0

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: remove unused class read_log_and_missing_error

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

crimson/osd: remove "force_new" from ms_get_authorizer()

it was removed in e62f45d1, so update crimson-osd accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: ec saves a write access to the memory under most circumstances

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #26046 from liewegas/wip-37976

mgr/telemetry: fix total_objects

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: drop "total_objects" field

This field was removed from df output a while back in
342f309

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25712 from familyuu/configuration

mgr/dashboard: Add a 'clear filter' button to configuration page

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/diskprediction_cloud: fix divide by zero when total_size is 0

Fix devide by zero when total_size is o.

Ref ticket: https://tracker.ceph.com/issues/37736

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

mgr/telemetry: fix total_objects

This field was removed from df output a while back in
342f309645df886fb96eb401634e38376553e6d9

Fixes: http://tracker.ceph.com/issues/37976
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26031 into master

* refs/pull/26031/head:
mgr/BaseMgrModule: fix mon commands targetting a specific monitor
mgr/ActivePyModules: fix RestoreThread calls for get('devices')

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: unmount store after service.shutdown()

in service.shutdown(), the store's cache is accessed, so we need to
destroy the store after service.shutdown()

Fixes: http://tracker.ceph.com/issues/37975
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25490 from Aran85/fix-ec-using-cache

osd: keep using cache even if op will invalid cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25920 from wjwithagen/wjw-fix-freebsd-numa

common/numa: Add shim routines for NUMA on FreeBSD.