mgr/dashboard: awsauth: fix python3 string decode problem

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 7f332a396299825eb078c8935449b816ce613d6d)

Merge PR #21867 into mimic

* refs/pull/21867/head:
qa/tasks/cephfs: add test for renewing stale session
client: invalidate caps and leases when session becomes stale
client: fix race in concurrent readdir

qa/tasks/cephfs: add test for renewing stale session

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 5688476513a78cf9ab2cf3b1f65e6244f05ea73d)

client: invalidate caps and leases when session becomes stale

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/23894
(cherry picked from commit 8b2e7d834ccf2a4ff6c7aa3d4aeee07ebe36fb59)

client: fix race in concurrent readdir

For a large directory, program needs to issue multiple readdir
syscalls to get all dentries. When there are multiple programs
read the directory concurrently. Following sequence of events
can happen.

 - program calls readdir with pos = 2. ceph sends readdir request
   to mds. The reply contains N1 entries. ceph adds these N1 entries
   to readdir cache.
 - program calls readdir with pos = N1+2. The readdir is satisfied
   by the readdir cache, N2 entries are returned. (Other program
   calls readdir in the middle, which fills the cache)
 - program calls readdir with pos = N1+N2+2. ceph sends readdir
   request to mds. The reply contains N3 entries and it reaches
   directory end. ceph adds these N3 entries to the readdir cache
   and marks directory complete.

The second readdir call does not update dirp->cache_index. ceph adds
the last N3 entries to wrong places.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/23894
(cherry picked from commit 01e23c178d068a3983c58cf115d57f6e1cc06255)

Merge PR #21866 into mimic

* refs/pull/21866/head:
mds: include nfiles/nsubdirs of directory inode in MClientCaps

mds: include nfiles/nsubdirs of directory inode in MClientCaps

Directory inode's dirstat gets updated by request reply, but not by
cap message. This causes problem for following case.

1. MDS modifies a directory
2. MDS issues CEPH_CAP_ANY_SHARED to client
3. The client satifies stat(2) by its cached metadata.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/23855
(cherry picked from commit ee2c628f6783954e9b25fab8ac9b572a58666a91)

Merge PR #21857 into mimic

* refs/pull/21857/head:
qa: move snap-hierarchy out of snaps workunits
qa: try snap format upgrade with multimds cluster
qa: add mds deactivation procedure for upgrades

qa: move snap-hierarchy out of snaps workunits

The snapshot hierarchy it leaves behind can't be cleaned up by `rm -rf` which
breaks workunit cleanup. So, don't run this as part of normal snaps test.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4d37b0ee8d9594977aeae02c536b5f5fcc187e33)

qa: try snap format upgrade with multimds cluster

Fixes: http://tracker.ceph.com/issues/24002
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0b466cb2e6ccdeb56199cb889f24cc590aedff04)

qa: add mds deactivation procedure for upgrades

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 6a788bf203dc07d32f299ce488b054addaae4f75)

Merge pull request #21832 from tchaikov/mimic-23627

librados: block MgrClient::start_command until mgrmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librados: timeout on mgr_command()

because the mgrclient will be waiting for the mgrmap if the mgrmap
is not available, when the client is about to send a mgr command.
and monitor will drop the subscription requests if the client does not
have enough cap for reading mon, so unlike mon commands, the client
won't get an EACCES return code in this case.

in this change, a timeout machinary is introduced. and the client
will wait for "rados-mon-op-timeout" before it gives up. if this
setting is 0, it will wait forever.

Fixes: https://tracker.ceph.com/issues/23627
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit eaa1179965c840c0f935c570e146395583f198b6)

librados: fix locking on get_required_monitor_features

This wasn't taking the MonClient lock: should use
with_monmap to protect access to MonClient::monmap.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ef517d0ea0dd4754fe725ae8d6f01209b58130be)

librados: config mgrclient for pre-luminous cluster

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8fd500d736bc3c868736537d04940c627587ec63)

mgr/MgrClient: add mgr_optional mode

This is for use when talking to pre-luminous
clusters, where we should not block waiting
for MgrMap because it might never come.

Fixes: https://tracker.ceph.com/issues/23627
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit bae47183c2621577ba564d29c82a0517ab8a7f54)

Merge pull request #21786 from dillaman/wip-23966

osdc: invoke notify finish context on linger commit failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD: drop extra/wrong *unregister_pg*

Signed-off-by: xiexingguo <xie.xingguo@gmail.com>
(cherry picked from commit 6ee6fa79b5fab532dea22399afdcceeb6624019d)

Merge pull request #21753 from liewegas/wip-upgrade-enospc

qa/suites/upgrade/luminous-x/stress-split: avoid enospc

Merge pull request #21824 from tchaikov/mimic-cmake-jni

mimic: cmake: use javac -h for creating JNI native headers

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge branch 'mimic' of github.com:ceph/ceph into mimic

cmake: bump up JDK version to 1.8

as JDK 7 is EOL. see
http://www.oracle.com/technetwork/java/eol-135779.html and
https://www.java.com/en/download/faq/java_7.xml

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit df27ed3a3806aaba41ca7bb1019e586d0b01a049)

cmake: use javac -h for creating JNI native headers

JDK 1.10 does not offer javah anymore, so we need to use "javac -h" or
add_jar(... GENERATE_NATIVE_HEADERS) instead.

Fixes: http://tracker.ceph.com/issues/24012
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit edc504b9d446cc1739f3e8e087914cfe66d6cac3)

mgr: Include daemon details in SLOW_OPS output

Currently there is no way to see which daemons were involved in a slow
op after the op has cleared. This change allows us to record which
daemons were implicated in the logs.

Partially fixes: http://tracker.ceph.com/issues/23205

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit b5263176dea6d204c2caa52043cefc94dd5e475f)

qa/suites/upgarde/luminous-x/stress-split: 8 osds (not 6)

Signed-off-by: Sage Weil <sage@redhat.com>

osdc: invoke notify finish context on linger commit failure

Fixes: http://tracker.ceph.com/issues/23966
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21557 from jan--f/mgr-prometheus-fix-metadata-labels

mgr: prometheus fix metadata labels

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21553 from dragonylffly/wip-improve-rados-ls

tools/rados: improve the ls command usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21795 from wido/influx-string-split

mgr/influx: Only split string on first occurence of dot (.)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21798 from liewegas/wip-23980

osd/PG: fix uninit read in Incomplete::react(AdvMap&)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21771 from majianpeng/osd-remove-pgslot

osd: fix bug which cause can't erase OSDShardPGSlot.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

13.1.0

Merge pull request #21546 from dzafman/wip-22881

osd: process _scan_snaps() with all snapshots with head

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #21374 into master

* refs/pull/21374/head:
qa: add test for snap format upgrade
mds: initialize SnapServer::snaprealm_v2_since after journal replay
mds: properly distinguish cap update from snap flush
mds: update dev document of cephfs snapshot
doc: add release notes for cephfs snapshot
mds: allow snapshot by default for new filesystem
mds: close past parents after snaprealm format gets converted
mds: automaticly allow multi-active MDS after scrubbing all inodes
mds: don't mark primary dentry damaged if inode has been repaired
mds: upgrade snaprealm format during scrub
mds: allow scrubbing mdsdir
mds: cleanup scrub code
mds: show health warning if multimds with old format snapshots
mds: automaticly allow multi-active MDS after removing all old snapshots
mds: disallow multi-active MDS if snapshot was ever created by pre-mimic mds
mds: validate SnapInfo::long_name before using it
mds: don't bump snaptable last_snap when renaming snapshot
mds: properly save snaptable after upgrading version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add test for snap format upgrade

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21677 from Devp00l/wip-convert-floating-values

mgr/dashboard: Convert floating values to bytes

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21780 from votdev/use_bootstrap_css

mgr/dashboard: Use Bootstrap CSS

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/influx: Only split string on first occurence of dot (.)

Service names are not always osd.X or mon.X, they might be
rgw.radosgw.rgw1

This would lead to:

  Unhandled exception from module 'influx' while running on mgr.mon01: too many values to unpack

Only split on the first dot as the rest is the service name

Fixes: http://tracker.ceph.com/issues/23996
Signed-off-by: Wido den Hollander <wido@42on.com>

osd/PG: fix uninit read in Incomplete::react(AdvMap&)

If a PG is incomplete when the pool is deleted we'll dereference invalid
iterators here.

Fixes: http://tracker.ceph.com/issues/23980
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21684 from liewegas/wip-23769

mon: fix slow op warning on mon, improve slow op warnings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21694 from sebastian-philipp/dashboard-review-pylint

mgr/dashboard: Clean up Pylint warnings

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21774 from sebastian-philipp/dashboard-mypy-fixes

mgr/dashboard: fix two type errors found by mypy

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21787 from tspmelo/fix-checkbox-fork-awesome

mgr/dashboard: Change font-family of checkbox

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge PR #21789 into master

* refs/pull/21789/head:
qa: fix blacklisted check for test_lifecycle

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix blacklisted check for test_lifecycle

Caused by: 36f89c5acf0385f13be2f39bc8d30621f4bf0115

Fixes: http://tracker.ceph.com/issues/23975
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21791 from cbodley/wip-23974

rgw: raise log level on coroutine shutdown errors

rgw: raise log level on coroutine shutdown errors

Fixes: http://tracker.ceph.com/issues/23974
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mon/MDSMonitor: no_reply on MMDSLoadTargets

If we don't note that we don't reply then we don't close out the routed
mon request and the op will appear as slow on the forwarding mon.

Fixes: http://tracker.ceph.com/issues/23769
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Change font-family of checkbox

awesome-bootstrap-checkbox was still trying to use Font Awesome,
this will force it to use Fork Awesome.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21763 from dillaman/wip-23955

librbd: prevent watcher from unregistering with in-flight actions

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #21065 into master

* refs/pull/21065/head:
qa/cephfs: test if evicted client unmounts without hanging
qa/tasks: allow custom timeout for umount_wait()
client: don't hang when MDS sessions are evicted

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21768 from zy751713126/pybind-note

pybind: add return note in rbd.pyx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21701 from Devp00l/wip-kv-table-render-object

mgr/dashboard: Enable object rendering in KV-table

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21781 from LenzGr/mgr-dashboard-documentation

doc: Update dashboard feature list (added RGW management)

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #21762 into master

* refs/pull/21762/head:
mds: mark new root inode dirty

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21652 into master

* refs/pull/21652/head:
client: use common interp of st_nlink for dirs

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21715 into master

* refs/pull/21715/head:
mds: don't report slow request for blocked filelock request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21010 into master

* refs/pull/21010/head:
common: refactor for array size
common: Use ARRAY_SIZE macro
common: compress return logic
common: Remove multiple assignment
common: Remove unnecessary semicolon
common: addr_parsing: Cleanup and Refactor Code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Use Bootstrap CSS

Replace custom CSS class by Bootstrap.

Signed-off-by: Volker Theile <vtheile@suse.com>

doc: Update dashboard feature list (RGW management)

Updated feature list in the dashboard documentation to also
mention the RGW user/bucket management capabilities.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21351 from votdev/dashboard_rgw_management

mgr/dashboard: Add RGW user and bucket management features

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Enable object rendering in KV-table

Now it's possible to render objects as values in our key-value table.

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: fix two type errors found by mypy

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Refactored formatter service

Added tests which caused a refactorization of the service.

Signed-off-by: Stephan Müller <smueller@suse.com>

osd: fix bug which cause can't erase OSDShardPGSlot.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/dashboard: Convert floating values to bytes

Now the 'formatter' service can convert floating values to bytes.

Signed-off-by: Stephan Müller <smueller@suse.com>

pybind: add return note in rbd.pyx

Signed-off-by: Zheng Yin <zhengyin@huayun.com>

Merge pull request #21756 from liewegas/wip-23949

mon/OSDMonitor: set FLAG_SELFMANAGED_SNAPS on cephfs snap removal

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21757 from liewegas/wip-osdc-warning

osdc/Objecter: fix warning

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: mark new root inode dirty

Fixes: http://tracker.ceph.com/issues/23960
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21696 from jan--f/prometheus-file_sd-consider-defaultport

pybing/mgr/prometheus: return default port if config-key get returns …

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #21749 from b-ranto/wip-expose-time-perfcount

prometheus: Handle the TIME perf counter type metrics

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #21627 from jcsp/wip-mgr-dashboard-ssl

mgr/dashboard: add TLS

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

librbd: prevent watcher from unregistering with in-flight actions

Fixes: http://tracker.ceph.com/issues/23955
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc/Objecter: fix warning

Fallout from 891f5192427a4a783d5d7194fc2556dfdc1a0ed2

Signed-off-by: Sage Weil <sage@redhat.com>

mon: log SLOW_OPS locally

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: dump mgr health checks to debug log

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: print daemon_health_metrics to debug log

Signed-off-by: Sage Weil <sage@redhat.com>

osd: print desc for oldest slow op to osd log

Signed-off-by: Sage Weil <sage@redhat.com>

osd: log 'slow op' debug messages for individual slow ops

Otherwise it is very hard to identify which OSD ops are slow when we've
seen a SLOW_OPS health warning in a qa run.

Notably, without this, bugs like http://tracker.ceph.com/issues/23769
are very challenging to track down.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set FLAG_SELFMANAGED_SNAPS on cephfs snap removal

CephFS uses a different path to remove selfmanaged snaps than librados,
so while the librados path goes through pg_pool_t::remove_unmanaged_snap(),
we open code the snap addition to the pool's removed_snaps here.  If we
don't set FLAG_SELFMANAGED_SNAPS at that time, we will implicitly set it
during decode and get a CRC mismatch.

Fix by explicitly setting FLAG_SELFMANAGED_SNAPS flag here.

Fixes: http://tracker.ceph.com/issues/23949
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: dump osdmaps if crc doesn't match

Dump both the json and hexdump at debug level 20.

Hunting http://tracker.ceph.com/issues/23949

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21742 from liewegas/wip-23940

osdc/Objecter: fix recursive locking in _finish_command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21745 from liewegas/wip-pg-removal-race

osd: fix _process handling for pg vs slot race

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20678 from ceph/wip-s3a-fix

fix s3atests that are failing for sometime

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20894 from ZVampirEM77/wip-multisite-cleanup

rgw: some cleanup for sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21647 from yehudasa/wip-23859

rgw: fix for issue #21647

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21648 from yehudasa/wip-cloud-sync-7

rgw: cloud sync fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21678 from idiv-biodiversity/wip-doc-scrub_load_threshold

doc: fix error in osd scrub load threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade/stress-split/4-workloads/radosbench: use less disk space

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21727 from trociny/wip-23929

librbd: release lock executing deep copy progress callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: don't report slow request for blocked filelock request

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/22428

Merge PR #21719 into master

* refs/pull/21719/head:
mds: trim log during shutdown to clean metadata

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21720 into master

* refs/pull/21720/head:
mds: kick rdlock if waiting for dirfragtreelock

Reviewed-by: Zheng Yan <zyan@redhat.com>

prometheus: Handle the TIME perf counter type metrics

This patch correctly sets the PERFCOUNTER_MASK to 3 so that the
PERFCOUNTER_TIME metrics are not ignored by the mgr_module code. It also
converts the TIME metrics from nanoseconds to seconds just like the ceph
perf dump does and exposes the metrics via prometheus module.

Signed-off-by: Boris Ranto <branto@redhat.com>

common: refactor for array size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: fix _process handling for pg vs slot race

We could see the slot with a different PG than we expected if the old
PG was removed and a new one was instantiated in its place.  We can't
just pick up the new PG pointer, however, since it isn't locked.

Fix by retrying with the slot's new pg (possibly null!).  Move this check
below the other cases so that we know we are otherwise consistent with
the slot, since the next pass around we might get pg==null and skip the
to_process.empty() and requeue_seq checks entirely.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: release lock executing deep copy progress callback

Fixes: http://tracker.ceph.com/issues/23929
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21706 from liewegas/wip-23860

osd/PG: fix DeferRecovery vs AllReplicasRecovered race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: kick rdlock if waiting for dirfragtreelock

Fixes: https://tracker.ceph.com/issues/23919
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>