doc: Fix typo in ceph-fuse

Fixes: https://github.com/ceph/ceph/pull/21616#pullrequestreview-122923127
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #21845 from tchaikov/wip-gcc-warnings

mds,messages: silence -Wclass-memaccess warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #21856 from tchaikov/wip-drop-dpdk-submodule

dpdk: drop dpdk submodule

Reviewed-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge PR #21712 into master

* refs/pull/21712/head:
qa/tasks/cephfs: add test for renewing stale session
client: invalidate caps and leases when session becomes stale
client: fix race in concurrent readdir

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21688 into master

* refs/pull/21688/head:
client: change vxattr flags field to unsigned int

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21668 into master

* refs/pull/21668/head:
mds: include nfiles/nsubdirs of directory inode in MClientCaps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21858 from yuriw/wip-yuriw-crontab

qa/tests: fixed typo

Merge pull request #21707 from cbodley/wip-qa-upgrade-ragweed

qa/rgw: move ragweed upgrade test into upgrade/luminous-x

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21794 from rjfd/wip-dashboard-fix-awsauth

mgr/dashboard: awsauth: fix python3 string decode problem

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Lenz Grimmer <LGrimmer@suse.com>

dpdk: drop dpdk submodule

we are using spdk/dpdk for async msgr's dpdk backend since 01a9f178. so
no need to keep dpdk submodule anymore.

Fixes: http://tracker.ceph.com/issues/24032
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: fixed typo

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21784 from votdev/fix_rgw_backend

mgr/dashboard: Refactor RGW backend

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #21839 from yuriw/wip-yuriw-crontab

qa/test: Added rados, rbd and fs to run two time a week only

mgr/dashboard: Refactor RGW backend

- Do some polishing in the docs.
- Refactor RgwClient::is_service_online() method. The system is considered as online if the response structure is valid. The response content itself is not validated in this case.
- Relocate NoCredentialsException and derive it from RequestException.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #21673 from votdev/refactor_perf_counters

mgr/dashboard: Refactor perf counters

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21582 from jan--f/test-case-strtoll-hex

test/strtol: add test case for parsing hex numbers

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: silence -Wformat-truncation= warnings

this change silences following warnings:

: warning: ‘%s’ directive output may be truncated writing up to 127
bytes into a region of size between 109 and 119 [-W
format-truncation=]
     snprintf(m_id_str, SysTestRunnable::ID_STR_SZ, "process_%d%s",
m_id, extra);
                                                    ^~~~~~~~~~~~~~
~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: silence -Wparentheses warning

this change silences following warning:

warning: unnecessary parentheses in declaration of ‘os’ [-Wparentheses]
   std::ostringstream(os);
                     ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds,messages: silence -Wclass-memaccess warnings

this change silences warnings like:

warning: ‘void* memset(void*, int, size_t)’ writing to an object of
non-trivial type ‘struct uuid_d’; use assignment instead [-Wcla
ss-memaccess]
   memset(&z.cluster_fsid, 1, sizeof(z.cluster_fsid));
                                                    ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21748 from b-ranto/wip-fix-prom-shutdown

prometheus: Fix prometheus shutdown/restart

Reveiwed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21793 from b-ranto/wip-expose-objects

prometheus: Expose number of degraded/misplaced/unfound objects

Reveiwed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21837 from tchaikov/wip-gcc-8

cmake,common,filestore: silence gcc-8 warnings/errors

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/test: Added rados, rbd and fs to run two time a week only

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21827 from yuriw/wip-yuriw-crontab

qa/tests: Added mimic runs, removed large suites (rados, rbd, etc) ru…

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/filestore: silence -Wformat-truncation= warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: silence error from libstdc++ in gcc-8

this change silences following error

stl_tree.h:452:21: error: static assertion failed: comparison object
must be invocable with two arguments of key type
static_assert(__is_invocable<_Compare&, const _Key&, const _Key&>{},
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages,mon,osd: silence gcc-8 warnings related to memset()

this silences warnings like:

warning: ‘void* memset(void*, int, size_t)’ writing to an object of
non-trivial type ‘struct uuid_d’; use assignment instead [-Wcl\
ass-memaccess]

uuid_d only contains boost::uuids::uuid, which is "nil" initialized in
uuid_d's ctor. so we don't need to bother with memset() it with 0.
the same applies to entity_inst_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,filestore: silence GCC-8 warnings

should catch polymorphic exceptions by reference. this silences warnings
like:

error: catching polymorphic type ‘class std::RetryException’ by value
[-Werror=catch-value=]
     } catch (std::RetryException) {

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: specify missing settings for dpdk

* after upgrading to gcc-8, seems dpdk requires more settings to compile.
  in which, CONFIG_RTE_MAX_VFIO_GROUPS=64 is copied from config/common_base.
* librte_bus_pci.a depends on librte_pci.a, so reorder the libraries
  in DPDK_LIBRARIES.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21637 from ceph/wip-da-SCA-20180423

misc: mark constructors as explicit

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19041 from dragonylffly/wip-list-objects-in-a-pg

tools/rados: allow list objects in a specific pg in a pool

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21765 from yunfeiguan/wip-23962

pybind/ceph_daemon: expand the order of magnitude of daemonperf statistics to ZB

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21815 from xiexingguo/wip-23921

crush, osd: handle multiple parents properly when applying pg upmaps

Reviewed-by: huangjun <hjwsm1989@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21811 from tchaikov/wip-23627

librados: block MgrClient::start_command until mgrmap

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21790 from dalgaaf/SCA-override-specifier

misc: mark functions with 'override' specifier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21829 from tchaikov/wip-qa-cephfs-snap

qa: fix test on "ceph fs set cephfs allow_new_snaps"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: fix test on "ceph fs set cephfs allow_new_snaps"

snapshot is now enabled for new fs in 81d0554e, so this test should be
updated accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: Added mimic runs, removed large suites (rados, rbd, etc) runs on luminous

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21816 from xiexingguo/wip-wrong-unregister-pg

osd/OSD: drop extra/wrong *unregister_pg*

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21822 from tchaikov/wip-24012

cmake: use javac -h for creating JNI native headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: bump up JDK version to 1.8

as JDK 7 is EOL. see
http://www.oracle.com/technetwork/java/eol-135779.html and
https://www.java.com/en/download/faq/java_7.xml

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use javac -h for creating JNI native headers

JDK 1.10 does not offer javah anymore, so we need to use "javac -h" or
add_jar(... GENERATE_NATIVE_HEADERS) instead.

Fixes: http://tracker.ceph.com/issues/24012
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'gh/mimic'

osd/OSD: drop extra/wrong *unregister_pg*

Signed-off-by: xiexingguo <xie.xingguo@gmail.com>

mgr: Include daemon details in SLOW_OPS output

Currently there is no way to see which daemons were involved in a slow
op after the op has cleared. This change allows us to record which
daemons were implicated in the logs.

Partially fixes: http://tracker.ceph.com/issues/23205

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit b5263176dea6d204c2caa52043cefc94dd5e475f)

crush, osd: handle multiple parents properly when applying pg upmaps

Fixes: http://tracker.ceph.com/issues/23921
Signed-off-by: xiexingguo <xie.xingguo@gmail.com>

Merge pull request #21750 from badone/wip-summarise-osds-when-issuing-slow-request-warning-post-luminous

mgr: Include daemon details in SLOW_OPS output

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

librados: timeout on mgr_command()

because the mgrclient will be waiting for the mgrmap if the mgrmap
is not available, when the client is about to send a mgr command.
and monitor will drop the subscription requests if the client does not
have enough cap for reading mon, so unlike mon commands, the client
won't get an EACCES return code in this case.

in this change, a timeout machinary is introduced. and the client
will wait for "rados-mon-op-timeout" before it gives up. if this
setting is 0, it will wait forever.

Fixes: https://tracker.ceph.com/issues/23627
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21520 from jan--f/doc-fix-pg-state-names

doc/rados/operations/pg-states: fix PG state names

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21557 from jan--f/mgr-prometheus-fix-metadata-labels

mgr: prometheus fix metadata labels

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21797 from tomkukral/remove-cz-mirror

doc: remove CZ mirror

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: remove CZ mirror

CZ mirror is no longer running because company sponsoring this mirror
isn't able to provide it anymore.

I'm removing this mirror from list of official mirrors.

Signed-off-by: Tomáš Kukrál <tom@6shore.net>

librados: fix locking on get_required_monitor_features

This wasn't taking the MonClient lock: should use
with_monmap to protect access to MonClient::monmap.

Signed-off-by: John Spray <john.spray@redhat.com>

librados: config mgrclient for pre-luminous cluster

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/MgrClient: add mgr_optional mode

This is for use when talking to pre-luminous
clusters, where we should not block waiting
for MgrMap because it might never come.

Fixes: https://tracker.ceph.com/issues/23627
Signed-off-by: John Spray <john.spray@redhat.com>

doc/rados/operations/pg-states: fix PG state names

Change pg state names according to osd/osd_types.cc as this is what
ceph -s and the prometheus exporter will present to users.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21461 from yaozongyou/calculate-stddev-on-the-fly

common: calculate stddev on the fly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21805 into master

* refs/pull/21805/head:
qa: move snap-hierarchy out of snaps workunits
qa: try snap format upgrade with multimds cluster
qa: add mds deactivation procedure for upgrades

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21807 from smithfarm/wip-python-encoding

script: build-integration-branch: avoid Unicode error

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19714 from qwren/qatzip-compressor

compressor: add QAT support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21242 from majianpeng/objecter-calc-budget

osdc: Change 'bool budgeted' to 'int budget' to avoid recalculating.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21487 from yanghonggang/kstore-statistic

os/kstore: support db statistic

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21299 from yanghonggang/pool-create

librados: Reject the invalid pool create request at client side, rath…

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #21759 into master

* refs/pull/21759/head:
script: Fix ceph-debug-docker.sh script create --tag in correct format

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move snap-hierarchy out of snaps workunits

The snapshot hierarchy it leaves behind can't be cleaned up by `rm -rf` which
breaks workunit cleanup. So, don't run this as part of normal snaps test.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: try snap format upgrade with multimds cluster

Fixes: http://tracker.ceph.com/issues/24002
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add mds deactivation procedure for upgrades

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21553 from dragonylffly/wip-improve-rados-ls

tools/rados: improve the ls command usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21795 from wido/influx-string-split

mgr/influx: Only split string on first occurence of dot (.)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21798 from liewegas/wip-23980

osd/PG: fix uninit read in Incomplete::react(AdvMap&)

Reviewed-by: Kefu Chai <kchai@redhat.com>

script: build-integration-branch: avoid Unicode error

When run on PRs that have non-ASCII characters in their titles, the script
fails like this when run in a non UTF-8 environment:

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2026' in
position 651: ordinal not in range(256)

Since failing like that is not useful, avoid the error by forcing ASCII
encoding and filtering out any non-ASCII characters.

Fixes: http://tracker.ceph.com/issues/24003
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21766 from majianpeng/osd-remove-useless-conditon

osd: Remove useless conditon.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: huangjun <hjwsm1989@gmail.com>

Merge pull request #21771 from majianpeng/osd-remove-pgslot

osd: fix bug which cause can't erase OSDShardPGSlot.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: add ragweed to luminous-x upgrade suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: remove rgw/upgrade suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21546 from dzafman/wip-22881

osd: process _scan_snaps() with all snapshots with head

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #21374 into master

* refs/pull/21374/head:
qa: add test for snap format upgrade
mds: initialize SnapServer::snaprealm_v2_since after journal replay
mds: properly distinguish cap update from snap flush
mds: update dev document of cephfs snapshot
doc: add release notes for cephfs snapshot
mds: allow snapshot by default for new filesystem
mds: close past parents after snaprealm format gets converted
mds: automaticly allow multi-active MDS after scrubbing all inodes
mds: don't mark primary dentry damaged if inode has been repaired
mds: upgrade snaprealm format during scrub
mds: allow scrubbing mdsdir
mds: cleanup scrub code
mds: show health warning if multimds with old format snapshots
mds: automaticly allow multi-active MDS after removing all old snapshots
mds: disallow multi-active MDS if snapshot was ever created by pre-mimic mds
mds: validate SnapInfo::long_name before using it
mds: don't bump snaptable last_snap when renaming snapshot
mds: properly save snaptable after upgrading version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add test for snap format upgrade

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21677 from Devp00l/wip-convert-floating-values

mgr/dashboard: Convert floating values to bytes

Reviewed-by: Volker Theile <vtheile@suse.com>

prometheus: Expose number of degraded/misplaced/unfound objects

These come from the pg dump by polling the pg_stas_sum.

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr: Expose pg_sum in pg_summary

Signed-off-by: Boris Ranto <branto@redhat.com>

tools/rados: allow list objects in a pg

When the cluster is not healthy, normally only a very few pgs encounter
problems, and the operator wants to know which objects may potentially
get lost.

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #21780 from votdev/use_bootstrap_css

mgr/dashboard: Use Bootstrap CSS

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/influx: Only split string on first occurence of dot (.)

Service names are not always osd.X or mon.X, they might be
rgw.radosgw.rgw1

This would lead to:

  Unhandled exception from module 'influx' while running on mgr.mon01: too many values to unpack

Only split on the first dot as the rest is the service name

Fixes: http://tracker.ceph.com/issues/23996
Signed-off-by: Wido den Hollander <wido@42on.com>

osd/PG: fix uninit read in Incomplete::react(AdvMap&)

If a PG is incomplete when the pool is deleted we'll dereference invalid
iterators here.

Fixes: http://tracker.ceph.com/issues/23980
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21684 from liewegas/wip-23769

mon: fix slow op warning on mon, improve slow op warnings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21694 from sebastian-philipp/dashboard-review-pylint

mgr/dashboard: Clean up Pylint warnings

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21774 from sebastian-philipp/dashboard-mypy-fixes

mgr/dashboard: fix two type errors found by mypy

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21787 from tspmelo/fix-checkbox-fork-awesome

mgr/dashboard: Change font-family of checkbox

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: awsauth: fix python3 string decode problem

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr: Include daemon details in SLOW_OPS output

Currently there is no way to see which daemons were involved in a slow
op after the op has cleared. This change allows us to record which
daemons were implicated in the logs.

Partially fixes: http://tracker.ceph.com/issues/23205

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mds: include nfiles/nsubdirs of directory inode in MClientCaps

Directory inode's dirstat gets updated by request reply, but not by
cap message. This causes problem for following case.

1. MDS modifies a directory
2. MDS issues CEPH_CAP_ANY_SHARED to client
3. The client satifies stat(2) by its cached metadata.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/23855

qa/tasks/cephfs: add test for renewing stale session

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

client: invalidate caps and leases when session becomes stale

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/23894

client: fix race in concurrent readdir

For a large directory, program needs to issue multiple readdir
syscalls to get all dentries. When there are multiple programs
read the directory concurrently. Following sequence of events
can happen.

 - program calls readdir with pos = 2. ceph sends readdir request
   to mds. The reply contains N1 entries. ceph adds these N1 entries
   to readdir cache.
 - program calls readdir with pos = N1+2. The readdir is satisfied
   by the readdir cache, N2 entries are returned. (Other program
   calls readdir in the middle, which fills the cache)
 - program calls readdir with pos = N1+N2+2. ceph sends readdir
   request to mds. The reply contains N3 entries and it reaches
   directory end. ceph adds these N3 entries to the readdir cache
   and marks directory complete.

The second readdir call does not update dirp->cache_index. ceph adds
the last N3 entries to wrong places.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/23894

Merge PR #21789 into master

* refs/pull/21789/head:
qa: fix blacklisted check for test_lifecycle

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix blacklisted check for test_lifecycle

Caused by: 36f89c5acf0385f13be2f39bc8d30621f4bf0115

Fixes: http://tracker.ceph.com/issues/23975
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21791 from cbodley/wip-23974

rgw: raise log level on coroutine shutdown errors

rgw: raise log level on coroutine shutdown errors

Fixes: http://tracker.ceph.com/issues/23974
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mon/MDSMonitor: no_reply on MMDSLoadTargets

If we don't note that we don't reply then we don't close out the routed
mon request and the op will appear as slow on the forwarding mon.

Fixes: http://tracker.ceph.com/issues/23769
Signed-off-by: Sage Weil <sage@redhat.com>