mds: add perf counter for finisher of MDSRank
        in all flash cluster, there are a lot of reqs in the queue.

Signed-off-by: simon gao <simon29rock@gmail.com>

ceph-mds: dump all info of ceph_file_layout, InodeStoreBase, frag_info_t, inode_t

Signed-off-by: simon gao <simon29rock@gmail.com>

mds: avoid sending too many osd requests at once after mds restarts
Fixes: http://tracker.ceph.com/issues/40028
Signed-off-by: simon gao <simon29rock@gmail.com>

Merge pull request #27407 from tchaikov/wip-rpm-python3-el7

rpm: use python3.4 on RHEL7 by default

Merge pull request #27403 from cyberang3l/many_objects_per_pg_docs

doc: update documentation for the MANY_OBJECTS_PER_PG warning

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27409 from cbodley/wip-vstart-rgw-debug

vstart: only add --debug-ms=1 in RGWDEBUG

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

vstart: only add --debug-ms=1 in RGWDEBUG

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: should PYTHON3_VERSION_STRING of libpython3

and make sure the version matches exactly the requested one

in future, we should use FindPython.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Update documentation for the MANY_OBJECTS_PER_PG warning

The current documentation for the MANY_OBJECTS_PER_PG warning
states that The threshold can be raised to silence the health
warning by adjusting the mon_pg_warn_max_object_skew config
option on the monitors. It seems that this is not true (at least)
since the luminous times, and this option should be adjusted on
the managers.

I encountered this problem and I spend quite sometime injecting
the mon_pg_warn_max_object_skew to the monitors, added the option
ceph.conf and restarted the monitors several times but the warning
was not going away. I had to download the code to see what's
happening and I found out this:

$ git grep -A 3 mon_pg_warn_max_object_skew src/common/options.cc
src/common/options.cc:1480:    Option("mon_pg_warn_max_object_skew", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
src/common/options.cc-1481-    .set_default(10.0)
src/common/options.cc-1482-    .set_description("max skew few average in objects per pg")
src/common/options.cc-1483-    .add_service("mgr"),

After I restarted the ceph-mgr service, the warning went away.

Signed-off-by: Vangelis Tasoulas <vangelis@tasoulas.net>

Merge PR #27396 into master

* refs/pull/27396/head:
doc: fixed typo in leadership names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27397 into master

* refs/pull/27397/head:
doc: fixed caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rpm: use python3.4 on RHEL7 by default

python3.4 is the native python3 before 7.6

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27400 from cbodley/wip-39118

rgw: limit entries in remove_olh_pending_entries()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #26874 into master

* refs/pull/26874/head:
OSD: OSDMapRef access by multiple threads is unsafe

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27132 into master

* refs/pull/27132/head:
os/bluestore: new bluestore_debug_enforce_settings option.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27317 into master

* refs/pull/27317/head:
kv: make delete range optional on number of keys

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27327 into master

* refs/pull/27327/head:
mon/MonmapMonitor: clean up empty created stamp in monmap
common/buffer: fix warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: limit entries in remove_olh_pending_entries()

If there are too many entries to send in a single osd op, the osd rejects
the request with EINVAL. This error happens in follow_olh(), which means
that requests against the object logical head (requests with no version
id) can't be resolved to the current object version. In multisite, this
also causes data sync to get stuck in retries

Fixes: http://tracker.ceph.com/issues/39118
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: fixed caps

Signed-off-by: Servesha Dudhgaonkar <sdudhgao@redhat.com>

doc: fixed typo in leadership names

Signed-off-by: Servesha Dudhgaonkar <sdudhgao@redhat.com>

Merge pull request #27254 from ricardoasmarques/improve-iscsi-overview

mgr/dashboad: Improve iSCSI overview page

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27292 from sebastian-philipp/kubejacker-simplify

src/script/kubejacker: Fix and simplify

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #27121 from sebastian-philipp/orchestrator-doc-cli-fix-service-add

doc/orchestrator: Fix broken bullet points

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27379 from xenolinux/wip-fix-bug1

doc : fixed capitalization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26710 from cyx1231st/wip-seastar-msgr-protocolv2

crimson/net: lossy connection for ProtocolV2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27383 from tchaikov/wip-cmake-libpython3

cmake: check for MAJOR.MINOR version of python3

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/net: more compact code with then()

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: clean auth related interfaces

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: scoped enum for proto_type

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: utilize lowres_system_clock with better performance

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: set policy from peer_type

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: fix Socket::read_exactly(0) case

Do not raise read_eof when try to read 0 bytes.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: fix perf_crimson_msgr to support v2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: msgr should wait before shutdown

test_concurrent_dispatch() should wait for server->wait() before
shutting down its messengers.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: unit test for v2 protocol

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: message frame exchanges (lossy part)

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add keepalive timestamps to connection

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: protocolv2 handshake frame exchanges

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: ProtocolV2 auth part

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: dummy auth for protocol v2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add AuthConnectionMeta to Protocol

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add peer_global_id to Connection

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: banner exchange and HelloFrame

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: support frame read/write for ProtocolV2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add target_addr to SocketConnection

represent the peer target_addr accepted/connected

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add get_mytype() to Messenger

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: add global_seq in ProtocolV2

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: skeleton code for ProtocolV2 logic

crimson ProtocolV2 class is following a state-machine design style:
* states are defined in ProtocolV2::state_t;
* call `execute_<state_name>()` methods to trigger different states;
* V2 logics are implemented in each execute_<state_name>() methods, and
  with explicit transitions to other states at the end of the execute_*;
* each state is associated with a write state defined in Protocol.h:
  - none: not allowed to send;
  - delay: messages can be queued, but will be delayed to send;
  - open: dispatch queued message/keepalive/ack;
  - drop: not send any messages, drop them all.

crimson ProtocolV2 is alike async ProtocolV2, with some considerations:
* explicit and encapsulated client/server handshake workflow.
* futurized-exception-based fault handling, which can interrupt protocol
  workflow at any time in each state.
* introduced SERVER_WAIT state, meaning to wait for peer-client's socket
  to reset or be replaced, and expect no further reads.
* introduced an explicit REPLACING state, async-msgr would be at the
  NONE state during replacing.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: apple-to-apple server async-msgr

The simplest async-msgr server which will have the same behavior with
crimson-msgr server for apple-to-apple performance test.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: improved perf tool for crimson-msgr

New features:
* --jobs: start multiple client messengers from core #1 ~ #jobs;
* --core: can assign server core to get away from busy client cores;
* --rounds: a client will send <rounds>/<jobs> messages;

Improved:
* Better configuration report;
* Report individual client results plus a summary;
* Validate if CPU number is sufficient before running;
* Sleep 1 second while connecting, so it won't hurt performance;
* Simplify client logic and bug fixes;

Removed unecessary features:
* finish_decode() for MOSDOp;
* keepalive ratio;

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: fix messenger creation

sharded data structures should only be allocated in core#0, or the
program will hang during exit.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: create SocketConnection based on address type

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: use entity_addrvec_t::front() inside messenger

Do not expect legacy-addr in order to bind v2 address.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: added msgr2 setting for SocketConnection

Provide entrypoint for the future ProtocolV2 class.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: implement factory method of Socket

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: introduce SocketFRef

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

crimson/net: introduce protocol-level abstraction

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

rpm: install python%{python3_pkgversion}-devel on el7

as RHEL 7.7 support a native python3.6, we cannot hardwire to python3.4
on RHEL7/CentOS7 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for MAJOR.MINOR version of python3

We can only check for MAJOR.MINOR version of python3 since
FindPython3Libs does not support checking for MAJOR.MINOR.PATCH version
of python3. We also need to make sure we use the PYTHON3 versions of
these variables.

This should fix a regression introduced by c961e00.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge PR #27266 into master

* refs/pull/27266/head:
cephfs-shell: Fix hidden files and directories list by ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27380 from ricardoasmarques/fix-iscsi-target-submission

mgr/dashboard: Fix iSCSI target submission

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #27198 into master

* refs/pull/27198/head:
vstart: generate environment script suitable for sourcing
cephfs-shell: fix mode of script in tree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fix iSCSI target submission

cdIsInGroup should not be included in the request
and username/password should be an empty string
(instead of null)

Fixes: https://tracker.ceph.com/issues/39110
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #27373 into master

* refs/pull/27373/head:
doc: Add usage for shortcuts command in cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27303 into master

* refs/pull/27303/head:
mds: add Octopus cephfs feature bit
mds: verify current release has a cephfs feature bit
mds: add cephfs feature bit for Nautilus
mon: generalize and refactor lambda use
qa: test featureful client with mimic base
qa: remove obsolete snap upgrade tests

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #27238 into master

* refs/pull/27238/head:
mds: update projected_version when upgrading snaptable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26881 into master

* refs/pull/26881/head:
mds: remove SimpleLock::replicate_relax()
mds: avoid check session connection's features when issuing caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Log `gateway_name` on `iscsi_client`

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Display number of sessions on iSCSI overview page

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Adds 'tcmu-runner' to the 'user:rbd' backstore descr

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Disable backstore dropdown

iSCSI backstore is always visible but is disabled
if only one option is available.

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Improve iSCSI overview page

iSCSI overview page will now use information
obtained from ceph-iscsi.

Fixes: https://tracker.ceph.com/issues/39024
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27358 from tchaikov/wip-multi-python3

rpm,cmake: use specified python3 version if any

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27362 from ektich/ektich-patch-1

doc/rados/configuration: fix typo in mon-lookup-dns

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27338 from tchaikov/wip-crimson-wait-for-active

crimson/osd: lower debug level on i/o path

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27378 from ricardoasmarques/fix-iscsi-disk-diff

mgr/dashboard: Fix iSCSI disk diff calculation

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27262 from ricardoasmarques/check-gateway-in-use

mgr/dashboard: Check if gateway is in use before deletion

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27248 from ricardoasmarques/display-target-sessions

mgr/dashboard: Display the number of iSCSI active sessions

Reviewed-by: Ricardo Dias <rdias@suse.com>

doc : fixed capitalization

Signed-off-by: Servesha Dudhgaonkar <sdudhgao@redhat.com>

mgr/dashboard: Fix iSCSI disk diff calculation

The separator is now a '/' instead of '.'

Fixes: https://tracker.ceph.com/issues/39109
Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc/rados/configuration/mon-lookup-dns: fix typo

Signed-off-by: Vanush "Misha" Paturyan <ektich@gmail.com>

doc: Add usage for shortcuts command in cephfs-shell

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #27350 from tspmelo/wip-fix-nfs-transport

mgr/dashboard: Fix deletion of NFS transports properties

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27326 from dillaman/wip-librados-unitest-logging

test/librados_test_stub: ensure the log flusher thread is started

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26016 from falcon78921/wip-doc-37957

doc: added section on creating RESTful API user

Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm: pass %{python3_version} to cmake

to force cmake to use the python3 and python3 modules for building
python3 bindings

on the debian side, it's okay to continue using "-DWITH_PYTHON3=ON", as

- cmake does normalize "ON" to 3
- debian's cmake extension lives on /usr/lib/python3/dist-packages/
  not in a specific /usr/lib/python3.x/dist-packages directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for libpython of the same version of interpreter

actually cython or python3-cython's dependencies should take care of it.
but we should at least get it right on our side if we check it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use specified python3 version if any

use might have multiple python3 installed, some of them has/have all
dependencies installed and is good enough for building Ceph. we should
not always use the latest python installed in the system and complain that
there is missing dependencies, even if user has installed all the
python3 dependencies for the older python3.

put in other words, if user only installs cython module for python3.4, but
she has both python3.6 and python3.4 in her system. we should not force
her to uninstall python3.6 for installing Ceph.

this change also aligns with MGR_PYTHON_VERSION. i am not applying the
same change to WITH_PYTHON2, because python2 is already stablized. and distros
are not likely to release new python2 releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: add Octopus cephfs feature bit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: verify current release has a cephfs feature bit

Fixes: http://tracker.ceph.com/issues/39077
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add cephfs feature bit for Nautilus

Fixes: http://tracker.ceph.com/issues/39078
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon: generalize and refactor lambda use

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test featureful client with mimic base

Fixes: http://tracker.ceph.com/issues/39020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove obsolete snap upgrade tests

Direct upgrades from Luminous to Octopus are not supported. These snap format
upgrade tests are now only going to be run in the mimic/nautilus branches.

Fixes: http://tracker.ceph.com/issues/39020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27353 into master

* refs/pull/27353/head:
include/ceph_features: fix typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #27293 from tspmelo/wip-nfsv4

mgr/dashboard: Fix NFS pseudo validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26439 from taoCH/wip-fix-librgw-empty-etag

rgw_file: save etag and acl info in setattr

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

include/ceph_features: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27141 into master

* refs/pull/27141/head:
mon/OSDMonitor: fix osd boot feature vs require_osd_release check
include/ceph_features: retire 7 other old features
include/ceph_features: retire ERASURE_CODE_PLUGINS_V2
include/ceph_features: retire OSD_ERASURE_CODES
include/ceph_features: update comment to align with N+2 upgrades
include/ceph_features: adjust whitespace for retired and now usable features
mon: remove check for jewel mons
mds/FSMap: remove support for encoding jewel FSMap
include/ceph_features: enable SERVER_OCTOPUS
test/cli/osdmaptool/feature-set-unset-list: add octopus to output
test/cli/osdmaptool/feature-set-unset-list: change unknown feature bit
qa/releases/octopus.yaml: add octopus upgrade final step
osd/OSDMap: octopus encoding features
mon/OSDMonitor: add mon_debug_no_require_octopus
mon/OSDMonitor: allow 'osd require-osd-release octopus'
mon: add ondisk incompat octopus feature
mon/mon_types: add mon feature for octopus
include/ceph_features: SERVER_O -> SERVER_OCTOPUS

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Fix NFS pseudo validation

Fixes: http://tracker.ceph.com/issues/39063
Signed-off-by: Tiago Melo <tmelo@suse.com>