Merge PR #23291 into master

* refs/pull/23291/head:
tests: add Cephx-resembling scenarios to crypto tests.
tests: differentiate AES testing regarding text size.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22879 into master

* refs/pull/22879/head:
doc/rados/operations/user-management: document 'network' clause of moncap and osdcap
mds/SessionMap: make Session::connection private
mds/MDSAuthCaps: parse and enforce network restriction
mds/MDSAuthCaps: pass addr to is_capable
mds/SessionMap: track socket_addr for each Session
osd/OSDCap: enforce network restriction
osd/OSDCap: take addr arg to is_capable
osd/OSDCap: parse 'network' clause in grant
mon: adapt MonCap network checks to addrvecs
mon/MonCap: enforce network constraint (if present)
mon/MonCap: take addr for MonCap::is_capable
mon/MonCap: parse 'network ...' suffix
osd: add peer_socket_addr to Session
mon: use addrvec for RoutedRequest and MForward
msg/Connection: add get_peer_socket_addr()
common/ipaddr: add network_contains() helper
common/ipaddr: add parse_network helper
common/ipaddr: expose netmask_ipv[46]

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #23510 from tchaikov/wip-cmake-boost-build

cmake: fix build WITH_SYSTEM_BOOST=ON

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23542 into master

* refs/pull/23542/head:
qa: fix mon config section

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix mon config section

Fixes: fc6e683832125a1042638f21b623e29d0d4c34d8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23444 into master

* refs/pull/23444/head:
cephfs-shell: avoid sys.argv modification
tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados/operations/user-management: document 'network' clause of moncap and osdcap

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: make Session::connection private

Ensure all setters go through set_connection()

Signed-off-by: Sage Weil <sage@redhat.com>

cephfs-shell: avoid sys.argv modification

In multiple locations in the code.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds/MDSAuthCaps: parse and enforce network restriction

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSAuthCaps: pass addr to is_capable

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: track socket_addr for each Session

Make sure socket_addr is populated when connection is set.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: enforce network restriction

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: take addr arg to is_capable

Not enforced yet; just plumbing!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: parse 'network' clause in grant

This mirrors what we do in the MonCap (but, naturally, is a slightly
different since we're using a the spirit parse in a totally different way).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: adapt MonCap network checks to addrvecs

Breaking this out in a separate commit to make the backport a bit easier.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: enforce network constraint (if present)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: take addr for MonCap::is_capable

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: parse 'network ...' suffix

Allow an optional network constraint to any grant.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add peer_socket_addr to Session

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use addrvec for RoutedRequest and MForward

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: add get_peer_socket_addr()

Return the addr that is actually being used for this socket (among the
potentially many in the addrvec).

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: add network_contains() helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: add parse_network helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: expose netmask_ipv[46]

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23066 into master

* refs/pull/23066/head:
include/utime.h: remove improper comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23513 into master

* refs/pull/23513/head:
doc: updated contact email for Ceph pull requests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23280 into master

* refs/pull/23280/head:
cephfs: LeaseStat versioning
cephfs: DirStat versioning
mds: make has_feature const function
cephfs: InodeStat versioning
mds: add new feature bit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23538 into master

* refs/pull/23538/head:
mds: mark beacons as high priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23527 into master

* refs/pull/23527/head:
mds: use fast dispatch to handle MDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23457 into master

* refs/pull/23457/head:
qa: ignore normal slow IO warnings for cephfs

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: ignore normal slow IO warnings for cephfs

Generally the slow warnings we get are just over the threshold. These warnings
are related to deploying multiple Ceph daemons side-by-side. Let's see how we
do with two minutes.

Ignoring the warnings entirely is unsatisfactory as they serve as a useful
canary in the coal mine when you see warnings for ops > some unreasonably large
amount of time.

Fixes: http://tracker.ceph.com/issues/26900
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23416 from ZVampirEM77/wip-mfa-cmd-and-options

radosgw-admin: add mfa related command and options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.

Fixes:http://tracker.ceph.com/issues/26853
Fixes:http://tracker.ceph.com/issues/26855

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge pull request #22563 from ifed01/wip-ifed-bdev-minsize

os/bluestore: fix bluefs extent miscalculations on small slow device

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23294 from noahdesu/timely-mgr-health-updates

mgr: timely health updates between monitor and manager

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23257 from ifed01/wip-ifed-fix-rcoll-race

os/bluestore: fix race between remove_collection and object removals

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23492 from liewegas/wip-mgrc-warning

mgr/MgrClient: make some noise for a user if no mgr daemon is running

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

cephfs: LeaseStat versioning

* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* Dropped LeaseStat::encode().
* The old format is maintained for backward compatibility.
* Created Locker::encode_lease() for encoding and dropped the duplicates.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs: DirStat versioning

* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* Dropped DirStat::encode().
* The old format is maintained for backward compatibility.
* Created CDir::encode_dirstat() for encoding and dropped the duplicates.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #23500 into master

* refs/pull/23500/head:
debian: mark python-ceph-argparse Arch = all
rpm: package cephfs-shell for fedora
tools/cephfs,deb: package cephfs-shell
cmake: install script and egg-info files of cephfs-shell
tools/cephfs: add setup.py for cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23480 into master

* refs/pull/23480/head:
tools/cephfs: fix mkdirs(exist_ok) call
tools/cephfs: cephfs-shell cleanup
tools/cephfs: run cephfs-shell thru autopep8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: mark beacons as high priority

The mons already mark beacon replies as high priority (via default mon message
priority). We should expect that the mons handle our beacons at the same
priority so the MDS doesn't wrongly get marked laggy.

Fixes: http://tracker.ceph.com/issues/26899
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22868 from iliul/modify-log-level

rgw: Fix log level of gc_iterate_entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Fix log level of gc_iterate_entries

Fixes: http://tracker.ceph.com/issues/23801
Signed-off-by: iliul <liul.stone@gmail.com>

mds: use fast dispatch to handle MDSBeacon

Current mds handles MDSBeacon messages through MDSRank::ms_dispatch().
MDSRank::ms_dispatch() locks mds_lock at the very beginning. This means
that long running task (such as processing finished contexts) can delay
MMDSBeacon. Using fast dispatch for MDSBeacon can avoid this issue.

Fixes: http://tracker.ceph.com/issues/23519
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #22838 into master

* refs/pull/22838/head:
kv/RocksDBStore: Handle nullptr if clock cache is chosen.
kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri pool.
src/kv: Initial import of a custom RocksDB cache.
src/rocksdb: switch back to master branch.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22970 into master

* refs/pull/22970/head:
os/bluestore: limit OSD memory usage by tuning the cache size.

Reviewed-by: Sage Weil <sage@redhat.com>

radosgw-admin: add mfa related command and options

command:
+ mfa create
+ mfa list
+ mfa get
+ mfa remove
+ mfa check
+ mfa resync

options:
+ --totp-serial
+ --totp-seed
+ --totp-second
+ --totp-window
+ --totp-pin

Signed-off-by: Enming.Zhang <enming.zhang@umcloud.com>

Merge pull request #22967 from linuxbox2/wip-slo-nosize

rgw: Swift SLO size_bytes member is optional

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23056 from stiopaa1/fixRgwAdminUser

rgw: fix dependencies/target_link_libraries

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23282 from zhangsw/fix-rgw-server-header

rgw: add Http header 'Server' in response headers.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23405 from joke-lee/WIP-RGW-BUCKET-NOTIFY

rgw: add s3 notification sub resources

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23434 from joke-lee/wip-cors-rule-num-limit

rgw: cors rules num limit

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23210 from alfredodeza/wip-bluestore-sizing

doc: document sizing for `block.db`

Reviewed-by: Ben England <bengland@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23511 from excellentkf/WIP-test

common/options: correct the description of rgw_enable_lc_threads option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: make has_feature const function

Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs: InodeStat versioning

* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* The old format is maintained for backward compatibility.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #23496 from bstillwell/crush_doc_fixes

doc: Fix a couple typos and improve diagram formatting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23477 from liewegas/wip-kv-merge-name

kv/KeyValueDB: return const char* from MergeOperator::name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23474 from tchaikov/wip-config-expand-meta

common: expand meta in parse_argv()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23481 from Yan-waller/wip-walle-180808-smooth-intervals

mgr,mon: fix to apply changed mon_stat_smooth_intervals

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23482 from Yan-waller/wip-walle-180808-mgr-deadlock

mgr: fix beacon interruption caused by deadlock

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23487 from xiexingguo/wip-hb-peers-v2

osd/OSD: choose heartbeat peers more carefully

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

kv/RocksDBStore: Handle nullptr if clock cache is chosen.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri pool.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

src/kv: Initial import of a custom RocksDB cache.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #23514 from bstillwell/operations_doc_fixes

doc: Multiple spelling fixes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: Multiple spelling fixes

I ran a lot of the docs through aspell and found a number of spelling problems.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

os/bluestore: limit OSD memory usage by tuning the cache size.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #23471 into master

* refs/pull/23471/head:
mon/PGMap: fix spacing around pretty-printed SI units
include/types: render SI units adjacent to number

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #23473 into master

* refs/pull/23473/head:
OSD: translate the error result

Reviewed-by: Sage Weil <sage@redhat.com>

doc: updated contact email for Ceph pull requests

Updated the contact email address for Ceph pull requests (per @johnwilkins' request).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: Fix a couple typos and improve diagram formatting

I found a couple misspelled words in the crush-map documentation, and also
tweaked the formatting of the CRUSH hierarchy diagram to center some of the
entries.

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>

mds: add new feature bit

Added new feature bit for InodeStat, DirStat and LeaseStat versioning

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>

tools/cephfs: fix mkdirs(exist_ok) call

we need to use pathlib.Path.mkdirs(parents=True, exist_ok=True) or
os.mkdirs(path), but not both.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: cephfs-shell cleanup

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: run cephfs-shell thru autopep8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23422 into master

* refs/pull/23422/head:
log/Log: fix buffer overflow

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/options: correct the description of rgw_enable_lc_threads option

Signed-off-by: excellentkf <likefei@cmss.chinamobile.com>

Merge pull request #23466 from tspmelo/wip-update-packages

mgr/dashboard: Update frontend packages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

debian: mark python-ceph-argparse Arch = all

as the python{3,}-ceph-argparse is pure python

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: package cephfs-shell for fedora

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs,deb: package cephfs-shell

change `#!/usr/bin/env python3` to `#!/usr/bin/python3` as per
https://www.debian.org/doc/packaging-manuals/python-policy/programs.html#interpreter-directive

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix build WITH_SYSTEM_BOOST=ON

FindBoost.cmake from upstream cmake now finds python libraries like

find_package(Boost 1.67 python36)

and it export targets like Boost::python36

but we are still linking against Boost::python, so to be compatible
with FindBoost.cmake, we need to update BuildBoost.cmake and
mgr/CMakeLists.txt accordingly. in other words, to export
Boost::python36 and to link Boost::python36.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install script and egg-info files of cephfs-shell

egg-info offers requires.txt, which is parsed by dh_python3 to prepare
the dependencies for the cephfs-shell packaging. also, the meta-info
in the .egg allows user to use eggs if they wish. see
https://wiki.debian.org/Python/FAQ#How_should_we_package_Python_eggs.3F
.

Fixes: http://tracker.ceph.com/issues/26852
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: add setup.py for cephfs-shell

easier to manage its dependency this way, also pave its road to pypi.
please note, cephfs is not hosted by pypi yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23431 from falcon78921/wip-doc-rados-pool

doc: fixed links in Pools section

Reviewed-by: John Wilkins <jowilkin@redhat.com>

osd/OSD.cc: force updating heartbeat peers periodically

Because the cluster topology may change (e.g., because we add some
new racks, hosts and disks) and we want the existing osds are then
aware of new incoming osds, guaranteeing osds are always trying to
do heartbeat as wide as possible(e.g., across all racks, hosts etc.).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush/CrushWrapper: check against dead crush rule

We currently remove a rule without adjusting the **rules** array
because we don't recycle the rule_no and hence there can be
holes in the "rules" array.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD.cc: choose heartbeat peers by failure domain

By default, monitor requires at least two valid failure votes/reports from
different hosts to mark an OSD down, which turns out to be impossible sometimes
for a replicated-pool of size of 2 in those clusters made up of hosts
with contiguous labeled OSDs.

This patch instead does a breadth-first search based on the highest level of
failure domain at cluster-wide, to try to make heartbeat peers can cover all failure domains
whenever possible, which can hopefully help accelerating osd failure detection
in the above case..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23441 from tchaikov/wip-cmake-rocksdb-cleanup

cmake: rocksdb related cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23462 into master

* refs/pull/23462/head:
doc: cleanup in cephfs/capabilities

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon/PGMap: fix spacing around pretty-printed SI units

"1k foo" not "1kfoo"

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23489 from alfredodeza/wip-cv-mgr

ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/MgrClient: make some noise for a user if no mgr daemon is running

Otherwise a cli command may unexpectedly hang forever with not output
because no ceph-mgr is running, which can be confusing for an admin.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23475 into master

* refs/pull/23475/head:
qa/standalone/crush/crush-choose-args: run mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23478 into master

* refs/pull/23478/head:
mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23267 from ifed01/wip-ifed-rm-fio-dir

test/fio:  pass config params to object store in a different manner

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>