librbd: handle CEPH_NOSNAP case outside of deep_copy::util::compute_snap_map

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add function to wait writes unblocked

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use newly added method to list watchers in remove request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: async request to list watchers

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow to remove already opened image

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: shortcut to skip v2 probing when opening v1 image

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add flags to open request

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add assert_snapc_seq method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add "migration" trash image source

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add "migration" trash image source

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: methods to set/get/remove image migration header

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23532 from alfredodeza/wip-rm24960

ceph-volume:  expand auto engine for single type devices on filestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23402 from cbodley/wip-23659

qa/rgw: patch keystone requirements.txt

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #23426 into master

* refs/pull/23426/head:
client: privatize some class members
client: reorganize Client header

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #23470 into master

* refs/pull/23470/head:
tools/cephfs-shell: unable to copy files to a sub-directory from local file system

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs-shell: unable to copy files to a sub-directory from local file system

Fixes:http://tracker.ceph.com/issues/26874

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge PR #23561 into master

* refs/pull/23561/head:
tools/cephfs: do not split args if it's already a list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume lvm.zap loop over all vgs from a pv to destroy them

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23552 into master

* refs/pull/23552/head:
ceph_test_objectstore: queue split on parent pg sequencer
os/bluestore: fix split vs finish_write race

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume tests.util ensure dmcrypt_close will skip non-existent paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.encryption if crypt path does not exist, skip it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap unmount lvs and close dmcrypt on partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph_test_objectstore: queue split on parent pg sequencer

This matches the OSD.

It also ensure that we drain transactions that precede the split, which
will include any finish_write calls that might otherwise attach the
Buffer to the wrong cache shard: _txc_finish() calls finish_write and
passes the cache shard without the cache shard lock held, but may block
waiting for split_cache() which then changes the destination collection's
shard.  Once it gets the lock and proceeds it would operate on the wrong
cache shard, leading to a failed assert later when the sharedblob is
trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23539 from cyx1231st/wip-seastar-msgr-cyx

crimson/net: fix unittest_seastar_messenger errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22457 from p-na/dash-settings

mgr/dashboard: Add backend support for changing dashboard configuration settings via the REST API

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/net: fix unittest_seastar_messenger errors

fix several bugs that may fail unittest_seastar_messenger:
1. keep holding the connection reference until closed.
2. zero h.reply before start to use it.
3. set server-side features correctly.
4. add missing return to link write with flush.
5. remove an empty lambda operation.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

tools/cephfs: do not split args if it's already a list

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23537 into master

* refs/pull/23537/head:
tools/cephfs-shell:fixing the issue in using  with_argparse.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23519 from falcon78921/wip-doc-troubleshoot-osds

doc: cleaned up troubleshooting OSDs documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #23401 from cbodley/wip-rgw-getomapkeys2-more

rgw: RGWRadosGetOmapKeysCR uses 'more' flag from omap_get_keys2()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23406 into master

* refs/pull/23406/head:
TestOSDMap: Test with negative upmap values
osd/OSDMap: Disallow negative values in _apply_upmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #23512 into master

* refs/pull/23512/head:
osd: vary tick interval +/- 5% to avoid scrub livelocks
osd: tick at OSD_TICK_INTERVAL, not heartbeat interval

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23515 into master

* refs/pull/23515/head:
os/bluestore_tool: print actual usage in BlueFS::dump_block_extents.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23516 into master

* refs/pull/23516/head:
os/bluestore: check bluefs_extents consistency (DB vs. BlueFS)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23517 into master

* refs/pull/23517/head:
os/bluestore: uniform loging format for bluefs_extent_t.
os/bluestore_tool: handle fsck's returned  status properly.
os/bluestore_tool: fix multiple extents handling in BlueFS::log_dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23536 into master

* refs/pull/23536/head:
mds: report lagginess at lower debug
MDSMonitor: note beacons and cluster changes at low dbg level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: fix split vs finish_write race

In _tcx_finish(), we were looking at the right Cache for the collection,
and then calling finish_write with that Cache and taking the lock.  This
could race with a split_cache() such that after we got the lock the
collection was not on a different cache.  This would in turn lead to a
failed assertion later on in _rm_buffer when the sharedblob was trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23291 into master

* refs/pull/23291/head:
tests: add Cephx-resembling scenarios to crypto tests.
tests: differentiate AES testing regarding text size.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22879 into master

* refs/pull/22879/head:
doc/rados/operations/user-management: document 'network' clause of moncap and osdcap
mds/SessionMap: make Session::connection private
mds/MDSAuthCaps: parse and enforce network restriction
mds/MDSAuthCaps: pass addr to is_capable
mds/SessionMap: track socket_addr for each Session
osd/OSDCap: enforce network restriction
osd/OSDCap: take addr arg to is_capable
osd/OSDCap: parse 'network' clause in grant
mon: adapt MonCap network checks to addrvecs
mon/MonCap: enforce network constraint (if present)
mon/MonCap: take addr for MonCap::is_capable
mon/MonCap: parse 'network ...' suffix
osd: add peer_socket_addr to Session
mon: use addrvec for RoutedRequest and MForward
msg/Connection: add get_peer_socket_addr()
common/ipaddr: add network_contains() helper
common/ipaddr: add parse_network helper
common/ipaddr: expose netmask_ipv[46]

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #23510 from tchaikov/wip-cmake-boost-build

cmake: fix build WITH_SYSTEM_BOOST=ON

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.functional batch add filestore sceanrios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23542 into master

* refs/pull/23542/head:
qa: fix mon config section

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix mon config section

Fixes: fc6e683832125a1042638f21b623e29d0d4c34d8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23444 into master

* refs/pull/23444/head:
cephfs-shell: avoid sys.argv modification
tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados/operations/user-management: document 'network' clause of moncap and osdcap

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: make Session::connection private

Ensure all setters go through set_connection()

Signed-off-by: Sage Weil <sage@redhat.com>

cephfs-shell: avoid sys.argv modification

In multiple locations in the code.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds/MDSAuthCaps: parse and enforce network restriction

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSAuthCaps: pass addr to is_capable

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: track socket_addr for each Session

Make sure socket_addr is populated when connection is set.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: enforce network restriction

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: take addr arg to is_capable

Not enforced yet; just plumbing!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: parse 'network' clause in grant

This mirrors what we do in the MonCap (but, naturally, is a slightly
different since we're using a the spirit parse in a totally different way).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: adapt MonCap network checks to addrvecs

Breaking this out in a separate commit to make the backport a bit easier.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: enforce network constraint (if present)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: take addr for MonCap::is_capable

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: parse 'network ...' suffix

Allow an optional network constraint to any grant.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add peer_socket_addr to Session

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use addrvec for RoutedRequest and MForward

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: add get_peer_socket_addr()

Return the addr that is actually being used for this socket (among the
potentially many in the addrvec).

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: add network_contains() helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: add parse_network helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: expose netmask_ipv[46]

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23066 into master

* refs/pull/23066/head:
include/utime.h: remove improper comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23513 into master

* refs/pull/23513/head:
doc: updated contact email for Ceph pull requests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23280 into master

* refs/pull/23280/head:
cephfs: LeaseStat versioning
cephfs: DirStat versioning
mds: make has_feature const function
cephfs: InodeStat versioning
mds: add new feature bit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23538 into master

* refs/pull/23538/head:
mds: mark beacons as high priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23527 into master

* refs/pull/23527/head:
mds: use fast dispatch to handle MDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23457 into master

* refs/pull/23457/head:
qa: ignore normal slow IO warnings for cephfs

Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: report lagginess at lower debug

To help diagnose laggy daemons with performance tuned debug settings.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

MDSMonitor: note beacons and cluster changes at low dbg level

These messages are essential for diagnosing the reason why the MDSMonitor is
kicking MDSs out of the MDSMap. They should also be rare enough that the extra
verbosity is not noticable.

Fixes: http://tracker.ceph.com/issues/26898
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore normal slow IO warnings for cephfs

Generally the slow warnings we get are just over the threshold. These warnings
are related to deploying multiple Ceph daemons side-by-side. Let's see how we
do with two minutes.

Ignoring the warnings entirely is unsatisfactory as they serve as a useful
canary in the coal mine when you see warnings for ops > some unreasonably large
amount of time.

Fixes: http://tracker.ceph.com/issues/26900
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: vary tick interval +/- 5% to avoid scrub livelocks

If you have two pgs that need to scrub on two OSDs, each the primary
for one pg and the replica for the other, you can end up in a livelock:

- both osds locally reserve a scrub slot
- both osds send a scrub schedule request
- both scrub requests are rejected
- both osds wait exactly 1 second
- repeat

Seems a bit unlikely, but I've seen test cases where it goes on more an
hour.

Fixes: http://tracker.ceph.com/issues/26890
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23416 from ZVampirEM77/wip-mfa-cmd-and-options

radosgw-admin: add mfa related command and options

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.

Fixes:http://tracker.ceph.com/issues/26853
Fixes:http://tracker.ceph.com/issues/26855

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:fixing the issue in using  with_argparse.

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

ceph-volume lvm.zap iterate over multiple lvs on a pv

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22563 from ifed01/wip-ifed-bdev-minsize

os/bluestore: fix bluefs extent miscalculations on small slow device

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23294 from noahdesu/timely-mgr-health-updates

mgr: timely health updates between monitor and manager

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23257 from ifed01/wip-ifed-fix-rcoll-race

os/bluestore: fix race between remove_collection and object removals

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23492 from liewegas/wip-mgrc-warning

mgr/MgrClient: make some noise for a user if no mgr daemon is running

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

cephfs: LeaseStat versioning

* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* Dropped LeaseStat::encode().
* The old format is maintained for backward compatibility.
* Created Locker::encode_lease() for encoding and dropped the duplicates.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs: DirStat versioning

* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* Dropped DirStat::encode().
* The old format is maintained for backward compatibility.
* Created CDir::encode_dirstat() for encoding and dropped the duplicates.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #23500 into master

* refs/pull/23500/head:
debian: mark python-ceph-argparse Arch = all
rpm: package cephfs-shell for fedora
tools/cephfs,deb: package cephfs-shell
cmake: install script and egg-info files of cephfs-shell
tools/cephfs: add setup.py for cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23480 into master

* refs/pull/23480/head:
tools/cephfs: fix mkdirs(exist_ok) call
tools/cephfs: cephfs-shell cleanup
tools/cephfs: run cephfs-shell thru autopep8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: mark beacons as high priority

The mons already mark beacon replies as high priority (via default mon message
priority). We should expect that the mons handle our beacons at the same
priority so the MDS doesn't wrongly get marked laggy.

Fixes: http://tracker.ceph.com/issues/26899
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.functional add xenial batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add bluestore,filestore playbooks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22868 from iliul/modify-log-level

rgw: Fix log level of gc_iterate_entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Fix log level of gc_iterate_entries

Fixes: http://tracker.ceph.com/issues/23801
Signed-off-by: iliul <liul.stone@gmail.com>

mds: use fast dispatch to handle MDSBeacon

Current mds handles MDSBeacon messages through MDSRank::ms_dispatch().
MDSRank::ms_dispatch() locks mds_lock at the very beginning. This means
that long running task (such as processing finished contexts) can delay
MMDSBeacon. Using fast dispatch for MDSBeacon can avoid this issue.

Fixes: http://tracker.ceph.com/issues/23519
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge PR #22838 into master

* refs/pull/22838/head:
kv/RocksDBStore: Handle nullptr if clock cache is chosen.
kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri pool.
src/kv: Initial import of a custom RocksDB cache.
src/rocksdb: switch back to master branch.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22970 into master

* refs/pull/22970/head:
os/bluestore: limit OSD memory usage by tuning the cache size.

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume tests.functional add centos7 filestore batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add centos7 bluestore batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: updated troubleshooting OSDs documentation

Restructured OSD event descriptions for better clarification.
Added a link to the write() system call reference.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

qa/rgw: raise default curl timeout for multisite suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: vary omap osd-side limit

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use get_omap_keys2

Making looping less fragile.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.util assert new as_int,as_float helper methods

Signed-off-by: Alfredo Deza <adeza@redhat.com>