common: add templated Context factory

For use to create Contexts only when needed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add const version of get_payload

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: use queue of messages for dispatch

This avoids an allocation for each new message (amortized via deque).

Additionally, use std::move to avoid atomic update to the Message ref.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: dispatch intrusive_ptr Messages

This codifies the giving of a reference to the Dispatcher and helps avoid
memory leaks. Old-style dispatch is kept to allow older code to continue
working.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: use deque for dispatcher pointers

To improve cache locality.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23564 from falcon78921/wip-docs-links-1

doc: fixed broken urls

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23240 into master

* refs/pull/23240/head:
qa/suites/rados, qa/workunits/rados: Add suite/workunit for ceph-crash
add ceph-crash service
common/options: enable mgr 'crash' module by default
global/signal_handler: add 'done' file to signal crashdump is ready

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15831 from trociny/wip-18430-1

librbd: support migrating images with minimal downtime

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop snap lock before sending copyup request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: restart object deep copy with flatten

if copyup request was appended with non empty writes when the deep
copy was in flight.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: update in-memory migration snap map after creating snap

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: AsyncOperation may not be reused

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: migrate: skip update object map for snaps that don't overlap

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: don't update object map if object deep copy returned ENOENT

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: open migration image parent and store its pointer in ImageCtx

Use the pointer in copyup and migrate requests, when deep copying an
object.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use 'open snap by id' in clone request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use 'open snap by id' in deep copy image copy request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use 'open snap by id' in refresh parent request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow to open snapshot by id

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: separate read vs write migration parent overlaps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: corrected usage of ImageState::open flag parameter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migration should keep parent linkage by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move shared common clone parent logic to state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: normalize clone state machine method names and debug logs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update executing state description with migration % complete

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: make status show migration details if it is in progress

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: show when parent is migration source in info output

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: get_parent_info: don't try to find parent snap name for CEPH_NOSNAP

(it is possible now when the parent is a migration source)

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: CLI to migrate images

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: API for image migration

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: API for image migration

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow 'migrating' internal feature on image create

We want this feature set from the start to avoid races when a client
is opening an image being prepared for migration.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: handle CEPH_NOSNAP case outside of deep_copy::util::compute_snap_map

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add function to wait writes unblocked

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use newly added method to list watchers in remove request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: async request to list watchers

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow to remove already opened image

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: shortcut to skip v2 probing when opening v1 image

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add flags to open request

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add assert_snapc_seq method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add "migration" trash image source

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add "migration" trash image source

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: methods to set/get/remove image migration header

Signed-off-by: Mykola Golub <mgolub@suse.com>

doc: added :ref: directive for osdmaptool

Added :ref: directive for osdmaptool utility.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added osdmaptool cross reference

Added osdmaptool cross reference for docs/rados/operations/control.rst.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #23532 from alfredodeza/wip-rm24960

ceph-volume:  expand auto engine for single type devices on filestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23402 from cbodley/wip-23659

qa/rgw: patch keystone requirements.txt

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #23426 into master

* refs/pull/23426/head:
client: privatize some class members
client: reorganize Client header

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #23470 into master

* refs/pull/23470/head:
tools/cephfs-shell: unable to copy files to a sub-directory from local file system

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs-shell: unable to copy files to a sub-directory from local file system

Fixes:http://tracker.ceph.com/issues/26874

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge PR #23561 into master

* refs/pull/23561/head:
tools/cephfs: do not split args if it's already a list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume lvm.zap loop over all vgs from a pv to destroy them

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23552 into master

* refs/pull/23552/head:
ceph_test_objectstore: queue split on parent pg sequencer
os/bluestore: fix split vs finish_write race

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume tests.util ensure dmcrypt_close will skip non-existent paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.encryption if crypt path does not exist, skip it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap unmount lvs and close dmcrypt on partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph_test_objectstore: queue split on parent pg sequencer

This matches the OSD.

It also ensure that we drain transactions that precede the split, which
will include any finish_write calls that might otherwise attach the
Buffer to the wrong cache shard: _txc_finish() calls finish_write and
passes the cache shard without the cache shard lock held, but may block
waiting for split_cache() which then changes the destination collection's
shard.  Once it gets the lock and proceeds it would operate on the wrong
cache shard, leading to a failed assert later when the sharedblob is
trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23539 from cyx1231st/wip-seastar-msgr-cyx

crimson/net: fix unittest_seastar_messenger errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22457 from p-na/dash-settings

mgr/dashboard: Add backend support for changing dashboard configuration settings via the REST API

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/net: fix unittest_seastar_messenger errors

fix several bugs that may fail unittest_seastar_messenger:
1. keep holding the connection reference until closed.
2. zero h.reply before start to use it.
3. set server-side features correctly.
4. add missing return to link write with flush.
5. remove an empty lambda operation.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

doc: updated reference link for osdmaptool utility

Reference link for osdmaptool utility was broken (404 error).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: updated link for submitting patches

Updated link for Submitting Patches (404 error).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

tools/cephfs: do not split args if it's already a list

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23537 into master

* refs/pull/23537/head:
tools/cephfs-shell:fixing the issue in using  with_argparse.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23519 from falcon78921/wip-doc-troubleshoot-osds

doc: cleaned up troubleshooting OSDs documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/suites/rados, qa/workunits/rados: Add suite/workunit for ceph-crash

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #23401 from cbodley/wip-rgw-getomapkeys2-more

rgw: RGWRadosGetOmapKeysCR uses 'more' flag from omap_get_keys2()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23406 into master

* refs/pull/23406/head:
TestOSDMap: Test with negative upmap values
osd/OSDMap: Disallow negative values in _apply_upmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #23512 into master

* refs/pull/23512/head:
osd: vary tick interval +/- 5% to avoid scrub livelocks
osd: tick at OSD_TICK_INTERVAL, not heartbeat interval

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23515 into master

* refs/pull/23515/head:
os/bluestore_tool: print actual usage in BlueFS::dump_block_extents.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23516 into master

* refs/pull/23516/head:
os/bluestore: check bluefs_extents consistency (DB vs. BlueFS)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23517 into master

* refs/pull/23517/head:
os/bluestore: uniform loging format for bluefs_extent_t.
os/bluestore_tool: handle fsck's returned  status properly.
os/bluestore_tool: fix multiple extents handling in BlueFS::log_dump

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23536 into master

* refs/pull/23536/head:
mds: report lagginess at lower debug
MDSMonitor: note beacons and cluster changes at low dbg level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: fix split vs finish_write race

In _tcx_finish(), we were looking at the right Cache for the collection,
and then calling finish_write with that Cache and taking the lock.  This
could race with a split_cache() such that after we got the lock the
collection was not on a different cache.  This would in turn lead to a
failed assertion later on in _rm_buffer when the sharedblob was trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23291 into master

* refs/pull/23291/head:
tests: add Cephx-resembling scenarios to crypto tests.
tests: differentiate AES testing regarding text size.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22879 into master

* refs/pull/22879/head:
doc/rados/operations/user-management: document 'network' clause of moncap and osdcap
mds/SessionMap: make Session::connection private
mds/MDSAuthCaps: parse and enforce network restriction
mds/MDSAuthCaps: pass addr to is_capable
mds/SessionMap: track socket_addr for each Session
osd/OSDCap: enforce network restriction
osd/OSDCap: take addr arg to is_capable
osd/OSDCap: parse 'network' clause in grant
mon: adapt MonCap network checks to addrvecs
mon/MonCap: enforce network constraint (if present)
mon/MonCap: take addr for MonCap::is_capable
mon/MonCap: parse 'network ...' suffix
osd: add peer_socket_addr to Session
mon: use addrvec for RoutedRequest and MForward
msg/Connection: add get_peer_socket_addr()
common/ipaddr: add network_contains() helper
common/ipaddr: add parse_network helper
common/ipaddr: expose netmask_ipv[46]

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #23510 from tchaikov/wip-cmake-boost-build

cmake: fix build WITH_SYSTEM_BOOST=ON

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.functional batch add filestore sceanrios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23542 into master

* refs/pull/23542/head:
qa: fix mon config section

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix mon config section

Fixes: fc6e683832125a1042638f21b623e29d0d4c34d8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23444 into master

* refs/pull/23444/head:
cephfs-shell: avoid sys.argv modification
tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados/operations/user-management: document 'network' clause of moncap and osdcap

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: make Session::connection private

Ensure all setters go through set_connection()

Signed-off-by: Sage Weil <sage@redhat.com>

cephfs-shell: avoid sys.argv modification

In multiple locations in the code.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds/MDSAuthCaps: parse and enforce network restriction

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSAuthCaps: pass addr to is_capable

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: track socket_addr for each Session

Make sure socket_addr is populated when connection is set.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: enforce network restriction

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: take addr arg to is_capable

Not enforced yet; just plumbing!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDCap: parse 'network' clause in grant

This mirrors what we do in the MonCap (but, naturally, is a slightly
different since we're using a the spirit parse in a totally different way).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: adapt MonCap network checks to addrvecs

Breaking this out in a separate commit to make the backport a bit easier.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: enforce network constraint (if present)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: take addr for MonCap::is_capable

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCap: parse 'network ...' suffix

Allow an optional network constraint to any grant.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add peer_socket_addr to Session

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use addrvec for RoutedRequest and MForward

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: add get_peer_socket_addr()

Return the addr that is actually being used for this socket (among the
potentially many in the addrvec).

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: add network_contains() helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: add parse_network helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/ipaddr: expose netmask_ipv[46]

Signed-off-by: Sage Weil <sage@redhat.com>