rbd: add protected in snap list

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

Merge PR #23468 into master

* refs/pull/23468/head:
Allow to unittest shell using transcript

Merge PR #22555 into master

* refs/pull/22555/head:
msg: define MFoo::create helper
msg: add msgref cast method
msg: cleanup factory/ref definition in messages
mds: use message factory to avoid leaks
msg: add factory method to correctly construct smart_ptr
mds: remove dead MDS-MDS forwarding code
mds: manage Message lifetime with intrusive_ptr
common: add templated Context factory
msg: add const version of get_payload
msg: use queue of messages for dispatch
msg: dispatch intrusive_ptr Messages
msg: use deque for dispatcher pointers

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23551 into master

* refs/pull/23551/head:
qa: use recent kernel to kernel build testing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23608 from tchaikov/wip-librbd-test

test/librbd: fix -Wsign-compare warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: use recent kernel to kernel build testing

Fixes: http://tracker.ceph.com/issues/24679
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23597 into master

* refs/pull/23597/head:
qa: add cephfs-shell skeleton test case

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add cephfs-shell skeleton test case

Right now just tests that "help" works.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21286 from joke-lee/wip-auth-uri-encode-decode

rgw:  should recode  canonical_uri when caculate s3 v4 auth

Merge pull request #23533 from wjwithagen/wjw-fix-random_shuffle

osd: make random shuffle comply with C++17

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23547 from yaozongyou/mon-get-rank

mon/MonMap: remove duplicate code in get_rank

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

test/librbd: fix -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23445 from ricardoasmarques/wip-rbd-actions-disable

mgr/dashboard: Disable RBD actions during task execution

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23485 from javacruft/misc-32bit-fixes

build: 32 bit architecture fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23581 from JiYou/fix.examples.Makefile

examples/Makefile: add -Wno-unused-parameter to avoid compile error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Allow to unittest shell using transcript

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge pull request #23600 from wjwithagen/wjw-fix-freebsd-manual

doc: fix/cleanup freebsd osd disk creation

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix/cleanup osd disk creation

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

msg: define MFoo::create helper

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

examples/Makefile: add -Wno-unused-parameter to avoid compile error

Signed-off-by: You Ji <youji@ebay.com>

msg: add msgref cast method

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: cleanup factory/ref definition in messages

This eliminates duplicate code definitions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use message factory to avoid leaks

Fixes: http://tracker.ceph.com/issues/24306
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add factory method to correctly construct smart_ptr

This is to avoid easy leaks of the form:

    boost::intrusive_ptr(new M*(...));

where a ref is leaked because constructing the intrusive_ptr adds a ref.

It is expected that eventually all constructors move to protected/private so
that the message's factory is the only way to build a new message (safely).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove dead MDS-MDS forwarding code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: manage Message lifetime with intrusive_ptr

This change turned out to be far more extensive than I hoped but the end result
should prevent all Message-related memory leaks. I believe I fixed several
incidentally.

Fixes: http://tracker.ceph.com/issues/24306
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23592 into master

* refs/pull/23592/head:
tools/cephfs-shell:used cmd2 inbuilt colorize method instead of colorama

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs-shell:used cmd2 inbuilt colorize method instead of colorama

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge pull request #23580 from tchaikov/wip-man-page-parse-desc

man: skip directive starting with ".."

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

man: skip directive starting with ".."

the target definition of hyper link looks like:

.. _a-reference-label:

there is chance that we need to add `:ref:` referencing man pages,
and add target definition at the beginning of the manapges for the
references. in that case, we need to skip the ref directives when
looking for the title of the manpage.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add templated Context factory

For use to create Contexts only when needed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add const version of get_payload

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: use queue of messages for dispatch

This avoids an allocation for each new message (amortized via deque).

Additionally, use std::move to avoid atomic update to the Message ref.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: dispatch intrusive_ptr Messages

This codifies the giving of a reference to the Dispatcher and helps avoid
memory leaks. Old-style dispatch is kept to allow older code to continue
working.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: use deque for dispatcher pointers

To improve cache locality.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23564 from falcon78921/wip-docs-links-1

doc: fixed broken urls

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23240 into master

* refs/pull/23240/head:
qa/suites/rados, qa/workunits/rados: Add suite/workunit for ceph-crash
add ceph-crash service
common/options: enable mgr 'crash' module by default
global/signal_handler: add 'done' file to signal crashdump is ready

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15831 from trociny/wip-18430-1

librbd: support migrating images with minimal downtime

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop snap lock before sending copyup request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: restart object deep copy with flatten

if copyup request was appended with non empty writes when the deep
copy was in flight.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: update in-memory migration snap map after creating snap

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: AsyncOperation may not be reused

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: migrate: skip update object map for snaps that don't overlap

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: don't update object map if object deep copy returned ENOENT

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: open migration image parent and store its pointer in ImageCtx

Use the pointer in copyup and migrate requests, when deep copying an
object.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use 'open snap by id' in clone request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use 'open snap by id' in deep copy image copy request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use 'open snap by id' in refresh parent request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow to open snapshot by id

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: separate read vs write migration parent overlaps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: corrected usage of ImageState::open flag parameter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: migration should keep parent linkage by default

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move shared common clone parent logic to state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: normalize clone state machine method names and debug logs

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update executing state description with migration % complete

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: make status show migration details if it is in progress

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: show when parent is migration source in info output

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: get_parent_info: don't try to find parent snap name for CEPH_NOSNAP

(it is possible now when the parent is a migration source)

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: CLI to migrate images

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: API for image migration

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: API for image migration

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow 'migrating' internal feature on image create

We want this feature set from the start to avoid races when a client
is opening an image being prepared for migration.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: handle CEPH_NOSNAP case outside of deep_copy::util::compute_snap_map

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add function to wait writes unblocked

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: use newly added method to list watchers in remove request

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: async request to list watchers

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow to remove already opened image

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: shortcut to skip v2 probing when opening v1 image

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add flags to open request

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add assert_snapc_seq method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add "migration" trash image source

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add "migration" trash image source

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: methods to set/get/remove image migration header

Signed-off-by: Mykola Golub <mgolub@suse.com>

doc: added :ref: directive for osdmaptool

Added :ref: directive for osdmaptool utility.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: added osdmaptool cross reference

Added osdmaptool cross reference for docs/rados/operations/control.rst.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #23532 from alfredodeza/wip-rm24960

ceph-volume:  expand auto engine for single type devices on filestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23402 from cbodley/wip-23659

qa/rgw: patch keystone requirements.txt

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #23426 into master

* refs/pull/23426/head:
client: privatize some class members
client: reorganize Client header

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #23470 into master

* refs/pull/23470/head:
tools/cephfs-shell: unable to copy files to a sub-directory from local file system

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tools/cephfs-shell: unable to copy files to a sub-directory from local file system

Fixes:http://tracker.ceph.com/issues/26874

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

Merge PR #23561 into master

* refs/pull/23561/head:
tools/cephfs: do not split args if it's already a list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume lvm.zap loop over all vgs from a pv to destroy them

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23552 into master

* refs/pull/23552/head:
ceph_test_objectstore: queue split on parent pg sequencer
os/bluestore: fix split vs finish_write race

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume tests.util ensure dmcrypt_close will skip non-existent paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.encryption if crypt path does not exist, skip it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap unmount lvs and close dmcrypt on partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph_test_objectstore: queue split on parent pg sequencer

This matches the OSD.

It also ensure that we drain transactions that precede the split, which
will include any finish_write calls that might otherwise attach the
Buffer to the wrong cache shard: _txc_finish() calls finish_write and
passes the cache shard without the cache shard lock held, but may block
waiting for split_cache() which then changes the destination collection's
shard.  Once it gets the lock and proceeds it would operate on the wrong
cache shard, leading to a failed assert later when the sharedblob is
trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23539 from cyx1231st/wip-seastar-msgr-cyx

crimson/net: fix unittest_seastar_messenger errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22457 from p-na/dash-settings

mgr/dashboard: Add backend support for changing dashboard configuration settings via the REST API

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/net: fix unittest_seastar_messenger errors

fix several bugs that may fail unittest_seastar_messenger:
1. keep holding the connection reference until closed.
2. zero h.reply before start to use it.
3. set server-side features correctly.
4. add missing return to link write with flush.
5. remove an empty lambda operation.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

doc: updated reference link for osdmaptool utility

Reference link for osdmaptool utility was broken (404 error).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: updated link for submitting patches

Updated link for Submitting Patches (404 error).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

tools/cephfs: do not split args if it's already a list

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23537 into master

* refs/pull/23537/head:
tools/cephfs-shell:fixing the issue in using  with_argparse.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23519 from falcon78921/wip-doc-troubleshoot-osds

doc: cleaned up troubleshooting OSDs documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/suites/rados, qa/workunits/rados: Add suite/workunit for ceph-crash

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #23401 from cbodley/wip-rgw-getomapkeys2-more

rgw: RGWRadosGetOmapKeysCR uses 'more' flag from omap_get_keys2()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23406 into master

* refs/pull/23406/head:
TestOSDMap: Test with negative upmap values
osd/OSDMap: Disallow negative values in _apply_upmap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #23512 into master

* refs/pull/23512/head:
osd: vary tick interval +/- 5% to avoid scrub livelocks
osd: tick at OSD_TICK_INTERVAL, not heartbeat interval

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23515 into master

* refs/pull/23515/head:
os/bluestore_tool: print actual usage in BlueFS::dump_block_extents.

Reviewed-by: Sage Weil <sage@redhat.com>