librbd: allow to provide sparse read result

Signed-off-by: Mykola Golub <mgolub@suse.com>

osdc/striper: make possible to assemble sparse read result

Signed-off-by: Mykola Golub <mgolub@suse.com>

librados_test_stub: pass snap context to "create" method

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add sparse_copyup method

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Remove messages.xlf (#28020)

mgr/dashboard: Remove messages.xlf

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27976 from mykaul/patch-2

doc: Update link to Red Hat documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Remove messages.xlf

Removes the messages.xlf file from the git repository, due to a lot of merge conflicts.
It won't cause any problems, because the file is not important for building or the translation during runtime.

Fixes: https://tracker.ceph.com/issues/38003
Signed-off-by: Sebastian Krah <skrah@suse.com>

Merge pull request #26654 from sebastian-philipp/orchestrator-progress

mgr/orchestrator: add progress events to all orchestrators

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc: Update link to Red Hat documentation

The previous link was to version 2, which is a bit older. Fix to version 3.

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Merge pull request #27855 from tchaikov/wip-ceph-release

src/: define ceph_release_t and use it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27689 from Jeegn-Chen/wip-39390-fix

os/filestore: assure sufficient leaves in pre-split

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27925 from tchaikov/wip-38628

mgr/restful: do not use filter() for list

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #27962 from xiexingguo/wip-jira-3756

mgr/DaemonServer: safe-to-destroy - do not consider irrelevant pgs

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28036 from neha-ojha/wip-fix-ar-doc

doc/dev/osd_internals/async_recovery: update cost calculation

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28034 from letterwuyu/forward_incompat_pg_missing

ceph-object-corpus: forward_incompat pg_missing_item and pg_missing_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-object-corpus: forward_incompat pg_missing_item and pg_missing_t

Signed-off-by: lishuhao <lishuhao@unitedstack.com>

doc/dev/osd_internals/async_recovery: update cost calculation

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #27688 into master

* refs/pull/27688/head:
pybind: added lseek()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27482 into master

* refs/pull/27482/head:
client: fix fuse client hang because its bad session PipeConnection

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27761 into master

* refs/pull/27761/head:
qa: use skipTest method instead of exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27480 into master

* refs/pull/27480/head:
mds: check dir fragment to split dir if mkdir makes it oversized.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #27765 from linuxbox2/wip-rgw-crytpexcept

rgw: crypto: throw DigestException from Digest and HMAC

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27843 into master

* refs/pull/27843/head:
os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27716 into master

* refs/pull/27716/head:
cephfs-shell: fix string decoding for ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28013 from cbodley/wip-39611

rgw: use explicit to_string() overload for boost::string_ref

Reviewed-by: Nathan Cutler <nculter@suse.com>

mgr/dashboard: fix help message for minimum blob size in pool form (#28017)

mgr/dashboard: fix help message for minimum blob size in pool form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27813 from ricardoasmarques/do-not-log-iscsi-get-requests

mgr/dashboard: iSCSI GET requests should not be logged

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27946 from ricardoasmarques/iscsi-form-ipv6

mgr/dashboard: Fix iSCSI form when using IPv6

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: iSCSI GET requests should not be logged

Fixes: https://tracker.ceph.com/issues/39492
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/orchestrator: add progress events to all orchestrators

Now, progress events are part of `WriteCompletion` istead of part of the orchestrator module.

It does not yet provide a way to just show orchestrator events.

Also fixes issue in the SSH orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #27873 from rzarzynski/wip-crimson-writes

crimson: add support for basic write path

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/thread: include reactor.hh to avoid linking error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add support for basic write path, part 2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make ObjectState mutable. Evict mutated state on error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: implement bits needed for basic write path.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27413 from liu-chunmei/ceph_seastar_io_read

crimson/osd: support write pid_file when osd start

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix help message for minimum blob size in pool form

A value less than maximum blob size is required.

Fixes: http://tracker.ceph.com/issues/39624
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #27665 from badone/wip-improve-dump_pgstate_history-json

osd: Improve dump_pgstate_history json output

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28015 from tchaikov/wip-install-deps.sh

install-deps.sh: install libboost-test for seastar

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Discard comand should be issued before space released.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/39621

install-deps.sh: install libboost-test for seastar

see
https://github.com/scylladb/seastar/blob/5858fe30c16a1a059200c129d20180ceb0199d99/cmake/SeastarDependencies.cmake#L58-L68

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support write pid_file when osd start

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #27979 from yuriw/wip-yuriw-crontab-master

qa/test: reduce over all number of runs

Merge pull request #27941 from dzafman/wip-39011

doc: Describe recovery and backfill prioritizations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: use explicit to_string() overload for boost::string_ref

Fixes: http://tracker.ceph.com/issues/39611
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27981 from dillaman/wip-39598

qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27911 into master

* refs/pull/27911/head:
os/bluestore: dump onode meta before "no spanning blob" assertion.
os/bluestore: move _dump_xxx methods out of BlueStore class

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27929 into master

* refs/pull/27929/head:
os/bluestore: be verbose about objects that existing on rmcoll
osd/PrimaryLogPG: disallow ops on objects with an empty name
osd/PG: fix cleanup of pgmeta-like objects on PG deletion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26488 from mogeb/per-pool-scrub-status

mon: show no[deep-]scrub flags per pool in the status

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

qa/test: reduce over all number of runs

We kill thousands of queued jobs every week, so why do we even schedule them ?
Another point was that we run numerous of tests as part of PRs testing on released versions anyway, so it's duplicating effort

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: All RBD features are supported by 'tcmu:runner' (#28000)

mgr/dashboard: All RBD features are supported by 'tcmu:runner'

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27830 from dzafman/wip-38124

osd: Don't include user changeable flag in snaptrim related assert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27841 from richael02/multiNVMESSD

vstart.sh: enable creating multiple OSDs backed by spdk backend

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28008 from tchaikov/wip-seastar

seastar: pick up changes for better performance

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: do not build unit test of seastar

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up changes for better performance

to be specific, a78fb44c96e2912c6f39b2151f94a0bb2b5796a6 helps to
improve the performance of future implementation -- with this change
future can always reference its local state without checking its `_promise`
and dereferencing it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard/qa: Fix a few linting issues in dashboard backend tests (#27865)

mgr/dashboard/qa: Fix a few linting issues in dashboard backend tests

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #27947 from smithfarm/wip-39579

pybind/tox: pass additional command line arguments through to tox

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: All RBD features are supported by 'tcmu:runner'

Fixes: https://tracker.ceph.com/issues/39607
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27969 from yuvalif/wip-yuval-add-tenant-to-user

rgw: add tenant as parameter to User in multisite tests

Merge pull request #27987 from tchaikov/wip-crimson-osd-shutdown

crimson/osd: shutdown services in the right order

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: object state can be manually evicted from OSD's cache.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: retrieve object state once per MOSDOp.

Justification behind the change is behaviour of classical OSD.
It calls PrimaryLogPG::find_object_context() far before going
through OSDOps in ::do_osd_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add support for basic write path, part 1.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: add inclusion guard in Transaction.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: PG caches ObjectState instead of object_info_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: PG::do_osd_op() takes OSDOp by reference, not pointer.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: PG::do_osd_op throws about unsupported op.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: handle hot CaynStore exceptions without throwing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27931 from yuvalif/pubsub_info_update_fix

rgw/pubsub: fix doc on updates. fix multi-notifications

rgw: add tenant as parameter to User in multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/dashboard: Fix iSCSI form when using IPv6

Fixes: https://tracker.ceph.com/issues/39578
Signed-off-by: Ricardo Marques <rimarques@suse.com>

crimson/osd: shutdown services in the right order

we should stop config service *after* osd is stopped, as osd depends on
a working and alive config subsystem when stopping itself. for instance,
the destructor of AuthRegistry unregisters itself from the ObserverMgr,
which is in turn a member variable of ConfigProxy, so if ConfigProxy is
destroyed before we destroy mon::Client, we will have a segfault with
following backtrace

ObserverMgr<ceph::md_config_obs_impl<ceph::common::ConfigProxy>
>::remove_observer(ceph::md_config_obs_impl<ceph::common::ConfigProxy>*)
at /var/ssd/ceph/build/../src/common/config_obs_mgr.h:78
AuthRegistry::~AuthRegistry() at
/var/ssd/ceph/build/../src/crimson/common/config_proxy.h:101
 (inlined by) AuthRegistry::~AuthRegistry() at
/var/ssd/ceph/build/../src/auth/AuthRegistry.cc:28
ceph::mon::Client::~Client() at
/var/ssd/ceph/build/../src/crimson/mon/MonClient.h:44
ceph::mon::Client::~Client() at
/var/ssd/ceph/build/../src/crimson/mon/MonClient.h:44
OSD::~OSD() at /usr/include/c++/9/bits/unique_ptr.h:81

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: Improve dump_pgstate_history json output

Fixes: http://tracker.ceph.com/issues/38846
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27975 from tchaikov/wip-pidfile_write

global/pidfile: pass string_view instead of ConfigProxy to pidfile_wr…

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27932 into master

* refs/pull/27932/head:
osd: clean up osdmap sharing
osd: minor cleanup in dequeue_op

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

vstart.sh: enable creating multiple OSDs backed by spdk backend

Currently vstart.sh only support deploying one OSD based on NVMe SSD.
The following two cases will cause errors:
1.There are 2 more NVMe SSDs from the same vendor on the machine
2.Trying to deploy 2 more OSDs if we only get 1 pci_id available

Add the support for allowing deploying multiple OSDs on a machine with
multiple NVME SSDs.

Change-Id: I6016435c1438bb4d16aff31f4575e03ccd3c9b3d

Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>

Merge pull request #26152 from noonedeadpunk/mgr/zabbix_discovery

mgr/zabbix Added pools discovery and per-pool statistics

Reviewed-by: Wido den Hollander <wido@42on.com>

rgw/pubsub: revert the RGWSysObjectCtx change

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

qa/workunits/rbd: wait for rbd-nbd unmap to complete

The "unmap" request is asynchronous, so wait for a short amount
of time for the "rbd-nbd" daemon process to exit.

Fixes: http://tracker.ceph.com/issues/39598
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Apply suggestions from code review

Signed-off-by: Dmitriy Rabotjagov <noonedeadpunk@ya.ru>

Merge pull request #27967 from liewegas/wip-fix-rgw-bias

rgw/rgw_tools: fix 'osd pool set' json syntax

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Allow the decrease of pg's of an existing pool (#27785)

mgr/dashboard: Allow the decrease of pg's of an existing pool

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27977 from cbodley/wip-39597

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27968 from iotcg/master

vstart.sh: correct ceph-run path

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Fixes: http://tracker.ceph.com/issues/39597
Signed-off-by: Casey Bodley <cbodley@redhat.com>

global/pidfile: pass string_view instead of ConfigProxy to pidfile_write()

there is no need to pass ConfigProxy to this function. and passing a
string_view also make it easier to reuse this function.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27970 from Liuchang0812/duplicate-tags-bl

rgw: eliminates duplicated tags_bl var

Reviewed-by: Casey Bodley <cbodley@redhat.com>

pybind/tox: pass additional command line arguments through to tox

Fixes: 9426f1f2045d0ae0f319530c3dc3a9240d838d07
Fixes: https://tracker.ceph.com/issues/39579
Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix small typo in dashboard documentation (#27850)

doc: fix small typo in dashboard documentation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

vstart.sh: correct ceph-run path

ceph-run is in the same directory as vstart.sh. It's often that
vstart.sh is run under build directory. Without giving the right
directory, ceph-run file can't be found.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

rgw: eliminates duplicated tags_bl var

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #27538 from iotcg/radosgw

init-radosgw: use ceph-conf to get cluster configuration value

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: be verbose about objects that existing on rmcoll

This is always a bug (OSD doesn't try to remove a collection unless it
thinks it is empty), and not seeing it at default debug levels makes it
hard to track down.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: disallow ops on objects with an empty name

These may conflict with pgmeta objects.  And are going to cause other
problems later (e.g., https://tracker.ceph.com/issues/38724).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: fix cleanup of pgmeta-like objects on PG deletion

If an object has an empty 'name' field, it "looks" like a pgmeta object,
and the PG cleanup code was skipping it.  However, we were letting these
objects get created.

Fix by only skipping *our* pgmeta object.  If there are other pgmeta-like
objects in the PG collection, clean them up.

Fixes: https://tracker.ceph.com/issues/38724
Signed-off-by: Sage Weil <sage@redhat.com>

rgw/rgw_tools: fix 'osd pool set' json syntax

Fixes af30e3a512dd1e5c51639d1f823b99ed9b43fb50

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/pubsub: cleanup tests for multiple notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw: RGWUserPubSub::write invalidates its obj_ctx

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/DaemonServer: safe-to-destroy - do not consider irrelevant pgs

otherwise ceph osd safe-to-destroy would say NO even if we want to
destroy an down+out osd from an actually active+clean pool!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>