doc/dev/osd_internals/async_recovery: update cost calculation

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #27843 into master

* refs/pull/27843/head:
os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27716 into master

* refs/pull/27716/head:
cephfs-shell: fix string decoding for ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28013 from cbodley/wip-39611

rgw: use explicit to_string() overload for boost::string_ref

Reviewed-by: Nathan Cutler <nculter@suse.com>

mgr/dashboard: fix help message for minimum blob size in pool form (#28017)

mgr/dashboard: fix help message for minimum blob size in pool form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27813 from ricardoasmarques/do-not-log-iscsi-get-requests

mgr/dashboard: iSCSI GET requests should not be logged

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27946 from ricardoasmarques/iscsi-form-ipv6

mgr/dashboard: Fix iSCSI form when using IPv6

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: iSCSI GET requests should not be logged

Fixes: https://tracker.ceph.com/issues/39492
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27873 from rzarzynski/wip-crimson-writes

crimson: add support for basic write path

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/thread: include reactor.hh to avoid linking error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add support for basic write path, part 2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make ObjectState mutable. Evict mutated state on error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: implement bits needed for basic write path.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27413 from liu-chunmei/ceph_seastar_io_read

crimson/osd: support write pid_file when osd start

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix help message for minimum blob size in pool form

A value less than maximum blob size is required.

Fixes: http://tracker.ceph.com/issues/39624
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #27665 from badone/wip-improve-dump_pgstate_history-json

osd: Improve dump_pgstate_history json output

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28015 from tchaikov/wip-install-deps.sh

install-deps.sh: install libboost-test for seastar

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Discard comand should be issued before space released.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/39621

install-deps.sh: install libboost-test for seastar

see
https://github.com/scylladb/seastar/blob/5858fe30c16a1a059200c129d20180ceb0199d99/cmake/SeastarDependencies.cmake#L58-L68

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support write pid_file when osd start

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #27979 from yuriw/wip-yuriw-crontab-master

qa/test: reduce over all number of runs

Merge pull request #27941 from dzafman/wip-39011

doc: Describe recovery and backfill prioritizations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: use explicit to_string() overload for boost::string_ref

Fixes: http://tracker.ceph.com/issues/39611
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27981 from dillaman/wip-39598

qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27911 into master

* refs/pull/27911/head:
os/bluestore: dump onode meta before "no spanning blob" assertion.
os/bluestore: move _dump_xxx methods out of BlueStore class

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27929 into master

* refs/pull/27929/head:
os/bluestore: be verbose about objects that existing on rmcoll
osd/PrimaryLogPG: disallow ops on objects with an empty name
osd/PG: fix cleanup of pgmeta-like objects on PG deletion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26488 from mogeb/per-pool-scrub-status

mon: show no[deep-]scrub flags per pool in the status

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

qa/test: reduce over all number of runs

We kill thousands of queued jobs every week, so why do we even schedule them ?
Another point was that we run numerous of tests as part of PRs testing on released versions anyway, so it's duplicating effort

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: All RBD features are supported by 'tcmu:runner' (#28000)

mgr/dashboard: All RBD features are supported by 'tcmu:runner'

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27830 from dzafman/wip-38124

osd: Don't include user changeable flag in snaptrim related assert

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27841 from richael02/multiNVMESSD

vstart.sh: enable creating multiple OSDs backed by spdk backend

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28008 from tchaikov/wip-seastar

seastar: pick up changes for better performance

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: do not build unit test of seastar

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up changes for better performance

to be specific, a78fb44c96e2912c6f39b2151f94a0bb2b5796a6 helps to
improve the performance of future implementation -- with this change
future can always reference its local state without checking its `_promise`
and dereferencing it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard/qa: Fix a few linting issues in dashboard backend tests (#27865)

mgr/dashboard/qa: Fix a few linting issues in dashboard backend tests

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #27947 from smithfarm/wip-39579

pybind/tox: pass additional command line arguments through to tox

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: All RBD features are supported by 'tcmu:runner'

Fixes: https://tracker.ceph.com/issues/39607
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27969 from yuvalif/wip-yuval-add-tenant-to-user

rgw: add tenant as parameter to User in multisite tests

Merge pull request #27987 from tchaikov/wip-crimson-osd-shutdown

crimson/osd: shutdown services in the right order

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: object state can be manually evicted from OSD's cache.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: retrieve object state once per MOSDOp.

Justification behind the change is behaviour of classical OSD.
It calls PrimaryLogPG::find_object_context() far before going
through OSDOps in ::do_osd_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add support for basic write path, part 1.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: add inclusion guard in Transaction.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: PG caches ObjectState instead of object_info_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: PG::do_osd_op() takes OSDOp by reference, not pointer.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: PG::do_osd_op throws about unsupported op.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: handle hot CaynStore exceptions without throwing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27931 from yuvalif/pubsub_info_update_fix

rgw/pubsub: fix doc on updates. fix multi-notifications

rgw: add tenant as parameter to User in multisite tests

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/dashboard: Fix iSCSI form when using IPv6

Fixes: https://tracker.ceph.com/issues/39578
Signed-off-by: Ricardo Marques <rimarques@suse.com>

crimson/osd: shutdown services in the right order

we should stop config service *after* osd is stopped, as osd depends on
a working and alive config subsystem when stopping itself. for instance,
the destructor of AuthRegistry unregisters itself from the ObserverMgr,
which is in turn a member variable of ConfigProxy, so if ConfigProxy is
destroyed before we destroy mon::Client, we will have a segfault with
following backtrace

ObserverMgr<ceph::md_config_obs_impl<ceph::common::ConfigProxy>
>::remove_observer(ceph::md_config_obs_impl<ceph::common::ConfigProxy>*)
at /var/ssd/ceph/build/../src/common/config_obs_mgr.h:78
AuthRegistry::~AuthRegistry() at
/var/ssd/ceph/build/../src/crimson/common/config_proxy.h:101
 (inlined by) AuthRegistry::~AuthRegistry() at
/var/ssd/ceph/build/../src/auth/AuthRegistry.cc:28
ceph::mon::Client::~Client() at
/var/ssd/ceph/build/../src/crimson/mon/MonClient.h:44
ceph::mon::Client::~Client() at
/var/ssd/ceph/build/../src/crimson/mon/MonClient.h:44
OSD::~OSD() at /usr/include/c++/9/bits/unique_ptr.h:81

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: Improve dump_pgstate_history json output

Fixes: http://tracker.ceph.com/issues/38846
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27975 from tchaikov/wip-pidfile_write

global/pidfile: pass string_view instead of ConfigProxy to pidfile_wr…

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27932 into master

* refs/pull/27932/head:
osd: clean up osdmap sharing
osd: minor cleanup in dequeue_op

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

vstart.sh: enable creating multiple OSDs backed by spdk backend

Currently vstart.sh only support deploying one OSD based on NVMe SSD.
The following two cases will cause errors:
1.There are 2 more NVMe SSDs from the same vendor on the machine
2.Trying to deploy 2 more OSDs if we only get 1 pci_id available

Add the support for allowing deploying multiple OSDs on a machine with
multiple NVME SSDs.

Change-Id: I6016435c1438bb4d16aff31f4575e03ccd3c9b3d

Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>

Merge pull request #26152 from noonedeadpunk/mgr/zabbix_discovery

mgr/zabbix Added pools discovery and per-pool statistics

Reviewed-by: Wido den Hollander <wido@42on.com>

rgw/pubsub: revert the RGWSysObjectCtx change

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

qa/workunits/rbd: wait for rbd-nbd unmap to complete

The "unmap" request is asynchronous, so wait for a short amount
of time for the "rbd-nbd" daemon process to exit.

Fixes: http://tracker.ceph.com/issues/39598
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Apply suggestions from code review

Signed-off-by: Dmitriy Rabotjagov <noonedeadpunk@ya.ru>

Merge pull request #27967 from liewegas/wip-fix-rgw-bias

rgw/rgw_tools: fix 'osd pool set' json syntax

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Allow the decrease of pg's of an existing pool (#27785)

mgr/dashboard: Allow the decrease of pg's of an existing pool

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27977 from cbodley/wip-39597

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27968 from iotcg/master

vstart.sh: correct ceph-run path

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Fixes: http://tracker.ceph.com/issues/39597
Signed-off-by: Casey Bodley <cbodley@redhat.com>

global/pidfile: pass string_view instead of ConfigProxy to pidfile_write()

there is no need to pass ConfigProxy to this function. and passing a
string_view also make it easier to reuse this function.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27970 from Liuchang0812/duplicate-tags-bl

rgw: eliminates duplicated tags_bl var

Reviewed-by: Casey Bodley <cbodley@redhat.com>

pybind/tox: pass additional command line arguments through to tox

Fixes: 9426f1f2045d0ae0f319530c3dc3a9240d838d07
Fixes: https://tracker.ceph.com/issues/39579
Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix small typo in dashboard documentation (#27850)

doc: fix small typo in dashboard documentation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

vstart.sh: correct ceph-run path

ceph-run is in the same directory as vstart.sh. It's often that
vstart.sh is run under build directory. Without giving the right
directory, ceph-run file can't be found.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

rgw: eliminates duplicated tags_bl var

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #27538 from iotcg/radosgw

init-radosgw: use ceph-conf to get cluster configuration value

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: be verbose about objects that existing on rmcoll

This is always a bug (OSD doesn't try to remove a collection unless it
thinks it is empty), and not seeing it at default debug levels makes it
hard to track down.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: disallow ops on objects with an empty name

These may conflict with pgmeta objects.  And are going to cause other
problems later (e.g., https://tracker.ceph.com/issues/38724).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: fix cleanup of pgmeta-like objects on PG deletion

If an object has an empty 'name' field, it "looks" like a pgmeta object,
and the PG cleanup code was skipping it.  However, we were letting these
objects get created.

Fix by only skipping *our* pgmeta object.  If there are other pgmeta-like
objects in the PG collection, clean them up.

Fixes: https://tracker.ceph.com/issues/38724
Signed-off-by: Sage Weil <sage@redhat.com>

rgw/rgw_tools: fix 'osd pool set' json syntax

Fixes af30e3a512dd1e5c51639d1f823b99ed9b43fb50

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/pubsub: cleanup tests for multiple notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw: RGWUserPubSub::write invalidates its obj_ctx

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27885 from b-ranto/wip-restful-perf-counters

restful: Expose perf counters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27891 from tchaikov/wip-mgr-pgp-num-change-throttling

mgr/DaemonServer: refactor pgp_num changes throttling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27960 from yuriw/wip-yuriw-crontab-master

qa/tests: removed all runs on ovh

Merge pull request #27896 from ivancich/wip-add-rgw-period-comments

rgw: add some comments to rgw code to help explain functionality

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27956 from yuriw/wip-yuriw-smoke-master

qa/tests - cleaned up distro settings

qa/tests: removed all runs on ovh

and consolidated `smoke` runs re: https://github.com/ceph/ceph/pull/27956

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: add some comments to rgw code to help explain functionality

Comments to explain the role of RGWPeriod and the use of prefixes in
gc log entries are added. Some other comments are reformatted for 80
columns.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27838 from yuvalif/wip-yuval-pubsub_teuthology-new

rgw/pubsub: add multisite pubsub tests to teuthology

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tests - cleaned up distro settings

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #27448 from ricardoasmarques/upgrade-ceph-iscsi-version

mgr/dashboard: Support ceph-iscsi config v9

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27887 from smithfarm/wip-bci-len255plus

script/backport-create-issue: handle long Redmine issue names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27895 from dillaman/wip-librbd-zero-copy

librbd: support zero-copy writes via the C API

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #16929 from amitkumar50/cov-rgw-1

rgw: Initialize member variables in rgw_sync.h, rgw_rados.h

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc: Describe recovery and backfill prioritizations

Fixes: http://tracker.ceph.com/issues/39011
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27927 from tchaikov/wip-util.collect_sys_info-is-optional

test/common/test_util: skip it if /etc/os-release does not exist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd: clean up osdmap sharing

- always use the Session::last_sent_epoch value, both for clients and osds
- get rid of the stl map<> of peer epochs
- consolidate all map sharing into a single maybe_share_map()
- optionally take a lower bound on the peer's epoch, for use when it is
  available (e.g., when we are handling a message that specifies what
  epoch the peer had when it sent the message)
- use const OSDMapRef& where possible
- drop osd->is_active() check, since we no longer have any dependency on
  OSD[Service] state beyond our osdmap

The old callchain was convoluted, partly because it was needlessly
separated into several layers of helpers, and partly because the tracking
for clients and peer OSDs was totally different.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: minor cleanup in dequeue_op

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27897 from ivancich/wip-add-editor-comments-rgw-services

rgw: add editor directive comments to rgw services source files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27820 from smithfarm/wip-39361

rgw: drop cloud sync module logs attrs from the log

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

test: do not conditionize on GTEST_HAS_PARAM_TEST

value-parameterized is supported on all platforms, so drop this check

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: s/INSTANTIATE_TEST_CASE_P/INSTANTIATE_TEST_SUITE_P/

the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/pubsub: fix doc on updates. attempt to fix multi-notifications

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

qa/valgrind.supp: added false-positive in boost::queue::lockfree

Signed-off-by: Jason Dillaman <dillaman@redhat.com>