qa/workunits/rbd: fixups for the new krbd discard behavior

With discard_granularity set to alloc_size, we no longer get object
size alignment from blk_bio_discard_split().

This assumption is pretty deeply ingrained in krbd_data_pool.sh, so
make it explicit.  For krbd_fallocate.sh, just fix the expectation.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_fallocate.sh: zero can be munged to truncate

The test case is issuing discards that span two objects: the tail of
the first object is truncated, the head of the second object is zeroed.
These discards aren't serial, so there is a race:

  discard i ~ i + 1: truncate i, zero i + 1
  discard i + 1 ~ i + 2: truncate i + 1, zero i + 2

can be executed as

  truncate i + 1, zero i + 2, truncate i, zero i + 1

For object i + 1, the sequence ends up being truncate tail, then zero
head.  This zero op is munged to truncate on the OSD, resulting in size
0 instead of OBJECT_SIZE / 2.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #27150 from tchaikov/wip-ceph.spec.in/immutable-object-cache

rpm: immutable-object-cache related changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27126 from tchaikov/wip-monclient-srv-priority

mon/MonClient: add proper SRV priority support

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25808 from joke-lee/fix_dump_usage_categories_info

rgw: usage dump_unsigned instead dump_int in dump_usage_categories_info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26070 from iliul/fix-bug-38002

rgw: Replace COMPLETE_MULTIPART_MAX_LEN with rgw_max_put_param_size

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26947 from IrekFasikhov/fixRGWMultiDeleteObject

rgw: fix RGWDeleteMultiObj::verify_permission()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #26634 from theanalyst/scheduler-beast-ctx

rgw: build async scheduler only when beast is built

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25451 from nathan-weinberg/gateway-tests

mgr/dashboard: Added breadcrumb tests to Object Gateway menu items

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26850 from nathan-weinberg/nfs-breadcrumb

mgr/dashboard: Added breadcrumb tests to NFS menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

rpm,deb: let immutable-object-cache depend on librados2

ceph-common depends on librados2, but ceph-immutable-object-cache only
uses librados2's C++ API, so librados2 would suffice.

rpmbuild is able to figure out the .so dependencies, and let
ceph-immutable-object-cache depend on the package which offers
librados2, but we'd better make sure immutable-object-cache is linked
against the librados2 of the same version, to ensure that librados2
offers the correct C++ symbols. as we only version the C++ API in the
librados2 after nautilus.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27090 from xiexingguo/wip-config-mon-core

mon/ConfigMonitor: make 'num' of 'config reset' optional; allow target version 0

Reviewed-by: Sage Weil <sage@redhat.com>

rpm: remove libradospp1 from immutable-object-cache's Requires

* remove "libradospp1" from immutable-object-cache's runtime
  dependencies. this package was removed in 65c8733b

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: do not use "-n" for immutable-object-cache

as immutable-object-cache has the prefix of "ceph", no need to use "-n"
option. so this change is but a cleanup.

* use "%package immutable-object-cache" not "%package
  ceph-immutable-object-cache" for naming subpackage of
  "ceph-immutable-object-cache". this would result in package name of
  "ceph-immutable-object-cache". see
  http://ftp.rpm.org/max-rpm/s1-rpm-subpack-spec-file-changes.html
* the same applies to other directives with "-n" options, see
  http://ftp.rpm.org/max-rpm/s1-rpm-inside-package-directive.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigMonitor: make 'num' of 'config reset' command optional

This way it can be used to fast cancel/undo the last command.
Also make the tip message a litter bit nicer..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27147 into master

* refs/pull/27147/head:
qa/workunits/mon/config.sh: s|bin/ceph|ceph|

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26531 from majianpeng/msg-async-optimization

msg/async: optimizations

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27107 from neha-ojha/wip-38845

mon/Monitor.cc: print min_mon_release correctly

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27097 from tchaikov/wip-option-sv-as-keys

common/config: use string_view for keys

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26220 from ashitakasam/master

common/Formatter: escape printed buffer in XMLFormatter::dump_format_va()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27138 into master

* refs/pull/27138/head:
qa/suites/upgrade/nautilus-x: misc updates

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/workunits/mon/config.sh: s|bin/ceph|ceph|

so we can verify that ceph is able to reject wrong debug levels.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27143 from dzafman/wip-38896

doc: Fix the pg states and auto repair config options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix the pg states and auto repair config options

Fixes: http://tracker.ceph.com/issues/38896
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27140 from neha-ojha/wip-fix-cron

qa/crontab/teuthology-cronjobs: fix suite-branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #27129 from yuriw/wip-yuriw-subsets-master

qa/tests: added the subset clause for nautilus branch

qa/crontab/teuthology-cronjobs: fix suite-branch

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/nautilus-x: misc updates

- start with msgr2 enabled (defaults)
- run nautilus branch for workunits
- drop msgr2 enable step at teh end
- add octopus placeholder (although it is empty for now)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26939 into master

* refs/pull/26939/head:
os/bluestore: unconditionally cap chunks returned by allocator to 2^31
os/bluestore: start using 64-bit intervals for bitmap allocator
os/bluestore: make bluestore interval base template.
tests/fastbmap_alloc: UT to reproduce 4G allocation bug
os/bluestore: os/bluestore: implement dump for bitmap allocator
os/bluestore be more tolerant to lack of space for bluefs.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27086 into master

* refs/pull/27086/head:
msg: remove unused header file in Messenger.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27111 into master

* refs/pull/27111/head:
pybind/rados: should pass "name" to cstr()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27112 into master

* refs/pull/27112/head:
qa/suites: do not test luminous-x upgrade path

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27047 from tianshan/fix_38811

rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27102 from adamemerson/wip-one-rados-handle-to-rule-them-all

One Rados Handle to Rule Them All

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25999 from cbodley/wip-37945

rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26998 from cbodley/wip-38328

rgw: don't crash on missing /etc/mime.types

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26952 from cbodley/wip-rgw-bucket-tagging-notsupp

rgw: reject bucket tagging requests and document unsupported

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26722 from cbodley/wip-rgw-sync-counters

rgw multisite: add perf counters to data sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27008 from ofriedma/wip-34308-beast-tcp-nodelay

rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27105 from neha-ojha/wip-21174

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26819 from tchaikov/wip-rpm-more-recommends

rpm: use Recommends on fedora also

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27108 from tchaikov/wip-crimson-vstart.sh

 vstart.sh: add an option to use crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

This is particularly important as the crt might have changed before
being passed to merge_object_divergent_entries().

Fixes: http://tracker.ceph.com/issues/38894
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27083 from neha-ojha/wip-38635

qa/objectstore: test with reduced value of osd_memory_target

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

vstart.sh: pass osd specific options to OSD

--osd-args : for passing osd specific args

this allows us to pass seastar related options to crimson-osd

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

qa/tests: added the subset clause for nuatilis branch
--subset x/2999 should generate ~250 jobs as result

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites: do not test luminous-x upgrade path

in this change,

* suites/rados/upgrade: luminous-x-singleton => mimic-x-singleton
* suites/upgrade: luminous-x => nautilus-x

we support upgrade from n to n+2 release. otherwise monitor refuses to
do so:

mon.a@-1(probing) e1 current monmap has min_mon_release 15 (luminous)
which is >2 releases older than me 15 (octopus), stopping.

Fixes: https://tracker.ceph.com/issues/38845
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: take all-0 weights as a special case

if all weights of monitors are 0, it means the administrator want to use
uniform distribution for the load balance. see page.2 of
https://www.ietf.org/rfc/rfc2782.txt .

so remove the default weight of 10.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: reject bucket tagging requests and document unsupported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/weighted_shufft: rewrite it to avoid copyright issue

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: respect priority in SRV

before this change, we always choose the monitors with the lowest
priority and ignore the ones with higher priorty. but per
https://www.ietf.org/rfc/rfc2782.txt

> A client MUST attempt to contact the target host with the
> lowest-numbered priority it can reach

so, if a monitor is not reachable, we should try the ones with the
lowest ones *reachable*.

after this change, MonClient will memorize the monitors have been
contacted. and will continue hunting the ones not tried if the last
batch fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25545 from zhouyuan/wip-ceph-immutable-obj-cache-daemon

tools: adding ceph level immutable obj cache daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26675 from trociny/wip-rbd-io-simple-scheduler

librbd: simple scheduler plugin for object dispatcher layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: Adding tcp_nodelay option to Beast

beast frontend option to set the TCP_NODELAY socket option to match the tcp_nodelay option in civetweb.

Fixes: https://tracker.ceph.com/issues/34308
Signed-off-by: Or Friedmann <ofriedma@redhat.com>

mon/ConfigMonitor: kill improper assert

otherwise it can be reliably triggerred by the following command:

  ceph config reset 0

There are two ways to fix this problem:
- add some more santicy checks and prevent user from passing in
  any kind of non-positive (not permitted) value
- kill the assert instead

Option 2 sounds to be the preferred choice for me since "0" in general
is the natural representation of "nothing" and hence could always
be used to revert everything to its default or initiative state.

But perhaps I am the only one?

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27098 from ming416/master

mgr/dashboard: auth ttl expired error

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #27070 into master

* refs/pull/27070/head:
osd/OSDMap: add 'zone' to default crush map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27020 into master

* refs/pull/27020/head:
common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27021 into master

* refs/pull/27021/head:
msg: remove XioMessenger
qa/suites/rados/thrash-old-clients: add nautilus
qa/suites/rados/thrash-old-clients: add mimic v1 variant
qa/suites/rados/thrash-old-clients: add mimic
qa/suites/rados/thrash-old-clients: collapse msgr and client choice
qa: remove simplemessenger tests
ceph_test_msgr: remove simple
msg: remove SimpleMessenger

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27101 into master

* refs/pull/27101/head:
mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Neha Ojha <nojha@redhat.com>

mon/Monitor.cc: print min_mon_release correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: auth ttl expired error

Fixes: https://tracker.ceph.com/issues/38428
Signed-off-by: ming416 <geenature@163.com>

Merge pull request #26940 from xiexingguo/wip-monc-add-con

mon/MonClient: weight-based mon selection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26955 from liewegas/wip-slow-add

crush: various fixes for weight-sets, the osd_crush_update_weight_set option, and tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26950 from xiexingguo/wip-mgr-fixes

mgr/Mgr: kill redundant sub_unwant call

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27068 from hjwsm1989/upmap-filter-out-underfull-buckets

crush: add root_bucket to identify underfull buckets

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27067 from SUSE/wip-fix-38788

cmake: remove cython 0.29's subinterpreter check during install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26923 from majianpeng/msg-optimize-check-loopback-con

msg/async/AsyncConnection: optimize check loopback connection.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27076 from MariusSchiffer/ceph-mgr-python3-fix

mgr/balancer: Python 3 compatibility fix

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/rados: should pass "name" to cstr()

it's a regression introduced by 6cb23f9c

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add an option to use crimson-osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27106 from dalgaaf/dalgaaf-wip-fix-lrc-doc

doc: fix LRC documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27071 from tchaikov/wip-crimson-osd-peering

crimson/osd: add minimal state machine for PG peering

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: add minimal PG recovery FSM

after this change, PG is able to go clean as primary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: advance pg in consume_map()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: update info.stats

PG.info.stats is sent to mgr periodically for PG stats reporting,
also, it is sent to peer OSD in MOSDPGLog message when PG performs
peering. we need to keep it updated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add more peering facilities

* add PG::should_restart_peering() for telling if we should start
  a peering at seeing a new mapping
* add PG::start_peering_interval() for starting a peering
* add PG::activate() for activating a PG
* add PG::on_activated() to be called once a PG is activated
* add PG::send_to_osd() for sending message to given OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add operator<< for PG

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: add peer_activated

add facilities for updating peer_activated. it is used for tracking the
peers has activated and committed. once all of them ack the activation
proposed by primary, the PG is considered activated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for sending notify

we need to notify the primary osd that a PG hosted by it has changed,
and it needs to start peering for it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: maintain PG::want_acting

* add PG::choose_acting()
* add PG::proc_replica_info()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add PG::is_{acting,up}()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wait osdmap before processing peering evt

the constraint that

> peering messages/events from peers must remain ordered by peer

is still enforced. because, the peering message from a certain peer
will block the succeeding messages from this peer, if the latest osdmap
possessed by OSD is not new enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::_send_active()

so PG can request monitor to update the OSD's up_thru in osdmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGLog in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGQuery in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGInfo in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGNotify in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: keep track of last_peering_reset

last_peering_reset is used to track the epoch when the most recent peering
started. it is used to tell if a message is targeting a changed PG. the
sender of a peering message will attach it. and the recipient will check
it against its own last_peering_reset. for instance, if a message claims
that that its map epoch is 16, while our last_peering_reset is 18, we will
consider this message stale, and probably drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config_values: use string_view for keys

the contained char sequences are either static char strings or strings
in `Option` instances. so no need to keep another copy of string around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigMonitor: use MConfig(map&&)

less memcpy this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config,mon/ConfigMap: use transparent comparator

use transparent comparator as the comparator in the map of settings, so
we can find the values in a map<string,string> by string_view.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: let ConfigMap::generate_entity_map() return the map

C++17 enforces copy elision, so no need to pass the output parameter by
reference/pointer anymore. also, it would be easier if we change the
output's type by putting something like

auto output = config.generate_entity_map(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ConfUtil: use string_view as key when accessing settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: use string_view as key when accessing settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,librbd: use string_view for keys of config

the contained char sequences are either static char strings or strings
in `Option` instances. so no need to keep another copy of string around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix LRC documentation

Recovery from a failure in jerasure need only k reads and
not k+m-1.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw: Get rid of num_rados_handles infrastructure in RGWSI_RADOS

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Get rid of num_rados_handles infrastructure in RGWRados

Since we aren't supporting multiple handles, don't have the overhead.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove rgw_num_rados_handles option

This has been deprecated for some time and underlies much of the
complexity of the RADOS service.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>