osd/PG: move down peers out from peer_purged

In purge_strays(), we'll aggressively clear stray_set and
add all related peers into peer_purged.

However, if the corrsponding peer is down and becomes
up again, (unconditionally) adding it to peer_purged
will prevent primary from re-purging it.
(See Active::react(const MNotifyRec& notevt))

On consuming a new osdmap, let's move any down peers out from
peer_purged simutaneously. This way we can lower the risk
of leaving any leftover PGs behind.

Related-to: http://tracker.ceph.com/issues/38931
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: introduce all_missing_unfound helper

We use pg_log.missing to track each peer's missing objects separately,
whereas missing_loc records the location of all (probably existing) good copies
for both primary and replicas' missing objects. Hence an item from
pg_log.missing or missing_loc is of different meaning and is not comparable.

During recovery, we can skip recovering primary only if
- primary is good, e.g., has no missing at all
- or all of the primary's missing objects do exist in missing_loc and are
  currently unfound

Obviously, the current "all missing objects are unfound" checker is broken.
Fix by introducing an independent all_missing_unfound helper to make the
count of missing objects that are currently unfound correct.

Fixes: http://tracker.ceph.com/issues/38784
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26850 from nathan-weinberg/nfs-breadcrumb

mgr/dashboard: Added breadcrumb tests to NFS menu

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27090 from xiexingguo/wip-config-mon-core

mon/ConfigMonitor: make 'num' of 'config reset' optional; allow target version 0

Reviewed-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: make 'num' of 'config reset' command optional

This way it can be used to fast cancel/undo the last command.
Also make the tip message a litter bit nicer..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27147 into master

* refs/pull/27147/head:
qa/workunits/mon/config.sh: s|bin/ceph|ceph|

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26531 from majianpeng/msg-async-optimization

msg/async: optimizations

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27107 from neha-ojha/wip-38845

mon/Monitor.cc: print min_mon_release correctly

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27097 from tchaikov/wip-option-sv-as-keys

common/config: use string_view for keys

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26220 from ashitakasam/master

common/Formatter: escape printed buffer in XMLFormatter::dump_format_va()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27138 into master

* refs/pull/27138/head:
qa/suites/upgrade/nautilus-x: misc updates

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/workunits/mon/config.sh: s|bin/ceph|ceph|

so we can verify that ceph is able to reject wrong debug levels.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27143 from dzafman/wip-38896

doc: Fix the pg states and auto repair config options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix the pg states and auto repair config options

Fixes: http://tracker.ceph.com/issues/38896
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27140 from neha-ojha/wip-fix-cron

qa/crontab/teuthology-cronjobs: fix suite-branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #27129 from yuriw/wip-yuriw-subsets-master

qa/tests: added the subset clause for nautilus branch

qa/crontab/teuthology-cronjobs: fix suite-branch

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/upgrade/nautilus-x: misc updates

- start with msgr2 enabled (defaults)
- run nautilus branch for workunits
- drop msgr2 enable step at teh end
- add octopus placeholder (although it is empty for now)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26939 into master

* refs/pull/26939/head:
os/bluestore: unconditionally cap chunks returned by allocator to 2^31
os/bluestore: start using 64-bit intervals for bitmap allocator
os/bluestore: make bluestore interval base template.
tests/fastbmap_alloc: UT to reproduce 4G allocation bug
os/bluestore: os/bluestore: implement dump for bitmap allocator
os/bluestore be more tolerant to lack of space for bluefs.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27086 into master

* refs/pull/27086/head:
msg: remove unused header file in Messenger.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27111 into master

* refs/pull/27111/head:
pybind/rados: should pass "name" to cstr()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27112 into master

* refs/pull/27112/head:
qa/suites: do not test luminous-x upgrade path

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27047 from tianshan/fix_38811

rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27102 from adamemerson/wip-one-rados-handle-to-rule-them-all

One Rados Handle to Rule Them All

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25999 from cbodley/wip-37945

rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26998 from cbodley/wip-38328

rgw: don't crash on missing /etc/mime.types

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26952 from cbodley/wip-rgw-bucket-tagging-notsupp

rgw: reject bucket tagging requests and document unsupported

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26722 from cbodley/wip-rgw-sync-counters

rgw multisite: add perf counters to data sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27008 from ofriedma/wip-34308-beast-tcp-nodelay

rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27105 from neha-ojha/wip-21174

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26819 from tchaikov/wip-rpm-more-recommends

rpm: use Recommends on fedora also

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27108 from tchaikov/wip-crimson-vstart.sh

 vstart.sh: add an option to use crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

This is particularly important as the crt might have changed before
being passed to merge_object_divergent_entries().

Fixes: http://tracker.ceph.com/issues/38894
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27083 from neha-ojha/wip-38635

qa/objectstore: test with reduced value of osd_memory_target

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

vstart.sh: pass osd specific options to OSD

--osd-args : for passing osd specific args

this allows us to pass seastar related options to crimson-osd

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

qa/tests: added the subset clause for nuatilis branch
--subset x/2999 should generate ~250 jobs as result

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites: do not test luminous-x upgrade path

in this change,

* suites/rados/upgrade: luminous-x-singleton => mimic-x-singleton
* suites/upgrade: luminous-x => nautilus-x

we support upgrade from n to n+2 release. otherwise monitor refuses to
do so:

mon.a@-1(probing) e1 current monmap has min_mon_release 15 (luminous)
which is >2 releases older than me 15 (octopus), stopping.

Fixes: https://tracker.ceph.com/issues/38845
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: reject bucket tagging requests and document unsupported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25545 from zhouyuan/wip-ceph-immutable-obj-cache-daemon

tools: adding ceph level immutable obj cache daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26675 from trociny/wip-rbd-io-simple-scheduler

librbd: simple scheduler plugin for object dispatcher layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: Adding tcp_nodelay option to Beast

beast frontend option to set the TCP_NODELAY socket option to match the tcp_nodelay option in civetweb.

Fixes: https://tracker.ceph.com/issues/34308
Signed-off-by: Or Friedmann <ofriedma@redhat.com>

mon/ConfigMonitor: kill improper assert

otherwise it can be reliably triggerred by the following command:

  ceph config reset 0

There are two ways to fix this problem:
- add some more santicy checks and prevent user from passing in
  any kind of non-positive (not permitted) value
- kill the assert instead

Option 2 sounds to be the preferred choice for me since "0" in general
is the natural representation of "nothing" and hence could always
be used to revert everything to its default or initiative state.

But perhaps I am the only one?

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27098 from ming416/master

mgr/dashboard: auth ttl expired error

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #27070 into master

* refs/pull/27070/head:
osd/OSDMap: add 'zone' to default crush map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27020 into master

* refs/pull/27020/head:
common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27021 into master

* refs/pull/27021/head:
msg: remove XioMessenger
qa/suites/rados/thrash-old-clients: add nautilus
qa/suites/rados/thrash-old-clients: add mimic v1 variant
qa/suites/rados/thrash-old-clients: add mimic
qa/suites/rados/thrash-old-clients: collapse msgr and client choice
qa: remove simplemessenger tests
ceph_test_msgr: remove simple
msg: remove SimpleMessenger

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27101 into master

* refs/pull/27101/head:
mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Neha Ojha <nojha@redhat.com>

mon/Monitor.cc: print min_mon_release correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: auth ttl expired error

Fixes: https://tracker.ceph.com/issues/38428
Signed-off-by: ming416 <geenature@163.com>

Merge pull request #26940 from xiexingguo/wip-monc-add-con

mon/MonClient: weight-based mon selection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26955 from liewegas/wip-slow-add

crush: various fixes for weight-sets, the osd_crush_update_weight_set option, and tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26950 from xiexingguo/wip-mgr-fixes

mgr/Mgr: kill redundant sub_unwant call

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27068 from hjwsm1989/upmap-filter-out-underfull-buckets

crush: add root_bucket to identify underfull buckets

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27067 from SUSE/wip-fix-38788

cmake: remove cython 0.29's subinterpreter check during install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26923 from majianpeng/msg-optimize-check-loopback-con

msg/async/AsyncConnection: optimize check loopback connection.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27076 from MariusSchiffer/ceph-mgr-python3-fix

mgr/balancer: Python 3 compatibility fix

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/rados: should pass "name" to cstr()

it's a regression introduced by 6cb23f9c

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add an option to use crimson-osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27106 from dalgaaf/dalgaaf-wip-fix-lrc-doc

doc: fix LRC documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27071 from tchaikov/wip-crimson-osd-peering

crimson/osd: add minimal state machine for PG peering

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: add minimal PG recovery FSM

after this change, PG is able to go clean as primary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: advance pg in consume_map()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: update info.stats

PG.info.stats is sent to mgr periodically for PG stats reporting,
also, it is sent to peer OSD in MOSDPGLog message when PG performs
peering. we need to keep it updated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add more peering facilities

* add PG::should_restart_peering() for telling if we should start
  a peering at seeing a new mapping
* add PG::start_peering_interval() for starting a peering
* add PG::activate() for activating a PG
* add PG::on_activated() to be called once a PG is activated
* add PG::send_to_osd() for sending message to given OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add operator<< for PG

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: add peer_activated

add facilities for updating peer_activated. it is used for tracking the
peers has activated and committed. once all of them ack the activation
proposed by primary, the PG is considered activated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for sending notify

we need to notify the primary osd that a PG hosted by it has changed,
and it needs to start peering for it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: maintain PG::want_acting

* add PG::choose_acting()
* add PG::proc_replica_info()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add PG::is_{acting,up}()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wait osdmap before processing peering evt

the constraint that

> peering messages/events from peers must remain ordered by peer

is still enforced. because, the peering message from a certain peer
will block the succeeding messages from this peer, if the latest osdmap
possessed by OSD is not new enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::_send_active()

so PG can request monitor to update the OSD's up_thru in osdmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGLog in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGQuery in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGInfo in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGNotify in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: keep track of last_peering_reset

last_peering_reset is used to track the epoch when the most recent peering
started. it is used to tell if a message is targeting a changed PG. the
sender of a peering message will attach it. and the recipient will check
it against its own last_peering_reset. for instance, if a message claims
that that its map epoch is 16, while our last_peering_reset is 18, we will
consider this message stale, and probably drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config_values: use string_view for keys

the contained char sequences are either static char strings or strings
in `Option` instances. so no need to keep another copy of string around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/ConfigMonitor: use MConfig(map&&)

less memcpy this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config,mon/ConfigMap: use transparent comparator

use transparent comparator as the comparator in the map of settings, so
we can find the values in a map<string,string> by string_view.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: let ConfigMap::generate_entity_map() return the map

C++17 enforces copy elision, so no need to pass the output parameter by
reference/pointer anymore. also, it would be easier if we change the
output's type by putting something like

auto output = config.generate_entity_map(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ConfUtil: use string_view as key when accessing settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: use string_view as key when accessing settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,librbd: use string_view for keys of config

the contained char sequences are either static char strings or strings
in `Option` instances. so no need to keep another copy of string around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix LRC documentation

Recovery from a failure in jerasure need only k reads and
not k+m-1.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw: Get rid of num_rados_handles infrastructure in RGWSI_RADOS

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Get rid of num_rados_handles infrastructure in RGWRados

Since we aren't supporting multiple handles, don't have the overhead.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove rgw_num_rados_handles option

This has been deprecated for some time and underlies much of the
complexity of the RADOS service.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: don't crash on missing /etc/mime.types

lack of mime types is not a fatal error. when a Content-Type header
is not provided in swift's PutObj, it uses this mime type mapping
to guess a content type based on the object's suffix

Fixes: http://tracker.ceph.com/issues/38328
Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Some devices appear like so:

E: ID_MODEL=LVM PV LUClYG-Oyte-jcM6-npfZ-ncsl-ycL0-bkOH0m on /dev/sdn
E: ID_MODEL_ENC=WDC\x20WDS200T2B0A-00SM50\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_SERIAL=WDC_WDS200T2B0A-00SM50_183503800168
E: ID_SERIAL_SHORT=183503800168

To avoid upsetting our prioritization, defer to ID_MODEL_ENC over ID_MODEL
if ID_MODEL has "LVM PV" in it (as opposed to just using ID_SERIAL).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27044 into master

* refs/pull/27044/head:
common: add bool log_to_file option

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd/OSDMap: add 'zone' to default crush map

This lets use have a root -> region -> zone -> host hierarchy, like you
will see in the public cloud.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: ensure only one copy of initial service map

It's possible for create_pending to be called multiple times.

Fixes: http://tracker.ceph.com/issues/38839
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27048 into master

* refs/pull/27048/head:
doc: add changelog for nautilus

Reviewed-by: Sage Weil <sage@redhat.com>

test/librbd: sequential write aio test

Signed-off-by: Mykola Golub <mgolub@suse.com>

tools: move options parsing into ObjectCacheStore for immutable obj cache daemon

This makes the options parsing more easy to read. SimplePolicy will get those
options from ObjectCacheStore.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

building: adding missing ceph-immtable-object-cache-dbg rule

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: fix SimplyPolicy memory leak

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: cleanup code style and delete useless data

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: new cache sub-dir rule for immutable obj cache daemon

do not use static sub-dirs for cache, instead we create the sub-dir
dynamically with the first two charactor of crc32(file_name)

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: cleanup RO

- replace useless callback with bind function.
- remove session_id and m_session_map.
- fixed memory leak when session occur error
- change type conversion.
- add prefix

Signed-off-by: Dehao Shang <dehao.shang@intel.com>