osd: Fixes for 64-bit PG state

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27143 from dzafman/wip-38896

doc: Fix the pg states and auto repair config options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix the pg states and auto repair config options

Fixes: http://tracker.ceph.com/issues/38896
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27140 from neha-ojha/wip-fix-cron

qa/crontab/teuthology-cronjobs: fix suite-branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #27129 from yuriw/wip-yuriw-subsets-master

qa/tests: added the subset clause for nautilus branch

qa/crontab/teuthology-cronjobs: fix suite-branch

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #26939 into master

* refs/pull/26939/head:
os/bluestore: unconditionally cap chunks returned by allocator to 2^31
os/bluestore: start using 64-bit intervals for bitmap allocator
os/bluestore: make bluestore interval base template.
tests/fastbmap_alloc: UT to reproduce 4G allocation bug
os/bluestore: os/bluestore: implement dump for bitmap allocator
os/bluestore be more tolerant to lack of space for bluefs.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27086 into master

* refs/pull/27086/head:
msg: remove unused header file in Messenger.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27111 into master

* refs/pull/27111/head:
pybind/rados: should pass "name" to cstr()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27112 into master

* refs/pull/27112/head:
qa/suites: do not test luminous-x upgrade path

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27047 from tianshan/fix_38811

rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27102 from adamemerson/wip-one-rados-handle-to-rule-them-all

One Rados Handle to Rule Them All

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25999 from cbodley/wip-37945

rgw: Fix S3 compatibility bug when CORS is not found

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26998 from cbodley/wip-38328

rgw: don't crash on missing /etc/mime.types

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26952 from cbodley/wip-rgw-bucket-tagging-notsupp

rgw: reject bucket tagging requests and document unsupported

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26722 from cbodley/wip-rgw-sync-counters

rgw multisite: add perf counters to data sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27008 from ofriedma/wip-34308-beast-tcp-nodelay

rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27105 from neha-ojha/wip-21174

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26819 from tchaikov/wip-rpm-more-recommends

rpm: use Recommends on fedora also

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #27108 from tchaikov/wip-crimson-vstart.sh

 vstart.sh: add an option to use crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PGLog.h: print olog_can_rollback_to before deciding to rollback

This is particularly important as the crt might have changed before
being passed to merge_object_divergent_entries().

Fixes: http://tracker.ceph.com/issues/38894
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27083 from neha-ojha/wip-38635

qa/objectstore: test with reduced value of osd_memory_target

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

vstart.sh: pass osd specific options to OSD

--osd-args : for passing osd specific args

this allows us to pass seastar related options to crimson-osd

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

qa/tests: added the subset clause for nuatilis branch
--subset x/2999 should generate ~250 jobs as result

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites: do not test luminous-x upgrade path

in this change,

* suites/rados/upgrade: luminous-x-singleton => mimic-x-singleton
* suites/upgrade: luminous-x => nautilus-x

we support upgrade from n to n+2 release. otherwise monitor refuses to
do so:

mon.a@-1(probing) e1 current monmap has min_mon_release 15 (luminous)
which is >2 releases older than me 15 (octopus), stopping.

Fixes: https://tracker.ceph.com/issues/38845
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: reject bucket tagging requests and document unsupported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25545 from zhouyuan/wip-ceph-immutable-obj-cache-daemon

tools: adding ceph level immutable obj cache daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26675 from trociny/wip-rbd-io-simple-scheduler

librbd: simple scheduler plugin for object dispatcher layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: Adding tcp_nodelay option to Beast

beast frontend option to set the TCP_NODELAY socket option to match the tcp_nodelay option in civetweb.

Fixes: https://tracker.ceph.com/issues/34308
Signed-off-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #27098 from ming416/master

mgr/dashboard: auth ttl expired error

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #27070 into master

* refs/pull/27070/head:
osd/OSDMap: add 'zone' to default crush map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27020 into master

* refs/pull/27020/head:
common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27021 into master

* refs/pull/27021/head:
msg: remove XioMessenger
qa/suites/rados/thrash-old-clients: add nautilus
qa/suites/rados/thrash-old-clients: add mimic v1 variant
qa/suites/rados/thrash-old-clients: add mimic
qa/suites/rados/thrash-old-clients: collapse msgr and client choice
qa: remove simplemessenger tests
ceph_test_msgr: remove simple
msg: remove SimpleMessenger

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27101 into master

* refs/pull/27101/head:
mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: auth ttl expired error

Fixes: https://tracker.ceph.com/issues/38428
Signed-off-by: ming416 <geenature@163.com>

Merge pull request #26940 from xiexingguo/wip-monc-add-con

mon/MonClient: weight-based mon selection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26955 from liewegas/wip-slow-add

crush: various fixes for weight-sets, the osd_crush_update_weight_set option, and tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26950 from xiexingguo/wip-mgr-fixes

mgr/Mgr: kill redundant sub_unwant call

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27068 from hjwsm1989/upmap-filter-out-underfull-buckets

crush: add root_bucket to identify underfull buckets

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27067 from SUSE/wip-fix-38788

cmake: remove cython 0.29's subinterpreter check during install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26923 from majianpeng/msg-optimize-check-loopback-con

msg/async/AsyncConnection: optimize check loopback connection.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27076 from MariusSchiffer/ceph-mgr-python3-fix

mgr/balancer: Python 3 compatibility fix

Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/rados: should pass "name" to cstr()

it's a regression introduced by 6cb23f9c

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add an option to use crimson-osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27106 from dalgaaf/dalgaaf-wip-fix-lrc-doc

doc: fix LRC documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27071 from tchaikov/wip-crimson-osd-peering

crimson/osd: add minimal state machine for PG peering

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: add minimal PG recovery FSM

after this change, PG is able to go clean as primary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: advance pg in consume_map()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: update info.stats

PG.info.stats is sent to mgr periodically for PG stats reporting,
also, it is sent to peer OSD in MOSDPGLog message when PG performs
peering. we need to keep it updated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add more peering facilities

* add PG::should_restart_peering() for telling if we should start
  a peering at seeing a new mapping
* add PG::start_peering_interval() for starting a peering
* add PG::activate() for activating a PG
* add PG::on_activated() to be called once a PG is activated
* add PG::send_to_osd() for sending message to given OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add operator<< for PG

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: add peer_activated

add facilities for updating peer_activated. it is used for tracking the
peers has activated and committed. once all of them ack the activation
proposed by primary, the PG is considered activated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add facilities for sending notify

we need to notify the primary osd that a PG hosted by it has changed,
and it needs to start peering for it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: maintain PG::want_acting

* add PG::choose_acting()
* add PG::proc_replica_info()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add PG::is_{acting,up}()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wait osdmap before processing peering evt

the constraint that

> peering messages/events from peers must remain ordered by peer

is still enforced. because, the peering message from a certain peer
will block the succeeding messages from this peer, if the latest osdmap
possessed by OSD is not new enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::_send_active()

so PG can request monitor to update the OSD's up_thru in osdmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGLog in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGQuery in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGInfo in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDPGNotify in OSD

will handle it in PG in follow-up changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: keep track of last_peering_reset

last_peering_reset is used to track the epoch when the most recent peering
started. it is used to tell if a message is targeting a changed PG. the
sender of a peering message will attach it. and the recipient will check
it against its own last_peering_reset. for instance, if a message claims
that that its map epoch is 16, while our last_peering_reset is 18, we will
consider this message stale, and probably drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix LRC documentation

Recovery from a failure in jerasure need only k reads and
not k+m-1.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw: Get rid of num_rados_handles infrastructure in RGWSI_RADOS

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Get rid of num_rados_handles infrastructure in RGWRados

Since we aren't supporting multiple handles, don't have the overhead.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove rgw_num_rados_handles option

This has been deprecated for some time and underlies much of the
complexity of the RADOS service.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: don't crash on missing /etc/mime.types

lack of mime types is not a fatal error. when a Content-Type header
is not provided in swift's PutObj, it uses this mime type mapping
to guess a content type based on the object's suffix

Fixes: http://tracker.ceph.com/issues/38328
Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Some devices appear like so:

E: ID_MODEL=LVM PV LUClYG-Oyte-jcM6-npfZ-ncsl-ycL0-bkOH0m on /dev/sdn
E: ID_MODEL_ENC=WDC\x20WDS200T2B0A-00SM50\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_SERIAL=WDC_WDS200T2B0A-00SM50_183503800168
E: ID_SERIAL_SHORT=183503800168

To avoid upsetting our prioritization, defer to ID_MODEL_ENC over ID_MODEL
if ID_MODEL has "LVM PV" in it (as opposed to just using ID_SERIAL).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27044 into master

* refs/pull/27044/head:
common: add bool log_to_file option

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd/OSDMap: add 'zone' to default crush map

This lets use have a root -> region -> zone -> host hierarchy, like you
will see in the public cloud.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: ensure only one copy of initial service map

It's possible for create_pending to be called multiple times.

Fixes: http://tracker.ceph.com/issues/38839
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27048 into master

* refs/pull/27048/head:
doc: add changelog for nautilus

Reviewed-by: Sage Weil <sage@redhat.com>

test/librbd: sequential write aio test

Signed-off-by: Mykola Golub <mgolub@suse.com>

tools: move options parsing into ObjectCacheStore for immutable obj cache daemon

This makes the options parsing more easy to read. SimplePolicy will get those
options from ObjectCacheStore.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

building: adding missing ceph-immtable-object-cache-dbg rule

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: fix SimplyPolicy memory leak

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: cleanup code style and delete useless data

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: new cache sub-dir rule for immutable obj cache daemon

do not use static sub-dirs for cache, instead we create the sub-dir
dynamically with the first two charactor of crc32(file_name)

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: cleanup RO

- replace useless callback with bind function.
- remove session_id and m_session_map.
- fixed memory leak when session occur error
- change type conversion.
- add prefix

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: eliminate session_map race between CacheSession

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: make cache file dir number configurable

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: make cache watermark configurable

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: make dedicated thread configurable for RO

At cache client side of RO, make dedicated woker thread configurable
Also, cleanup useless member of CacheClient class

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: adjust code style of RO

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: clean up data structure of immutable obj cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: refactor ObjectCacheRequest of RO

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: return real cache status on lookup in immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: fix librados compatibility

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: clean up cache dir for immutable obj cache

- user defined full path for cache dir
- hash objects into subfolders with crc32 of its name

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: maintain cache size counter for immutable obj cache daemon

Do not assume 4M object size since we allow to cache any-sized objects.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: templatize create_rados_callback

Also cleanup unused template codes

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: use specific message for different ops in immutable obj cache daemon

use different types of message for different ops to be more efficient

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: cleanup IPC message for immutable obj cache daemon

using ceph serialization(encode/decode) framework instead of building
new data structure

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: set namespace & snap_id before promote from RADOS in immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: format code and clean up

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: adding signal handling for immutable obj cache daemon

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tests: reduce testing resources occupied for immutable obj cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

tools: follow current codes to modify UT

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: release dedicated thread resource

Addtional, enable all UT

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: add connect fails to fault

Signed-off-by: Dehao Shang <dehao.shang@intel.com>