crimson/os/seastore: change Scanner to ExtentReader

This commit makes Scanner an extent reader that route read requests to the corresponding
backing devices according to the device ids encapsulated in the segment ids.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: make segment managers device-id aware

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: introduce device id into segment_id_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43166 from rzarzynski/wip-crimson-no-subproc

 crush, crimson: don't support crush_location_hook as crimson lacks SubProcess

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43346 from hjwsm1989/fix-52765

tools/ceph-kvstore-tool: fix segfaults when repair the rocksdb

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43452 from adk3798/agent-perf1

cephadm: agent: subtract average time of previous iterations off wait time

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43401 from romoh/master

src/cephadm: Add support for mariner

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge pull request #43443 from tchaikov/wip-elapsed-in-double

msg/async/ProtocolV2: cast usec to double when converting from nsec

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: dongdong tao <dongdong.tao@canonical.com>

Merge pull request #43343 from trociny/wip-crushdiff-compat

tools/crushdiff: support old format json dump

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43324 from neha-ojha/wip-52640

common/options/global.yaml.in: add runtime flag for mon_max_pg_per_osd

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #43232 from Rethan/feat-bluestore-fail-eio

os/bluestore: add option for bluestore fail eio

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #43163 into master

* refs/pull/43163/head:
qa: fsync dir for asynchronous creat on stray tests
qa: refactor and generalize create_n_files
qa: only set frag confs for workloads
mds: improve debugging for fragment size check

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #43462 from Zhiwei-Dai/wip-test-erasure-code-for-aarch64

qa: support isal ec test for aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43444 from idryomov/wip-read-from-replica-comment

librados,librbd: make it clear that replica reads are safe for general use

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43137 from rhcs-dashboard/fix-52480-master

mgr/dashboard: clean-up controllers

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43459 from cyx1231st/wip-seastore-optimize-record-overhead

crimson/os/seastore: reduce write amplification from record overhead and implement placement hint

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa: support isal ec test for aarch64

modified:   qa/standalone/erasure-code/test-erasure-code-plugins.sh
new file:   qa/suites/rados/thrash-erasure-code-isa/arch/aarch64.yaml

Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>

crimson/os/seastore: implement placement_hint HOT/COLD/REWRITE

The current strategy is to inline allocate non-REWRITE extents in order
to reduce the record overhead. The observation shows the transactions
from user is usually very small and doesn't deserve to be written in ool
records, which requires at least a block to store the record metadata.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: rename ool_placement_hint_t

To placement_hint_t as the hint is not limited to ool extents.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43248 from inspur-wyq/wip-doc

doc/rbd/vault.rst: fix typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43373 from gregsfortytwo/wip-mon-stretch-new-tiebreaker-52126

mon: Allow specifying new tiebreaker monitors

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43331 from adk3798/agent-drain

mgr/cephadm: fix host drain with agent

Reviewed-by: Nizamudeen A <nia@redhat.com>

qa: fsync dir for asynchronous creat on stray tests

Use the enhanced create_n_files to dedup code. Also split the large test
into three.

Fixes: https://tracker.ceph.com/issues/52606
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: refactor and generalize create_n_files

Few things:

- Allow calling fsync on directory (to support async create kernel).
- Allow immediately unlinking the created file (for stray testing).
- Close any file descriptors created.
- Write unique content (the i variable) to each file.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: only set frag confs for workloads

Otherwise these local conf overrides prevent functional testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: improve debugging for fragment size check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43442 from linuxbox2/wip-rgwadmin-logfix

qa/rgw: fix ops log tests to handle non-bucket ops (which are now valid)

Merge pull request #43448 from rzarzynski/wip-crimson-monmap-moredbgs

crimson, monmap: inform about errors when interacting with DNS

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cephadm: agent: subtract average time of previous iterations off wait time

We want the agent to actually report metadata at the rate we set
it for. Before this, that rate was just being used as the wait time
between iterations so the actual time between iterations was the
given interval plus the time to gather metadata. Now the time between
reports should actually be roughly the given interval.

Signed-off-by: Adam King <adking@redhat.com>

msg/async/ProtocolV2: cast usec to double when converting from nsec

to be consistent with the same logic in V1, and more importantly, it is
more correct.

the change was suggested by Ilya Dryomov <idryomov@redhat.com>.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #43231 into master

* refs/pull/43231/head:
qa: fix promotion test

Reviewed-by: Ramana Raja <rraja@redhat.com>

crimson, monmap: inform about errors in MonMap::init_with_dns_srv().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, monmap: limit the scope of the seastar namespace inclusion.

More consistency that way. This also enables us to define the usual
`logger()` function without having a clash with `seastar::logger`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43384 from bosc0/master

mgr: Add check to prevent mgr from crashing

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41445 from guits/cephadm_quick_patch

cephadm: add daemon_name in daemon description

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43335 from liewegas/debug-51815

mon,auth: fix proposal (and mon db rebuild) of rotating secrets

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43379 from myoungwon/wip-evict-range

osd: trim corresponding chunks on tier_evict

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43378 from ivancich/wip-build-without-mgr

rados: build minimally when "WITH_MGR" is off

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43360 from jianwei1216/fix-shard-threads-cannot-wakeup-bug

osd: fix shard-threads cannot wakeup bug

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43307 from taodd/recv_stamp_fix

msg/async/ProtocolV2: Set the recv_stamp at the beginning of receiving a message

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Dan Hill <daniel.hill@canonical.com>

Merge pull request #43380 from myoungwon/wip-dont-promote-dirty

osd: do not promote objects if op is is_dirty

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43435 from rzarzynski/wip-crimson-pickv2only

crimson/osd: pick only v2 addresses.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

librbd: clarify rbd_read_from_replica_policy vs older options

If rbd_read_from_replica_policy is set appropriately, setting
rbd_balance/localize_snap_reads or rbd_balance/localize_parent_reads
is redundant.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librados: OPERATION_BALANCE/LOCALIZE_READS comment is outdated

With the introduction of min_last_complete_ondisk on the OSD side,
it is safe to use these for non-immutable data.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/rgw: fix ops log tests to handle non-bucket ops (which are now valid)

After 3863eb89512f1698b8e56f1f1ffc78a6ca8d5826--rgw: permit logging of
list-bucket (and any other no-bucket op-- the radosgw ops-log
contains entries for ops with no associated buckets--e.g., list_buckets.
When examining such a log object in the radosgw_admin task, don't assert
that it has any bucket name.

Fixes: https://tracker.ceph.com/issues/52647
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #43256 from rhcs-dashboard/fix-48845-master

qa/mgr/dashboard/test_pool: don't check HEALTH_OK

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: clean-up controllers

Fixes: https://tracker.ceph.com/issues/52589
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42964 from trociny/wip-52448

osd: re-cache peer_bytes on every peering state activate

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/osd: pick only v2 addresses.

Otherwise we'll run into an assertion failure when binding:

```
INFO  2021-10-05 15:14:51,075 [shard 0] osd - picked address v2:0.0.0.0:0/0
INFO  2021-10-05 15:14:51,075 [shard 0] osd - picked address v1:0.0.0.0:0/0
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-8025-geb9c0f59/rpm/el8/BUILD/ceph-17.0.0-8025-geb9c0f59/src/crimson/net/SocketMessenger.cc:89: crimson::errorator<crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<std::errc::address_in_use>))>, crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<std::errc::address_not_available>))> >::future<> crimson::net::SocketMessenger::try_bind(const entity_addrvec_t&, uint32_t, uint32_t): Assertion `addrs.size() == 1' failed.
Aborting on shard 0.
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #43399 into master

* refs/pull/43399/head:
mgr/volumes/fs/operations: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43398 into master

* refs/pull/43398/head:
mgr/volumes/fs: fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43397 into master

* refs/pull/43397/head:
mgr/volumes/fs: fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43395 into master

* refs/pull/43395/head:
mgr/volumes/fs/operations/versions: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43392 into master

* refs/pull/43392/head:
mgr/volumes/fs/operations: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43391 into master

* refs/pull/43391/head:
mgr/volumes/fs/vol_spec.py: fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43390 into master

* refs/pull/43390/head:
mgr/volumes/fs/operations : Fix flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43387 into master

* refs/pull/43387/head:
mgr/volumes: fix various flake8 issues

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43375 into master

* refs/pull/43375/head:
mgr/volumes/fs: add extra blank line

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>

Merge PR #43357 into master

* refs/pull/43357/head:
mgr/nfs: stick to lazy evaluation of logger messages
mgr/nfs: change _cmd_rgw_export_create_rgw() name
mgr/nfs: don't log fsal keys
mgr/nfs: Add more debug log messages

Reviewed-by: Varsha Rao <rvarsha016@gmail.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

qa: fix promotion test

The test is not needing to check that the new MDS becomes active, only
that a replacement occurs.

Fixes: https://tracker.ceph.com/issues/52677
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42977 from amathuria/wip-amathuri-scrub-duration

osd/scrub: Add scrub duration to pg dump stats

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

osd: wake up more shard workers when new osdmap is consumed

Reproduce:
(1) ceph cluster not running any client IO
(2) only ceph osd in osd.14 operation

Reason:
(1) one shard-queue has three shard-threads
(2) one or some PeeringOp's epoch > osdmap's epoch held by current osd,
    and these PeeringOp _add_slot_waiter()
(3) shard-queue become empty and three shard-threads cond.wait()
(4) new osdmap consume and it _wake_pg_slot()
    Problem in here
1> OSDShard::consume() exec loop all pg's slot wait
     and requeue more than one PeeringOp to shard-queue
        2> but it only notify one shard-thread to wakeup,
           the other two shard-threads continue cond.wait()
3> OSD::ShardedOpWQ::_enqueue() found the shard-queue not empty
   and not notify all shard-thread to wakeup

     In a period of time, only one shard-thread of 3 shard-threads is running.

Fixes: https://tracker.ceph.com/issues/52781
Signed-off-by: Jianwei Zhang <jianwei1216@qq.com>
Change-Id: I4617db2fd95082007e6d9fa2b60f17f2a6296b5b

Merge pull request #43175 from ideepika/wip-supress-jaeger-cmake-logs

cmake,rgw: use jaeger-base target, as a dependency for building dbstore

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: update stretch mode documentation for new commands

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: dump tiebreaker_mon in MonMap

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: udpate MonMap::last_changed on stretch commands which didn't

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: remove deleted monitors from the disallowed_leaders set

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: don't let users remove the tiebreaker monitor from the map

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: add "mon set_new_tiebreaker" command, so stretch clusters can replace it

Fixes: https://tracker.ceph.com/issues/52126
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/cephadm: Add mariner as a supported distro.

Using cephadm install commands on Mariner will result in an error
"Distro not supported" though the packages are supported and can be
installed manually. This patch adds mariner as a supported distro.
Fix validated on mariner.

Signed-off-by: Roaa Sakr <romoh@microsoft.com>

Merge pull request #43410 from zdover23/wip-doc-2021-10-04-min-hardware-typo

docs/start: fix min. hardware rec. typo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

docs/start: fix min. hardware rec. typo

The minimum hardware recommendations said that we
recommended a minimum of "24GB+ per daemon".

I changed this to read "2-4GB+ per daemon".

Reddit actually caught this one:
https://www.reddit.com/r/ceph/comments/q14ey2/cephs_mon_minimum_requirements/

Signed-off-by: Zac Dover <zac.dover@gmail.com>

cmake,rgw: use jaeger-base target, as a dependency for building dbstore

* dbstore also has tracing dependencies and missing linking for libraries
* use `using` keyword in common/tracer.cc to silence unused method
  warnings
* log building of external project jaegertracing, opentracing into build
  file, (done to reduce external project build verbosity)

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43402 from liewegas/doc-general-rep

doc/foundation: Martin is new general member rep

Merge PR #43383 into master

* refs/pull/43383/head:
doc: fixed typo in terminology

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/foundation: Martin is new general member rep

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/volumes/fs/operations: fix various flake8 issues

Fixes: https://tracker.ceph.com/issues/51390
Signed-off-by: Zoey <jingyasu@my.yorku.ca>

mgr/volumes/fs/vol_spec.py: fix flake8 issues

Fixes: https://tracker.ceph.com/issues/51384
Signed-off-by: Eunice Lee <eunice337@gmail.com>

mgr/volumes/fs: fix flake8 issues

Fixes: https://tracker.ceph.com/issues/51407
Signed-off-by: Hargun Kaur <hargunalone@gmail.com>

mgr/volumes/fs/operations/versions: fix various flake8 issues

Fixes: https://tracker.ceph.com/issues/51403
Signed-off-by: Gokcen Iskender <gokceng@gmail.com>

mgr/volumes/fs: fix flake8 issues

Fixes: https://tracker.ceph.com/issues/51381
Signed-off-by: Eunice Lee <eunice337@gmail.com>

mgr/volumes/fs/operations: fix various flake8 issues

Fixes: https://tracker.ceph.com/issues/51398
Signed-off-by: Gokcen Iskender <gokceng@gmail.com>

mgr/volumes/fs/operations : Fix flake8 issues
Fixes : https://tracker.ceph.com/issues/51396

Signed-off-by: Anuradha Kulkarni <anuradhak1990@gmail.com>

mgr/volumes: fix various flake8 issues

Fixes: https://tracker.ceph.com/issues/51380
Signed-off-by: Almen Ng <almen000509@gmail.com>

ceph-monstore-tool: include rotating keys in rebuilt mon

This avoids an initial period after mon restart in which there are no
rotating keys and authentication can fail.

Signed-off-by: Sage Weil <sage@newdream.net>

mon,auth: fix proposal of rotating keys

Instead of updating the live CephxKeyServer's rotating_keys and also
including them in a paxos proposal, propose new keys only in the proposal,
and only make them live once they are committed.  This keeps mons fully in
sync and avoids any inconsistency between the live behavior and committed
state (e.g., stale or divergent keys being applied and passed out to
daemons).

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/singleton/rebuild-mon-db: debug auth 30

Hunting https://tracker.ceph.com/issues/51815

Signed-off-by: Sage Weil <sage@newdream.net>

mon,auth: debug missing service secrets

Hunting https://tracker.ceph.com/issues/51815

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41255 from rhcs-dashboard/angular-11-upgrade

mgr/dashboard: Angular v11 Migration

Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42716 from sshambar/bug-52002

mgr/dashboard,prometheus: fix handling of server_addr

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: sshambar <NOT@FOUND>

Merge pull request #43277 from anthonyeleven/anthonyeleven/fix-47839

ceph: add example BlueStore migration via the ceph-objectstore-tool dup op

Reviewed-by: Zac Dover <zac.dover@gmail.com>

MetricCollector.h: Add check to prevent mgr from crashing

Fixes: https://tracker.ceph.com/issues/52801
Signed-off-by: Aswin Toni <aswin.toni@cern.ch>

doc: fixed typo in terminology

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/scrub: Add scrub duration to pg dump stats

Addition of a new column, SCRUB_DURATION, to the pg stats that stores the time taken for a PG scrub.

Fixes: https://tracker.ceph.com/issues/52605
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>

mgr/dashboard: Remove i18n Warning about duplicate id

Fixes: https://tracker.ceph.com/issues/50268
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Taking care of deprecation warnings and other errors

1. deprecation  group is deprecated: This API is not typesafe and can result in issues with Closure Compiler renaming. Use the `FormBuilder#group` overload with `AbstractControlOptions` instead.

2. cypress/integration/rgw/users.po.ts(113,23): error TS2345: Argument of type 'void' is not assignable to parameter of type 'string'.

3. Cannot find module 'isomorphic-form-data' from 'node_modules/swagger-ui/dist/swagger-ui.js'

Fixes: https://tracker.ceph.com/issues/50268
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Angular v11 Migration

Fixes: https://tracker.ceph.com/issues/50268
Signed-off-by: Nizamudeen A <nia@redhat.com>

osd: do not promote objects if op is is_dirty

manifest object contains object_info_t even if the object
is flushed, so OSD can reply the result of is_dirty.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: trim corresponding chunks on tier_evict

This commit trims chunks on the object even if the size of
the chunks is not equal to the object size.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>