msg/async/dpdk:add the handling of DPDK initialization failure

If rte_eal_init returns with failure,the waiting msgr-worker thread is
woken up for exception handling.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #44404 from inspur-wyq/wip-doc-mgr-zabbix

doc/mgr/zabbix.rst: fix typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/mgr/zabbix.rst: fix typos

Signed-off-by: wangyunqing <wangyunqing@inspur.com>

Merge pull request #44378 from cyx1231st/wip-seastore-onode-cleanup

crimson/os/seastore: fix potential leak for onodes to live across transactions

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #44311 from ifed01/wip-ifed-cleanup-onode-pin

os/bluestore: get rid of fake onode nref increment for pinned entry

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #44303 from 5cs/fix-new-state-weight

mon/OSDMonitor: fix incorrect op between osd state and weight

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #44391 from anthonyeleven/anthonyeleven/correct-quay-url

doc/install: Update quay.io URL for ceph-ci builds

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/install: Update quay.io URL for ceph-ci builds

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #44352 from SMIL-Infra/journald

common: fix fmt::format_to deprecated warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42896 from ifed01/wip-ifed-bluefs-improve-log

os/bluestore: dump alloc unit size on bluefs allocation failure.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #44353 from tchaikov/wip-str-map

common/str_map: reimplement get_str_list() using for_each_pair

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44330 from rzarzynski/wip-osd-drop-includes

osd: drop unnecessary recovery / scrub includes from OSD.cc.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44364 from rhcs-dashboard/e2e-fixups

mgr/dashboard: fix timeout error in dashboard cephadm e2e job

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #44322 into master

* refs/pull/44322/head:
mds: skip directory size checks for reintegration
qa: test reintegration with directory limits

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #44375 from wjwithagen/wjw-fix-missing-utility

common: add missing #include <utility>

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/dashboard: fix timeout error in dashboard cephadm e2e job

1. Fix the timeout error happening in the dashboard e2e job
2. Take care of the flaky force maintenance check

Most of the time our test is getting timed out while searching for an item
in the table. Its because `.clear().type()` is not clearing the content
in the search field sometimes and that creates a wrong data to be
entered into the search field and it starts searching based on this
wrong name. To avoid this I am explicitly clearing the search area
before typing.

Fixes: https://tracker.ceph.com/issues/53672
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44365 from tchaikov/wip-seastar

seastar: pick up upstream change which includes cryptopp fix

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #44324 into master

* refs/pull/44324/head:
qa: set quincy require-osd-release to avoid health warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #44313 into master

* refs/pull/44313/head:
mds: support '~mds{rank number}' for dump tree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43881 into master

* refs/pull/43881/head:
osdc: add set_error in BufferHead, when split set_error to right

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44372 from DimitriPapadopoulos/codespell_examples

examples: typos found by codespell

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44241 from kamoltat/wip-ksirivad-pool-bulk-flag

mon: osd pool create <pool-name> with --bulk flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44385 from trociny/wip-53677

qa/tasks: improve backfill_toofull test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44360 from SMIL-Infra/fix-ios-frontend

mgr/dashboard: fix white screen on Safari

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: huww98 <NOT@FOUND>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43598 from ideepika/wip-opentelemetry

migrate from using opentracing-cpp to opentelemetry-cpp static as distributed tracing API

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44334 from aclamk/wip-aclamk-better-bluestore-perf

os/bluestore: Better readability of perf output

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

examples: typos found by codespell

Signed-off-by: Dimitri Papadopoulos <dpo.ceph@mail.com>

Merge pull request #44317 from aclamk/aclamk-fix-bluefs-import

Fix ceph-bluestore-tool bluefs-import command

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44328 from yaarith/gather_crashinfo-indexerror

mgr/telemetry: check that backtrace is not empty when redacting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44327 from yaarith/gather_device_report-indexerror

mgr/telemetry: catch also IndexError in gather_device_report()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #44314 from locallocal/master

osd: remove unused variable legacy_infos_oid in the function PG::peek_map_epoch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43686 from myoungwon/wip-ceph-dedup-tool-make-dedup-obj

tool: add  chunk/object-dedup command

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/tasks: improve backfill_toofull test

1) Write more data to the pool so we operate with larger ratios.
2) Round up ratios when truncating.

Fixes: https://tracker.ceph.com/issues/53677
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44190 from rhcs-dashboard/grafana-regex

monitoring/grafana: improve grafana unit tests variable substitution

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44362 from rhcs-dashboard/kcli-e2e-improvements

mgr/dashboard: cephadm e2e job: improvements

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43838 from rhcs-dashboard/fix-53128-master

mgr/dashboard: Cluster Expansion - Review Section: fixes and improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore: fix potential leak for onodes to live across transactions

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43942 from rhcs-dashboard/host-drain-frontend

mgr/dashboard: Implement drain host functionality in dashboard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Implement drain host functionality in dashboard

Fixes: https://tracker.ceph.com/issues/51587
Signed-off-by: Nizamudeen A <nia@redhat.com>

common: Fix missing utility include

See:
    https://en.cppreference.com/w/cpp/utility/move

Detected on FreeBSD/Clang/libc++:
/home/jenkins/workspace/ceph-master-compile/src/common/deleter.h:111:43: error: no member named 'move' in namespace 'std'
  impl(deleter next) : refs(1), next(std::move(next)) {}

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Added ReleasesNotes and documentation

Add Release Notes and remove any `profile`
related stuff in the autoscaler documentation
and replace it with `bulk` flag.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #42750 from aclamk/wip-aclamk-bluefs-log-incremental

os/bluestore: incremental update mode for bluefs log

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44368 from ceph/fix-api-tests

mgr/dashboard: run-backend-api-tests.sh: Older setuptools

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

os/bluestore/bluefs: Add tracking of bluefs log in noop replay mode

Keep updating bluefs log when printing content of bluefs replay log.
Without this modification we only have initial content of log.
Log can be printed by 'ceph-bluestore-tool bluefs-log-dump'.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore/bluefs: Sync BlueFS log with its allocation delta

BlueFS log is the only file that we can append to.
When we append to file we must take into consideration previously commited allocations,
otherwise update will be miscalculated.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

test/objectstore/bluefs_test: Add test for continuation of previous BlueFS log

Added test that verifies that in update mode we properly pick up delta.
BlueFS log is the only file that can be appended to, but it is done in very indirect way.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluefs: allow incremental file metadata updates in bluefs log

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

qa/workunits/cephtool: modified test.sh

Included bulk flag in the test script
such it tests the `get` and `set` of
the flag

Signed-off-by: Kamoltat <ksirivad@redhat.com>

qa/workunits/mon/pg_autoscaler: modified test script

Modified test scrtipt to include `bulk` and
remove all `profile` options.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

pg_autoscaler/test: Modified unit-test for bulk flag

Modified the unit-test cases to account for bulk
flag and remove any `profile` related things.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

mon: osd pool create <pool-name> with --bulk flag

Creating the pool with `--bulk` will allow
the pg_autoscaler to use the `scale-down`
mode on.

Creating pool:

`ceph osd pool create <pool-name> --bulk`

Get var:

`ceph osd pool get <pool-name> bulk`

Set var:

`ceph osd pool set <pool-name> bulk=true/false/1/0`

Removed `autoscale_profile` and incorporate bulk flag
into calculating `final_pg_target` for each pool.

bin/ceph osd pool autoscale-status no longer has
`PROFILE` column but has `BULK` instead.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge PR #44315 into master

* refs/pull/44315/head:
doc/cephfs: mds default cache memory limit is now 4GB

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: Older setuptools

https://github.com/pypa/setuptools/issues/2353

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #42416 from galsalomon66/wip-test-albin-error-status

RGW/s3select : new-s3select-responds presto-alignments output-serialization

seastar: pick up upstream change which includes cryptopp fix

update seastar submodule to upstream master, so we don't need to
use our own fork.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

os/bluestore: get rid of fake onode nref increment for pinned entry

Looks like this isn't necessary any more after fixing
https://tracker.ceph.com/issues/53002

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

mgr/dashboard: cephadm e2e job: improvements

- Avoid jenkins env. error: Input file /workdir/./src/pybind/mgr/dashboard/ci/cephadm/ceph_cluster.yml not found.Leaving...
- More meaningful log output.

Fixes: https://tracker.ceph.com/issues/53666
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Cluster Expansion - Review Section: fixes and improvements

Ensure "Storage capacity" keeps the "Description : Value" approach ("Number of devices: X" and "Raw Capacity: Y" in different lines).Correct issue with "host by services" host count

Fixes: https://tracker.ceph.com/issues/53128
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: fix white screen on Safari

Safari do not support lookbehind in regular expression.

Fixes: https://tracker.ceph.com/issues/53665
Signed-off-by: 胡玮文 <huww98@outlook.com>

Merge pull request #44350 from Huber-ming/fix_annotation_misspelling

rgw: fix misspelling of 'effort'

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

common: fix fmt::format_to deprecated warning

Signed-off-by: 胡玮文 <huww98@outlook.com>

mgr: move DaemonState methods into .cc

for faster compilation, and for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr: refactor DaemonState::set_metadata()

restructure DaemonState::set_metadata() using for_each_pair() for better
readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

common/str_map: reimplement get_str_list() using for_each_pair

to avoid creating a temporary list<string> and then dropping it
on the floor after iterating through it for collecting the kv
pairs in it.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44277 from fengchunsong/dpdk-rte-exit

msg/async/dpdk: replacing rte_exit with ceph_assert to avoid exit fai…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

using fmt for message formatting

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

fix comments; enum-class; typo; documentation

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

modification for error-handling and boto3

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

adding/modifying documentation for output-serialization, s3select-response, syntax support, error-handling

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3select: point to s3select/master(after merging pr #90)

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

in the case Presto is push-down query, it also send output-serialization parameter('&#10;'), that should be handle as '\n'

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3tests.yaml points to ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

RGW: Implement continuation, progress, stats, end s3select response

    RGW/S3select: Implement output-serializationi. user may request different CSV defintions
    for output (field delimiter, row delimiter, quote handling.

    RGW/S3select: Implement presto-alignments. presto-application sends
    queries with table-alias,case insensitive, and with no-semicolon at the
    end of statement.

    RGW/s3select: zero object size issue

Signed-off-by: Albin Antony <aantony@redhat.com>
Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

rgw: fix misspelling of 'effort'

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge pull request #43139 from josephsawaya/rook-orch-qa

qa/tasks/rook: test reapplication of drive groups stored in mgr

Merge PR #44337 into master

* refs/pull/44337/head:
mon: prevent new sessions during shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43872 from Huber-ming/reshard

rgw: init bucket index only if putting bucket instance info succeeds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43817 from joke-lee/UploadPartCopy-src-not-exist

rgw: fix UploadPartCopy error code when src object not exist and src bucket not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43259 from paulreece42/wip-rgw-xmlfix

rgw: allow CompletedMultipartUpload in XML

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41119 from Huber-ming/ls_mp

rgw: fix error ListMultipart allways returns StorageClass:STANDARD

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44336 from sebastian-philipp/cephadm-fix-test-facts

mgr/cephadm: Fix test_facts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44338 from rzarzynski/wip-crimson-abort_on_LOST_REVERT

crimson/osd: explicitly abort on LOST_REVERT in PG log.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44340 from rzarzynski/wip-crimson-backfill-discarding

crimson/osd: honor discarding old backfill messages

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44345 from xxhdx1985126/wip-onode-root-fix

crimson/os/seastore: reset onode in 'SeaStore::repeat_with_onode' before the transaction gets destroyed

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: reset onode in 'SeaStore::repeat_with_onode' before the transaction gets destroyed

Onodes hold references to the onode tree extents. And if it's referencing the root extent, that root
extent is cached in the onode trees root_tracker which caches onode tree roots by transaction address.
Than root_tracker entry only gets removed when the onode(or the corresponding "super") is destroyed.
On the other hand, two non-concurrent transactions can occupy the same address. So if an onode gets destroyed
after its transaction is destroyed, there will be a chance that another transaction occupying the same
address get that not-yet-destroyed and may-be-outdated onode.

BTW, Since we already cache extents in transactions, might want to drop onode tree root_tracker later?

Fixes: https://tracker.ceph.com/issues/53651
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction: make sure the transaction's write_set is a subset of its read_set

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

src/vstart.sh: correct shell syntax error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge PR #44228 into master

* refs/pull/44228/head:
qa/suites/orch/cephadm/osds: test 'ceph cephadm osd activate'
mgr/cephadm/services/osd: skip found osds that already have daemons
mgr/cephadm: allow activation of OSDs that have previously started

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa/tasks/rook: test reapplication of drive groups stored in mgr

This commit adds testing for the drive_group_loop in the Rook orchestrator
that reapplies drive groups that were applied previously.

This test removes an OSD, zaps the underlying device then waits for the OSD
to be re-created by the drive_group_loop.

This commit also updates the rook test suite to test v1.7.2 instead of 1.7.0
since `orch device zap` is only supported from v1.7.2 onwards.

Fixes: https://tracker.ceph.com/issues/53501
Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

qa/suites/orch/cephadm/osds: test 'ceph cephadm osd activate'

Make sure this command behaves when the /var/lib/ceph osd.NNN dir is
removed.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42009 from pritha-srivastava/wip-rgw-sts-correct-read-obj-perm

rgw/sts: fix read_obj_policy permission evaluation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #44331 from ronen-fr/wip-rf-tips187

crimson/osd: removing an unneeded make_unique()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement op discarding for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: generalize PG::old_peering_msg() to handle query_epoch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dissect the lpr check into PG::old_peering_msg().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #44280 from chrisphoffman/warn_pg

mon: Omit MANY_OBJECTS_PER_PG warning when autoscaler is on

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/osd: explicitly abort on LOST_REVERT in PG log.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43901 from pcuzner/snmp-notifier

mgr/cephadm: Add snmp-gateway service support

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Fix test_facts

Wasn't executed before

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>