crimson/os/seastore: clean up logging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44403 from cyx1231st/wip-fix-crimson-args

crimson/osd: fix and improve args processing

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44396 from cyx1231st/wip-seastore-fix-seastar-runner

crimson/test: fix SeastarRunner when app is not started

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44274 from fengchunsong/dpdk-dfx

msg/async/dpdk: add commands to obtain the NIC status and statistics

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

msg/async/dpdk:add commands to obtain the NIC status and statistics

Commands are added to obtain the network adapter status and statistics
for debugging network adapter packet loss and mbuf insufficiency issues.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #44276 from fengchunsong/dpdk-affinity

common/numa: Skip the DPDK thread when setting NUMA affinity

Merge PR #44342 into master

* refs/pull/44342/head:
mds: trigger stray reintegration when loading dentry
qa: test that scrub causes reintegration

Reviewed-by: Xiubo Li <xiubli@redhat.com>

common/numa: Skip the DPDK thread when setting NUMA affinity

The CPU affinity of the DPDK thread has been set during DPDK initialization.
Do not modify the DPDK affinity when setting NUMA affinity.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Han Fengzhe <hanfengzhe@hisilicon.com>

msg/async: refactory rename_thread for DPDKStack

The thread_name of the DPDK thread has been set during DPDK initialization.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #44243 from ideepika/wip-deepika-iscsi

qa/rbd: update the cephadm required distro

Reviewed-by: Melissa Li <melissali@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/osd: print message when customize parameters

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: allow seastar application to print its helps

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: --help-seastar does no longer belong to unknown_args

Now that app_template handles --help-seastar, and prints out all
app-level usages if --help is included in command line options. There is
no need to add a dedicated --help-seastar by ourselves and translate it
to --help.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44411 from tchaikov/wip-crimson-doc

doc: remove references to SEASTAR_DEFAULT_ALLOCATOR

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #44333 from rzarzynski/wip-crimson-fix-recovery-discarding

crimson/osd: implement op discarding for pglog-based recovery.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #44275 from fengchunsong/dpdk-nic

msg/async/dpdk: add NIC whitelist configuration

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44285 from fengchunsong/dpdk-rte-init

msg/async/dpdk: handle DPDK initialization failure properly

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

msg/async/dpdk:add the handling of DPDK initialization failure

If rte_eal_init returns with failure,the waiting msgr-worker thread is
woken up for exception handling.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Han Fengzhe <hanfengzhe@hisilicon.com>

doc: remove references to SEASTAR_DEFAULT_ALLOCATOR

now that https://github.com/scylladb/seastar/commit/067681b70172ab4a6676431b623e8bd0caab6dc8
has been merged upstream, and our src/seastar submodule includes
this commit. there is no need to worry that the seastar's lockless
allocator does not work with seastar::alien anymore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson/test: fix SeastarRunner when app is not started

Support the case when the SeastarRunner isn't able to start the app,
for example, when start with --help.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

msg/async/dpdk: add NIC whitelist configuration

Each DPDK process uses some exclusive network adapters.The network adapters to
be used are specified in the whitelist to prevent data sharing between multiple
DPDK process.The following is an example:
1)Configure a single NIC.
-a 0000:7d:010 or --allow=0000:7d:010
2)Configure the Bond Network Adapter
ms_dpdk_port_options=--allow=0000:7d:01.0 --allow=0000:7d:02.6
--vdev=net_bonding0,mode=2,slave=0000:7d:01.0,slave=0000:7d:02.6

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Reviewed-by: luo rixin <luorixin@huawei.com>
Reviewed-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #44404 from inspur-wyq/wip-doc-mgr-zabbix

doc/mgr/zabbix.rst: fix typos

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/mgr/zabbix.rst: fix typos

Signed-off-by: wangyunqing <wangyunqing@inspur.com>

Merge pull request #44378 from cyx1231st/wip-seastore-onode-cleanup

crimson/os/seastore: fix potential leak for onodes to live across transactions

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #44311 from ifed01/wip-ifed-cleanup-onode-pin

os/bluestore: get rid of fake onode nref increment for pinned entry

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #44303 from 5cs/fix-new-state-weight

mon/OSDMonitor: fix incorrect op between osd state and weight

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #44391 from anthonyeleven/anthonyeleven/correct-quay-url

doc/install: Update quay.io URL for ceph-ci builds

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/install: Update quay.io URL for ceph-ci builds

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #44352 from SMIL-Infra/journald

common: fix fmt::format_to deprecated warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42896 from ifed01/wip-ifed-bluefs-improve-log

os/bluestore: dump alloc unit size on bluefs allocation failure.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #44353 from tchaikov/wip-str-map

common/str_map: reimplement get_str_list() using for_each_pair

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44330 from rzarzynski/wip-osd-drop-includes

osd: drop unnecessary recovery / scrub includes from OSD.cc.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44364 from rhcs-dashboard/e2e-fixups

mgr/dashboard: fix timeout error in dashboard cephadm e2e job

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #44322 into master

* refs/pull/44322/head:
mds: skip directory size checks for reintegration
qa: test reintegration with directory limits

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #44375 from wjwithagen/wjw-fix-missing-utility

common: add missing #include <utility>

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/dashboard: fix timeout error in dashboard cephadm e2e job

1. Fix the timeout error happening in the dashboard e2e job
2. Take care of the flaky force maintenance check

Most of the time our test is getting timed out while searching for an item
in the table. Its because `.clear().type()` is not clearing the content
in the search field sometimes and that creates a wrong data to be
entered into the search field and it starts searching based on this
wrong name. To avoid this I am explicitly clearing the search area
before typing.

Fixes: https://tracker.ceph.com/issues/53672
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44365 from tchaikov/wip-seastar

seastar: pick up upstream change which includes cryptopp fix

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #44324 into master

* refs/pull/44324/head:
qa: set quincy require-osd-release to avoid health warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #44313 into master

* refs/pull/44313/head:
mds: support '~mds{rank number}' for dump tree

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #43881 into master

* refs/pull/43881/head:
osdc: add set_error in BufferHead, when split set_error to right

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44372 from DimitriPapadopoulos/codespell_examples

examples: typos found by codespell

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44241 from kamoltat/wip-ksirivad-pool-bulk-flag

mon: osd pool create <pool-name> with --bulk flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44385 from trociny/wip-53677

qa/tasks: improve backfill_toofull test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44360 from SMIL-Infra/fix-ios-frontend

mgr/dashboard: fix white screen on Safari

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: huww98 <NOT@FOUND>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43598 from ideepika/wip-opentelemetry

migrate from using opentracing-cpp to opentelemetry-cpp static as distributed tracing API

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44334 from aclamk/wip-aclamk-better-bluestore-perf

os/bluestore: Better readability of perf output

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

examples: typos found by codespell

Signed-off-by: Dimitri Papadopoulos <dpo.ceph@mail.com>

Merge pull request #44317 from aclamk/aclamk-fix-bluefs-import

Fix ceph-bluestore-tool bluefs-import command

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44328 from yaarith/gather_crashinfo-indexerror

mgr/telemetry: check that backtrace is not empty when redacting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44327 from yaarith/gather_device_report-indexerror

mgr/telemetry: catch also IndexError in gather_device_report()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #44314 from locallocal/master

osd: remove unused variable legacy_infos_oid in the function PG::peek_map_epoch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43686 from myoungwon/wip-ceph-dedup-tool-make-dedup-obj

tool: add  chunk/object-dedup command

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/tasks: improve backfill_toofull test

1) Write more data to the pool so we operate with larger ratios.
2) Round up ratios when truncating.

Fixes: https://tracker.ceph.com/issues/53677
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44190 from rhcs-dashboard/grafana-regex

monitoring/grafana: improve grafana unit tests variable substitution

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44362 from rhcs-dashboard/kcli-e2e-improvements

mgr/dashboard: cephadm e2e job: improvements

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43838 from rhcs-dashboard/fix-53128-master

mgr/dashboard: Cluster Expansion - Review Section: fixes and improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore: fix potential leak for onodes to live across transactions

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43942 from rhcs-dashboard/host-drain-frontend

mgr/dashboard: Implement drain host functionality in dashboard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Implement drain host functionality in dashboard

Fixes: https://tracker.ceph.com/issues/51587
Signed-off-by: Nizamudeen A <nia@redhat.com>

common: Fix missing utility include

See:
    https://en.cppreference.com/w/cpp/utility/move

Detected on FreeBSD/Clang/libc++:
/home/jenkins/workspace/ceph-master-compile/src/common/deleter.h:111:43: error: no member named 'move' in namespace 'std'
  impl(deleter next) : refs(1), next(std::move(next)) {}

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Added ReleasesNotes and documentation

Add Release Notes and remove any `profile`
related stuff in the autoscaler documentation
and replace it with `bulk` flag.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #42750 from aclamk/wip-aclamk-bluefs-log-incremental

os/bluestore: incremental update mode for bluefs log

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #44368 from ceph/fix-api-tests

mgr/dashboard: run-backend-api-tests.sh: Older setuptools

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

os/bluestore/bluefs: Add tracking of bluefs log in noop replay mode

Keep updating bluefs log when printing content of bluefs replay log.
Without this modification we only have initial content of log.
Log can be printed by 'ceph-bluestore-tool bluefs-log-dump'.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore/bluefs: Sync BlueFS log with its allocation delta

BlueFS log is the only file that we can append to.
When we append to file we must take into consideration previously commited allocations,
otherwise update will be miscalculated.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

test/objectstore/bluefs_test: Add test for continuation of previous BlueFS log

Added test that verifies that in update mode we properly pick up delta.
BlueFS log is the only file that can be appended to, but it is done in very indirect way.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluefs: allow incremental file metadata updates in bluefs log

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

qa/workunits/cephtool: modified test.sh

Included bulk flag in the test script
such it tests the `get` and `set` of
the flag

Signed-off-by: Kamoltat <ksirivad@redhat.com>

qa/workunits/mon/pg_autoscaler: modified test script

Modified test scrtipt to include `bulk` and
remove all `profile` options.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

pg_autoscaler/test: Modified unit-test for bulk flag

Modified the unit-test cases to account for bulk
flag and remove any `profile` related things.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

mon: osd pool create <pool-name> with --bulk flag

Creating the pool with `--bulk` will allow
the pg_autoscaler to use the `scale-down`
mode on.

Creating pool:

`ceph osd pool create <pool-name> --bulk`

Get var:

`ceph osd pool get <pool-name> bulk`

Set var:

`ceph osd pool set <pool-name> bulk=true/false/1/0`

Removed `autoscale_profile` and incorporate bulk flag
into calculating `final_pg_target` for each pool.

bin/ceph osd pool autoscale-status no longer has
`PROFILE` column but has `BULK` instead.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge PR #44315 into master

* refs/pull/44315/head:
doc/cephfs: mds default cache memory limit is now 4GB

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

pybind/mgr/dashboard/run-backend-api-tests: Older setuptools

https://github.com/pypa/setuptools/issues/2353

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #42416 from galsalomon66/wip-test-albin-error-status

RGW/s3select : new-s3select-responds presto-alignments output-serialization

seastar: pick up upstream change which includes cryptopp fix

update seastar submodule to upstream master, so we don't need to
use our own fork.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

os/bluestore: get rid of fake onode nref increment for pinned entry

Looks like this isn't necessary any more after fixing
https://tracker.ceph.com/issues/53002

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

mgr/dashboard: cephadm e2e job: improvements

- Avoid jenkins env. error: Input file /workdir/./src/pybind/mgr/dashboard/ci/cephadm/ceph_cluster.yml not found.Leaving...
- More meaningful log output.

Fixes: https://tracker.ceph.com/issues/53666
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: Cluster Expansion - Review Section: fixes and improvements

Ensure "Storage capacity" keeps the "Description : Value" approach ("Number of devices: X" and "Raw Capacity: Y" in different lines).Correct issue with "host by services" host count

Fixes: https://tracker.ceph.com/issues/53128
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: fix white screen on Safari

Safari do not support lookbehind in regular expression.

Fixes: https://tracker.ceph.com/issues/53665
Signed-off-by: 胡玮文 <huww98@outlook.com>

Merge pull request #44350 from Huber-ming/fix_annotation_misspelling

rgw: fix misspelling of 'effort'

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

common: fix fmt::format_to deprecated warning

Signed-off-by: 胡玮文 <huww98@outlook.com>

mgr: move DaemonState methods into .cc

for faster compilation, and for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr: refactor DaemonState::set_metadata()

restructure DaemonState::set_metadata() using for_each_pair() for better
readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

common/str_map: reimplement get_str_list() using for_each_pair

to avoid creating a temporary list<string> and then dropping it
on the floor after iterating through it for collecting the kv
pairs in it.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44277 from fengchunsong/dpdk-rte-exit

msg/async/dpdk: replacing rte_exit with ceph_assert to avoid exit fai…

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

using fmt for message formatting

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

fix comments; enum-class; typo; documentation

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

typo

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

modification for error-handling and boto3

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

adding/modifying documentation for output-serialization, s3select-response, syntax support, error-handling

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3select: point to s3select/master(after merging pr #90)

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

in the case Presto is push-down query, it also send output-serialization parameter('&#10;'), that should be handle as '\n'

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

s3tests.yaml points to ceph-master

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

RGW: Implement continuation, progress, stats, end s3select response

    RGW/S3select: Implement output-serializationi. user may request different CSV defintions
    for output (field delimiter, row delimiter, quote handling.

    RGW/S3select: Implement presto-alignments. presto-application sends
    queries with table-alias,case insensitive, and with no-semicolon at the
    end of statement.

    RGW/s3select: zero object size issue

Signed-off-by: Albin Antony <aantony@redhat.com>
Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

rgw: fix misspelling of 'effort'

Signed-off-by: Huber-ming <zhangsm01@inspur.com>

Merge pull request #43139 from josephsawaya/rook-orch-qa

qa/tasks/rook: test reapplication of drive groups stored in mgr

Merge PR #44337 into master

* refs/pull/44337/head:
mon: prevent new sessions during shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43872 from Huber-ming/reshard

rgw: init bucket index only if putting bucket instance info succeeds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43817 from joke-lee/UploadPartCopy-src-not-exist

rgw: fix UploadPartCopy error code when src object not exist and src bucket not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43259 from paulreece42/wip-rgw-xmlfix

rgw: allow CompletedMultipartUpload in XML

Reviewed-by: Casey Bodley <cbodley@redhat.com>