librbd: flush requests could race past initiation of write ops

Now that IO is being processed by multiple threads, it's possible
that a write operation that was issued prior to a flush would not
have been started prior to the processing of the flush.

Fixes: https://tracker.ceph.com/issues/47050
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reset collection state after move operation

A technical requirement of move operations requires that the collection
be reset to a known state before being used again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35789 from Reperator/fix-ceph-objectstore-tool-docs

doc/man/8/ceph-objectstore-tool.rst: remove duplicate lines

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36742 from hulksmaaash/patch-2

doc/rados: remove statement that ISA plugin only works on Intel chips

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36496 from Daniel-Pivonka/cephadm-44926

mgr/cephadm: rgw update period after realm, zone creation

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36716 from jmolmo/dashboard_ports_2

cephadm: Allow users to use a custom dashboard ssl port

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36721 from ceph/47036-sass-colors-typescript

mgr/dashboard: use SASS colors in TypeScript

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>

Merge pull request #36735 from rhcs-dashboard/fix-this-week-endpoint

mgr/dashboard: remove "This week/month/year" and "Today" time stamps of Grafana in dashboard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #36740 from varshar16/wip-fix-nfs-bug-47009

qa/tasks/nfs: Change the mds name and delete created nfs clusters

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.

Merge pull request #36694 from shun-s/fix-linger-op-stray

osdc: fix lingerOp stray

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: remove "This week/month/year" and "Today" time stamps of Grafana in dashboard

Fixes: https://tracker.ceph.com/issues/47049
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #36465 from majianpeng/bluefs-avoid-flush-much-data

os/bluestore/BlueRocksEnv: Avoid flushing too much data at once.

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #36593 from gregsfortytwo/wip-46914

mon: mark pgtemp messages as no_reply more consistenly in preprocess_…

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36675 from tchaikov/wip-fix-option-typo

common/options: fix a typo in option description

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35613 from changchengx/str_list

use for_each_substr to avoid redundant operation by calling get_str_set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36573 from changchengx/osdmonitor

mon: use string_veiw to glimpse string and remove unused variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36680 from tchaikov/wip-mgr-disk-predict

pybind/mgr/diskprediction_cloud: remove mgr-diskprediction-cloud

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #36534 from changchengx/formatter

common/Formatter: don't construct string if string_view work

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #36420 from qwren/qat-compressor

compressor: fix compile error for QAT

Reviewed-By: Kefu Chai <kchai@redhat.com>

doc/rados: remove statement that ISA plugin only works on Intel chips

ISA-L supports aarch64 (Arm), not just Intel processors, as does Ceph (see #32171).

Signed-off-by: Robbie Williamson <robbie.williamson@arm.com>

mon: mark pgtemp messages as no_reply more consistently in preprocess_pgtemp

If a message is forwarded, it's conceivable the leader's and peon's evaluation
will disagree about whether the message is useful or not, which could result
in the leader ignoring it and the peon having a dangling forwarded message.
Fix this by marking the op as no_reply whenever ignoring it.

Fixes: https://tracker.ceph.com/issues/46914
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

qa/tasks/nfs: Instead of 'orch ls' use 'orch ps'

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/suites/rados/test_orch_cli: Change mds name

As MDS name should not start with numeric digit.

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/nfs: Delete nfs cluster after exports are created

Fixes: https://tracker.ceph.com/issues/47009
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #36477 from p-se/fix-downstream-alertmanager

cephadm: Support custom alertmanager container

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Allow users to use a custom dashboard ssl port

This modification allows the user to create a new bootstrap cluster using a predefined SSl port for the dashboard.
If firewall is enabled, any new manager daemon deployed in new hosts will take care of open the required ports for all the services enabled in the manager.

Two new parameters for cephadm tool (aka binary or standalone):
Command bootstrap:
--ssl-dashboard-port SSL_DASHBOARD_PORT
                      Port number used to connect with dashboard using SSL
Command deploy:
--tcp-ports TCP_PORTS
                      List of tcp ports to open in the host firewall

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #35756 from ShyamsundarR/tr-46163

mgr/volumes: Use snapshot root directory attrs when creating clone root

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge pull request #36485 from adk3798/cephadm-44252

mgr/cephadm: mgr or mds scale-down should prefer non-active daemons

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36723 from tchaikov/wip-osd-writesame

crimson/osd: support writesame op

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #36711 into master

* refs/pull/36711/head:
Revert "vstart_runner: rotate logs between testcases"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Revert "vstart_runner: rotate logs between testcases"

Fixes: https://tracker.ceph.com/issues/47032
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: write log when sending "osd_failure" msg

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support writesame op

it is tested by TestIoctx.test_aio_writesame. so let's implement it as
well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: use SASS colors in TypeScript

NOTE: Enabled TS compiler option esModuleInterop for the unit tests to pass, which is also highly recommended:
https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-7.html#support-for-import-d-from-cjs-from-commonjs-modules-with---esmoduleinterop
This implied updating lodash & momentjs imports (unneeded ones have been removed).

Fixes: https://tracker.ceph.com/issues/47036
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #36719 from ofriedma/wip-reshard-logs

rgw: log resharding events at level 1 (formerly 20)

rgw: log resharding events at level 1 (formerly 20)
log resharding events at level 1 (formerly 20)
Signed-off-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #36381 from dang/wip-dang-zipper-7

Zipper 7 - chop list items

Merge pull request #36703 from tchaikov/wip-crimson-errorator-omap

crimson/osd: errorator omap_get_values() ops

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36702 from tchaikov/wip-doc-crimson

doc/dev/crimson: fix a typo

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #36699 from ceph/47008-table-footer-size

mgr/dashboard: restore table footer size

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #36706 from Daniel-Pivonka/cephadm-rootuserdocfix

doc/cephadm: remove ssh user limitation troubleshooting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: errorator omap_get_values() ops

* os/alienstore: do not panic if retval is less than 0. instead,
  translate ENOENT to enoent and other erros to input_output_error
* os/FuturizedStore: errorator omap_get_values() variants, so
  they return errorator::future<> instead of plain seastar::future<>
* os/seastore: update accordingly.
* osd/pg_backend: use enodata for OI without omap in it. better
  performance, and better readability this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr/diskprediction: add a configuration section

also link to monitoring, as diskprediction cannot function without
enabling monitoring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/devicehealth: drop the cloud model of disk prediction

as the service offered by https://www.diskprophet.com/ is not accessible
anymore, the cloud model of disk prediction is not available. so let's
drop the code handling "cloud" model.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: drop test for diskprediction_cloud

this module is removed, so no need to test its settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: drop commented code

the test for diskprediction_cloud is never enabled, and the used
cloud-based service is not reachable anymore. let's just remove the dead
code.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: stop referencing diskprediction_cloud

diskprediction_cloud is removed from master, so no need to have it in
the exclude list anymore

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr/diskprediction.rst: remove references to diskprediction_cloud

since https://www.diskprophet.com is not accessible anymore, and this
feature is not available at the time of writing. let's drop the related
document.

Fixes: https://tracker.ceph.com/issues/46863
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/diskprediction_cloud: remove mgr-diskprediction-cloud

the service offered by https://www.diskprophet.com is not accessible
anymore. before we have a replacement of it, let's drop
mgr-diskprediction-cloud. and remove its packaging.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36531 from tchaikov/wip-alpine

alpine: remove stale Alpine build recipe

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36688 from coreycb/master

librbd: Align rbd_write_zeroes declarations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/cephadm: remove ssh user limitation troubleshooting

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

mgr/cephadm: rgw update period after realm, zone creation

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #35824 into master

* refs/pull/35824/head:
vstart_runner.py: rotate logs between testcases
vstart.sh: write logrotate conf and rotate logs
vstart_runner: run one test method at a time...

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/crimson: fix a typo

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36533 into master

* refs/pull/36533/head:
mon: store mon updates in ceph context for future MonMap instantiation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36530 from tchaikov/wip-cmake-python2

cmake: drop WITH_PYTHON2 option

Reviewed-By: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: mgr or mds scale-down should prefer non-active daemons

When removing daemons during a mgr/mds scale-down, prefer to remove
standby daemons so the active daemon is not killed

Fixes: https://tracker.ceph.com/issues/44252
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #36543 from jschmid1/prevent_path_from_non_data_devices

python-common: ban paths from non-data-devices

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36568 from jschmid1/remove_osd_spec

mgr/orchestrator: remove traces of ceph orch osd spec

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36541 from jschmid1/save_osdspec_affinity_to_store

mgr/cephadm: save osdspec_affinity to dd store

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36584 from jschmid1/get_daemons_by_type

mgr/cephadm: add by_type daemon retrieval method

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: restore table footer size

Fixes: https://tracker.ceph.com/issues/47008
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Zipper - add const accessors for svc and ctl

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #34918 from bk201/wip-45397

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #36674 from tchaikov/wip-doc-crimson

doc/dev/crimson: note about script/ceph-debug-docker.sh

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart_runner.py: rotate logs between testcases

Fixes: https://tracker.ceph.com/issues/46059
Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart.sh: write logrotate conf and rotate logs

Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: run one test method at a time...

instead of running entire testsuite at once; this would allow us more
control on the test runner.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

osdc: fix lingerOp stray

  when linger ping failed with error, like ENOTCONN
  last_error set to error.
  after that, last_error will never recovery to succecss(0),
  even reconnecting successfully, which stops from sending linger ping to osd.
  as a result, this normal client ** can't receive notify message **
  after osd_client_watch_timeout goes away.

Fixes: https://tracker.ceph.com/issues/47004
Signed-off-by: Song Shun <song.shun3@zte.com.cn>

Merge pull request #36686 from sebastian-philipp/ceaphdm-upgrade-pytest

mgr/cephadm: Add tests/test_upgrade.py

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #36629 from yison/fix-mem-leak-nvmedevice

blk/spdk/NVMEDevice.cc: fix a memory leak issue

Reviewed-By: Kefu Chai <kchai@redhat.com>

mgr/cephadm: use fixed config file path as argument

Fixes other alertmanager containers which do not default to look for a
configuration file in `/etc/alertmanager/alertmanager.yml`.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

alpine: remove stale Alpine build recipe

alpine build recipe is stale and does not work with the latest Ceph,
also the APKBUILD for Ceph can be found at alphine's aports repo, see
https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/ceph.

so instead keeping a stale version, let's drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36515 into master

* refs/pull/36515/head:
libcephfs: ignore restoring the open files limit
msg: throw a system error when center.init fails

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36525 into master

* refs/pull/36525/head:
client: switch to use ceph_mutex_is_locked_by_me always

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36552 into master

* refs/pull/36552/head:
mds: fix kcephfs parse dirfrag's ndist is always 0

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36617 into master

* refs/pull/36617/head:
mds: revert the decode version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Zipper - Make sure to use std::string everywhere

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW - Turn on debug logging for tempest tests

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Fix crash in RGWHandler_REST_S3Website::error_handler

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Fix read_user_manifest_part

Was using Rados directly; use Zipper instead.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Fix retry_raced_bucket_write

Was using Rados directly; use Zipper instead.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Removing getRados() - RGWGetObjLayout

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - convert RGWBulkUploadOp::handle_dir to use create_bucket

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Remove duplicate remove_bucket() code

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Rename get_bi() to get_key()

Provides consistency, and get_bi() wasn't a useful name.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - forward_request_to_master in RGWStore

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Implement Delete op on RGWObject

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Implement Read op on RGWObject

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - implement set_attrs and delete_attrs in RGWObject

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - Implement check_quota in RGWBucket

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Zipper - implement RGWBucket::List

Create a list op for RGWBucket, and use it.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

librbd: Align rbd_write_zeroes declarations

Fixes: https://tracker.ceph.com/issues/46928
Signed-off-by: Corey Bryant <corey.bryant@canonical.com>

Merge pull request #36605 from rhcs-dashboard/fix-46920-master

mgr/dashboard/api: reduce amount of daemon logs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36312 from ishanrai05/bug-1

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/cephadm: also mock `ceph vesions`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: add test_upgrade.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/tests: move with_service to fixtures.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: Also mock the config store in pytest

This will be used to dry-run mgr/cephadm/upgrade.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>