mgr/cephadm: block removing last instance of _admin label

Fixes: https://tracker.ceph.com/issues/54425
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #45191 from neesingh-rh/cephfs-top_doc_fix

doc/cephfs: Typographical error.

Merge pull request #45136 from adk3798/extra-all

mgr/cephadm: extend extra_container_args to other service types

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #45166 from pritha-srivastava/wip-rgw-bucket-sync-run

rgw: add OPT_BUCKET_SYNC_RUN to gc_ops_list, so that

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephfs: Typographical error.

Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

rgw: add OPT_BUCKET_SYNC_RUN to gc_ops_list, so that
gc is initialised and send_chain does not crash.

Also deleting objects inline in case gc is uninitialised.

Fixes: https://tracker.ceph.com/issues/54417
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #44941 from ronen-fr/wip-rf-scrubBEv1_2

osd/scrub: limiting scrubber-backend external interfaces

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43957 from petrutlucian94/windows_ipv6

msg: Fix Windows IPv6 support

Merge pull request #44870 from phlogistonjohn/jjm-nfs-cleanups2

mgr/nfs: support managing exports without orchestration enabled

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #44845 from phlogistonjohn/jjm-nfs-cleanups

mgr/nfs: various minor cleanups

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #45113 from adk3798/minimize-image

cephadm: still set container_image when --no-assimilate-config is provided in bootstrap

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #45127 from rkachach/fix_issue_54026_part1

mgr/cephadm: Adding python natsort module

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45132 from melissa-kun-li/asyncssh_error_messages

mgr/cephadm: reduce log level for asyncssh error messages

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: extend extra_container_args to other service types

Otherwise, without this change, this can only be used for mgr,
mon and crash (daemons without their own service spec class)

Fixes: https://tracker.ceph.com/issues/54390
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #45041 from cyx1231st/wip-crimson-cleanup-epm-segment-writer

crimson/os/seastore: introduce SegmentAllocator for Journal and ExtentPlacementManager

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #45109 from cbodley/wip-rgw-dbstore-strings

rgw/dbstore: use string_view for string constants

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #45126 from guits/fix-bug-cv-bz2056719

ceph-volume: abort when passed devices have partitions

osd/scrub: fix some scrubber-backend formatting

no code changes.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: cleanup scrubber-be's usage of the size-of-disk service

Limit the amount of scrubber code that directly calls the
PG-Backend services - in this case, the logical-size to
on-disk-size translation.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: clean-up scrubber-be interfaces

Esp - the shallow/deep-error counters.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: make the scrubber-be the sole owner of the authoritative set

simplifying the interfaces used by the scrubber backend.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: remove I/O (store) direct access from the Scrubber Backend

Target: improving Scrubber BE testability by reducing the number and
complexity of the external APIs it uses.

(note - first step of many)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os/seastore/epm: integrate SegmentAllocator with Writer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: introduce SegmentAllocator and integrate with Journal

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/journal: cleanup, move commit_to to RecordSubmitter

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: replace condition_variable by a shared_promise

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/epm: simplify gating writes for Writer

Dropped open_segment_wrapper_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: drop unnecessary future from get_segment()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #45121 from yaarith/fix-wear-level-empty-page

mgr/devicehealth: skip null pages when extracting wear level

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45118 from sseshasa/wip-fix-osd-bench-fill-non-zeros

osd: Write non-zero data as part of osd benchmark test.

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #45010 from cyx1231st/wip-seastore-types-cleanup

crimson/os/seastore/seastore_types: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/mgr/nfs: document that nfs exports related mgr call requirements

A recent change in the mgr/nfs module should enable the functioning
of export management commands/API calls as long as the rados namespaces
and objects have been already established. Document this fact, noting
that now only the `ceph nfs cluster ...` calls *require* an
orchestration module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: support managing exports without orchestration enabled

This change allows the `ceph nfs export ...` commands to function
without the entire mgr/nfs subsystem requiring orchestration to be
enabled.  When there's no orchestration available, the code falls back
to examining the namespaces in the ".nfs" rados pool to determine what
cluster_id values are valid.

This change does not add support for creating the rados objects and
namespace needed to manage a nfs cluster. As discussed with the
orchestration group on 2022-01-22, rook does not need the mgr module to
establish the namespace. So, for now, we'll defer the work needed to
create the namespace/objects when orchestration is disabled.

Fixes: https://tracker.ceph.com/issues/54043
Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: fix typo in error message

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: add unit test for normalize_path

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: change method format_path to function normalize_path

This function was not using self and thus has no need to be a method.
While we're at it, rename it to normalize_path because that's what
it is doing.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: clean up rados object naming code

The naming of rados objects used to store the nfs config was spread
all over the code, including inline f-strings, not-static methods,
etc.
This change unifies the naming by putting constant string prefixes
and name generating functions into the utils.py file.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: make _check_rados_notify a function

This was previously a staticmethod. This static method was only used by
NFSRados object. Staticmethods are nearly always better implemented as
functions, which is done so here.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/nfs: limit dependency of NFSRados object

Previously, the NFSRados object accepted the "Module" as the
first argument but only used the rados attribute (type rados.Rados).
It's better to limit the scope of types when reasonably possible
so we can see what the true dependencies are. So we restrict
NFSRados to accepting a rados.Rados as the argument.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: still set container_image when --no-assimilate-config is provided

Fixes: https://tracker.ceph.com/issues/54141
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #44911 from Matan-B/wip-matanb-pool-name

mon/OSDMonitor: Restrict pool names beggining with a dot

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44321 from Alexponomarev7/fix-static-librados-build

cmake: add ceph-common DENABLE_SHARED=OFF support

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: reduce log level for asyncssh error messages

Fixes: https://tracker.ceph.com/issues/54132
Signed-off-by: Melissa Li <melissali@redhat.com>

Merge pull request #45023 from rkachach/fix_issue_54198

mgr/cephadm: Show an error when invalid format

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44928 from rkachach/fix_issue_54184

mgr/cephadm: validating service_id for MDS

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44896 from rkachach/fix_issue_53540

mgr/cephadm: Adding AGE field to device ls cmd

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #45119 from rhcs-dashboard/fix-dashboard_frontend_vulnerabilities-master

mgr/dashboard: fix front-end vulnerabilities

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

ceph-volume: abort when passed devices have partitions

ceph-volume doesn't prevent from using db and/or wal devices
with existing partitions on them.
This can lead to a data loss situation.

Fixes: https://tracker.ceph.com/issues/54376
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #45062 from cbodley/wip-54303

librbd: avoid get_callback_adapter() for tcp_stream::async_connect()

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45031 from cbodley/wip-54279

common: replace BitVector::NoInitAllocator with wrapper struct

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45086 from trociny/wip-54258

rbd-mirror: make mirror properly detect pool replayer needs restart

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45106 from idryomov/wip-image-replayer-stop

rbd-mirror: synchronize with in-flight stop in ImageReplayer::stop()

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/cephadm: Adding python natsort module
Needed by: https://tracker.ceph.com/issues/54026

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

mgr/dashboard: fix front-end vulnerabilities

Fixes: https://tracker.ceph.com/issues/54367
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/devicehealth: skip null pages when extracting wear level

Some devices have null pages in their ata_device_statistics struct; skip
those pages in order to avoid an AttributeError when extracting device's
wear level.

Fixes: https://tracker.ceph.com/issues/51554
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #45117 from idryomov/wip-rbd-trash-purge-error-typo

mgr/rbd_support: fix a typo in the error message

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #45068 from ronen-fr/wip-rf-minimal-digestupd-fix

osd/scrub: stop sending bogus digest-update event messages

Reviewed-by: Neha Ojha <nojha@redhat.com>

osd: Write non-zero data as part of osd benchmark test.

An optimization (see PR: https://github.com/ceph/ceph/pull/43337) was made
in BlueStore to avoid writing bufferlists made up of zeros. The osd
benchmark used zero filled bufferlists and this resulted in inflated osd
benchmark results.

This issue is fixed by using bufferlists filled with non-zero values.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
Fixes: https://tracker.ceph.com/issues/54364

mgr/rbd_support: fix a typo in the error message

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/cephadm: Adding AGE field to device ls cmd
Fixes: https://tracker.ceph.com/issues/53540
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

rgw/dbstore: DBOpPrepareInfo::query_str is a view

in `SQLiteDB::InitPrepareParams()`, `DBOpInfo::query_str` is assigned to
`DBOpPrepareInfo::query_str`:

  p_params.op.query_str = params->op.query_str;

by making `DBOpPrepareInfo::query_str` a view, we avoid allocating a
copy of the `std::string`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/dbstore: use `const char*` for sqlite3_bind_parameter_index()

all string constants passed to sqlite3_bind_parameter_index() are now
`static constexpr const char*` instead of `std::string`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/dbstore: Schema() functions are static

these functions only refer to static member variables, so can be static
themselves

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/dbstore: remove unused 'string Op' argument to InitializeParams()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: using MDSSPec instead of ServiceSpec
Fixes: https://tracker.ceph.com/issues/54184
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #44985 from rhcs-dashboard/dashboard-turns-telemetry-off

mgr/dashboard: dashboard turns telemetry off when configuring report

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarit@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45022 from kalebskeithley/54266

rgw: cmake configure error on fedora-37/rawhide

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #44905 from mgfritch/cephadm-prom-chown

cephadm: chown the prometheus data dir during redeploy

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #45054 from rkachach/fix_issue_46655

mgr/cephadm: Delete ceph.target if last cluster

Reviewed-by: Michael Fritch mfritch@suse.com

rbd-mirror: synchronize with in-flight stop in ImageReplayer::stop()

Complete on_finish right away only if the replayer is stopped (meaning
that it is legible to be restarted immediately, possibly from on_finish
itself).  This is the behaviour pretty much anyone would assume and
also what ImageReplayer::restart() relies on.

Fixes: https://tracker.ceph.com/issues/54344
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: turn m_on_stop_finish into a list of Contexts

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: manual stop should take precedence over regular stop

Somewhat similar to commit 0a3794e56256 ("rbd-mirror: make stop
properly cancel restart"), make it so that a) if a manual stop is
joined to regular stop, the stop becomes manual and b) if a regular
stop is joined to a manual stop, the stop stays manual.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd-mirror: straighten ImageReplayer::stop() a bit

- don't default on_finish parameter
- m_restart_requested is set in ImageReplayer::restart() which is the
  only restart=true call site, so setting m_restart_requested here is
  redundant
- is_stopped_() can't be true in is_running_() branch
- on_finish->complete(0) in the end is unreachable

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44873 from kotreshhr/clone-inherit-quota-files

mgr/volumes: Inherit file quota attr to clone

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44854 from kotreshhr/subvolumegroup_ls

mgr/volumes: Fix subvolumegroup ls

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/seastore_types: pretty print fundamental types

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/seastore_types: unify NULL/MAX/default values

Mostly for paddr_t and journal_seq_t, and use P_ADDR_NULL and
JOURNAL_SEQ_NULL where possible, with related cleanups.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #45089 from athanatos/sjust/wip-seastore-journal-refactor

seastore: seperate Journal interface from SegmentedJournal implementation

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #45033 from anthonyeleven/anthonyeleven/releases-tweaks

doc/releases: Improve capitalization for Octopus

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #44994 from ljflores/wip-telemetry-device-assertion-failure

mgr/telemetry: handle empty device report when "send" is triggered

seastore: seperate Journal interface from SegmentedJournal implementation

A subsequent PR will introduce a CircularBoundedJournal implementation
for fast nvme devices.

SegmentCleaner no longer needs a reference to Journal, so dispense with
the set_segment_provider machinery and simply pass it in the
constructor.

Move responsibility for finding the journal segments into the journal
itself.  This does mean that we check the segment headers on the journal
device twice, but that should be a neglible amount of overhead on mount.

SegmentCleaner::init_segments no longer needs to return Journal
segments, so merge with mount().

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: Contact Info should be visible only when Ident channel is checked

Fixes:https://tracker.ceph.com/issues/54133
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>

Merge pull request #44936 from javacruft/build-compress-common-dep

cmake: add dependency compress->common

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

cmake: add dependency compression->common

The compression code depends on headers generated during the build
of the common code for Ceph.

Ensure that CMake understands this dependency, avoiding a race
between generation of header files in common and compilation of the
compression module.

Signed-off-by: James Page <james.page@ubuntu.com>

rgw/dbstore: use string_view for string constants

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44608 from liumiaomiaoIntel/qatchange

common: fix compilation and function issues about compressor and crypto to enable latest QAT driver

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44904 from chrisphoffman/wip_rbd_default_map_override

tools/rbd: expand where option rbd_default_map_options can be set

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #45085 from tchaikov/crimson-fmtlib-v8

crimson: fixes for compiling with fmtlib v8

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43388 from e337/wip-doc-50499

rbd-mirror: make mirror properly detect pool replayer needs restart

When a PoolReplayer detects remote pool metadata change it
sets "stopping" flag expecting the Mirror will restart it.

Although setting "stopping" flag makes the PoolReplayer::run
thread to terminate, the thread's is_started function will still
return true until join is called (and reset the thread id).

This made impossible for the Mirror to detect (by calling
PoolReplayer::is_running) that the PoolReplayer needed restart.

Fixes: https://tracker.ceph.com/issues/54258
Signed-off-by: Mykola Golub <mgolub@suse.com>

crimson: specialize fmt::formatter<>() for crimson types

otherwise we'd have FTBFS like

/usr/include/fmt/core.h:1727:3: error: static_assert failed due to requirement 'formattable' "Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api$
  static_assert(
  ^
/usr/include/fmt/core.h:1853:23: note: in instantiation of function template specialization 'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, fmt::detail::type::custom_type, crimson::os:$
        data_{detail::make_arg<

please note, delta_op_t is lifted out of the templated outer class
to avoid the headache of specialization of template of template in
another namespace.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa/suites/krbd: add rbd_default_map_options override coverage

Add coverage to test precedence, override, and option merge on rbd map.

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

qa/suites/krbd: rename rxbounce subsuite

A new job that doesn't want ms_mode to be set underneath it is about to
be added.  Rename rxbounce to ms_modeless to make this purpose obvious.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: support pool and image level overrides for rbd_default_map_options

Fixes: https://tracker.ceph.com/issues/52850
Signed-off-by: Christopher Hoffman <choffman@redhat.com>

crimson/os/seastore: drop compile-time fmt validation

_format() was deprecated in fmtlib v8, see also
https://github.com/fmtlib/fmt/pull/2646

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

compressor: fix compilation issues about QATzip

Signed-off-by: Miaomiao Liu <miaomiao.liu@intel.com>
Signed-off-by: Hualong Feng <hualong.feng@intel.com>

Merge pull request #44933 from liu-chunmei/seastore-support-sparse-read

crimson/seastore: support sparse_read

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44955 from aclamk/wip-vselector-check-fix

os/bluestore/bluefs: Fix improper vselector tracking in _flush_special()

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

crimson/seastore: put ObjectDataHandler instance into do_with

make ObjectDataHandler instance alive before sub functions are resolved.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/seastore:: add fiemap and sparse read unit test

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>