qa/suites/orch: add test for smb mgr module basic deploy with resources

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/suites/orch: add test for smb mgr module basic deploy with cli

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/suites/orch: add test for smb mgr module ad deploy with resources

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/suites/orch: add test for smb mgr module ad deploy with cli

Signed-off-by: John Mulligan <jmulligan@redhat.com>

qa/tasks/cephadm: add a wait_for_service_not_present task func

Add a wait_for_service_not_present task function that will wait until a
given service name is not present in the list of running cephadm
services. This is intended for testing service cleanup operations.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

doc/mgr: add documentation for new smb mgr module

Add initial documentation for the new smb mgr module. It doesn't cover
every possible thing or expected future changes but it should cover
the basics of interacting with the module from the cli.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

ceph.spec.in: add smb module and python-dataclasses dependency

The only distro ceph squid+ is building for at the moment that does not
already have a python version that includes dataclasses is centos/rhel
8. Add a dependency for the backport package on rhel8.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: use black & isort on the smb module

Provide tox envs that check or reformat code with black and isort,
currently applied to only the new smb module.
This is similar to what we recently did for enabling tox in the
cephadmlib dir as it only applies to new code. However, other modules
that want to opt-in to automated, python-community-wide typical,
stop-thinking-and-let-tools-do-it approach to code formatting can
be added to the new envs later on.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_smb.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_handler.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_rados_store.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_resources.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_resourcelib.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_validation.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add unit tests file tests/test_enums.py

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: create a smb tests dir

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: extend module.py with real smb mgmt methods

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add cli.py wrapper funcs for cli/mgr interaction

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add handler.py the main business logic & transaction core

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add results.py for result reporting types

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add fs.py functions for interacting with file systems

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add external.py intermediate funcs for external config store

The external stores are used to share configuration with the samba
containers.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add internal.py intermediate funcs for internal config store

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add resources.py for defining resource classes

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add resourcelib.py an internal resource mgmt lib

While I like the workflow that `ceph orch apply` provides I find the
code a little too "loose". Create a new minimalistic un/re-structuring
library that partly inspired by my work with Go, cephadm, and a little
from pydantic. But without adding any dependencies beyond python's
dataclasses.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add validation.py for generic module validation funcs

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add rados_store.py wrapping rados apis

Add a config store that wraps rados apis such that the values are
stored in rados objects.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add mon_store.py for wrapping the mon key-store

Add a config store based on wrapping the mon's key-store.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add config_store.py defining generic in-memory stores

The config store abstraction is defined in proto.py the config_store.py
configuration stores meet this protocol with wrappers around in memory
structures.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add proto.py for protcol classes and generic types

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr/smb: add enums.py for general smb module enums

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: add new smb module to tox.ini

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: stub in a new smb mgr module

Currently, this module does nothing but serve as a placeholder for
the future smb manager module.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm: handle user_sources uri values in smb daemon

When a smb daemon is being configured it may have user_sources - a
field containing uris that are supplemental configurations expected
to define users and/or groups for a non-AD member server. Ensure these
uris get passed to the env var for the config uris to get processed.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: add support for user_sources smb spec field

Pass the user_sources spec field to the smb service being deployed.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: add a user_sources field to smb service spec

We had a mechanism for passing primary configs and join sources to the
smb service but need a way to pass configs containing user (and group)
definitions for non-AD scenarios.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56926 from samarahu/rgw-update-boost-redis

RGW: boost/redis: Point to 1.85 tag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56486 from soumyakoduri/wip-skoduri-lc-stall

rgw/lc: advance head if the current entry doesn't exist

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #56860 from clwluvw/rgw-crypt-ssl

rgw: eliminate ssl enforcement for sse-s3 encryption

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56868 from clwluvw/request-payer

rgw: exclude logging of request payer for 403 requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56394 from cbodley/wip-rgw-cache-size

rgw: increase default metadata cache size for accounts

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #56606 from guojidan/num_bytes

crimson/osd/ops_executer: fix snap overlap range error

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56942 from Svelar/fix_tox_cephadm

test/cephadm: extend timeout again

Merge pull request #57057 from adk3798/no-downgrade

doc/cephadm: remove downgrade reference from upgrade docs

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #56644 from tchaikov/wip-warning-not-fatal

test: do not consider warning from Sanitizer as fatal

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55950 from ifed01/wip-ifed-omap-pcounters

os/bluestore: add omap_set* perf counters

Reviewed-by: akupczyk@ibm.com
Reviewed-by: rfriedma@ibm.com

Merge pull request #57071 from zdover23/wip-doc-2024-04-24-rados-troubleshooting-pg

doc/rados: improve t-shooting pg

Merge pull request #57051 from zdover23/wip-doc-2024-04-23-rados-dual-stack

doc/rados: remove dual-stack docs

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: improve t-shooting pg

Incorporate Anthony D'Atri's suggestions from
https://github.com/ceph/ceph/pull/57022 into the text in
doc/rados/troubleshooting/troubleshooting-pg.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56745 from ronen-fr/wip-rf-pgtell

tests/scripts: use 'tell pg deep-scrub pgid' instead of 'tell pgid deep-scrub'

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56420 from rzarzynski/wip-tools-rados-get-with-offset

tools: rados get takes --offset now

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56067 from myoungwon/wip-ceph-dedup-daemon

tool: separating ceph-dedup-tool into tool and daemon

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55949 from NitzanMordhai/wip-nitzan-safe-to-destroy-test-build-vars

test/osd/safe-to-destroy.sh: Add checks for the test

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54979 from pdvian/wip-mgr-optracker

mgr: integrate optracker in ceph-mgr

Merge pull request #57053 from idryomov/wip-labeler-rbd

.github: sync the list of paths for rbd label with CODEOWNERS

Reviewed-by: Laura Flores <lflores@redhat.com>

doc/cephadm: remove downgrade reference from upgrade docs

This has been in here for years, but cephadm will block
attempted upgrades to lower versions and we generally
don't want people to think this is supported or safe.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #56986 from cbodley/wip-65551

rgw/account: add bucket_quota to RGWAccountInfo

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Oguzhan Ozmen <oozmen@bloomberg.net>

Merge pull request #57038 from adk3798/mixin-max-nvme-gws-per-group

monitoring/ceph-mixin: set NVMeoFMaxGatewaysPerGroup to 4

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #56990 from cbodley/wip-doc-rgw-realm-change

doc/rgw: add release note for changes to rgw_realm init

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55983 from lxbsz/wip-64572

qa/fsx: use the master branch to build the xfstest-dev

rgw/lc: cleanup duplicate code

Cleaning up duplicate code around updating head

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/lc: advance head if the current entry doesn't exist

This is extension to https://github.com/ceph/ceph/pull/47595.

When skipping the non-exist LC entry, instead of get_next_entry(),
use advance_head() to fetch the next entry. In case the cycle
is finished for that shard, head should be reset to avoid lc process
being stuck in indefinite loop.

Fixes: https://tracker.ceph.com/issues/65160
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

.github: sync the list of paths for rbd label with CODEOWNERS

With https://github.com/actions/labeler/issues/483 still pending, do it
manually.

A couple of entries matching both files and directories were expanded
since, unlike fnmatch(3), minimatch treats slashes specially:

- src/include/rbd* -> src/include/rbd* src/include/rbd/**
- src/rbd*         -> src/rbd* src/rbd*/**
- src/test/rbd*    -> src/test/rbd* src/test/rbd*/**

A bunch of entries under src/tools were combined into:

- src/tools/rbd*/**

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

CODEOWNERS: assign ceph-rbd-mirror systemd service to RBD

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: remove dual-stack docs

Remove references to dual-stack mode in
doc/rados/configuration/network-config-ref.rst and
doc/rados/configuration/msgr2.rst. This feature seems to have been
planned but never to have been completely implemented.

See the tracker issue listed below for an email exchange detailing the
confusion caused by the presence in the documentation of this
now-removed information.

Fixes: https://tracker.ceph.com/issues/65631
Signed-off-by: Zac Dover <zac.dover@proton.me>

test/cephadm: extend timeout limit again to fix timeout issue

When run make check(arm64) on omani series servers,
Shows:
python: timeout after 60 seconds
INFO     root:call_wrappers.py:284 Non-zero exit code 124 from /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/cephadm/.tox/py3/bin/python -c for i in range(1000000): print(i, flush=True)

Seems like 60s is not enough, extend to 300s would be fine.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #57033 from gbregman/main

mgr/cephadm: Some changes to nvmeof conf file defaults

Merge PR #54067 into main

* refs/pull/54067/head:
mds: add dispatch killpoint and delay configs

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #56047 into main

* refs/pull/56047/head:
mgr: process map before notifying clients

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #56638 into main

* refs/pull/56638/head:
qa: do not iterate list being modified
qa: remove unnecessary background job cleanup

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56669 into main

* refs/pull/56669/head:
qa: make teardown behavior consistent for kclient/fuse mounts

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56723 into main

* refs/pull/56723/head:
mds: use mds_cache_quiesce_decay_rate to init quiesce_counter

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57010 into main

* refs/pull/57010/head:
qa: update quiesce tests to expect ipolicy lock
mds: add missing policylock to test F_QUIESCE_BLOCK

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57022 from zdover23/wip-doc-2024-04-22-rados-operations-pg-troubleshooting

doc/rados: remove redundant pg repair commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: apply default quota config on account creation

add new default quota config options for accounts analogous to
rgw_user_default_quota_max_objects/size. apply the default bucket quota
config options as-is

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57032 from stackhpc/doc-update-ms-bind-port-max

doc/rados: fix outdated value for ms_bind_port_max

Merge pull request #57036 from stackhpc/fix-typo-radosgw-metrics

doc/radosgw: fix typo in metrics.rst

monitoring/ceph-mixin: set NVMeoFMaxGatewaysPerGroup to 4

Recommendation from the nvmeof team

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #56448 from Svelar/arm_pass_check_shard_select

test/test_mempool: skip check_shard_select test on ARM64

doc/radosgw: fix typo in metrics.rst

Signed-off-by: Pierre Riteau <pierre@stackhpc.com>

mgr/cephadm: some changes to nvmeof conf file defaults
Fixes https://tracker.ceph.com/issues/65566

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: some changes to nvmeof conf file defaults
Fixes https://tracker.ceph.com/issues/65566

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

doc/rados: fix outdated value for ms_bind_port_max

The highest port number used by OSD or MDS daemons was increased from
7300 to 7568 in [1] but the documentation still refers to 7300 in
multiple locations.

[1] https://github.com/ceph/ceph/pull/42210

Fixes: https://tracker.ceph.com/issues/65609
Signed-off-by: Pierre Riteau <pierre@stackhpc.com>

Merge pull request #57016 from zdover23/wip-doc-2024-04-21-rados-operations-crush-map

doc/rados: add bucket rename command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: remove redundant pg repair commands

Incorporate the material in /doc/rados/operations/pg-repair into
/doc/rados/troubleshooting/troubleshooting-pg. Remove
/doc/rados/operations/pg-repair from the documentation. Redirect all
links to the old location to the new location.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: add bucket rename command

Add "ceph osd crush rename bucket" command. This commit is made in
response to a request from Michele Giacomoli.

Fixes: https://tracker.ceph.com/issues/65599
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56353 from myoungwon/wip-apply-shallow-copy-rbm-overwrite

crimson/os/seastore: avoid new allocation when overwriting data in RBM for performance

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #56855 from Svelar/asan_osd_types

test/osd/types: free PriorSet::pcontdec memory

Merge pull request #56025 from xxhdx1985126/wip-seastore-onode-loc-key

crimson/osd/osd_operations/client_requests: we don't support rados locator keys

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #56904 from zdover23/wip-doc-2024-04-16-security-cves

doc/security: update CVE list

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/security: update CVE list

Update the list of CVEs in doc/security/cves.rst.

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56916 from Matan-B/wip-matanb-crimson-recovery-misc

crimson/osd/replicated_backend: misc fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56848 from xxhdx1985126/wip-65453

crimson/osd/recovery_backends: discard outdated recovery ops

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #56277 from amathuria/wip-amat-crimson-fix-pool-compression

crimson: Add support for pool compression

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #56611 from Matan-B/wip-matanb-crimson-replicated-peering

crimson/osd/replicated_recovery_backend: prepare_pull use pg_info

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #56875 from xxhdx1985126/wip-65474

crimson/osd/osd_meta: load incremental osdmap from "inc_osdmap.XXX"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57012 from tchaikov/wip-python-common-pep8

python-common: PEP8 related cleanups

Reviewed-by: John Mulligan <jmulligan@redhat.com>

python-common: remove spaces around `=`

to be more PEP8 compliant

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

python-common: remove trailing spaces

for better readability.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa: update quiesce tests to expect ipolicy lock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>