dpdk: drop dpdk submodule

we are using spdk/dpdk for async msgr's dpdk backend since 01a9f178. so
no need to keep dpdk submodule anymore.

Fixes: http://tracker.ceph.com/issues/24032
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32992 from ceph/wip-revert-dpdk

Revert "dpdk: drop dpdk submodule"

Revert "dpdk: drop dpdk submodule"

This reverts commit cb8087dfac31b8490fefdfca28d389b7b9901ef8.

The seastar submodule's .gitmodules links to `../dpdk` which is no longer present after removing dpdk from ceph.git's .gitmodules.

```
<dwfreed> the ceph/seastar repo uses awful URLs for the submodules
<dwfreed> and those awful URLs are the real reason it's failing
<dwfreed> dgalloway: ^^^
<dwfreed> seastar's .gitmodules references repos in the parent directory, so that when it's checked out as a submodule of ceph, you don't download the repos twice (and git will probably also use references instead of duplicating the local .git); however, ceph doesn't have a submodule for dpdk anymore
<dwfreed> so seastar's referencing a dpdk repo that doesn't exist
<dgalloway> i think i follow.  so you're suggesting revert https://github.com/ceph/ceph/commit/cb8087dfac31b8490fefdfca28d389b7b9901ef8 ?
<dwfreed> yep
<dwfreed> that'd be one way to fix it
...
<joshd> dgalloway: I'd suggest revert for now, and let the crimson folks figure out the longer term fix when they're back
```

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge PR #32958 into master

* refs/pull/32958/head:
qa/suites/rados/singleton/all/lost-unfound*: whitelist SLOW_OPS

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32789 into master

* refs/pull/32789/head:
test: fix unused assert's variable in ceph_test_osd_stale_read.cc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32961 into master

* refs/pull/32961/head:
qa/standalone/osd/osd-bench: debug bluestore

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32968 into master

* refs/pull/32968/head:
qa/suites/rados/verify: debug monc = 20

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32771 from xxcs/xxcs07

rgw: fix multipart upload's error response

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge PR #32878 into master

* refs/pull/32878/head:
cephadm: share code between 'pull' and 'inspect-image'
mgr/cephadm: upgrade: pull image after upgrade start, and for each host
cephadm: add inspect-image command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #30969 into master

* refs/pull/30969/head:
mds : clean up data written to unsafe inodes

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32397 into master

* refs/pull/32397/head:
mds: Move StrayManager initializations to its header
mds: Remove extra spaces in StrayManager header.
mds: Reorganize structure members in StrayManager header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32969 into master

* refs/pull/32969/head:
qa/suites/rados/cephadm: explicitly test many distros

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32854 into master

* refs/pull/32854/head:
qa: fix testing kernel branch link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32801 into master

* refs/pull/32801/head:
cephfs-shell: Fix ls -l

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: Change project name to "Ceph Dashboard" (#32959)

mgr/dashboard: Change project name to "Ceph Dashboard"

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32978 from tchaikov/wip-seastar

seastar: pickup change to fix cgroups V2 support

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #32972 into master

* refs/pull/32972/head:
python-common/ceph/deployment/translate: use 'prepare' instead of 'batch' for trivial case
qa/tasks/cephadm: pass short dev name to osd prepare
mgr/cephadm: fix detection of just-created OSDs
mgr/cephadm: properly indent raise conditions
mgr/cephadm: add warning to other orchestrators
mgr/cephadm: separate acceptance criterias for Devices
mgr/cephadm: fix typos
mgr/cephadm: move utils in test/utils.py
mgr/ssh: increase disk size to 20G
drivegroups: add support for drivegroups + tests
mgr/orch_cli: allow multiple drivegroups
drivegroups: translate disk spec to ceph-volume call

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #32980 from ceph/gregsfortytwo-doc-rm-hadoop-2

doc: remove the CephFS-Hadoop instructions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: fix multipart upload's error response
Fixes: https://tracker.ceph.com/issues/43751
Signed-off-by: GaryHyg <huygbj@inspur.com>

Merge pull request #32967 from rhcs-dashboard/ignore-python-common-build

.gitignore: ignore /src/python-common/build

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #30777 from GaryHyg/origin/master

rgw: dmclock: wait until the request is handled.

Reviewed-By: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #32669 from IlsooByun/rel_cb_arg

rgw: release unused callback argument

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #32476 from theanalyst/rgw/multisite/user-policy

rgw: user policy: forward write requests to master zone

Reviewed-By: Casey Bodley <cbodley@redhat.com>

doc: remove the CephFS-Hadoop instructions

These have not aged gracefully, and in particular include instructions
for setting pool size 1 to let Hadoop control the replication — but I've
heard reports of users setting up multiple size-1 pools and then wondering
where their data went when an OSD dies.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32370 from AlexZzz/prometheus_pg_status_bypool

mgr/prometheus: report per-pool pg states

Merge PR #32975 into master

* refs/pull/32975/head:
mon/ConfigMonitor: only propose if leader

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32970 into master

* refs/pull/32970/head:
cephadm: include daemon/unit id in unit name

Reviewed-by: Michael Fritch <mfritch@suse.com>

seastar: pickup change to fix cgroups V2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32945 from dzafman/wip-43864

test: Update pg log test for new trimming behavior

Reviewed-by: Neha Ojha <nojha@redhat.com>

mon/ConfigMonitor: only propose if leader

Introduced by 08fcf01c040e0caaf5b418d300f774bf8c3033da, which activated
this (broken) code path.

Fixes: https://tracker.ceph.com/issues/43892
Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: share code between 'pull' and 'inspect-image'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32623 into master

* refs/pull/32623/head:
ceph_release: octopus rc 15.1.0

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/cephadm: explicitly test many distros

Include _latest.yaml in a few cases here to be a bit future-proof.

cephadm-smoke/ is *just* a cephadm bring-up, and includes el7.  cephadm/
installs packages and runs a real workload.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32828 from leseb/raw-db-wal

ceph-volume: add db and wal support to raw mode

python-common/ceph/deployment/translate: use 'prepare' instead of 'batch' for trivial case

This works around ceph-volumes lack of support for preexisting LVs in the
batch mode.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include daemon/unit id in unit name

This makes the log messages more descriptive.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32941 from cbodley/wip-qa-rgw-multisite-pubsub-checkpoint

qa/rgw: multisite checkpoints consider pubsub zone

Merge PR #32943 into master

* refs/pull/32943/head:
qa/tasks/ceph_manager: fix chmod on log dir during pg export copy

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/verify: debug monc = 20

Hunting https://tracker.ceph.com/issues/43882

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32932 from smanjara/add-bucket-chown-nonmaster

rgw/multisite: warn if bucket chown command is run on non-master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

.gitignore: ignore /src/python-common/build

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #32957 from tchaikov/wip-mgr-cephadm

mgr/cephadm: init attrs created by settattr()

Merge pull request #32893 from sebastian-philipp/orchestrator-document-client-usage

mgr/orchestrator: Add doc about how to use OrchestratorClientMixin

Reviewed-by: Zac Dover <zac.dover@gmail.com>

mgr/grafana: sum pg states for cluster

Also, revert table formatting.

Signed-off-by: Aleksei Zakharov <zaharov@selectel.ru>

monitoring/grafana,prometheus: add per-pool pg states support

Signed-off-by: Aleksei Zakharov <zaharov@selectel.ru>

mgr/prometheus: pg counters per pool descriptions

Update pg metrics descriptions to show that we have per
pool stats now.

Signed-off-by: Aleksei Zakharov <zakharov.a.g@yandex.ru>

mgr/prometheus: pg count by pool

If we have all other stats by pool, it's better to have total
count by pool too. We always can sum() all of total, but it's
hard to count by-pool total.

Signed-off-by: Aleksei Zakharov <zakharov.a.g@yandex.ru>

mgr/prometheus: report per-pool pg states

This commit adds per-pool pg states metrics
with unique 'pool_id' label.

Signed-off-by: Aleksei Zakharov <zakharov.a.g@yandex.ru>

Merge PR #32939 into master

* refs/pull/32939/head:
mon/MgrMonitor.cc: add always_on_modules to the output of "ceph mgr module ls"

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/osd-bench: debug bluestore

Looking for https://tracker.ceph.com/issues/43888

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop "type: ignore" annotations

since these options are already initialized in the constructor, there is
no need to ignore their types when checking those lines using mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: init attrs created by settattr()

to address the test falure caused by mypy:
```
mypy run-test: commands[0] | mypy --config-file=../../mypy.ini ansible/module.py cephadm/module.py mgr_module.py mgr_util.py orchestrator.py orchestrator_cli/module.py rook/module.py
test_orchestrator/module.py
cephadm/module.py: note: In member "_check_for_strays" of class "CephadmOrchestrator":
cephadm/module.py:596: error: "CephadmOrchestrator" has no attribute "warn_on_stray_hosts"
cephadm/module.py:596: error: "CephadmOrchestrator" has no attribute "warn_on_stray_services"
cephadm/module.py:599: error: "CephadmOrchestrator" has no attribute "warn_on_stray_services"
Found 3 errors in 1 file (checked 8 source files)
```
see also https://github.com/python/mypy/issues/5719

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add bucket_chown to list of commands that are recommended to be run on master zone.

Fixes: https://tracker.ceph.com/issues/43884
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge PR #32823 into master

* refs/pull/32823/head:
common/bl: fix memory corruption in bufferlist::claim_append()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32940 into master

* refs/pull/32940/head:
qa: remove rados/basic/tasks/rgw_snaps.yml

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #32942 into master

* refs/pull/32942/head:
osd: dispatch_context and queue split finish on early bail-out

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton/all/lost-unfound*: whitelist SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31879 from joke-lee/rgw-sts-post-upload-auth

rgw: use STSEngine::authenticate when post upload with x_amz_security_token

Merge pull request #31214 from IlsooByun/lc_stale_head

rgw: prevent LC from reading stale head when transitioning object

ceph-volume: add db and wal support to raw mode

Using the raw mode, the OSD preparation can now be called with extra
arguments to specify a block device for either rocksdb db or wal.

The code expects the device to be available and no validation will be
performed. Also, the entire block device will be consumed.

The goal is run this on Kubernetes with Rook and OSD are running on PVC.
Users will request PVC from a certain size, the request will be
acknowledged and a block will be created, then attached to the machine.
Later on consumed by Rook's preparation job.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #31700 from rishabh-d-dave/cv-list-using-dash-S

ceph-volume: refactor listing.py

Merge pull request #32808 from theanalyst/rgw/test-iam-bits

test: modify iam tests to use a function to set bits

Reviewed-By: Adam Emersen <aemerson@redhat.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Change project name to "Ceph Dashboard"

The Dashboard's "About" page still referred to the
application as "Ceph Manager Dashboard".
Changed the `projectName` constant to
"Ceph Dashboard" to resolve this.

Fixes: https://tracker.ceph.com/issues/43840
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: smart: add support for NVMe (#32708)

mgr/dashboard: smart: add support for NVMe

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32938 from trociny/wip-fix-freebsd-build

librbd: fix build on freebsd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: Update pg log test for new trimming behavior

Fixes: https://tracker.ceph.com/issues/43864
Signed-off-by: David Zafman <dzafman@redhat.com>

qa/rgw: multisite checkpoints consider pubsub zone

pubsub zones are configured to only sync_from the master zone, so bucket
checkpoints should ignore its sync status with respect to other zones

Fixes: https://tracker.ceph.com/issues/43768
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/cephadm: pass short dev name to osd prepare

Zap needs a full path, but create/prepare needs the VG/LV
only if it is an existing LV.

We'll make c-v more friendly later.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32502 into master

* refs/pull/32502/head:
pybind/mgr: Add ceph_module.pyi to improve type checking

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #32903 into master

* refs/pull/32903/head:
mgr/devicehealth: fix telemetry stops sending device reports after 48 hours

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32365 into master

* refs/pull/32365/head:
mon: Don't put session during feature change

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32897 into master

* refs/pull/32897/head:
osd/PG: do not leak cluster message when there's no con

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31686 from yehudasa/wip-rgw-sync-bucket-6

rgw: bucket granularity sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/ceph_manager: fix chmod on log dir during pg export copy

With cephadm, we should chmod both /var/log/ceph and /var/log/ceph/$fsid.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: bucket sync: invalidate hints data if raced with another writer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd: dispatch_context and queue split finish on early bail-out

If we bail out of advance_pg early because there is an upcoming merge, we
still need to dispatch_context() on rctx before we drop the PG lock.  And
the rctx that we submit needs to include the on_applied finisher comit
to call _finish_splits.

This is noticeable (at least) when there is a split and merge that are
both known.  When we process the split, the new child is added to new_pgs.
When we get to the merge epoch, we stop early and take the bail-out
path.

Fix by adding a dispatch_context call for this path.  And further make sure
that both dispatch_context callers in this function queue up the
new_pgs event.

Fixes: https://tracker.ceph.com/issues/43825
Signed-off-by: Sage Weil <sage@redhat.com>

qa: remove rados/basic/tasks/rgw_snaps.yml

rgw_snaps tasks should not be running in the rados suite.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: fix compilation after rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon/MgrMonitor.cc: add always_on_modules to the output of "ceph mgr module ls"

Makes it easier to know what the always_on_modules for each release are, without
having to look at the code.

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw: don't include empty buckets in potential related buckets

Being used by the sync hint system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep sync_policy as optional member inside RGWBucketInfo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move a few basic types into rgw_basic_types.{h,cc}

To help with the fight against circular dependencies

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: initiate bucket meta sync if can't find sync handler

If bucket sync handler returns -ENOENT, try to get bucket info. This can
trigger bucket meta sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trim bilog: send bucket param instead of source-bucket param

For backward compatibility, so that older rgws could handle it appropriately.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: don't allow replication api on legacy config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

PendingReleaseNotes: rgw, bucket granularity

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clarify api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: get_hint_entities(), call read_bucket_info once

once per bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: no create param in find directional/symmetrical

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: don't use optional for some members

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a potential leak

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove return std::move()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() create default acls if not found

Create default acls if the original object did not have acl header.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove superfluous alias

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: doc fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: find first key with longest matching prefix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix bucket sync markers for non-trivial case

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket autotrim: fetch aggregated bilog info of all bucket targets

Add a new option to the bilog status request that allows returning the
min status for all buckets in a zone that sync from a specific bucket.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket trim: only fetch status from relevant zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: use list instead of vector

Keeping pointers into entries inside the container, vector might be
reallocated.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>