rgw: more lifecycle code cleanup and tooling

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle utility coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: have sub name as event oid prefix

dest should be unique for each subscription

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add a new lc get command

Useful for debugging lifecycle issues

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: create notif via REST api sets dest bucket

This got lost somewhere in the refactoring work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine set_state() also sets retcode

Generally recode is needed for the caller, but current might need it too.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: adjust event encoding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_json_formattable: extend for formattable encoding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/ceph_json: formattable encoding fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: events hold a generic formattable

Instead of encoding specific object store events, make it
hold a generic formattable struct, so that pubsub can be
used in the future for registering other event types.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/formatter: json formattable is now also a formatter

This way we can construct a formattable structure by encoding
objects into it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub: use effective conf

the effective conf has the defaults set if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanups and data structure changes, api changes

split user topics and bucket topics, can have multiple buckets per topic
and multiple topics per bucket. Buckets don't hold subscription info.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: override basic rest handler callbacks

these shouldn't do anything there.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: doesn't support quota

Check if handler supports quota, pubsub doesn't.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync module instance specifies whether user writes supported

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: REST api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add args.get_int()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub: pull sub, rm event

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: store event in bucket index

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub config fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: multiple fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: add start_sync() callback

The existing init_sync() is only called when the sync is first initialized,
need a call for whenever sync runs.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: fetch user subs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub related new commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define pubsub user structures

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch topics via a cr, other changes

this will allow fetching topics info from rados, so that we
don't need to hold in in memory.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: more events coverage

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: introduce cr singleton, use for subscriptions init

Prevent concurrent initalizations of the same subscription

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub config cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: register pubsub sync module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: distribution of events to subscriptions

more work required

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: message storage functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new cr for simple object put

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: new object put command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: easy data access helper class

Only does put object right now.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: subscription init and more

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new utility crs

 - get user info
 - create bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial const changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async cr template for crs that return data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create user cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: template helper for async crs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: user tools don't require formatter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a dummy flusher

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial pub sub work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25404 from ptozys/master

async/dpdk: when enable dpdk, multiple message queue defect

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #25460 from tchaikov/wip-krb5-off

auth: do not use GSS/KRB5 if ! HAVE_GSSAPI

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25298 from smanjara/wip-rgw-import

qa/rgw: add missing import line

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25429 from alfredodeza/wip-bz1644847

ceph-volume zap devices associated with an OSD ID and/or OSD FSID

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
Reviewed-by: Sebastien Han <shan@redhat.com>

Merge pull request #25354 from ceph/wip-rm37409

ceph-volume: test with multiple NVME drives

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25120 from Devp00l/wip-ecp-api-test-failure

mgr/dashboard/qa: Fix ECP creation test

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25363 from zmc/wip-fix-dash-ecp-test

mgr/dashboard: Fix a broken ECP controller test

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25391 from SUSE/wip-orchestrator-deepsea-cleanup

mgr/deepsea: check for inflight completions when starting event reader, cleanup logging and comments

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard/qa: Fix ECP creation test

The current solution fails on our CI-system as some outputs can have
more values and some parameters like 'w' can vary in different
environments.

As this was only tested before in a vstart cluster environment it
worked.

Through this commit only the given attributes we know to be there,
will be tested.

Fixes: https://tracker.ceph.com/issues/37275
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24790 from sileht/doc-fix

doc/mgr/dashboard: fix typo in mgr ssl setup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25427 from tspmelo/wip-silent-xi18n

mgr/dashboard: Disable showing xi18n's progress

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

pybind/rados: silence GCC warning

otherwise we will have following warnings:

[1/1] Cythonizing rados.pyx
warning: rados.pyx:91:4: 'rados_t' redeclared
warning: rados.pyx:92:4: 'rados_config_t' redeclared
warning: rados.pyx:93:4: 'rados_ioctx_t' redeclared

these types are already defined by librados.h, so no need to define them
again in rados.pyx .

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: do not use GSS/KRB5 if ! HAVE_GSSAPI

WITH_GSSAPI is a configure-time setting, so we should not compile krb5
related sources if this option is disabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25434 from majianpeng/osd-getpg-reserve

osd/OSD: preallocate for _get_pgs/_get_pgids to avoid reallocate.

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25420 from majianpeng/buffer-repeated-condtion

common/buffer: remove repeated condtion-check.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25424 from rzarzynski/wip-bl-c_str_tracker

common: hint bufferlist's buffer_track_c_str accordingly.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25439 from p-na/pna-wip-cleanup

mgr/dashboard: Remove unused code

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #25425 into master

* refs/pull/25425/head:
test/common/blkdev-udevadm-info-samples: add a few test cases
ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL
common/blkdev: micron SSDs don't include vendor name in ID_SERIAL
ceph-volume: update get_device_id to match in-tree implementation
common/blkdev: improve device_id generation

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

doc: fix typo in mgr ssl setup

Unlike `config set`, `config-key set` does not take the <who>
argument.

This change removes it from mgr ssl setup.

Signed-off-by: Mehdi Abaakouk <sileht@sileht.net>

Merge pull request #25312 from falcon78921/wip-doc-37451

doc: broken link on troubleshooting-mon page

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: broken link on troubleshooting-mon page
Fixes: https://tracker.ceph.com/issues/37451
This fix involves adding :ref: labels to the add-or-rm-mons.rst
page. I also added :ref: labels for other headings within
add-or-rm-mons.rst (for future reference).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #25410 from dragonylffly/rb

tools/rados: fix memory leak in error path

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25393 from Aran85/fix-agent-state

osd: avoid using null agent_state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25370 from majianpeng/sharedlru-addfunc

common/shared_cache: bumps it to the front of the LRU if key existed.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25275 into master

* refs/pull/25275/head:
osdc/Journaler: make try_read_entry store data in contiguous memory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25270 into master

* refs/pull/25270/head:
mds: use uniform write handler for PQ

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24991 into master

* refs/pull/24991/head:
mds: disallow dumping huge caches to formatter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24940 into master

* refs/pull/24940/head:
qa: add test for getfattr ceph.dir.pin
client: support getfattr ceph.dir.pin extended attribute

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.functional.batch symlink test_zap to all batch scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #24748 into master

* refs/pull/24748/head:
qa: use 6h timeout for pjd test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.functional.batch add test_zap yaml to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional.batch create a separate test_zap playbook

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap update success message for OSD IDs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add zapping by osd-id examples

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add zapping by osd-id examples

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25347 from smithfarm/wip-37485

build/ops: rpm: make ceph-grafana-dashboards own its directories

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #25414 from ivancich/wip-rgw-user-stats-mismatch

rgw: fix bad user stats on versioned bucket after reshard

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

test/common/blkdev-udevadm-info-samples: add a few test cases

The erwan1 one unfortunately has no model information :(.  The other two
work with the current implementation, although erwan.v1.sdb has
"Shared_PERC8" for the model, and I suspect the Shared_ prefix is not
part of the real model?

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: micron SSDs don't include vendor name in ID_SERIAL

According to Ryan Meredith <rmeredith@micron.com>, the NVMes begin with
MTFD, but their other devices use the standard Micron_$model_$serial.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: update get_device_id to match in-tree implementation

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: improve device_id generation

A few interesting cases:

1. autriche (my desktop)

$ sudo udevadm info /dev/nvme0n1 | grep ID_
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=c83d5616-676b-4667-bcf3-c82fd4fc7e64
E: ID_SERIAL=Samsung SSD 960 EVO 250GB_S3ESNX0J958081E
E: ID_SERIAL_SHORT=S3ESNX0J958081E

- no ID_MODEL or ID_VENDOR
-> use ID_SERIAL

2. my dev box

gnit:~ (master) 09:09 AM $ udevadm info /dev/nvme0n1 | grep ID_
E: ID_FS_TYPE=xfs
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=860d4503-9c9d-4c24-af09-4266b7717a5c
E: ID_FS_UUID_ENC=860d4503-9c9d-4c24-af09-4266b7717a5c
E: ID_MODEL=INTEL SSDPEDMD400G4
E: ID_PATH=pci-0000:82:00.0-nvme-1
E: ID_PATH_TAG=pci-0000_82_00_0-nvme-1
E: ID_SERIAL=INTEL SSDPEDMD400G4_CVFT520200G7400BGN
E: ID_SERIAL_SHORT=CVFT520200G7400BGN
E: ID_WWN=nvme.8086-43564654353230323030473734303042474e-494e54454c205353445045444d443430304734-00000001

- no ID_VENDOR
-> ID_MODEL + ID_SERIAL_SHORT

gnit:~ (master) 09:12 AM $ udevadm info /dev/sda | grep ID_
E: ID_ATA=1
E: ID_ATA_DOWNLOAD_MICROCODE=1
E: ID_ATA_FEATURE_SET_HPA=1
E: ID_ATA_FEATURE_SET_HPA_ENABLED=1
E: ID_ATA_FEATURE_SET_PM=1
E: ID_ATA_FEATURE_SET_PM_ENABLED=1
E: ID_ATA_FEATURE_SET_SECURITY=1
E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0
E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=2
E: ID_ATA_FEATURE_SET_SECURITY_FROZEN=1
E: ID_ATA_FEATURE_SET_SMART=1
E: ID_ATA_FEATURE_SET_SMART_ENABLED=1
E: ID_ATA_ROTATION_RATE_RPM=0
E: ID_ATA_SATA=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1
E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1
E: ID_ATA_WRITE_CACHE=1
E: ID_ATA_WRITE_CACHE_ENABLED=1
E: ID_BUS=ata
E: ID_MODEL=INTEL_SSDSC2BB240G4
E: ID_MODEL_ENC=INTEL\x20SSDSC2BB240G4\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
E: ID_PART_TABLE_TYPE=dos
E: ID_PART_TABLE_UUID=bb35118c
E: ID_PATH=pci-0000:00:1f.2-ata-1
E: ID_PATH_TAG=pci-0000_00_1f_2-ata-1
E: ID_REVISION=D2010355
E: ID_SERIAL=INTEL_SSDSC2BB240G4_BTWL3414034J240NGN
E: ID_SERIAL_SHORT=BTWL3414034J240NGN
E: ID_TYPE=disk
E: ID_WWN=0x55cd2e404b4e47d8
E: ID_WWN_WITH_EXTENSION=0x55cd2e404b4e47d8

- no ID_VENDOR
-> ID_MODEL + ID_SERIAL_SHORT

3. mira lab machine (old areca controller in JBOD mode, I think)

root@mira055:~# udevadm info /dev/sdb | grep ID_
E: ID_BUS=scsi
E: ID_MODEL=HUS724040ALA640
E: ID_MODEL_ENC=HUS724040ALA640\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PART_TABLE_UUID=957b2db6-de5c-46cb-a672-243fa12d55b2
E: ID_PATH=pci-0000:01:00.0-scsi-0:0:0:1
E: ID_PATH_TAG=pci-0000_01_00_0-scsi-0_0_0_1
E: ID_REVISION=R001
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PN1334PBH5JMJS
E: ID_SERIAL=2001b4d2058da3a00
E: ID_SERIAL_SHORT=001b4d2058da3a00
E: ID_TYPE=disk
E: ID_VENDOR=HGST
E: ID_VENDOR_ENC=HGST\x20\x20\x20\x20

- ID_VENDOR and ID_MODEL
- ID_MODEL doesn't include vendor name + _
- ID_SERIAL and _SHORT are junk
- ID_SCSI_SERIAL has the serial number!  wth
-> ID_VENDOR + ID_MODEL + ID_SCSI_SHORT

Added a bunch of udevadm output samples.

So,

- if ID_VENDOR + ID_MODEL + ID_SCSI_SERIAL are present, use them.
- if ID_MODEL + ID_SERIAL_SHORT are present, use them
- if ID_SERIAL is present, use it.
- fail

Signed-off-by: Sage Weil <sage@redhat.com>

build/ops: rpm: make ceph-grafana-dashboards own its directories on SUSE

This is only required to get the spec file to build in the OpenSUSE
Build Service (OBS). Also, this change could potentially make the
package impossible to install together with grafana (if the latter
owns the same directories with different ownership/permissions).
Therefore, make the change specific to SUSE.

Fixes: http://tracker.ceph.com/issues/37485
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>

mgr/dashboard: Remove unused code

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #25415 from dillaman/wip-rbd-cli-integration

test/cli-integration/rbd: added new parent image attributes

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25234 from jan--f/prometheus-add-iface-objstore-metadata

mgr/prometheus: add interface and objectstore to osd metadata

osd/OSD: preallocate for _get_pgs/_get_pgids to avoid reallocate.

num_pgs is the most accurate value fore reserve when call _get_pgs &
_get_pgids.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #19454 from ifed01/wip-ifed-pool-sizes

osd: per-pool osd stats collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25243 into master

* refs/pull/25243/head:
osdc/Objecter: update op_target_t::paused in _calc_target

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25399 into master

* refs/pull/25399/head:
osd: requeue waiting peering events from deleted slots in unprime_split_children

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25428 from trociny/wip-open-flags

librbd: corrected usage of ImageState::open flag parameter

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25430 from cbodley/wip-pr-22539

rgw: rgwgc:process coredump in some special case

rgw: rgwgc:process coredump in some special case。 Gc processes obja, objb, objc in order and pool of objb is deleted (obja and objc is in the same pool and pool exits). RGW will coredump as ctx->io_ctx_impl is an empty point during delete objc.
Fixes:http://tracker.ceph.com/issues/23199

Signed-off-by: zhaokun <develop@hikdata.com>