rgw: pubsub: send missing http headers in certain rest requests

PUT and DELETE requests still need to make sure http status code
is returned.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: dump topics as array

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some added logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't use self_ref

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: list events returns empty list if bucket doesn't exist

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common: add missing get_str_set() implementation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustments following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

docs/rgw: document pubsub

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: trivial push notifications

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: filter events

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/str_list: get_str_set(), add second param for template

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: define second param for std::set encode_json() template

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: fix events distribution

check was located in the wrong place

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: remove warning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: fix encoding of json config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub topic get command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: dest bucket registers with lifecycle

So that events expire after X days

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more lifecycle code cleanup and tooling

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle utility coroutine

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: have sub name as event oid prefix

dest should be unique for each subscription

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add a new lc get command

Useful for debugging lifecycle issues

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: create notif via REST api sets dest bucket

This got lost somewhere in the refactoring work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutine set_state() also sets retcode

Generally recode is needed for the caller, but current might need it too.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: adjust event encoding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test_json_formattable: extend for formattable encoding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/ceph_json: formattable encoding fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: events hold a generic formattable

Instead of encoding specific object store events, make it
hold a generic formattable struct, so that pubsub can be
used in the future for registering other event types.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/formatter: json formattable is now also a formatter

This way we can construct a formattable structure by encoding
objects into it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub: use effective conf

the effective conf has the defaults set if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cleanups and data structure changes, api changes

split user topics and bucket topics, can have multiple buckets per topic
and multiple topics per bucket. Buckets don't hold subscription info.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: override basic rest handler callbacks

these shouldn't do anything there.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: doesn't support quota

Check if handler supports quota, pubsub doesn't.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync module instance specifies whether user writes supported

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: REST api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add args.get_int()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub: pull sub, rm event

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: store event in bucket index

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub config fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: multiple fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: add start_sync() callback

The existing init_sync() is only called when the sync is first initialized,
need a call for whenever sync runs.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: fetch user subs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: pubsub related new commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define pubsub user structures

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch topics via a cr, other changes

this will allow fetching topics info from rados, so that we
don't need to hold in in memory.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: more events coverage

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: introduce cr singleton, use for subscriptions init

Prevent concurrent initalizations of the same subscription

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub config cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: register pubsub sync module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: distribution of events to subscriptions

more work required

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: message storage functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new cr for simple object put

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: new object put command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: easy data access helper class

Only does put object right now.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: subscription init and more

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new utility crs

 - get user info
 - create bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial const changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async cr template for crs that return data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create user cr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: template helper for async crs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: user tools don't require formatter

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create a dummy flusher

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial pub sub work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25404 from ptozys/master

async/dpdk: when enable dpdk, multiple message queue defect

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #25460 from tchaikov/wip-krb5-off

auth: do not use GSS/KRB5 if ! HAVE_GSSAPI

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25298 from smanjara/wip-rgw-import

qa/rgw: add missing import line

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25429 from alfredodeza/wip-bz1644847

ceph-volume zap devices associated with an OSD ID and/or OSD FSID

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
Reviewed-by: Sebastien Han <shan@redhat.com>

Merge pull request #25354 from ceph/wip-rm37409

ceph-volume: test with multiple NVME drives

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25120 from Devp00l/wip-ecp-api-test-failure

mgr/dashboard/qa: Fix ECP creation test

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25363 from zmc/wip-fix-dash-ecp-test

mgr/dashboard: Fix a broken ECP controller test

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25391 from SUSE/wip-orchestrator-deepsea-cleanup

mgr/deepsea: check for inflight completions when starting event reader, cleanup logging and comments

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard/qa: Fix ECP creation test

The current solution fails on our CI-system as some outputs can have
more values and some parameters like 'w' can vary in different
environments.

As this was only tested before in a vstart cluster environment it
worked.

Through this commit only the given attributes we know to be there,
will be tested.

Fixes: https://tracker.ceph.com/issues/37275
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24790 from sileht/doc-fix

doc/mgr/dashboard: fix typo in mgr ssl setup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25427 from tspmelo/wip-silent-xi18n

mgr/dashboard: Disable showing xi18n's progress

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

pybind/rados: silence GCC warning

otherwise we will have following warnings:

[1/1] Cythonizing rados.pyx
warning: rados.pyx:91:4: 'rados_t' redeclared
warning: rados.pyx:92:4: 'rados_config_t' redeclared
warning: rados.pyx:93:4: 'rados_ioctx_t' redeclared

these types are already defined by librados.h, so no need to define them
again in rados.pyx .

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: do not use GSS/KRB5 if ! HAVE_GSSAPI

WITH_GSSAPI is a configure-time setting, so we should not compile krb5
related sources if this option is disabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25434 from majianpeng/osd-getpg-reserve

osd/OSD: preallocate for _get_pgs/_get_pgids to avoid reallocate.

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25420 from majianpeng/buffer-repeated-condtion

common/buffer: remove repeated condtion-check.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25424 from rzarzynski/wip-bl-c_str_tracker

common: hint bufferlist's buffer_track_c_str accordingly.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25439 from p-na/pna-wip-cleanup

mgr/dashboard: Remove unused code

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #25425 into master

* refs/pull/25425/head:
test/common/blkdev-udevadm-info-samples: add a few test cases
ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL
common/blkdev: micron SSDs don't include vendor name in ID_SERIAL
ceph-volume: update get_device_id to match in-tree implementation
common/blkdev: improve device_id generation

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

doc: fix typo in mgr ssl setup

Unlike `config set`, `config-key set` does not take the <who>
argument.

This change removes it from mgr ssl setup.

Signed-off-by: Mehdi Abaakouk <sileht@sileht.net>

Merge pull request #25312 from falcon78921/wip-doc-37451

doc: broken link on troubleshooting-mon page

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: broken link on troubleshooting-mon page
Fixes: https://tracker.ceph.com/issues/37451
This fix involves adding :ref: labels to the add-or-rm-mons.rst
page. I also added :ref: labels for other headings within
add-or-rm-mons.rst (for future reference).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #25410 from dragonylffly/rb

tools/rados: fix memory leak in error path

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25393 from Aran85/fix-agent-state

osd: avoid using null agent_state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25370 from majianpeng/sharedlru-addfunc

common/shared_cache: bumps it to the front of the LRU if key existed.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #25275 into master

* refs/pull/25275/head:
osdc/Journaler: make try_read_entry store data in contiguous memory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25270 into master

* refs/pull/25270/head:
mds: use uniform write handler for PQ

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24991 into master

* refs/pull/24991/head:
mds: disallow dumping huge caches to formatter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24940 into master

* refs/pull/24940/head:
qa: add test for getfattr ceph.dir.pin
client: support getfattr ceph.dir.pin extended attribute

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.functional.batch symlink test_zap to all batch scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #24748 into master

* refs/pull/24748/head:
qa: use 6h timeout for pjd test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests.functional.batch add test_zap yaml to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional.batch create a separate test_zap playbook

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap update success message for OSD IDs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man/ceph-volume add zapping by osd-id examples

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add zapping by osd-id examples

Signed-off-by: Alfredo Deza <adeza@redhat.com>