From: Yuval Lifshitz Date: Wed, 18 Nov 2020 16:43:16 +0000 (+0200) Subject: rgw/notification: support GetTopicAttributes API X-Git-Tag: v15.2.13~10^2~24^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F40812%2Fhead;p=ceph.git rgw/notification: support GetTopicAttributes API fixes: https://tracker.ceph.com/issues/46296 Signed-off-by: Yuval Lifshitz (cherry picked from commit 3906884aa66b7b6c976d6165cc3b5dfaa8f754c4) Conflicts: PendingReleaseNotes src/rgw/rgw_rest_pubsub.cc --- diff --git a/PendingReleaseNotes b/PendingReleaseNotes index b50111565262..e4eb04ad80dc 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -6,37 +6,5 @@ is enabled. This helps with the monitor logs on larger clusters, that may get many 'osd.X reported immediately failed by osd.Y' messages, and confuse tools. -15.2.9 ------- -* MGR: progress module can now be turned on/off, using the commands: - ``ceph progress on`` and ``ceph progress off``. - -* New bluestore_rocksdb_options_annex config parameter. Complements - bluestore_rocksdb_options and allows setting rocksdb options without repeating - the existing defaults. - -15.2.8 ------- -* $pid expansion in config paths like `admin_socket` will now properly expand - to the daemon pid for commands like `ceph-mds` or `ceph-osd`. Previously only - `ceph-fuse`/`rbd-nbd` expanded `$pid` with the actual daemon pid. - -* ceph-volume: The ``lvm batch` subcommand received a major rewrite. This closed - a number of bugs and improves usability in terms of size specification and - calculation, as well as idempotency behaviour and disk replacement process. - Please refer to https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/ for - more detailed information. - -* MON: The cluster log now logs health detail every ``mon_health_to_clog_interval``, - which has been changed from 1hr to 10min. Logging of health detail will be - skipped if there is no change in health summary since last known. - -* The ``ceph df`` command now lists the number of pgs in each pool. - -* The ``bluefs_preextend_wal_files`` option has been removed. - -* It is now possible to specify the initial monitor to contact for Ceph tools - and daemons using the ``mon_host_override`` config option or - ``--mon-host-override `` command-line switch. This generally should only - be used for debugging and only affects initial communication with Ceph's - monitor cluster. +* An AWS-compliant API: "GetTopicAttributes" was added to replace the existing "GetTopic" API. The new API + should be used to fetch information about topics used for bucket notifications. diff --git a/doc/radosgw/notifications.rst b/doc/radosgw/notifications.rst index 3e5d4fcdaaed..f9cbe187a2cd 100644 --- a/doc/radosgw/notifications.rst +++ b/doc/radosgw/notifications.rst @@ -139,10 +139,67 @@ The topic ARN in the response will have the following format: arn:aws:sns::: +Get Topic Attributes +```````````````````` + +Returns information about a specific topic. This includes push-endpoint information, if provided. + +:: + + POST + + Action=GetTopicAttributes + &TopicArn= + +Response will have the following format: + +:: + + + + + + User + + + + Name + + + + EndPoint + + + + TopicArn + + + + OpaqueData + + + + + + + + + +- User: name of the user that created the topic +- Name: name of the topic +- EndPoint: JSON formatted endpoint parameters, including: + - EndpointAddress: the push-endpoint URL + - EndpointArgs: the push-endpoint args + - EndpointTopic: the topic name that should be sent to the endpoint (may be different than the above topic name) + - HasStoredSecret: "true" if if endpoint URL contain user/password information. In this case request must be made over HTTPS. If not, topic get request will be rejected +- TopicArn: topic ARN +- OpaqueData: the opaque data set on the topic + Get Topic Information ````````````````````` Returns information about specific topic. This includes push-endpoint information, if provided. +Note that this API is now deprecated in favor of the AWS compliant `GetTopicAttributes` API. :: @@ -164,6 +221,7 @@ Response will have the following format: + @@ -177,10 +235,11 @@ Response will have the following format: - User: name of the user that created the topic - Name: name of the topic - EndpointAddress: the push-endpoint URL -- if endpoint URL contain user/password information, request must be made over HTTPS. If not, topic get request will be rejected. - EndpointArgs: the push-endpoint args -- EndpointTopic: the topic name that should be sent to the endpoint (mat be different than the above topic name) +- EndpointTopic: the topic name that should be sent to the endpoint (may be different than the above topic name) +- HasStoredSecret: "true" if endpoint URL contain user/password information. In this case request must be made over HTTPS. If not, topic get request will be rejected - TopicArn: topic ARN +- OpaqueData: the opaque data set on the topic Delete Topic ```````````` diff --git a/doc/radosgw/pubsub-module.rst b/doc/radosgw/pubsub-module.rst index 377afca6dbcc..cf3749b3756c 100644 --- a/doc/radosgw/pubsub-module.rst +++ b/doc/radosgw/pubsub-module.rst @@ -221,7 +221,8 @@ Response will have the following format (JSON): "oid_prefix":"", "push_endpoint":"", "push_endpoint_args":"", - "push_endpoint_topic":"" + "push_endpoint_topic":"", + "stored_secret":"", }, "arn":"" "opaqueData":"" diff --git a/src/rgw/rgw_pubsub.cc b/src/rgw/rgw_pubsub.cc index ca58a7f0384f..a25544bd7723 100644 --- a/src/rgw/rgw_pubsub.cc +++ b/src/rgw/rgw_pubsub.cc @@ -312,6 +312,26 @@ void rgw_pubsub_topic::dump_xml(Formatter *f) const encode_xml("OpaqueData", opaque_data, f); } +void encode_xml_key_value_entry(const std::string& key, const std::string& value, Formatter *f) { + f->open_object_section("entry"); + encode_xml("key", key, f); + encode_xml("value", value, f); + f->close_section(); // entry +} + +void rgw_pubsub_topic::dump_xml_as_attributes(Formatter *f) const +{ + f->open_array_section("Attributes"); + std::string str_user; + user.to_str(str_user); + encode_xml_key_value_entry("User", str_user, f); + encode_xml_key_value_entry("Name", name, f); + encode_xml_key_value_entry("EndPoint", dest.to_json_str(), f); + encode_xml_key_value_entry("TopicArn", arn, f); + encode_xml_key_value_entry("OpaqueData", opaque_data, f); + f->close_section(); // Attributes +} + void encode_json(const char *name, const rgw::notify::EventTypeList& l, Formatter *f) { f->open_array_section(name); @@ -372,6 +392,22 @@ void rgw_pubsub_sub_dest::dump_xml(Formatter *f) const encode_xml("EndpointTopic", arn_topic, f); } +std::string rgw_pubsub_sub_dest::to_json_str() const +{ + // first 2 members are omitted here since they + // dont apply to AWS compliant topics + JSONFormatter f; + f.open_object_section(""); + encode_json("EndpointAddress", push_endpoint, &f); + encode_json("EndpointArgs", push_endpoint_args, &f); + encode_json("EndpointTopic", arn_topic, &f); + encode_json("HasStoredSecret", stored_secret, &f); + f.close_section(); + std::stringstream ss; + f.flush(ss); + return ss.str(); +} + void rgw_pubsub_sub_config::dump(Formatter *f) const { encode_json("user", user, f); diff --git a/src/rgw/rgw_pubsub.h b/src/rgw/rgw_pubsub.h index db7e79010270..6d2fac71b17e 100644 --- a/src/rgw/rgw_pubsub.h +++ b/src/rgw/rgw_pubsub.h @@ -399,6 +399,7 @@ struct rgw_pubsub_sub_dest { void dump(Formatter *f) const; void dump_xml(Formatter *f) const; + std::string to_json_str() const; }; WRITE_CLASS_ENCODER(rgw_pubsub_sub_dest) @@ -472,6 +473,7 @@ struct rgw_pubsub_topic { void dump(Formatter *f) const; void dump_xml(Formatter *f) const; + void dump_xml_as_attributes(Formatter *f) const; bool operator<(const rgw_pubsub_topic& t) const { return to_str().compare(t.to_str()); diff --git a/src/rgw/rgw_rest_pubsub.cc b/src/rgw/rgw_rest_pubsub.cc index 8c757b7c9c6e..4363f185222d 100644 --- a/src/rgw/rgw_rest_pubsub.cc +++ b/src/rgw/rgw_rest_pubsub.cc @@ -19,6 +19,7 @@ #define dout_context g_ceph_context #define dout_subsys ceph_subsys_rgw +static const char* AWS_SNS_NS("https://sns.amazonaws.com/doc/2010-03-31/"); // command (AWS compliant): // POST @@ -76,14 +77,14 @@ public: } const auto f = s->formatter; - f->open_object_section_in_ns("CreateTopicResponse", "https://sns.amazonaws.com/doc/2010-03-31/"); + f->open_object_section_in_ns("CreateTopicResponse", AWS_SNS_NS); f->open_object_section("CreateTopicResult"); encode_xml("TopicArn", topic_arn, f); - f->close_section(); + f->close_section(); // CreateTopicResult f->open_object_section("ResponseMetadata"); encode_xml("RequestId", s->req_id, f); - f->close_section(); - f->close_section(); + f->close_section(); // ResponseMetadata + f->close_section(); // CreateTopicResponse rgw_flush_formatter_and_reset(s, f); } }; @@ -105,14 +106,14 @@ public: } const auto f = s->formatter; - f->open_object_section_in_ns("ListTopicsResponse", "https://sns.amazonaws.com/doc/2010-03-31/"); + f->open_object_section_in_ns("ListTopicsResponse", AWS_SNS_NS); f->open_object_section("ListTopicsResult"); encode_xml("Topics", result, f); - f->close_section(); + f->close_section(); // ListTopicsResult f->open_object_section("ResponseMetadata"); encode_xml("RequestId", s->req_id, f); - f->close_section(); - f->close_section(); + f->close_section(); // ResponseMetadat + f->close_section(); // ListTopicsResponse rgw_flush_formatter_and_reset(s, f); } }; @@ -158,6 +159,47 @@ public: } }; +// command (AWS compliant): +// POST +// Action=GetTopicAttributes&TopicArn= +class RGWPSGetTopicAttributes_ObjStore_AWS : public RGWPSGetTopicOp { +public: + int get_params() override { + const auto topic_arn = rgw::ARN::parse((s->info.args.get("TopicArn"))); + + if (!topic_arn || topic_arn->resource.empty()) { + ldout(s->cct, 1) << "GetTopicAttribute Action 'TopicArn' argument is missing or invalid" << dendl; + return -EINVAL; + } + + topic_name = topic_arn->resource; + return 0; + } + + void send_response() override { + if (op_ret) { + set_req_state_err(s, op_ret); + } + dump_errno(s); + end_header(s, this, "application/xml"); + + if (op_ret < 0) { + return; + } + + const auto f = s->formatter; + f->open_object_section_in_ns("GetTopicAttributesResponse", AWS_SNS_NS); + f->open_object_section("GetTopicAttributesResult"); + result.topic.dump_xml_as_attributes(f); + f->close_section(); // GetTopicAttributesResult + f->open_object_section("ResponseMetadata"); + encode_xml("RequestId", s->req_id, f); + f->close_section(); // ResponseMetadata + f->close_section(); // GetTopicAttributesResponse + rgw_flush_formatter_and_reset(s, f); + } +}; + // command (AWS compliant): // POST // Action=DeleteTopic&TopicArn= @@ -187,11 +229,11 @@ public: } const auto f = s->formatter; - f->open_object_section_in_ns("DeleteTopicResponse", "https://sns.amazonaws.com/doc/2010-03-31/"); + f->open_object_section_in_ns("DeleteTopicResponse", AWS_SNS_NS); f->open_object_section("ResponseMetadata"); encode_xml("RequestId", s->req_id, f); - f->close_section(); - f->close_section(); + f->close_section(); // ResponseMetadata + f->close_section(); // DeleteTopicResponse rgw_flush_formatter_and_reset(s, f); } }; @@ -321,16 +363,14 @@ RGWOp* RGWHandler_REST_PSTopic_AWS::op_post() { return new RGWPSListTopics_ObjStore_AWS(); if (action.compare("GetTopic") == 0) return new RGWPSGetTopic_ObjStore_AWS(); + if (action.compare("GetTopicAttributes") == 0) + return new RGWPSGetTopicAttributes_ObjStore_AWS(); } return nullptr; } int RGWHandler_REST_PSTopic_AWS::authorize(const DoutPrefixProvider* dpp) { - /*if (s->info.args.exists("Action") && s->info.args.get("Action").find("Topic") != std::string::npos) { - // TODO: some topic specific authorization - return 0; - }*/ return RGW_Auth_S3::authorize(dpp, store, auth_registry, s); } diff --git a/src/rgw/rgw_rest_pubsub.h b/src/rgw/rgw_rest_pubsub.h index f2f63356048b..b55815bf6727 100644 --- a/src/rgw/rgw_rest_pubsub.h +++ b/src/rgw/rgw_rest_pubsub.h @@ -27,7 +27,6 @@ class RGWHandler_REST_PSTopic_AWS : public RGWHandler_REST { const rgw::auth::StrategyRegistry& auth_registry; const std::string& post_body; void rgw_topic_parse_input(); - //static int init_from_header(struct req_state *s, int default_formatter, bool configurable_format); protected: RGWOp* op_post() override; public: diff --git a/src/test/rgw/rgw_multi/tests_ps.py b/src/test/rgw/rgw_multi/tests_ps.py index 83e88f783fc6..aa190d507ee5 100644 --- a/src/test/rgw/rgw_multi/tests_ps.py +++ b/src/test/rgw/rgw_multi/tests_ps.py @@ -865,6 +865,10 @@ def test_ps_s3_topic_on_master(): assert_equal(topic_arn, result['GetTopicResponse']['GetTopicResult']['Topic']['TopicArn']) assert_equal(endpoint_address, result['GetTopicResponse']['GetTopicResult']['Topic']['EndPoint']['EndpointAddress']) # Note that endpoint args may be ordered differently in the result + result = topic_conf3.get_attributes() + assert_equal(topic_arn, result['Attributes']['TopicArn']) + json_endpoint = json.loads(result['Attributes']['EndPoint']) + assert_equal(endpoint_address, json_endpoint['EndpointAddress']) # delete topic 1 result = topic_conf1.del_config() @@ -873,6 +877,12 @@ def test_ps_s3_topic_on_master(): # try to get a deleted topic _, status = topic_conf1.get_config() assert_equal(status, 404) + try: + topic_conf1.get_attributes() + except: + print('topic already deleted - this is expected') + else: + assert False, 'topic 1 should be deleted at this point' # get the remaining 2 topics result, status = topic_conf1.get_list() diff --git a/src/test/rgw/rgw_multi/zone_ps.py b/src/test/rgw/rgw_multi/zone_ps.py index 84413f5565d8..3adfc8fd6878 100644 --- a/src/test/rgw/rgw_multi/zone_ps.py +++ b/src/test/rgw/rgw_multi/zone_ps.py @@ -187,6 +187,7 @@ class PSTopicS3: POST ?Action=CreateTopic&Name=[&OpaqueData=[&push-endpoint=&[=...]]] POST ?Action=ListTopics POST ?Action=GetTopic&TopicArn= + POST ?Action=GetTopicAttributes&TopicArn= POST ?Action=DeleteTopic&TopicArn= """ def __init__(self, conn, topic_name, region, endpoint_args=None, opaque_data=None): @@ -239,6 +240,10 @@ class PSTopicS3: dict_response = xmltodict.parse(data) return dict_response, status + def get_attributes(self): + """get topic attributes""" + return self.client.get_topic_attributes(TopicArn=self.topic_arn) + def set_config(self): """set topic""" result = self.client.create_topic(Name=self.topic_name, Attributes=self.attributes)