]> git-server-git.apps.pok.os.sepia.ceph.com Git - ceph.git/commit
rgw: support kafka 4.x brokers 69666/head
authorShreeJejurikar <shreemj8@gmail.com>
Sat, 20 Jun 2026 10:58:49 +0000 (16:28 +0530)
committerYuval Lifshitz <ylifshit@ibm.com>
Wed, 22 Jul 2026 10:29:04 +0000 (10:29 +0000)
commitefe54523a45ef1ba1f824dd55705a42a00fbd73e
treeee203e36e732d72011fd0440accefe73ce145270
parent179b4aeed129851a3e64e89b231381a8502fdb97
rgw: support kafka 4.x brokers

librdkafka submodule

Vendors librdkafka v2.12.1 as a submodule (default) and teaches the
notification teuthology task to set up Kafka 4.x KRaft brokers. The
distro-shipped librdkafka 1.6.1 cannot SCRAM-auth against Kafka 4.x
brokers (#75900); the bundled version fixes that. Build with
WITH_SYSTEM_RDKAFKA=ON to opt back into the distro library (floor 2.6.1).
follwinf changes needed with teh submodule:
* build kafka_stub only in case of system librdkafka
* use the same header location in submodule and system
* supress compilation warning to avoid -WError issues

2.12.1 matches Fedora(44)'s current librdkafka-devel, so the vendored copy
tracks a version that distro packagers have already vetted rather than
whichever release happens to be latest upstream.

The teuthology version matrix picks one of {3.9.2, 4.1, 4.2, 4.3} per run.
3.9.2 covers the last Zookeeper-based release for the migration cohort.
4.1/4.2/4.3 track Apache's current three supported minor lines and are
resolved to the latest patch at test time via dlcdn.apache.org/kafka/, so
new patches within a supported minor need no PR update. A new minor line
(4.4) requires renaming one YAML.
* new java version (17) is needed for kafka 4.2 and up (we wil use it on
  all version)
* kerberos task need to run before the kafka install task

After pulling this branch, fetch the new submodule:

    git submodule update --init src/librdkafka

Fixes: https://tracker.ceph.com/issues/77336
Signed-off-by: ShreeJejurikar <shreemj8@gmail.com>
Co-authored-by: Yuval Lifshitz <ylifshit@ibm.com>
22 files changed:
.gitmodules
CMakeLists.txt
ceph.spec.in
debian/control
debian/rules
doc/radosgw/notifications.rst
qa/suites/rgw/notifications/tasks/kafka/0-install.yaml
qa/suites/rgw/notifications/tasks/kafka/1-kerberos.yaml [new file with mode: 0644]
qa/suites/rgw/notifications/tasks/kafka/1-versions$/.qa [new symlink]
qa/suites/rgw/notifications/tasks/kafka/1-versions$/3.9.2.yaml [new file with mode: 0644]
qa/suites/rgw/notifications/tasks/kafka/1-versions$/4.1.yaml [new file with mode: 0644]
qa/suites/rgw/notifications/tasks/kafka/1-versions$/4.2.yaml [new file with mode: 0644]
qa/suites/rgw/notifications/tasks/kafka/1-versions$/4.3.yaml [new file with mode: 0644]
qa/suites/rgw/notifications/tasks/kafka/2-notification-tests.yaml [new file with mode: 0644]
qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml [deleted file]
qa/tasks/kafka.py
qa/tasks/kafka_failover.py
src/librdkafka [new submodule]
src/rgw/CMakeLists.txt
src/test/CMakeLists.txt
src/test/rgw/CMakeLists.txt
src/test/rgw/bucket_notification/README.rst