mgr/k8sevents: Initial ceph -> k8s events integration

Adds a mgr module to provide integration between
Ceph and the kubernetes events API within the rook-ceph
namespace. It provides several commands to view event
status including;

ceph k8sevents list .... show all k8s related events
ceph k8sevents ceph .... show events generated from this
                         module
ceph k8sevents status .. show status of the tracker threads
                         and counts of tracked events

Events sent to kubernetes are sourced from the
a) clog: ceph healtchecks and admin commands
b) explicit checks: hosts, pools and OSD states are checked
   every 'n' seconds

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #26547 into master

* refs/pull/26547/head:
Added validation of zabbix_host to support hostnames, IPv4 and IPv6.
mgr/zabbix: Documentation added.
mgr/zabbix: Adds possibility to send data to multiple zabbix servers.

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge PR #27667 into master

* refs/pull/27667/head:
src/test: fix creating two different objects for testing chunked object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28200 into master

* refs/pull/28200/head:
pybind/mgr: add slash to prometheus illegal metric name characters
pybind/mgr: fix format for rbd-mirror prometheus metrics

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28180 from athanatos/sjust/wip-crimson-peering

Integrate PeeringState into crimson, fix related bugs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28068 from cbodley/wip-qa-rgw-s3a

qa/rgw: don't use ceph-ansible in s3a-hadoop suite

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #28300 into master

* refs/pull/28300/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/rgw: add dnsmasq back to s3a-hadoop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw task can override --rgw-dns-name on the command line

the value for rgw_dns_name isn't known until a machine is assigned, so
it can't be set via 'override: conf:'. add a per-client config option
to the rgw task so it can add the endpoint's hostname and/or s3website
hostname on the radosgw command line

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: remove ceph-ansible from s3a-hadoop suite

Fixes: http://tracker.ceph.com/issues/39706
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: use default ports (80 or 443) unless overridden

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: allow rgw client config to override port

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28146 from smanjara/wip-39551

rgw/multisite:RGWListBucketIndexesCR for data full sync pagination

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26915 from dvanders/dvanders_doc_heap

doc: describe metadata_heap cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28359 from tchaikov/wip-split-ceph-dencoder

tools/ceph-dencoder: split types.h into smaller pieces

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: describe metadata_heap cleanup

Fixes: http://tracker.ceph.com/issues/18174
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #28305 from rzarzynski/wip-bug-38827

qa/valgrind.supp: generalize the whiterule for aes-128-gcm to help rgw suite.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25498 from cbodley/wip-27217

rgw: parse_copy_location defers url-decode

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #16659 from rzarzynski/wip-rgw-20797

rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12709 from rzarzynski/wip-rgw-18364-dbgs

rgw: improve debugs on the path of RGWRados::cls_bucket_head.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12470 from rzarzynski/wip-rgw-18241

rgw: dump s3_code as the Code response element in RGWDeleteMultiObj_ObjStore_S3

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28339 from tchaikov/wip-cmake-asan

cmake/modules: replace ";" with " " in compile flags

Reviewed-by: Yingxin Cheng <yingxincheng@gmail.com>

tools/ceph-dencoder: split types.h into smaller pieces

instead of putting all types in a single compilation unit, split them
into smaller groups, in hope to reduce the memory footprint of compiler
when compiling ceph_dencoder.cc.

sometimes, GCC just fails to compile this source file.

Fixes: http://tracker.ceph.com/issues/39595
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #28354 into master

* refs/pull/28354/head:
doc: Add warning that the root directory cannot be fragmented

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/: clarify rebuilt_missing_with_deletes and set in init

Updates PeeringState::init to set may_include_deletes_in_missing when
possible to avoid needlessly rebuilding the missing set on the next map.
The previous behavior was also subtly wrong as until the the next map
the osd would be possibly adding deletes to the missing set without the
flag having been set in the missing set on disk

This patch also renames the flag to reflect the fact that it simply
denotes wheter the may_include_deletes_in_missing omap entry needs to be
written out.  This may happen during rebuild_missing_set_with_deletes,
but it'll also happen more commonly during init now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: hook PeeringState into pg

This patch:
1) Adds partially implemented stubs for PeeringState::PeeringListener
2) Revamps the message handling to batch the peering events from
   a single message.
3) Replaces the message handling with hand-offs to PeeringState
4) Adds support for MOSDPGCreate2 enabling creation of pgs.
5) Introduces ShardServices for exposing osd machinery to the pgs.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/main: add --debug option

Really, what we need to do is examine the ceph conf and set
the right logger levels based on configured log levels, but
this'll suffice for now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: clear exists in remove

Signed-off-by: Samuel Just <sjust@redhat.com>

cyan_store: map -ENOENT to 0 for _remove

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/ProtocolV1: close connection when replaced

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "cmake: define BOOST_SP_DISABLE_THREADS for crimson"

For whatever reason, this patch appears to cause the reactor/peering
to hang much of the time.  Reverting until we have time to debug it.

This reverts commit a09a4f342c0a0fddcae29fe6c8d97e3b353f8c0c.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/assert.cc: flush cout before abort

Seastar logger() instances don't flush the channel, so recent messages
won't be present otherwise.

Signed-off-by: Samuel Just <sjust@redhat.com>

dout: let <= 20 correspond to debug for crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: add --redirect-output option to redirect output for --nodaemon

Useful for crimson which doesn't currently support writing directly
to a log file.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: use configured ceph-osd uuid in mkfs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/Connection: add peer_is_* helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/SocketMessenger: wait for accepter thread to die on shutdown

Otherwise, it ends up trying to operator<< *this after it's been
deleted.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/mon/MonClient: handle mon client connection race

The first to establish a connection and close the others needs
to ensure that the others don't mess with broken state.  To that
end, leave a cleared pending_conns map while the conns are being
closed.  Also, take care to clean up outstanding promises and
fallout from conns disappearing.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/cyan_store: run t.get_on_* contexts

PeeringState makes liberal use of those callbacks, run them
upon completion of do_transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/cyan_store: add some omap machinery

Adds transaction methods for PeeringState.

Mainly omap_set, but also a stub for collection hint and the other omap
update methods.  Also adds omap_get and debugging to dump the
transaction upon encountering a problem.

Also fixes up std and ceph::bufferlist namespace qualifications.

Signed-off-by: Samuel Just <sjust@redhat.com>

os/cyan_collection: add pragma once

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move pg/osd state machinery into osd_types for use in crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

PGLog: add cyan_store machinery for reading the log

Signed-off-by: Samuel Just <sjust@redhat.com>

doc: Add warning that the root directory cannot be fragmented

Documents: #39704

Signed-off-by: Nathan Fish <lordcirth@gmail.com>

Merge PR #28199 into master

* refs/pull/28199/head:
qa/tasks: upgrade the check for -c sudo option in vstart_runner.py

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28108 from linuxbox2/wip-rgwfile-reqid

Wip rgwfile reqid:  absorbs rgw_file: allocate new id for continued request #25664

Merge pull request #28135 from dillaman/wip-39946

qa/workunits/rbd: remove 'fast-diff' from dynamic features test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28295 from dillaman/wip-39952

librbd: do not unblock IO prior to growing object map during resize

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28296 from mcv21/doc-profile-rbd

doc: note explicitly that "profile rbd" allows blacklisting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake/modules: replace ";" with " " in compile flags

as the flags should be a string, not a list

tested using

cmake -DCMAKE_LINKER=gold -DWITH_ASAN=ON -DWITH_ASAN_LEAK=ON \
-DCMAKE_BUILD_TYPE=Debug

on ubuntu xenial

without this change, cmake will fail with following warning:

CMake Error at src/CMakeLists.txt:112 (message):
  The compiler /usr/bin/c++ has no C++17 support.

-- Configuring incomplete, errors occurred!

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28328 from tchaikov/wip-fix-ceph-helper-test

qa/standalone/ceph-helpers: resurrect all OSD before waiting for health

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28338 from Jing-Scott/dev/modify-scrub-comments

osd: update comment as sub_op_scrub_map has been removed

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: the sub_op_scrub_map has been removed
sub_op_scrub_map replaced by do_replica_scrub_map, sync the comments.

Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

Merge pull request #25099 from Thingee/master

doc: Adding US-Mid-West Mirror to docs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25530 into master

* refs/pull/25530/head:
mgr/diskprediction: Add diskprediction local plugin dependencies

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

doc/install/mirrors: Adding US-Mid-West Mirror to docs

Signed-off-by: Mike Perez <thingee@gmail.com>

Merge pull request #27898 from alimaredia/wip-rgw-thread-yield-context-get-bucket-info

Thread optional yield context through  get_bucket_info call path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

PeeringState: permit init_from_disk_state to return a value

This way it can return a future.

Signed-off-by: Samuel Just <sjust@redhat.com>

PGPeeringEvent: move MLogRec constructor definition to .cc and add unique_ptr

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd/: add appropriate OSDMapRef defs for seastar to OSDMap,osd_types

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: modernize accept_buffered_messages

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: Move PeeringCtx out of PeeringState to permit forward declaration

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os: Move CollectionRef out of CyanStore

This way, we don't need cyan_store.h to get CollectionRef

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #27957 into master

* refs/pull/27957/head:
json: JSONDecoder::err inherits from std::runtime_error
rgw: RGWXMLDecoder::err inherits from std::runtime_error

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #28317 from tchaikov/wip-crimson-net-remove-redundant-move

crimson/net: remove redundant std::move()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28289 from tchaikov/wip-kill-gcc-9-warnings

common,os: address string truncated warnings from GCC-9

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28055 from SUSE/wip-fix-39662

mgr/PyModuleRegistry: log error if we can't find any modules to load

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/standalone/ceph-helpers: resurrect all OSD before waiting for health

address the regression introduced by e62cfceb
in e62cfceb, we wanted to test the newly introduced TOO_FEW_OSDS
warning, so we increased the number of OSD to the size of pool, so if
the number of OSD is less than pool size, monitor will send a warning
message.

but we need to bring all OSDs back if we are expecting a healthy
cluster. in this change, all OSDs are resurrect before
`wait_for_health_ok`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/run-standalone.sh: s/Makefile/CMakeCache.txt/

as we could use a generator other than Make, for instance ninja.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: do not unblock IO prior to growing object map during resize

This could result in a small race condition where IO is able to write
beyond the current extent of the object map, resulting in an assertion
failure.

Fixes: http://tracker.ceph.com/issues/39952
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: parse_copy_location defers url-decode

don't url-decode until after we search for the ?, or we'll truncate
object names that contain a url-encoded ?

Fixes: http://tracker.ceph.com/issues/27217
Signed-off-by: Casey Bodley <cbodley@redhat.com>

crimson/net: remove redundant std::move()

see https://en.cppreference.com/w/cpp/language/copy_elision

it also silences warning like:

src/crimson/net/ProtocolV2.cc:146:26: warning: redundant move in return
statement [-Wredundant-move]
  146 |       return std::move(bl);
      |                          ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27806 from ashitakasam/add-osd-alarm

osd: Better error message when OSD count is less than osd_pool_default_size

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28278 from tchaikov/wip-buffer-remove-advance-size_t

buffer.h: remove list::iterator_impl::advance(size_t)

Reviewed-by: James Page <james.page@ubuntu.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28251 from xiexingguo/wip-balancer-new-osd-weight

mgr/balancer: fix initial weight-set value for newly created osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28107 from ifed01/wip-ifed-add-devices

os/bluestore: show device name in "osd metadata output"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28277 from javacruft/32bit-compat-fixes

common: avoid use of size_t in options

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28250 from tchaikov/wip-silence-warnings

mon,test: silence warnings from GCC and test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr: add slash to prometheus illegal metric name characters

(replace with underscore)

While here, change the sequence of replaces to a single regex, as it
becomes too long.

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr: fix format for rbd-mirror prometheus metrics

Fixes: https://tracker.ceph.com/issues/39977
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25729 from the78mole/wip-32-bit-arm-fixes

cmake: boost fixes for ARM 32 bit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28255 from tchaikov/wip-convenience

common: no need to include ceph_assert.h

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28270 from NTmatter/patch-2

blobhash.h: remove extra [[fallthrough]]

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "qa: remove mon valgrind check in rgw verfiy suite"

This reverts commit d25d6ee0af0c85715e0ef8ce377f0b78a91f7ad7.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/valgrind.supp: fix the name for aes-128-gcm whiterules.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/valgrind.supp: generalize the whiterule for aes-128-gcm.

Fixes: http://tracker.ceph.com/issues/38827
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #27799 into master

* refs/pull/27799/head:
common/utime: don't pass %z to utime if there is a 'Z'
test: test json encode/decode of utime_t
mgr/ssh: parse new datetime
mgr/devicehealth: parse new datetime
mgr/crash: parse both old and new timestamp formats
mgr/telemetry: use cluster-provided timestamp unmolested
mon/MonMap: dump timestamps in UTC
qa/tasks/ceph: tolerate 'T' or ' ' as date and time separator
PendingReleaseNotes: note about change to ISO 8601 throughout
test/cli: update regexs for timestamps
log/LogClock: render timestamp in ISO 8601 format
common/ceph_time: stringify in ISO 8601 format
unittest_utime: add tests
common/utime: make parse() handle (our) ISO 8601 output
include/utime: make default string rendering ISO 8601 conformant
include/utime: remove unused s[n]printf methods
include/utime: make gmtime() output conformant ISO 8601

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

common/utime: don't pass %z to utime if there is a 'Z'

This fails on centos 7 for some reason.

If it's UTC, we don't need to parse the tz portion, so avoid the issue by
leaving the %z out of the fmt string.

Signed-off-by: Sage Weil <sage@redhat.com>

test: test json encode/decode of utime_t

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/ssh: parse new datetime

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/devicehealth: parse new datetime

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: parse both old and new timestamp formats

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/telemetry: use cluster-provided timestamp unmolested

The cluster stamp is now ISO 8601; just use that.

(The isoformat() puts a : in +hh:mm the timezone offset, which is slightly
different than what Ceph does; just pass Ceph's value through for
consistency.)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonMap: dump timestamps in UTC

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: tolerate 'T' or ' ' as date and time separator

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: note about change to ISO 8601 throughout

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli: update regexs for timestamps

Signed-off-by: Sage Weil <sage@redhat.com>

log/LogClock: render timestamp in ISO 8601 format

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_time: stringify in ISO 8601 format

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_utime: add tests

Signed-off-by: Sage Weil <sage@redhat.com>