]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
2 years agoMerge pull request #47170 from guits/dep-dev-inventory
Guillaume Abrioux [Thu, 28 Jul 2022 20:19:31 +0000 (22:19 +0200)]
Merge pull request #47170 from guits/dep-dev-inventory

ceph-volume: report slave devices in inventory

2 years agoceph-volume/tests: migrate to pyfakefs 47170/head
Guillaume Abrioux [Wed, 27 Jul 2022 11:41:41 +0000 (13:41 +0200)]
ceph-volume/tests: migrate to pyfakefs

ceph-volume unit tests shouldn't actually create contents on the
filesystem from where it runs (even though they are written in a tmp
dir), let's use pyfakefs.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoceph-volume/tests: fix test_path_is_valid()
Guillaume Abrioux [Wed, 27 Jul 2022 12:28:10 +0000 (14:28 +0200)]
ceph-volume/tests: fix test_path_is_valid()

When ceph-volume tests are run from a host where there's no `/dev/sda`
device, this test fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoceph-volume: report slave devices in inventory
Guillaume Abrioux [Tue, 19 Jul 2022 12:06:10 +0000 (12:06 +0000)]
ceph-volume: report slave devices in inventory

`ceph-volume inventory` currently reports the following:

```

Device Path               Size        rotates available Model name
/dev/mapper/mpatha        50.00 GB    True    True
/dev/mapper/mpathaa       6.00 GB     True    True
/dev/mapper/mpathab       6.00 GB     True    True
/dev/mapper/mpathac       6.00 GB     True    True
/dev/mapper/mpathad       6.00 GB     True    True
/dev/mapper/mpathae       6.00 GB     True    True
```

whereas something like following would be useful:

```

Device Path               Size         Device nodes    rotates available Model name
/dev/mapper/mpatha        50.00 GB     sdf,sde         True    True
/dev/mapper/mpathaa       6.00 GB      sdbe,sdat       True    True
/dev/mapper/mpathab       6.00 GB      sdav,sdbf       True    True
/dev/mapper/mpathac       6.00 GB      sdbb,sdbl       True    True
/dev/mapper/mpathad       6.00 GB      sdas,sdbc       True    True
/dev/mapper/mpathae       6.00 GB      sdax,sdbh       True    True
```

Fixes: https://tracker.ceph.com/issues/56624
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoMerge pull request #47169 from guits/improve-mpath-reporting
Guillaume Abrioux [Thu, 28 Jul 2022 18:11:30 +0000 (20:11 +0200)]
Merge pull request #47169 from guits/improve-mpath-reporting

ceph-volume: improve mpath devices reporting

2 years agoMerge pull request #46893 from guits/ceph-volume-symlinks-guits
Guillaume Abrioux [Thu, 28 Jul 2022 18:11:03 +0000 (20:11 +0200)]
Merge pull request #46893 from guits/ceph-volume-symlinks-guits

ceph-volume: allow symlinks as devices

2 years agoMerge pull request #47310 from tchaikov/ceph-context-formatting
Kefu Chai [Thu, 28 Jul 2022 14:39:33 +0000 (22:39 +0800)]
Merge pull request #47310 from tchaikov/ceph-context-formatting

common/ceph_context: fix the formatting

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2 years agoMerge pull request #46996 from rhcs-dashboard/html-injection-bug
Pere Diaz Bou [Thu, 28 Jul 2022 07:38:41 +0000 (09:38 +0200)]
Merge pull request #46996 from rhcs-dashboard/html-injection-bug

mgr/dashboard: added pattern validaton for form input

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2 years agoMerge pull request #46970 from ronen-fr/wip-rf-unlocked-logs
Ronen Friedman [Thu, 28 Jul 2022 05:48:37 +0000 (08:48 +0300)]
Merge pull request #46970 from ronen-fr/wip-rf-unlocked-logs

osd/scrub: pre-prepare the description text of the current state

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
2 years agoMerge pull request #46024 from sirtoobii/patch-2
zdover23 [Thu, 28 Jul 2022 04:38:28 +0000 (14:38 +1000)]
Merge pull request #46024 from sirtoobii/patch-2

doc/radosgw/STS: sts_key and user capabilities

Reviewed-by: Zac Dover <zac.dover@gmail.com>
2 years agoMerge pull request #44039 from richael02/update-isal-crypto-source
Kefu Chai [Thu, 28 Jul 2022 02:29:28 +0000 (10:29 +0800)]
Merge pull request #44039 from richael02/update-isal-crypto-source

.gitmodules: update isa-l_crypto source address

Reviewed-by: Yibo Cai <yibo.cai@arm.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #46420 from pcuzner/add-disk-rescan
Adam King [Wed, 27 Jul 2022 23:50:59 +0000 (19:50 -0400)]
Merge pull request #46420 from pcuzner/add-disk-rescan

mgr/cephadm: Add disk rescan feature to the orchestrator

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
2 years agoMerge pull request #47286 from tchaikov/wip-fmt-ranges
Kefu Chai [Wed, 27 Jul 2022 23:50:43 +0000 (07:50 +0800)]
Merge pull request #47286 from tchaikov/wip-fmt-ranges

osd: use fmt/ranges.h for printing ranges

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agoMerge pull request #46883 from adk3798/custom-config
Adam King [Wed, 27 Jul 2022 23:49:44 +0000 (19:49 -0400)]
Merge pull request #46883 from adk3798/custom-config

mgr/cephadm: support for miscellaneous config files for daemons

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
2 years agoMerge pull request #46955 from hookak/monitoring-stack
Adam King [Wed, 27 Jul 2022 23:48:09 +0000 (19:48 -0400)]
Merge pull request #46955 from hookak/monitoring-stack

cephadm: Add promtail and loki when bootstrap

Reviewed-by: Adam King <adking@redhat.com>
2 years agoMerge pull request #47088 from adk3798/cephadm-logging
Adam King [Wed, 27 Jul 2022 23:47:16 +0000 (19:47 -0400)]
Merge pull request #47088 from adk3798/cephadm-logging

cephadm: reduce spam to cephadm.log

Reviewed-by: John Mulligan <jmulligan@redhat.com>
2 years agocommon/ceph_context: fix the formatting 47310/head
Kefu Chai [Wed, 27 Jul 2022 23:46:55 +0000 (07:46 +0800)]
common/ceph_context: fix the formatting

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #47178 from adk3798/logrotate-owner
Adam King [Wed, 27 Jul 2022 23:46:04 +0000 (19:46 -0400)]
Merge pull request #47178 from adk3798/logrotate-owner

cephadm: add "su root root" to cephadm.log logrotate config

Reviewed-by: Guillaume Abrioux gabrioux@redhat.com
2 years agoMerge pull request #46618 from aclamk/wip-dout-for-alien
Kefu Chai [Wed, 27 Jul 2022 23:45:39 +0000 (07:45 +0800)]
Merge pull request #46618 from aclamk/wip-dout-for-alien

crimson/alienstore: Add douts

Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agoMerge pull request #47269 from adk3798/fix-repo-test
Adam King [Wed, 27 Jul 2022 23:45:11 +0000 (19:45 -0400)]
Merge pull request #47269 from adk3798/fix-repo-test

qa/workunits/cephadm: update test_repos master -> main

Reviewed-by: Guillaume Abrioux gabrioux@redhat.com
2 years ago.gitmodules: update isa-l_crypto source address 44039/head
Richael Zhuang [Mon, 22 Nov 2021 02:37:48 +0000 (10:37 +0800)]
.gitmodules: update isa-l_crypto source address

Update address to https://github.com/intel/isa-l_crypto for it's
moved under intel repo.

Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>
2 years agoMerge pull request #44499 from SMIL-Infra/fix-missed-lease-ack
Kefu Chai [Wed, 27 Jul 2022 23:40:09 +0000 (07:40 +0800)]
Merge pull request #44499 from SMIL-Infra/fix-missed-lease-ack

osd/PeeringState: fix missed recheck_readable from laggy

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #45135 from lmlg/fix-32-bit
Kefu Chai [Wed, 27 Jul 2022 23:39:00 +0000 (07:39 +0800)]
Merge pull request #45135 from lmlg/fix-32-bit

rgw,test: Fix build errors in 32 bit platforms

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2 years agoMerge pull request #43005 from wxypro/timeout_conn_msg_opt
Kefu Chai [Wed, 27 Jul 2022 23:38:14 +0000 (07:38 +0800)]
Merge pull request #43005 from wxypro/timeout_conn_msg_opt

msg: Add target_addr to log when connections timed out

Reviewed-by: Kefu Chai <kchai@redhat.com>
2 years agoMerge pull request #44040 from richael02/update-isal-crypto-version
Kefu Chai [Wed, 27 Jul 2022 23:34:59 +0000 (07:34 +0800)]
Merge pull request #44040 from richael02/update-isal-crypto-version

crypto: update isa-l_crypto version

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #44762 from Zhiwei-Dai/wip-zlib-compressor-isal-for-aarch64
Kefu Chai [Wed, 27 Jul 2022 23:34:04 +0000 (07:34 +0800)]
Merge pull request #44762 from Zhiwei-Dai/wip-zlib-compressor-isal-for-aarch64

compressor/zlib: isa-l optimization for zlib algorithm on aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoceph-volume: support symlinks as devices 46893/head
Jan Sobczak [Fri, 20 Nov 2020 13:33:13 +0000 (14:33 +0100)]
ceph-volume: support symlinks as devices

This makes ceph-volume support passing symlinks as devices.

Fixes: https://tracker.ceph.com/issues/49103
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoceph-volume: drop self.abspath in Device()
Guillaume Abrioux [Fri, 8 Jul 2022 11:47:00 +0000 (13:47 +0200)]
ceph-volume: drop self.abspath in Device()

seems useless to have both self.path and self.abspath attributes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoMerge pull request #47167 from guits/drop-bs-tool-call
Guillaume Abrioux [Wed, 27 Jul 2022 14:34:08 +0000 (16:34 +0200)]
Merge pull request #47167 from guits/drop-bs-tool-call

ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()

2 years agoMerge pull request #47290 from aisakaki/wip-omap-clear
Yingxin [Wed, 27 Jul 2022 09:54:30 +0000 (17:54 +0800)]
Merge pull request #47290 from aisakaki/wip-omap-clear

crimson/os/seastore/omap: remove unnecessary extent loads when clearing

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2 years agoceph-volume: fix is_ceph_disk_member() 47167/head
Guillaume Abrioux [Tue, 26 Jul 2022 14:24:31 +0000 (16:24 +0200)]
ceph-volume: fix is_ceph_disk_member()

`dev['NAME']` can't match `part` given that it's the name of the
parent device being compared to the partition name.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoosd/scrub: pre-prepare the description text of the current state 46970/head
Ronen Friedman [Tue, 5 Jul 2022 13:42:33 +0000 (13:42 +0000)]
osd/scrub: pre-prepare the description text of the current state

The existing code accesses the states of the scrub FSM via the
standard "state iterator", which is definitely not thread-safe.
It is also a bit time consuming.

As it happened, new code that tried to create the state string
while not holding the PG lock - crashed the OSD.
This PR changes the relevant lines to allow for safe and fast
access to the state description text.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2 years agoMerge pull request #46755 from aravind-wdc/zns
Samuel Just [Wed, 27 Jul 2022 05:08:18 +0000 (22:08 -0700)]
Merge pull request #46755 from aravind-wdc/zns

Crimson: Update zns support

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agocrimson/os/seastore/omap: remove unnecessary extent loads when clearing. 47290/head
Xinyu Huang [Tue, 26 Jul 2022 23:53:41 +0000 (23:53 +0000)]
crimson/os/seastore/omap: remove unnecessary extent loads when clearing.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
2 years agoMerge pull request #47285 from rzarzynski/wip-crimson-monc-discard_result
Kefu Chai [Wed, 27 Jul 2022 00:58:42 +0000 (08:58 +0800)]
Merge pull request #47285 from rzarzynski/wip-crimson-monc-discard_result

crimson/monc: replace then_unpack() with discard_result() if possible

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agorgw,test: Fix build errors in 32 bit platforms 45135/head
Luciano Lo Giudice [Wed, 23 Feb 2022 19:01:58 +0000 (16:01 -0300)]
rgw,test: Fix build errors in 32 bit platforms

When building on 32 bit platforms (tested on armhf), 2 errors were detected:
The first involves the compiler being unable to disambiguate a call to
deserialize an integer (apparently because there isn't a specialization for it).
The second involves invalid usage of the builtin reinterpret_cast.

Signed-off-by: Luciano Lo Giudice <luciano.logiudice@canonical.com>
2 years agoosd: use fmt/ranges.h for printing ranges 47286/head
Kefu Chai [Tue, 26 Jul 2022 16:38:31 +0000 (00:38 +0800)]
osd: use fmt/ranges.h for printing ranges

now that fmt/ranges.h provides a generic mechinary for printing
ranges. let's just use it.

the output format is changed a little bit when it comes to
set and map, but as long as it makes sense, and we don't use
a tool to parse the logging messages for extracting information
from the messages printed by fmt::format(), we should be safe.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #47287 from tchaikov/wip-crimson-cleanup
Samuel Just [Tue, 26 Jul 2022 19:33:00 +0000 (12:33 -0700)]
Merge pull request #47287 from tchaikov/wip-crimson-cleanup

crimson/osd: remove unused variable

Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agocrimson/osd: remove unused variable 47287/head
Kefu Chai [Tue, 26 Jul 2022 16:41:24 +0000 (00:41 +0800)]
crimson/osd: remove unused variable

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agocrimson/monc: replace then_unpack() with discard_result() if possible 47285/head
Radoslaw Zarzynski [Tue, 26 Jul 2022 16:24:12 +0000 (16:24 +0000)]
crimson/monc: replace then_unpack() with discard_result() if possible

More understandable this way.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agoMerge pull request #47279 from rzarzynski/wip-crimson-fix-repopreply-capture
Kefu Chai [Tue, 26 Jul 2022 15:24:50 +0000 (23:24 +0800)]
Merge pull request #47279 from rzarzynski/wip-crimson-fix-repopreply-capture

crimson/osd: don't capture a local, on-stack address of MOSDRepOpReply

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #47277 from rzarzynski/wip-crimson-renew_on_auth_reply
Kefu Chai [Tue, 26 Jul 2022 15:23:44 +0000 (23:23 +0800)]
Merge pull request #47277 from rzarzynski/wip-crimson-renew_on_auth_reply

crimson/monc: renew tickets and rotating keys on MAuthReply

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #47082 from rzarzynski/wip-doc-crimson-cephadm
zdover23 [Tue, 26 Jul 2022 14:45:09 +0000 (00:45 +1000)]
Merge pull request #47082 from rzarzynski/wip-doc-crimson-cephadm

doc/crimson: document installing crimson with cephadm

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2 years agoceph-volume: filter out non-existing devices
Guillaume Abrioux [Mon, 25 Jul 2022 15:19:01 +0000 (17:19 +0200)]
ceph-volume: filter out non-existing devices

[1] has broken teuthology tests.

```
[root@smithi097 /]# ls -l /sys/block/
total 0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jul 25 14:12 nvme0n1 -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1c1n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2c2n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme2/nvme2c2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys2/nvme2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3c3n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme3/nvme3c3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys3/nvme3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4c4n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme4/nvme4c4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys4/nvme4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:12 sda -> ../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda
[root@smithi097 /]#
```

devices like `nvme1c1n1` exist in `/sys/block` but aren't present in `/dev`

```
[root@smithi097 /]# ls -l /dev/nvme*
crw------- 1 root root  10, 122 Jul 25 14:16 /dev/nvme-fabrics
crw------- 1 root root 240,   0 Jul 25 14:12 /dev/nvme0
brw-rw---- 1 root disk 259,   0 Jul 25 14:15 /dev/nvme0n1
crw------- 1 root root 240,   1 Jul 25 14:16 /dev/nvme1
brw-rw---- 1 root disk 259,   2 Jul 25 15:24 /dev/nvme1n1
crw------- 1 root root 240,   2 Jul 25 14:16 /dev/nvme2
brw-rw---- 1 root disk 259,   4 Jul 25 15:24 /dev/nvme2n1
crw------- 1 root root 240,   3 Jul 25 14:16 /dev/nvme3
brw-rw---- 1 root disk 259,   6 Jul 25 15:24 /dev/nvme3n1
crw------- 1 root root 240,   4 Jul 25 14:16 /dev/nvme4
brw-rw---- 1 root disk 259,   8 Jul 25 15:24 /dev/nvme4n1
[root@smithi097 /]#
```

If a device isn't actually present in `/dev`, `get_block_devs_sysfs()` shouldn't return it.

[1] https://github.com/ceph/ceph/commit/c7f017b21ade3762ba5b7b9688bed72c6b6

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoceph-volume: drop `ceph-bluestore-tool` call in Device._parse()
Guillaume Abrioux [Mon, 18 Jul 2022 14:54:17 +0000 (14:54 +0000)]
ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()

disk.has_bluestore_label() does the same thing without a subprocess call.

Fixes: https://tracker.ceph.com/issues/56622
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2 years agoMerge pull request #46945 from rhcs-dashboard/rbd-e2e
Nizamudeen A [Tue, 26 Jul 2022 14:21:06 +0000 (19:51 +0530)]
Merge pull request #46945 from rhcs-dashboard/rbd-e2e

mgr/dashboard: rbd-mirroring e2e tests

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2 years agodoc, crimson: document installing crimson with cephadm 47082/head
Radosław Zarzyński [Wed, 13 Jul 2022 12:33:06 +0000 (14:33 +0200)]
doc, crimson: document installing crimson with cephadm

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/osd: comment on `dispatched` in OSD::ms_dispatch() 47279/head
Radoslaw Zarzynski [Tue, 26 Jul 2022 12:46:01 +0000 (12:46 +0000)]
crimson/osd: comment on `dispatched` in OSD::ms_dispatch()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agoMerge PR #47218 into main
Patrick Donnelly [Tue, 26 Jul 2022 12:59:56 +0000 (08:59 -0400)]
Merge PR #47218 into main

* refs/pull/47218/head:
mon/MDSMonitor: fix check for standby reversion
mds: update comment about standby-replay daemons holding a rank
qa: test for standby-replay removal in MDSMonitor::prepare_beacon
mds: add inject config for synthetic health message

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
2 years agocrimson/osd: move `conn` into the closure in OSD::ms_dispatch()
Radoslaw Zarzynski [Tue, 26 Jul 2022 12:45:07 +0000 (12:45 +0000)]
crimson/osd: move `conn` into the closure in OSD::ms_dispatch()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/osd: don't capture a local, on-stack address of MOSDRepOpReply
Radoslaw Zarzynski [Tue, 26 Jul 2022 12:18:55 +0000 (12:18 +0000)]
crimson/osd: don't capture a local, on-stack address of MOSDRepOpReply

This flaw can explain the following crash observed at Sepia:

```
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-13661-g3a14f283/rpm/el8/BUILD/ceph-17.0.0-13661-g3a14f283/x86_64-redhat-linux-gnu/boost/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = MOSDRepOpReply]: Assertion `px != 0' failed.
Aborting on shard 0.
Backtrace:
 0# gsignal in /lib64/libc.so.6
 1# abort in /lib64/libc.so.6
 2# 0x00007FB1346F4CD9 in /lib64/libc.so.6
 3# 0x00007FB13471A3F6 in /lib64/libc.so.6
 4# crimson::osd::OSD::handle_rep_op_reply(seastar::shared_ptr<crimson::net::Connection>, boost::intrusive_ptr<MOSDRepOpReply>) in ceph-osd
 5# 0x000055C4100FF8D0 in ceph-osd
 6# 0x000055C410100639 in ceph-osd
 7# crimson::osd::OSD::ms_dispatch(seastar::shared_ptr<crimson::net::Connection>, boost::intrusive_ptr<Message>) in ceph-osd
 8# crimson::net::ChainedDispatchers::ms_dispatch(seastar::shared_ptr<crimson::net::Connection>, boost::intrusive_ptr<Message>) in ceph-osd
 9# 0x000055C4165220EF in ceph-osd
10# 0x000055C416532791 in ceph-osd
11# 0x000055C4165328B2 in ceph-osd
12# 0x000055C40FDD33D5 in ceph-osd
13# 0x000055C40FDEE2CE in ceph-osd
14# 0x000055C40FDEEEF0 in ceph-osd
15# 0x000055C41CA41215 in ceph-osd
16# 0x000055C41CA97959 in ceph-osd
17# 0x000055C41CBF73A0 in ceph-osd
18# 0x000055C41CBF93F5 in ceph-osd
19# 0x000055C41C629A46 in ceph-osd
20# 0x000055C41C62E012 in ceph-osd
21# main in ceph-osd
22# __libc_start_main in /lib64/libc.so.6
23# _start in ceph-osd
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/monc: renew_rotating_keyring() doesn't assume the clock is monotonic 47277/head
Radoslaw Zarzynski [Tue, 26 Jul 2022 09:39:08 +0000 (09:39 +0000)]
crimson/monc: renew_rotating_keyring() doesn't assume the clock is monotonic

According to the `seastar::lowres_system_clock` reference:

> This clock has the same granularity as lowres_clock,
> but it is not required to be monotonic and its time
> points correspond to system time.

we should similar check as the classical `MonClient` does:

```cpp
  if ((now > last_rotating_renew_sent) &&
      double(now - last_rotating_renew_sent) < 1) {
    ldout(cct, 10) << __func__ << " called too often (last: "
                   << last_rotating_renew_sent << "), skipping refresh" << dendl;
    return 0;
  }
```

This check is rather paranoidal and the main reason behind it
in crimson is replicating the classical behaviour.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/monc: unify handling auth_service_ticket_ttl with classical OSD
Radoslaw Zarzynski [Tue, 26 Jul 2022 09:31:38 +0000 (09:31 +0000)]
crimson/monc: unify handling auth_service_ticket_ttl with classical OSD

In the classical `MonClient` the `auth_service_ticket_ttl` is lower
bounded to `30` units.

```cpp
  utime_t now = ceph_clock_now();
  utime_t cutoff = now;
  cutoff -= std::min(30.0, cct->_conf->auth_service_ticket_ttl / 4.0);
  utime_t issued_at_lower_bound = now;
  issued_at_lower_bound -= cct->_conf->auth_service_ticket_ttl;
  if (!rotating_secrets->need_new_secrets(cutoff)) {
    ldout(cct, 10) << "_check_auth_rotating have uptodate secrets (they expire after " << cutoff << ")" << dendl;
    rotating_secrets->dump_rotating();
    return 0;
  }
```

The unification affects also the debug mesages.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/monc: improve debugs around Connection::renew_tickets()
Radoslaw Zarzynski [Mon, 25 Jul 2022 14:48:07 +0000 (14:48 +0000)]
crimson/monc: improve debugs around Connection::renew_tickets()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/monc: renew tickets and rotating keys on MAuthReply
Radoslaw Zarzynski [Mon, 25 Jul 2022 14:42:00 +0000 (14:42 +0000)]
crimson/monc: renew tickets and rotating keys on MAuthReply

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2 years agocrimson/zns: manage zone size, capacity and device size 46755/head
Aravind Ramesh [Wed, 13 Jul 2022 09:26:58 +0000 (14:56 +0530)]
crimson/zns: manage zone size, capacity and device size

zone_size and zone_capacity are reported in 512B sectors by the
kernel. Update the superblock correctly with zone size/capacity
in bytes.
Currently we are using stat() to get the device size, but it
fails on block devices to get the size. Use BLKGETSIZE ioctl
to get the actual size of the zns device in sectors and convert
to bytes.

Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
2 years agocmake: add rbd-mirror executable to run-make-check 46945/head
Deepika Upadhyay [Mon, 4 Jul 2022 15:27:29 +0000 (20:57 +0530)]
cmake: add rbd-mirror executable to run-make-check

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit e2b46020317e4d4e37c281a6b1817711e0826a37)

2 years agomgr/dashboard: rbd-mirroring e2e tests
Nizamudeen A [Mon, 4 Jul 2022 07:39:43 +0000 (13:09 +0530)]
mgr/dashboard: rbd-mirroring e2e tests

Signed-off-by: Nizamudeen A <nia@redhat.com>
2 years agoMerge pull request #47168 from guits/get-devices-vgs-per-dev
Guillaume Abrioux [Tue, 26 Jul 2022 07:56:32 +0000 (09:56 +0200)]
Merge pull request #47168 from guits/get-devices-vgs-per-dev

ceph-volume: do not call get_device_vgs() per devices

2 years agoMerge pull request #47239 from cyx1231st/wip-seastore-cleaner-alloc-trim
Yingxin [Tue, 26 Jul 2022 07:18:22 +0000 (15:18 +0800)]
Merge pull request #47239 from cyx1231st/wip-seastore-cleaner-alloc-trim

crimson/os/seastore: cleanups and improvements to trimming

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2 years agoAdd promtail and loki when bootstrap 46955/head
jinhong.kim [Mon, 4 Jul 2022 11:59:46 +0000 (20:59 +0900)]
Add promtail and loki when bootstrap

Signed-off-by: jinhong.kim <jinhong.kim0@navercorp.com>
2 years agoMerge pull request #46554 from adamemerson/wip-up-the-boost
Kefu Chai [Mon, 25 Jul 2022 23:39:10 +0000 (07:39 +0800)]
Merge pull request #46554 from adamemerson/wip-up-the-boost

build: Bump Boost version to 1.79

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2 years agoMerge pull request #47003 from liu-chunmei/seastore-list-objects
Liu-Chunmei [Mon, 25 Jul 2022 23:14:59 +0000 (16:14 -0700)]
Merge pull request #47003 from liu-chunmei/seastore-list-objects

crimson/seastore: fix list objects

Reviewed-by: Samuel Just <sjust@redhat.com>
2 years agoMerge pull request #47217 from tchaikov/wip-ntstatus
Kefu Chai [Mon, 25 Jul 2022 22:56:36 +0000 (06:56 +0800)]
Merge pull request #47217 from tchaikov/wip-ntstatus

common/win32,dokan: include bcrypt.h for NTSTATUS

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2 years agoqa/workunits/cephadm: update test_repos master -> main 47269/head
Adam King [Mon, 25 Jul 2022 20:46:39 +0000 (16:46 -0400)]
qa/workunits/cephadm: update test_repos master -> main

Missed this one, it seems the test was still passing until
recently but I guess the "master" builds are now gone

Signed-off-by: Adam King <adking@redhat.com>
2 years agodoc/cephadm: custom config files documentation 46883/head
Adam King [Tue, 28 Jun 2022 20:52:05 +0000 (16:52 -0400)]
doc/cephadm: custom config files documentation

Signed-off-by: Adam King <adking@redhat.com>
2 years agomgr/cephadm: unit testing for custom config files
Adam King [Tue, 28 Jun 2022 20:30:58 +0000 (16:30 -0400)]
mgr/cephadm: unit testing for custom config files

Signed-off-by: Adam King <adking@redhat.com>
2 years agomgr/cephadm: allow mounting custom conf files
Adam King [Mon, 27 Jun 2022 20:52:14 +0000 (16:52 -0400)]
mgr/cephadm: allow mounting custom conf files

Fixes: https://tracker.ceph.com/issues/56394
Signed-off-by: Adam King <adking@redhat.com>
2 years agosrc/python_common: Add CustomConfig class
Adam King [Mon, 27 Jun 2022 18:26:37 +0000 (14:26 -0400)]
src/python_common: Add CustomConfig class

Signed-off-by: Adam King <adking@redhat.com>
2 years agoMerge pull request #47164 from Dazul/add-config-availability-drive-group
Adam King [Mon, 25 Jul 2022 20:19:33 +0000 (16:19 -0400)]
Merge pull request #47164 from Dazul/add-config-availability-drive-group

mgr/cephadm: add parsing for config on osd specs

Reviewed-by: Adam King <adking@redhat.com>
2 years agoMerge pull request #47120 from yehudasa/wip-56576
Casey Bodley [Mon, 25 Jul 2022 19:49:07 +0000 (15:49 -0400)]
Merge pull request #47120 from yehudasa/wip-56576

rgw: bucket sync: fetch source bucket info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2 years agoMerge pull request #47083 from cbodley/wip-rgw-sysobjctx-rm
Casey Bodley [Mon, 25 Jul 2022 19:03:35 +0000 (15:03 -0400)]
Merge pull request #47083 from cbodley/wip-rgw-sysobjctx-rm

rgw: sysobj service cleanup

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2 years agoMerge pull request #35870 from soumyakoduri/json_fix
Casey Bodley [Mon, 25 Jul 2022 16:17:58 +0000 (12:17 -0400)]
Merge pull request #35870 from soumyakoduri/json_fix

JSONFormattable incorrectly sets a string starting with digit as int

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2 years agoMerge pull request #47255 from tchaikov/wip-fmt-containers
Kefu Chai [Mon, 25 Jul 2022 15:41:03 +0000 (23:41 +0800)]
Merge pull request #47255 from tchaikov/wip-fmt-containers

include/types_fmt: use fmt::join() to format containers

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2 years agocrimson/os/seastore/async_cleaner: fix and improve update_journal_tails() 47239/head
Yingxin Cheng [Mon, 25 Jul 2022 08:28:42 +0000 (16:28 +0800)]
crimson/os/seastore/async_cleaner: fix and improve update_journal_tails()

* Accept JOURNAL_SEQ_NULL dirty_tail when it is pending.
* Enforce monotone increasing to the journal tails.
* Enforce journal tails to be always behind the journal head.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agocrimson/os/seastore/cache: add logs to debug alloc_tail trimming
Yingxin Cheng [Mon, 25 Jul 2022 08:26:09 +0000 (16:26 +0800)]
crimson/os/seastore/cache: add logs to debug alloc_tail trimming

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agoMerge pull request #47256 from anthonyeleven/anthonyeleven/how-to-fix-pg-autoscaler...
Anthony D'Atri [Mon, 25 Jul 2022 14:17:18 +0000 (07:17 -0700)]
Merge pull request #47256 from anthonyeleven/anthonyeleven/how-to-fix-pg-autoscaler-empty-output

doc/radosgw/operations: Document how to fix empty autoscale-status output

2 years agoinclude/types_fmt: use fmt::join() to format containers 47255/head
Kefu Chai [Sun, 24 Jul 2022 16:16:33 +0000 (00:16 +0800)]
include/types_fmt: use fmt::join() to format containers

* add formatter for classes with stream insertion operator.
  as in fmtlib v6.1.2, which is the one shipped with ubuntu focal,
  fmt::join() requires the printed element to have formatter.
  see https://github.com/fmtlib/fmt/issues/2040 and
  https://github.com/fmtlib/fmt/issues/1462
* use template parameter pack to represent container template argument,
  simpler this way. also, this enables us to print specialized
  classes which uses non-default template parameters.
* use fmt::join() to print container elements. see also
  https://fmt.dev/latest/api.html#_CPPv4I0EN3fmt4joinE9join_viewIN6detail10iterator_tI5RangeEEN6detail10sentinel_tI5RangeEEERR5Range11string_view

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agofmt: pickup v8.1.1
Kefu Chai [Mon, 25 Jul 2022 12:42:58 +0000 (20:42 +0800)]
fmt: pickup v8.1.1

update fmt submodule so we can use fmt::join() without defining
formatter specialization for types which already have `operator<<()`.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agocmake: disable FMT_USE_TZSET for WIN32
Kefu Chai [Mon, 25 Jul 2022 13:57:57 +0000 (21:57 +0800)]
cmake: disable FMT_USE_TZSET for WIN32

otherwise we'd have following FTBFS with MinGW, like:

../src/fmt/include/fmt/chrono.h: In lambda function:
../src/fmt/include/fmt/chrono.h:952:5: error: '_tzset' was not declared
in this scope; did you mean 'tzset'?
  952 |     _tzset();
      |     ^~~~~~
      |     tzset

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agomon/MDSMonitor: fix check for standby reversion 47218/head
Patrick Donnelly [Thu, 21 Jul 2022 20:34:05 +0000 (16:34 -0400)]
mon/MDSMonitor: fix check for standby reversion

A standby-replay daemon always has a rank, so this check is completely
wrong. If a beacon from a standby-replay daemon reaches
MDSMonitor::prepare_beacon, it will always be evicted/removed by the
mons. This is rare (usually a reply occurs directly from
MDSMonitor::preprocess_beacon) but can happen in certain circumstances,
like a health warning included in the beacon.

Fixes: https://tracker.ceph.com/issues/56666
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2 years agomds: update comment about standby-replay daemons holding a rank
Patrick Donnelly [Fri, 22 Jul 2022 17:22:01 +0000 (13:22 -0400)]
mds: update comment about standby-replay daemons holding a rank

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2 years agoqa: test for standby-replay removal in MDSMonitor::prepare_beacon
Patrick Donnelly [Thu, 21 Jul 2022 20:28:08 +0000 (16:28 -0400)]
qa: test for standby-replay removal in MDSMonitor::prepare_beacon

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2 years agomds: add inject config for synthetic health message
Patrick Donnelly [Thu, 21 Jul 2022 19:46:58 +0000 (15:46 -0400)]
mds: add inject config for synthetic health message

This will be used to force the MDSMonitor to process a beacon in
MDSMonitor::prepare_beacon.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2 years agocmake: bump up required fmt version
Kefu Chai [Mon, 25 Jul 2022 12:23:14 +0000 (20:23 +0800)]
cmake: bump up required fmt version

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2 years agocrimson/os/seastore/cache: write cleaner dirty tail if the oldest dirty extent is...
Yingxin Cheng [Mon, 25 Jul 2022 08:20:04 +0000 (16:20 +0800)]
crimson/os/seastore/cache: write cleaner dirty tail if the oldest dirty extent is still pending

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agocrimson/os/seastore: drop journal_seq_t NO_DELTAS
Yingxin Cheng [Mon, 25 Jul 2022 05:35:37 +0000 (13:35 +0800)]
crimson/os/seastore: drop journal_seq_t NO_DELTAS

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agocrimson/os/seastore: improve logs
Yingxin Cheng [Mon, 25 Jul 2022 05:22:32 +0000 (13:22 +0800)]
crimson/os/seastore: improve logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2 years agodoc/radosgw/operations: Document how to fix empty autoscale-status output 47256/head
Anthony D'Atri [Mon, 25 Jul 2022 03:14:02 +0000 (20:14 -0700)]
doc/radosgw/operations: Document how to fix empty autoscale-status output

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
2 years agocephadm: Add tests for rescan function 46420/head
Paul Cuzner [Mon, 27 Jun 2022 06:41:27 +0000 (18:41 +1200)]
cephadm: Add tests for rescan function

Adds unittets for the rescan code

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2 years agodocs: Add docs for new host rescan feature
Paul Cuzner [Mon, 30 May 2022 01:55:27 +0000 (13:55 +1200)]
docs: Add docs for new host rescan feature

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2 years agomgr/cephadm: Add new host rescan command
Paul Cuzner [Mon, 30 May 2022 01:54:33 +0000 (13:54 +1200)]
mgr/cephadm: Add new host rescan command

Adds a host rescan command to invoke cephadm's
rescan-disks subcommand

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2 years agoMerge pull request #47014 from ktdreyer/no-system-pmdk-s390x
Ilya Dryomov [Sun, 24 Jul 2022 07:37:55 +0000 (09:37 +0200)]
Merge pull request #47014 from ktdreyer/no-system-pmdk-s390x

ceph.spec.in: disable system_pmdk on s390x

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2 years agoMerge pull request #47242 from idryomov/wip-56676
Ilya Dryomov [Sat, 23 Jul 2022 09:33:47 +0000 (11:33 +0200)]
Merge pull request #47242 from idryomov/wip-56676

librbd: tweak misleading "image is still primary" error message

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2 years agocrimson/test: add more list_objects test in test_seastore 47003/head
chunmei-liu [Sat, 16 Jul 2022 06:09:42 +0000 (23:09 -0700)]
crimson/test: add more list_objects test in test_seastore

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
Signed-off-by: Samuel Just <sjust@redhat.com>
2 years agocrimson/seastore: fix test_seastore
chunmei-liu [Fri, 8 Jul 2022 08:52:09 +0000 (01:52 -0700)]
crimson/seastore: fix test_seastore

collection default has pool(0), and NO_SHARD,
so the ghobject_t should have the same setting.
and ghobject_t default hash not within 1<<(32-4),
so set collection bits to be 0.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2 years agocrimson/seastore: fix list_onodes bug
chunmei-liu [Fri, 15 Jul 2022 05:16:27 +0000 (22:16 -0700)]
crimson/seastore: fix list_onodes bug

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2 years agocrimson/seastore: calculate list_objects ranges by coll_range_(temp)_start/end
chunmei-liu [Wed, 6 Jul 2022 09:10:39 +0000 (02:10 -0700)]
crimson/seastore: calculate list_objects ranges by coll_range_(temp)_start/end

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
Signed-off-by: Samuel Just <sjust@redhat.com>
2 years agoMerge pull request #46711 from 5cs/fix-backoff-retries
J. Eric Ivancich [Fri, 22 Jul 2022 18:39:22 +0000 (14:39 -0400)]
Merge pull request #46711 from 5cs/fix-backoff-retries

rgwlc: fix backoff retries counter

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2 years agoMerge pull request #47005 from 5cs/fix-rm-LCEntry
J. Eric Ivancich [Fri, 22 Jul 2022 18:38:11 +0000 (14:38 -0400)]
Merge pull request #47005 from 5cs/fix-rm-LCEntry

rgwlc: fix removing LCEntry

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>