]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
3 years agorgw:adminops: add test case for 'info' section
Matt Benjamin [Thu, 16 Sep 2021 13:57:06 +0000 (09:57 -0400)]
rgw:adminops: add test case for 'info' section

Add 'info' section test case to the radosgw_admin_test.py qa
task.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
3 years agorgw: introduce new /{admin}/info api resource
Matt Benjamin [Tue, 29 Jun 2021 17:54:35 +0000 (13:54 -0400)]
rgw: introduce new /{admin}/info api resource

The new resource returns an array of informational data, which
currently consists of the RADOS cluster fsid as "cluster_fsid."

Fixes: https://tracker.ceph.com/issues/51432
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
3 years agorgw:sal: expose cluster fsid in RGWStore
Matt Benjamin [Tue, 29 Jun 2021 13:23:40 +0000 (09:23 -0400)]
rgw:sal: expose cluster fsid in RGWStore

Adds a get_cluster_id() method to RGWStore, provided librados
cluster_fsid in the RADOS implementation.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
3 years agoMerge pull request #44001 from votdev/issue_53317_fix_proxy_conf
Alfonso Martínez [Thu, 9 Dec 2021 15:27:03 +0000 (16:27 +0100)]
Merge pull request #44001 from votdev/issue_53317_fix_proxy_conf

mgr/dashboard: API docs UI does not work with Angular dev server

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
3 years agoMerge pull request #44237 from pritha-srivastava/wip-rgw-abort-multipart-bypass-gc
Matt Benjamin [Thu, 9 Dec 2021 15:05:16 +0000 (10:05 -0500)]
Merge pull request #44237 from pritha-srivastava/wip-rgw-abort-multipart-bypass-gc

rgw: deleting objects inline in case bypass_gc is specified

3 years agoMerge pull request #44127 from cyx1231st/wip-seastore-merge-journal-headers-2
Samuel Just [Thu, 9 Dec 2021 07:51:30 +0000 (23:51 -0800)]
Merge pull request #44127 from cyx1231st/wip-seastore-merge-journal-headers-2

crimson/os/seastore: make JournalSubmitter aware of header merging

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/journal: make JournalSubmitter aware of header merging 44127/head
Yingxin Cheng [Mon, 29 Nov 2021 02:17:05 +0000 (10:17 +0800)]
crimson/os/seastore/journal: make JournalSubmitter aware of header merging

Introduce an option seastore_journal_batch_preferred_fullness for
JournalSubmitter to control the preferred fullness to trigger batch
flushes.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: refactor get_encoded_length_after()
Yingxin Cheng [Mon, 29 Nov 2021 02:11:09 +0000 (10:11 +0800)]
crimson/os/seastore: refactor get_encoded_length_after()

To return a more informative record_group_size_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: debug print journal write details
Yingxin Cheng [Fri, 26 Nov 2021 05:13:56 +0000 (13:13 +0800)]
crimson/os/seastore: debug print journal write details

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: measure records write in journal
Yingxin Cheng [Thu, 25 Nov 2021 03:06:05 +0000 (11:06 +0800)]
crimson/os/seastore: measure records write in journal

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: fix record metrics in cache
Yingxin Cheng [Thu, 25 Nov 2021 01:23:53 +0000 (09:23 +0800)]
crimson/os/seastore: fix record metrics in cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore/cache: misc cleanup
Yingxin Cheng [Thu, 9 Dec 2021 03:09:27 +0000 (11:09 +0800)]
crimson/os/seastore/cache: misc cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: add debug logs to print transaction name
Yingxin Cheng [Mon, 22 Nov 2021 08:55:02 +0000 (16:55 +0800)]
crimson/os/seastore: add debug logs to print transaction name

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: don't print detail if the extent is not ready 44041/head
Yingxin Cheng [Mon, 22 Nov 2021 02:55:22 +0000 (10:55 +0800)]
crimson/os/seastore: don't print detail if the extent is not ready

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: print extent length
Yingxin Cheng [Mon, 22 Nov 2021 02:51:38 +0000 (10:51 +0800)]
crimson/os/seastore: print extent length

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: move implementation to seastore_types.cc
Yingxin Cheng [Mon, 22 Nov 2021 02:50:06 +0000 (10:50 +0800)]
crimson/os/seastore: move implementation to seastore_types.cc

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: merge records metadata if they are grouped
Yingxin Cheng [Fri, 19 Nov 2021 05:17:43 +0000 (13:17 +0800)]
crimson/os/seastore: merge records metadata if they are grouped

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: mark out empty transactions
Yingxin Cheng [Fri, 19 Nov 2021 05:30:06 +0000 (13:30 +0800)]
crimson/os/seastore: mark out empty transactions

TODO: avoid write if the transaction is empty.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: refactor, introduce record_t and record_group_t with sizes
Yingxin Cheng [Fri, 19 Nov 2021 05:17:16 +0000 (13:17 +0800)]
crimson/os/seastore: refactor, introduce record_t and record_group_t with sizes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: scan records based on record_locator_t
Yingxin Cheng [Tue, 16 Nov 2021 08:47:12 +0000 (16:47 +0800)]
crimson/os/seastore: scan records based on record_locator_t

Record may not have its own base if headers are merged.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: add more checks when read record_header_t
Yingxin Cheng [Tue, 16 Nov 2021 08:15:42 +0000 (16:15 +0800)]
crimson/os/seastore: add more checks when read record_header_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: misc cleanup and reformat
Yingxin Cheng [Tue, 16 Nov 2021 06:23:15 +0000 (14:23 +0800)]
crimson/os/seastore: misc cleanup and reformat

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: add logs in ExtentReader
Yingxin Cheng [Tue, 16 Nov 2021 02:14:05 +0000 (10:14 +0800)]
crimson/os/seastore: add logs in ExtentReader

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agoMerge pull request #43919 from ronen-fr/wip-rf-test-nodeep
Yuri Weinstein [Wed, 8 Dec 2021 21:04:57 +0000 (13:04 -0800)]
Merge pull request #43919 from ronen-fr/wip-rf-test-nodeep

osd/scrub (& qa/standalone): test for scrub behavior when no-scrub is set but no-deep-scrub is not

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>
3 years agoMerge pull request #43305 from heylinn/ceph_rundir_sysvinit
Yuri Weinstein [Wed, 8 Dec 2021 21:04:14 +0000 (13:04 -0800)]
Merge pull request #43305 from heylinn/ceph_rundir_sysvinit

init-ceph: create /var/run/ceph for sysvinit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge pull request #44090 from sseshasa/wip-fix-require-osd-release
Yuri Weinstein [Wed, 8 Dec 2021 21:02:50 +0000 (13:02 -0800)]
Merge pull request #44090 from sseshasa/wip-fix-require-osd-release

osd/OSDMap: Add health warning if 'require-osd-release' != current release

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
3 years agoMerge pull request #44244 from cyx1231st/wip-seastore-refine-metrics
Samuel Just [Wed, 8 Dec 2021 20:02:45 +0000 (12:02 -0800)]
Merge pull request #44244 from cyx1231st/wip-seastore-refine-metrics

crimson/os/seastore: refine transaction metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44242 from liu-chunmei/crimson-fix-heartbeat-addrs
Samuel Just [Wed, 8 Dec 2021 20:02:04 +0000 (12:02 -0800)]
Merge pull request #44242 from liu-chunmei/crimson-fix-heartbeat-addrs

crimson/osd: fix heartbeat front and back blank ip

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44098 from ifed01/wip-ifed-dump-alloc-unit
Igor Fedotov [Wed, 8 Dec 2021 19:21:40 +0000 (22:21 +0300)]
Merge pull request #44098 from ifed01/wip-ifed-dump-alloc-unit

os/bluestore: dump bluestore/bluefs alloc unit sizes with perf dump

Reviewed-by: Laura Flores <lflores@redhat.com>
3 years agoMerge pull request #43840 from ifed01/wip-ifed-verbose-open-col
Igor Fedotov [Wed, 8 Dec 2021 19:19:22 +0000 (22:19 +0300)]
Merge pull request #43840 from ifed01/wip-ifed-verbose-open-col

osd,bluestore: gracefully handle a failure during meta collection load

Reviewed-by: jdurgin@redhat.com
Reviewed-by: nojha@redhat.com
3 years agoMerge pull request #39691 from aclamk/wip-bs-compression-blob-64k
Igor Fedotov [Wed, 8 Dec 2021 19:17:38 +0000 (22:17 +0300)]
Merge pull request #39691 from aclamk/wip-bs-compression-blob-64k

os/bluestore: Set new compression blob size to 64K

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
3 years agoMerge pull request #44213 from zdover23/wip-doc-2021-12-05-hardware-recommendations...
zdover23 [Wed, 8 Dec 2021 19:14:37 +0000 (05:14 +1000)]
Merge pull request #44213 from zdover23/wip-doc-2021-12-05-hardware-recommendations-removing-journal

doc/start: remove journal info from hardware recs

Reviewed-by: Dan van der Ster <daniel.vanderster@cern.ch>
3 years agoMerge pull request #44232 from Huber-ming/admin_fixlogs
Daniel Gryniewicz [Wed, 8 Dec 2021 16:34:41 +0000 (11:34 -0500)]
Merge pull request #44232 from Huber-ming/admin_fixlogs

radosgw-admin: fix some error logs

3 years agoMerge pull request #43915 from qiuxinyidian/dev-rgw
Daniel Gryniewicz [Wed, 8 Dec 2021 16:22:28 +0000 (11:22 -0500)]
Merge pull request #43915 from qiuxinyidian/dev-rgw

rgw: when radosgw-admin stating user, add user exists judging

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
3 years agoMerge pull request #43834 from Huber-ming/admin_bilog
Daniel Gryniewicz [Wed, 8 Dec 2021 16:19:12 +0000 (11:19 -0500)]
Merge pull request #43834 from Huber-ming/admin_bilog

radosgw-admin: supplement help documents with 'bilog autotrim'

3 years agodoc/start: remove journal info from hardware recs 44213/head
Zac Dover [Sun, 5 Dec 2021 14:44:09 +0000 (00:44 +1000)]
doc/start: remove journal info from hardware recs

This PR removes mentions of journaling from the hardware
recommendations.

Journaling was a FileStore-related practice. BlueStore is
the default backend for Ceph OSDs and has been since
Luminous. The documentation should reflect that.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
3 years agoMerge pull request #44093 from melissa-kun-li/ssh-non-root-user
Sebastian Wagner [Wed, 8 Dec 2021 14:51:36 +0000 (15:51 +0100)]
Merge pull request #44093 from melissa-kun-li/ssh-non-root-user

mgr/cephadm: support bootstrap with non-root ssh-user

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge PR #44162 into master
Sage Weil [Wed, 8 Dec 2021 12:11:53 +0000 (07:11 -0500)]
Merge PR #44162 into master

* refs/pull/44162/head:
mgr: only queue notify events that modules ask for
pybind/mgr: annotate which events modules consume
pybind/mgr: introduce NotifyType enum
mgr: stop issuing events that no modules consume

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge PR #44196 into master
Sage Weil [Wed, 8 Dec 2021 12:10:21 +0000 (07:10 -0500)]
Merge PR #44196 into master

* refs/pull/44196/head:
mon/MgrStatMonitor: do not spam subscribers (mgr) with service_map

Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge PR #44207 into master
Sage Weil [Wed, 8 Dec 2021 12:10:08 +0000 (07:10 -0500)]
Merge PR #44207 into master

* refs/pull/44207/head:
mgr/ActivePyModule: avoid with_gil where possible
mgr/ActivePyModules: push without_gil_t down into blocks

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #43542 from rzarzynski/wip-crimson-net-ms_learn_from_peer
Radoslaw Zarzynski [Wed, 8 Dec 2021 10:24:24 +0000 (11:24 +0100)]
Merge pull request #43542 from rzarzynski/wip-crimson-net-ms_learn_from_peer

crimson/net: add support for ms_learn_addr_from_peer.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
3 years agoMerge pull request #44104 from guits/guits-cephadm-skip-cv-restorecon
Sebastian Wagner [Wed, 8 Dec 2021 09:25:49 +0000 (10:25 +0100)]
Merge pull request #44104 from guits/guits-cephadm-skip-cv-restorecon

cephadm: pass `CEPH_VOLUME_SKIP_RESTORECON=yes`

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge pull request #44239 from guits/guits-add-skip-needs-root
Sébastien Han [Wed, 8 Dec 2021 08:56:49 +0000 (09:56 +0100)]
Merge pull request #44239 from guits/guits-add-skip-needs-root

ceph-volume: make it possible to skip needs_root()

3 years agoMerge pull request #44245 from tchaikov/wip-crimson-seastar
Kefu Chai [Wed, 8 Dec 2021 08:50:20 +0000 (16:50 +0800)]
Merge pull request #44245 from tchaikov/wip-crimson-seastar

seastar: pick up change to fix FTBFS with old cryptopp

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/net: add support for ms_learn_addr_from_peer. 43542/head
Radoslaw Zarzynski [Thu, 14 Oct 2021 13:55:23 +0000 (13:55 +0000)]
crimson/net: add support for ms_learn_addr_from_peer.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agoMerge pull request #44179 from athanatos/sjust/wip-pin-race
Samuel Just [Wed, 8 Dec 2021 07:43:17 +0000 (23:43 -0800)]
Merge pull request #44179 from athanatos/sjust/wip-pin-race

crimson/os/seastore: initialize logical pins before exposing to cache

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
3 years agocrimson/osd: s/seastar::fprint()/fmt::print()/ 44245/head
Kefu Chai [Wed, 8 Dec 2021 07:30:10 +0000 (15:30 +0800)]
crimson/osd: s/seastar::fprint()/fmt::print()/

otherwise, we'd have warnings like:

./src/crimson/osd/main.cc:106:16: error: 'fprint<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > &>' is deprecated: use std::format_to() or fmt::print() [-Werror,-Wdeprecated-declarations]
      seastar::fprint(std::cerr, "already have key in keyring: %s\n", path);
               ^

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 years agoseastar: pick up change to fix FTBFS with old cryptopp
Kefu Chai [Wed, 8 Dec 2021 06:03:41 +0000 (14:03 +0800)]
seastar: pick up change to fix FTBFS with old cryptopp

Signed-off-by: Kefu Chai <kefu@xsky.com>
3 years agoMerge pull request #44186 from qiuxinyidian/rgw-de
Yuval Lifshitz [Wed, 8 Dec 2021 06:05:32 +0000 (08:05 +0200)]
Merge pull request #44186 from qiuxinyidian/rgw-de

rgw: add object null point judging when listing pubsub  topics

3 years agocrimson/os/seastore: measure the number of conflicting transactions by srcs 44244/head
Yingxin Cheng [Tue, 7 Dec 2021 08:48:03 +0000 (16:48 +0800)]
crimson/os/seastore: measure the number of conflicting transactions by srcs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/os/seastore: differentiate cleaner trim/reclaim transactions
Yingxin Cheng [Mon, 6 Dec 2021 08:33:47 +0000 (16:33 +0800)]
crimson/os/seastore: differentiate cleaner trim/reclaim transactions

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/osd: fix heartbeat front and back blank ip 44242/head
chunmei-liu [Tue, 7 Dec 2021 23:31:24 +0000 (15:31 -0800)]
crimson/osd: fix heartbeat front and back blank ip

when ceph.conf not set public ip & cluster ip, heartbeat will get blank ip address. when osd::_send_boot , classic osd will check if heartbeat front and back addrs are blank ip, if they are blank ip, will use public ip which is learned from mon to set into them. So implement them in crimson osd.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
3 years agoMerge pull request #44147 from rzarzynski/wip-crimson-new-seastar
Kefu Chai [Wed, 8 Dec 2021 00:56:11 +0000 (08:56 +0800)]
Merge pull request #44147 from rzarzynski/wip-crimson-new-seastar

crimson: bump up Seastar to recent master and fix FTBFS

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
3 years agoMerge pull request #44095 from Matan-B/wip-matanb-local-workunits
Neha Ojha [Tue, 7 Dec 2021 21:34:15 +0000 (13:34 -0800)]
Merge pull request #44095 from Matan-B/wip-matanb-local-workunits

doc/dev: Running workunits locally

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
3 years agoMerge pull request #43940 from TRYTOBE8TME/wip-rgw-empty-config
Yuval Lifshitz [Tue, 7 Dec 2021 19:20:33 +0000 (21:20 +0200)]
Merge pull request #43940 from TRYTOBE8TME/wip-rgw-empty-config

src/rgw: Empty configuration support

3 years agoMerge pull request #43665 from zenomri/wip-omri-multipart-trace
Yuval Lifshitz [Tue, 7 Dec 2021 19:19:27 +0000 (21:19 +0200)]
Merge pull request #43665 from zenomri/wip-omri-multipart-trace

rgw/tracer: Multipart upload trace

3 years agoMerge pull request #44156 from rzarzynski/wip-crimson-fix-process_op-sequencing
Samuel Just [Tue, 7 Dec 2021 15:53:21 +0000 (07:53 -0800)]
Merge pull request #44156 from rzarzynski/wip-crimson-fix-process_op-sequencing

crimson/osd: fix sequencing issues in ClientRequest::process_op.

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44223 from rzarzynski/wip-crimson-fix-pullinfo-on-push
Samuel Just [Tue, 7 Dec 2021 15:52:41 +0000 (07:52 -0800)]
Merge pull request #44223 from rzarzynski/wip-crimson-fix-pullinfo-on-push

crimson/osd: don't assume a pull must happen if there is no push.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
3 years agoMerge pull request #44224 from rzarzynski/wip-crimson-clean-msghs
Samuel Just [Tue, 7 Dec 2021 15:52:02 +0000 (07:52 -0800)]
Merge pull request #44224 from rzarzynski/wip-crimson-clean-msghs

crimson/osd: clean the recovery message-related header inclusion.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44184 from rzarzynski/wip-crimson-internal_client_request-fix...
Samuel Just [Tue, 7 Dec 2021 15:49:08 +0000 (07:49 -0800)]
Merge pull request #44184 from rzarzynski/wip-crimson-internal_client_request-fix-hobj

crimson/osd: fix assertion failure in InternalClientRequest.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
3 years agorgw: deleting objects inline in case bypass_gc is specified 44237/head
Pritha Srivastava [Tue, 7 Dec 2021 11:01:25 +0000 (16:31 +0530)]
rgw: deleting objects inline in case bypass_gc is specified
for bucket remove command.

fixes: https://tracker.ceph.com/issues/53512

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
3 years agoceph-volume: make it possible to skip needs_root() 44239/head
Guillaume Abrioux [Tue, 7 Dec 2021 14:18:10 +0000 (15:18 +0100)]
ceph-volume: make it possible to skip needs_root()

Add the possibility to skip the `needs_root()` decorator.
See linked tracker for details.

Fixes: https://tracker.ceph.com/issues/53511
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
3 years agoMerge pull request #44145 from rhcs-dashboard/fix-frontend-vulnerabilities
Alfonso Martínez [Tue, 7 Dec 2021 14:02:42 +0000 (15:02 +0100)]
Merge pull request #44145 from rhcs-dashboard/fix-frontend-vulnerabilities

mgr/dashboard: fix frontend deps' vulnerabilities

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
3 years agocrimson/osd: fix sequencing issues in ClientRequest::process_op. 44156/head
Radoslaw Zarzynski [Tue, 30 Nov 2021 19:30:32 +0000 (19:30 +0000)]
crimson/osd: fix sequencing issues in ClientRequest::process_op.

The following crash has been observed in one of the runs at Sepia:

```
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-8898-ge57ad63c/rpm/el8/BUILD/ceph-17.0.0-8898-ge57ad63c/src/crimson/osd/osd_operation_sequencer.h:123: void crimson::osd::OpSequencer::finish_op_in_order(crimson::osd::ClientRequest&): Assertion `op.get_id() > last_completed_id' failed.
Aborting on shard 0.
Backtrace:
Reactor stalled for 1807 ms on shard 0. Backtrace: 0xb14ab 0x46e57428 0x46bc450d 0x46be03bd 0x46be0782 0x46be0946 0x46be0bf6 0x12b1f 0x137341 0x3fdd6a92 0x3fddccdb 0x3fdde1ee 0x3fdde8b3 0x3fdd3f2b 0x3fdd4442 0x3fdd4c3a 0x12b1f 0x3737e 0x21db4 0x21c88 0x2fa75 0x3b769527 0x3b8418af 0x3b8423cb 0x3b842ce0 0x3b84383d 0x3a116220 0x3a143f31 0x3a144bcd 0x46b96271 0x46bde51a 0x46d6891b 0x46d6a8f0 0x4681a7d2 0x4681f03b 0x39fd50f2 0x23492 0x39b7a7dd
 0# gsignal in /lib64/libc.so.6
 1# abort in /lib64/libc.so.6
 2# 0x00007FB9FB946C89 in /lib64/libc.so.6
 3# 0x00007FB9FB954A76 in /lib64/libc.so.6
 4# 0x00005595E98E6528 in ceph-osd
 5# 0x00005595E99BE8B0 in ceph-osd
 6# 0x00005595E99BF3CC in ceph-osd
 7# 0x00005595E99BFCE1 in ceph-osd
 8# 0x00005595E99C083E in ceph-osd
 9# 0x00005595E8293221 in ceph-osd
10# 0x00005595E82C0F32 in ceph-osd
11# 0x00005595E82C1BCE in ceph-osd
12# 0x00005595F4D13272 in ceph-osd
13# 0x00005595F4D5B51B in ceph-osd
14# 0x00005595F4EE591C in ceph-osd
15# 0x00005595F4EE78F1 in ceph-osd
16# 0x00005595F49977D3 in ceph-osd
17# 0x00005595F499C03C in ceph-osd
18# main in ceph-osd
19# __libc_start_main in /lib64/libc.so.6
20# _start in ceph-osd
```

The sequence of events provides at least two clues:
  - the op no. 32 finished before the op no. 29 which was waiting
    for `ObjectContext`,
  - the op no. 29 was a short-living one -- it wasn't waiting even
    on `obc`.

```
rzarzynski@teuthology:/home/teuthworker/archive/rzarzynski-2021-11-22_22:01:32-rados-master-distro-basic-smithi$ less ./6520106/remote/smithi115/log/ceph-osd.3.log.gz
...
DEBUG 2021-11-22 22:32:24,531 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4.f0fb5e1d (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]): start
DEBUG 2021-11-22 22:32:24,531 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4.f0fb5e1d (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]): in repeat
...
DEBUG 2021-11-22 22:32:24,546 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4.f0fb5e1d (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]) same_interval_since: 21
DEBUG 2021-11-22 22:32:24,546 [shard 0] osd - OpSequencer::start_op: op=29, last_started=27, last_unblocked=27, last_completed=27
...
DEBUG 2021-11-22 22:32:24,621 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4.81addbad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]): start
DEBUG 2021-11-22 22:32:24,621 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4.81addbad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]): in repeat
...
DEBUG 2021-11-22 22:32:24,626 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4.81addbad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]) same_interval_s
ince: 21
DEBUG 2021-11-22 22:32:24,626 [shard 0] osd - OpSequencer::start_op: op=32, last_started=29, last_unblocked=29, last_completed=27
<note that op 32 is very short living>
DEBUG 2021-11-22 22:32:24,669 [shard 0] osd - OpSequencer::finish_op_in_order: op=32, last_started=32, last_unblocked=32, last_completed=27
...
DEBUG 2021-11-22 22:32:24,671 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4:b5dbb581:::smithi11538976-13:head {write 601684~619341 in=619341b, stat} snapc 0={} RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]): destroying
...
DEBUG 2021-11-22 22:32:24,722 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4:b87adf0f:::smithi11538976-9:head {read 0~1} snapc 0={} RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]): got obc lock
...
INFO  2021-11-22 22:32:24,723 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4:b87adf0f:::smithi11538976-9:head {read 0~1} snapc 0={} RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]) obc.get()=0x6190000d5780
...
DEBUG 2021-11-22 22:32:24,753 [shard 0] osd - OpSequencer::finish_op_in_order: op=29, last_started=32, last_unblocked=32, last_completed=32
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-8898-ge57ad63c/rpm/el8/BUILD/ceph-17.0.0-8898-ge57ad63c/src/crimson/osd/osd_operation_sequencer.h:123: void crimson::osd::OpSequencer::finish_op_in_order(crimson::osd::ClientRequest&): Assertion `op.get_id() > last_completed_id' failed.
Aborting on shard 0.
```

This could be explained in a scenario where:
  - op no. 29 skipped stages of the execution pipeline while
  - it wrongly informed `OpSequencer` the execution was in-order.

Static analysis shows there are multiple problems of this genre
in the `ClientRequest::process_op()` and its callees with the most
recently merged one being the path for `PG::already_complete()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/os/seastore/lba_manager: initialize lba node pins using get_extent 44179/head
Samuel Just [Thu, 2 Dec 2021 21:45:47 +0000 (21:45 +0000)]
crimson/os/seastore/lba_manager: initialize lba node pins using get_extent

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore: initialize logical pins before exposing to cache
Samuel Just [Thu, 2 Dec 2021 03:25:15 +0000 (19:25 -0800)]
crimson/os/seastore: initialize logical pins before exposing to cache

Otherwise, another task may get a reference to the extent before
we've set the pin.

Fixes: https://tracker.ceph.com/issues/53267
Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #44231 from xxhdx1985126/wip-cpu-profile
Samuel Just [Tue, 7 Dec 2021 06:21:55 +0000 (22:21 -0800)]
Merge pull request #44231 from xxhdx1985126/wip-cpu-profile

crimson/os/seastore: fix compiler error for gcc > 9 and clang13

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoradosgw-admin: fix some error logs 44232/head
Huber-ming [Tue, 7 Dec 2021 05:59:47 +0000 (13:59 +0800)]
radosgw-admin: fix some error logs

Signed-off-by: Huber-ming <zhangsm01@inspur.com>
3 years agocrimson/os/seastore: fix compiler error for gcc > 9 and clang13 44231/head
Xuehan Xu [Tue, 7 Dec 2021 04:27:14 +0000 (12:27 +0800)]
crimson/os/seastore: fix compiler error for gcc > 9 and clang13

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
3 years agoMerge pull request #44222 from ceph/wip-m2r
David Galloway [Mon, 6 Dec 2021 17:58:22 +0000 (12:58 -0500)]
Merge pull request #44222 from ceph/wip-m2r

doc: Use older mistune

3 years agocrimson/osd: clean the recovery message-related header inclusion. 44224/head
Radoslaw Zarzynski [Mon, 6 Dec 2021 17:29:39 +0000 (17:29 +0000)]
crimson/osd: clean the recovery message-related header inclusion.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/osd: don't assume a pull must happen if there is no push. 44223/head
Radoslaw Zarzynski [Mon, 6 Dec 2021 16:54:41 +0000 (16:54 +0000)]
crimson/osd: don't assume a pull must happen if there is no push.

In the classical OSD the `ReplicatedRecoveryBackend::recover_object()`
divides into two main flows: pull and push:

```cpp
int ReplicatedBackend::recover_object(
  const hobject_t &hoid,
  // ...
  )
{
  dout(10) << __func__ << ": " << hoid << dendl;
  RPGHandle *h = static_cast<RPGHandle *>(_h);
  if (get_parent()->get_local_missing().is_missing(hoid)) {
    ceph_assert(!obc);
    // pull
    prepare_pull(
      v,
      hoid,
      head,
      h);
  } else {
    ceph_assert(obc);
    int started = start_pushes(
      hoid,
      obc,
      h);
    // ...
  }
  return 0;
}
```

Pulls may also enter the push path (`C_ReplicatedBackend_OnPullComplete`)
but push handling doesn't draw any assumption on that. What's important,
`recover_object()` may result in no pulls and pushes.

This isn't the case of crimson as its implementation of the push path
asserts that, if no push is scheduled, `PullInfo` must be allocated.
This patch reworks this logic to reflects the classical one and to avoid
crashes like the following one:

```
DEBUG 2021-12-01 18:43:00,220 [shard 0] osd - recover_object: loaded obc: 3:4e058a2e:::smithi13839607-45:head
WARN  2021-12-01 18:43:00,220 [shard 0] none - intrusive_ptr_add_ref(p=0x6190000d7f80, use_count=3)
WARN  2021-12-01 18:43:00,220 [shard 0] none - intrusive_ptr_release(p=0x6190000d7f80, use_count=4)
TRACE 2021-12-01 18:43:00,220 [shard 0] osd - call_with_interruption_impl clearing interrupt_cond: 0x60300012b210,N7crimson3osd20IOInterruptConditionE
TRACE 2021-12-01 18:43:00,220 [shard 0] osd - call_with_interruption_impl: may_interrupt: false, local interrupt_condintion: 0x60300012b210, global interrupt_cond: 0x0,N7crimson3osd20IOInterruptConditionE
TRACE 2021-12-01 18:43:00,220 [shard 0] osd - set: interrupt_cond: 0x60300012b210, ref_count: 1
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-8902-g52fd47fe/rpm/el8/BUILD/ceph-17.0.
0-8902-g52fd47fe/src/crimson/osd/replicated_recovery_backend.cc:84: ReplicatedRecoveryBackend::maybe_push_shards(const hobject_t&, eversion_t)::<lambda()>: Assertion `recovery.pi' failed.
Aborting on shard 0.
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agodoc: Use older mistune 44222/head
David Galloway [Mon, 6 Dec 2021 15:32:56 +0000 (10:32 -0500)]
doc: Use older mistune

https://github.com/miyakogi/m2r/issues/66

Signed-off-by: David Galloway <dgallowa@redhat.com>
3 years agoMerge pull request #43870 from benhanokh/restore_alloc_file
benhanokh [Sun, 5 Dec 2021 07:47:49 +0000 (09:47 +0200)]
Merge pull request #43870 from benhanokh/restore_alloc_file

NCB::refresh allocation-file after FSCK remove

3 years agoBlueStore: Fix a bug when FSCK is invoked in mount()/umount()/mkfs() with DEEP option 43870/head
Gabriel BenHanokh [Mon, 8 Nov 2021 17:12:40 +0000 (19:12 +0200)]
BlueStore: Fix a bug when FSCK is invoked in mount()/umount()/mkfs() with DEEP option
Fixes: https://tracker.ceph.com/issues/53185
NCB mishandles fsck DEEP in mount()/umount()/mkfs() case causing it to remove the allocation-file without destaging a new copy (which will cost us a full rebuild on startup)
There are also few confiliting calls to open_db()/close_db() passing inconsistent read-only flag

We fix both issues by storing open-db type (read-only/read-write) and using it for close-db (which won't pass read-only flag anymore)
We also move allocation-file destage to close-db so it will be refreshed after being removed by fsck and such

Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>
3 years agoMerge PR #44155 into master
Sage Weil [Sat, 4 Dec 2021 17:52:45 +0000 (12:52 -0500)]
Merge PR #44155 into master

* refs/pull/44155/head:
mgr: limit changes to pg_num

Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge PR #44108 into master
Sage Weil [Sat, 4 Dec 2021 17:52:09 +0000 (12:52 -0500)]
Merge PR #44108 into master

* refs/pull/44108/head:
mgr: fix locking for MetadataUpdate::finish

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
3 years agoMerge pull request #44204 from songtongshuai/sts_ceph
Mykola Golub [Sat, 4 Dec 2021 16:25:02 +0000 (18:25 +0200)]
Merge pull request #44204 from songtongshuai/sts_ceph

test/librbd: add get_group test

Reviewed-by: Mykola Golub <mgolub@suse.com>
3 years agoMerge pull request #44189 from zdover23/wip-doc-2021-12-02-documenting-ceph
zdover23 [Sat, 4 Dec 2021 05:53:42 +0000 (15:53 +1000)]
Merge pull request #44189 from zdover23/wip-doc-2021-12-02-documenting-ceph

doc/start: update documenting-ceph.rst (1 of x)

Reviewed-by: Laura Flores <lflores@redhat.com>
3 years agoMerge PR #44017 into master
Sage Weil [Fri, 3 Dec 2021 17:42:57 +0000 (12:42 -0500)]
Merge PR #44017 into master

* refs/pull/44017/head:
mgr/cephadm: Do not propogate access logs from cherrypy

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge pull request #44139 from linuxbox2/wip-rgw-lcselect
Matt Benjamin [Fri, 3 Dec 2021 17:22:20 +0000 (12:22 -0500)]
Merge pull request #44139 from linuxbox2/wip-rgw-lcselect

rgwlc: permit lifecycle processing for a single bucket

3 years agoMerge PR #44132 into master
Sage Weil [Fri, 3 Dec 2021 17:05:13 +0000 (12:05 -0500)]
Merge PR #44132 into master

* refs/pull/44132/head:
mgr/prometheus: define module options for standby

Reviewed-by: Laura Flores <lflores@redhat.com>
3 years agotest/librbd: add get_group test 44204/head
songtongshuai_yewu [Fri, 3 Dec 2021 14:29:03 +0000 (09:29 -0500)]
test/librbd: add get_group test

Signed-off-by: songtongshuai_yewu <songtongshuai_yewu@cmss.chinamobile.com>
3 years agodoc/start: update documenting-ceph.rst (1 of x) 44189/head
Zac Dover [Thu, 2 Dec 2021 14:15:44 +0000 (00:15 +1000)]
doc/start: update documenting-ceph.rst (1 of x)

This PR updates the content on documenting-ceph,
which is, as of December 2021, in need of an
update.

This is the first of what I estimate to be three
to five PRs against this .rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
3 years agoPendingReleaseNotes: Release note 'require-osd-release' health warning 44090/head
Sridhar Seshasayee [Fri, 3 Dec 2021 09:55:32 +0000 (15:25 +0530)]
PendingReleaseNotes: Release note 'require-osd-release' health warning

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoosd/OSDMap: Add health warning if 'require-osd-release' != current release
Sridhar Seshasayee [Mon, 22 Nov 2021 15:16:02 +0000 (20:46 +0530)]
osd/OSDMap: Add health warning if 'require-osd-release' != current release

After all OSDs are upgraded to a new release, generate a health warning if
the 'require-osd-release' flag doesn't match the the new release version.
This will result in the cluster showing a warning in the health state until
the flag is set properly.

Fixes: https://tracker.ceph.com/issues/51984
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
3 years agoMerge pull request #44103 from majianpeng/librbd-pwl-flush-by-blockgurad
Deepika Upadhyay [Fri, 3 Dec 2021 09:35:12 +0000 (15:05 +0530)]
Merge pull request #44103 from majianpeng/librbd-pwl-flush-by-blockgurad

librbd/cache/pwl: Using BlockGuard control overlap ops order when flu…

Reviewed-by: Mykola Golub mgolub@suse.com
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
3 years agoMerge pull request #44144 from majianpeng/librbd-fix-discard-granularity
Deepika Upadhyay [Fri, 3 Dec 2021 06:21:08 +0000 (11:51 +0530)]
Merge pull request #44144 from majianpeng/librbd-fix-discard-granularity

librbd: fix discard granularity for pwl cache

Reviewed-by: mgolub@suse.com
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
3 years agomgr: only queue notify events that modules ask for 44162/head
Sage Weil [Thu, 2 Dec 2021 15:23:45 +0000 (10:23 -0500)]
mgr: only queue notify events that modules ask for

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agopybind/mgr: annotate which events modules consume
Sage Weil [Thu, 2 Dec 2021 15:23:22 +0000 (10:23 -0500)]
pybind/mgr: annotate which events modules consume

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agopybind/mgr: introduce NotifyType enum
Sage Weil [Thu, 2 Dec 2021 15:22:48 +0000 (10:22 -0500)]
pybind/mgr: introduce NotifyType enum

Note that we don't annotate the dashboard NotificationQueue because it is
used internally by the dashboard with other events.

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoMerge pull request #43848 from dvanders/doc_bench
zdover23 [Thu, 2 Dec 2021 23:37:51 +0000 (09:37 +1000)]
Merge pull request #43848 from dvanders/doc_bench

doc: add disk benchmarking and cache tuning recommendations

Reviewed-by: Zac Dover <zac.dover@gmail.com>
3 years agoMerge pull request #43691 from curtbruns/use_optimal_for_min_alloc
Yuri Weinstein [Thu, 2 Dec 2021 23:28:17 +0000 (15:28 -0800)]
Merge pull request #43691 from curtbruns/use_optimal_for_min_alloc

os/bluestore: Set min_alloc_size to optimal io size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
3 years agoMerge pull request #43578 from ronen-fr/wip-rf-log-options
Yuri Weinstein [Thu, 2 Dec 2021 23:27:18 +0000 (15:27 -0800)]
Merge pull request #43578 from ronen-fr/wip-rf-log-options

common: hide internal logger configuration strings from clients

Reviewed-by: Laura Flores <lflores@redhat.com>
3 years agoMerge pull request #43493 from myoungwon/wip-52872
Yuri Weinstein [Thu, 2 Dec 2021 23:26:24 +0000 (15:26 -0800)]
Merge pull request #43493 from myoungwon/wip-52872

test: increase retry duration when calculating manifest ref. count

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43944 from aclamk/wip-aclamk-fix-crush-location-hook-50659
Yuri Weinstein [Thu, 2 Dec 2021 23:25:10 +0000 (15:25 -0800)]
Merge pull request #43944 from aclamk/wip-aclamk-fix-crush-location-hook-50659

crush: Fix segfault in update_from_hook

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
3 years agomon/MgrStatMonitor: do not spam subscribers (mgr) with service_map 44196/head
Sage Weil [Thu, 2 Dec 2021 22:46:26 +0000 (17:46 -0500)]
mon/MgrStatMonitor: do not spam subscribers (mgr) with service_map

We are comparing the monmap epoch to the service_map epoch!

Fixes: https://tracker.ceph.com/issues/53479
Signed-off-by: Sage Weil <sage@newdream.net>
3 years agoos/bluestore: report amount of loaded collections. 43840/head
Igor Fedotov [Mon, 8 Nov 2021 12:18:11 +0000 (15:18 +0300)]
os/bluestore: report amount of loaded collections.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
3 years agoosd: report error and fail if store returns no meta collection
Igor Fedotov [Mon, 8 Nov 2021 12:14:46 +0000 (15:14 +0300)]
osd: report error and fail if store returns no meta collection

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
3 years agoMerge pull request #44194 from ceph/wip-yuriw-crontab-master
Yuri Weinstein [Thu, 2 Dec 2021 20:23:28 +0000 (12:23 -0800)]
Merge pull request #44194 from ceph/wip-yuriw-crontab-master

qa/tests: switch all gibba machines to smithi

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>