]>
git.apps.os.sepia.ceph.com Git - ceph.git/log
John Mulligan [Fri, 10 Dec 2021 13:19:59 +0000 (08:19 -0500)]
python-common: add test inputs verifying count & count-per-host >= 1
This adds unit new test inputs, local to python-common that verify the
correct error messages are raised when count == 0 and count_per_host ==
0.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 10 Dec 2021 13:16:19 +0000 (08:16 -0500)]
python-common: make count & count-per-host >= 1 checks consistent
The previous version of the validate function had a incorrect error
statement that suggested the count must be >1 when it should have
been >=1. This confusion was possibly due to using "n < 1" on
one line and "n <= 0" on another line. Since both values are supposed
to be integers this change corrects the error message and makes
the comparisons on the lines both use "n < 1" (since I find it easier
to see that the check "n < 1" is the inverse of the error text
asserting "n >= 1").
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 8 Dec 2021 20:37:11 +0000 (15:37 -0500)]
python-common: add unit test func for invalid yaml inputs
I didn't find a preexisting test function for this so I added a
new test that is fed yaml snippets and expected error messages.
This verifies some of the recently added validation for
count and cound_per_host under the placement spec.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 8 Dec 2021 20:33:54 +0000 (15:33 -0500)]
python-common: add int value validation for count and count_per_host
Add additional validation for the count and count_per_host fields
sourced from YAML.
Fixes: https://tracker.ceph.com/issues/50524
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Casey Bodley [Fri, 10 Dec 2021 16:48:39 +0000 (11:48 -0500)]
Merge pull request #44262 from cbodley/wip-53325
test/rgw: disable cls_rgw_gc test cases with defer_gc()
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Matt Benjamin [Fri, 10 Dec 2021 13:46:09 +0000 (08:46 -0500)]
Merge pull request #44206 from linuxbox2/wip-rgwlc-9969
rgwlc: remove magic debug blocks for clearing stale lc entries
Alfonso Martínez [Fri, 10 Dec 2021 07:46:43 +0000 (08:46 +0100)]
Merge pull request #43991 from rhcs-dashboard/introduce-haroxy-metrics-rgw
mgr/dashboard: introduce HAProxy metrics for RGW
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Soumya Koduri [Fri, 10 Dec 2021 05:10:55 +0000 (10:40 +0530)]
Merge pull request #43258 from soumyakoduri/wip-skoduri-dbstore-mp
rgw/dbstore: Multipart upload APIs
Matt Benjamin [Thu, 9 Dec 2021 23:35:03 +0000 (18:35 -0500)]
Merge pull request #42104 from linuxbox2/wip-rgwadminops-fsid
rgw: expose RADOS cluster_fsid via adminops
Matt Benjamin [Thu, 9 Dec 2021 18:24:52 +0000 (13:24 -0500)]
rgw:adminops: remove "import json" from radosgw_admin_rest.py
This is perhaps erring a bit on the side of cosmetic fixes.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Yuri Weinstein [Thu, 9 Dec 2021 16:27:09 +0000 (08:27 -0800)]
Merge pull request #44220 from vshankar/wip-53487
qa: exclude `nofallback` mount option when using v1-style syntax
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Casey Bodley [Thu, 9 Dec 2021 15:58:12 +0000 (10:58 -0500)]
test/rgw: disable cls_rgw_gc test cases with defer_gc()
Fixes: https://tracker.ceph.com/issues/53325
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Matt Benjamin [Tue, 7 Dec 2021 15:53:30 +0000 (10:53 -0500)]
rgw:adminops: slightly generalize /info
Adds a get_name() method to rgw::sal::Store, by which each store
returns its unique name in lowercase.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Thu, 16 Sep 2021 15:11:16 +0000 (11:11 -0400)]
rgw:sal: implement (placeholder) get_cluster_fsid in DBStore
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Thu, 16 Sep 2021 13:57:06 +0000 (09:57 -0400)]
rgw:adminops: add test case for 'info' section
Add 'info' section test case to the radosgw_admin_test.py qa
task.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Tue, 29 Jun 2021 17:54:35 +0000 (13:54 -0400)]
rgw: introduce new /{admin}/info api resource
The new resource returns an array of informational data, which
currently consists of the RADOS cluster fsid as "cluster_fsid."
Fixes: https://tracker.ceph.com/issues/51432
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Tue, 29 Jun 2021 13:23:40 +0000 (09:23 -0400)]
rgw:sal: expose cluster fsid in RGWStore
Adds a get_cluster_id() method to RGWStore, provided librados
cluster_fsid in the RADOS implementation.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Alfonso Martínez [Thu, 9 Dec 2021 15:27:03 +0000 (16:27 +0100)]
Merge pull request #44001 from votdev/issue_53317_fix_proxy_conf
mgr/dashboard: API docs UI does not work with Angular dev server
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Matt Benjamin [Thu, 9 Dec 2021 15:05:16 +0000 (10:05 -0500)]
Merge pull request #44237 from pritha-srivastava/wip-rgw-abort-multipart-bypass-gc
rgw: deleting objects inline in case bypass_gc is specified
Avan Thakkar [Wed, 17 Nov 2021 23:26:12 +0000 (04:56 +0530)]
mgr/dashboard: introduce HAProxy metrics for RGW
Fixes: https://tracker.ceph.com/issues/53311
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Venky Shankar [Mon, 6 Dec 2021 05:17:26 +0000 (00:17 -0500)]
qa: exclude `nofallback` mount option when using v1-style syntax
Otherwise, certain upgrade tests fail which install pacific
or earlier releases since the mount helper does not understand
this mount option, thereby passing it to the kernel which would
does not handle this config causing mount to fail in tests.
Note that this mount config is only used during teuthology tests
to catch v2-style syntax implementation bugs in the kernel.
Fixes: http://tracker.ceph.com/issues/53487
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Samuel Just [Thu, 9 Dec 2021 07:51:30 +0000 (23:51 -0800)]
Merge pull request #44127 from cyx1231st/wip-seastore-merge-journal-headers-2
crimson/os/seastore: make JournalSubmitter aware of header merging
Reviewed-by: Samuel Just <sjust@redhat.com>
Yingxin Cheng [Mon, 29 Nov 2021 02:17:05 +0000 (10:17 +0800)]
crimson/os/seastore/journal: make JournalSubmitter aware of header merging
Introduce an option seastore_journal_batch_preferred_fullness for
JournalSubmitter to control the preferred fullness to trigger batch
flushes.
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Mon, 29 Nov 2021 02:11:09 +0000 (10:11 +0800)]
crimson/os/seastore: refactor get_encoded_length_after()
To return a more informative record_group_size_t.
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Fri, 26 Nov 2021 05:13:56 +0000 (13:13 +0800)]
crimson/os/seastore: debug print journal write details
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Thu, 25 Nov 2021 03:06:05 +0000 (11:06 +0800)]
crimson/os/seastore: measure records write in journal
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Thu, 25 Nov 2021 01:23:53 +0000 (09:23 +0800)]
crimson/os/seastore: fix record metrics in cache
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Thu, 9 Dec 2021 03:09:27 +0000 (11:09 +0800)]
crimson/os/seastore/cache: misc cleanup
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Mon, 22 Nov 2021 08:55:02 +0000 (16:55 +0800)]
crimson/os/seastore: add debug logs to print transaction name
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Mon, 22 Nov 2021 02:55:22 +0000 (10:55 +0800)]
crimson/os/seastore: don't print detail if the extent is not ready
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Mon, 22 Nov 2021 02:51:38 +0000 (10:51 +0800)]
crimson/os/seastore: print extent length
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Mon, 22 Nov 2021 02:50:06 +0000 (10:50 +0800)]
crimson/os/seastore: move implementation to seastore_types.cc
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Fri, 19 Nov 2021 05:17:43 +0000 (13:17 +0800)]
crimson/os/seastore: merge records metadata if they are grouped
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Fri, 19 Nov 2021 05:30:06 +0000 (13:30 +0800)]
crimson/os/seastore: mark out empty transactions
TODO: avoid write if the transaction is empty.
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Fri, 19 Nov 2021 05:17:16 +0000 (13:17 +0800)]
crimson/os/seastore: refactor, introduce record_t and record_group_t with sizes
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Tue, 16 Nov 2021 08:47:12 +0000 (16:47 +0800)]
crimson/os/seastore: scan records based on record_locator_t
Record may not have its own base if headers are merged.
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Tue, 16 Nov 2021 08:15:42 +0000 (16:15 +0800)]
crimson/os/seastore: add more checks when read record_header_t
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Tue, 16 Nov 2021 06:23:15 +0000 (14:23 +0800)]
crimson/os/seastore: misc cleanup and reformat
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Tue, 16 Nov 2021 02:14:05 +0000 (10:14 +0800)]
crimson/os/seastore: add logs in ExtentReader
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yuri Weinstein [Wed, 8 Dec 2021 21:04:57 +0000 (13:04 -0800)]
Merge pull request #43919 from ronen-fr/wip-rf-test-nodeep
osd/scrub (& qa/standalone): test for scrub behavior when no-scrub is set but no-deep-scrub is not
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>
Yuri Weinstein [Wed, 8 Dec 2021 21:04:14 +0000 (13:04 -0800)]
Merge pull request #43305 from heylinn/ceph_rundir_sysvinit
init-ceph: create /var/run/ceph for sysvinit
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Yuri Weinstein [Wed, 8 Dec 2021 21:02:50 +0000 (13:02 -0800)]
Merge pull request #44090 from sseshasa/wip-fix-require-osd-release
osd/OSDMap: Add health warning if 'require-osd-release' != current release
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Samuel Just [Wed, 8 Dec 2021 20:02:45 +0000 (12:02 -0800)]
Merge pull request #44244 from cyx1231st/wip-seastore-refine-metrics
crimson/os/seastore: refine transaction metrics
Reviewed-by: Samuel Just <sjust@redhat.com>
Samuel Just [Wed, 8 Dec 2021 20:02:04 +0000 (12:02 -0800)]
Merge pull request #44242 from liu-chunmei/crimson-fix-heartbeat-addrs
crimson/osd: fix heartbeat front and back blank ip
Reviewed-by: Samuel Just <sjust@redhat.com>
Igor Fedotov [Wed, 8 Dec 2021 19:21:40 +0000 (22:21 +0300)]
Merge pull request #44098 from ifed01/wip-ifed-dump-alloc-unit
os/bluestore: dump bluestore/bluefs alloc unit sizes with perf dump
Reviewed-by: Laura Flores <lflores@redhat.com>
Igor Fedotov [Wed, 8 Dec 2021 19:19:22 +0000 (22:19 +0300)]
Merge pull request #43840 from ifed01/wip-ifed-verbose-open-col
osd,bluestore: gracefully handle a failure during meta collection load
Reviewed-by: jdurgin@redhat.com
Reviewed-by: nojha@redhat.com
Igor Fedotov [Wed, 8 Dec 2021 19:17:38 +0000 (22:17 +0300)]
Merge pull request #39691 from aclamk/wip-bs-compression-blob-64k
os/bluestore: Set new compression blob size to 64K
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
zdover23 [Wed, 8 Dec 2021 19:14:37 +0000 (05:14 +1000)]
Merge pull request #44213 from zdover23/wip-doc-2021-12-05-hardware-recommendations-removing-journal
doc/start: remove journal info from hardware recs
Reviewed-by: Dan van der Ster <daniel.vanderster@cern.ch>
Soumya Koduri [Mon, 22 Nov 2021 16:58:19 +0000 (22:28 +0530)]
rgw/dbstore: Multipart upload APIs
For multipart upload processing, below is the method applied -
MultipartUpload::Init - create head object entry for meta obj (src_obj_name + "." + upload_id)
[ Meta object stores all the parts upload info]
MultipartWriter::process - create all data/tail objects with obj_name same as
meta obj (so that they can all be identified & deleted during abort)
MultipartUpload::Abort - Just delete meta obj .. that will indirectly delete all the uploads
associated with that upload id / meta obj so far.
MultipartUpload::Complete - Create head object of the original object (if not exists).
Rename all data/tail object entries' obj name to orig object name and update metadata of the orig object.
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Daniel Gryniewicz [Wed, 8 Dec 2021 16:34:41 +0000 (11:34 -0500)]
Merge pull request #44232 from Huber-ming/admin_fixlogs
radosgw-admin: fix some error logs
Daniel Gryniewicz [Wed, 8 Dec 2021 16:22:28 +0000 (11:22 -0500)]
Merge pull request #43915 from qiuxinyidian/dev-rgw
rgw: when radosgw-admin stating user, add user exists judging
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Daniel Gryniewicz [Wed, 8 Dec 2021 16:19:12 +0000 (11:19 -0500)]
Merge pull request #43834 from Huber-ming/admin_bilog
radosgw-admin: supplement help documents with 'bilog autotrim'
Zac Dover [Sun, 5 Dec 2021 14:44:09 +0000 (00:44 +1000)]
doc/start: remove journal info from hardware recs
This PR removes mentions of journaling from the hardware
recommendations.
Journaling was a FileStore-related practice. BlueStore is
the default backend for Ceph OSDs and has been since
Luminous. The documentation should reflect that.
Signed-off-by: Zac Dover <zac.dover@gmail.com>
Sebastian Wagner [Wed, 8 Dec 2021 14:51:36 +0000 (15:51 +0100)]
Merge pull request #44093 from melissa-kun-li/ssh-non-root-user
mgr/cephadm: support bootstrap with non-root ssh-user
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Sage Weil [Wed, 8 Dec 2021 12:11:53 +0000 (07:11 -0500)]
Merge PR #44162 into master
* refs/pull/44162/head:
mgr: only queue notify events that modules ask for
pybind/mgr: annotate which events modules consume
pybind/mgr: introduce NotifyType enum
mgr: stop issuing events that no modules consume
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Sage Weil [Wed, 8 Dec 2021 12:10:21 +0000 (07:10 -0500)]
Merge PR #44196 into master
* refs/pull/44196/head:
mon/MgrStatMonitor: do not spam subscribers (mgr) with service_map
Reviewed-by: Neha Ojha <nojha@redhat.com>
Sage Weil [Wed, 8 Dec 2021 12:10:08 +0000 (07:10 -0500)]
Merge PR #44207 into master
* refs/pull/44207/head:
mgr/ActivePyModule: avoid with_gil where possible
mgr/ActivePyModules: push without_gil_t down into blocks
Reviewed-by: Kefu Chai <kchai@redhat.com>
Radoslaw Zarzynski [Wed, 8 Dec 2021 10:24:24 +0000 (11:24 +0100)]
Merge pull request #43542 from rzarzynski/wip-crimson-net-ms_learn_from_peer
crimson/net: add support for ms_learn_addr_from_peer.
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Sebastian Wagner [Wed, 8 Dec 2021 09:25:49 +0000 (10:25 +0100)]
Merge pull request #44104 from guits/guits-cephadm-skip-cv-restorecon
cephadm: pass `CEPH_VOLUME_SKIP_RESTORECON=yes`
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Sébastien Han [Wed, 8 Dec 2021 08:56:49 +0000 (09:56 +0100)]
Merge pull request #44239 from guits/guits-add-skip-needs-root
ceph-volume: make it possible to skip needs_root()
Kefu Chai [Wed, 8 Dec 2021 08:50:20 +0000 (16:50 +0800)]
Merge pull request #44245 from tchaikov/wip-crimson-seastar
seastar: pick up change to fix FTBFS with old cryptopp
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Radoslaw Zarzynski [Thu, 14 Oct 2021 13:55:23 +0000 (13:55 +0000)]
crimson/net: add support for ms_learn_addr_from_peer.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Samuel Just [Wed, 8 Dec 2021 07:43:17 +0000 (23:43 -0800)]
Merge pull request #44179 from athanatos/sjust/wip-pin-race
crimson/os/seastore: initialize logical pins before exposing to cache
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Kefu Chai [Wed, 8 Dec 2021 07:30:10 +0000 (15:30 +0800)]
crimson/osd: s/seastar::fprint()/fmt::print()/
otherwise, we'd have warnings like:
./src/crimson/osd/main.cc:106:16: error: 'fprint<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > &>' is deprecated: use std::format_to() or fmt::print() [-Werror,-Wdeprecated-declarations]
seastar::fprint(std::cerr, "already have key in keyring: %s\n", path);
^
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Kefu Chai [Wed, 8 Dec 2021 06:03:41 +0000 (14:03 +0800)]
seastar: pick up change to fix FTBFS with old cryptopp
Signed-off-by: Kefu Chai <kefu@xsky.com>
Yuval Lifshitz [Wed, 8 Dec 2021 06:05:32 +0000 (08:05 +0200)]
Merge pull request #44186 from qiuxinyidian/rgw-de
rgw: add object null point judging when listing pubsub topics
Yingxin Cheng [Tue, 7 Dec 2021 08:48:03 +0000 (16:48 +0800)]
crimson/os/seastore: measure the number of conflicting transactions by srcs
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Yingxin Cheng [Mon, 6 Dec 2021 08:33:47 +0000 (16:33 +0800)]
crimson/os/seastore: differentiate cleaner trim/reclaim transactions
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
chunmei-liu [Tue, 7 Dec 2021 23:31:24 +0000 (15:31 -0800)]
crimson/osd: fix heartbeat front and back blank ip
when ceph.conf not set public ip & cluster ip, heartbeat will get blank ip address. when osd::_send_boot , classic osd will check if heartbeat front and back addrs are blank ip, if they are blank ip, will use public ip which is learned from mon to set into them. So implement them in crimson osd.
Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
Kefu Chai [Wed, 8 Dec 2021 00:56:11 +0000 (08:56 +0800)]
Merge pull request #44147 from rzarzynski/wip-crimson-new-seastar
crimson: bump up Seastar to recent master and fix FTBFS
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Neha Ojha [Tue, 7 Dec 2021 21:34:15 +0000 (13:34 -0800)]
Merge pull request #44095 from Matan-B/wip-matanb-local-workunits
doc/dev: Running workunits locally
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Yuval Lifshitz [Tue, 7 Dec 2021 19:20:33 +0000 (21:20 +0200)]
Merge pull request #43940 from TRYTOBE8TME/wip-rgw-empty-config
src/rgw: Empty configuration support
Yuval Lifshitz [Tue, 7 Dec 2021 19:19:27 +0000 (21:19 +0200)]
Merge pull request #43665 from zenomri/wip-omri-multipart-trace
rgw/tracer: Multipart upload trace
Samuel Just [Tue, 7 Dec 2021 15:53:21 +0000 (07:53 -0800)]
Merge pull request #44156 from rzarzynski/wip-crimson-fix-process_op-sequencing
crimson/osd: fix sequencing issues in ClientRequest::process_op.
Reviewed-by: Samuel Just <sjust@redhat.com>
Samuel Just [Tue, 7 Dec 2021 15:52:41 +0000 (07:52 -0800)]
Merge pull request #44223 from rzarzynski/wip-crimson-fix-pullinfo-on-push
crimson/osd: don't assume a pull must happen if there is no push.
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Samuel Just [Tue, 7 Dec 2021 15:52:02 +0000 (07:52 -0800)]
Merge pull request #44224 from rzarzynski/wip-crimson-clean-msghs
crimson/osd: clean the recovery message-related header inclusion.
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Samuel Just [Tue, 7 Dec 2021 15:49:08 +0000 (07:49 -0800)]
Merge pull request #44184 from rzarzynski/wip-crimson-internal_client_request-fix-hobj
crimson/osd: fix assertion failure in InternalClientRequest.
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Pritha Srivastava [Tue, 7 Dec 2021 11:01:25 +0000 (16:31 +0530)]
rgw: deleting objects inline in case bypass_gc is specified
for bucket remove command.
fixes: https://tracker.ceph.com/issues/53512
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
Guillaume Abrioux [Tue, 7 Dec 2021 14:18:10 +0000 (15:18 +0100)]
ceph-volume: make it possible to skip needs_root()
Add the possibility to skip the `needs_root()` decorator.
See linked tracker for details.
Fixes: https://tracker.ceph.com/issues/53511
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
Alfonso Martínez [Tue, 7 Dec 2021 14:02:42 +0000 (15:02 +0100)]
Merge pull request #44145 from rhcs-dashboard/fix-frontend-vulnerabilities
mgr/dashboard: fix frontend deps' vulnerabilities
Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Radoslaw Zarzynski [Tue, 30 Nov 2021 19:30:32 +0000 (19:30 +0000)]
crimson/osd: fix sequencing issues in ClientRequest::process_op.
The following crash has been observed in one of the runs at Sepia:
```
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/
17.0.0-8898-ge57ad63c /rpm/el8/BUILD/
ceph-17.0.0-8898-ge57ad63c /src/crimson/osd/osd_operation_sequencer.h:123: void crimson::osd::OpSequencer::finish_op_in_order(crimson::osd::ClientRequest&): Assertion `op.get_id() > last_completed_id' failed.
Aborting on shard 0.
Backtrace:
Reactor stalled for 1807 ms on shard 0. Backtrace: 0xb14ab 0x46e57428 0x46bc450d 0x46be03bd 0x46be0782 0x46be0946 0x46be0bf6 0x12b1f 0x137341 0x3fdd6a92 0x3fddccdb 0x3fdde1ee 0x3fdde8b3 0x3fdd3f2b 0x3fdd4442 0x3fdd4c3a 0x12b1f 0x3737e 0x21db4 0x21c88 0x2fa75 0x3b769527 0x3b8418af 0x3b8423cb 0x3b842ce0 0x3b84383d 0x3a116220 0x3a143f31 0x3a144bcd 0x46b96271 0x46bde51a 0x46d6891b 0x46d6a8f0 0x4681a7d2 0x4681f03b 0x39fd50f2 0x23492 0x39b7a7dd
0# gsignal in /lib64/libc.so.6
1# abort in /lib64/libc.so.6
2# 0x00007FB9FB946C89 in /lib64/libc.so.6
3# 0x00007FB9FB954A76 in /lib64/libc.so.6
4# 0x00005595E98E6528 in ceph-osd
5# 0x00005595E99BE8B0 in ceph-osd
6# 0x00005595E99BF3CC in ceph-osd
7# 0x00005595E99BFCE1 in ceph-osd
8# 0x00005595E99C083E in ceph-osd
9# 0x00005595E8293221 in ceph-osd
10# 0x00005595E82C0F32 in ceph-osd
11# 0x00005595E82C1BCE in ceph-osd
12# 0x00005595F4D13272 in ceph-osd
13# 0x00005595F4D5B51B in ceph-osd
14# 0x00005595F4EE591C in ceph-osd
15# 0x00005595F4EE78F1 in ceph-osd
16# 0x00005595F49977D3 in ceph-osd
17# 0x00005595F499C03C in ceph-osd
18# main in ceph-osd
19# __libc_start_main in /lib64/libc.so.6
20# _start in ceph-osd
```
The sequence of events provides at least two clues:
- the op no. 32 finished before the op no. 29 which was waiting
for `ObjectContext`,
- the op no. 29 was a short-living one -- it wasn't waiting even
on `obc`.
```
rzarzynski@teuthology:/home/teuthworker/archive/rzarzynski-2021-11-22_22:01:32-rados-master-distro-basic-smithi$ less ./
6520106 /remote/smithi115/log/ceph-osd.3.log.gz
...
DEBUG 2021-11-22 22:32:24,531 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4.
f0fb5e1d (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]): start
DEBUG 2021-11-22 22:32:24,531 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4.
f0fb5e1d (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]): in repeat
...
DEBUG 2021-11-22 22:32:24,546 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4.
f0fb5e1d (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]) same_interval_since: 21
DEBUG 2021-11-22 22:32:24,546 [shard 0] osd - OpSequencer::start_op: op=29, last_started=27, last_unblocked=27, last_completed=27
...
DEBUG 2021-11-22 22:32:24,621 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4.
81addbad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]): start
DEBUG 2021-11-22 22:32:24,621 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4.
81addbad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]): in repeat
...
DEBUG 2021-11-22 22:32:24,626 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4.
81addbad (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]) same_interval_s
ince: 21
DEBUG 2021-11-22 22:32:24,626 [shard 0] osd - OpSequencer::start_op: op=32, last_started=29, last_unblocked=29, last_completed=27
<note that op 32 is very short living>
DEBUG 2021-11-22 22:32:24,669 [shard 0] osd - OpSequencer::finish_op_in_order: op=32, last_started=32, last_unblocked=32, last_completed=27
...
DEBUG 2021-11-22 22:32:24,671 [shard 0] osd - client_request(id=32, detail=m=[osd_op(client.4371.0:49 4.d 4:
b5dbb581 :::smithi11538976-13:head {write 601684~619341 in=
619341b , stat} snapc 0={} RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e23) v8]): destroying
...
DEBUG 2021-11-22 22:32:24,722 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4:
b87adf0f :::smithi11538976-9:head {read 0~1} snapc 0={} RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]): got obc lock
...
INFO 2021-11-22 22:32:24,723 [shard 0] osd - client_request(id=29, detail=m=[osd_op(client.4371.0:36 4.d 4:
b87adf0f :::smithi11538976-9:head {read 0~1} snapc 0={} RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e23) v8]) obc.get()=0x6190000d5780
...
DEBUG 2021-11-22 22:32:24,753 [shard 0] osd - OpSequencer::finish_op_in_order: op=29, last_started=32, last_unblocked=32, last_completed=32
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/
17.0.0-8898-ge57ad63c /rpm/el8/BUILD/
ceph-17.0.0-8898-ge57ad63c /src/crimson/osd/osd_operation_sequencer.h:123: void crimson::osd::OpSequencer::finish_op_in_order(crimson::osd::ClientRequest&): Assertion `op.get_id() > last_completed_id' failed.
Aborting on shard 0.
```
This could be explained in a scenario where:
- op no. 29 skipped stages of the execution pipeline while
- it wrongly informed `OpSequencer` the execution was in-order.
Static analysis shows there are multiple problems of this genre
in the `ClientRequest::process_op()` and its callees with the most
recently merged one being the path for `PG::already_complete()`.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Samuel Just [Thu, 2 Dec 2021 21:45:47 +0000 (21:45 +0000)]
crimson/os/seastore/lba_manager: initialize lba node pins using get_extent
Signed-off-by: Samuel Just <sjust@redhat.com>
Samuel Just [Thu, 2 Dec 2021 03:25:15 +0000 (19:25 -0800)]
crimson/os/seastore: initialize logical pins before exposing to cache
Otherwise, another task may get a reference to the extent before
we've set the pin.
Fixes: https://tracker.ceph.com/issues/53267
Signed-off-by: Samuel Just <sjust@redhat.com>
Samuel Just [Tue, 7 Dec 2021 06:21:55 +0000 (22:21 -0800)]
Merge pull request #44231 from xxhdx1985126/wip-cpu-profile
crimson/os/seastore: fix compiler error for gcc > 9 and clang13
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Huber-ming [Tue, 7 Dec 2021 05:59:47 +0000 (13:59 +0800)]
radosgw-admin: fix some error logs
Signed-off-by: Huber-ming <zhangsm01@inspur.com>
Xuehan Xu [Tue, 7 Dec 2021 04:27:14 +0000 (12:27 +0800)]
crimson/os/seastore: fix compiler error for gcc > 9 and clang13
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
David Galloway [Mon, 6 Dec 2021 17:58:22 +0000 (12:58 -0500)]
Merge pull request #44222 from ceph/wip-m2r
doc: Use older mistune
Radoslaw Zarzynski [Mon, 6 Dec 2021 17:29:39 +0000 (17:29 +0000)]
crimson/osd: clean the recovery message-related header inclusion.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Radoslaw Zarzynski [Mon, 6 Dec 2021 16:54:41 +0000 (16:54 +0000)]
crimson/osd: don't assume a pull must happen if there is no push.
In the classical OSD the `ReplicatedRecoveryBackend::recover_object()`
divides into two main flows: pull and push:
```cpp
int ReplicatedBackend::recover_object(
const hobject_t &hoid,
// ...
)
{
dout(10) << __func__ << ": " << hoid << dendl;
RPGHandle *h = static_cast<RPGHandle *>(_h);
if (get_parent()->get_local_missing().is_missing(hoid)) {
ceph_assert(!obc);
// pull
prepare_pull(
v,
hoid,
head,
h);
} else {
ceph_assert(obc);
int started = start_pushes(
hoid,
obc,
h);
// ...
}
return 0;
}
```
Pulls may also enter the push path (`C_ReplicatedBackend_OnPullComplete`)
but push handling doesn't draw any assumption on that. What's important,
`recover_object()` may result in no pulls and pushes.
This isn't the case of crimson as its implementation of the push path
asserts that, if no push is scheduled, `PullInfo` must be allocated.
This patch reworks this logic to reflects the classical one and to avoid
crashes like the following one:
```
DEBUG 2021-12-01 18:43:00,220 [shard 0] osd - recover_object: loaded obc: 3:
4e058a2e :::smithi13839607-45:head
WARN 2021-12-01 18:43:00,220 [shard 0] none - intrusive_ptr_add_ref(p=0x6190000d7f80, use_count=3)
WARN 2021-12-01 18:43:00,220 [shard 0] none - intrusive_ptr_release(p=0x6190000d7f80, use_count=4)
TRACE 2021-12-01 18:43:00,220 [shard 0] osd - call_with_interruption_impl clearing interrupt_cond: 0x60300012b210,N7crimson3osd20IOInterruptConditionE
TRACE 2021-12-01 18:43:00,220 [shard 0] osd - call_with_interruption_impl: may_interrupt: false, local interrupt_condintion: 0x60300012b210, global interrupt_cond: 0x0,N7crimson3osd20IOInterruptConditionE
TRACE 2021-12-01 18:43:00,220 [shard 0] osd - set: interrupt_cond: 0x60300012b210, ref_count: 1
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/
17.0.0-8902-g52fd47fe /rpm/el8/BUILD/ceph-17.0.
0-8902-g52fd47fe /src/crimson/osd/replicated_recovery_backend.cc:84: ReplicatedRecoveryBackend::maybe_push_shards(const hobject_t&, eversion_t)::<lambda()>: Assertion `recovery.pi' failed.
Aborting on shard 0.
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
David Galloway [Mon, 6 Dec 2021 15:32:56 +0000 (10:32 -0500)]
doc: Use older mistune
https://github.com/miyakogi/m2r/issues/66
Signed-off-by: David Galloway <dgallowa@redhat.com>
benhanokh [Sun, 5 Dec 2021 07:47:49 +0000 (09:47 +0200)]
Merge pull request #43870 from benhanokh/restore_alloc_file
NCB::refresh allocation-file after FSCK remove
Gabriel BenHanokh [Mon, 8 Nov 2021 17:12:40 +0000 (19:12 +0200)]
BlueStore: Fix a bug when FSCK is invoked in mount()/umount()/mkfs() with DEEP option
Fixes: https://tracker.ceph.com/issues/53185
NCB mishandles fsck DEEP in mount()/umount()/mkfs() case causing it to remove the allocation-file without destaging a new copy (which will cost us a full rebuild on startup)
There are also few confiliting calls to open_db()/close_db() passing inconsistent read-only flag
We fix both issues by storing open-db type (read-only/read-write) and using it for close-db (which won't pass read-only flag anymore)
We also move allocation-file destage to close-db so it will be refreshed after being removed by fsck and such
Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>
Sage Weil [Sat, 4 Dec 2021 17:52:45 +0000 (12:52 -0500)]
Merge PR #44155 into master
* refs/pull/44155/head:
mgr: limit changes to pg_num
Reviewed-by: Neha Ojha <nojha@redhat.com>
Sage Weil [Sat, 4 Dec 2021 17:52:09 +0000 (12:52 -0500)]
Merge PR #44108 into master
* refs/pull/44108/head:
mgr: fix locking for MetadataUpdate::finish
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Mykola Golub [Sat, 4 Dec 2021 16:25:02 +0000 (18:25 +0200)]
Merge pull request #44204 from songtongshuai/sts_ceph
test/librbd: add get_group test
Reviewed-by: Mykola Golub <mgolub@suse.com>
zdover23 [Sat, 4 Dec 2021 05:53:42 +0000 (15:53 +1000)]
Merge pull request #44189 from zdover23/wip-doc-2021-12-02-documenting-ceph
doc/start: update documenting-ceph.rst (1 of x)
Reviewed-by: Laura Flores <lflores@redhat.com>
Matt Benjamin [Fri, 3 Dec 2021 18:06:10 +0000 (13:06 -0500)]
rgwlc: remove magic debug blocks for clearing stale lc entries
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Sage Weil [Fri, 3 Dec 2021 17:42:57 +0000 (12:42 -0500)]
Merge PR #44017 into master
* refs/pull/44017/head:
mgr/cephadm: Do not propogate access logs from cherrypy
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Matt Benjamin [Fri, 3 Dec 2021 17:22:20 +0000 (12:22 -0500)]
Merge pull request #44139 from linuxbox2/wip-rgw-lcselect
rgwlc: permit lifecycle processing for a single bucket
Sage Weil [Fri, 3 Dec 2021 17:05:13 +0000 (12:05 -0500)]
Merge PR #44132 into master
* refs/pull/44132/head:
mgr/prometheus: define module options for standby
Reviewed-by: Laura Flores <lflores@redhat.com>