Dan van der Ster [Thu, 12 Nov 2020 16:14:37 +0000 (17:14 +0100)]
common/options: bluefs_buffered_io=true by default
Enable bluefs_buffered_io again because it makes a huge user-visible
improvement in metadata intensive scenarios, such as but not limited to
PG deletion.
In our environment, deleting PGs from 4 hybrid OSDs (sharing one SATA SSD block.db) saturates
the block.db at 350MB/s reads and causes slow reqs and flapping on the OSDs.
Those OSDs have 3GB osd_target_memory.
Enabling bluefs_buffered_io drops the SSD IO down to <1MBps and the OSDs
are performant again. (The underlying PG deletion inefficiency is being
solved separately, but the page cache is so much more effective than
the bluestore cache in this scenario).
Lastly, remove the comment about swap. We should separately advise
operators to disable swap on OSD machines, as it is much better in
our experience to OOM and restart than to chug along swapping.
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch> Related-to: https://tracker.ceph.com/issues/45765 Related-to: https://tracker.ceph.com/issues/47044
(cherry picked from commit 5ec8e8e63d409860c35e24a192090ac2b70af8f6)
Patrick Donnelly [Wed, 24 Mar 2021 23:11:03 +0000 (16:11 -0700)]
Merge PR #40317 into pacific
* refs/pull/40317/head:
cephsqlite: add julian day offset in milliseconds
doc: add libcephsqlite
ceph.spec,debian: package libcephsqlite
test/libcephsqlite,qa: add tests for libcephsqlite
libcephsqlite: rework architecture and backend
SimpleRADOSStriper: wait for finished aios after write
SimpleRADOSStriper: add new minimal async striper
mon: define simple-rados-client-with-blocklist profile
librados: define must renew lock flag
common: add timeval conversion for durations
Revert "libradosstriper: add function to read into char*"
test_libcephsqlite: test random inserts
cephsqlite: fix compiler errors
cmake: improve build inst for cephsqlite
libcephsqlite: sqlite interface to RADOS
libradosstriper: add function to read into char*
Sage Weil [Wed, 24 Mar 2021 15:34:03 +0000 (10:34 -0500)]
Merge PR #40094 into pacific
* refs/pull/40094/head:
rgw/kms/vault - PendingReleaseNotes pointer
rgw/kms/vault - s3tests for both old and new test logic.
rgw/kms/vault - rework unit test logic for new transit logic.
rgw/kms/vault - 0 terminate before rapidjson
rgw/kms/vault - document configuration for new transit logic
rgw/kms/vault - new transit logic - fix compat logic
rgw/kms/vault - define attribute for new transit logic
rgw/kms/vault - "compat" option
rgw/kms/vault - encryption context - first part
rgw/kms/vault - define attribute to store encryption context
rgw/kms/vault - share get/set attr between rgw_crypt.cc and rgw_kms.cc
rgw/kms/vault - relax configuration parsing for rgw_crypt_vault_secret_engine
rgw/kms/vault - need libicu to make canonical json for encryption contexts.
rgw/kms/kmip - document configuration for a new feature: kmip kms
rgw/kms/kmip - rgw / kmip test integration.
rgw/kms/kmip - rgw / kmip test integration.
rgw/kms/kmip - rgw / kmip test integration.
rgw/kms/kmip - correct documentation.
rgw/kms/kmip - pykmip.py needs to make keys too.
rgw/kms/kmip - pykmip.py should actually run pykmip.
rgw/kms/kmip - python3 changes for testing.
rgw/kms/kmip - string handling cleanup.
teuthology/rgw: pykmip task
kmip: first pass at implementation logic.
kmip: configuration options.
Including cmake build logic inside of libkmip.
cmake glue to build libkmip.
Added libkmip as a submodule.
Yin Congmin [Thu, 18 Mar 2021 13:33:07 +0000 (21:33 +0800)]
librbd/cache/pwl: fix bug of flush request blocked by deferd IO
Flush requests do not need to be queued behind the defer_io queue,
should be issued immediately. Otherwise, there will be a deadlock
scenario in which dirty data is waiting for flush req, flush req is
waiting for defer_io empty, and defer_io is waiting for dirty data
persistence to release space. So this sometimes occur when the cache
is small but the IO is large or the queue depth is large.
Alfonso Martínez [Tue, 23 Mar 2021 10:14:11 +0000 (11:14 +0100)]
mgr/dashboard: fix error notification shown when no rgw daemons are running.
- Adapted code to changes introduced in: https://github.com/ceph/ceph/pull/40220
- Improved error handling.
- Increased test coverage.
- Some refactoring.
- Simplified documentation about setting default daemon host and port.
Fixes: https://tracker.ceph.com/issues/49655 Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 58253a0002f8722abecaaf58161f6494fbe0eaa0)
Kefu Chai [Fri, 12 Mar 2021 04:02:22 +0000 (12:02 +0800)]
ceph.spec: build with system libpmem on fedora and el8
* build with WITH_SYSTEM_PMDK=ON on fedora, as f32 and f33 ship
libpmem1.8 and libpmem1.9 respectively. and we need libpmem v1.7
* build with WITH_SYSTEM_PMDK=ON on el8, as el8 and CentOS8 AppStream
ships libpmem v1.6,
quote from nvml.spec:
> By design, PMDK does not support any 32-bit architecture.
> Due to dependency on some inline assembly, PMDK can be compiled only
> on these architectures:
> - x86_64
> - ppc64le (experimental)
> - aarch64 (unmaintained, supporting hardware doesn't exist?)
so far, only x86_64 and ppc64le packages are built.
see also,
https://src.fedoraproject.org/rpms/nvml/blob/rawhide/f/nvml.spec
Sage Weil [Tue, 23 Mar 2021 00:20:48 +0000 (19:20 -0500)]
Merge PR #40184 into pacific
* refs/pull/40184/head:
qa/suites/rados/cephadm/orchestrator_cli: random-distro$ -> 0-random-distro$
qa/suites/rados/cephadm/smoke-roleless: distro -> 0-distro
qa/distros/podman: install kubic once per host, in parallel
qa/suites/fs/multiclient: use clients: not all: for pexec
Sage Weil [Tue, 23 Mar 2021 00:20:39 +0000 (19:20 -0500)]
Merge PR #40202 into pacific
* refs/pull/40202/head:
qa/suites/rados/cephadm/upgrade: wait for rgw servicemap entries to refresh
mgr/cephadm: identify iscsi service by the pool
qa/distros/podman: install containernetworking-plugins along with podman
python-common: Validate characters in service_id for container names
qa/suites/rados/cephadm/smoke-roleless: deploy additional daemon types
cephadm: fix a minor typo in logging message
qa/suites/rados/cephadm/dashboard: test on centos
cephadm: use debug verbosity during container exec
mgr/cephadm/upgrade: do not repeat crash message
mgr/cephadm/upgrade: a little less verbose
mgr/cephadm: don't log not-ok-to-stop at ERR level
mgr/cephadm: is presumed -> appears
mgr/cephadm: don't double-log ok-to-stop results
mgr/cephadm/upgrade: include upgrade progress in ceph -s
mgr/cephadm: clean up misc messages
mgr/cephadm/configcheck: do not spam info every minute
mgr/cephadm: stop conflicting daemon when deploying to a specific port
mgr/cephadm: make DaemonPlacement print nicer
mgr/cephadm: fix --force remove comment
mgr/cephadm/schedule: choose an IP from a subnet list
mgr/cephadm: rgw: clean up config and config-key values on removal
mgr/cephadm: rgw: drop .crt extension when storing cert in config-key
mgr/cephadm/services: allow beast/civetweb to bind to a particular IP
python-common: add 'networks' property to ServiceSpec
mgr/cephadm/schedule: match placement ip only combination with port
mgr/cephadm: less noise about refreshing hosts
mgr/cephadm: fall back to service spec port if none on DaemonDescription
mgr/cephadm: fix redeploy when daemons have ip:port
mgr/cephadm/schedule: add test case
qa/suites/rados/cephadm/smoke-roleless: add rgw test on many ports
doc/cephadm/rgw: update docs to show count-per-host
mgr/cephadm: add support for rgw_frontend_type (beast or civetweb)
mgr/cephadm: remove ssl_frontend_ssl_key from RGWSpec
mgr/cephadm: fix beast private key config option
mgr/cephadm: fix rgw ssl cert/key config-key path
mgr/cephadm/schedule: dynamically assign ports for rgw
mgr/cephadm/schedule: only 1 port in DaemonPlacement
mgr/cephadm: move rgw frontend logic into RgwService
mgr/cephadm/schedule: return DaemonPlacement instead of HostPlacementSpec
mgr/cephadm/schedule: remove unused methods
mgr/cephadm: propagate ip:port from CephadmDaemoNDeploySpec to deployment
cephadm: populate ports if known and not included in unit.meta
mgr/cephadm: gather and report ports in 'orch ps' output
qa/suites/rados/cephadm/orchestrator_cli: random-distro$ -> 0-random-distro$
qa/suites/rados/cephadm/smoke-roleless: distro -> 0-distro
qa/distros/podman: install kubic once per host, in parallel
qa/suites/fs/multiclient: use clients: not all: for pexec
mgr/cephadm: add info to 'ceph orch upgrade status' in cephadm
Reviewed-by: Michael Fritch <mfritch@suse.com> Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Sage Weil [Tue, 23 Mar 2021 00:20:30 +0000 (19:20 -0500)]
Merge PR #40279 into pacific
* refs/pull/40279/head:
mgr/cephadm: identify rgw, cepfs-mirror in servicemap
mgr/ServiceMap: adjust 'ceph -s' summary
rgw: register daemons in servicemap by gid; include id
cephadm: fix rbd-mirror auth name
Kefu Chai [Mon, 22 Mar 2021 06:49:13 +0000 (14:49 +0800)]
qa/distros/podman: install containernetworking-plugins along with podman
/etc/cni/net.d/87-podman-bridge.conflist tries to load "bridge",
"firewall", "tuning" and "portmap" plugins, which are provided by
containernetworking-plugins package.
Patrick Donnelly [Thu, 28 Jan 2021 23:04:01 +0000 (15:04 -0800)]
SimpleRADOSStriper: add new minimal async striper
This was developed because the two other striper implementations were
unsuitable for libcephsqlite:
- libradosstriper: while the async APIs exist, its current protocol
requires synchronously locking an object for every write/read whether
that operation is async or not. For this reason, it's too far too slow
for latency sensitive applications.
- osdc/Filer: this requires the object name to be an inode number. It
also comes with other overhead burden which is not necessary for
libcephsqlite including caching/buffering.
SimpleRADOSStriper aims to be a minimalistic heavily asynchronous
striper. One way it achieves this is through the use of exclusive locks
to protect access to the striped objects. Most metadata updates are
deferred until the striped file is unlocked, flushed, (or closed). All
reads/writes are asynchronous (but a read implicitly gathers async
striped reads for each op). Writes are not buffered. Reads are not
cached. There is no readahead.
SimpleRADOSStriper aims to be compatible with the rados binary --striper
option for extracting files out of RADOS but it should not be used
otherwise.
Patrick Donnelly [Mon, 22 Feb 2021 03:19:25 +0000 (19:19 -0800)]
librados: define must renew lock flag
This flag already exists in cls_lock but was not made externally
available via librados. Additionally, internally cls_lock refers to the
_RENEW flag as _MAY_RENEW, add an alias for librados to match.
This library provides a SQLite front-end to the RADOS objects.
This effort will help alleviate the restriction on number of key-value pairs
that can be stored in an object.
This interface is a generic one without any constraint on the database
schema either. Library clients can enforce any schema and use SQLite API
to store data in the database backed by RADOS Objects.
Sage Weil [Thu, 18 Mar 2021 16:45:48 +0000 (11:45 -0500)]
mon/MgrStatMonitor: ignore MMgrReport from non-active mgr
If it's not the active mgr, we should ignore it.
Since the mgr instance is best identified by the gid, add that to the
message. (We can't use the source_addrs for the message since that is
the MgrStandby monc addr, not the active mgr addrs in the MgrMap.)
This fixes a problem where a just-demoted mgr report gets processed and a
new mgr gets a ServiceMap with an epoch >= its pending map. (At least,
that is my theory!)
Josh Durgin [Wed, 17 Mar 2021 18:51:27 +0000 (14:51 -0400)]
common/options: turn off bluestore_fsck_quick_fix_on_mount by default
This option enables 3 conversions:
1) pool stats, added in nautilus
2) per-pool omap, added in octopus
3) per-pg omap (replacing (2)) in pacific
Upgrading the long running cluster in sepia from octopus to pacific
resulted in conversion (3). This conversion isn't particularly useful
yet since the follow-on optimization of pg removal aren't in pacific
yet.
This took 25 minutes for the SSD-based osds with <10GB of omap. That's
a lot of disruption, and some clusters have 10x that much omap data.
Upgrades going from nautilus to pacific will miss the finer-grained
stats granularity, but isn't such an important feature it's worth
causing potential availability problems.
In the future we can orchestrate these format changes via cephadm/rook
to minimize the impact on the whole cluster, e.g. going an osd at a
time or doing it during an off-peak period, and not necessarily at the
same time as an upgrade.
Sage Weil [Sun, 21 Mar 2021 18:25:06 +0000 (13:25 -0500)]
Merge PR #40247 into pacific
* refs/pull/40247/head:
common: reset last_log_sent when clog_to_monitors is updated
logclient: move LogChannel::set_log_to_monitors(bool v) to LogClient.cc
Sage Weil [Sun, 21 Mar 2021 14:38:49 +0000 (09:38 -0500)]
Merge PR #40129 into pacific
* refs/pull/40129/head:
osd: PeeringState: implement an acting_set_writeable() function
osd: PeeringState: fix a boolean conditional direction
osd: PeeringState: fix stretch peering so PGs can go peered but not active
osd: PeeringState: don't add acting-set OSDs to candidate set in stretch mode
osd: PeeringState: fix calc_replicated_acting_stretch() syntax/logic
osd: PeeringState: respect stretch peering constraints for async recovery
osd: PeeringState: add a comment about using size as a proxy for activateable
osd: check for is_stretch_pool() in stretch_set_can_peer()
scripts: some additions to help with local testing
script: set_up_stretch_mode: include OSDs in root=default so pg creation works
myoungwon oh [Wed, 24 Feb 2021 14:16:39 +0000 (23:16 +0900)]
osd: recover unreadable snapshots when handling manifest object
The manifest object needs adjacent clones to increment/decrement
refcount when modifying the object. So, recovering the clones is needed
if the adjacent clones are unreadable.
Kefu Chai [Sat, 20 Mar 2021 05:00:01 +0000 (13:00 +0800)]
install-deps.sh: remove existing ceph-libboost of different version
we install different versions of precompiled ceph-libboost packages
for different branches when building and testing them on ubuntu test
nodes. for instance,
- nautilus: v1.72
- octopus, pacific: v1.73
they share the same set of test nodes. and these ceph-libboost packages
conflict with each other, because they install files to the same places.
in order to avoid the confliction, we should uninstall existing packages
before installing a different version of ceph-libboost packages.
ceph-libboost${version}-dev is a package providing the shared headers of
boost library, so, in this change we check if it is installed before
returning or removing the existing packages.
Sage Weil [Fri, 19 Mar 2021 12:21:18 +0000 (08:21 -0400)]
mgr/ServiceMap: adjust 'ceph -s' summary
- Do not list individual daemon ids as this won't scale for larger
clusters
- Do not contemplate multile daemons of the same type that register with
different "daemon_type" -- not until we actually have any that do that.
- Present counts by various groupings: distinct hosts and rgw zones to
start.
services:
mon: 1 daemons, quorum a (age 4m)
mgr: x(active, since 3m)
osd: 1 osds: 1 up (since 3m), 1 in (since 3m)
cephfs-mirror: 1 daemon active (1 hosts)
rbd-mirror: 2 daemons active (1 hosts)
rgw: 2 daemons active (1 hosts, 1 zones)
Sage Weil [Fri, 19 Mar 2021 12:25:23 +0000 (08:25 -0400)]
rgw: register daemons in servicemap by gid; include id
Registering by gid allows multiple radosgw instances to share an auth
key/identity. Including the id in the metadata allows them to still be
identified by name (even if not uniquely).
projects / ceph.git / log