Marcus Watts [Mon, 5 Mar 2018 21:10:29 +0000 (16:10 -0500)]
rgw: fix memory fragmentation problem reading data from client.
mg_read returns 0 on EOF. In some versions of civetweb, mg_read can
return "short" reads. The logic in the rest of ceph depends on a read
always filling its buffer when possible. So loop here and fill the
buffer. Looping to fill the buffer means we must also track when
mg_read returns EOF, because after that further calls to mg_read will return -1.
Fixes: https://tracker.ceph.com/issues/23207 Signed-off-by: Marcus Watts <mwatts@redhat.com>
(cherry picked from commit d9a150b1f1f9ec3952555186722c9b13d7448e2d)
Conflicts:
src/rgw/rgw_civetweb.cc:
- master RGWCivetWeb::RGWCivetWeb() becomes RGWMongoose::RGWMongoose() in jewel,
with slightly different initializer list; adapt mg_read() call for jewel
- master RGWCivetWeb::read_data() becomes RGWMongoose::read_data() in jewel,
and inside this function we return -EIO in jewel, instead of throwing an exception
rocksdb's makefile tries to detect jemalloc automatically and compiles
with it enabled if it's found, but if ceph disable jemalloc, it won't
link against jemalloc. this causes link failure if we compile ceph
without jemalloc enabled in a host with jemalloc library installed.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts:
src/kv/Makefile.am: this change is not cherry-picked from
master. as master does not use autotools anymore. and master is
using rocksdb's cmake script for building, the latter does not
jemalloc by default.
Conflicts:
src/common/options.cc - file does not exist in jewel; manually
backported the change to src/common/config_opts.h
src/common/shared_cache.hpp - use jewel programming convention
for accessing config option values
--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.
Conflicts:
src/ceph-detect-init/CMakeLists.txt
src/ceph-disk/CMakeLists.txt: trivial resolution
src/pybind/mgr/dashboard/CMakeLists.txt: dashboard2 is not
in luminous, so drop this change.
src/tools/setup-virtualenv.sh:
src/ceph-detect-init/Makefile.am:
src/ceph-disk/Makefile.am: these files were dropped in
master, so update them also.
Conflicts:
src/mds/Locker.cc - fbda12b1eefeebae1e1c64b01bc8f222c0d4c13d has not
been backported to jewel, so session->check_access() call has fewer arguments
than in master
src/mds/Server.cc - 3fe5e734be822492446a403ab176965d2be737fc has not
been backported to jewel, so there is no "bool blacklisted = false;" assignment
statement
Yan, Zheng [Fri, 21 Oct 2016 03:38:44 +0000 (11:38 +0800)]
mds: fix CDir::log_mark_dirty()
CDir::log_mark_dirty() moves dirfrag to current log segment's dirty
dirfrag list, but it does not submit any log event. Old log segments
(that include events which dirty the dirfrag) may get expired before
the dirfrag gets committed. If MDS crashes, the changes in expired
log segments get lost.
ceph: do link/rename semantic checks after srcdn is readable
For hard link, source inode must not be directory. For rename,
types of source/destination inodes must match. If srcdn is replica
and we do these checks while it's not readble, it's possible that
wrong source inode is used in these checks.
Conflicts:
src/rgw/rgw_keystone.cc - functions shifted and automatic merge failed, corrected by manual merge
src/rgw/rgw_keystone.h - trivial merge conflicts, class elements unrelated to this pr are different in jewel
Casey Bodley [Fri, 2 Mar 2018 19:22:30 +0000 (14:22 -0500)]
rgw: s3website error handler uses original object name
the s3website error handler needs to use the same object name for
redirect handling that retarget() does, but s->object.name may be
modified based on get_effective_key()
If you directly call the operator new function, you must also directly
call the operator delete function, and must manually call the
destructor as well.
Matt Benjamin [Tue, 27 Mar 2018 17:49:53 +0000 (13:49 -0400)]
radosgw: conditionally link with tcmalloc
The radosgw program should be linked with tcmalloc (or
tcmalloc_minimal) when selected by the build (i.e, whenever other
Ceph daemons such as ceph-osd would be.
Just adding $(LIBTCMALLOC) to the linkage directive is sufficient
to do this.
Fixes: http://tracker.ceph.com/issues/23469 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Nathan Cutler [Tue, 13 Mar 2018 16:36:00 +0000 (17:36 +0100)]
legal: remove doc license ambiguity
The license applicable to the files under doc/ was originally declared (in the
top-level file COPYING) to be "Creative Commons Attribution-ShareAlike (CC
BY-SA)" by ed0653b493a3f919a3abc37a0aa9b5aa29ae0b0e
This license declaration omitted a version number.
Some time later ef7418421b3748c712019c8aedd02b8005c1e1ea was merged, mentioning
CC-BY-SA-1.0 as one of the Ceph source code licenses. Although the purpose of that
commit was only to summarize the canonical license information from COPYING, it
unintentionally became the only place in the source code where the doc license
version was specified.
In March 2018, the Debian project warned that CC-BY-SA-1.0 does not meet its
criteria for inclusion in the "free" section of the Debian distribution. (For
that, at least version 3.0 must be used.)
This new commit removes the doc license ambiguity by setting it to CC-BY-SA-3.0
in all places where the license is mentioned.
The exact spelling of the license name is taken from https://spdx.org/licenses/
Josh Durgin [Tue, 6 Mar 2018 01:04:10 +0000 (20:04 -0500)]
tools: Add pg log trim command to ceph-objectstore-tool
Add test script that verifies the command in qa/standalone/osd
Fixes: http://tracker.ceph.com/issues/23242 Signed-off-by: Josh Durgin <jdurgin@redhat.com> Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b01e4ea5e2c1d2e9e997f53621aba73a7f7c9241)
Conflicts:
src/tools/ceph_objectstore_tool.cc
Different ops
Need ObjectStore::Sequencer
Use apply_transaction()
qa/standalone/special/ceph_objectstore_tool.py
Different ops
Conflicts:
src/common/legacy_config_opts.h src/common/options.cc
Use src/common/config_opts.h in Jewel
src/test/ceph_objectstore_tool.py (different ops)
src/tools/ceph_objectstore_tool.cc (different ops, no PastIntervals)
src/test/osd/repro_long_log.sh
Moved from ../qa/standalone/osd
Use Jewel ceph-helpers.sh location
No ceph-mgr in Jewel
Single test that creates 51 write entries and trims to 30
Since web frontends may signal an error when requests are malformed or so, let
us double check this and raise errors early. The current user of this is
civetweb frontend; which can potentially return null from `parse_http_headers`
when a HTTP header without a ":" is supplied at which point headers.value is
null which can lead to undefined behaviour later in RGW.
Conflicts: this change was introduced to master by 8fc67075. but that commit
involves a lot of py3 compatibility changes, which are not the focus of
jewel or this fix. so i am only cherry-pick the change writing buffer
returned by asock to stdout.
since we use http manager which in turn uses curl and uses curl multi
interfaces. While curl is initialized at the first call of curl_easy_init() this
method isn't guaranteed to be safe when multiple threads may call the function
since curl_global_init isn't reentrant. Calling curl_global_init via
rgw::curl::setup_curl which additionally sets up ssl interfaces etc. when
openssl is used as curl's ssl backend. Similarly moving rgw target link to
accomodate this change.
Conflicts:
src/rgw/CMakeLists.txt
beast submodule related changes which are not a part of this patchset
src/rgw/rgw_admin.cc
trivial whitespace conflicts #
openssl <= 1.02 requires explicit callbacks for locking which libcurl doesn't
set. This causes random segmentation faults when openssl uses its global
structures across multiple threads. Providing a simple mutex lock/unlock
functions as a callback. We determine whether openssl is used for libcurl via
curl-config utility which should be installed as a part of our curl development
headers package. We also additionally check that the openssl version is < 1.1.0
which alleviates the need for these callbacks. In this patchset we have done the
following:
- move all curl related global init functionality under rgw::curl namespace
since libcurl may need to set up various ssl libraries etc during its init
- introduce WITH_CURL_OPENSSL in cmake
this checks the backend curl is deployed with using curl-config. Since curl
devel is expected to be installed anyway, this binary should be available and
can help identify the ssl backend curl was compiled with.
- we only setup the locks if beast/civetweb aren't terminated with ssl, since
these libraries setup the locks anyway and we want to prevent double
initialization of openssl. Also we pass in ~CURL_GLOBAL_SSL making curl not
initialize openssl if civetwb/beast is initializing them. Unfortunately this
flag is a noop from curl >= 7.57 wherein both the libraries will end up
initializing openssl anyway, which might override certain settings like error
strings if using openssl < 1.1
Conflicts:
- CMakeLists.txt
libcryptopp drop in master conflict
- src/rgw/rgw_main.cc
conflict as we removed the ac related headers in Luminous
Conflicts:
CMake isn't supported in jewel, while the CMake changes were cherried,
CMakelists for rgw does not exist in jewel and this commit doesn't fix that.
So though CMake changes are backported, cmake build is untested and expected
to be broken
src/include/config-h.in.cmake
src/rgw/rgw_main.cc
Marcus Watts [Tue, 27 Feb 2018 07:53:37 +0000 (02:53 -0500)]
rgw: reuse CURL* for keystone.
When using keystone with https (the recommended setting),
it is inefficient to start up a new SSL connection for
each and every operation. Keeping a CURL* structure around
should reduce the cost of doing this. This logic tries
to do so, but it also tries to free them fairly aggressively
(5-10 seconds). This should still greatly reduce load on
keystone at peak times while not tying up excess resources.
Fixes: https://tracker.ceph.com/issues/23162 Signed-off-by: Marcus Watts <mwatts@redhat.com>
(cherry picked from commit 0c5cee1977eac232045571f4850e64be93271aec)
Conflicts:
src/rgw/rgw_common.h
trivial conflict as lowercase_dash_http_attr isnt a part of L
projects / ceph.git / log