Marcus Watts [Thu, 23 Feb 2017 07:30:52 +0000 (02:30 -0500)]
rgw/openssl fix: xenial autoconf logic problem: gcc/ld got too smart...
On xenial, cc -o foo -lssl -lcrypto doesn't always
record libssl.so libcrypto.so as runtime library dependencies.
It is necessary to actually *use* a function from the library before
it gets recorded. The ld(1) options "--as-needed" and "no-as-needed"
control this. Evidently the default has changed in xenial.
That caused my smart "soname" detecting logic for openssl to
stop working. To make it work, the test program has to
actually reference routines or variables inside the library.
This is a quick fix for xenial / autoconf. There needs to be
a better fix for cmake and master.
For civetweb: accept a range of port numbers joined with '+'.
Port numbers may include an ipaddress: prefix and 's' suffix.
Additionally, use "mg_get_local_addr" to correctly deduce host port per
incoming connection.
civetweb can accept connections on multiple ports, some of which
might have SSL turned on and some not. Both s3 and swift have various
authorization protocols in which the port number matters. In the generic
radosgw frontend process, each frontend only has one port number, but
we should want to have both ssl and non-ssl connections managed within
one rgw frontend, because the thread pool is also per front-end, and
that *is* a scarce resource.
So, this patch enables the use of multiple ports with a single civetweb
frontend. To indicate https: append an 's' to portno. To use multiple
ports, use +. So 80+443s indicates use of the usual default http ports.
The parsed port is not stored in the frontend structure,
So instead, this patch adds logic to use the results of
mg_get_local_addr() on a per-connection basis insetad of the generic
front-end port number. This will affect "v4" s3 authorization, and also
affect swift pre-signed URLs.
mg_get_local_addr() is a new customization to civetweb; that submodule
was updated (in a temporary repository) by the previous commit to this.
Marcus Watts [Tue, 20 Dec 2016 05:22:02 +0000 (00:22 -0500)]
rgw: s3: secure_port should override port, also apply ssl default right.
Without https, only port is set. With https, secure_port and port are
both set to the same value. The previous logic looked at port first and
had overly simplified conditional logic which was liable to try to apply
both non-default cases. The correct behavior is: look secure_port first,
and if secure_port is set, then only check to see if it's a non-default
port.
Marcus Watts [Wed, 2 Nov 2016 04:18:48 +0000 (00:18 -0400)]
rgw: Get civetweb ssl enhancement: wip-listen3 = mg_get_local_addr
The logic inside of radosgw that computes aws v4 signatures wants to know
what server port the client connected. The patch to civetweb patch adds a
call mg_get_local_addr() which will permit that code to actually find out
on what address a connection was received, rather than merely guessing
based on configuration as it previously did.
Marcus Watts [Fri, 13 Jan 2017 08:39:57 +0000 (03:39 -0500)]
rgw: civetweb/openssl: automagic: load libssl.so and libcrypto.so by soname.
If building with radosgw, always look for openssl library (even when
building with nss). Then, use objdump to fetch SONAME from the copies
of libssl and libcrypto that were found. When building civetweb; pass
the library soname values in as the libraries to load with "dlopen".
The issue reported here against master
http://tracker.ceph.com/issues/16535
reflects stuff that doesn't seem to have made it into jewel
(which had: -lssl -lcrypto hard-wired into it.)
Still, since people were pretty riled about making that change,
this puts things in congruence to the final result in master.
(which is: runtime shared object load of ssl crypto by soname.)
Marcus Watts [Mon, 11 Jul 2016 21:54:07 +0000 (17:54 -0400)]
rgw: civetweb/openssl: Load libssl.so and libcrypto.so by soname.
If building with radosgw, always look for openssl library (even when
building with nss). Then, use objdump to fetch SONAME from the copies
of libssl and libcrypto that were found. When building civetweb; pass
the library soname values in as the libraries to load with "dlopen".
This is a problem that went away for a bit, but came back with some
changes for
http://tracker.ceph.com/issues/16535
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1341775 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1258961 Fixes: http://tracker.ceph.com/issues/11239 Signed-off-by: Marcus Watts <mwatts@redhat.com>
(cherry picked from commit 7caa0bd002110b62514da83a37a2a3deb841267a)
Marcus Watts [Fri, 4 Nov 2016 03:33:44 +0000 (23:33 -0400)]
rgw: cmake: remove useless civetweb include path side effect.
For 'target_include_directories" for the cmake object library
'civetweb_common_objs', change from PUBLIC to PRIVATE. This doesn't
break anything, so it wasn't doing anything useful. If it has it
any effect, it would be to cause everything that linked against this
"library" to also use the indictated include path. Which would be great
except everything in ceph wants to include "civetweb/civetweb.h" and
not "civetweb.h". We already make separate arrangements elsewhere for
that to work. Additionally, static object libraries in cmake aren't
really libraries, so I'm not entirely sure this even does anything.
So: making this public is not useful, and could be harmful. Making it
private makes this only take effect for building civetweb.c itself,
exactly the effect we we require, and no more.
Sage Weil [Thu, 22 Dec 2016 18:05:22 +0000 (13:05 -0500)]
qa/tasks/workunit: clear clone dir before retrying checkout
If we checkout ceph-ci.git, and don't find a branch,
we'll try again from ceph.git. But the checkout will
already exist and the clone will fail, so we'll still
fail to find the branch.
The same can happen if a previous workunit task already
checked out the repo.
Fix by removing the repo before checkout (the first and
second times). Note that this may break if there are
multiple workunit tasks running in parallel on the same
role. That is already racy, so if it's happening, we'll
want to switch to using a truly unique clonedir for each
instantiation.
Samuel Just [Mon, 12 Dec 2016 18:35:38 +0000 (10:35 -0800)]
PG: fix cached_removed_snaps bug in PGPool::update after map gap
5798fb3bf6d726d14a9c5cb99dc5902eba5b878a actually made 15943 worse
by always creating an out-of-date cached_removed_snaps value after
a map gap rather than only in the case where the the first map after
the gap did not remove any snapshots.
Jeff Layton [Tue, 20 Dec 2016 19:44:04 +0000 (14:44 -0500)]
ceph_disk: fix a jewel checkin test break
Silly python:
ceph_disk/main.py:173:1: E305 expected 2 blank lines after class or function definition, found 1
ceph_disk/main.py:5011:1: E305 expected 2 blank lines after class or function definition, found 1
Jeff Layton [Tue, 20 Dec 2016 13:17:21 +0000 (08:17 -0500)]
client: drop setuid/setgid bits on ownership change
When we hold exclusive auth caps, then the client is responsible for
handling changes to the mode. Make sure we remove any setuid/setgid
bits on an ownership change.
Jeff Layton [Tue, 20 Dec 2016 13:16:43 +0000 (08:16 -0500)]
mds: clear setuid/setgid bits on ownership changes
If we get a ownership change, POSIX mandates that you clear the
setuid and setgid bits unless you are "appropriately privileged", in
which case the OS is allowed to leave them intact.
Linux however always clears those bits, regardless of the process
privileges, as that makes it simpler to close some potential races.
Have ceph do the same.
Jeff Layton [Tue, 20 Dec 2016 13:07:23 +0000 (08:07 -0500)]
client: set metadata["root"] from mount method when it's called with a pathname
Currently, we only set the root properly config file or the
--client_metadata command line option. If a userland client program
tries to call ceph_mount with a pathname, it's not being properly
set.
Since we already hold the mutex, we can just update it directly.
Sage Weil [Wed, 14 Dec 2016 17:18:29 +0000 (12:18 -0500)]
tasks/workunit: remove kludge to use git.ceph.com
This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo. Remove it entirely. We'll see if
github.com is better at handling our load than it used to
be!
Sage Weil [Thu, 8 Dec 2016 00:25:55 +0000 (18:25 -0600)]
msg/simple/Pipe: avoid returning 0 on poll timeout
If poll times out it will return 0 (no data to read on socket). In 165e5abdbf6311974d4001e43982b83d06f9e0cc we changed tcp_read_wait from
returning -1 to returning -errno, which means we return 0 instead of -1
in this case.
This makes tcp_read() get into an infinite loop by repeatedly trying to
read from the socket and getting EAGAIN.
Fix by explicitly checking for a 0 return from poll(2) and returning
EAGAIN in that case.
Nathan Cutler [Thu, 24 Nov 2016 10:25:35 +0000 (11:25 +0100)]
thrashosds: try ceph-objectstore-tool for 10 minutes
If ceph-objectstore-tool binary is not present, it's likely because we're in
the middle of an upgrade. Do not try to run the binary until we verify that
it's really present. If it is absent, spend up to 10 minutes waiting for it to
appear.
Before this patch there was quite a large window for a race to occur. This
patch doesn't entirely eliminate it, but drastically reduces it.
Nathan Cutler [Sat, 3 Dec 2016 12:29:56 +0000 (13:29 +0100)]
build/ops: fix undefined crypto references with --with-xio
Only with --with-xio, RPM build fails due to undefined references to various
symbols starting with "PK11_" in ./.libs/libcommon.a(Crypto.o) in several
of the unit tests.
Samuel Just [Mon, 14 Nov 2016 19:50:23 +0000 (11:50 -0800)]
OSDMonitor: only reject MOSDBoot based on up_from if inst matches
If the osd actually restarts, there is no guarrantee that the epoch will
advance past up_from. If the inst is different, it can't really be a
dup. At worst, it might be a queued MOSDBoot from a previous inst, but
in that case, the real inst would see itself marked up, and then back
down causing it to try booting again.
Loic Dachary [Tue, 29 Nov 2016 08:49:15 +0000 (09:49 +0100)]
upgrade/hammer-x: verify shec before the full upgrade
The hammer-x/stress-split-erasure-code upgrade sequence comes from
hammer-x/stress-split and was modified to fully upgrade the cluster. It
previously upgraded only half of it. Verifying that the shec plugin is
not available and that trying to set it does not crash the OSD or the
MON must be tried before the upgrade is complete.
[Updated write_file to use all feaetures]
[Updated OSDMonitor.cc to use mon->quorum_features instead of the
mon->get_quorum_con_featuers() helper]
[trivial conflict from removed write_file and read_file]
Sage Weil [Mon, 28 Nov 2016 15:29:40 +0000 (10:29 -0500)]
upgrade/hammer-x/parallel: white 'failed to encode'
The problem here has nothing to do with osdmap
encoding, but that hammer -> jewel makes the systemd
transition and installing the package starts
the mons.. before the osds. I'm not sure what
the workaround for that is but the osdmap issue
appears okay, so ignore this for now.
Loic Dachary [Fri, 18 Nov 2016 07:06:02 +0000 (08:06 +0100)]
tests: save 9 characters for asok paths
For vstart.sh powered tests, save 9 characters in the path name
by replacing testdir/test- with td/t-
60 characters imposed by jenkins
9 characters for src/test
5 characters for td/t-
33 left (instead of 24) for the test to create asok such as out/client.admin.25327.asok
Moving these files outside of the build directory is a bad idea because
tests should only create/use files within the builddir and not write
outside of this directory. Doing so would make things more complicated
for cleanup in case the test fail and create other problems as a
consequence (filling out disk space, conflicting directories between
runs etc.).
For ceph-helpers.sh tests replace testdir with td, saving 5 characters.
This is not strictly necessary but keeps the directory names consistent:
if the developer wants to get rid of all the test leftovers, it is
enough to remove the a single directory: td.
projects / ceph.git / log