qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 549d993d3fd8ffffa280ed4a64aca41d1c6f2da1)

Conflicts:
qa/tasks/cram.py (trivial resolution)

Merge pull request #12983 from ceph/wip-cherry-pick-4vasu

qa: Wip cherry pick https://github.com/ceph/ceph/pull/12969

Add ceph-create-keys to explicitly create admin/bootstrap keys

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit 68f9b7eb3c0548c88650f67fb72c6ff9bc0f3ead)

Remove debug overrides

the high level of debug for mon/osd is causing remoto to hang during get key

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit f7dcc74cd3f119a2f65584fdb544c08d115f8c39)

use the create option during instantiation

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit be836bb30960000468c79e08fb416ceefd79d7db)

Merge pull request #12210 from ddiss/tracker18049_ceph_disk_trigger_flock_timeout_jewel

jewel: systemd/ceph-disk: reduce ceph-disk flock contention

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #12959 from SUSE/wip-18545-jewel

jewel: [teuthology] update Ubuntu image url after ceph.com refactor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6d17befb3bbc3d83c9d23d763ad95e1e7b2e4be0)

Merge pull request #12745 from SUSE/wip-18386-jewel

jewel: tests: use ceph-jewel branch for s3tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12912 from liewegas/wip-workunits-jewel

qa/tasks/workunits: backport misc fixes to jewel

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/workunit: clear clone dir before retrying checkout

If we checkout ceph-ci.git, and don't find a branch,
we'll try again from ceph.git. But the checkout will
already exist and the clone will fail, so we'll still
fail to find the branch.

The same can happen if a previous workunit task already
checked out the repo.

Fix by removing the repo before checkout (the first and
second times).  Note that this may break if there are
multiple workunit tasks running in parallel on the same
role.  That is already racy, so if it's happening, we'll
want to switch to using a truly unique clonedir for each
instantiation.

Fixes: http://tracker.ceph.com/issues/18336
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2a7013cd5a033c5be43350505d75f088e831e201)

qa/tasks/workunit: retry on ceph.git if checkout fails

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 72d73b8c8836ae35c518fa09f44805a74038f02a)

qa/tasks/workunit.py: add CEPH_BASE env var

Root of git checkout

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 27b8eac24922f8b4bd065e6e7f0bc8e2ba37b5d5)

qa/tasks/workunit: leave workunits inside git checkout

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4602884ab8f5a256d13091f7239d938990482d95)

Merge pull request #12791 from athanatos/wip-15943-jewel

jewel: crash adding snap to purged_snaps in ReplicatedPG::WaitingOnReplicas (part 2)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12868 from athanatos/wip-17899-jewel

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

Reviewed-by: Sage Weil <sage@redhat.com>

PG: fix cached_removed_snaps bug in PGPool::update after map gap

5798fb3bf6d726d14a9c5cb99dc5902eba5b878a actually made 15943 worse
by always creating an out-of-date cached_removed_snaps value after
a map gap rather than only in the case where the the first map after
the gap did not remove any snapshots.

Introduced: 5798fb3bf6d726d14a9c5cb99dc5902eba5b878a
Fixes: http://tracker.ceph.com/issues/15943
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 5642e7e1b3bb6ffceddacd2f4030eb13a17fcccc)

qa/config/rados.yaml: enable osd_debug_verify_cached_snaps

Also, make map gaps more likely.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit d4b6615a49e4635113f9ba900e9c57147b224357)

PG::handle_advance_map: add debugging option to verify cached_removed_snaps

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit aeb8fef92469831d94f06db457a4ba15b5b0e3c5)

tests: use ceph-jewel branch for s3tests

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge remote-tracking branch 'ceph/jewel-next' into jewel

Merge pull request #12591 from jtlayton/wip-18308-jewel

Clear setuid bits on ownership changes

Merge branch 'jewel' into wip-18308-jewel

Merge pull request #12592 from jtlayton/wip-18307-jewel

Fix mount root for ceph_mount users and change tarball format

ceph_disk: fix a jewel checkin test break

Silly python:

    ceph_disk/main.py:173:1: E305 expected 2 blank lines after class or function definition, found 1
    ceph_disk/main.py:5011:1: E305 expected 2 blank lines after class or function definition, found 1

Signed-off-by: Jeff Layton <jlayton@redhat.com>

automake: convert to tar-pax

We hit some recent build issues with the merge of ceph-qa-suite into
the main repo. The ustar format barfs on >100 character symlink
paths.

Convert to using "tar-pax" which should make it use the posix format.
Any build machine that we're reasonably targeting should support it.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: drop setuid/setgid bits on ownership change

When we hold exclusive auth caps, then the client is responsible for
handling changes to the mode. Make sure we remove any setuid/setgid
bits on an ownership change.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 18d2499d6c85a10b4b54f3b8c335cddf86c4588f)

mds: clear setuid/setgid bits on ownership changes

If we get a ownership change, POSIX mandates that you clear the
setuid and setgid bits unless you are "appropriately privileged", in
which case the OS is allowed to leave them intact.

Linux however always clears those bits, regardless of the process
privileges, as that makes it simpler to close some potential races.
Have ceph do the same.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 6da72500882d9749cb2be6eaa2568e6fe6e5ff4d)

client: set metadata["root"] from mount method when it's called with a pathname

Currently, we only set the root properly config file or the
--client_metadata command line option. If a userland client program
tries to call ceph_mount with a pathname, it's not being properly
set.

Since we already hold the mutex, we can just update it directly.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 9f8810008c82eebe6e354e7e321e33a3dcba8407)

Merge pull request #12454 from liewegas/qa-suite-jewel

jewel: merge ceph-qa-suite

merge ceph-qa-suite

move ceph-qa-suite dirs into qa/

Revert "tasks/workunit.py: depth 1 clone"

This reverts commit e6f61ea9f19d0f1fad4a6547775fa80616eeeb89.

tasks/workunit.py: depth 1 clone

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4faf77a649cb3f8ddf497ca81937b3dbf63a18dc)

tasks/workunit: remove kludge to use git.ceph.com

This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo.  Remove it entirely.  We'll see if
github.com is better at handling our load than it used to
be!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 159c455a0326eef2c017b3e3cf510f918b5ec76c)

tasks/ceph: restore context of osd mount path before mkfs

all newly created files and directories under the mount dir inherit the
SELinux type of their parent directory. so we need to set it before
mkfs.

Fixes: http://tracker.ceph.com/issues/16800
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 53225d5272a1d35d4183fcfa55a139f55f77e122)

10.2.5

Merge pull request #11865 from dachary/wip-17710-jewel

jewel: multisite: race between ReadSyncStatus and InitSyncStatus leads to EIO errors

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12376 from liewegas/wip-msgr-eagain-loop-jewel

msg/simple/Pipe: avoid returning 0 on poll timeout

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg/simple/Pipe: avoid returning 0 on poll timeout

If poll times out it will return 0 (no data to read on socket).  In
165e5abdbf6311974d4001e43982b83d06f9e0cc we changed tcp_read_wait from
returning -1 to returning -errno, which means we return 0 instead of -1
in this case.

This makes tcp_read() get into an infinite loop by repeatedly trying to
read from the socket and getting EAGAIN.

Fix by explicitly checking for a 0 return from poll(2) and returning
EAGAIN in that case.

Fixes: http://tracker.ceph.com/issues/18184
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6c3d015c6854a12cda40673848813d968ff6afae)

Merge pull request #12033 from dachary/wip-17926-jewel

jewel: ceph-disk --dmcrypt create must not require admin key

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11968 from ddiss/jewel_next_flush_evict_snaps

jewel: tools: snapshotted RBD extent objects can't be manually evicted from a cache tier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12151 from dachary/wip-18011-jewel

jewel: test fails due to The UNIX domain socket path

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12296 from SUSE/wip-18133-jewel

jewel: build/ops: fix undefined crypto references with --with-xio

Reviewed-by: Loic Dachary <ldachary@redhat.com>

10.2.4

Merge pull request #11997 from Abhishekvrshny/wip-17876-jewel

jewel: osd: update_log_missing does not order correctly with osd_ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11944 from SUSE/wip-17866-jewel

jewel: osd: Add config option to disable new scrubs during recovery

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11672 from linuxbox2/jewel-17663

jewel: rgw_rest_s3:  apply missed base64 try-catch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11953 from SUSE/wip-17885-jewel

jewel: test: temporarily disable fork()'ing tests

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11884 from SUSE/wip-17754-jewel

jewel: ceph-create-keys loops forever

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11529 from SUSE/wip-17600-jewel

jewel: common: Improve linux dcache hash algorithm

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12167 from liewegas/wip-osdmap-encoding-jewel

jewel: osd: condition OSDMap encoding on features

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #1290 from SUSE/wip-18014-jewel

thrashosds: try ceph-objectstore-tool for 10 minutes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

thrashosds: try ceph-objectstore-tool for 10 minutes

If ceph-objectstore-tool binary is not present, it's likely because we're in
the middle of an upgrade. Do not try to run the binary until we verify that
it's really present. If it is absent, spend up to 10 minutes waiting for it to
appear.

Before this patch there was quite a large window for a race to occur. This
patch doesn't entirely eliminate it, but drastically reduces it.

Fixes: http://tracker.ceph.com/issues/18014
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 862b47faac1fc9f05ee3322ee4b65cf3d3d666c5)

build/ops: fix undefined crypto references with --with-xio

Only with --with-xio, RPM build fails due to undefined references to various
symbols starting with "PK11_" in ./.libs/libcommon.a(Crypto.o) in several
of the unit tests.

Fixes: http://tracker.ceph.com/issues/18133
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12067 from SUSE/wip-17953-jewel

jewel: mon: OSDMonitor: only reject MOSDBoot based on up_from if inst matches

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1297 from ceph/wip-14.04

suites/rados: s/trusty/"14.04"/

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

If the osd actually restarts, there is no guarrantee that the epoch will
advance past up_from.  If the inst is different, it can't really be a
dup.  At worst, it might be a queued MOSDBoot from a previous inst, but
in that case, the real inst would see itself marked up, and then back
down causing it to try booting again.

Fixes: http://tracker.ceph.com/issues/17899
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 033ad5b46c0492134e72a8372e44e3ef1358d2df)

Merge pull request #12207 from jdurgin/wip-librados-setxattr-overload-jewel

librados: remove new setxattr overload to avoid breaking the C++ ABI

Reviewed-by: Sage Weil <sage@redhat.com>

suites/rados: s/trusty/"14.04"/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1296 from dachary/wip-ceph-coverage-jewel

upgrade: ceph-test is needed for ceph-coverage

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

upgrade/infernalis-client-x: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8122494530589e27df929652e38c74137d6d823a)

Merge pull request #12267 from dachary/wip-17904-jewel

 jewel: Error EINVAL: removing mon.a at 172.21.15.16:6789/0, there will be 1 monitors

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1284 from ceph/jewel-name-limits

drop broken name length config args

Merge pull request #1292 from ceph/jewel-avoid-xenial

rados: avoid xenial for upgrade tests

mon: MonmapMonitor: drop unnecessary 'goto' statements

Signed-off-by: Joao Eduardo Luis <joao@suse.de>
(cherry picked from commit 20dcb597e35e6961db81831facefbe22cecddec3)

mon: MonmapMonitor: return success when monitor will be removed

Fixes: http://tracker.ceph.com/issues/17725
Signed-off-by: Joao Eduardo Luis <joao@suse.de>
(cherry picked from commit c9d46cfbf2512bc3495c6901de2b8f711bef9bae)

Merge pull request #11999 from Abhishekvrshny/wip-17904-jewel

jewel: Error EINVAL: removing mon.a at 172.21.15.16:6789/0, there will be 1 monitors

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12001 from dachary/wip-17915-jewel

jewel: filestore: can get stuck in an unbounded loop during scrub

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12174 from SUSE/wip-18025-jewel

rgw: don't store empty chains in gc

Merge pull request #11758 from Abhishekvrshny/wip-17784-jewel

jewel: osd crashes when radosgw-admin bi list --max-entries=1 command runing

upgrade/client-upgrade: correct distros/ location

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrade: fix distro symlinks

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrade: specify centos or trusty (not xenial)

Signed-off-by: Sage Weil <sage@redhat.com>

rados: avoid ubuntu xenial on upgrade tests

Not all of the older package builds are present for xenial.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1152 from ceph/wip-objecstore

rados/objectstore/objectstore.yaml: skip bluestore tests

Merge pull request #1288 from dachary/wip-shec-upgrade-jewel

upgrade/hammer-x: verify shec before the full upgrade

Reviewed-by: Sage Weil <sage@redhat.com>

systemd/ceph-disk: reduce ceph-disk flock contention

"ceph-disk trigger" invocation is currently performed in a mutually
exclusive fashion, with each call first taking an flock on the path
/var/lock/ceph-disk. On systems with a lot of osds, this leads to a
large amount of lock contention during boot-up, and can cause some
service instances to trip the 120 second timeout.

Take an flock on a device specific path instead of /var/lock/ceph-disk,
so that concurrent "ceph-disk trigger" invocations are permitted for
independent osds. This greatly reduces lock contention and consequently
the chance of service timeout. Per-device concurrency restrictions
required for http://tracker.ceph.com/issues/13160 are maintained.

Fixes: http://tracker.ceph.com/issues/18060
Signed-off-by: David Disseldorp <ddiss@suse.de>
(cherry picked from commit 8a62cbc074b711275cfd57b372bfb35f6a017833)

upgrade/hammer-x: verify shec before the full upgrade

The hammer-x/stress-split-erasure-code upgrade sequence comes from
hammer-x/stress-split and was modified to fully upgrade the cluster. It
previously upgraded only half of it. Verifying that the shec plugin is
not available and that trying to set it does not crash the OSD or the
MON must be tried before the upgrade is complete.

Signed-off-by: Loic Dachary <loic@dachary.org>

librados: remove new setxattr overload to avoid breaking the C++ ABI

Fixes: http://tracker.ceph.com/issues/18058
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit b8ff781ddcf737882163cf56d7c9b11e815fb699)

Conflicts:
src/include/rados/librados.hpp (trivial namespace change in removed line)

Merge pull request #1287 from ceph/jewel-failed-to-encode

upgrade/hammer-x: encoding fixes (jewel)

Reviewed-by: Samuel Just <sjust@redhat.com>

crush: condition latest tunable encoding on features

This avoids throwing hammer OSDMap encodings off.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9e5ff86487bd1f5979866b5e16300dd4a3979f97)

crush/CrushWrapper: encode with features

No behavior change yet; just fixing callers.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b7c9e055848c8aa951bc48c957cff3ef323ea441)

[Updated write_file to use all feaetures]
[Updated OSDMonitor.cc to use mon->quorum_features instead of the
 mon->get_quorum_con_featuers() helper]
[trivial conflict from removed write_file and read_file]

Conflicts:
src/crush/CrushWrapper.h
src/mgr/PyModules.cc
src/mon/OSDMonitor.cc
src/tools/ceph_monstore_tool.cc

crush/CrushWrapper: drop unused 'lean' encode() argument

No callers, no users.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 638a38bbb55c07ad0358a35a56418e66874d1c26)

Conflicts:
src/crush/CrushWrapper.h

[trivial conflict due to removal of write_file and read_file]

upgrade/hammer-x/stress-split-*: disable sighup injection

We already did this for stress-split; do the same
here.  It triggers a File closed exception when the greenlet
is joined.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: white 'failed to encode'

The problem here has nothing to do with osdmap
encoding, but that hammer -> jewel makes the systemd
transition and installing the package starts
the mons.. before the osds.  I'm not sure what
the workaround for that is but the osdmap issue
appears okay, so ignore this for now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1285 from ceph/jewel-failed-to-encode

upgrade/hammer-x: do not whitelist 'failed to encode map'

upgrade/hammer-x/parallel: upgrade osds first

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: do not whitelist 'failed to encode map'

Well, on parallel.  For the others, keep it in
place because we don't upgrade osds first (we are testing
other things).

Signed-off-by: Sage Weil <sage@redhat.com>

drop broken name length config args

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2a44c3d20de9a75065c271e9ad8dfceeed1186d9)

rgw: don't store empty chains in gc

Fixes: http://tracker.ceph.com/issues/17897
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7c8f1b9edc921711fa30345bbecea2dcd5de1229)

upgrade/hammer-x: fix symlinks

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: encode pg_pool_t like hammer if features indicate hammer

If the target features are missing the new OSDOp encoding, the
first feature we added post-hammer, encode like hammer.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2f8cfb632823ba4e63eaff394392d6af7979d7c8)

osd/osd_types: conditional pg_pool_t encoding

Align this with decode.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 01d9e8a20bbc3c039f67b040da95018e2c7b00b6)

Merge pull request #1273 from ceph/wip-whitelist-crc

whitelist CRC mismatch

Merge pull request #1281 from ceph/wip-jewel-debug-fuse

upgrade/hammer-x: debug mds

upgrade/hammer-x: debug mds

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1280 from ceph/wip-jewel-debug-fuse

upgrade/hammer-x: debug client

upgrade/hammer-x: debug client

Signed-off-by: Sage Weil <sage@redhat.com>