cmake: test-ceph-helpers working

Moved all the libraries in CMAKE_BINARY_DIR/lib
and all the binaries in CMAKE_BINARY_DIR/bin. Set
various environment variables for test-ceph-helpers.
Put those variables throughout
qa/workunits/ceph-helpers.sh.

NOTE: This is a very rough draft of these fixes.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Reorganized test dir, added cmake functions

make check targets now use add_ceph_test, or
add_ceph_unittest to be added to CTest. Renamed and
reordered targets in tests/CMakeLists.txt

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #8606 from dillaman/wip-15457

rbd: loosen image spec validation requirements

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8603 from rmechler/wip-15362

pybind/rados: fix omap method return values

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

release-notes: document rbd cli name validation override

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: fail if cannot extract spec from legacy "--image" optional

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: optionally disable spec validation when creating images/snapshots

Setting the "rbd-validate-names" config propery to false will disable
spec validation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add support for relaxed image spec validation

Operations which create images or snapshots still fully enforce
the spec validation.  All other operations use a relaxed validation
format.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: new rbd image spec validation option

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Removed parentheses for if statement.

Signed-off-by: Roland Mechler <rmechler@cisco.com>

Merge pull request #8588 from yehudasa/wip-rgw-period-id

rgw: try to use current period id in a few more cases

Merge pull request #8566 from cbodley/wip-rgw-period-reflect

rgw: RGWPeriod::reflect() sets master zonegroup as default
Reviewed-by: Orit Wasserman<owasserm@redhat.com>

update release-notes manually

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8585 from cbodley/wip-rgw-reload-usage

rgw: call rgw_log_usage_finalize() on reconfiguration

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8600 from liewegas/wip-rados-lfn

ceph_test_rados: make long name ~300 chars, (not ~800)

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8604 from gregsfortytwo/wip-fsmap-decode-fix

mds: FSmap decode fix

Reviewed-by: Sage Weil <sage@redhat.com>
Tested-by: Sage Weil <sage@redhat.com>

Merge pull request #8565 from dillaman/wip-15471

librbd: fix handling of proxied maintenance operations during shut down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: do not return a failure if a peer cannot be notified of updateReviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: do not return a failure if a peer cannot be notified of update

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8511 from dillaman/wip-15436

librbd: IO deadlock when dynamically enabling/disabling features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8494 from dillaman/wip-15352

journal: possible race condition during live replay

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Fixes for python omap method return values.

Signed-off-by: Roland Mechler <rmechler@cisco.com>

FSMap: guard the ever_enabled_multiple decode

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph_test_rados: make long name ~300 chars, (not ~800)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8594 from dachary/wip-releases

release-notes: v10.1.2 release notes

cmake: pass RULE_LAUNCHER_* to cythonReviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: pass RULE_LAUNCHER_* to cython

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Robin H. Johnson <robin.johnson@dreamhost.com>

librbd: dynamically disabling exclusive lock might leave lingering request

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8534 from kmroz/wip-rgw-rados-cleanup-1

rgw-rados: return RGWSystemMetaObj init directly
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

release-notes: v10.1.2 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: v10.1.2 release notes (draft)

cd src/script
virtualenv v
source v/bin/activate
pip install githubpy GitPython requests
GITHUB_ACCESS_TOKEN=xxxxxxx ./ceph-release-notes \
    -r tags/v10.1.1..tags/v10.1.2 $(git rev-parse --show-toplevel)

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #8381 from ceph/wip-cython-tarball-build

cmake/pybind: fix include paths for cephfs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8569 from jdurgin/wip-dev-package-doc

doc: reinstate accidentally removed section header

Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #8587 from emenguy/fix_dependencies_documentation

doc: fix dependencies

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8584 from dillaman/wip-15479

test: image replayer needs dummy remote mirror peer uuid

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: try to use current period id in a few more cases

If missing period id, try to use the current period id. This is needed
for sync agent compatibility.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8188 from ceph/wip-systemd-restart-defaults

systemd: Use the same restart limits as upstart

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: fix dependencies

Signed-off-by: Etienne Menguy <etienne.menguy@corp.ovh.com>

systemd: Use the same restart limits as upstart

Currently, the systemd daemons are not restarted on failure. This patch
adds this functionality and sets the defaults to those defined in
upstart. This resolves to 3 fails per 30 minutes for osd, mon and mds
and 5 fails per 30 seconds for radosgw.

Signed-off-by: Boris Ranto <branto@redhat.com>

rgw: call rgw_log_usage_finalize() on reconfiguration

RGWRealmReloader was calling rgw_log_usage_init() with the new RGWRados
instance, but never cleaned up the previous one with
rgw_log_usage_finalize()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: image replayer needs dummy remote mirror peer uuid

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8498 from liewegas/wip-bluestore

os/bluestore: fix shortened bluefs paths in env mirror case

rgw: RGWPeriod::reflect() sets master zonegroup as default

if RGWPeriod::reflect() is called right after a 'realm pull', it will
create the zonegroup objects but not set any as default.  so the next
time RGWRados is initialized, create_defaults() will add a new 'default'
zonegroup and set it as the master.  this causes a 'period update' to
find multiple master zonegroups and fail with EINVAL

reflect() now calls set_as_default(exclusive=true) on the master
zonegroup to guarantee that a default is set

Signed-off-by: Casey Bodley <cbodley@redhat.com>

journal: race possible when re-watching a journal object

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fix shortened bluefs paths in env mirror case

If we are mirroring the rocksdb env, don't shorten the
paths.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8556 from liewegas/wip-ext4

doc: recommend against ext4

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #8574 from tchaikov/wip-readable-test-fix

test/encoding/readable: use [ for "test" not ((

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #8567 from cbodley/wip-rgw-period-pull-url

radosgw-admin: allow 'period pull --url' without full multisite config
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

test/encoding/readable: use [ for "test" not ((

and use '&&' instead, easier to understand.

this also fixes following error in the ubuntu gitbuilder

error: Added files:
+ cat .git/added-files
src/0

where sh points to dash.

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: potential for double-free of context on shut down

The context associated with a scheduled watch might be freed
by two ObjectPlayers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: possible race condition during live replay

When two objects are being actively watched, it was possible for
the watch context to complete before the second watch was
associated.

Fixes: http://tracker.ceph.com/issues/15352
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: reinstate accidentally removed section header

Accidentally removed in 63be401a411ffc7c2f78e450a29c69eee1af02d3

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

radosgw-admin: 'realm pull' only accepts --url

'realm pull' is defined as a 'raw storage op', so cannot use the
--remote argument to refer to endpoints from the period or local
zonegroup

refactored send_to_url() slightly to make it easier to call directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: allow 'period pull --url' without full multisite config

to support 'period pull --remote', radosgw-admin needs a complete
multisite configuration to look up remote endpoints by name

however, 'period pull' will also accept a --url argument, which doesn't
require a working configuration. this patch splits the handling of these
two cases, so that 'period pull --url' can succeed when run on a cluster
that hasn't yet been fully configured

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8520 from yuyuyu101/wip-15440

msg/async: AsyncConnection: avoid is_connected require connection's lock

librbd: cancel ops before blocking IO

Ops might have in-flight IO -- blocking IO after canceling the ops
will result in the in-flight IO being flushed.  Shutdown also requires
an intermediate state where is still acts like it downs the lock
until after all ops are canceled and all IO is flushed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: delay invalidation of exclusive lock pointer

Certain IO and maintenance ops code paths have an expectation
that the exclusive lock pointer will be valid while in-flight.
Let the exclusive lock state machine clean up the pointer after
it has flushed all IO and canceled all ops.

Fixes: http://tracker.ceph.com/issues/15471
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: cleanly fail bench-write upon failure

Fixes: http://tracker.ceph.com/issues/15456
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: gracefully handle blacklisted client in ImageWatcher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8559 from liewegas/wip-15643

rgw: fix unsafe c_str() usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

10.1.2

doc/configuration/filesystem-recommendations: recommend against ext4

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: mention ext4 in the release notes

Signed-off-by: Sage Weil <sage@redhat.com>

doc/start/os-recommendations: drop ancient 'syncfs' note

...and with it it's mention of ext4.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rados/configuration: minor cleanup of inline xattr description

- whitespace
- it's not omap

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/rgw_rados: use to_str() instead of c_str()

bufferlist::c_str() doesn't guarantee null termination.

Fixes: http://tracker.ceph.com/issues/15463
Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add list::to_str()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8552 from tchaikov/wip-15426

PG: set epoch_created and parent_split_bits for child pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8521 from stiopaa1/checkgenerated

check-generated.sh: can't source bash from sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge branch 'jewel'

Merge remote-tracking branch 'origin/greg-fs-testing-411' into jewel

doc: do not include ext4 as an example

Signed-off-by: Sage Weil <sage@redhat.com>

doc/8/ceph-deploy: do not include ext4 as an example value

Signed-off-by: Sage Weil <sage@redhat.com>

doc: list ext4 last

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev: remove ancient filestore-filesystem-compat doc

This describes a problem we fixed years ago.

Signed-off-by: Sage Weil <sage@redhat.com>

PG: set epoch_created and parent_split_bits for child pg

if we have a wrong "created", OSD receiving an osd_pg_create message
could fail to project its history and crash. a bad split_bits could
mislead the monitor and osd, and hence cause problems.
* set child.info.history.epoch_created to osdmap.get_epoch() instead of
  using the parent's epoch_created
* set child.info.stats.parent_split_bits to the calculated split_bits
  instead using the parent's split_bits

Fixes: http://tracker.ceph.com/issues/15426
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/Makefile.am: fix build with ccache

as a work-around of https://bugs.python.org/issue8027, pass ${CC} and
${LDSHARED} to distutils, so we can still have a working compiling
command line even ${CC} is "ccache gcc".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8544 from xiexingguo/xxg-wip-15464

test: fix ut test failure caused by lfn change

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: fix ut test failure caused by lfn change

"https://github.com/ceph/ceph/pull/8496/commits/755c685f2d09c36b53bc39589a77fa73faba5149"
change the lfn_parse_object_name() return type from bool to int but does not
change the relevant test correspondingly.

Fixes: http://tracker.ceph.com/issues/15464
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librbd: do not return a failure if a peer cannot be notified of update

Maintenance ops that fail to notify a peer of the update should not
report a failure to the original user.

Fixes: http://tracker.ceph.com/issues/15351
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'jewel-15402' of git://github.com/ukernel/ceph into greg-fs-testing

Do not ignore standby_for_fscid when selecting a standby.

https://github.com/ceph/ceph/pull/8536

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Fixed ceph-common install.

Fixes: #15438
Signed-off-by: John Wilkins <jowilkin@redhat.com>

librbd: fixed bug in disabling non-primary image mirroring

librbd: fixed bug in disabling non-primary image mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8496 from athanatos/wip-15312

os/filestore: force lfn attrs to be written atomically, restructure name length limits

mds: take standby_for_fscid into account in FSMap::find_unused()

Fixes: http://tracker.ceph.com/issues/15402
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #8529 from oritwas/wip-rgw-fix-zone-set

radosgw-admin: allow

librbd: Fixed bug in disabling non-primary image mirroring

It is only possible to remove image from rbd_mirroring object if the
mirroring image state is set to DISABLING.

Signed-off-by: Ricardo Dias <rdias@suse.com>

rgw-rados: return RGWSystemMetaObj init directly

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #8508 from JiYou/fix-ceph-disk

ceph-disk: do not always fail when re-using a partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/jewel'

ceph-disk: fix PrepareData.set_type() args should be class member.

Signed-off-by: You Ji <youji@ebay.com>

ceph-disk: fix spelling mistake geattr to getattr.

Signed-off-by: You Ji <youji@ebay.com>

ceph-disk: fix lockbox set_or_create_partition()

Signed-off-by: You Ji <youji@ebay.com>

Merge pull request #8507 from kmroz/wip-rgw-admin-cleanup

rgw-admin: remove unused iterator and fix error message

radosgw-admin: fix name checking

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: allow setting zone when there is no realm

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: pass RULE_LAUNCHER_* to cython

* pass the environment variables using `env` to pass environment
  varibles with space(s) in them to the COMMAND in add_custom_target.
  otherwise, cmake will try to quote the space with "\". this breaks the
  generated command line.
* add a comment for ccache to note that we do not expect ccache to speed
  up linking. we use it as the linker's launcher to workaround
  https://bugs.python.org/issue8027. to be specific,
  distutils.UnixCCompiler.link overwrites the first linker CLI's arg
  using the the first C++ compiler's CLI arg, if "env" is not used to
  launch the linker. this breaks the cythonization of our pybind APIs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8522 from swamireddy/10.1.1-mailmap

mailmap update

Reviewed-by: Loic Dachary <ldachary@redhat.com>

AsyncConnection: avoid is_connected require connection's lock

Fixes: http://tracker.ceph.com/issues/15440
Signed-off-by: Haomai Wang <haomai@xsky.com>

check-generated.sh: can't source bash from sh

'source' is not sh command and executing bash commands in sh won't
work. We could try to replace it with:
'. ../qa/workunits/ceph-helpers.sh'
but the above file uses bash.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #8519 from dachary/wip-xenial

tests: add Ubuntu 16.04 xenial dockerfile

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: exclusive lock might be disabled while waiting to acquire lock

Disabling exclusive lock will now acquire the exclusive lock before
proceeding.  If this resulted in another client releasing the lock
and re-requesting it, support for exclusive locking might be disabled
while a request is pending.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>