Merge pull request #7884 from dillaman/wip-14663

librbd: allocate new journal tag after acquiring exclusive lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7772 from xinxinsh/wip-14789

librbd: truncate does not need to mark the object as existing in the object map

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8003 from liewegas/wip-ceph-disk-doc

man/8/ceph-disk: fix formatting issue

Merge pull request #7927 from oritwas/wip-rgw-fix-default

rgw: RGWZoneParams::create should not handle -EEXIST error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7768 from xinxinsh/wip-fix-zero

osd: filestore: FALLOC_FL_PUNCH_HOLE must be used with FALLOC_FL_KEEP_SIZE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7844 from xiexingguo/xxg-wip-14922

osd: replicatedpg: break out loop if we encounter fatal error during do_pg_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7864 from dillaman/wip-librados-aio-notify

librados: race condition on aio_notify completion handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7928 from branch-predictor/bp-radostool-reduce-mem

tools/rados: reduce "rados put" memory usage by op_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7967 from tchaikov/wip-si-cast-coverity

common/strtol.cc: fix the coverity warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7948 from avnerbh/fix_avoid_enqueue_dup_code

msg: remove duplicated code - local_delivery will now call 'enqueue'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7890 from liewegas/wip-reweight

mon: osd [test-]reweight-by-{pg,utilization} command updates

Reviewed-by: Samuel Just <sjust@redhat.com>

mon: remove range=100 from reweight-by-* commands

Specifying the range makes it skip instead of error when you give
it a bad value.

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7847 from wido/mirroring-fixes

Updated scripts and documentation for mirrors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add script to test connectivity and availability of Ceph mirrors

Signed-off-by: Wido den Hollander <wido@42on.com>

mirror: Add examle rsync daemon configuration

This can be used by mirrors to offer a rsync service to end-users

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: Updated docs with additional mirror information

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #7552 from dachary/wip-14669-key-management

ceph-disk: key management support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7995 from majianpeng/bluestore-dump-onode

osd: bluestore: allow _dump_onode dynamic accept log level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: Allow _dump_onode dynamic accept log level.

Make _dump_onode dynamic accept different log level in different case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7996 from majianpeng/bluestore-fullextent-overwrite

osd: bluestore: for overwrite a extent, allocate new extent on min_alloc_size write

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: For overwrite a extent, allocate new extent rather than WAL.

This bug introduce by commit:b0b4b6de362f0edf. This commit forget this
case which overwrite(0, bluestore_min_alloc_size).
For this case it  need a new extent rather than WAL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7960 from SUSE/wip-docdev-mon

doc/dev: continue writing Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: Continue writing Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7986 from liewegas/wip-xxhash

xxhash: use clone of xxhash.git; add .gitignore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xxHash: add .gitignore for build artifacts

Signed-off-by: Sage Weil <sage@redhat.com>

.gitmodules: use github.com/ceph/xxHash clone

This lets us add our own commits, if necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7966 from dachary/wip-release-notes

release-notes: draft v10.0.4 release notes

Merge pull request #7880 from SUSE/wip_keyring_permisions_for_mon_deamon_14950

mon: fix keyring permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7121 from efirs/ef_eversion_t_no_sprintf

osd: Replace snprintf with faster implementation in eversion_t::get_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7141 from jecluis/wip-mon-clockskew

mon: make clock skew checks sane

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7899 from liewegas/wip-osd-markdown

test/osd: Relax the timing intervals in osd-markdown.sh

Merge pull request #7900 from liewegas/wip-bluefs-alignment

osd: bluefs: fix alignment for odd page sizes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7904 from liewegas/wip-log-delete

log: use delete[]

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

journal: possible race condition during fetch playback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: fix errors introduced by rebase to master

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: clean up playback notification handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated test cases for librbd journal tag allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: properly handle tag transition

Now that the tag concept has been re-used for delineating epochs for
librbd, we need playback to properly handle the cases where the active
playback tag abruptly ends and a newer tag is inserted in the first
splay offset object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: additional debug messages surrounding commit updates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improved debugging for journal client / tag processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate new journal tag when acquiring exclusive lock

The journal tag tracks a write epoch within the image, so before
allowing writes to proceed, ensure the journal entries will be
stored under a new epoch.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add tag handling to journal state machine

The journal will not retrieve the tag class for the image within
the journal in addition to the most recently allocated tag. Also
added helper methods to allocate new tags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to retrieve in-memory client data

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: local pool id no longer stored in client registration

The remote pool name should match the remote pool name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update journal client and tag data structures

The mirror peer now holds remote's image id as well as a collection
of bootstrap sync points.  The tag now contains uuids to map back
to the owning pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate new uuid when enabling mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: support for uuid to represent a mirrored pool

This uuid will be used as the client id for remote peer image
journals.  This will allow the uuid to be also utilized within
the librbd journal tag for tracking epoch ownership.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7742 from ErwanAliasr1/evelu-spec

Adding build requires

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

common/strtol.cc: fix the coverity warnings

* promote the compared types properly to address the signed/unsigned
  comparison warnings. this also fixes the potential problems of
  slicing a compared type down to a "smaller" type before the
  comparison.
* check for the width of resulting type and shift bits caused by SI
  prefix to avoid the -Wshift-count-overflow warnings. this again
  is a potential issue, as shifting n bits of an integer of m bits
  width, where n >= m, leads to undefined behaviour.
* add a test for the 2nd fixed issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: skip truncate request if object does not exist

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

scripts: ceph-release-note rewrite PR title case

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.4 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7959 from kmroz/wip-add-rgw-to-stopsh

script: add missing stop_rgw variable to stop.sh script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'wip-conf-root' of https://github.com/liewegas/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/common/config.cc

Merge pull request #7914 from dillaman/wip-tests-template-specialization

test: correct librbd errors discovered with unoptimized cmake build

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: Moving Cython into distro specific area

To get the spec file into a consistent state, let's move Cython into the distro
specific are of the spec file.

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding redhat-rpm-config

Some part of the rados compilation process is expecting redhat-hardened-cc1.
That's surely because of the FORTIFY compilation flag.

redhat-hardened-cc1 is part of the redhat-rpm-config so let's put it as a build
dependency.

The typical output when this issue is occuring :

cd ./pybind/rados; CPPFLAGS="-iquote \/ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=\"/usr/local/lib\" -DCEPH_PKGLIBDIR=\"/usr/local/lib/ceph\" -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT    " CFLAGS="-iquote \/ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith  -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong  -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -g -O2" LDFLAGS="-L\/ceph/src/.libs -Wl,--as-needed  -Wl,-z,relro -Wl,-z,now  -latomic_ops " CYTHON_BUILD_DIR="/ceph/src/build" /bin/python ./setup.py build \
--build-base /ceph/src/build \
--verbose
Compiling rados.pyx because it changed.
Cythonizing rados.pyx
  CXXLD    libradosstriper.la
  CXXLD    librbd.la
  CXXLD    librgw.la
running build
running build_ext
building 'rados' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -iquote /ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -g -O2 -iquote /ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=/usr/local/lib -DCEPH_PKGLIBDIR=/usr/local/lib/ceph -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT -fPIC -I/usr/include/python2.7 -c /ceph/src/build/rados.c -o /ceph/src/build/temp.linux-x86_64-2.7/ceph/src/build/rados.o
gcc: erreur: /usr/lib/rpm/redhat/redhat-hardened-cc1: Aucun fichier ou dossier de ce type
error: command 'gcc' failed with exit status 1

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding btrfs build require

When doing a 'make check', 'qa/workunits/ceph-helpers.sh' does expect the 'btrfs'
command to be available. If not, that make many tests failing.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7628 from liewegas/wip-journal-block-size

os/filestore/FileJournal: set block size via config option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7705 from renhwsky/renhw-wip-pgmonitor-get_rule_avail

mon/pgmonitor: use appropriate forced conversions in get_rule_avail

Merge pull request #7775 from branch-predictor/bp-bl-eradication

os/ObjectStore: implement more efficient get_encoded_bytes()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7845 from aiicore/pthread_setname_fix

common: set thread name from correct thread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7850 from jupiturliu/kstore-latency

osd: kstore: latency breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7861 from tchaikov/wip-14910

qa/workunits/cephtool/test.sh: wait longer in ceph_watch_start()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7863 from branch-predictor/bp-faster-name-gen

common/obj_bencher.cc: faster object name generation

Reviewed-by: Sage Weil <sage@redhat.com>

stop: Add missing stop_rgw variable

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7954 from SUSE/wip-docdev

doc/dev: various refinements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: start Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev: various refinements

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7951 from badone/wip-cls_hello-grammar

cls_hello: Fix grammatical error in description comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rados: reduce rados put memory usage by op_size

indata.append(char*, len) causes buffer "buf" to be physically copied
to new memory block. Make static buffer pointing to buf so we don't need
any intermediate copies. This reduces memory usage by op_size, and also
cpu usage by a bit (no memmove necessary).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

cmake: Remove duplicate find_package libcurl line.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7946 from SUSE/wip-doc-dev-teuth

doc/dev: integrate testing into the narrative

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cls_hello: Fix grammatical error in description comment

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

msg: remove duplicated code - local_delivery will now call 'enqueue'

This simplify the code and clarify that for each local message,
 'local_delivery_thread' simply tries to 'fast_dispatch' it,
 or it will 'enqueue' it for the dispatch_thread (like any standard message)

Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #7862 from theanalyst/rgw/doc-insecure-keystone

doc: rgw explain keystone's verify ssl switch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7887 from cbodley/wip-rgw-test-multi-period

test/rgw: add multisite test for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7827 from cbodley/wip-rgw-period-meta-logs

rgw: fixes for per-period metadata logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7938 from dmick/jewel

debian/changelog: Remove stray 'v' in version

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 0f7730c8f444657d4aaffa28c12bbcd602a2e3b8)

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

rgw: fix for RGWCoroutinesManager::run() return value

the 'int ret' variable of the inner scope was shadowing an 'int ret'
variable in the outer scope, so we weren't propagating any of the error
codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use current period for InitSyncStatus

the InitSyncStatus coroutine records the position to start incremental
sync after finishing a full sync.  this should be the master's marker
from the current period, rather than its oldest log period

this also adds a check to run_sync() that restarts a full sync if it
sees that our sync period is behind the master's oldest log period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: meta log rest handlers avoid get_log()

RGWMetadataManager::get_log() will allocate a log and keep it in memory.
this could lead to a potential denial of service by making requests with
lots of different period ids

RGWMetadataLog if effectively stateless (the only state is a set of
modified_shards, which are not touched by any of the rest api calls), so
we can use a temporary instead of calling get_log()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pull first log period from master if not found locally

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: implement find_oldest_log_period for RGWMetadataManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period history behaves with no current_period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: initialize async_rados before meta_mgr

RGWMetadataManager::init() will depend on async_rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>