doc: Add French mirror

Signed-off-by: Wido den Hollander <wido@42on.com>

mirrors: Change contact e-mail address for se.ceph.com

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #8003 from liewegas/wip-ceph-disk-doc

man/8/ceph-disk: fix formatting issue

Merge pull request #7927 from oritwas/wip-rgw-fix-default

rgw: RGWZoneParams::create should not handle -EEXIST error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7768 from xinxinsh/wip-fix-zero

osd: filestore: FALLOC_FL_PUNCH_HOLE must be used with FALLOC_FL_KEEP_SIZE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7844 from xiexingguo/xxg-wip-14922

osd: replicatedpg: break out loop if we encounter fatal error during do_pg_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7864 from dillaman/wip-librados-aio-notify

librados: race condition on aio_notify completion handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7928 from branch-predictor/bp-radostool-reduce-mem

tools/rados: reduce "rados put" memory usage by op_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7967 from tchaikov/wip-si-cast-coverity

common/strtol.cc: fix the coverity warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7948 from avnerbh/fix_avoid_enqueue_dup_code

msg: remove duplicated code - local_delivery will now call 'enqueue'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7890 from liewegas/wip-reweight

mon: osd [test-]reweight-by-{pg,utilization} command updates

Reviewed-by: Samuel Just <sjust@redhat.com>

mon: remove range=100 from reweight-by-* commands

Specifying the range makes it skip instead of error when you give
it a bad value.

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7847 from wido/mirroring-fixes

Updated scripts and documentation for mirrors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add script to test connectivity and availability of Ceph mirrors

Signed-off-by: Wido den Hollander <wido@42on.com>

mirror: Add examle rsync daemon configuration

This can be used by mirrors to offer a rsync service to end-users

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: Updated docs with additional mirror information

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #7552 from dachary/wip-14669-key-management

ceph-disk: key management support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7995 from majianpeng/bluestore-dump-onode

osd: bluestore: allow _dump_onode dynamic accept log level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: Allow _dump_onode dynamic accept log level.

Make _dump_onode dynamic accept different log level in different case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7996 from majianpeng/bluestore-fullextent-overwrite

osd: bluestore: for overwrite a extent, allocate new extent on min_alloc_size write

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: For overwrite a extent, allocate new extent rather than WAL.

This bug introduce by commit:b0b4b6de362f0edf. This commit forget this
case which overwrite(0, bluestore_min_alloc_size).
For this case it  need a new extent rather than WAL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7960 from SUSE/wip-docdev-mon

doc/dev: continue writing Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: Continue writing Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7986 from liewegas/wip-xxhash

xxhash: use clone of xxhash.git; add .gitignore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xxHash: add .gitignore for build artifacts

Signed-off-by: Sage Weil <sage@redhat.com>

.gitmodules: use github.com/ceph/xxHash clone

This lets us add our own commits, if necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7966 from dachary/wip-release-notes

release-notes: draft v10.0.4 release notes

Merge pull request #7880 from SUSE/wip_keyring_permisions_for_mon_deamon_14950

mon: fix keyring permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7121 from efirs/ef_eversion_t_no_sprintf

osd: Replace snprintf with faster implementation in eversion_t::get_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7141 from jecluis/wip-mon-clockskew

mon: make clock skew checks sane

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7899 from liewegas/wip-osd-markdown

test/osd: Relax the timing intervals in osd-markdown.sh

Merge pull request #7900 from liewegas/wip-bluefs-alignment

osd: bluefs: fix alignment for odd page sizes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7904 from liewegas/wip-log-delete

log: use delete[]

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7742 from ErwanAliasr1/evelu-spec

Adding build requires

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

common/strtol.cc: fix the coverity warnings

* promote the compared types properly to address the signed/unsigned
  comparison warnings. this also fixes the potential problems of
  slicing a compared type down to a "smaller" type before the
  comparison.
* check for the width of resulting type and shift bits caused by SI
  prefix to avoid the -Wshift-count-overflow warnings. this again
  is a potential issue, as shifting n bits of an integer of m bits
  width, where n >= m, leads to undefined behaviour.
* add a test for the 2nd fixed issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts: ceph-release-note rewrite PR title case

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.4 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7959 from kmroz/wip-add-rgw-to-stopsh

script: add missing stop_rgw variable to stop.sh script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'wip-conf-root' of https://github.com/liewegas/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/common/config.cc

Merge pull request #7914 from dillaman/wip-tests-template-specialization

test: correct librbd errors discovered with unoptimized cmake build

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: Moving Cython into distro specific area

To get the spec file into a consistent state, let's move Cython into the distro
specific are of the spec file.

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding redhat-rpm-config

Some part of the rados compilation process is expecting redhat-hardened-cc1.
That's surely because of the FORTIFY compilation flag.

redhat-hardened-cc1 is part of the redhat-rpm-config so let's put it as a build
dependency.

The typical output when this issue is occuring :

cd ./pybind/rados; CPPFLAGS="-iquote \/ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=\"/usr/local/lib\" -DCEPH_PKGLIBDIR=\"/usr/local/lib/ceph\" -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT    " CFLAGS="-iquote \/ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith  -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong  -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -g -O2" LDFLAGS="-L\/ceph/src/.libs -Wl,--as-needed  -Wl,-z,relro -Wl,-z,now  -latomic_ops " CYTHON_BUILD_DIR="/ceph/src/build" /bin/python ./setup.py build \
--build-base /ceph/src/build \
--verbose
Compiling rados.pyx because it changed.
Cythonizing rados.pyx
  CXXLD    libradosstriper.la
  CXXLD    librbd.la
  CXXLD    librgw.la
running build
running build_ext
building 'rados' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -iquote /ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -g -O2 -iquote /ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=/usr/local/lib -DCEPH_PKGLIBDIR=/usr/local/lib/ceph -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT -fPIC -I/usr/include/python2.7 -c /ceph/src/build/rados.c -o /ceph/src/build/temp.linux-x86_64-2.7/ceph/src/build/rados.o
gcc: erreur: /usr/lib/rpm/redhat/redhat-hardened-cc1: Aucun fichier ou dossier de ce type
error: command 'gcc' failed with exit status 1

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding btrfs build require

When doing a 'make check', 'qa/workunits/ceph-helpers.sh' does expect the 'btrfs'
command to be available. If not, that make many tests failing.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7628 from liewegas/wip-journal-block-size

os/filestore/FileJournal: set block size via config option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7705 from renhwsky/renhw-wip-pgmonitor-get_rule_avail

mon/pgmonitor: use appropriate forced conversions in get_rule_avail

Merge pull request #7775 from branch-predictor/bp-bl-eradication

os/ObjectStore: implement more efficient get_encoded_bytes()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7845 from aiicore/pthread_setname_fix

common: set thread name from correct thread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7850 from jupiturliu/kstore-latency

osd: kstore: latency breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7861 from tchaikov/wip-14910

qa/workunits/cephtool/test.sh: wait longer in ceph_watch_start()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7863 from branch-predictor/bp-faster-name-gen

common/obj_bencher.cc: faster object name generation

Reviewed-by: Sage Weil <sage@redhat.com>

stop: Add missing stop_rgw variable

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7954 from SUSE/wip-docdev

doc/dev: various refinements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: start Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev: various refinements

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7951 from badone/wip-cls_hello-grammar

cls_hello: Fix grammatical error in description comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rados: reduce rados put memory usage by op_size

indata.append(char*, len) causes buffer "buf" to be physically copied
to new memory block. Make static buffer pointing to buf so we don't need
any intermediate copies. This reduces memory usage by op_size, and also
cpu usage by a bit (no memmove necessary).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

cmake: Remove duplicate find_package libcurl line.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7946 from SUSE/wip-doc-dev-teuth

doc/dev: integrate testing into the narrative

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cls_hello: Fix grammatical error in description comment

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

msg: remove duplicated code - local_delivery will now call 'enqueue'

This simplify the code and clarify that for each local message,
 'local_delivery_thread' simply tries to 'fast_dispatch' it,
 or it will 'enqueue' it for the dispatch_thread (like any standard message)

Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #7862 from theanalyst/rgw/doc-insecure-keystone

doc: rgw explain keystone's verify ssl switch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7887 from cbodley/wip-rgw-test-multi-period

test/rgw: add multisite test for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7827 from cbodley/wip-rgw-period-meta-logs

rgw: fixes for per-period metadata logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7938 from dmick/jewel

debian/changelog: Remove stray 'v' in version

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 0f7730c8f444657d4aaffa28c12bbcd602a2e3b8)

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

rgw: fix for RGWCoroutinesManager::run() return value

the 'int ret' variable of the inner scope was shadowing an 'int ret'
variable in the outer scope, so we weren't propagating any of the error
codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use current period for InitSyncStatus

the InitSyncStatus coroutine records the position to start incremental
sync after finishing a full sync.  this should be the master's marker
from the current period, rather than its oldest log period

this also adds a check to run_sync() that restarts a full sync if it
sees that our sync period is behind the master's oldest log period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: meta log rest handlers avoid get_log()

RGWMetadataManager::get_log() will allocate a log and keep it in memory.
this could lead to a potential denial of service by making requests with
lots of different period ids

RGWMetadataLog if effectively stateless (the only state is a set of
modified_shards, which are not touched by any of the rest api calls), so
we can use a temporary instead of calling get_log()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pull first log period from master if not found locally

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: implement find_oldest_log_period for RGWMetadataManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period history behaves with no current_period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: initialize async_rados before meta_mgr

RGWMetadataManager::init() will depend on async_rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWStatObjCR for async raw_obj_stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: only write meta sync shard markers if can_adjust_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetaSyncCR wakeups thread-safe

now that the shards will be coming and going after startup, they need to
be reference counted and protected by a mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: add read_current_period_id() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #7918 from ceph/wip-rgw-payer-yehuda

rgw: link payer info to usage logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: add ceph-detect-init(8) source to dist tarball

Prior to this commit, the tarball from "make dist" did not include the
ceph-detect-init(8) man page rST source.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7929 from yehudasa/wip-submodules-fix

submodules: revert an accidental change

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7910 from ceph/wip-14955

buffer: increment history alloc as well in raw_combined

Reviewed-by: Sage Weil <sage@redhat.com>

submodules: revert an accidental change

revert changes that went in in commit f1ac0de5bfd974258c257470feae5ff03054873e.
This affects ceph-erasure-code-corpus, civetweb

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWZoneParams::create should not handle -EEXIST error

It should be handled by the caller

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7916 from ceph/wip-14975

rgw: adjust error code when bucket does not exist in copy operation
rgw: don't override error when initializing zonegroup

Fixes: #14975
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-disk: protect destroy with the activate lock

So that it does not race against an activation.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: map after luks formatting

dmcrypt_map is idempotent, it does not matter that it is called multiple
times when activating or preparing a device. It is however important to
not cryptsetup remove the device after preparing it because partprobe
may not re-send another add udev event to luksOpen / create it again.

On Ubuntu 14.04, no extra add udev event is sent after the partition was
acknowledged by the kernel. On CentOS 7.2 an extra udev event is sent
for each existing partition, regardless of what the kernel already
knows.

The format() function is just a call to map() to limit the extent of the
refactor. The dmcrypt_map/unmap functions should be refactored into a
device class and the format() function can then be made more specific.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: verify ceph-disk lockbox activation scenarii

Simulate the cases where the activation (via udev running trigger)
sequences are:

* journal then lockbox
* data then lockbox
* lockbox

All of them must end with the OSD verfied to be up.

Signed-off-by: Loic Dachary <loic@dachary.org>