man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7847 from wido/mirroring-fixes

Updated scripts and documentation for mirrors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add script to test connectivity and availability of Ceph mirrors

Signed-off-by: Wido den Hollander <wido@42on.com>

mirror: Add examle rsync daemon configuration

This can be used by mirrors to offer a rsync service to end-users

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: Updated docs with additional mirror information

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #7552 from dachary/wip-14669-key-management

ceph-disk: key management support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7995 from majianpeng/bluestore-dump-onode

osd: bluestore: allow _dump_onode dynamic accept log level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: Allow _dump_onode dynamic accept log level.

Make _dump_onode dynamic accept different log level in different case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7996 from majianpeng/bluestore-fullextent-overwrite

osd: bluestore: for overwrite a extent, allocate new extent on min_alloc_size write

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: For overwrite a extent, allocate new extent rather than WAL.

This bug introduce by commit:b0b4b6de362f0edf. This commit forget this
case which overwrite(0, bluestore_min_alloc_size).
For this case it  need a new extent rather than WAL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7960 from SUSE/wip-docdev-mon

doc/dev: continue writing Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: Continue writing Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7986 from liewegas/wip-xxhash

xxhash: use clone of xxhash.git; add .gitignore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xxHash: add .gitignore for build artifacts

Signed-off-by: Sage Weil <sage@redhat.com>

.gitmodules: use github.com/ceph/xxHash clone

This lets us add our own commits, if necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7966 from dachary/wip-release-notes

release-notes: draft v10.0.4 release notes

Merge pull request #7880 from SUSE/wip_keyring_permisions_for_mon_deamon_14950

mon: fix keyring permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7121 from efirs/ef_eversion_t_no_sprintf

osd: Replace snprintf with faster implementation in eversion_t::get_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7141 from jecluis/wip-mon-clockskew

mon: make clock skew checks sane

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7899 from liewegas/wip-osd-markdown

test/osd: Relax the timing intervals in osd-markdown.sh

Merge pull request #7900 from liewegas/wip-bluefs-alignment

osd: bluefs: fix alignment for odd page sizes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7904 from liewegas/wip-log-delete

log: use delete[]

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7742 from ErwanAliasr1/evelu-spec

Adding build requires

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts: ceph-release-note rewrite PR title case

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.4 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7959 from kmroz/wip-add-rgw-to-stopsh

script: add missing stop_rgw variable to stop.sh script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'wip-conf-root' of https://github.com/liewegas/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/common/config.cc

Merge pull request #7914 from dillaman/wip-tests-template-specialization

test: correct librbd errors discovered with unoptimized cmake build

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: Moving Cython into distro specific area

To get the spec file into a consistent state, let's move Cython into the distro
specific are of the spec file.

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding redhat-rpm-config

Some part of the rados compilation process is expecting redhat-hardened-cc1.
That's surely because of the FORTIFY compilation flag.

redhat-hardened-cc1 is part of the redhat-rpm-config so let's put it as a build
dependency.

The typical output when this issue is occuring :

cd ./pybind/rados; CPPFLAGS="-iquote \/ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=\"/usr/local/lib\" -DCEPH_PKGLIBDIR=\"/usr/local/lib/ceph\" -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT    " CFLAGS="-iquote \/ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith  -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong  -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -g -O2" LDFLAGS="-L\/ceph/src/.libs -Wl,--as-needed  -Wl,-z,relro -Wl,-z,now  -latomic_ops " CYTHON_BUILD_DIR="/ceph/src/build" /bin/python ./setup.py build \
--build-base /ceph/src/build \
--verbose
Compiling rados.pyx because it changed.
Cythonizing rados.pyx
  CXXLD    libradosstriper.la
  CXXLD    librbd.la
  CXXLD    librgw.la
running build
running build_ext
building 'rados' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -iquote /ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -g -O2 -iquote /ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=/usr/local/lib -DCEPH_PKGLIBDIR=/usr/local/lib/ceph -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT -fPIC -I/usr/include/python2.7 -c /ceph/src/build/rados.c -o /ceph/src/build/temp.linux-x86_64-2.7/ceph/src/build/rados.o
gcc: erreur: /usr/lib/rpm/redhat/redhat-hardened-cc1: Aucun fichier ou dossier de ce type
error: command 'gcc' failed with exit status 1

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding btrfs build require

When doing a 'make check', 'qa/workunits/ceph-helpers.sh' does expect the 'btrfs'
command to be available. If not, that make many tests failing.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7628 from liewegas/wip-journal-block-size

os/filestore/FileJournal: set block size via config option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7705 from renhwsky/renhw-wip-pgmonitor-get_rule_avail

mon/pgmonitor: use appropriate forced conversions in get_rule_avail

Merge pull request #7775 from branch-predictor/bp-bl-eradication

os/ObjectStore: implement more efficient get_encoded_bytes()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7845 from aiicore/pthread_setname_fix

common: set thread name from correct thread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7850 from jupiturliu/kstore-latency

osd: kstore: latency breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7861 from tchaikov/wip-14910

qa/workunits/cephtool/test.sh: wait longer in ceph_watch_start()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7863 from branch-predictor/bp-faster-name-gen

common/obj_bencher.cc: faster object name generation

Reviewed-by: Sage Weil <sage@redhat.com>

stop: Add missing stop_rgw variable

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7954 from SUSE/wip-docdev

doc/dev: various refinements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: start Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev: various refinements

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7951 from badone/wip-cls_hello-grammar

cls_hello: Fix grammatical error in description comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Remove duplicate find_package libcurl line.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7946 from SUSE/wip-doc-dev-teuth

doc/dev: integrate testing into the narrative

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cls_hello: Fix grammatical error in description comment

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7862 from theanalyst/rgw/doc-insecure-keystone

doc: rgw explain keystone's verify ssl switch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7887 from cbodley/wip-rgw-test-multi-period

test/rgw: add multisite test for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7827 from cbodley/wip-rgw-period-meta-logs

rgw: fixes for per-period metadata logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

rgw: fix for RGWCoroutinesManager::run() return value

the 'int ret' variable of the inner scope was shadowing an 'int ret'
variable in the outer scope, so we weren't propagating any of the error
codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use current period for InitSyncStatus

the InitSyncStatus coroutine records the position to start incremental
sync after finishing a full sync.  this should be the master's marker
from the current period, rather than its oldest log period

this also adds a check to run_sync() that restarts a full sync if it
sees that our sync period is behind the master's oldest log period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: meta log rest handlers avoid get_log()

RGWMetadataManager::get_log() will allocate a log and keep it in memory.
this could lead to a potential denial of service by making requests with
lots of different period ids

RGWMetadataLog if effectively stateless (the only state is a set of
modified_shards, which are not touched by any of the rest api calls), so
we can use a temporary instead of calling get_log()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pull first log period from master if not found locally

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: implement find_oldest_log_period for RGWMetadataManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period history behaves with no current_period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: initialize async_rados before meta_mgr

RGWMetadataManager::init() will depend on async_rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWStatObjCR for async raw_obj_stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: only write meta sync shard markers if can_adjust_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetaSyncCR wakeups thread-safe

now that the shards will be coming and going after startup, they need to
be reference counted and protected by a mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: add read_current_period_id() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #7918 from ceph/wip-rgw-payer-yehuda

rgw: link payer info to usage logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: add ceph-detect-init(8) source to dist tarball

Prior to this commit, the tarball from "make dist" did not include the
ceph-detect-init(8) man page rST source.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7929 from yehudasa/wip-submodules-fix

submodules: revert an accidental change

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7910 from ceph/wip-14955

buffer: increment history alloc as well in raw_combined

Reviewed-by: Sage Weil <sage@redhat.com>

submodules: revert an accidental change

revert changes that went in in commit f1ac0de5bfd974258c257470feae5ff03054873e.
This affects ceph-erasure-code-corpus, civetweb

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7916 from ceph/wip-14975

rgw: adjust error code when bucket does not exist in copy operation
rgw: don't override error when initializing zonegroup

Fixes: #14975
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-disk: protect destroy with the activate lock

So that it does not race against an activation.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: map after luks formatting

dmcrypt_map is idempotent, it does not matter that it is called multiple
times when activating or preparing a device. It is however important to
not cryptsetup remove the device after preparing it because partprobe
may not re-send another add udev event to luksOpen / create it again.

On Ubuntu 14.04, no extra add udev event is sent after the partition was
acknowledged by the kernel. On CentOS 7.2 an extra udev event is sent
for each existing partition, regardless of what the kernel already
knows.

The format() function is just a call to map() to limit the extent of the
refactor. The dmcrypt_map/unmap functions should be refactored into a
device class and the format() function can then be made more specific.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: verify ceph-disk lockbox activation scenarii

Simulate the cases where the activation (via udev running trigger)
sequences are:

* journal then lockbox
* data then lockbox
* lockbox

All of them must end with the OSD verfied to be up.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: improve trigger verbosity

The ceph-disk activate errors were ignored and not displayed. Capture
stdout/stderr and display it if the exit code is non zero. Also fail
when an activate fails.

Pass the --verbose flag to activate, if given to trigger.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: make ceph-disk workunit resilient to non ascii

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: implement deactivate --once

It is mostly useful for test purposes.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: verify ceph-disk activate-lockbox

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: protect list with activate lock

list may try to mount partitions to figure out the OSD id and other
details. If it does so while the OSD is activated, it will race and lead
to errors, either for activation or for list.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: implement list for lockbox

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: update ceph-disk to refer to ceph-disk --help

The ceph-disk page is often obsolete, mostly because maintaining
it requires a significant amount of copy/paste and re-formatting.
Now that the --help of ceph-disk has been updated to include a more
verbose explanation of each subcommand, simplify the man page to
give an overview of the subcommands and suggest the --help is used
to get more information.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: add a description for each subcommand

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: implement lockbox key management

Instead of storing the dmcrypt keys in the /etc/ceph/dmcrypt-keys
directory, they are stored in the monitor. If a machine with
OSDs created with ceph-disk prepare --dmcrypt is lost, it does
not contain the key that would allow to decrypt their content.

The dmcrypt key is retrieved from the monitor using a different keyring
for each OSD. It is stored in a small partition called the lockbox. At
boot time the lockbox is mounted

    /var/lib/ceph/osd-lockbox/$uuid

and used when the $uuid partition is detected by udev to map it with
cryptsetup.

The OSDs that were prepared prior to the lockbox implementation are
supported by looking up the key found in /etc/ceph/dmcrypt-keys before
looking in /var/lib/ceph/osd-lockbox/$uuid.

http://tracker.ceph.com/issues/14669 Fixes: #14669

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: simplify trigger

The ceph-disk trigger deals with dmcrypt mapping which is redundant with
what ceph-disk activate-* does when the --dmcrypt flag is set. Remove
the dmcrypt mapping code and add the --dmcrypt flag to ceph-disk
activate-* where relevant.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: re-using an OSD partition bugous test

When re-using an OSD partition, the test of the partition type is done
against a non-existent variable.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: make dmcrypt_unmap idempotent

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: destroy does not unmap block/journal

The dmcrypt boolean needs to be set for destroy to ensure the
journal/block partitions are 'cryptsetup remove'.

Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: display the function name with the log messages

Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: avoid showing payer when payer == owner

If the payer matches the owner it shows the owner only.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #7786 from ceph/wip-rgw-indexless

rgw: indexless buckets (Yehuda Sadeh)

 - can define a policy, for which buckets are indexless
 - users can then create buckets under the specified placement target
 - indexless buckets will not be synced across zones
 - does not work with (s3) versioned buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: indexless buckets

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: configurable index type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't override error when initializing zonegroup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>