mon/PGMonitor: make pg refresh debug output more informative

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: drop unused nocreating flag for stat_pg_{add,sub}

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop create events based on same_primary_since, not interval

Creates are resent when the primary changes, not the interval.
Ensure we don't drop something we shouldn't.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop unused primary arg to handle_pg_peering_evt

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use handle_pg_peering_evt for pg creations

This ensures we captures all of the PG resurrection logic.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop unused from arg from handle_pg_peering_evt

Signed-off-by: Sage Weil <sage@redhat.com>

osd: only pass history to handle_pg_peering_evt

We were passing the whole pg_info, but we only need the
history.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8003 from liewegas/wip-ceph-disk-doc

man/8/ceph-disk: fix formatting issue

Merge pull request #7927 from oritwas/wip-rgw-fix-default

rgw: RGWZoneParams::create should not handle -EEXIST error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7768 from xinxinsh/wip-fix-zero

osd: filestore: FALLOC_FL_PUNCH_HOLE must be used with FALLOC_FL_KEEP_SIZE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7844 from xiexingguo/xxg-wip-14922

osd: replicatedpg: break out loop if we encounter fatal error during do_pg_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7864 from dillaman/wip-librados-aio-notify

librados: race condition on aio_notify completion handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7928 from branch-predictor/bp-radostool-reduce-mem

tools/rados: reduce "rados put" memory usage by op_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7967 from tchaikov/wip-si-cast-coverity

common/strtol.cc: fix the coverity warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7948 from avnerbh/fix_avoid_enqueue_dup_code

msg: remove duplicated code - local_delivery will now call 'enqueue'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7890 from liewegas/wip-reweight

mon: osd [test-]reweight-by-{pg,utilization} command updates

Reviewed-by: Samuel Just <sjust@redhat.com>

mon: remove range=100 from reweight-by-* commands

Specifying the range makes it skip instead of error when you give
it a bad value.

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7847 from wido/mirroring-fixes

Updated scripts and documentation for mirrors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add script to test connectivity and availability of Ceph mirrors

Signed-off-by: Wido den Hollander <wido@42on.com>

mirror: Add examle rsync daemon configuration

This can be used by mirrors to offer a rsync service to end-users

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: Updated docs with additional mirror information

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #7552 from dachary/wip-14669-key-management

ceph-disk: key management support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7995 from majianpeng/bluestore-dump-onode

osd: bluestore: allow _dump_onode dynamic accept log level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: Allow _dump_onode dynamic accept log level.

Make _dump_onode dynamic accept different log level in different case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7996 from majianpeng/bluestore-fullextent-overwrite

osd: bluestore: for overwrite a extent, allocate new extent on min_alloc_size write

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: For overwrite a extent, allocate new extent rather than WAL.

This bug introduce by commit:b0b4b6de362f0edf. This commit forget this
case which overwrite(0, bluestore_min_alloc_size).
For this case it  need a new extent rather than WAL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7960 from SUSE/wip-docdev-mon

doc/dev: continue writing Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: Continue writing Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7986 from liewegas/wip-xxhash

xxhash: use clone of xxhash.git; add .gitignore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xxHash: add .gitignore for build artifacts

Signed-off-by: Sage Weil <sage@redhat.com>

.gitmodules: use github.com/ceph/xxHash clone

This lets us add our own commits, if necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7966 from dachary/wip-release-notes

release-notes: draft v10.0.4 release notes

Merge pull request #7880 from SUSE/wip_keyring_permisions_for_mon_deamon_14950

mon: fix keyring permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7121 from efirs/ef_eversion_t_no_sprintf

osd: Replace snprintf with faster implementation in eversion_t::get_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7141 from jecluis/wip-mon-clockskew

mon: make clock skew checks sane

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7899 from liewegas/wip-osd-markdown

test/osd: Relax the timing intervals in osd-markdown.sh

Merge pull request #7900 from liewegas/wip-bluefs-alignment

osd: bluefs: fix alignment for odd page sizes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7904 from liewegas/wip-log-delete

log: use delete[]

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7742 from ErwanAliasr1/evelu-spec

Adding build requires

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

common/strtol.cc: fix the coverity warnings

* promote the compared types properly to address the signed/unsigned
  comparison warnings. this also fixes the potential problems of
  slicing a compared type down to a "smaller" type before the
  comparison.
* check for the width of resulting type and shift bits caused by SI
  prefix to avoid the -Wshift-count-overflow warnings. this again
  is a potential issue, as shifting n bits of an integer of m bits
  width, where n >= m, leads to undefined behaviour.
* add a test for the 2nd fixed issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

scripts: ceph-release-note rewrite PR title case

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.4 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7959 from kmroz/wip-add-rgw-to-stopsh

script: add missing stop_rgw variable to stop.sh script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'wip-conf-root' of https://github.com/liewegas/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/common/config.cc

Merge pull request #7914 from dillaman/wip-tests-template-specialization

test: correct librbd errors discovered with unoptimized cmake build

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: Moving Cython into distro specific area

To get the spec file into a consistent state, let's move Cython into the distro
specific are of the spec file.

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding redhat-rpm-config

Some part of the rados compilation process is expecting redhat-hardened-cc1.
That's surely because of the FORTIFY compilation flag.

redhat-hardened-cc1 is part of the redhat-rpm-config so let's put it as a build
dependency.

The typical output when this issue is occuring :

cd ./pybind/rados; CPPFLAGS="-iquote \/ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=\"/usr/local/lib\" -DCEPH_PKGLIBDIR=\"/usr/local/lib/ceph\" -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT    " CFLAGS="-iquote \/ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith  -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong  -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -g -O2" LDFLAGS="-L\/ceph/src/.libs -Wl,--as-needed  -Wl,-z,relro -Wl,-z,now  -latomic_ops " CYTHON_BUILD_DIR="/ceph/src/build" /bin/python ./setup.py build \
--build-base /ceph/src/build \
--verbose
Compiling rados.pyx because it changed.
Cythonizing rados.pyx
  CXXLD    libradosstriper.la
  CXXLD    librbd.la
  CXXLD    librgw.la
running build
running build_ext
building 'rados' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -iquote /ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -g -O2 -iquote /ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=/usr/local/lib -DCEPH_PKGLIBDIR=/usr/local/lib/ceph -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT -fPIC -I/usr/include/python2.7 -c /ceph/src/build/rados.c -o /ceph/src/build/temp.linux-x86_64-2.7/ceph/src/build/rados.o
gcc: erreur: /usr/lib/rpm/redhat/redhat-hardened-cc1: Aucun fichier ou dossier de ce type
error: command 'gcc' failed with exit status 1

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding btrfs build require

When doing a 'make check', 'qa/workunits/ceph-helpers.sh' does expect the 'btrfs'
command to be available. If not, that make many tests failing.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7628 from liewegas/wip-journal-block-size

os/filestore/FileJournal: set block size via config option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7705 from renhwsky/renhw-wip-pgmonitor-get_rule_avail

mon/pgmonitor: use appropriate forced conversions in get_rule_avail

Merge pull request #7775 from branch-predictor/bp-bl-eradication

os/ObjectStore: implement more efficient get_encoded_bytes()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7845 from aiicore/pthread_setname_fix

common: set thread name from correct thread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7850 from jupiturliu/kstore-latency

osd: kstore: latency breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7861 from tchaikov/wip-14910

qa/workunits/cephtool/test.sh: wait longer in ceph_watch_start()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7863 from branch-predictor/bp-faster-name-gen

common/obj_bencher.cc: faster object name generation

Reviewed-by: Sage Weil <sage@redhat.com>

stop: Add missing stop_rgw variable

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7954 from SUSE/wip-docdev

doc/dev: various refinements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: start Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev: various refinements

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7951 from badone/wip-cls_hello-grammar

cls_hello: Fix grammatical error in description comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rados: reduce rados put memory usage by op_size

indata.append(char*, len) causes buffer "buf" to be physically copied
to new memory block. Make static buffer pointing to buf so we don't need
any intermediate copies. This reduces memory usage by op_size, and also
cpu usage by a bit (no memmove necessary).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

cmake: Remove duplicate find_package libcurl line.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7946 from SUSE/wip-doc-dev-teuth

doc/dev: integrate testing into the narrative

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cls_hello: Fix grammatical error in description comment

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

msg: remove duplicated code - local_delivery will now call 'enqueue'

This simplify the code and clarify that for each local message,
 'local_delivery_thread' simply tries to 'fast_dispatch' it,
 or it will 'enqueue' it for the dispatch_thread (like any standard message)

Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #7862 from theanalyst/rgw/doc-insecure-keystone

doc: rgw explain keystone's verify ssl switch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7887 from cbodley/wip-rgw-test-multi-period

test/rgw: add multisite test for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7827 from cbodley/wip-rgw-period-meta-logs

rgw: fixes for per-period metadata logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7938 from dmick/jewel

debian/changelog: Remove stray 'v' in version

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit 0f7730c8f444657d4aaffa28c12bbcd602a2e3b8)

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

rgw: fix for RGWCoroutinesManager::run() return value

the 'int ret' variable of the inner scope was shadowing an 'int ret'
variable in the outer scope, so we weren't propagating any of the error
codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use current period for InitSyncStatus

the InitSyncStatus coroutine records the position to start incremental
sync after finishing a full sync.  this should be the master's marker
from the current period, rather than its oldest log period

this also adds a check to run_sync() that restarts a full sync if it
sees that our sync period is behind the master's oldest log period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: meta log rest handlers avoid get_log()

RGWMetadataManager::get_log() will allocate a log and keep it in memory.
this could lead to a potential denial of service by making requests with
lots of different period ids

RGWMetadataLog if effectively stateless (the only state is a set of
modified_shards, which are not touched by any of the rest api calls), so
we can use a temporary instead of calling get_log()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pull first log period from master if not found locally

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: implement find_oldest_log_period for RGWMetadataManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period history behaves with no current_period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: initialize async_rados before meta_mgr

RGWMetadataManager::init() will depend on async_rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWStatObjCR for async raw_obj_stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: only write meta sync shard markers if can_adjust_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetaSyncCR wakeups thread-safe

now that the shards will be coming and going after startup, they need to
be reference counted and protected by a mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: add read_current_period_id() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #7918 from ceph/wip-rgw-payer-yehuda

rgw: link payer info to usage logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: add ceph-detect-init(8) source to dist tarball

Prior to this commit, the tarball from "make dist" did not include the
ceph-detect-init(8) man page rST source.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7929 from yehudasa/wip-submodules-fix

submodules: revert an accidental change

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7910 from ceph/wip-14955

buffer: increment history alloc as well in raw_combined

Reviewed-by: Sage Weil <sage@redhat.com>

submodules: revert an accidental change

revert changes that went in in commit f1ac0de5bfd974258c257470feae5ff03054873e.
This affects ceph-erasure-code-corpus, civetweb

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>