rbd-mirror: integrate single thread pool for all processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: own the lifecycle of the journaler's threads

This is the first step in merging the journal threads into the
librbd singleton thread pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: use provided work queue and timer

This avoids the need to open two threads per journaler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: pass metadata instead of finisher to FutureImpl

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7959 from kmroz/wip-add-rgw-to-stopsh

script: add missing stop_rgw variable to stop.sh script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'wip-conf-root' of https://github.com/liewegas/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/common/config.cc

Merge pull request #7914 from dillaman/wip-tests-template-specialization

test: correct librbd errors discovered with unoptimized cmake build

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7628 from liewegas/wip-journal-block-size

os/filestore/FileJournal: set block size via config option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7705 from renhwsky/renhw-wip-pgmonitor-get_rule_avail

mon/pgmonitor: use appropriate forced conversions in get_rule_avail

Merge pull request #7775 from branch-predictor/bp-bl-eradication

os/ObjectStore: implement more efficient get_encoded_bytes()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7845 from aiicore/pthread_setname_fix

common: set thread name from correct thread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7850 from jupiturliu/kstore-latency

osd: kstore: latency breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7861 from tchaikov/wip-14910

qa/workunits/cephtool/test.sh: wait longer in ceph_watch_start()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7863 from branch-predictor/bp-faster-name-gen

common/obj_bencher.cc: faster object name generation

Reviewed-by: Sage Weil <sage@redhat.com>

stop: Add missing stop_rgw variable

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7954 from SUSE/wip-docdev

doc/dev: various refinements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: start Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/dev: various refinements

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7951 from badone/wip-cls_hello-grammar

cls_hello: Fix grammatical error in description comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Remove duplicate find_package libcurl line.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7946 from SUSE/wip-doc-dev-teuth

doc/dev: integrate testing into the narrative

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cls_hello: Fix grammatical error in description comment

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/dev: integrate testing into the narrative

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7862 from theanalyst/rgw/doc-insecure-keystone

doc: rgw explain keystone's verify ssl switch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7887 from cbodley/wip-rgw-test-multi-period

test/rgw: add multisite test for meta sync across periods

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7827 from cbodley/wip-rgw-period-meta-logs

rgw: fixes for per-period metadata logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7941 from SUSE/wip-teuos-intro

doc: detailed description of bugfixing workflow

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: detailed description of bugfixing workflow

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7933 from ceph/wip-doc-dist-ceph-detect-init

doc: add ceph-detect-init(8) source to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7937 from saru95/patch--1

ceph_daemon.py: Resolved ImportError to work with python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7936 from ceph/wip-fix-debian-build

debian/changelog: Remove stray 'v' in version

ceph_daemon.py: Resolved ImportError to work with python3

Signed-off-by: Sarthak Munshi <sarthakmunshi@gmail.com>

rgw: fix for RGWCoroutinesManager::run() return value

the 'int ret' variable of the inner scope was shadowing an 'int ret'
variable in the outer scope, so we weren't propagating any of the error
codes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use current period for InitSyncStatus

the InitSyncStatus coroutine records the position to start incremental
sync after finishing a full sync.  this should be the master's marker
from the current period, rather than its oldest log period

this also adds a check to run_sync() that restarts a full sync if it
sees that our sync period is behind the master's oldest log period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: meta log rest handlers avoid get_log()

RGWMetadataManager::get_log() will allocate a log and keep it in memory.
this could lead to a potential denial of service by making requests with
lots of different period ids

RGWMetadataLog if effectively stateless (the only state is a set of
modified_shards, which are not touched by any of the rest api calls), so
we can use a temporary instead of calling get_log()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: pull first log period from master if not found locally

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: implement find_oldest_log_period for RGWMetadataManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: period history behaves with no current_period

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: initialize async_rados before meta_mgr

RGWMetadataManager::init() will depend on async_rados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWStatObjCR for async raw_obj_stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: only write meta sync shard markers if can_adjust_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetaSyncCR wakeups thread-safe

now that the shards will be coming and going after startup, they need to
be reference counted and protected by a mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: add read_current_period_id() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian/changelog: Remove stray 'v' in version

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #7918 from ceph/wip-rgw-payer-yehuda

rgw: link payer info to usage logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: add ceph-detect-init(8) source to dist tarball

Prior to this commit, the tarball from "make dist" did not include the
ceph-detect-init(8) man page rST source.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7929 from yehudasa/wip-submodules-fix

submodules: revert an accidental change

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #7910 from ceph/wip-14955

buffer: increment history alloc as well in raw_combined

Reviewed-by: Sage Weil <sage@redhat.com>

submodules: revert an accidental change

revert changes that went in in commit f1ac0de5bfd974258c257470feae5ff03054873e.
This affects ceph-erasure-code-corpus, civetweb

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7916 from ceph/wip-14975

rgw: adjust error code when bucket does not exist in copy operation
rgw: don't override error when initializing zonegroup

Fixes: #14975
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: avoid showing payer when payer == owner

If the payer matches the owner it shows the owner only.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #7786 from ceph/wip-rgw-indexless

rgw: indexless buckets (Yehuda Sadeh)

 - can define a policy, for which buckets are indexless
 - users can then create buckets under the specified placement target
 - indexless buckets will not be synced across zones
 - does not work with (s3) versioned buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: indexless buckets

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: configurable index type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't override error when initializing zonegroup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adjust error code when bucket does not exist in copy operation

Fixes: #14975
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test: use StrEq for C-style string matching

If compiler optimizations are enabled, strings will be copied,
resulting in different memory addresses.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common/buffer: correct list_iterator::operator!= behavior

It was actually behaving as if it were operator==.  This results
in loops running off the end of empty bufferlists.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7764 from rzarzynski/wip-rgw-etag-dlo

rgw: multiple improvements regarding etag calculation for SLO/DLO of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test: librbd template specializations leaking between translation units

A CMake build with optimizations disabled can result in a template
specialization for one test being reused by another test in a
different translation unit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

buffer: increment history alloc as well in raw_combined

Introduced in 73dcd26fc8798e09d93520c583d63edb28ffc631

Fixes: #14955
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'gh/jewel'

v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7897 from branch-predictor/bp-bench-errors-fatal

common/obj_bencher.cc: make verify error fatal

Reviewed-by: Loic Dachary <ldachary@redhat.com>

global: complain about parse errors after we open the log

Signed-off-by: Sage Weil <sage@redhat.com>

config: more parse_errors into md_config_t

Signed-off-by: Sage Weil <sage@redhat.com>

common/obj_bencher.cc: make verify error fatal

When run without "--no-verify", all verification errors are noted,
but they are not forwarded/reported anywhere else but to cerr, which
will cause automated testing to ignore them. Make seq_read_bench and
rand_read_bench return -EIO on any verification error which will,
in turn, return it back to caller.

Fixes: #14971
Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7757 from ukernel/jewel-14684

mon/MDSMonitor.cc: properly note beacon when health metrics changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7882 from branch-predictor/bp-unconfuse-health-1

mon: unconfuse object count skew message

Reviewed-by: Sage Weil <sage@redhat.com>

PGMonitor: unconfuse object count skew message

"Pool <pool> has too few pgs" is okay assuming it does not take other
pools into account. And since it does, it is confusing in the following
scenario:

1. Create two pools, one with small pg count and one with large
   pg count
2. Put a whole lot of objects in smaller pool, resulting in "too few
   pgs" warning on that pool, which is expected behavior.
3. Put a whole lot of objects in larger pool, warning goes away.
   Suddenly smaller pool has plenty of PGs?

Current message suggests adding more nodes (or PGs) to pool, when
actually it's warning about significantly more objects in that
particular pool than in the other pools.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7898 from oritwas/wip-cmake-add-fs_types

cmake: add common/fs_types.cc to libcommon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7881 from aclamk/ceph-df-used-fix

mon: fix calculation of %USED

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7892 from majianpeng/bluestore-leak-unwritten-extent

osd: bluestore: fix check for write falling within the same extent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7886 from stiopaa1/blueStoreOverrides

osd: bluestore: add 'override' to virtual functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7895 from majianpeng/bluestore-trim-overlay

os/bluestore/BlueStore: Don't leak trim overlay data before write.

os/bluestore/BlueStore: Don't leak trim overlay data before write.

Suppose: bluestore_overlay_max_length=bluestore_min_alloc_size;
bluestore_overlay_max = 2;
For the following ops:
write(off=0, len=4096) --->write into overlay
write(off=4096, len=4096)-->write into overlay
write(off=0, len=bluestore_min_alloc_size)-->because overlay_map.size()
>=2, it allocate a extent.

It should trim overlay data(0,4096) &(4096, 4096),and then write(0,
bluestore_min_alloc_size).
But the original code don't trim overlay data.
This make the later read data is orignal data rather that new data.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: add common/fs_types.cc to libcommon

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore/BlueStore: Fix bug when calc offset & end whether locate in the a extent.

Suppose: bluestore_overlay_max_length == bluestore_min_alloc_size
The orignal code which calc content of written whether locate in a
extent:
(offset / min_alloc_size) == (offset + length) /min_alloc_size
This will make the case which offset=0 & length =min_alloc_size locate
in the different extent.
In fact, this content is in the same extent.

Change end = offset + length - 1 make work.

Fixes: #14954
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/obj_bencher.cc: use more readable constant instead of magic number

When clean_up_slow() fails, it returns "-5" which is equal to -EIO.
Change it in source, so it's not confusing for someone who does not
remember all error codes (functionality remains the same).

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #7098 from liewegas/wip-file-layout

mds, client: add namespace to file_layout_t (previously ceph_file_layout)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Bluestore.h:Added 'override' to virtual functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

test/rgw: add multisite test for meta sync across periods

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7849 from ceph/wip-cmake

Wip cmake

cmake: simplified heap_profiler_objs dependencies

The heap_profiler object library only needs to be
added to mon,mds, and osd instead of the numerous
targets it is part of

Signed-off-by: Ali Maredia <amaredia@redhat.com>

client/Inode: fix has_dir_layout

Compare to default file_layout_t() -- not zeroed memory.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7867 from SUSE/wip-14940

RPM: move %post(un) ldconfig calls to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #7869 from rzarzynski/wip-rgw-aws4-puthash

rgw: calculate payload hash in RGWPutObj_ObjStore only when necessary.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7612 from liewegas/wip-buffer-combined

buffer: raw_combined allocations buffer and ref count together

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7690 from xiexingguo/xxg-wip-14804

common/TrackedOp: fix inaccurate counting for slow requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7704 from xiexingguo/xxg-wip-14814

osd: filestore: fix wrong scope of result code for error cases during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7812 from xiexingguo/xxg-wip-14872

osd: fix wrong return type of find_osd_on_ip()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7815 from branch-predictor/bp-conditional-crc

msg/async: don't calculate msg header crc when not needed

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7711 from xiexingguo/xxg-wip-14817

osd: filestore: fix result code overwritten for clone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7790 from tchaikov/wip-log-segv

log: segv in a portable way

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7842 from xiexingguo/xxg-wip-14921

osd: fix overload of '==' operator for pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

[MON] Fixed calculation of %USED. Now it is shows (space used by all replicas)/(raw space available on OSDs). Before it was (size of pool)/(raw space available on OSDs).

Signed-off-by: Adam Kupczyk akupczyk@mirantis.com