Merge pull request #14510 from liewegas/wip-bluefs-sync-write

os/bluestore/BlueFS: add bluefs_sync_write option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14305 from liewegas/wip-zipkin-partial

osd: zipkin tracing

Merge pull request #14981 from cbodley/wip-rgw-bi-list-plain

cls/rgw: list_plain_entries() stops before bi_log entries
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: list_plain_entries() stops before bi_log entries

list_plain_entries() was using encode_obj_versioned_data_key() to set
its end_key, which gives a prefix of BI_BUCKET_OBJ_INSTANCE_INDEX[=2]

that range between start_key and end_key would not only span the
BI_BUCKET_OBJS_INDEX[=0] prefixes, but BI_BUCKET_LOG_INDEX[=1] prefixes
as well. this can result in list_plain_entries() trying and failing to
decode a rgw_bi_log_entry as a rgw_bucket_dir_entry

Fixes: http://tracker.ceph.com/issues/19876
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14051 from SpandanKumarSahu/bug#17195

rbd: stop indefinite thread waiting in krbd udev handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

CMakeLists.txt: WITH_BLKIN off for now

We are unconditionally linking lttng-ust, which make selinux complain. We
should either

 - fix selinux rules and unconditionally link
 - dlopen at runtime based on an option (like we do for the current
   tracepoints)
 - ???

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix test

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix cmake compile options

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: remove installed files

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix -fPIC option

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: drop boost dependency

Signed-off-by: Sage Weil <sage@redhat.com>

common/zipkin_trace: fix blkin_trace_info encoding

- encode the same regardless of whether it is compiled in (!)
- encode in endian- and struct alignment-safe way.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Message: make {encode,decode}_trace unconditional

These are protocol features and cannot vary based on our compilation.
Encode and decode unconditionally.  The callers have already guarded these
field additions with a message version bump and are conditionally calling
decode_trace.

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: fix libcommon link to blkin

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: default WITH_BLKIN=ON

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: build blkin instead of using find_package

Signed-off-by: Casey Bodley <cbodley@redhat.com>

src/blkin: add blkin submodule

Signed-off-by: Sage Weil <sage@redhat.com>

librados: Overload aio_operate with blkin trace info parameter

This will enable passing trace information from rbd to librados for
aio_reads and aio_writes.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

cmake: Add blkin libs to target_link_libraries

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

blkin: add config variable osdc_blkin_trace_all

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to librados and Objecter

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add config variable osd_blkin_trace_all

when set, Message::decode_trace() will always create a trace for
incoming messages, even if they didn't pass trace information

Signed-off-by: Casey Bodley <cbodley@redhat.com>

filestore: Initialize Zipkin traces when there is a store trace

Avoids creating orphaned traces.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

blkin: add traces to FileStore/Journal

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to ECBackend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to ReplicatedBackend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: set up tracing in PGs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: set up tracing in the OSD

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: osd op messages carry trace information

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to AsyncMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to XioMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to SimpleMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: Messenger integration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

msg: pass Connection to decode_message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add header and initialization

zipkin_trace.h is a wrapper around ztracer.hpp, which provides a stub
implementation when WITH_BLKIN is not defined

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: add module and WITH_BLKIN option for blkin

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Build support for BlkKin (LTTng + Zipkin) tracing

 * Adds --with-blkin to autoconf (default without)
 * Adds BLKIN_LIBS to necessary automake files
 * Adds documentation for testing BlkKin tracing

Signed-off-by: Andrew Shewmaker <agshew@gmail.com>
Signed-off-by: Marios-Evaggelos Kogias <marioskogias@gmail.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14834 from runsisi/wip-fix-double-free

librbd: minor fixes for image trash move

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14830 from runsisi/wip-open-cleanup

librbd: remove redundant check for image id emptiness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14927 from tchaikov/wip-19427

logclient: use the seq id of the 1st log entry when resetting session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14920 from tchaikov/wip-14579

mgr: add machinery for python modules to send MCommands to daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13681 from liewegas/wip-epochs

osd: tag fast dispatch messages with min_epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14273 from cbodley/wip-rgw-beast-new-parser

rgw: update Beast for streaming reads in asio frontend

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rename frontend from asio to beast

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update Beast for streaming reads in asio frontend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update Beast submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14931 from tchaikov/wip-19771

qa/tasks/ceph_manager: always fix pgp_num when done with thrashosd task

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14973 from ifed01/wip-bluestore-kvsync-cond

os/bluestore: remove unused condition variable

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove unused condition variable

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14963 from dillaman/wip-19858

rbd-mirror: failover and failback of unmodified image results in split-brain

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14965 from dmick/wip-19865

test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14465 from dillaman/wip-librbd-devstack-tempest

qa/workunits: switch to OpenStack Ocata release for RBD testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

If RBD_FEATURES is not in the environment, set *features to 0 in
get_features(); callers rely on a valid return value.  (This was
breaking on arm64.)

Fixes: http://tracker.ceph.com/issues/19865
Signed-off-by: Dan Mick <dan.mick@redhat.com>

librbd: add no-op event when promoting an image

The rbd-mirror process needs an event in the journal
to properly detect the transition between primary and
non-primary state between peers.

Fixes: http://tracker.ceph.com/issues/19858
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not delay propagation of demote events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: prevent infinite loop when computing replay status

If the image had a non-primary predecessor epoch whose tag tid
duplicates an epoch within its own journal, an infinite loop
would result.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14373 from rzarzynski/wip-rgw-19520

rgw: fix broken /crossdomain.xml, /info and /healthcheck of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9950 from weiqiaomiao/wqm-wip-fetchobj-req

rgw: should delete in_stream_req  if conn->get_obj(...) return not zero value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14601 from Jing-Scott/fix-bulkupload-multisite

rgw: fix forward request for bulkupload to be applied in multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14945 from dillaman/wip-19811

rbd-mirror: ensure missing images are re-synced when detected

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14935 from dillaman/wip-rbd-cli-mirror-pool

rbd: removed spurious error message from mirror pool commands

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14400 from ceph/wip-cd-1node

qa/tasks: few fixes to get ceph-deploy 1node to working state

Merge pull request #14936 from cbodley/wip-19834

rgw: RGWPeriodPusher spawns http thread before cr thread

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14908 from guihecheng/rgw_file-cleanup

rgw_file: cleanup virtual keyword on derived functions

Merge pull request #14891 from tchaikov/wip-librbd-test-odr

librbd: do not instantiate templates while building tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not instantiate templates while building tests

to avoid violation of ODR

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14844 from liewegas/wip-bluestore-bitalloc

unittest_alloc: add test_alloc_big

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

unittest_alloc: add test_alloc_big

Ensure that big allocations work.

Verifies http://tracker.ceph.com/issues/16662

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: add bluefs_sync_write option

If we have a fast device we can do our writes using synchronous
IO instead of aio.  Most of the time rocksdb is doing sync writes
anyway (write and then fsync from the same thread).

Note that this might not be the case when using the
bluestore_sync_submit_transaction mode... that probably should
not be combined with bluefs_sync_write.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14953 from tchaikov/wip-librbd-silence-Wunused-var

librbd: silence -Wunused-variable warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14407 from cbodley/wip-rgw-admin-remote

rgw: fix handling of --remote in radosgw-admin period commands
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

librbd: silence -Wunused-variable warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11179 from rzarzynski/wip-rgw-swift-formpost

rgw: add support for FormPost of Swift API
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14949 from zhsj/fix-cryptopp-build

common: fix building against libcryptopp

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: fix building against libcryptopp

libnspr is only needed for libnss, move it to if defined(USE_NSS)
section.

ceph::crypto::shutdown's params should be same with declaration.

Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>

Merge pull request #14923 from joscollin/wip-dmclocktest-function-error

dmclock: include missing <functional> header.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14948 from Liuchang0812/wip-kill-duplicated-cmake-commands

cmake: kill duplicated cmake commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14951 from smithfarm/wip-19853

cmake: align cmake names of library packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Update rgw_rest.cc

fix indentation.
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: align cmake names of library packages

The cmake package names of libraries are case-sensitive. The name used in
FIND_PACKAGE_HANDLE_STANDARD_ARGS - e.g. "cryptopp" - must match the name used
in the module name - "Findcryptopp.cmake" - as well as the name used in
find_package() call - "find_package(cryptopp REQUIRED)"

Note that the term "package" here refers to a cmake abstraction, *not* an
actual RPM or Debian package.

Fixes: http://tracker.ceph.com/issues/19853
Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: kill duplicated cmake commands

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #14947 from dmick/wip-arm-compressor

compressor/zlib: fix plugin for non-Intel arches

Reviewed-by: Kefu Chai <kchai@redhat.com>

compressor/zlib: fix plugin for non-Intel arches

unittest_compressor was failing on arm64 because the zlib
compressor was never initialized, even though it works fine in
non-isal mode

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14363 from majianpeng/finisher-optimize

common/Finisher: batch handle perfcounter && only send signal when waiter existed.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14627 from badone/wip-peering-state-timing

osd: Implement peering state timing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14709 from majianpeng/luestore-txc-finalize-kv

os/blueStore: In osd_tp_thread, call _txc_finalize_kv.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14932 from markhpc/wip-rocksdb-compaction-readahead

common/config_opts.h: compaction readahead for bluestore/rocksdb

Reviewed-by: Sage Weil <sage@redhat.com>

test/librbd: remove duplicated explicit template specializations

to avoid violation of ODR

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: ensure missing images are re-synced when detected

Fixes: http://tracker.ceph.com/issues/19811
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

default vpm is trusty and for master we need latest which is xenial

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

install mgr on the node

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #14879 from wjwithagen/wip-wjw-reduce-noise-3

cls/log/cls_log.cc: reduce logging noise

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14928 from drunkard/master

doc: remove deprecated subcommand in man/8/ceph.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14723 from neha-ojha/objclass-sdk

osd: objclass sdk

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14917 from yuriw/wip-add-overrite-cls

qa/added overrides

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls/log/cls_log.cc: reduce logging noise

 - The other reference in the source as already at 20.
      ./src/cls/timeindex/cls_timeindex.cc:85:
CLS_LOG(20, "storing entry at %s", index.c_str());

   And we need not always know where in the log items are stored.
   So it looks like a leftover debug feature.

Fixes: http://tracker.ceph.com/issues/19835
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14913 from liewegas/wip-19826

mon: do not prime_pg_temp creating pgs; clean up pg create conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/Finisher: No waite_for_empty no signal.

Avoid every time to send a signal because wait_for_empty is rare event.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rgw: RGWPeriodPusher spawns http thread before cr thread

Fixes: http://tracker.ceph.com/issues/19834
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/BlueFS: _write_super is always synchronous

Signed-off-by: Sage Weil <sage@redhat.com>