rgw: when create_bucket use the same num_shards with info.num_shards

pr #14388 only fix the num_shards in BucketInfo, "init_bucket_index" function still use local num_shards

Fixes: http://tracker.ceph.com/issues/19745
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #14984 from dmick/wip-osd-dup

test/osd/osd-dup.sh: lower wb fd throttle limits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14907 from ukernel/wip-19820

ceph-fuse: use user space permission check by default

Merge pull request #14976 from trociny/wip-19871

rbd-nbd: relax size check for newer kernel versions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14656 from chardan/jfw-wip-halflife_atomic_t-rodan

rbd, librbd: migrate atomic_t to std::atomic

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14925 from fangyuxiangGL/fix_unneccesary_write

rgw: don't do unneccesary write if buffer with zero length

Merge pull request #14990 from wjin/fix

osd: clean nonused work queue

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14962 from liewegas/wip-denc-dump

include/denc: support ENCODE_DUMP

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14995 from SUSE/wip-mgr-rest-exit

pybind/mgr/rest: completely terminate cherrypy in shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/denc: clean up ENCODE_DUMP a bit

- PATH_MAX for string
- ENCODE_DUMP -> ENCODE_DUMP_PATH
- some comments!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14999 from ivancich/wip-mod-git-subtree-README

doc: Re-word the warnings about using git subtrees.

doc: Re-word the warnings about using git subtrees.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rbd_mirror: migrate std::atomic<unsigned> to std::atomic<bool>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

librbd: migrate atomic_t to std::atomic<>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

tools (rbd*): migrate atomic_t to std::atomic<>

Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

pybind/mgr/rest: completely terminate cherrypy in shutdown

cherrypy.engine.stop() doesn't actually completely kill cherrypy,
which means cherrypy.engine.block() never returns, so this module's
thread never completes cleanly.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #14956 from liewegas/wip-bdev-ioc-lock

os/bluestore: fix use after free race with aio_wait

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jianpeng Ma <jianpeng@intel.com>

osd: clean nonused work queue

it was restructured by 3cc48278bf0ee5c9535d04b60a661f988c50063b

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

Merge pull request #14919 from tchaikov/wip-19825

mon: check is_shutdown() in timer callbacks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14968 from TsaiJin/wip-lock-pg-when-scrub-available

osd: sched_scrub() lock pg only if all scrubbing conditions are fulfilled

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14970 from dachary/wip-crush-documentation

crush: update documentation for negative choose step

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14979 from liewegas/wip-19874

mon: fix prime_pg_temp overrun
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14958 from ovh/bp-fix-finisher-warning

common/Finisher: fix uninitialized variable warning
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14793 from dmick/wip-prio-and-daemonperf

ceph: perfcounter priorities and daemonperf updates to use them

Merge pull request #14510 from liewegas/wip-bluefs-sync-write

os/bluestore/BlueFS: add bluefs_sync_write option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14305 from liewegas/wip-zipkin-partial

osd: zipkin tracing

messages/MCommand: fix type on decode

Wow, this has been broken since v0.38, but apparently
the message never made it into the object corpus so
we never noticed!

In reality the bug is harmless: decode_message() will
set_header which clobbers whatever version the default
ctor fills in, so this only affects ceph-dencoder's
test.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-object-corpus: kraken objects

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/osd-dup.sh: lower wb fd throttle limits

osd-dup.sh was losing OSDs to EMFILE, mostly on arm64 (not clear why).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14981 from cbodley/wip-rgw-bi-list-plain

cls/rgw: list_plain_entries() stops before bi_log entries
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

test/encoding/generate-corpus-objects: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rgw: list_plain_entries() stops before bi_log entries

list_plain_entries() was using encode_obj_versioned_data_key() to set
its end_key, which gives a prefix of BI_BUCKET_OBJ_INSTANCE_INDEX[=2]

that range between start_key and end_key would not only span the
BI_BUCKET_OBJS_INDEX[=0] prefixes, but BI_BUCKET_LOG_INDEX[=1] prefixes
as well. this can result in list_plain_entries() trying and failing to
decode a rgw_bi_log_entry as a rgw_bucket_dir_entry

Fixes: http://tracker.ceph.com/issues/19876
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/encoding/generate-corpus-objects.sh: simplify object corpus population

Script that generates corpus objects.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rgw/run-s3tests.sh: run s3-tests

This works out of the box with a vstart environment and

 RGW=1 ../src/vstart.sh -n -l
 PATH=bin:$PATH ../qa/workunits/rgw/run-s3tests.sh

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14051 from SpandanKumarSahu/bug#17195

rbd: stop indefinite thread waiting in krbd udev handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: remove useless IOContext::num_reading

If we are a syncrhonous read, we don't need this: we don't aio_wait for
sync reads.  If we are an aio_read, we are in the aio_running count anyway,
and there is also no purpose for this counter.

I'm a bit unsure about the NVME use of this counter; I switched it to use
num_running (pretty sure we aren't mixing reads and writes on a single
IOContext) *but* it might make more sense to switch to a private counter.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix racy aio_{wake,wait}() on IOContext

Thread 1 (_do_read)                 Thread 2 (_aio_thread)
queues aio
ioc->aio_wait()
  locks ioc->lock
  num_waiting++
                                    finishes aios
                                    ioc->aio_wake
                                      if (num_waiting)
                                        lock ioc->lock (blocks)
  num_running == 0, continue
  ioc->unlock
                                        ioc->lock taken
do_read destroys ioc
                                        use after free, may block forever

The last bit of the race may vary; the key thing is that thread 2 is
taking a lock that thread 1 can destroy; thread 2 doesn't have it pinned
in memory.

Fix this by simplifying the aio_wake, aio_wait.  Since it is mutually
exclusive with a callback completion, we can avoid calling it at all when
a callback in present, and focus on keeping it simple.

Avoid use-after-free by making sure the last decrement happens under
the lock in the aio_wake/wait case.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: remove weird aio_wake from sync read()

I'm not entirely sure why we were ever signaling the cond here; the read
is synchronous!

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: WITH_BLKIN off for now

We are unconditionally linking lttng-ust, which make selinux complain. We
should either

 - fix selinux rules and unconditionally link
 - dlopen at runtime based on an option (like we do for the current
   tracepoints)
 - ???

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix test

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix cmake compile options

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: remove installed files

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: fix -fPIC option

Signed-off-by: Sage Weil <sage@redhat.com>

blkin: drop boost dependency

Signed-off-by: Sage Weil <sage@redhat.com>

common/zipkin_trace: fix blkin_trace_info encoding

- encode the same regardless of whether it is compiled in (!)
- encode in endian- and struct alignment-safe way.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Message: make {encode,decode}_trace unconditional

These are protocol features and cannot vary based on our compilation.
Encode and decode unconditionally.  The callers have already guarded these
field additions with a message version bump and are conditionally calling
decode_trace.

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: fix libcommon link to blkin

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: default WITH_BLKIN=ON

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: build blkin instead of using find_package

Signed-off-by: Casey Bodley <cbodley@redhat.com>

src/blkin: add blkin submodule

Signed-off-by: Sage Weil <sage@redhat.com>

librados: Overload aio_operate with blkin trace info parameter

This will enable passing trace information from rbd to librados for
aio_reads and aio_writes.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

cmake: Add blkin libs to target_link_libraries

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

blkin: add config variable osdc_blkin_trace_all

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to librados and Objecter

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add config variable osd_blkin_trace_all

when set, Message::decode_trace() will always create a trace for
incoming messages, even if they didn't pass trace information

Signed-off-by: Casey Bodley <cbodley@redhat.com>

filestore: Initialize Zipkin traces when there is a store trace

Avoids creating orphaned traces.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

blkin: add traces to FileStore/Journal

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to ECBackend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to ReplicatedBackend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: set up tracing in PGs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: set up tracing in the OSD

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: osd op messages carry trace information

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to AsyncMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to XioMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add traces to SimpleMessenger

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: Messenger integration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

msg: pass Connection to decode_message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

blkin: add header and initialization

zipkin_trace.h is a wrapper around ztracer.hpp, which provides a stub
implementation when WITH_BLKIN is not defined

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: add module and WITH_BLKIN option for blkin

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Build support for BlkKin (LTTng + Zipkin) tracing

 * Adds --with-blkin to autoconf (default without)
 * Adds BLKIN_LIBS to necessary automake files
 * Adds documentation for testing BlkKin tracing

Signed-off-by: Andrew Shewmaker <agshew@gmail.com>
Signed-off-by: Marios-Evaggelos Kogias <marioskogias@gmail.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14834 from runsisi/wip-fix-double-free

librbd: minor fixes for image trash move

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14830 from runsisi/wip-open-cleanup

librbd: remove redundant check for image id emptiness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14927 from tchaikov/wip-19427

logclient: use the seq id of the 1st log entry when resetting session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14920 from tchaikov/wip-14579

mgr: add machinery for python modules to send MCommands to daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13681 from liewegas/wip-epochs

osd: tag fast dispatch messages with min_epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSDMapMapping: add assert

This is easier to debug than walking off into memory we don't own!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix prime_pg_temp condition

We have to make sure the *pg* exists, not just the pool, in the previous
map, or else we'll get a funky overrun in OSDMapMapping::get().

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14273 from cbodley/wip-rgw-beast-new-parser

rgw: update Beast for streaming reads in asio frontend

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rbd-nbd: relax size check for newer kernel versions

Fixes: http://tracker.ceph.com/issues/19871
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rgw: rename frontend from asio to beast

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update Beast for streaming reads in asio frontend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: update Beast submodule

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14931 from tchaikov/wip-19771

qa/tasks/ceph_manager: always fix pgp_num when done with thrashosd task

Reviewed-by: Sage Weil <sage@redhat.com>

crush: update documentation for negative choose step

And remove an obsolete comment.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #14973 from ifed01/wip-bluestore-kvsync-cond

os/bluestore: remove unused condition variable

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove unused condition variable

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14963 from dillaman/wip-19858

rbd-mirror: failover and failback of unmodified image results in split-brain

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14965 from dmick/wip-19865

test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14465 from dillaman/wip-librbd-devstack-tempest

qa/workunits: switch to OpenStack Ocata release for RBD testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

osd/OSD.cc: lock the pg only when all conditional checks are passed when
scheduling a  scrub

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>

mon: check is_shutdown() in timer callbacks

introduce a helper class: C_MonContext, and initialize all timer events
using it, to ensure that they do check is_shutdown() before doing their
work.

Fixes: http://tracker.ceph.com/issues/19825
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

If RBD_FEATURES is not in the environment, set *features to 0 in
get_features(); callers rely on a valid return value.  (This was
breaking on arm64.)

Fixes: http://tracker.ceph.com/issues/19865
Signed-off-by: Dan Mick <dan.mick@redhat.com>

mon/Elector: call cancel_timer() in shutdown()

instead of doing it manually.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: add no-op event when promoting an image

The rbd-mirror process needs an event in the journal
to properly detect the transition between primary and
non-primary state between peers.

Fixes: http://tracker.ceph.com/issues/19858
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not delay propagation of demote events

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

include/denc: support ENCODE_DUMP

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: prevent infinite loop when computing replay status

If the image had a non-primary predecessor epoch whose tag tid
duplicates an epoch within its own journal, an infinite loop
would result.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14373 from rzarzynski/wip-rgw-19520

rgw: fix broken /crossdomain.xml, /info and /healthcheck of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9950 from weiqiaomiao/wqm-wip-fetchobj-req

rgw: should delete in_stream_req  if conn->get_obj(...) return not zero value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14601 from Jing-Scott/fix-bulkupload-multisite

rgw: fix forward request for bulkupload to be applied in multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>