rgw: take placement_rule from bucket info in update_containers_stats

in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18615 from rgdoliveira/alpine

rbd-fuse: make sure PATH_MAX is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17363 from tchaikov/wip-ceph-disk

ceph-disk: unlock all partitions when activate

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #18585 from yaozongyou/fix-bluestore-tool-arg-bug

ceph-bluestore-tool: link target shouldn't ending with "\n"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17995 from scienceluo/wip-luo-fix-bluestore-metadata-branch

os/bluestore/BlueFS: Reduce unnecessary operations in collect_metadata

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd-fuse: make sure PATH_MAX is defined

On systems without glibc, as Alpine Linux, you might lack definition of
PATH_MAX. This patch adds the limits.h header to solve this issue.

Signed-off-by: Roberto Oliveira <robertoguimaraes8@gmail.com>

Merge pull request #18611 from liewegas/wip-interval-set

include/interval_set: parameterize by map type and kill btree_interval_set.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18507 from hjwsm1989/skip-generation-object

ceph-objectstore-tool: skip object with generated version

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18493 from liewegas/wip-unclean-degraded

mon/PGMap: 'unclean' does not imply damaged

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18553 from liewegas/wip-preempt-feature

osd/PG: remote recovery preemption, and new feature bit to condition it on

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18594 from zmc/wip-cephmetrics

suites/cephmetrics: Add Centos 7

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #18601 from dillaman/wip-21961

rbd-mirror: clean up spurious error log messages

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

osd/PG: handle spurious remote revoke or toofull

If we restart the recovery reservation process for some reason (e.g.,
local preemption), we may still get a racing REVOKE from the replicas in
the previous round.  Ignore.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: journal debug message should not be logged as error

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: avoid attempting to decode an empty client data buffer

Fixes: http://tracker.ceph.com/issues/21961
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18604 from dillaman/wip-21960

cls/journal: ensure tags are properly expired

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

unittest_interval_set: test with flat_map<> too

Signed-off-by: Sage Weil <sage@redhat.com>

include/btree_interval_set: remove it!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use interval_set<...,btree_map<...>>

This avoid the code duplication, yay!

Signed-off-by: Sage Weil <sage@redhat.com>

include/btree_map: clean up wrapper header around btree_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitMapAllocator: drop useless include

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: cbegin/cend -> begin/end

c{begin,end} don't exist for btree_map<>.

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: tolerate maps that invalidate iterator on change

These changes picked out of the diff between the original
btree_interval_set.h and interval_set.h (sadly I had it rolled into the
initial commit so it was tedious to identify these).

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: templating map used

Default to std::map for backward compat.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: handle RELEASE race with Remote*Reserved on replica

If we get a REQUEST and RELEASE in quick succession, we may process a
local Remote*Reserved back in the RepNotRecovering state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18602 from adamemerson/wip-optional-failure

rgw: Fix dereference of empty optional

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

osd/PG: fix Remote*Preempted handling in ReplicaActive

This was only half-implemented by c1e652a4547d89b889954ca4bdb82506ca0f1d83

Signed-off-by: Sage Weil <sage@redhat.com>

cls/journal: ensure tags are properly expired

Previously, if only the local image was using the journal or if
a disconnected peer was attached, the tag entries could not be
expired even if unreferenced.

Fixes: http://tracker.ceph.com/issues/21960
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-bluestore-tool: the link target should not ending with new line

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-bluestore-tool: the right action is prime-osd-dir not prime-osd-dev

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18592 from dillaman/wip-21956

cls/journal: fixed possible infinite loop in expire_tags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18607 from yaozongyou/fix-typo

doc: fix typo e.g,. => e.g.,

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: fix typo e.g,. => e.g.,

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18519 from liewegas/wip-21712-c

ceph_test_objectstore: do not change model for 0-length zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PG: send release to remote reservations in progress

If we have a REQUEST outstanding and then cancel, send a RELEASE to the
REQUEST target too, so that they too can clean up.  This avoids leaving
the replica in the RepRecovering state when the primary is restarting
the whole reservation process.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Fix dereference of empty optional

Due to the lack of a return, there was a case where an invalid ARN
could cause a dereference of an uninitialized boost::optional.

As a bit of defensive programming, restructure a couple functions to
make that kind of error impossible by ensuring the optional is only in
scope when it is initialized and relying less in early return on
error.

Fixes: http://tracker.ceph.com/issues/21962
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/rbd: run cls tests for all dependencies

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/journal: fixed possible infinite loop in expire_tags

Fixes: http://tracker.ceph.com/issues/21956
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18571 from linuxbox2/wip-rgwfile-21940

rgw_file:  set s->obj_size from bytes_written

Merge pull request #18449 from dzafman/wip-zafman-misc

mark_unfound_lost fix and some other minor changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Use a symlink for Ubuntu fragment

Signed-off-by: Zack Cerza <zack@redhat.com>

suites/cephmetrics: Add Centos 7

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #18402 from cbodley/wip-21831

rgw: fix for pause in beast frontend

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18328 from PCzhangPC/cplostmeta

librbd: cannot copy all image-metas if we have more than 64 key/value pairs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18419 from majianpeng/librbd-bug-fix

librbd misc cleanup.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18542 from trociny/wip-metadata-open

librbd: don't read metadata twice on image open

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18561 from dillaman/wip-librbd-test-notify

test/librbd: added update_features RPC message to test_notify

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18583 from idryomov/wip-qa-xfstests-except

qa/tasks/rbd: run all xfstests runs to completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18557 from tchaikov/wip-cmake-cleanup

cmake: check for aligned_alloc() instead of checking tcmalloc version

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18551 from liewegas/wip-object-stat-decode

osd/osd_types: fix object_stat_sum_t decode

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #18565 from liewegas/wip-prime-fixes

ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd/PG: handle slow Remote*Preempted events

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow preemption of remote recovery reservation too

We did backfill but not recovery.  This one is easier because we already
have a DeferRecovery event we can queue that does everything.

Condition on the RECOVERY_RESERVATION_2 feature.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: condition remote backfill preemption on RECOVERY_RESERVATION_2

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: define RECOVERY_RESERVATION_2

This will allow us to backport support for reservation preemption and
cancellations on the wire.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18518 from ceph/wip-cv-fix-default-flags

ceph-volume: ensure correct --filestore/--bluestore behavior

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18502 from optimistyzy/1024_yzy

bluestore/NVMEDevice: convert the legacy config opt related with SPDK

Merge pull request #18576 from tchaikov/wip-cmake-no-rados_a

cmake: do not link against librados.a

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: remove rados_a target

rados_a is librados.a, which is not neeeded anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link librbd against librados not rados_a

link librados.so in hope to reduce the size of librdb, also the size of
debug-info.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link librados tests against librados.so

* in hope to shrink the test, and the debug-info rpm package
* also add librados to radostest's target_link_libraries(),  otherwise we might
   run into linker order issue, like

Undefined reference in
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librados/test.cc:61
to `rados_application_enable'

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados/cmd.cc: avoid using non-librados facilities

a test for librados should be only depending on librados

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18543 from neha-ojha/wip-add-cbt-repo

qa: add cbt repo parameter

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18573 from neha-ojha/wip-add-cbt-workloads

qa/suites/rados/perf: add workloads

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: check for aligned_alloc()

in case developer is using patched tcmalloc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #18174 into master

* refs/pull/18174/head:
mds: no assertion on inode being purging in find_ino_peers()

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #18537 into master

* refs/pull/18537/head:
mds: set PRIO_USEFUL on num_sessions counter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/perf: add workloads

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw_file:  set s->obj_size from bytes_written

Required to store correct final accounted_size of objects in
RGWWriteWriteRequest::write_finish.

Fixes: http://tracker.ceph.com/issues/21940
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18524 from liewegas/wip-backfill-rops

osd/PG: fix recovery op leak

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18552 from liewegas/wip-mds-cluster-down

qa/workunits/rest/test.py: stop trying to test obsolte cluster_up/down

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18489 from cbodley/wip-21895

rgw: fix extra_data_len handling in PutObj filters

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

ceph-volume lvm.activate remove links on activate before priming

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-bluestore-tool: prime-osd-dir: update symlinks instead of bailing

If the symlink points to the right location, do nothing.  If it doesn't,
replace it.  If it's not a symlink, bail with EEXIST.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18451 from zmc/wip-cephmetrics

Add cephmetrics suite

Merge pull request #18198 from cbodley/wip-21735

rgw: fix error handling in ListBucketIndexesCR

qa/tasks/rbd: run all xfstests runs to completion

rbd.xfstests task allows spawning xfstests runs on multiple nodes.
Don't unwind task contexts if one of the runs fails -- let the other
runs finish.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Add cephmetrics suite

Signed-off-by: Zack Cerza <zack@redhat.com>

test/librbd: added update_features RPC message to test_notify

Fixes: http://tracker.ceph.com/issues/21936
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rest/test.py: stop trying to test obsolete 'mds *' commands

These were recently removed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18511 from smithfarm/wip-rpm-recommend-influxdb

build/ops: rpm: recommend python-influxdb with ceph-mgr

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #18182 from nrdmn/dashboard

mgr/dashboard: improve error handling

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: don't read metadata twice on image open

After adding get_metadata to RefreshRequest it has become redundant
in OpenRequest.

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18533 from tchaikov/wip-qa-cephtool

qa: do not "ceph fs get cephfs" w/o a cephfs fs

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

osd/PG: handle remote backfill recovation while waiting for other targets

If we have multiple targets, we may still be waiting on them when we get
a revocation.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: Reduce unnecessary operations in collect_metadata

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

ceph-volume lvm.activate ceph-bluestore-tool needs --cluster too

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18545 from batrick/pr18393-revert

Revert "ceph_context: re-expand admin_socket metavariables in child process"

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18496 from liewegas/wip-pgmap

build: remove PGMap.cc from libcommon

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: make scan recovery op cancellation match up reliably

Previously, there was only one time we would end up in this region of code:
when the backfill was rejected by the peer.  Previously that was apparently
reliably when we had an outstanding SCAN request, because we would
unconditionally cancle the MAX recovery op and clear waiting_on_backfill.

See 624aaf2a4ea9950153a89ff921e2adce683a6f51 for when this code appeared.

Now we have several similar paths, and we don't always have an outstanding
scan call (I don't think!).  Regardless, move most these three cases into
a common helper and make the finish_recovery_op completion conditional
on whether there is an outstanding SCAN.  This fixes a leak of a recovery
op when we defer while a scan is outstanding (this bug was recently
introduced by e708410542b0a52fbb29e14b76f49c94adbc0a59 and then
duplicated by 2463c6463d1ed38a2e15a0960ed1530a47851489).

Note that there is still one other time we register MAX ops: when we are
finishing backfill.  There, we start one per target.  But we will always
get back our reply and process it in the normal way (that old commit
did not change the timing for these).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: make recovering_oids a multiset

For multiple backfill targets, we start MAX multiple times.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18527 from xiexingguo/wip-hunt-scrub-mismatch

osd/PrimaryLogPG: several oi.extents fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18531 from majianpeng/msg-async

msg/async misc cleanup

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: record correctly l_msgr_send_bytes if "rc == 0".

If rc == 0, it mean all data of outcoming_bl whether old or new data.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/osd_types: fix object_stat_sum_t decode

Broken by 71bf04775bef90c9291bd825da626bc9de6f9ec1

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #18192 into master

* refs/pull/18192/head:
qa/cephfs: test ec data pool
qa/suites/fs/basic_functional/clusters: more osds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume lvm.activate consume cluster_name from lvm metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare persist cluster_name on lvm metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Revert "ceph_context: re-expand admin_socket metavariables in child process"

This commit introduces a regression where the admin socket specified in
ceph.conf overrides command line arguments or environment variables.

This reverts commit 1630f4bbef3a3ff6efb56acbaa9b8786b945b1e6.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18485 from liewegas/wip-remote-res-preemption

osd: fix recovery reservation bugs, and implement remote reservation preemption

Reviewed-by: David Zafman <dzafman@redhat.com>