qa/workunits/rados/test_rados_tool: fix copy paste error

Signed-off-by: Sage Weil <sage@redhat.com>

tests/rbd: add option to leave behind fsx test data

tests/rbd: add option to leave behind fsx test data

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests/rbd: add option to leave behind fsx test data

Previously, fsx deleted all test data upon successful completion. Add
an option to leave the data behind for further analysis.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #8936 from bstillwell/manual-install-doc-fixes1

doc: install/manual-deployment: use sudo on some commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8950 from stiopaa1/osd_passSharedByRef_shouldsharemap

osd/OSD: pass shared pointer by const reference

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8962 from beess/doc-fix1

doc: Fixes three hyphen errors and a syntax error.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8934 from liewegas/wip-bluestore-extent-flags

os/bluestore: drop transient extent flags

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8607 from ceph/wip-15372

msg/async: add missing DelayedDelivery and delay injection

Merge branch 'sharkcz-optional-tcmalloc'

Reviewed-by: Boris Ranto <branto@redhat.com>

fix tcmalloc handling in spec file

- there is no gperftools/tcmalloc on s390(x) yet
- explicitly disable tcmalloc when built without

Signed-off-by: Dan Horák <dan@danny.cz>

doc/rgw: fix hyperlinks which can't work

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/rgw: hyperlink of "Ceph Object Gateway" doesn't work

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: Remove redundant call to get_processors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

osd: remove useless set_role from split completion

This is done by PG::split_into().

Signed-off-by: Sage Weil <sage@redhat.com>

osd: respect shard when setting role on new pgs

This matches PG::start_peering_interval().

Signed-off-by: Sage Weil <sage@redhat.com>

osd: pay attention to shard when setting role

This matches start_peering_interval().

Signed-off-by: Sage Weil <sage@redhat.com>

create and check omap fsid

1. write osd_uuid to omap dir when doing filestore mkfs
2. check if omap fsid matches osd fsid when doing filestore mount
   (if there is no osd_uuid under omap, assume this as match)

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

Added option to deploy omap to a separate device(path)

In the HDD as OSD, SSD as journal test, we saw a great
throughput improvement if moving omap to a SSD device at
randwrite case.

This patch aim to add a config option 'filestore_omap_backend_path'
for users to configurate omap path before deploy.

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

ObjctStore: More accurate disk space usage calculation when considering 'whiteout'/empty/small objects.

This also adds cache protection against potential overflow with removed objects

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

msg/simple/Pipe: avoid infinite loop in Pipe::do_recv()

I found that, sometimes an OSD thread uses 100% CPU after cutting network
between OSD and client. recv(2) in Pipe::do_recv() keeps returning -EAGAIN,
which causes infinite loop. the call trace is:

 Pipe::do_recv (...)
 Pipe::buffered_recv (...)
 Pipe::tcp_read_nonblocking (...)
 Pipe::tcp_read (...)

Pipe::tcp_read() first calls Pipe::tcp_read_wait() to check if data is
avaliable. If there are prefetched data, Pipe::tcp_read_wait() return
immediately. Pipe::buffered_recv() is called, which reads data from the
prefetched data. If prefetched data isn't enough, Pipe::buffered_recv()
calls Pipe::do_recv() to read data from socket. But it's possble that
socket has no data at this time, so Pipe::do_recv() keeps retry.

The fix is simple, just not retry when recv(2) return -EAGAIN.

Fixes: #14120
Signed-off-by: Yan, Zheng <zyan@redhat.com>

show the  total pg on each osd.
ceph pg dump osds
dumped osds in format plain
osdstat kbused kbavail kb hb in hb out pg sum
0 143304204 79623420 234882056 [1,2] [] 330
1 143304204 79623420 234882056 [0,2] [] 295
2 143304148 79623476 234882056 [0,1] [] 303
sum 429912556 238870316 704646168

Fixes:#14183
Signed-off-by: songbaisen song.baisen@zte.com.cn

os/bluestore/KernelDevice: For buffer mode no need do bufferlist::rebuild.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: copy blacklist in better way.

Signed-off-by: song baisen song.baisen@zte.com.cn

mon/PGMonitor: don't get error code overwritten by 0

dump_stuck_pg_stats() method returns either 0 on success or
an negative error code on failure. So it is wrong to reset
"r" to 0.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/PGMonitor: fix inaccurate warn of unscrubbed pgs

1. we automatically perform deep-scrub at osd_deep_scrub_interval,
   not mon_scrub_interval(1week vs. 1day).

2. theoretically a pg can be in a combination state of both unscrubbed
   and un-deep-scrubbed.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/mon/mkfs.sh: clean up kv_backend file

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: remember kv backend type

If it is specified, use it; otherwise, go by the config
option.

Record the type when creating the store.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: {read,write}_meta

These are lifted from ObjectStore.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: move backend instantiation out of ctor

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: fix bug in dump output file

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove bad rocksdb option

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore::sync_entry check for stop in after wait

there is chance that the sync_entry() gets signaled in the
WaitInterval(max_interval) call because of sync_cond.Signal() call after
"stop = true" in umount(), so without this change, sync_entry() will
continue wait until min_interval is reached. this forces umount() to
wait even it has called d_force_sync(), and hence slows down the
progress of umount(). so we need to check for `stop` if we are not
signalled because of `force_sync`.

Fixes: http://tracker.ceph.com/issues/15695
Reported-by: Haomai Wang <haomaiwang@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: Fix SELinux relabel on fedora

The SELinux userspace utilities stopped providing versions when they
switched to CIL language. We need to use a different technique to
relabel the files.

Fixes: #15725
Signed-off-by: Boris Ranto <branto@redhat.com>

rgw: introduce rgw_list_bucket_min_readahead parameter for bucket listing from rados

Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.

If the option "rgw_swift_account_in_url" is being set to true,
both user and subuser IDs will be present in the X-Storage-Url
header generated by the implementation of Swift API's TempAuth.
This patch rectifies this behavior and makes that only the user
identifier will be placed there.

Fixes: http://tracker.ceph.com/issues/15667
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Test-enable XIO messenger and IB/RDMA support

Fixes: http://tracker.ceph.com/issues/15472
Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rados, test: Restrict usage of --prefix in cleanup

As a precaution to using cleanup for mass deletion of other
objects, only allow --prefix which begins with "benchmark_data."

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test rados cleanup command with and without --all

Signed-off-by: David Zafman <dzafman@redhat.com>

rados: Support --all for rados bench cleanup

Remove duplicate code in generate_object_name()
Need to handle multiple different pids during cleanup
Add same message from clean_up_slow() in clean_up()

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Fix ObjBencher clean_up when no prefix specified

By default remove all objects named benchmark_data_{hostname} when
there is no benchmark_last_metadata present.

Fixes: http://tracker.ceph.com/issues/15338
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test rados ls with --all option

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Find ceph command as CEPH_TOOL like rados command

Signed-off-by: David Zafman <dzafman@redhat.com>

debian,rpm: add jq build dependency

Needed since db517ba9176f1ee829453b016b1bd43c6054a555.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: reduce log level for image format 1 warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: do a full make check when --with-tests is given

The check-local target tests the CLI tools only.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop sysvinit bits from relabel_files function

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: replace literal paths with RPM macros

The only place we should write out literal paths is in the RPM scriptlets,
and there only for things that are not installed by this package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop udev/95-ceph-osd-alt.rules

This udev rules file was needed on older RHEL platforms, which are
unsupported as of jewel.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: global replace $RPM_BUILD_ROOT with %{buildroot}

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: put dependencies in alphabetical order

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: put /sbin/ldconfig into -p

This saves approximately 1 second per invocation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop sysvinit-specific dependencies

Obsolete in jewel and above.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: move boost-random dependency to appropriate section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: move unified python-sphinx build dependency

Now that the python-sphinx build dependency is unified, move it
to the proper section of the spec file.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop dead conditionals

This commit drops conditionals that no longer serve any purpose, since
jewel and above do not support the distro versions they are checking for.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop python-argparse dependency

This was only necessary for older (now unsupported) distro versions.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop _with_systemd

Ceph versions jewel and above only support systemd.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop init-ceph.in-fedora.patch

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop python_sitelib/sitearch conditional

This conditional was required to support older versions of RHEL/CentOS that are
no longer supported in infernalis and above.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

fix tcmalloc handling in spec file

- there is no gperftools/tcmalloc on s390(x) yet
- explicitly disable tcmalloc when built without

Merge pull request #8660 from xiexingguo/xxg-wip-docmultisite

doc/rgw: hyperlink of "Ceph Object Gateway" doesn't work

Reviewed-by: luo kexue <luo.kexue@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fixes three hyphen errors and a syntax error.

Signed-off-by: Zhuang Xiaochun <zhuangxc89@163.com>

Merge pull request #8954 from badone/wip-cmake-jj-fix

cmake: Remove redundant call to get_processors

cmake: Remove redundant call to get_processors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc/rgw: fix hyperlinks which can't work

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Some commands also need sudo to run

Included sudo on some commands which need it to run properly and also
fixed a few grammar problems.

Signed-off-by: Bryan Stillwell <bryan.stillwell@twcable.com>

Merge pull request #8834 from liewegas/wip-pg-role

osd: set pg role more carefully

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6421 from xuechendi/wip-omap-out

os/filestore: add option to deploy omap to a separate device(path)

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6664 from ifed01/wip-13848

osd: more accurate disk space usage calculation when considering 'whiteout'/empty/small objects.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6971 from ukernel/jewel-14120

msg/simple/Pipe: avoid infinite loop in Pipe::do_recv()

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7050 from songbaisen/wipshowpg

mon: show the pg count in df output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7872 from majianpeng/bluestore-write-buffer-rebuild

os/bluestore/KernelDevice: For buffer mode no need do bufferlist::rebuid

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8087 from songbaisen/song13

osd: copy blacklist in better way.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8800 from xiexingguo/xxg-wip-fixscrubwarn

mon/PGMonitor: fix inaccurate warn of unscrubbed pgs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8888 from liewegas/wip-mon-kv

mon: enable different backends (e.g., rocksdb)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #8905 from tchaikov/wip-15695

os/FileStore::sync_entry check for stop in after wait

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8923 from ceph/wip-selinux-relabel

rpm: Fix SELinux relabel on fedora

Reviewed-by: Petr Lautrbach <plautrba@redhat.com>

Merge pull request #8756 from Abhishekvrshny/fix-rgw-min-read-ahead

rgw: introduce rgw_list_bucket_min_readahead parameter for bucket list
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8726 from rzarzynski/wip-rgw-xstorageurl-fix

rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.

Fixes: http://tracker.ceph.com/issues/15667
Reviewed-by: Orit Wasserman <owassern@redhat.com>

Merge pull request #8557 from SUSE/wip-15472

Test-enable XIO messenger and IB/RDMA support

Reviewed-by: Roi Dayan <roi.dayan@gmail.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

Test-enable XIO messenger and IB/RDMA support

Fixes: http://tracker.ceph.com/issues/15472
Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8405 from dzafman/wip-15338

common: Fix ObjBencher clean_up when no prefix specified

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8935 from ceph/wip-jq

debian,rpm: add jq build dependency

Reviewed-by: David Zafman <dzafman@redhat.com>

osd/OSD: pass shared pointer by const reference

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #8932 from dillaman/wip-15577

librbd: reduce log level for image format 1 warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rpm: Fix SELinux relabel on fedora

The SELinux userspace utilities stopped providing versions when they
switched to CIL language. We need to use a different technique to
relabel the files.

Fixes: #15725
Signed-off-by: Boris Ranto <branto@redhat.com>

debian,rpm: add jq build dependency

Needed since db517ba9176f1ee829453b016b1bd43c6054a555.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8720 from ceph/wip-13445

rpm: drop support for unsupported distros

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

os/bluestore: drop useless wal check

Length is already trimmed to the current extent at the top of the
loop.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop COW_TAIL extent flag

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop COW_HEAD extent flag

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop UNWRITTEN extent flag

This was a transient flag that only existed in memory, and
muddles the ondisk state vs write logic separation.  Replace it
with a simple range that got allocated.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop dead overlay code

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unwritten handling in _do_read

These never reach disk.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: reduce log level for image format 1 warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.

If the option "rgw_swift_account_in_url" is being set to true,
both user and subuser IDs will be present in the X-Storage-Url
header generated by the implementation of Swift API's TempAuth.
This patch rectifies this behavior and makes that only the user
identifier will be placed there.

Fixes: http://tracker.ceph.com/issues/15667
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Fixes for list-inconsistent-*

Improve rados scrub output
Add epoch for future repair-read feature
Add testing of rados scrub output

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

test: Add testing of new scrub commands in rados

Test list-inconsistent-pg
Test list-inconsistent-obj
Test inconsistent-snapset

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #8916 from linuxbox2/xio-cmake

cmake: <UPPERCASED_NAME>_FOUND in Findxio.cmake

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8768 from cbodley/wip-rgw-admin-zonegroup

radosgw-admin: add missing --zonegroup-id to usage