journal: incorrectly computed object offset within set

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8786 from tchaikov/wip-btrfs-sudo

test: sudo to rm btrfs subvol

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #8994 from theanalyst/fix/15776

 rgw: log name instead of id for SystemMetaObj on failure

Merge pull request #8919 from stiopaa1/log_moveToPrivateOsdService

osd/OSD.h: change some data members to private

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8989 from flyd1005/wip-fix-python-warnings

cleanup: python: remove warnings of 'trailing whitespace' and 'new blank line at EOF'

Merge pull request #8992 from runsisi/wip-fix-dup-keygen

cls_journal: remove duplicated key generation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8991 from emenguy/doc_test-reweight-by-utilization

doc: adding test-reweight-by-utilization documentation

Merge pull request #8984 from stiopaa1/osd_removeUnorderedSet

osd/OSD.h: remove unneeded include file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8097 from aclamk/crushtool-pool-id

crushtool: add ability of precise testing of placement group calculation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8832 from stiopaa1/log_graylogmove

common/Graylog.cc: use std move to avoid copy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8920 from XinzeChi/wip-race-shutdown

osd: fix deadlock in OSD::_committed_osd_maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8922 from runsisi/wip-fix-lockdep-assert

lockdep: fix assertion expression if ran out of lock ids

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8967 from liewegas/wip-15760

osdc/Objecter: upper bound watch_check result

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8357 from liewegas/wip-osd-prestart

osd: update crush_location from ceph-osd on startup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8194 from tanghaodong25/fix_4

os/bluestore/KernelDevice.cc: small fix in buffer flush

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8349 from Aran85/prepare_entry_cleanup

os/filestore: prepare entry cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

osd: create osd id if it does not exist

Most tools create the osd id before trying to start ceph-osd.  Notably,
teuthology does not.  We could fix that, but we would be changing behavior,
as the osd boot will happily create the osd id on the fly for us.  Other
provisioning tools might rely on that behavior.  Instead, just allocate
the id sooner in the process (if necessary).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change osd_crush_initial_weight = 0 to mean weight to 0

Negative now means auto-weight, 0 means weight to 0.  Change the
default accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: update crush_location on startup from ceph-osd

Update the crush location from ceph-osd instead of relying on
kludgey bash in ceph-osd-prestart.sh.  Among other things, this
lets us get accurate statfs information from the ObjectStore
implementation instead of relying on 'df'.

Fixes: http://tracker.ceph.com/issues/15213
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: use cct's crush_location

Keep the observer so that we refresh our copy of the multimap.

Signed-off-by: Sage Weil <sage@redhat.com>

global: init crush_location on daemon start

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: add CrushLocation to cct

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushLocation: add class to manage crush_location

The crush_location can come from an explicitly set config,
a hook, or a simple fabricated default (root=default host=...).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8684 from vuhuong/wip-radosaio-copyout-data-xio

librados: copy out data to users' buffer for xio

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #8759 from xiexingguo/xxg-wip-fixobjecterrace

osdc/Objecter: fix race condition for sortbitwise flag detection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8826 from liewegas/wip-bluestore-bitmap-freelist

os/bluestore: bitmap-based freelist using merge operator

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: drop unnecessary spacing in rgw zg init log

Dropping unneeded space when we're printing the failed reading zg info
message

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: log name instead of id for SystemMetaObj on failure

Currently if we fail to read a SystemMetaObj we try to log the
MetaObject id, however this will not be set mostly as read_id has
failed, so we end up logging an empty id, changing this to log
the object name instead

Fixes: http://tracker.ceph.com/issues/15776
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

adding test-reweight-by-utilization documentation

Signed-off-by: Etienne Menguy <etienne.menguy@corp.ovh.com>

cls_journal: remove duplicated key generation

Signed-off-by: runsisi <runsisi@zte.com.cn>

cleanup: python: remove warnings of 'new blank line at EOF'

When applying ceph patches, some warnings reported, e.g.
qa/workunits/mon/caps.py:367: new blank line at EOF.

Signed-off-by: Li Peng <lip@dtdream.com>

cleanup: python: remove warnings of 'trailing whitespace'

When applying ceph patches, some warnings reported, e.g.
doc/scripts/gen_state_diagram.py:99: trailing whitespace.

Signed-off-by: Li Peng <lip@dtdream.com>

osd/OSD.h: remove unneeded include file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

osd/OSD.h: change some data members to private

Move some data members of the OSDService from public to private access
specifier.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #8912 from markhpc/wip-promote-throttle-fix

osd: fix reversed promote throttle default parameters

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8902 from xiexingguo/xxg-wip-cheanup-osdmon

mon/OSDMonitor: kill dead code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8290 from yangdongsheng/chain_xattr

Fix unittest for chain_xattr.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #8173 from ceph/buffer-api-visibility

buffer: fix iterator_impl visibility through typedef

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8289 from majianpeng/bluestore-zero-clear-tail-cache

os/bluestore/BlueStore: For _zero, check whether clear tail_bl.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8977 from liewegas/wip-ceph-tool

qa/workunits/rados/test_rados_tool: fix copy paste error

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8388 from dreamhost/wip-buildfix-mds-test

mds: build fix for LIBMDS consumer.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/rados/test_rados_tool: fix copy paste error

Signed-off-by: Sage Weil <sage@redhat.com>

tests/rbd: add option to leave behind fsx test data

tests/rbd: add option to leave behind fsx test data

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests/rbd: add option to leave behind fsx test data

Previously, fsx deleted all test data upon successful completion. Add
an option to leave the data behind for further analysis.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #8936 from bstillwell/manual-install-doc-fixes1

doc: install/manual-deployment: use sudo on some commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8950 from stiopaa1/osd_passSharedByRef_shouldsharemap

osd/OSD: pass shared pointer by const reference

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8962 from beess/doc-fix1

doc: Fixes three hyphen errors and a syntax error.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8934 from liewegas/wip-bluestore-extent-flags

os/bluestore: drop transient extent flags

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osdc/Objecter: upper bound watch_check result

This way we always return a safe upper bound on the amount of time
since we did a check.  Among other things, this prevents us from
returning a value of 0, which is confusing.

Fixes: http://tracker.ceph.com/issues/15760
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8607 from ceph/wip-15372

msg/async: add missing DelayedDelivery and delay injection

Merge branch 'sharkcz-optional-tcmalloc'

Reviewed-by: Boris Ranto <branto@redhat.com>

fix tcmalloc handling in spec file

- there is no gperftools/tcmalloc on s390(x) yet
- explicitly disable tcmalloc when built without

Signed-off-by: Dan Horák <dan@danny.cz>

doc/rgw: fix hyperlinks which can't work

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/rgw: hyperlink of "Ceph Object Gateway" doesn't work

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: Remove redundant call to get_processors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

osd: remove useless set_role from split completion

This is done by PG::split_into().

Signed-off-by: Sage Weil <sage@redhat.com>

osd: respect shard when setting role on new pgs

This matches PG::start_peering_interval().

Signed-off-by: Sage Weil <sage@redhat.com>

osd: pay attention to shard when setting role

This matches start_peering_interval().

Signed-off-by: Sage Weil <sage@redhat.com>

create and check omap fsid

1. write osd_uuid to omap dir when doing filestore mkfs
2. check if omap fsid matches osd fsid when doing filestore mount
   (if there is no osd_uuid under omap, assume this as match)

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

Added option to deploy omap to a separate device(path)

In the HDD as OSD, SSD as journal test, we saw a great
throughput improvement if moving omap to a SSD device at
randwrite case.

This patch aim to add a config option 'filestore_omap_backend_path'
for users to configurate omap path before deploy.

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

ObjctStore: More accurate disk space usage calculation when considering 'whiteout'/empty/small objects.

This also adds cache protection against potential overflow with removed objects

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

msg/simple/Pipe: avoid infinite loop in Pipe::do_recv()

I found that, sometimes an OSD thread uses 100% CPU after cutting network
between OSD and client. recv(2) in Pipe::do_recv() keeps returning -EAGAIN,
which causes infinite loop. the call trace is:

 Pipe::do_recv (...)
 Pipe::buffered_recv (...)
 Pipe::tcp_read_nonblocking (...)
 Pipe::tcp_read (...)

Pipe::tcp_read() first calls Pipe::tcp_read_wait() to check if data is
avaliable. If there are prefetched data, Pipe::tcp_read_wait() return
immediately. Pipe::buffered_recv() is called, which reads data from the
prefetched data. If prefetched data isn't enough, Pipe::buffered_recv()
calls Pipe::do_recv() to read data from socket. But it's possble that
socket has no data at this time, so Pipe::do_recv() keeps retry.

The fix is simple, just not retry when recv(2) return -EAGAIN.

Fixes: #14120
Signed-off-by: Yan, Zheng <zyan@redhat.com>

show the  total pg on each osd.
ceph pg dump osds
dumped osds in format plain
osdstat kbused kbavail kb hb in hb out pg sum
0 143304204 79623420 234882056 [1,2] [] 330
1 143304204 79623420 234882056 [0,2] [] 295
2 143304148 79623476 234882056 [0,1] [] 303
sum 429912556 238870316 704646168

Fixes:#14183
Signed-off-by: songbaisen song.baisen@zte.com.cn

os/bluestore/KernelDevice: For buffer mode no need do bufferlist::rebuild.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: copy blacklist in better way.

Signed-off-by: song baisen song.baisen@zte.com.cn

mon/PGMonitor: don't get error code overwritten by 0

dump_stuck_pg_stats() method returns either 0 on success or
an negative error code on failure. So it is wrong to reset
"r" to 0.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/PGMonitor: fix inaccurate warn of unscrubbed pgs

1. we automatically perform deep-scrub at osd_deep_scrub_interval,
   not mon_scrub_interval(1week vs. 1day).

2. theoretically a pg can be in a combination state of both unscrubbed
   and un-deep-scrubbed.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/mon/mkfs.sh: clean up kv_backend file

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: remember kv backend type

If it is specified, use it; otherwise, go by the config
option.

Record the type when creating the store.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: {read,write}_meta

These are lifted from ObjectStore.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: move backend instantiation out of ctor

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonitorDBStore: fix bug in dump output file

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove bad rocksdb option

Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore::sync_entry check for stop in after wait

there is chance that the sync_entry() gets signaled in the
WaitInterval(max_interval) call because of sync_cond.Signal() call after
"stop = true" in umount(), so without this change, sync_entry() will
continue wait until min_interval is reached. this forces umount() to
wait even it has called d_force_sync(), and hence slows down the
progress of umount(). so we need to check for `stop` if we are not
signalled because of `force_sync`.

Fixes: http://tracker.ceph.com/issues/15695
Reported-by: Haomai Wang <haomaiwang@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: Fix SELinux relabel on fedora

The SELinux userspace utilities stopped providing versions when they
switched to CIL language. We need to use a different technique to
relabel the files.

Fixes: #15725
Signed-off-by: Boris Ranto <branto@redhat.com>

rgw: introduce rgw_list_bucket_min_readahead parameter for bucket listing from rados

Signed-off-by: Abhishek Varshney <abhishek.varshney@flipkart.com>

rgw: fix printing wrong X-Storage-Url in Swift's TempAuth.

If the option "rgw_swift_account_in_url" is being set to true,
both user and subuser IDs will be present in the X-Storage-Url
header generated by the implementation of Swift API's TempAuth.
This patch rectifies this behavior and makes that only the user
identifier will be placed there.

Fixes: http://tracker.ceph.com/issues/15667
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Test-enable XIO messenger and IB/RDMA support

Fixes: http://tracker.ceph.com/issues/15472
Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rados, test: Restrict usage of --prefix in cleanup

As a precaution to using cleanup for mass deletion of other
objects, only allow --prefix which begins with "benchmark_data."

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test rados cleanup command with and without --all

Signed-off-by: David Zafman <dzafman@redhat.com>

rados: Support --all for rados bench cleanup

Remove duplicate code in generate_object_name()
Need to handle multiple different pids during cleanup
Add same message from clean_up_slow() in clean_up()

Signed-off-by: David Zafman <dzafman@redhat.com>

common: Fix ObjBencher clean_up when no prefix specified

By default remove all objects named benchmark_data_{hostname} when
there is no benchmark_last_metadata present.

Fixes: http://tracker.ceph.com/issues/15338
Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test rados ls with --all option

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Find ceph command as CEPH_TOOL like rados command

Signed-off-by: David Zafman <dzafman@redhat.com>

debian,rpm: add jq build dependency

Needed since db517ba9176f1ee829453b016b1bd43c6054a555.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: reduce log level for image format 1 warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: do a full make check when --with-tests is given

The check-local target tests the CLI tools only.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop sysvinit bits from relabel_files function

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: replace literal paths with RPM macros

The only place we should write out literal paths is in the RPM scriptlets,
and there only for things that are not installed by this package.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop udev/95-ceph-osd-alt.rules

This udev rules file was needed on older RHEL platforms, which are
unsupported as of jewel.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: global replace $RPM_BUILD_ROOT with %{buildroot}

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: put dependencies in alphabetical order

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: put /sbin/ldconfig into -p

This saves approximately 1 second per invocation.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop sysvinit-specific dependencies

Obsolete in jewel and above.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: move boost-random dependency to appropriate section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: move unified python-sphinx build dependency

Now that the python-sphinx build dependency is unified, move it
to the proper section of the spec file.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop dead conditionals

This commit drops conditionals that no longer serve any purpose, since
jewel and above do not support the distro versions they are checking for.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop python-argparse dependency

This was only necessary for older (now unsupported) distro versions.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop _with_systemd

Ceph versions jewel and above only support systemd.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: drop init-ceph.in-fedora.patch

Signed-off-by: Nathan Cutler <ncutler@suse.com>