Merge pull request #23647 from wjwithagen/wjw-fix-pidfile-err

common: be more informative if set PID-file fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23643 from tchaikov/wip-cmake-dmclock

dmclock, cmake: sync up with ceph/dmclock, dmclock related cleanups

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #23624 from david-z/wip-fix-mgr-fs-status

mgr/status: Add standby-replay MDS ceph version

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23576 from ifed01/wip-ifed-fix-perf-counters

os/bluestore: bluestore_buffer_hit_bytes perf counter doesn't reset

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21983 from chardan/jfw-wip-libradosstriper_ultimate-final-battle

libradosstriper: conditional compile

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23689 from tchaikov/wip-run-make-check-more-aio

run-make-check: increase fs.aio-max-nr to 1048576

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #23675 from tserlin/tserlin-require-fuse

debian/control: require fuse for ceph-fuse

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23655 from tchaikov/wip-silence-Wsign-compare-warnings

test: silence -Wsign-compare warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

run-make-check: increase fs.aio-max-nr to 1048576

the solution was suggested by Yingxin Cheng. otherwise we could have
EAGAIN returned by io_setup(2).

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23630 from wjwithagen/wjw-fix-rbd-ggate-kldload

test/rbd: rbd_ggate test improvements

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23684 from tchaikov/wip-doc-balancer

doc/rados/operations: add balancer.rst to TOC

Reviewed-by: John Spray <john.spray@redhat.com>

doc/rados/operations: add balancer.rst to TOC

this helps user to find the balancer feature, and
also addresses the FTBFS of doc like

Warning, treated as error:
/home/jenkins-build/build/workspace/ceph-pr-docs/doc/rados/operations/balancer.rst:document
isn't included in any toctree

this doc was moved from doc/mgr/balancer.rst to
doc/rados/operations/balancer.rst in
a78d600e6b368255096d6f0bbee8e53553307118

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23642 from tchaikov/wip-crimson-msgr

cmake,crimson/net: add keepalive support, and enable unittest_seastar_messenger in "make check"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/mon: no need to include OSD.h

what it needs is the definition of `struct statfs`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,test: update mClockPriorityQueue to accomodate the change in dmc

* cost is required to be a positive number
* PullPriorityQueue's construct uses an enum instead of a bool for
  specifying the behavior when request breaches the limit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

dmclock: pick up tag.reservation calc fix and cmake changes

see
- ceph/dmclock#58
- ceph/dmclock#60
- ceph/dmclock#61

Fixes: http://tracker.ceph.com/issues/26998
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: dmclock related cleanups

* do not include_directories() globally. use dmclock::dmclock's
  interface properties instead
* add_subdirectory(dmclock) instead of add_subdirectory(dmclock/src)
  for including its
* set dmclock_TEST instead of including the subdirectories under
  dmclock/
* objclass/class_api.cc includes PrimaryLogPG.h, which in turn includes
  indirect_intrusive_heap.h, so it'd better to compile class_api.cc as
  a part of osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #23635 into master

* refs/pull/23635/head:
mds: use monotonic clock in beacon
mds: simplify beacon init

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #21114 from colletj/colletj_rbd_image_timestamps

librbd: add rbd image access/modified timestamps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: add image access/last modified timestamps

Add access and modify timestamps and associated tests
to RBD images.

Access (resp. modify) timestamps are updated on
read (resp. write) operations. A configurable throttling
mechanism is implemented (default to 60s).

Signed-off-by: Julien Collet <julien.collet@cern.ch>

debian/control: require fuse for ceph-fuse

This is the Ubuntu equivalent for:

https://github.com/ceph/ceph/pull/17120
http://tracker.ceph.com/issues/21057

Installing "ceph-fuse" should pull in the "fuse" package automatically.

Signed-off-by: Thomas Serlin <tserlin@redhat.com>

Merge pull request #23649 from trociny/wip-26939

librbd: journaling unable request can not be sent to remote lock owner

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23311 from Songweibin/wip-fix-du-display

tools/rbd: minor fixes for rbd du display

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23638 from Songweibin/wip-filter-user-snapshots

rbd: filter out group/trash snapshots from snap_list

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23615 from dmick/wip-ceph-mgr-deb

debian/rules: fix ceph-mgr .pyc files left behind

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23606 from ifed01/wip-ifed-stupid-get-fragmentation-fix

os/bluestore: fix assertion in StupidAllocator::get_fragmentation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23587 from xiexingguo/wip-recover-order

osd/PrimaryLogPG: optimize recover order

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23558 from liewegas/wip-mgr-always-on

mgr: keep status, balancer always on

Merge pull request #23573 from rzarzynski/wip-msg-26937

msg: entity_addr_t::parse doesn't do memset(this, 0, ...) for clean-up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23641 from wjwithagen/wjw-fix-pthread_get_name_np

include/compat.h: make pthread_get_name_np work when available

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23658 into master

* refs/pull/23658/head:
qa: use cephfs standard configs for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test: silence -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23657 from trociny/wip-wsign-compare

test/librbd: fix compiler -Wsign-compare warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23591 from tchaikov/wip-lockless-auth

auth,common: add lockless auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23567 from tchaikov/wip-cmake-cleanup

cmake: fix Debug build `WITH_SEASTAR=ON`

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

common: be more informative if set PID-file fails

The errors returned by fcntl(...F_SETLK...) can be:
 -  -1 for any odd error
 -  EAGAIN if locking does not work because the file is already locked.

Differentiate in these to be more informative on what is going on.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23631 from cyx1231st/test-seastar-config

crimson/common: apply config changes also on shard.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/test: improve test_config with observers

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #23582 from badone/wip-remove-upstart

Remove upstart files and references

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/rbd: Improve/update rbd-ggate

* check if geom_gate can be loaded before doing the actual tests

Otherwise continuing does not make sense.
Major reason for this problem is due to mismatch between
kernel and module versions.

* After FreeBSD kernevel 1200078 ggate resizing is possible

So set the flag that resizing can be tested

* Only sudo commands that really need sudo

rbd-ggate list is available in regular user mode

* be a bit more verbose during testing and list the test purpose

* list-mapped is an option in rbd-nbd, not (yet) in rbd-ggate

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: should use lowercase for component name in FindSanitizers.cmake

and s/undefined-behavior/undefined_behavior/ to be compatible with
seastar

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: exercise keepalive also

allow one to pass options to unittest_seastar_messenger for specifying
- verbose
- keepalive-ratio
- rounds

so we can have better coverage of crimson/messenger. and make it less
chatty unless it's asked to do so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: add keepalive support to Connection

* do not mix buffered writes and zero-copy writes.

/root/ceph/src/seastar/include/seastar/core/iostream-impl.hh:114:
seastar::future<> seastar::output_stream::write(seastar::net::packet)
[with CharType = char]: Assertion `!_end && "Mixing buffered writes and
zero-copy writes not supported yet"' failed.
Aborting on shard 0.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add unittest_seastar_messenger to "make check"

as it's relatively stable at the moment. and as suggested by @cyx1231st
.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: use HAVE_FEATURE instead

so we can check the marker the bit's incarnation also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23654 from cbodley/wip-crypto-assert-always

crypto: use ceph_assert_always for assertions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22668 into master

* refs/pull/22668/head:
mds: reorganize and curate perf log counters
mds: show session load average in dump session
mds: allow request load average decay rate configurable via admin socket
mds: decay counter for tracking request load average
mds: add perf counters to track current open/stale sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23438 into master

* refs/pull/23438/head:
client: fix choose_target_mds for requests that do name lookup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23467 into master

* refs/pull/23467/head:
cephfs-journal-tool: enable purge_queue journal's event commands

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23529 into master

* refs/pull/23529/head:
mon: fix idempotent FS creation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #23543 into master

* refs/pull/23543/head:
tools/cephfs: make cephfs-data-scan prints the max used ino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23584 into master

* refs/pull/23584/head:
mds: don't modify filepath when printing
mds: hold slave request refernce when dumping MDRequestImpl

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23589 into master

* refs/pull/23589/head:
mds: more description for failed authpin
mds: cleanup CDir freezing/frozen tree check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use monotonic clock in beacon

Also update other parts of MDS which interact.

Fixes: http://tracker.ceph.com/issues/26959
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use cephfs standard configs for kclient

Specifically, qa/cephfs/begin.yaml.

Fixes: http://tracker.ceph.com/issues/26967
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/librbd: fix compiler -Wsign-compare warnings

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23315 from Devp00l/wip-clean-notification-messages

mgr/dashboard: Cleaner notifications

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23637 from noahdesu/qa-mgr-mod-path

qa: fix manager module paths

crypto: use ceph_assert_always for assertions

use ceph_assert_always() to enforce this comment in ceph_crypto.h:

This assert MUST NOT be compiled out, even on non-debug builds.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23639 from trociny/wip-24910

librbd: potential race on image create request complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23110 from Songweibin/wip-rbd-snap-rollback

librbd: use object map when doing snap rollback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: journaling unable request can not be sent to remote lock owner

Fixes: http://tracker.ceph.com/issues/26939
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23609 from ErwanAliasr1/evelu-ccache

run-make-check: Showing configuration before the build

crimson/common: apply config changes also on shard.0

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Remove upstart files and references

Nautilus is not supported on any distros still running upstart so these
are redundant.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #23601 from tchaikov/wip-pybind-freebsd-clang

pybind: do not check MFLAGS

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #23593 from jtlayton/wip-jlayton-26948

librados: add a rados_omap_iter_size function

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23588 from gmayyyha/ceph-df-json

mon/PGMap: command 'ceph df -f json' output add total_percent_used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23563 from yaozongyou/mon-ping-timeout

mon/MonClient: use mon_client_ping_timeout during ping_monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10305 from majianpeng/memdb-perfcounter

kv/MemDB: add perfcounter.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23628 from tchaikov/wip-cmake-gtest-cleanup

cmake: link against gtest in a better way

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

core: make pthread_get_name_np work when available

FreeBSD will have a pthread_get_name_np() in libpthread
Code for that was submitted

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cpp-btree: do not #define NDEBUG

use no_debug instead, the NDEBUG macro pollutes all source files
including this header.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados: do not use assert() for its side effect

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: potential race on image create request complete

Fixes: http://tracker.ceph.com/issues/24910
Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/tasks/mgr/test_module_selftest: use hello instead of status for disbled command test

Signed-off-by: Sage Weil <sage@redhat.com>

doc: move mgr/balancer to rados/operations/balancer

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: keep status, balancer always on

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23616 into master

* refs/pull/23616/head:
doc/releases/schedule: note Jewel EOL date

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23618 into master

* refs/pull/23618/head:
doc: updated infernalis release reference link
doc: added ref label for infernalis release notes
doc: used reST label for ceph-deploy reference
doc: added reference label for ceph-deploy page
doc: updated cross referencing locations link

Reviewed-by: Sage Weil <sage@redhat.com>

test: Fix test to use -gt instead of creating an empty file "0"

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #23636 into master

* refs/pull/23636/head:
messages: fix get_opname return type

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23620 from tchaikov/wip-dpdk-cmake

cmake: add dpdk::dpdk if dpdk is built or found

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: fix error processing images which have non-user snapshots

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: fix error purging non-user snapshots

Fixes:
  [root@ ~]# rbd snap rm img1@snap1
  [root@ ~]# rbd snap ls img1 -a
  SNAPID NAME                                 SIZE    TIMESTAMP                NAMESPACE
       4 f2e82bd1-e2ff-4a6b-aaef-5a12a2b23a30 100 MiB Sat Aug 18 08:48:34 2018 trash (snap1)
  [root@ ~]# rbd snap purge img1
  Removing all snapshots: 0% complete...failed.
  rbd: removing snaps failed: (2) No such file or directory

Signed-off-by: songweibin <song.weibin@zte.com.cn>

qa/ceph_objectstore_tool.py: set mgr module path

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

qa/standalone/ceph-helpers.sh: fix mgr module path

callers of get_python_path were not passing in a $1 parameter, so
ceph_lib was an empty string resulting in an invalid path to the built
cython modules. assume this is called from the `lib` parent directory.

pass path to the manager modules when starting ceph-mgr.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

messages: fix get_opname return type

It cannot be a constexpr.

Introduce-by: 00b867fae2e8e912119fc47348dee78631c6f66b / #23623
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23574 from cbodley/wip-26895

rgw multisite: bucket sync transitions back to StateInit on OP_SYNCSTOP

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #23575 into master

* refs/pull/23575/head:
tools/cephfs-shell:pep8 code clean
tools/cephfs-shell:resolved removing files using pattern inside sub-directories
tools/cephfs-shell:changed output representation for the commands cwd and ls
tools/cephfs-shell:resolved removing directories using pattern inside sub-directories

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18096 into master

* refs/pull/18096/head:
os/bluestore: factor VDO into bluestore's 'allocated' value
os/filestore: estimate omap_allocated
os/ObjectMap: add get_db() accessor
osd/OSDMap: include data, omap in 'osd df' output
os/bluestore: report omap and internal metadata usage
os/bluestore/BlueFS: drop get_fs_usage(); add get_used()
kv: add per-prefix space estimation
osd: decouple statfs update from hb peers, pg count
osd: drop useless OSDService::update_osd_stat()
osd/osd_types: separate accounting for data, omap, and metadata
osd: re-add get_num_pgs()

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

tools/cephfs-shell:pep8 code clean

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:resolved removing files using pattern inside sub-directories

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:changed output representation for the commands cwd and ls

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

tools/cephfs-shell:resolved removing directories using pattern inside sub-directories

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>

mds: simplify beacon init

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23623 from tchaikov/wip-inline-constants

messages: define HEAD_VERSION and COMPAT_VERSION inlined

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

messages: define HEAD_VERSION and COMPAT_VERSION inlined

otherwise we could have ../lib/libceph-common.so.0: undefined reference
to `MOSDPGNotify::HEAD_VERSION

when the compiler tries to be smart and pass them by reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/status: Add standby-replay MDS ceph version

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #23419 from tspmelo/fix-26856

mgr/dashboard: Add decorator to skip parameter encoding

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>