osd/PGPeeringEvent: note mon- vs peer-initiated pg creates

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch peering events (part 2)

This actually puts the remaining peering events into fast dispatch.  The
only remaining event is the pg create from the mon.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch peering events (part 1)

This is a big commit that lays out the infrastructure changes to fast
dispatch the remaining peering events.  It's hard to separate it all out
so this probably doesn't quite build; it's just easier to review as a
separate patch.

- lock ordering for pg_map has changed:
  before:
    OSD::pg_map_lock
      PG::lock
        ShardData::lock

  after:
    PG::lock
      ShardData::lock
        OSD::pg_map_lock

- queue items are now annotated with whether they can proceed without a
pg at all (e.g., query) or can instantiate a pg (e.g., notify log etc).

- There is some wonkiness around getting the initial Initialize event to
a newly-created PG.  I don't love it but it gets the job done for now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: queue null events without PG lock

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move part of wake_pg_waiters into helper

We'll need this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use MTrim peering event for trimming

This is simpler and cleaner than handling log trimming as a special case.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fast dispatch backfill and recovery reservation events

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move M{Backfill,Recovery}Reserve event logic into message

Better encapsulation!

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPeeringOp: add

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move peering event type out of PG class

We will create these directly from peering Messages shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: keep epoch, not map ref, of last osdmap for lsat persisted epoch

No need to pin the map in memory!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: remove old update_store_on_load()

This isn't needed post-luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge tag 'v13.0.2'

v13.0.2

Merge PR #21180 into master

* refs/pull/21180/head:
vstart_runner: examine check_status before error

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20460 from colletj/v1_image_creation_disallow

librbd: disallow creation of v1 image format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21202 from tchaikov/wip-rbd-replay

rbd-replay: remove boost dependency

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21142 from dragonylffly/wip-fix-ebusy

rbd-nbd: fix ebusy when do map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21157 from trociny/wip-23526

journal: limit number of appends sent in one librados op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: remove boost dependency

quite a few facilies are now available in standard library now after we
switched to C++17.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #20855 into master

* refs/pull/20855/head:
client: add the fuse parameter max_write

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21096 into master

* refs/pull/21096/head:
pybind/cephfs: added comments to cephfs.pyx

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21183 from neha-ojha/wip-minor-fix-perf-suite

qa/suites/perf-basic: add desc regarding test machines

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #21075 from ceph/wip-yuriw-try-crontab

test:qa:infra - teuthology crontab items as of 3/27/18

qa/suites/perf-basic: add desc regarding test machines

Signed-off-by: Neha Ojha <nojha@redhat.com>

journal: limit number of appends sent in one librados op

Fixes: http://tracker.ceph.com/issues/23526
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21165 from Songweibin/wip-drop-redefine-func

librbd: minor code cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

vstart_runner: examine check_status before error

Issue found by Rishabh Dave. Fix suggested by John Spray.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

CMakeLists: v13.0.2

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21143 from wjwithagen/wjw-ipaddr-ipv6

common: FreeBSD wants the correct struct selection for ipv6

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21136 from kungf/io_events

bluestore: fix aio_t::rval  type

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20820 from guzhongyan/nodels-improve

mon: node ls improvement

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20298 from markhpc/wip-osd-check_full

osd: don't process ostream strings when not debugging.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #21159 from tchaikov/wip-no-more-use-wheel

src: s/pip --use-wheel/pip/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

pybind/rbd: normalize errno exception

The code meant to do so but actually it does not ...

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: return error when r < 0

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: drop a duplicate definition function

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src: s/--use-wheel//

--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #20132 into master

* refs/pull/20132/head:
qa/cephfs: update TestDamage for open file table
mds: allow storing open file table in multiple omaps
mds: differentiate Anchor types to clarify purpose
mds: add perf counter for 'open ino' operation
mds: protect open file table against partial omap update
mds: add dirfrags whose child inodes have caps to open file table
mds: don't try prefetching destroyed inodes
mds: don't try opening inodes that haven't been created
mds: don't re-requeue open files to head of log
mds: use open file table to speed up mds recovery
mds: introduce open file table
mds: track how many clients/mds want caps for each inode
mds: cleanup MDCache::opening_inodes access
mds: cleanup CInode/CDir states definition

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21147 from rjfd/wip-dashboard-fix-notification-queue-test

mgr/dashboard: notification queue: fix priority tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: notification queue: fix priority tests

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20979 from ceph/wip-cd-smoke-fix

qa/tests: use ceph-deploy stable branch for single node tests

qa/tests: use stable branch for single node tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

common: FreeBSD wants the correct struct selection for ipv6

Lets see if this also works for Linux

Fixes: http://tracker.ceph.com/issues/21813
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd-nbd: fix ebusy when do map

When doing rbd-nbd map, if the Ceph service is not available,
the codes will wait on rados.connect(), unless killing the process.
In that case, the close_nbd logic is skipped with NBD_CLEAR_SOCK ioctl
not called. On the CentOS 7 kernel, it leaves nbd->file not cleared, which
causes the subsequent map requests return EBUSY, this patch fixes it
by connecting Ceph first prior to calling NBD_SET_SOCK ioctl

Fixes: http://tracker.ceph.com/issues/23528
Signed-off-by: Li Wang <laurence.liwang@gmail.com>

librbd/cli-integration: Permit V1 image creation

Signed-off-by: Julien COLLET <julien.collet@cern.ch>

Merge pull request #18317 from hitoshikamei/master-rbd

rbd: eager-thick provisioning support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20868 from shun-s/fix-same-cookie

librbd: fix refuse to release lock when cookie is the same at rewatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20935 from trociny/wip-get-name

librbd: add API function to get image name

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: change the type of aio_t:res to long

io_event::res was assigned to io_t::rval, but io_event::res
is long, if rval is int, this will cause overflow.

Fixes: http://tracker.ceph.com/issues/23527
Signed-off-by: kungf <yang.wang@easystack.cn>

Merge pull request #21117 from smithfarm/wip-23504

tests: remove TestPGLog ASSERT_DEATH test

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20927 into master

* refs/pull/20927/head:
doc: use actual entity address for clarity
doc: make minor grammatical rectifications

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21131 from trociny/wip-23502

qa/suites/rbd: set qemu task time_wait param

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/OSD: make dout in _check_full conditional

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #20983 into master

* refs/pull/20983/head:
ceph_authtool: add mode option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph_authtool: add mode option

We now have the ability to set the keyring file mode with the help of
'--mode MODE'. The mode needs to be specified in octal using the format:
0600.

Closes: http://tracker.ceph.com/issues/23513
Signed-off-by: Sébastien Han <seb@redhat.com>

qa/suites/rbd: set qemu task time_wait param

so workloads qemu_dynamic_features.sh and qemu_rebuild_object_map.sh,
which check if qemu is finished with periodicity 60 sec, have enough
time to detect this before the rbd image is removed.

Fixes: https://tracker.ceph.com/issues/23502
Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/tasks/qemu: add a parameter to wait for workloads detect qemu finished

In the case when a workload needs to detect qemu finished by running a
check with a periodicity of N sec it needs to set time_wait to 2 * N
in order to avoid races on finish.

Signed-off-by: Mykola Golub <mgolub@suse.com>

test:qa:infra - teuthology crontab items as of 3/27/18

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21121 from tchaikov/wip-no-relicensing

msg/async/dpdk: remove xsky copyright and LGPL copying

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #21105 from ashitakasam/wip-bug

rgw: comparison between signed and unsigned integer expressions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21074 from trociny/wip-librados_test_stub-append

test/librados_test_stub: always create copy of buffers passed to operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21116 from trociny/wip-23499

qa/workunits/rbd: remove sanity check in test_admin_socket.sh

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async/dpdk: remove xsky copyright and LGPL copying

it's not legitimate to relicense source code without original author's
permission. some of the source files are copied from seastar project
with minor changes. so xsky cannot claim the copyright of them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21107 from tchaikov/wip-pg-stat-cap-trimq

osd: cap snaptrimq_len at 2^32

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21106 from tchaikov/wip-compressor-zstd

compressor: add zstd back

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21097 from ceph/wip-ifed-fix-reclaim-cap-bug

os/bluestore: fix broken cap in _balance_bluefs_freespace()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20556 from yanghonggang/master

lttng: Trace rgw data transfer, bi entry and object header update processes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tests: remove TestPGLog ASSERT_DEATH test

Fixes: http://tracker.ceph.com/issues/23504
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/rbd: remove sanity check in test_admin_socket.sh

Fixes: https://tracker.ceph.com/issues/23499
Signed-off-by: Mykola Golub <mgolub@suse.com>

osd: cap snaptrimq_len at 2^32

(uint64_t)(1 << 31) evaluates to 0xffffffff80000000 instead of 0x80000000,
because the converted signed number is sign-extended.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21104 from neha-ojha/wip-minimal-perf-suite

qa/suites: add minimal performance suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #21092 from runsisi/wip-osdmon-log

mon/OSDMonitor: logging non-active osd id when handling osd beacon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21091 from runsisi/wip-elector-cleanup

mon/Elector: remove unused member fields start_stamp and ack_stamp

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21085 from xiexingguo/wip-kill-p2-macros

intarith: get rid of  P2* and ROUND_UP* macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21082 from zhsj/fix-java9

cmake: bump target jdk to 1.7

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #21062 from xiexingguo/wip-hobject-cmp

common/hobject: compare two objects' key directly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20955 from liewegas/wip-mgr-mkfs-grace

mon: double mon_mgr_mkfs_grace from 60s -> 120s

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20862 from wido/ipv6-local-address

common/ipaddr: Do not select link-local IPv6 addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21081 from majianpeng/rbd-remove-unused-func

osdc: remove unused function.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

compressor: add zstd back

it's a regression introduced by 16c3552a

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: comparison between signed and unsigned integer expressions

Fixes: Change the data type of 'c' from int to size_t in
src/rgw/rgw_civetweb.cc RGWCivetWeb::read_data

Signed-off-by: ashitakasam <694240887@qq.com>

qa/suites: add minimal performance suite

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #20040 from liewegas/wip-snapmapper

osd: make snapmapper warn+clean up instead of assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/cephfs: added comments to cephfs.pyx
Signed-off-by: Pavani Rajula<rpavani1998@gmail.com>

os/bluestore: fix broken cap in _balance_bluefs_freespace()

1 << 31 is cast to signed int that results in huge value when casting to
uint64_t making min() call ineffective.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21071 from mkoderer/vstart-fix

vstart.sh: Fix help text in vstart.sh

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20961 from smithfarm/wip-dashboard-rst

doc: mgr/dashboard.rst: mention ceph.conf and ceph mgr services

Reviewed-by: Ricardo Dias <rdias@suse.com>

osdc: remove unused function.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

vstart.sh: Fix help text

Signed-off-by: Marc Koderer <marc.koderer@sap.com>

Merge pull request #21077 from Ashid6/master

doc: typo fix

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20660 from Leeshine/wip-mon-op-tracker

mon,mgr: make  osd_metric more popular and report slow ops to mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21057 from tchaikov/wip-22114

mon: mark mgr reports as no_reply

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

doc: typo fix

Signed-off-by: Ashita Dashottar <AshitaDashottar6@gmail.com>

cmake: bump target jdk to 1.7

jdk 1.7 was released on 2011. For CentOS 6/7, both have openjdk
1.8. For Debian 8(jessie), Ubuntu 14.04(trusty) have openjdk 1.7.
These are the oldest distributions officially built for during
luminous release.

Since jdk 1.9/9, the minimal supported version is 1.6/6.
jdk 1.0/10(recently released) only supports 1.7/7.
http://openjdk.java.net/jeps/182

Fixes: http://tracker.ceph.com/issues/23458
Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>

doc: mgr/dashboard.rst: adjust TCP port wording

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon/OSDMonitor: logging non-active osd id when handling osd beacon

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #20976 from sebastian-philipp/dashboard-restcontroller-default-get

mgr/dashboard: Fix objects named `default` are inaccessible

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #20870 from rjfd/wip-dashboard-tasks

mgr/dashboard: asynchronous task support

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20884 from zouaiguo/wip_for_mgr_rbd_info

mgr/dashboard: add image id to mgr rbd info instead of block_name_prefix

Reviewed-by: Ricardo Dias <rdias@suse.com>

intarith: get rid of ROUND_UP* macros

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>