Merge pull request #19418 from tchaikov/wip-gcc-7-on-xenial

install-deps.sh, cmake: use GCC-7 on xenial also

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19402 from b-ranto/wip-fix-arm-build

test/librbd: Do not instantiate TrimRequest template class

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19437 from trociny/wip-mock-valgrind

test/rbd_mirror: "use of uninitialised value" valgrind warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19253 from oritwas/wip-rgw-22124

rgw: reshard should not update stats when linking new bucket instance

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/mimic-dev1'

Merge pull request #19435 from smithfarm/wip-20559

doc: globally change CRUSH ruleset to CRUSH rule

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: Do not instantiate TrimRequest template class

We include TrimRequest.cc in librbd tests at two places:
 - operation/test_mock_TrimRequest.cc
 - operation/test_mock_ResizeRequest.cc

That causes linking errors when doing the builds because some of the
instantiated classes are defined twice.

We can fix this by not instantiating the template class in the
TrimReqeust.cc file when including it in the tests.

Signed-off-by: Boris Ranto <branto@redhat.com>

test/rbd_mirror: "use of uninitialised value" valgrind warning

The `on_call` context serves as a barrier and should be completed
after the `on_start_ctx` context is assigned.

The warning was observed sporadically e.g. by repeating
WaitingOnNonLeaderAcquireLeader test under valgrind.

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19191 from tchaikov/wip-mon-cleanup

mon: remove_is_write_ready()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17438 from mikulely/fix-mstop

vstart.sh: fix mstop.sh can not stop rgw

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: link executables against ceph-common

so they can have access to libstdc++ if it is linked statically.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_STATIC_LIBSTDCXX option

to link libceph-common with libstdc++ if it is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #19432 into master

* refs/pull/19432/head:
doc: correct the new fs allow opts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: globally change CRUSH ruleset to CRUSH rule

Since kraken, Ceph enforces a 1:1 correspondence between CRUSH ruleset and
CRUSH rule, so effectively ruleset and rule are the same thing, although
the term "ruleset" still survives - notably in the CRUSH rule itself, where it
effectively denotes the number of the rule.

This commit updates the documentation to more faithfully reflect the current
state of the code.

Fixes: http://tracker.ceph.com/issues/20559
Signed-off-by: Nathan Cutler <ncutler@suse.com>

install-deps.sh: use GCC-7 on xenial also

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19416 from trociny/wip-flatten-cleanup

librbd: remove unused member in FlattenRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: correct the new fs allow opts

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #19426 from tchaikov/wip-cmake

cmake: disable FAIL_ON_WARNINGS for rocksdb

cmake: disable FAIL_ON_WARNINGS for rocksdb

otherwise -Werror=implicit-fallthrough will fail the build with GCC-7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #19373 into master

* refs/pull/19373/head:
client: drop duplicate friend

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19417 from tchaikov/wip-install-deps-gcc

install-deps.sh: install new gcc as the default the right way

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados: add missing openstack volumes

Signed-off-by: Sage Weil <sage@redhat.com>

install-deps.sh: install new gcc as the default the right way

* should install software-properties-common beforehand, otherwise
  the `add-apt-repository` command will not be available.
* the update-alternative commandline were copied from ceph-build,
  should remove the escape characters.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: remove unused member in FlattenRequest

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19398 from tchaikov/wip-install-deps-use-dts

install-deps.sh: use DTS on centos if GCC is too old

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19368 from dzafman/wip-22086

ceph-objectstore-tool: Add option "dump-import" to examine an export

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec: use dts-7 on rhel

devtoolset-7 is available for RHEL, but not for centos. only dts-6 is
available for centos.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph_objectstore_tool.py: Perform dump-import

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19382 from rishabh-d-dave/docfix-ceph-osd-ls

doc/man: add "ls" to "ceph osd" command's subcommands list

Merge pull request #19005 from shinobu-x/src_krbd

rbd: drop unnecessary using declaration, etc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: enable testing repo on centos on aarch64

the DTS-6 is only available in the testing repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19338 from vshankar/rbd-mirror-image-map-timestamp

rbd-mirror: persist image map timestamp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19383 from trociny/wip-snapshot-copy

librbd: deep_copy: don't create snapshots above snap_id_end

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: use DTS on centos if GCC is too old

please note, run-make-check.sh sources install-deps.sh here to import
the $PATH and other environmental variables, which could be changed by
the the DTS "enable" script.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19395 from tchaikov/wip-debian-control-upgrade

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19341 from tchaikov/wip-22301

ceph.spec: use devtoolset-6-gcc-c++ on aarch64

Merge pull request #19336 from amitkumar50/cov-misc-14

osd,os,io: Initializing C_ProxyChunkRead members,queue,request

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19295 from kungf/scrub_delayed

osd: change op delayed state to 'waiting for scrub'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19345 from ivancich/wip-bring-in-dmclock

osd: pull latest dmclock subtree

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19027 from tangwenjun3/wip-flush-reduce

os/bluestore: reduce meaningless flush

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19348 from adamemerson/wip-murdering-diagnostics

Add noreturn attribute to silence uninitialized warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

this is a follow-up of #19328. we need to get this change into 12.2.3.
so better off do the switch somewhere after 12.2.2 which has been
tagged, and before 12.2.3, which is not tagged yet.

please note, this is not targetting master, because i want to make
sure the change number (the <num> in << 12.2.2-<num>) is correct. it
does not hurt if it's not, as long as it is ">> 12.2.2", so the replace
machinery in 12.2.3 works, and it covers the releases where the
ceph-{osdomap,kvstore,monstore}-tool are not move yet. but why don't
make it more right?

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdf49ba664b2c1c0250f8931d24e504ec9274561)

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

The backport didn't make 12.2.2, but it will be in 12.2.3.

Fixes: http://tracker.ceph.com/issues/22319
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e0c814266fe653311673e07bd7c7dbb51d866f82)

Merge pull request #19356 from tchaikov/wip-18589-mimic-dev1

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18752 from liewegas/wip-peering-queue

osd: put peering events in main sharded wq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: deep_copy: move set_head to snapshot_copy state machine

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: deep_copy: don't create snapshots above snap_id_end

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

ceph-volume test logging ignored ceph.conf file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main warn on inability to load ceph.conf, don't raise

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api ensure that we can handle non-ceph tags in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm only consider 'ceph.' tags in logical volumes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration allow inlined comments for # and ;

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify that INI comments can be inlined

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create rollback osd creation when prepare or activate fails

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 07be6fe0ab7981ebdc34779a8ccd6220d640a549)

ceph-volume lvm.create fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6781d6498e01bfc89189c235540e5c2a2e6bc8ca)

ceph-volume lvm.prepare fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b17ad2f8fe617d219fdb2f46dd2675669f32f08)

ceph-volume lvm.prepare rollback osd when prepare fails and an osd id was generated

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 579d12806b2ffc693a4e70ba3eaf663fa281d35e)

ceph-volume lvm.common create a rollback_osd utility to cleanup failed osd prepare/create calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e533792e0dbafc160238c0f5783f4430e2705571)

Merge pull request #19244 from ceph/wip-cd-volume

qa/tasks: update ceph-deploy task to use newer ceph-volume syntax

Merge pull request #19276 from tchaikov/wip-volume-key

ceph-volume: do not use --key during mkfs

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19363 from ceph/wip-rm22282

ceph-volume: removed the explicit use of sudo

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc/man: mention "ls" in the ceph osd's subcommands list

Though "ls" command is explained and it's usage shown in the man page,
it is not mentioned in the subcommands list of "ceph osd" in the
beginning.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #19381 from ceph/fix_cephvol_doc

ceph-volume/doc: add missing subcommand in examples

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add missing subcommand in examples

Add the missing 'lvm' subcommand in examples in the documentation of
ceph-volume.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

rbd-mirror: persist image mapping timestamp on map update

Signed-off-by: Venky Shankar <vshankar@redhat.com>

rbd-mirror: include map timestamp in image map

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: drop duplicate friend

Dropped duplicate friend access providing, as the below warning appears:
ceph/src/client/Client.h:764:16: warning: ‘C_Client_Remount’ is already a friend of ‘Client’ [enabled by default]
   friend class C_Client_Remount;

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18276 from liewegas/wip-removed-snaps

mon,osd,osdc: refactor snap trimming (phase 1)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19364 from neha-ojha/wip-preserve-cbt

qa: preserve cbt task results

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-objectstore-tool: Add option "dump-import" to examine an export

Fixes: http://tracker.ceph.com/issues/22086
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: preserve cbt task results

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #19188 from adamwg/awg/rbd-watchers-api

librbd: Add a function to list image watchers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19337 from trociny/wip-22306

pybind/rbd: fix metadata functions error handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: Add debug info when creating ceph volumes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #18967 from shun-s/wip-rbd-cp-optimize

librbd: speed up sparse copy when object map is available

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19251 from MahatiC/eventtrace_librbd

librbd: add eventtrace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19280 from Songweibin/wip-parent-info

rbd: add parent info when moving child into trash bin

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19315 from trociny/wip-fsx-ggate

test/librbd: rbd-ggate mode for fsx

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #18791 into master

* refs/pull/18791/head:
mds: bump protocol for snapshot changes
mds: force client to do snapflush
mds: check existance of snaps when rebuilding need_snapflush
mds: properly setup need_snapflush for flushsnap messages
mds: properly update CInode::first and CDentry first
mds: don't create snap inode during trans-authority rename
mds: trim non-auth snap dentry during cache rejoin
mds: detach unlinked inode's snaprealm from its parent
mds: fix CInode::find_snaprealm()
mds: fix snap dentry discover
mds: introduce MDSMap::get_mds_set_lower_bound()
mds: properly distinguish snaptable TABLE_OP_CREATE from TABLE_OP_UPDATE
mds: properly set reply buffer for table request 'agree' message
mds: make change to mds table after corresponding event get logged

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19351 from ceph/wip-rm22281

ceph-volume rollback on failed OSD prepare/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: removed the explicit use of sudo

This will allow users to run ceph-volume on systems that do
not have sudo installed.

Fixes: http://tracker.ceph.com/issues/22282
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19344 from tchaikov/wip-we-need-cxx11-and-more

cmake: bail out if GCC version is less than 5.1

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: fix mstop.sh can not stop rgw

+ As mstop.sh find rgw processes by looking for pid file, so we need to
  specify the --pid-file option to let rgw create it.
+ Use rgw port instead of meaningless 'j k' in the file name
+ Add rgw asok support
+ Make vstart.sh respect 'radosgw' prefix (defined in mrgw.sh) instead of 'rgw'

before:

[root@ceph-node1]~/jiaying/ceph/build# ../src/mrun j ceph -w
2017-08-31 16:34:05.413931 7f69081e7700 -1 WARNING: all dangerous and experimental features are enabled.
2017-08-31 16:34:05.445024 7f69081e7700 -1 WARNING: all dangerous and experimental features are enabled.
  cluster:
    id:     9952bf4f-0829-49e3-b893-abe858b690e5
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum a
    mgr: x(active)
    mds: cephfs_a-1/1/1 up  {0=a=up:active}
    osd: 1 osds: 1 up, 1 in
    rgw: 1 daemon active

  data:
    pools:   6 pools, 48 pgs
    objects: 254 objects, 5964 bytes
    usage:   24549 MB used, 26625 MB / 51175 MB avail
    pgs:     48 active+clean

  io:
    client:   10919 B/s rd, 0 B/s wr, 10 op/s rd, 5 op/s wr

[root@ceph-node1]~/jiaying/ceph/build# ../src/mstop.sh j
2645521
pid=2645521
entity=mds pid=2645521
kill  2645521...2645132
pid=2645132
entity=mgr pid=2645132
kill  2645132...2645008
pid=2645008
entity=mon pid=2645008
kill  2645008...2645326
pid=2645326
entity=osd pid=2645326
kill  2645326...kill  2645326...#
[root@ceph-node1]~/jiaying/ceph/build# ps aux| grep radosg
root     2645832  1.0  0.0 1668120 40088 ?       Ssl  16:32   0:01 /root/jiaying/ceph/build/bin/radosgw -c /root/jiaying/ceph/build/run/j/ceph.conf --log-file=/root/jiaying/ceph/build/run/j/out/rgw.j.log --debug-ms=1 -n client.rgw --rgw_frontends=civetweb port=8001
root     2646217  0.0  0.0 112648   964 pts/3    S+   16:34   0:00 grep --color=auto radosg

after:

[root@ceph-node1]~ CEPH_NUM_MON=1 CEPH_NUM_OSD=1 CEPH_NUM_MDS=1 CEPH_NUM_MGR=1  CEPH_NUM_RGW=2 ../src/mstart.sh j  -n -X -l

[root@ceph-node1]~/jiaying/ceph/build# ../src/mstop.sh  j
2760395
pid=2760395
entity=mds pid=2760395 name=mds.a
kill  2760395...2760007
pid=2760007
entity=mgr pid=2760007 name=mgr.x
kill  2760007...2759883
pid=2759883
entity=mon pid=2759883 name=mon.a
kill  2759883...2760201
pid=2760201
entity=osd pid=2760201 name=osd.0
kill  2760201...kill  2760201...2760707
pid=2760707
entity=radosgw pid=2760707 name=radosgw.8001
kill  2760707...kill  2760707...2760714
pid=2760714
entity=radosgw pid=2760714 name=radosgw.8002
kill  2760714...kill  2760714...#

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #19343 from dillaman/wip-22321

ceph-dencoder: moved RBD types outside of RGW preprocessor guard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

osd: change op delayed state to 'waiting for scrub'

when release rwlock, the ops waiting for rwlock are thansfered to
waiting_for_scrub if the object was blocked by scrub, the delayed
state should be changed at the same time.

Signed-off-by: kungf <yang.wang@easystack.cn>

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

this is a follow-up of #19328. we need to get this change into 12.2.3.
so better off do the switch somewhere after 12.2.2 which has been
tagged, and before 12.2.3, which is not tagged yet.

please note, this is not targetting master, because i want to make
sure the change number (the <num> in << 12.2.2-<num>) is correct. it
does not hurt if it's not, as long as it is ">> 12.2.2", so the replace
machinery in 12.2.3 works, and it covers the releases where the
ceph-{osdomap,kvstore,monstore}-tool are not move yet. but why don't
make it more right?

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19333 from amitkumar50/cov-osd-4

osd: Initializing start_offset,last_offset,offset

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19334 from amitkumar50/cov-misc-13

test: Initializing ChunkReadOp members

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #18849 into master

* refs/pull/18849/head:
mds:improve mds perf counters can statistics filesystem operations number and latency

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18941 into master

* refs/pull/18941/head:
discard the mdsload clear after prep_rebalance in case we want to export it for debugging
make sure that MDBalancer uses heartbeat info from the same epoch

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19089 into master

* refs/pull/19089/head:
client: xlist link dentries instead of set
client: cleanup Dentry definition

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19092 into master

* refs/pull/19092/head:
mds: remove useless check in Migrator::export_dir
mds: ignore export pin for unlinked directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19218 into master

* refs/pull/19218/head:
client: quit on failed remount during dentry invalidate test

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19224 into master

* refs/pull/19224/head:
mds: use find instead of count in CInode::get_dirfrag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19237 into master

* refs/pull/19237/head:
mds: get rid of the redundant code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-objectstore-tool: Remove unused map section from log dump

Caused by: 44cb6d30640954c66fc03d4ce58b583ffd614951

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Fix output of section type number

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Improve ceph-objectstore-tool usage output

Remove unused argument test-align

Caused by: 1c8731c31eb53e17d8f363906438963ab5688fe0

Positional arguments are described by usage() because
the parsing code creates descriptions like "--object" for the
object positional argument.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19347 from ceph/wip-rm22326

ceph-volume: warn on missing ceph.conf file

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.create rollback osd creation when prepare or activate fails

Signed-off-by: Alfredo Deza <adeza@redhat.com>