Merge pull request #19366 from liewegas/wip-21557

osd: more debugging for snapmapper bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19367 from wjwithagen/wip-wjw-clang-xlist-string

include: xlist: Fix Clang error for missing string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19365 from liewegas/wip-tell

qa/tasks/ceph_manager: tolerate tell osd.* error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19353 from shinobu-x/common_clock

common: Remove ceph_clock_gettime, extern keyword

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19352 from chardan/jfw-wip-issue-20811

osd: migrate PGLOG_* macros to constexpr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19329 from ZVampirEM77/cleanup-revise-have

osd/PGLog: cleanup unused function revise_have

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19056 from adamemerson/wip-with-val

common/config: use with_val() for better performance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19114 from tchaikov/wip-rm-request-slow

mgr/PGMap: drop REQUEST_{SLOW,STUCK} HEALTH_WARNs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19229 from wido/mgr-influx

mgr/influx: PEP-8 and other fixes to Influx module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19472 from qrGitHub/wip-rgw-typo-woild

rgw: fix typo woild to would

Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: correct typo woild to would

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #17651 from ceph/wip-ceph-deploy-upgrade

qa/tests: Wip ceph deploy upgrade

Merge pull request #19144 from zhangsw/cleanup-rgwputobj-md5

rgw: no need to deal with md5 header in get_data.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19167 from joke-lee/Bucket_Policy_SourceIp_Condition

rgw: implement ipv4 aws:SourceIp condition for bucket policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19210 from Liuchang0812/rgw-camelcase

 rgw: use camelcase format in request headers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19236 from dplyakin/get-website-error-code

rgw: fix GET website response error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19250 from qrGitHub/wip-rgw-unused-param2

rgw: remove unused param in AWSGeneralAbstractor::get_auth_data_v4

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #19390 from yaozongyou/remove-duplicated-include

rgw,common: remove already included header files

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19391 from qrGitHub/wip-rgw-memory-optimize

rgw: optimize memory usage in function rgw_bucket::get_key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19430 from tianshan/fix_recursive_lock

rgw: fix recursive lock

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19418 from tchaikov/wip-gcc-7-on-xenial

install-deps.sh, cmake: use GCC-7 on xenial also

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19402 from b-ranto/wip-fix-arm-build

test/librbd: Do not instantiate TrimRequest template class

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19437 from trociny/wip-mock-valgrind

test/rbd_mirror: "use of uninitialised value" valgrind warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19253 from oritwas/wip-rgw-22124

rgw: reshard should not update stats when linking new bucket instance

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/mimic-dev1'

Merge pull request #19435 from smithfarm/wip-20559

doc: globally change CRUSH ruleset to CRUSH rule

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: Do not instantiate TrimRequest template class

We include TrimRequest.cc in librbd tests at two places:
 - operation/test_mock_TrimRequest.cc
 - operation/test_mock_ResizeRequest.cc

That causes linking errors when doing the builds because some of the
instantiated classes are defined twice.

We can fix this by not instantiating the template class in the
TrimReqeust.cc file when including it in the tests.

Signed-off-by: Boris Ranto <branto@redhat.com>

test/rbd_mirror: "use of uninitialised value" valgrind warning

The `on_call` context serves as a barrier and should be completed
after the `on_start_ctx` context is assigned.

The warning was observed sporadically e.g. by repeating
WaitingOnNonLeaderAcquireLeader test under valgrind.

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19191 from tchaikov/wip-mon-cleanup

mon: remove_is_write_ready()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17438 from mikulely/fix-mstop

vstart.sh: fix mstop.sh can not stop rgw

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix GET website response error code

Change NoSuchKey error code to NoSuchWebsiteConfiguration, when bucket doesn't have website configuration.

Fixes: http://tracker.ceph.com/issues/22272
Signed-off-by: Dmitry Plyakin <dplyakin@gmail.com>

cmake: link executables against ceph-common

so they can have access to libstdc++ if it is linked statically.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_STATIC_LIBSTDCXX option

to link libceph-common with libstdc++ if it is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #19432 into master

* refs/pull/19432/head:
doc: correct the new fs allow opts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: globally change CRUSH ruleset to CRUSH rule

Since kraken, Ceph enforces a 1:1 correspondence between CRUSH ruleset and
CRUSH rule, so effectively ruleset and rule are the same thing, although
the term "ruleset" still survives - notably in the CRUSH rule itself, where it
effectively denotes the number of the rule.

This commit updates the documentation to more faithfully reflect the current
state of the code.

Fixes: http://tracker.ceph.com/issues/20559
Signed-off-by: Nathan Cutler <ncutler@suse.com>

install-deps.sh: use GCC-7 on xenial also

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19416 from trociny/wip-flatten-cleanup

librbd: remove unused member in FlattenRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/influx: Print (in debug) time it takes to send data to Influx

On large clusters it might take a very long time to send data to Influx
due to the gathering and parsing of statistics.

By keeping a counter and printing it admins can adjust the interval if it
becomes to heavy for their cluster.

Signed-off-by: Wido den Hollander <wido@42on.com>

mgr/influx: Revise configuration of module

It's a lot like the Zabbix module and allows for setting configuration
options on run-time and also fetch them from the module.

A few additional commands have been registered to make sure it is easy
to interact with the module.

Signed-off-by: Wido den Hollander <wido@42on.com>

mgr/influx: Send fsid as a tag to Influx to identify cluster

This allows for multiple Ceph clusters to send their data to the
same Influx database.

Using the fsid values for different clusters can be queried from
Influx

Signed-off-by: Wido den Hollander <wido@42on.com>

mgr/influx: PEP-8 fixes to InfluxDB module

The code now scores:

  Your code has been rated at 8.94/10

Various indentation fixes, whitespaces and other PEP-8 related changes

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: correct the new fs allow opts

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

rgw: fix recursive lock

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #19426 from tchaikov/wip-cmake

cmake: disable FAIL_ON_WARNINGS for rocksdb

cmake: disable FAIL_ON_WARNINGS for rocksdb

otherwise -Werror=implicit-fallthrough will fail the build with GCC-7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #19373 into master

* refs/pull/19373/head:
client: drop duplicate friend

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19417 from tchaikov/wip-install-deps-gcc

install-deps.sh: install new gcc as the default the right way

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados: add missing openstack volumes

Signed-off-by: Sage Weil <sage@redhat.com>

install-deps.sh: install new gcc as the default the right way

* should install software-properties-common beforehand, otherwise
  the `add-apt-repository` command will not be available.
* the update-alternative commandline were copied from ceph-build,
  should remove the escape characters.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: remove unused member in FlattenRequest

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19398 from tchaikov/wip-install-deps-use-dts

install-deps.sh: use DTS on centos if GCC is too old

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19368 from dzafman/wip-22086

ceph-objectstore-tool: Add option "dump-import" to examine an export

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec: use dts-7 on rhel

devtoolset-7 is available for RHEL, but not for centos. only dts-6 is
available for centos.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph_objectstore_tool.py: Perform dump-import

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19382 from rishabh-d-dave/docfix-ceph-osd-ls

doc/man: add "ls" to "ceph osd" command's subcommands list

osd: migrate PGLOG_* macros to constexpr

Fixes: http://tracker.ceph.com/issues/20811
Signed-off-by: Jesse Williamson <jwilliamson@suse.de>

Merge pull request #19005 from shinobu-x/src_krbd

rbd: drop unnecessary using declaration, etc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: enable testing repo on centos on aarch64

the DTS-6 is only available in the testing repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19338 from vshankar/rbd-mirror-image-map-timestamp

rbd-mirror: persist image map timestamp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19383 from trociny/wip-snapshot-copy

librbd: deep_copy: don't create snapshots above snap_id_end

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: use DTS on centos if GCC is too old

please note, run-make-check.sh sources install-deps.sh here to import
the $PATH and other environmental variables, which could be changed by
the the DTS "enable" script.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19395 from tchaikov/wip-debian-control-upgrade

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19341 from tchaikov/wip-22301

ceph.spec: use devtoolset-6-gcc-c++ on aarch64

Merge pull request #19336 from amitkumar50/cov-misc-14

osd,os,io: Initializing C_ProxyChunkRead members,queue,request

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19295 from kungf/scrub_delayed

osd: change op delayed state to 'waiting for scrub'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19345 from ivancich/wip-bring-in-dmclock

osd: pull latest dmclock subtree

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19027 from tangwenjun3/wip-flush-reduce

os/bluestore: reduce meaningless flush

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19348 from adamemerson/wip-murdering-diagnostics

Add noreturn attribute to silence uninitialized warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

this is a follow-up of #19328. we need to get this change into 12.2.3.
so better off do the switch somewhere after 12.2.2 which has been
tagged, and before 12.2.3, which is not tagged yet.

please note, this is not targetting master, because i want to make
sure the change number (the <num> in << 12.2.2-<num>) is correct. it
does not hurt if it's not, as long as it is ">> 12.2.2", so the replace
machinery in 12.2.3 works, and it covers the releases where the
ceph-{osdomap,kvstore,monstore}-tool are not move yet. but why don't
make it more right?

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdf49ba664b2c1c0250f8931d24e504ec9274561)

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

The backport didn't make 12.2.2, but it will be in 12.2.3.

Fixes: http://tracker.ceph.com/issues/22319
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e0c814266fe653311673e07bd7c7dbb51d866f82)

rgw: optimize memory usage in function rgw_bucket::get_key

* In function rgw_bucket::get_key, the reserved 12 bytes is only useful
* for rgw_bucket_shard::get_key.

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

common: remove already included header files

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

rgw: remove already included header files

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #19356 from tchaikov/wip-18589-mimic-dev1

debian/control: adjust ceph-{osdomap,kvstore,monstore}-tool feature move

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: when no branch is specified, use wip branch under test for upgrade

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/tests: Fix mapped_role attribute where it gets overwritten for subsequent calls.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/tests: update upgrade tests from jewel->luminous to jewel->luminous->master

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #18752 from liewegas/wip-peering-queue

osd: put peering events in main sharded wq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

include: xlist: Fix Clang error for missing string

Clang complains:
home/jenkins/workspace/ceph-master/src/include/xlist.h:210:13: error: invalid operands to binary exp
ression ('std::ostream' (aka 'basic_ostream<char>') and 'const char *')
        oss << ", ";
        ~~~ ^  ~~~~
1 error generated.

include ostream to fix Clang error

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: deep_copy: move set_head to snapshot_copy state machine

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: deep_copy: don't create snapshots above snap_id_end

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

os/bluestore: Call get_str_map under with_val

Another heap allocation bites the dust.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore: Do not copy a string just to find its length

I really dislike whoever decided to disallow taking the address of
member functions in the standard library.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

msg: Use with_val, don't copy a string just to call find

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mon: Call get_json_str_map under wth_val

To avoid temporary heap allocations.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr: Call _list_modules under with_val

Avoid a temporary heap allocation.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv/RocksDB: Do comparisons under with_val

Rather than making a copy on the heap just to compare it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

kv/RocksDB: Use for_each_substr under with_val

Minimize the number of copies we make.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add with_val to md_config_t

Since passing back references that persist after we drop a lock is
unsafe, allow functions to be called on a reference under the lock.

This way we can compare strings or otherwise examine them without
needing to make a copy.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Disambiguate call to max

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume test logging ignored ceph.conf file

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main warn on inability to load ceph.conf, don't raise

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api ensure that we can handle non-ceph tags in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm only consider 'ceph.' tags in logical volumes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration allow inlined comments for # and ;

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify that INI comments can be inlined

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create rollback osd creation when prepare or activate fails

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 07be6fe0ab7981ebdc34779a8ccd6220d640a549)

ceph-volume lvm.create fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6781d6498e01bfc89189c235540e5c2a2e6bc8ca)

ceph-volume lvm.prepare fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b17ad2f8fe617d219fdb2f46dd2675669f32f08)

ceph-volume lvm.prepare rollback osd when prepare fails and an osd id was generated

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 579d12806b2ffc693a4e70ba3eaf663fa281d35e)