install-deps.sh: install prebuilt libboost dependencies for bionic also

* extract boost install function into a helper, so we don't need to
  repeat it when adding support for new distros.
* install ceph-libboost-* for bionic build also. we check for the
  existence in run-make-check.sh, so there is no need to update
  the build script elsewhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: correct gcc version info as major.minor.patch

On ubuntu 18.04, "gcc -dumpversion" output "7", it result in
that below check will return false:
    dpkg --compare-versions 7 ge 7.0
Then, this script will install other gcc.

Actully, the full gcc version "gcc -dumpfullversion" output is
"7.3.0", than below check will turn true:
    dpkg --compare-version 7.3.0 ge 7.0
So, there's no need to install other gcc.

In case of hitting error on ubuntu16.04, use below parameter:
   gcc -dumpfullversion -dumpversion

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge PR #20665 into master

* refs/pull/20665/head:
mgr/balancer: add crush_compat_metrics param

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25135 from cbodley/wip-test-log-garble

test: fix libc++ crash in Log.GarbleRecovery

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #22613 from zhangsw/fix-rgw-copyobj-tailtag

rgw: Copying object data should generate new tail tag for the new object.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23652 from theanalyst/rgw/awsv4/presign-slashes

rgw: s3: awsv4 drop special handling for x-amz-credential

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #25003 from ivancich/wip-update-reshard-params

rgw: reshard improvements

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #25102 from cbodley/wip-rgw-log-http-status

rgw: log http status with op prefix if available

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test: fix libc++ crash in Log.GarbleRecovery

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25007 from jan--f/c-v-dont-duplicated-rejected-reasons

ceph-volume: rename Device property valid to available

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25002 from yehudasa/wip-rgw-svc-5

rgw: fix uninitialized access

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24967 from bi-shun/fix-brotli

compressor: Fix build of Brotli Compressor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24803 from tspmelo/wip-i18n

mgr/dashboard: Add i18n support

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #14092 into master

* refs/pull/14092/head:
mgr/DaemonServer: fix session leak
mon/MonClient: ignore new mon commands while stopping
mgr/DeviceState: fix DeviceState initial refcount
qa/suites: valgrind ceph-mgr too

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25052 into master

* refs/pull/25052/head:
mgr: lock pg_map too for osd_pool_stats and notify_osdmap
mgr/DaemonServer: use with_osdmap_and_pgmap instead of nested calls
mgr/ClusterState: add with_osdmap_and_pgmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

compressor: modify brotli compression plugin.

* add "common/ceph_context.h" header file to CompressionPluginBrotli.cc
* set the brotli v1.0.7 stable version for compressor plugin use.

Signed-off-by: BI SHUN KE <aionshun@livemail.tw>

Merge pull request #25092 from jan--f/c-v-inventory-doc

doc: add ceph-volume inventory sections

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25053 from rhcs-dashboard/pytest-faulthandler

mgr/dashboard: add pytest plugin: faulthandler

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr: lock pg_map too for osd_pool_stats and notify_osdmap

Fixes: http://tracker.ceph.com/issues/36766
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: use with_osdmap_and_pgmap instead of nested calls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25046 from tchaikov/wip-16500

tests: do not check for invalid k/m combinations

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #25084 from s0nea/wip-dashboard-add-missing-test-suites

mgr/dashboard/qa: add missing dashboard suites

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

c-v device test: rename valid to available

Update test to reflect 8a80990471108b0920d1d8aa1239733ae2b20e9c

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

inventory test: rename valid to available

Fixes test to reflect 8a80990471108b0920d1d8aa1239733ae2b20e9c

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: rename Device property valid to available

This flag is used in the inventory reporting and available is deemed more
appropriate. Furthermore this fixes a bug where rejected_reasons
accumulated duplicate entries.

Fixes: http://tracker.ceph.com/issues/36701
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

doc: add ceph-volume inventory sections

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #24973 into master

* refs/pull/24973/head:
mgr: Separate diskprediction cloud plugin from the diskprediction plugin

Merge PR #20185 into master

* refs/pull/20185/head:
Common, log: Added functionality test for recovery after silenced (#20174)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #19987 into master

* refs/pull/19987/head:
qa/tasks/ceph: ignore failure when chown ceph:ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: Separate diskprediction cloud plugin from the diskprediction plugin

Separate diskprediction local cloud from the diskprediction plugin.
Devicehealth invoke device prediction function related on the global
configuration "device_failure_prediction_mode".

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>

mgr/DaemonServer: fix session leak

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: ignore new mon commands while stopping

If one thread is stopping and another threads tries to send a new mon
command, just ignore it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20683 from Aran85/wip-rados-loadgen-align

tools: add offset-align option to "rados" load-gen

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24799 from xiexingguo/wip-js-safe-to-destroy

mgr/DaemonServer: add js-output for "ceph osd safe-to-destroy"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24969 from joke-lee/doc-dump-journal-help-info

osd: add --dump-journal option in ceph-osd help info

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/ClusterState: add with_osdmap_and_pgmap

Several call sites need to lock both

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24866 from linuxbox2/wip-rm-aclparser

rgw: remove rgw_aclparser.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25108 from tchaikov/wip-bluestore-repair

tools/ceph_kvstore_tool: do not open rocksdb when repairing it

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #25070 from tchaikov/wip-25146

rocksdb: pick up a fix to be backward compatible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard/qa: add missing dashboard suites

Fixes: https://tracker.ceph.com/issues/37152
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

rocksdb: pick up a fix to be backward compatible

see https://github.com/facebook/rocksdb/pull/3488

also bump up the rocksdb version to v5.17.2

Fixes: http://tracker.ceph.com/issues/25146
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24901 from wjwithagen/wjw-fix-rbd-ggate-cmake

tool: link rbd-ggate against librados-cxx

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25060 from mooncak/fix_typos

doc: Fix some typos

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24999 from MrStupnikov/master-docfix-1358

doc: Put command template into literal block

Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code/lrc: error out if "l == 0"

l (locality) should not be 0, and k+m should be multiple of it

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code: return error using `ss` not derr

log is not very visible from user's perspective. we'd better return the
error message using the input parameter, if the technique is
unsupported.

also, remove the trailing newline, as `ss` will be sent back to user,
it's the the front-end's responsibility to format the error message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: do not check for invalid k/m combinations

This reverts commit c5bafae6fd10282f408695c7763b1277469123f6.

in 37f73607 and 3ec41030, we detect invalid "k". and we should always
detect invalid combinations of parameters in
ErasureCodeInterface::init(), and return -EINVAL instead of "autotune" them
as put in the commit message of c5bafae6, if the profile is invalid for
given erasure plugin. so ErasureCodeBench::encode() and
ErasureCodeBench::decode() will bail out with the error message returned by
ErasureCodeInterface::init() if the parameters are not valid.

also, i checked all ErasureCodeInterface::init() implementations, none
of them is overwriting the settings without returning a non-zero return
code. so we are not overriding or autotuning the settings specified by
user.

Fixes: http://tracker.ceph.com/issues/16500
Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: use stoi() instead of atoi()

stoi() is simpler if we need to convert std::string to integer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24996 from mcv21/bug/note_file_move

debian: correct ceph-common relationship with older radosgw package

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25048 from swinds24/master_fix_36764

ceph-mgr: hold lock while accessing the request list and submitting request

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24458 from rzarzynski/wip-osd-drop-osd_instructions

osd: drop the unused request_redirect_t::osd_instructions

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24141 from liu-chunmei/ceph_seastar_perfcounter

common,crimson: port perfcounters to seastar

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24858 from jecluis/wip-pr-19983-tests

qa/cephtool: test bounds on pool's `hit_set_*`

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25106 from tchaikov/wip-qa-envlibrados

qa: fix upgrade tests and test_envlibrados_for_rocksdb.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tools/ceph_kvstore_tool: do not open rocksdb when repairing it

before this change, the `need_open_db` parameter is passed to the
constructor of BlueStore as `min_alloc_size`. and rocksdb will fail to
repair because Repairer::Run() also tries to acquire the db lock, and it
will fail to do so if the lock file is already acquired by
BlueStore::_mount().

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: patch rocksdb the right way

quote from patch(1)

>       but usually just
>
>       patch -pnum <patchfile

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24979 from neha-ojha/wip-upgrade-bug

doc/releases/mimic.rst: make note of 13.2.2 upgrade bug

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25072 from tchaikov/wip-silence-sign-compare

os/tests: silence -Wsign-compare warning

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa: use FOUND_VAR to be backward compatible with cmake 2.8.12

before this change, we assume that the variable set if rados::radospp is
found will be radospp_FOUND, but this is a feature cmake 3, see
https://cmake.org/cmake/help/v3.3/module/FindPackageHandleStandardArgs.html

while the cmake shipped by centos is cmake 2.8.12, where the variable
name will be <UPPERCASED_NAME>_FOUND, see
https://cmake.org/cmake/help/v2.8.12/cmake.html#module:FindPackageHandleStandardArgs

in the test of test_envlibrados_for_rocksdb.sh, we are using cmake not
the cmake3 offered by EPEL7, so RADOSPP_FOUND will be set instead. that's why
executable env_librados_test will fail to link against rados::radospp.
as rados::radospp won't be defined if radospp_FOUND is not defined/set.

after this change, the 2nd mode of FIND_PACKAGE_HANDLE_STANDARD_ARGS()
is used instead to ensure that radospp_FOUND is defined even if cmake
2.8.12 is used.

also, the message() commands for debugging purpose are removed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites: add librados2 to "extra_packages" for upgrade tests

we use the playbook of "testnodes.yml" defined by ceph-cm-ansible for
initializing test nodes, and the role of "testnode" is used by
testnodes.yml. "testnode" requires "qemu-system-x86" or "qemu-kvm"
package to be installed. the qemu in turn depends on librbd1 and
librados2.

before librados3 was introduced, this worked perfectly. because in ceph
repo, qa/packages/packages.yaml defines the default set of packages the
"install" tasks should install. and in that yaml file, librados2 was
listed. so the package management system will overwrite the librados2
installed by ansible playbook with the version specified by the
"install" task, as apt/yum thinks this is what user requires explicitly,
so it's fine to install a different version of librados2.

after librados3 was introduced, librados2 was removed from
qa/packages/packages.yaml. because, by default, we need to install
librados3 instead of librados2 for ready a nautilus cluster. but the
problem is, the packge list also applies to "install" tasks installing
releases before nautilus, where we still need to replace the librados2
installed by ansible.

so, to address this issue, "librados2" is added to "extra_packages" of
the "install" tasks of tests installing old releases to install
librados2 explicitly instead of as a dependency of other ceph packages
like librbd1.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer: add js-output for "ceph osd safe-to-destroy"

E.g.:
 - case 1:
$ceph osd safe-to-destroy 0 1 2 -f json-pretty
Error EAGAIN: 12 pgs have unknown state; cannot draw any conclusions

 - case 2:
$ceph osd safe-to-destroy 0 1 2 -f json-pretty
{
    "safe_to_destroy": [
        0,
        1,
        2
    ],
    "active": [],
    "missing_stats": [],
    "stored_pgs": []
}

 - case 3:
$ceph osd safe-to-destroy 0 1 2 -f json-pretty
{
    "safe_to_destroy": [],
    "active": [
        0,
        1,
        2
    ],
    "missing_stats": [],
    "stored_pgs": []
}

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/dashboard: Trim HTML translations

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add locale files

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use I18N in TypeScript strings

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add missing HTML translations

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix existing translations

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add I18N section to HACKING.rst

Signed-off-by: Sebastian Krah <skrah@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25098 from ceph/rm-27062

ceph-volume: reject devices that have existing GPT headers

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: log http status with op prefix if available

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25093 from alfredodeza/wip-bz1644828

ceph-volume: remove LVs when using zap --destroy

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/tests: silence -Wsign-compare warning

silence warning like

In file included from
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/objectstore/store_test.cc:25:0:
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/googletest/googletest/include/gtest/gtest.h:
In instantiation of 'testing::AssertionResult
testing::internal::CmpHelperEQ(const char*, const char*, const T1&,
const T2&) [with T1 = int; T2 = long unsigned int]':

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: update tests after removing the LVPath arg validator

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062

ceph-volume: return lvs as vg/lv from ValidDevice if as_string is True

All code in prepare and activate expect lvs to be in that format.
Eventually we can refactor all that to use the Device class as well.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062

ceph-volume: adds lv_name property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062

ceph-volume: combine the LVPath arg validator with ValidDevice

This combines the two arg validators and makes both batch and
prepare/active both use ValidDevice.

This will also allow us to use lvs with a full path, like
/dev/vg/lv instead of always enforcing vg/lv.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062

ceph-volume: adds has_gpt_headers property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062

ceph-volume: add PTTYPE to blkid parser

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: rm#27062

Merge pull request #25094 from CharlotteRose/wip-fix-boxes

doc: Fixed the paragraph and boxes

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #24789 from Ranjitha-G/dashboard-test-erasure

mgr/dashboard: Add unit test case for controller/erasure_code_profile.py

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: Fixed the paragraph and boxes.

Signed-off-by: Scoots Hamilton <scoots@redhat.com>

mgr/dashboard: Add language selector

Add the language selector to the navigation.

Signed-off-by: Sebastian Krah <skrah@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add i18n providers

Prepare the dashboard to support i18n

Add ngx-translate polyfill to extract translations from the typescript files.
Add a npm script to extract marked strings to a translation template.

Signed-off-by: Sebastian Krah <skrah@suse.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24788 from Ranjitha-G/dashboard-hacking-doc

mgr/dashboard: Fix some setup steps in HACKING.rst

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

doc: Fix some typos

Signed-off-by: mooncake <xcoder@tenxcloud.com>

Merge pull request #25080 from smithfarm/wip-37089

tests: make ceph-admin-commands.sh log what it does

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25083 from iridescent-rsy/fix-typo

rbd: fix some typos

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd_mirror: fix a typo.

Signed-off-by: Shiyang Ruan <ruansy.fnst@cn.fujitsu.com>

rbd_mirror: restore the header annotation.

Signed-off-by: Shiyang Ruan <ruansy.fnst@cn.fujitsu.com>

rbd: fix typos.

Signed-off-by: Shiyang Ruan <ruansy.fnst@cn.fujitsu.com>

doc/releases/mimic.rst: make note of 13.2.2 upgrade bug

Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph-volume tests update tests to handle new SystemExit exceptions in main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api verify that Volume object can be removed

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm allow passing a Volume object to remove_lv

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional.lvm test full removal of LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main raise sys.exit to prevent garbage in help

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap remove vgs|lvs when --destroy is used

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #24490 into master

* refs/pull/24490/head:
mds: flush dirty dirfrags that weren't logged when deactivating mds
mds: use MDlog::trim_all() to trim log when deactivating mds
mds: don't cap log when there are replicated objects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tests: make ceph-admin-commands.sh log what it does

Fixes: http://tracker.ceph.com/issues/37089
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25057 from joke-lee/wip-ceph-dencoder-support

ceph-dencoder: add RGWRealm and RGWPeriod  support.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25071 from tchaikov/wip-install-dep-for-python-saml

install-deps.sh: install libtool-ltdl-devel for building python-saml

Reviewed-by: Ricardo Marques <rimarques@suse.com>