common: prefer std::size to ARRAY_SIZE macro

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #19964 from runsisi/wip-build

cmake: move ceph-osdomap-tool, ceph-monstore-tool out of ceph-test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24540 into master

* refs/pull/24540/head:
mgr/status: fix fs status subcommand did not show standby-replay MDS' perf info

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24709 into master

* refs/pull/24709/head:
PendingReleaseNotes: note about cephfs client state reclaim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24466 into master

* refs/pull/24466/head:
common: assert if buffer advance length overflow

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24529 into master

* refs/pull/24529/head:
mds: cleanup command descriptions

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24640 into master

* refs/pull/24640/head:
mds: remove MonClient reconnect

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24533 into master

* refs/pull/24533/head:
qa: add timeouts for remote ops for client mounts

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24679 from tspmelo/wip-36528

mgr/dashboard: Fix HttpClient Module imports in unit tests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24632 from Devp00l/wip-issue-36468

mgr/dashboard: Handle class objects as regular objects in KV-table

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24674 from sileht/master

ceph-volume: extract flake8 config

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23242 from linuxbox2/wip-rgw-inflight-ops

rgw: set default objecter_inflight_ops = 24576

Merge pull request #24692 from tchaikov/wip-cmake-with-asan

cmake: fixes to enable WITH_ASAN with clang and GCC

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24664 from tspmelo/wip-36513

mgr/dashboard: Fix missing failed restore notification

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24637 from ricardoasmarques/fix-rbd-actions-disable

mgr/dashboard: Fix RBD actions disable

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24612 from Tinatinak/fix_bug24548

mgr/dashboard: Fixed documentation link on RGW page

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix HttpClient Module imports in unit tests

Some of the tests were importing the wrong HttpClient Module and that was
causing random errors.

Fixes: https://tracker.ceph.com/issues/36528
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix missing failed restore notification

Fixes: http://tracker.ceph.com/issues/36513
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24544 from votdev/bug_24729

mgr/dashboard: Confirmation modal doesn't close

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24699 from rhcs-dashboard/add-tslint-rule

mgr/dashboard: Add TSLint rule "no-unused-variable"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24694 from tspmelo/wip-36544

mgr/dashboard: Fix problem with ErasureCodeProfileService

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24696 from dillaman/wip-36542

qa/tasks/qemu: use unique clone directory to avoid race with workunit

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #24698 into master

* refs/pull/24698/head:
Merge PR #24697 into nautilus
ceph_test_msgr: fix authorizer behavior
Merge pull request #24667 from liewegas/wip-ec-thrash-full
Merge PR #24689 into nautilus
qa/suites/rados/thrash-erasure-code*/thrashers/*: less likely resv rejection injection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24691 from iridescent-rsy/fix_typo

common: fix typos in BackoffThrottle

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24697 into nautilus

* refs/pull/24697/head:
ceph_test_msgr: fix authorizer behavior

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #24652 from badone/wip-undefined-gperftool-symbols-2

rpm: Use updated gperftools-libs at runtime

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/dashboard: Add TSLint rule
- TSlint no-unused-variable rule added.
- Cleanup: unused imports and variables.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #24591 from tspmelo/wip-testing-module

mgr/dashboard: Unit Tests cleanup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

ceph_test_msgr: fix authorizer behavior

Fixes breakage from this PR 2152d8ffb73a507a3d08d48b38c5a8e73f887138.

Fixes: http://tracker.ceph.com/issues/36495
Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_assert.h: do not pack assert params if WITH_ASAN

we pack the asset() params for smaller code size, but this creates a
inlined `assert_data_ctx` instance for every compilation unit which
call ceph_assert() defined in .h .

__PRETTY_FUNCTION__ is likely to be referenced by `assert_data_ctx`
sections which are included by different compiled object files. if the
ceph_assert() call is used by header file, then there will be multiple
`assert_data_ctx` sections sharing the same identifier. these sections are
defined as "COMDAT" group sections, i.e. common data sections. when linker
see multiple COMDAT sections with the same identifer, it will simply discard
the duplicated ones, and only keep a single copy of them. without enabling
ASan, GCC can always handle this problem just fine. but the dedup feature
does not work well with ASan. if ASan is enabled, and we link the objects
with the wrong order, some references will be pointing to the discarded
sections.

to address this issue, we could audit the link command line and inspect
all .o files to make sure they are properly ordered. but this is
non-trivial. as a workaround, in this change, the assert params are not
packed, and sent to the  __ceph_assert_fail() overrides which accepts
unpacked params directly, so the COMDAT section is not created.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/qemu: use unique clone directory to avoid race with workunit

If there is a workunit task associated with the same client, the two
tasks will attempt to clone the suite repo to the same directory.
Worse, if it's parallel tasks, the two clones will clobber each
other.

Fixes: http://tracker.ceph.com/issues/36542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24667 from liewegas/wip-ec-thrash-full

qa/suites/rados/thrash-erasure-code*/thrashers/*: less likely resv rejection injection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23103 from ifed01/wip-ifed-bluefs-migrate

os/bluestore: allow ceph-bluestore-tool to coalesce, add and migrate BlueFS backing volumes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23090 from mingshuaiwang/master

OSD: ceph-osd parent process need to restart log service after fork

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24689 into nautilus

* refs/pull/24689/head:
qa/tasks/ceph_manager: fix get_stuck_pgs from pg dump change

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24689 into master

* refs/pull/24689/head:
qa/tasks/ceph_manager: fix get_stuck_pgs from pg dump change
Merge PR #24625 into nautilus
qa/suites/rados/mgr/tasks/module_selftest: whitelist 'foo bar security'

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix RBD actions disable

Fixes: https://tracker.ceph.com/issues/36403
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Confirmation modal doesn't close

Fixes: https://tracker.ceph.com/issues/24729
Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix problem with ErasureCodeProfileService

ErasureCodeProfileService was being provided twice and that was causing
problems in production mode.

Fixes: https://tracker.ceph.com/issues/36544
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #20004 from mogeb/steady-clock-tools-rados

librados: use steady clock for rados_mon_op_timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24658 from tchaikov/wip-18202-rebased

blkdev: Rework API and add FreeBSD support

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

include/ceph_assert: always use __PRETTY_FUNCTION__ for C++

we've moved to GCC-7, no need to check for ancient compiler versions

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,ceph.in: preload libasan if WITH_ASAN

we need to preload libasan.so as the python exectuable is not likely to
be compiled with ASan enabled.
see:
https://github.com/google/sanitizers/wiki/AddressSanitizerAsDso#asan-and-ld_preload

just to ease the use of ASan, for fine-tuned behaviour, use
`ASAN_OPTIONS`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: extract get_cmake_variables()

so it can be reused

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should compile libzstd with -fPIC

otherwise we will have

/usr/bin/ld: libzstd/lib/libzstd.a(error_private.c.o): relocation
R_X86_64_32S against `.rodata' can not be used when making a shared
object; recompile with -fPIC

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass Sanitizers flags to linker for linking .so

see
https://github.com/google/sanitizers/wiki/AddressSanitizer#using-addresssanitizer

to be specific,

> In order to use AddressSanitizer you will need to compile and link your
> program using `clang` with the `-fsanitize=address` switch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass cflags to disutils using CC instead of CFLAGS

in python's distutils.ccompiler, linker_exe is composed using CC instead
of LDFLAGS. the latter only effects how it builds (shared) library.

and put CMAKE_C_FLAGS into the cflags for the compiler for building
python C extensions, it's more consistent this way. more importantly,
if we build with ASan enabled, the canary program, a.k.a. rados_dummy.c,
won't link without proper CFLAGS.

without this change, rados.so fails to build with errors like:

/usr/bin/ld: /var/ssd/ceph/build/lib/librados.so: undefined reference to
`__asan_stack_free_10'
/usr/bin/ld: /var/ssd/ceph/build/lib/librados.so: undefined reference to
`__asan_report_exp_store8'
...
...

clang: error: linker command failed with exit code 1 (use -v to see
invocation)

Link Error: RADOS library not found
make[3]: ***
[src/pybind/rados/CMakeFiles/cython_rados.dir/build.make:57:
src/pybind/rados/CMakeFiles/cython_rados] Error 1

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TextTable: define endrow

otherwise "cmake -DWITH_ASAN=ON -DCMAKE_BUILD_TYPE=Debug" will fail to
build with

/usr/bin/ld: //var/ssd/ceph/build/lib/libceph-common.so.0: undefined
reference to `TextTable::endrow'

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: fix typos in BackoffThrottle

Signed-off-by: Shiyang Ruan <ruansy.fnst@cn.fujitsu.com>

PendingReleaseNotes: note about cephfs client state reclaim

Fixes: http://tracker.ceph.com/issues/36394
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/tasks/ceph_manager: fix get_stuck_pgs from pg dump change

Fixes 95b7d2340c04dc7cf90085c89606b8c85a8f2803

Fixes: http://tracker.ceph.com/issues/36485
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #24184 into master

* refs/pull/24184/head:
mgr/DaemonServer: remove any upmaps on merging PGs
mgr/DaemonServer: prevent merge if either pg is remapped|upmap
mgr/DaemonServer: move pending merge check for more consistent code
qa/suites/rados/thrash*/thrashers/careful.yaml: thrash with mgr controller
mgr/DaemonServer: add option to bypass careful throttling for thrasher
PendingReleaseNotes: note about mgr/balancer/max_misplaced change
mgr/DaemonServer: remove stale/misleading check
mgr/DaemonServer: throttle pgp_num changes based on misplaced %
mgr/DaemonServer: block pg_num decrease(merge) until pgp_num is reduced
mgr/DaemonServer: adjust_pgs(): cosmetic change to debug output
mon/PGMap: add get_recovery_stats()
mgr/balancer: mgr/balancer/max_misplaced -> pg_max_misplaced
pybind/mgr/mgr_module: add get_option()
mgr/DaemonServer: allow pg_num increases that abort pending merges
mon/OSDMonitor: resent pre-nautilus client ops on aborted merge
mon/OSDMonitor: make pgp_num track pg_num more consistently

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #24654 into master

* refs/pull/24654/head:
osd: remove unused parameter 'dev' in OSD::mkfs function

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: remove any upmaps on merging PGs

Remove any pg_upmap[_items] on pgs that are merging to ensure that they
land on the same OSDs.

This is a bit sloppy: we *could* set the source upmap to match the target
upmap (vs potentially moving both PGs to a third location, and/or then
having the balancer move the resulting PG somewhere else again), but for
now assume upmaps are not a common case and Keep It Simple.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: prevent merge if either pg is remapped|upmap

Remapping means they could be on different OSDs.

Fixes: http://tracker.ceph.com/issues/36166
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: move pending merge check for more consistent code

No functional change, but this makes the code simpler to read.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash*/thrashers/careful.yaml: thrash with mgr controller

Thrash such that we still exercise the careful throttling in the mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: add option to bypass careful throttling for thrasher

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20581 from chrone81/patch-1

doc: Fix EC k=3 m=2 profile overhead calculation example.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22349 from gregsfortytwo/wip-24368-osd-restarts

systemd: only restart 3 times in 30 minutes, as fast as possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24646 from dillaman/wip-36500

rbd-mirror: always attempt to restart canceled status update task

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24670 from falcon78921/wip-doc-multisite-link

doc: updated wording from federated to multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24595 from cbodley/wip-24505

rgw: user stats account for resharded buckets

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #24673 from tchaikov/wip-librbd-test-os-cleanup

librbd,test: remove unused context_cb() function, silence GCC warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24645 from cbodley/wip-36496

cls/user: cls_user_remove_bucket writes modified header

Merge PR #24611 into master

* refs/pull/24611/head:
mgr/diskprediction: Solve predictor model file not backward compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24623 into master

* refs/pull/24623/head:
mgr: Mutex::Locker -> std::lock_guard
kv: Mutex::Locker -> std::lock_guard
librados: Mutex::Locker -> std::lock_guard
mon: Mutex::Locker -> std::lock_guard
osd: Mutex::Locker -> std::lock_guard

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24605 from tchaikov/wip-use-ccache-the-old-way

cmake: fix compilation with distcc and other compiler wrappers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

blkdev: FreeBSD support

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: overhaul API for better portability

* Turn the API into a class so it will work with GoogleMock
* Take file descriptors for all methods, instead of a mix of file
  descriptors, path names, and canonical device names.  It's more
  consistent and it will work better with FreeBSD.
* Split get_device_by_fd into block_device_partition and
  block_device_wholedisk
* Remove dead code

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: move test_get_blkdev_size to test_get_blkdev_props

Also, enhance blkdev test program to show more properties than just size

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: Add dedicated accessors for all property functions

Hide get_block_device_string_property and get_block_device_int_property
to lower risk of misuse.

Signed-off-by: Alan Somers <asomers@gmail.com>

blkdev: add an enum type for block device properties

Signed-off-by: Alan Somers <asomers@gmail.com>

Merge pull request #21155 from runsisi/wip-auth-cleanup

auth/cephx: minor code cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24672 from SUSE/wip-orchestrator-result-property

mgr/orchestrator: use result property in Completion classes

ceph-volume: extract flake8 config

It's preferable to use [flake8] section to configure flake8.

So external tools/editor can read the configuration.

Signed-off-by: Mehdi Abaakouk <sileht@sileht.net>

Merge pull request #24314 from rhcs-dashboard/dashboards

mgr/dashboard: Grafana dashboard updates and additions

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #24628 from Devp00l/wip-issue-36445

mgr/dashboard: Add missing test requirement "werkzeug"

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #24669 from falcon78921/wip-docs-36527

doc: add spacing to subcommand references

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19386 from ganeshmaharaj/qat-ceph-crypto

crypto: QAT based Encryption for RGW

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: remove unused context_cb() function

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/objectstore: silence "-Wsign-compare" warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: use result property in Completion classes

The _Completion class defined a get_result() method, but the orchestrator
CLI and Rook module both just expected a result property, so let's run with
the property version.

Signed-off-by: Tim Serong <tserong@suse.com>

auth/cephx: remove unused member method

Signed-off-by: runsisi <runsisi@zte.com.cn>

auth: remove unused forward declaration

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #24618 from CharlotteRose/wip-number-eviction-events

doc: numbered eviction situations

Reviewed-by: Jos Collin <jcollin@redhat.com>

crypto: Merge compilation of crypto accelerators under a single cmake.

- Control ISA-L and QAT under a crypto makefile
- Allow conditional compile of QAT.

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>

crypto: QAT Wrapper for Ceph Encryption

- Crypto plugin created and currently handles only AES_256_CBC
- Contents
-- Poller thread for QAT HW
-- Contiguous memory allocator using QAT's usdm driver
-- Do in-place crypto operation reducing mem usage.
-- Enable multi-QAT instance to allow parallel operations.

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>

doc: updated wording from federated to multisite

Updated the terminology from federated architecture
to multisite configuration. Updated the reference link for multisite
configuration.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: add spacing to subcommand references

Added appropriate spacing for subcommand
code references.

Fixes: https://tracker.ceph.com/issues/36527
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

qa/suites/rados/thrash-erasure-code*/thrashers/*: less likely resv rejection injection

For EC pools we have a lot of shards, and 30% probability on each one
means we are very like to repeatedly fail backfill reservations.. long
enough that teuthology gives up waiting.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24531 from alfredodeza/wip-rm36386

ceph-volume remove version reporting from help menu

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Fixed documentation link on RGW page

Fixes: https://tracker.ceph.com/issues/24548
Signed-off-by: Tina Kallio <tina.kallio@gmail.com>

Merge pull request #24453 from cbodley/wip-rgw-putobjprocessor

rgw: refactor PutObjProcessor stack

Merge pull request #23304 from smithfarm/wip-23776

 tools: ceph-authtool: report correct number of caps when creating keyring

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: numbered eviction situations

Signed-off-by: Scoots Hamilton <scoots@redhat.com>

Merge pull request #19571 from Fusl/patch-1

doc: doc/radosgw/s3.rst: Adding AWS S3 `Storage Class` as `Not Supported`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24660 from tchaikov/wip-crimson-cleanup

crimson: crimson/mon: remove timeout support from mon::Client::authenticate()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24626 from jtlayton/wip-mgr-urlparse

mgr/rook: fix urljoin import

Merge pull request #24604 from yangdongsheng/token_fix

common/Throttle: TokenBucketThrottle: use reference to m_blockers.front()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>