Merge pull request #24504 from ErwanAliasr1/evelu-ceph-volume-choose_disk

Additional work on ceph-volume to add some choose_disk capabilities

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #24703 into master

* refs/pull/24703/head:
common: prefer std::size to ARRAY_SIZE macro

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24712 into master

* refs/pull/24712/head:
mgr/DaemonServer: "osd safe-to-destroy" - more verbose output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24713 into master

* refs/pull/24713/head:
mon: drop repeated 'goodchars' and add osd crush ls testcase

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #24724 into master

* refs/pull/24724/head:
test/librados/aio: remove unused callbacks
test/librados/aio: wait for all completions properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24736 into master

* refs/pull/24736/head:
os/tests: should read size_t options using get_val<Option::size_t>()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24767 from votdev/issue_36581

mgr/dashboard/qa: Fix various vstart_runner.py issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24584 from wido/docs-zabbix-template

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24606 from p-na/osd-actions-fe

mgr/dashboard: Add support for managing individual OSD settings/characteristics in the frontend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: Update mgr/zabbix plugin documentation with link to Zabbix template

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #24297 from ukernel/wip-36192

osdc: reduce ObjectCacher's memory fragments

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24756 from tspmelo/wip-contribs

.githubmap, .mailmap, .organizationmap: update contributors

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #24727 from zmc/wip-dashboard-gzip

mgr/dashboard: Enable gzip compression

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

.githubmap, .mailmap, .organizationmap: update contributors

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard/qa: CephfsTest - admin_socket() got an unexpected keyword argument 'timeout'

Adapt method arguments of LocalRemote::run() according to teuthology.orchestra.run.run() (see https://github.com/ceph/teuthology/blob/master/teuthology/orchestra/run.py#L364) to be able to run QA tests locally in a vstart cluster.

Fixes: http://tracker.ceph.com/issues/36581
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #24720 from gauravsitlani/master

doc: added rbd default features

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: added rbd default features

Fixes: https://tracker.ceph.com/issues/15000
Added RBD default features description

Signed-off-by: Gaurav Sitlani <gauravsitlani@riseup.net>

Merge PR #24716 into master

* refs/pull/24716/head:
test/log: drop redundant test case
common/StackStringStream: don't reserve before every insert

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24283 into master

* refs/pull/24283/head:
client: support for exporting multiple subdirectories in faked mode

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24050 into master

* refs/pull/24050/head:
Revert "mds: adjust export size after renaming directory into freezing subtree"
mds: cleanup nested auth pins tracking

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24700 from cbodley/wip-27221

rgw: support server-side encryption when SSL is terminated in a proxy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24721 from cbodley/wip-librados-mono-timeout

librados: fix unitialized timeout in wait_for_osdmap

Reviewed-by: Mohamad Gebai <mgebai@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24680 from cbodley/wip-36537

rgw: fix vector index out of range in RGWReadDataSyncRecoveringShardsCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24677 from cbodley/wip-rgw-beast-derr

rgw: beast frontend logs socket errors at level 4

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #24731 into master

* refs/pull/24731/head:
rados: fix admin/build-doc warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24641 from ricardoasmarques/fix-cloning-message

mgr/dashboard: Fix long running RBD cloning / copying message

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24587 from ceph/wip-rm36363

ceph-volume: adds a --prepare flag to `lvm batch`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24750 from iotcg/master

doc: give pool name if default pool rbd is not created

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: give pool name if default pool rbd is not created

unless creating the default pool named to be "rbd",
it will fail without pointing the pool name when
using rbd command.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

ceph-volume: update man page for batch --prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: Fix long running RBD cloning / copying message

Fixes: https://tracker.ceph.com/issues/36484
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #24693 from idryomov/wip-is-queued

msg/async: fix is_queued() semantics

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add support for managing individual OSD settings/characteristics in the frontend

Fixes: http://tracker.ceph.com/issues/35448
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #24739 from tspmelo/wip-py3-itervalues

ceph.in: Add support for python 3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23962 from aclamk/fio-fix-compilation

objectstore/test/fio: Fixed fio compilation when tcmalloc is used.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Enable gzip compression

This is related to http://tracker.ceph.com/issues/36453. It is far from
a complete solution, but seems like a positive move.

I tested this change by first disabling my browser cache, and then used
the /docs endpoint to query /api/dashboard/health. Before compression:
  Content-Length: 60748
  Time: 615ms
After:
  Content-Length: 7505
  Time: 92ms

Then, I logged into the dashboard as normal and reloaded the page once I
was in. Some values for the reload operation before compression:
  Total page load time: 58.48s
  vendor.js Content-Length: 6486025
  vendor.js time: 48.09s
After:
  Total page load time: 14.55s
  vendor.js Content-Length: 1143178
  vendor.js time: 4.50s

Signed-off-by: Zack Cerza <zack@redhat.com>

DashboardTestCase: add assertHeaders()

Signed-off-by: Zack Cerza <zack@redhat.com>

LocalCephManager.admin_socket: add timeout kwarg

This fixes "TypeError: admin_socket() got an unexpected keyword argument
'timeout'". The value is never used.

Signed-off-by: Zack Cerza <zack@redhat.com>

ceph.in: Add support for python 3

This will allow the condition to work in both python 2 and 3 environments.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24738 from alfredodeza/wip-rm36492

ceph-volume: do not send (lvm) stderr/stdout to the terminal, use the logfile

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24665 into master

* refs/pull/24665/head:
common: remove unused ioprio thread/wq infrastructure

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #24701 into master

* refs/pull/24701/head:
os/bluestore: fix race between SharedBlobSet::lookup and SharedBlob::put

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #24707 into master

* refs/pull/24707/head:
build/ops: drop sgdisk build/runtime dependencies

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #24687 into master

* refs/pull/24687/head:
src/test: fix unordered manifest-unset op

Reviewed-by: Sage Weil <sage@redhat.com>

rados: fix admin/build-doc warnings

Fixed admin/build-doc warnings in librados.h
and rados_types.h

Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph-volume tests.api update monkeypatching to use **kw

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests allow **kw when monkeypatching

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm: do not spit out errors on the terminal from LVM

Signed-off-by: Alfredo Deza <adeza@redhat.com>

os/tests: should read size_t options using get_val<Option::size_t>()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24722 from zmc/rbd-redirect

mgr/dashboard: Redirect /block to /block/rbd

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Simplify frontend test helper

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: TableActionsComponent cleanup

Rename `buttonCondition` to `canBePrimary`

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Rename `DeletionModalComponent` to `CriticalConfirmationModalComponent`

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Cleanup OSD components

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

common: prefer std::size to ARRAY_SIZE macro

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #19964 from runsisi/wip-build

cmake: move ceph-osdomap-tool, ceph-monstore-tool out of ceph-test

Reviewed-by: Kefu Chai <kchai@redhat.com>

librados: fix unitialized timeout in wait_for_osdmap

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #24540 into master

* refs/pull/24540/head:
mgr/status: fix fs status subcommand did not show standby-replay MDS' perf info

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24709 into master

* refs/pull/24709/head:
PendingReleaseNotes: note about cephfs client state reclaim

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24466 into master

* refs/pull/24466/head:
common: assert if buffer advance length overflow

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24529 into master

* refs/pull/24529/head:
mds: cleanup command descriptions

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #24640 into master

* refs/pull/24640/head:
mds: remove MonClient reconnect

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24533 into master

* refs/pull/24533/head:
qa: add timeouts for remote ops for client mounts

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24679 from tspmelo/wip-36528

mgr/dashboard: Fix HttpClient Module imports in unit tests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24632 from Devp00l/wip-issue-36468

mgr/dashboard: Handle class objects as regular objects in KV-table

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test/librados/aio: remove unused callbacks

That callbacks are called is covered by many other tests.
These particular m_safe and m_complete fields were unused.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/librados/aio: wait for all completions properly

Using global semaphores was error-prone and unnecessary. Unbalanced
usage resulted in a race like:

aio_write object foo -> dropped on full OSD
aio_read object foo -> -ENOENT

The intent of the test was to wait for the write to complete, so
do that explicitly for each aio test and remove the semaphores.

Fixes: http://tracker.ceph.com/issues/24587
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24674 from sileht/master

ceph-volume: extract flake8 config

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Redirect /block to /block/rbd

Fixes: https://tracker.ceph.com/issues/36401
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #23242 from linuxbox2/wip-rgw-inflight-ops

rgw: set default objecter_inflight_ops = 24576

common: remove unused ioprio thread/wq infrastructure

This is all now obsolete, with no remaining users.

Move ceph_gettid into Thread.{cc,h}

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #24692 from tchaikov/wip-cmake-with-asan

cmake: fixes to enable WITH_ASAN with clang and GCC

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24664 from tspmelo/wip-36513

mgr/dashboard: Fix missing failed restore notification

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24637 from ricardoasmarques/fix-rbd-actions-disable

mgr/dashboard: Fix RBD actions disable

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24612 from Tinatinak/fix_bug24548

mgr/dashboard: Fixed documentation link on RGW page

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix HttpClient Module imports in unit tests

Some of the tests were importing the wrong HttpClient Module and that was
causing random errors.

Fixes: https://tracker.ceph.com/issues/36528
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix missing failed restore notification

Fixes: http://tracker.ceph.com/issues/36513
Signed-off-by: Tiago Melo <tmelo@suse.com>

mon: drop repeated 'goodchars' and add osd crush ls testcase

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mgr/DaemonServer: "osd safe-to-destroy" - more verbose output

For multiple inputs, we want to avoid a situation like:

```
$ ceph osd safe-to-destroy 0 1 2
Error EBUSY: OSD(s) 1,2 have 96 pgs currently mapped to them
$ ceph osd safe-to-destroy 0
Error EBUSY: OSD(s) 0 last reported they still store some PG data, and not all PGs are active+clean; we cannot be sure they aren't still needed.
```

The first command seems to be implying osd.0 is safe-to-destroy whereas it isn't.
This patch instead gives a complete report of each osd if multiple osds are specified.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24544 from votdev/bug_24729

mgr/dashboard: Confirmation modal doesn't close

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24699 from rhcs-dashboard/add-tslint-rule

mgr/dashboard: Add TSLint rule "no-unused-variable"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24694 from tspmelo/wip-36544

mgr/dashboard: Fix problem with ErasureCodeProfileService

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24696 from dillaman/wip-36542

qa/tasks/qemu: use unique clone directory to avoid race with workunit

Reviewed-by: Mykola Golub <mgolub@suse.com>

build/ops: drop sgdisk build/runtime dependencies

sgdisk was a dependency of ceph-disk, which has been ripped out.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24698 into master

* refs/pull/24698/head:
Merge PR #24697 into nautilus
ceph_test_msgr: fix authorizer behavior
Merge pull request #24667 from liewegas/wip-ec-thrash-full
Merge PR #24689 into nautilus
qa/suites/rados/thrash-erasure-code*/thrashers/*: less likely resv rejection injection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24691 from iridescent-rsy/fix_typo

common: fix typos in BackoffThrottle

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24697 into nautilus

* refs/pull/24697/head:
ceph_test_msgr: fix authorizer behavior

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #24652 from badone/wip-undefined-gperftool-symbols-2

rpm: Use updated gperftools-libs at runtime

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

os/bluestore: fix race between SharedBlobSet::lookup and SharedBlob::put

A                             B
SharedBlobSet::lookup()
  takes lock
  nref is not 0
                              SharedBlob::put()
                                --nref
returns SharedBlobRef,
  ++nref
                                takes cache lock
                                SharedBlobSet::remove
                                  takes lock
                                  removes
                                deletes SharedBlob

-> A ends up with a ref to deleted SharedBlob

Fix by verifying that nref is still zero in SharedBlobSet::remove(),
while we are holding the SharedBlobSet::lock.  The lock ensures that we
have increased the ref for the lookup before entering remove, so we can
verify that nref is still zero before removing it.  If not, we have
raced, and put() bails out and does nothing.

Fixes: http://tracker.ceph.com/issues/36526
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: aws:SecureTransport uses rgw_transport_is_secure()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SSE operations check rgw_transport_is_secure()

Fixes: http://tracker.ceph.com/issues/27221
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add helper function rgw_transport_is_secure()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add config option rgw_trust_forwarded_https

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/log: drop redundant test case

After the recent logging rework, ManyGatherLog and
ManyGatherLogStringAssign are identical barring the string.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

common/StackStringStream: don't reserve before every insert

Unlike ConcreteEntry, MutableEntry can be appended to.  Reserving the
exact number of elements before every append is harmful: vector will
likely reallocate each time and grow linearly instead of geometrically.
This results in quadratic behaviour when we spill past the preallocated
capacity and doesn't benefit the fast path in any way.

The new test case takes half a second with this patch and many hours
spinning in memmove without this patch.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: Add TSLint rule
- TSlint no-unused-variable rule added.
- Cleanup: unused imports and variables.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #24591 from tspmelo/wip-testing-module

mgr/dashboard: Unit Tests cleanup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

ceph_test_msgr: fix authorizer behavior

Fixes breakage from this PR 2152d8ffb73a507a3d08d48b38c5a8e73f887138.

Fixes: http://tracker.ceph.com/issues/36495
Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_assert.h: do not pack assert params if WITH_ASAN

we pack the asset() params for smaller code size, but this creates a
inlined `assert_data_ctx` instance for every compilation unit which
call ceph_assert() defined in .h .

__PRETTY_FUNCTION__ is likely to be referenced by `assert_data_ctx`
sections which are included by different compiled object files. if the
ceph_assert() call is used by header file, then there will be multiple
`assert_data_ctx` sections sharing the same identifier. these sections are
defined as "COMDAT" group sections, i.e. common data sections. when linker
see multiple COMDAT sections with the same identifer, it will simply discard
the duplicated ones, and only keep a single copy of them. without enabling
ASan, GCC can always handle this problem just fine. but the dedup feature
does not work well with ASan. if ASan is enabled, and we link the objects
with the wrong order, some references will be pointing to the discarded
sections.

to address this issue, we could audit the link command line and inspect
all .o files to make sure they are properly ordered. but this is
non-trivial. as a workaround, in this change, the assert params are not
packed, and sent to the  __ceph_assert_fail() overrides which accepts
unpacked params directly, so the COMDAT section is not created.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/qemu: use unique clone directory to avoid race with workunit

If there is a workunit task associated with the same client, the two
tasks will attempt to clone the suite repo to the same directory.
Worse, if it's parallel tasks, the two clones will clobber each
other.

Fixes: http://tracker.ceph.com/issues/36542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>