Merge pull request #23995 from tchaikov/wip-add-ceph-libboost-repo

install-deps,run-make-check: use ceph-libboost repo

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #24067 from libingyang-zte/master

doc: Fix typos in Developer Guide

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix Spelling Error in Developer Guide

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23919 from adamemerson/wip-end-to-warnings

Clean up warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24018 from dzafman/wip-35912

qa/standalone: Minor test improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24060 from alfredodeza/wip-rm35913

ceph-volume batch: allow --osds-per-device, default it to 1

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.lvm.strategies initial filestore validation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.tests conftest fixtures for custom vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.lvm add bluestore strategy cases

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify strategy validators for minimum size

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.filestore allow osds-per-device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

cls_cas: Fix format string warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #24025 into master

* refs/pull/24025/head:
mon/OSDMonitor: reject pg_num -> 0

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21611 into master

* refs/pull/21611/head:
osd/ECBackend: don't get result code of subchunk-read overwritten

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #23273 into master

* refs/pull/23273/head:
os/bluestore: handle spurious read errors

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23694 into master

* refs/pull/23694/head:
mon: remove pre-luminous health output compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

test: Remove unused variable

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mds: Remove pessimizing move

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Silence unused capture warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Fix sprintf overflow warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Quell unused variable warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Fix sign comparison warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #24035 into master

* refs/pull/24035/head:
osd/PG: avoid choose_acting picking want with > pool size items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #24052 into master

* refs/pull/24052/head:
doc: update mds stat output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24032 from liewegas/wip-pg-ls

mon/PGMap: include unknown PGs in 'pg ls'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #24040 from rzarzynski/wip-common-bl-drop-raw_mmap_pages

common: drop the unused buffer::raw_mmap_pages.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs: Do not catch polymorphic exception by value

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

crush: Quell unused variable warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph-volume lvm.batch.validators account for osds per device, include collocation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch.bluestore allow osds-per-device with MixedType

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24042 from liewegas/wip-merge-derr

osd: remove stray derr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23996 from mogeb/fmt-cmake

cmake: fix version matching for Findfmt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24013 from dzafman/wip-35845

test: Use a grep pattern that works across releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24045 from dillaman/wip-35702

librbd: fixed assert when flattening clone with zero overlap

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24030 from LenzGr/mgr-dashboard-documentation

doc: Updated Mgr Dashboard documentation

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24055 from sebastian-philipp/kubejacker-opensuse-mimic

script/kubejacker: Add openSUSE based images

Reviewed-by: John Spray <john.spray@redhat.com>

script/kubejacker: Add openSUSE based images

* Added executable bit of `kubejacker.sh`.
* Reduced the number of `RUN` commands
* Added zypper commands for SUSE.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: update mds stat output

Signed-off-by: Jos Collin <jcollin@redhat.com>

librbd: fixed assert when flattening clone with zero overlap

Fixes: http://tracker.ceph.com/issues/35702
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: reject pg_num -> 0

Otherwise it will finally crash the mgr process:
```
-2> 2018-09-11 16:23:31.272 7ff521d59700 -1 /clove/xxG/ceph/src/osd/osd_types.cc: In function 'pg_t pg_t::get_parent() const'
 thread 7ff521d59700 time 2018-09-11 16:23:31.272096
/clove/xxG/ceph/src/osd/osd_types.cc: 587: FAILED ceph_assert(bits)
```

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23941 from trociny/wip-23445

librbd: flatten operation should use object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22675 from nuyhah/fix_block_name_prefix

librbd: block_name_prefix is not created randomly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23790 from iridescent-rsy/myfix

librbd: remove template declaration of a non-template function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23853 from zy751713126/add_protect_snapshot_list

rbd: add protected in snap list

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24009 from trociny/wip-test-migration-other-pool

test/librbd: migration tests did not delete additional pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: remove stray derr

This was debug output that crept into the final commit.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: block_name_prefix is not created randomly

fixes: http://tracker.ceph.com/issues/24634
Signed-off-by: hyun ha hyun.ha@navercorp.com

mon/PGMap: allow 'pg ls unknown'

"unknown" is annoying because it is 0 and needs a special case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23702 from Rubab-Syed/rgw-orchestrator-rook

mgr/orchestrator: Add RGW service support

Reviewed-by: John Spray <john.spray@redhat.com>

mon/PGMap: include unknown PGs in 'pg ls [all]'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23908 into master

* refs/pull/23908/head:
client: retry remount on dcache invalidation failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23691 into master

* refs/pull/23691/head:
client:two ceph-fuse client, one can not list out files created by another.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23323 into master

* refs/pull/23323/head:
client: set f_files to the total number of files in the filesystem

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23314 into master

* refs/pull/23314/head:
mds: include session uptime when diplaying session list
mds: track average uptime of sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23954 from zy751713126/clear_snap_limit

rbd: simplified code to remove do_clear_limit function

Merge PR #23198 into master

* refs/pull/23198/head:
mds: optimize acquiring locks
mds: reduce memory fragments of EMetaBlob
mds: optimize memory allocation in CInode::encode_inodestat
mds: convert MDSCacheObject::ref_map to flat_map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23199 into master

* refs/pull/23199/head:
script: backport-create-issue: optionally take list of issue numbers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23728 into master

* refs/pull/23728/head:
doc: suppored features of kernel client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common: drop the unused buffer::raw_mmap_pages.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #23824 into master

* refs/pull/23824/head:
doc/cephfs: add notes on application best practices

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

script: backport-create-issue: optionally take list of issue numbers

Make the script optionally take a comma-separated list of issue numbers.
(Could be just one issue.)

Before this patch, backport-create-issue script insisted on looping over all
issues in Pending Backport status. This was cumbersome in cases when only
one issue (or a couple issues) needed to be processed.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PG: avoid choose_acting picking want with > pool size items

If the pool size recently changed, we might see an up that is larger than
the pool size.  Or, we might already have selected a primary that is not
part of up, and then add all up osds and end up with a want that is too
big.

Fixes: http://tracker.ceph.com/issues/35924
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21379 from pritha-srivastava/wip-rgw-user-policy

rgw: User Policy

ceph-volume lvm.batch.bluestore allow osds-per-device with SingleType

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.batch add --osds-per-device flag, defaulting to 1

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23497 from noahdesu/insights

mgr/insights: insights reporting module

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23146 from jcsp/wip-progress

mgr/progress: improve+test OSD out handling

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23897 from votdev/improve_delete_modal

mgr/dashboard: Make deletion dialog more touch device friendly

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #24016 from votdev/bug_35907

mgr/dashboard: Progress bar does not stop in TableKeyValueComponent

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #22987 into master

* refs/pull/22987/head:
common,rgw: rename sha1_digest_t
osd: decrement old chunk's reference count if the chunk has a reference.
src/test: add a unit test
osd: using fingerprint OID if fingerprint is set
osd: add flag interfaces in chunk_info_t
common/buffer.cc: add sha1 fingerprint
osd: add fingerprint property
mon: add a command to set fingerprint algorithm

Merge PR #24006 into master

* refs/pull/24006/head:
osd/OSD: clear ping_history on heartbeat_reset
mon/OSDMonitor: share new maps with even non-active osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #24010 into master

* refs/pull/24010/head:
osd/OSD: kick right merge source
mgr/DaemonServer: split should respect inflight creating pgs

Reviewed-by: Sage Weil <sage@redhat.com>

run-make-check,do_cmake: set BOOST_J if !WITH_SYSTEM_BOOS

it's also more readable if we can set the number of make jobs
in a single place.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: always add ${Boost_INCLUDE_DIRS} to include dirs

we need to -I${Boost_INCLUDE_DIRS} if WITH_SYSTEM_BOOST=ON and boost is
found by FindBoost.cmake.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: support multi-arch lib directory used along with BOOST_ROOT

by default, debian packaging creates ${prefix}/lib/x86_64-linux-gnu for
multi-arch support. but FindBoost.cmake does not check in such a
directory if $BOOST_ROOT is set. in this change, this directory is added
to the search path.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps,run-make-check: use ceph-libboost repo

ceph-libboost1.67 repo is setup to satisfy the libboost dependencies
used when building ceph. now it offers arm64 and amd64 builds for xenial.

* install-deps.sh: add ceph-libboost1.67 repo hosted on chacra if the
  building host does not have any of the listed ceph-libboost1.67
  packages installed
* run-make-check.sh: add "-DWITH_SYSTEM_BOOST=ON
  -DBOOST_ROOT=/opt/ceph" if include/boost/config.hpp is found
  under /opt/ceph

See-also: http://tracker.ceph.com/issues/25186
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules: add FindBoost.cmake

the FindBoost.cmake shipped by cmake 3.5.1 is not new enough to identify
the dependencies of boost 1.63 and up. so we need to copy the
FindBoost.cmake from cmake upstream. it supports up to boost 1.68.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23464 from wido/influx-optimize

mgr/influx: Use Queue to store points which need to be written

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #24019 from alfredodeza/wip-rm34535

ceph-volume  batch carve out lvs for bluestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: Updated Mgr Dashboard documentation

Fixed a typo, added a note about the embedding of Grafana dashboards
still being WIP.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

doc: suppored features of kernel client

Fixes: http://tracker.ceph.com/issues/27209
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #23957 from tchaikov/wip-crimson-logging

common,auth,crimson: add logging to crimson

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #23992 from badone/wip-librados-client-unique-ptr-compile-error

librados: Include memory for unique_ptr definition

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23723 from xiexingguo/wip-list-missing

osd/PrimaryLogPG: rename list_missing -> list_unfound command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23921 from croit/fix-35544

osd/OSDMap: add osd status to utilization dumper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23488 from xiaomanh/master

tools: correct the description of Allowed options in osdomap tool

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Updated Mgr Dashboard feature description, added references

Reworded the description of the user/role management feature, clarified
that it can be configured on both the CLI and the WebUI. Added labels for
referencing the user/role management and SSL configuration sections.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: Make deletion dialog more touch device friendly

* Refactor deletion dialog
* Add directives.module.ts to be able to use 'autofocus' in deletion dialog

Signed-off-by: Volker Theile <vtheile@suse.com>

qa: add task for progress module

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add 4th OSD to mgr test cluster

This is useful for testing progress module.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add tests for progress module

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/progress: no progress event on unmoved pgs

PGs may not be moved on osd out, if there is no suitable
location for them to move to.  In this situation
it doesn't make sense to have a progress event, as the
health warnings adequately communicate the situation.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/progress: fix PgRecoveryEvent completion cases

The event was previously not getting moved to the completed
list.  There are a couple more cases too:
 - When some pgs go away (a pool is removed) during the event
 - When the OSD comes back in after going out

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose osdmap pg_to_up_acting_osds

It's not efficient to have python calling this
O(pg_num) times to find the pgs for an OSD, but
I'm just shooting for something functional for now.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #23910 from votdev/improve_autofocus_directive

mgr/dashboard: Refactor autofocus directive

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

rbd: add protected in snap list

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

Merge pull request #23752 from ifed01/wip-ifed-fix-garbage-test

os/tests: fix garbageCollection test case from store_test suite.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24021 from libingyang-zte/master

doc: Fix Spelling Error of Radosgw

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Radosgw

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

os/bluestore: handle spurious read errors

Some kernels (4.9+) sometime fail to return data when reading
from a block device under memory pressure. This patch retries
the read if the checksum verification fails, tests show that
the first retried read succeeds in ~99.5% of the cases, so
3 attempts are made by default before giving up on the data.

Works-around: http://tracker.ceph.com/issues/22464
Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>