osd: do not include Messenger.h if not necessary

to speed up the compilation, do not include Messenger.h, and use forward
declaration instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22478 from liewegas/wip-24452

osd/PG: create new PGs from activate in last_peering_reset epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22428 from dzafman/wip-24396-fix

test: Use a file that should be on all OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPGScan: encode map_epcoh for query_epoch for pre-nautilus peers

Pre-nautilus OSDs do not create last_peering_reset reliably (due to not
having the previous commit's fix in place).  Avoid breaking them during
an upgrade by encoding the map_epcoh in place of the query_epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: create new PGs from activate in last_peering_reset epoch

If we create a new PG (e.g., a backfill target) in the current epoch, it
might be > last_peering_reset.  That can lead to last_peering_reset on
the replica having a higher last_peering_reset than the primary's, which
can then lead to future messages, like pg_scan during backfill, being
ignored.

Fixes: http://tracker.ceph.com/issues/24452
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22462 from liewegas/wip-bluestore-no-meta-paths

os/bluestore: don't store/use path_block.{db,wal} from meta

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22426 from alfredodeza/wip-rm24020

ceph-volume expand on the LVM API to create multiple LVs at different sizes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume util make chown links affect links as well as paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate chown bluestore symlinks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: make bad symlink target error messages more informative

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22439 from shaba/devel/fix-rocksdb

cmake: fix find system rockdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22430 from tchaikov/wip-find_package-oath

cmake: find liboath using the correct name

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #22453 from tspmelo/fix-pool-usage

mgr/dashboard: Fix pool usage not displaying on filesystem page

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22443 from dillaman/wip-doc-rbd-openstack

doc/rbd: corrected OpenStack Cinder permissions for Glance pool

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #22447 into master

* refs/pull/22447/head:
osd/OSDMap: fix pre-addrvec compat
mon/MgrMap: fix pre-addrvec compat
messages/MMgrBeacon: fix pre-addrvec compat

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: don't store/use path_block.{db,wal} from meta

This reverts most of 8d07fa5ad0772b4d979dc10695d230074d1f1e69.

ceph-volume goes to great lengths to ensure that the symlinks in the
osd dir are accurate.  Having these values here is an opportunity to
get them out of sync.  And that can happen very easily if the initial
mkfs was performed using a /dev/sdX device name (which is unstable
across reboots).  Even after ceph-volume corrects the symlink, bluestore
will continue to use the stale device path.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22403 from ricardoasmarques/wip-http-401

mgr/dashboard: Login failure should return HTTP 400

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22437 from alfredodeza/wip-cv-osd-destroy

ceph-volume tests destroy osds on monitor hosts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #22445 into master

* refs/pull/22445/head:
doc: fix release notes on snapshot upgrades

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

cmake: fix find system rockdb

- fix path for version.h
- fix typo ROCKDB -> ROCKSDB

Signed-off-by: Alexey Shabalin <shaba@altlinux.org>

mgr/dashboard: Fix pool usage not displaying on filesystem page

Reference to the template was undefined during the init of the component.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22451 from wwformat/wip-gcc7-aarch64

install-deps.sh: support install gcc7 in xenial aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: support install gcc7 in xenial aarch64

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>

osd/OSDMap: fix pre-addrvec compat

This was broken by b47d9135d5345816037899cd1d12a5a20b54e6c1.  Old code
does not decode an addrvec as an addr.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMap: fix pre-addrvec compat

This was broken by 7f787704cdcd55df18d204c5251cdd6a3cedd604.  Old code
does not decode an addrvec as an addr.

Take the opportunity to remove the old_available_modules field.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MMgrBeacon: fix pre-addrvec compat

This was broken by 7f787704cdcd55df18d204c5251cdd6a3cedd604.  Old code
doesn't not decode an addrvec as an addr.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22388 into master

* refs/pull/22388/head:
ceph-mds: Fix usage
doc: ceph-mds option updates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22338 into master

* refs/pull/22338/head:
doc: update toctree for add-remove-mds.rst
doc: create add/remove metadata server

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fix release notes on snapshot upgrades

Fixes: https://tracker.ceph.com/issues/24435
Introduced-by: d78dda7b812c63b5504f69e9df78bea4fbb37394.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests functional osd destroy on mons only for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rbd: corrected OpenStack Cinder permissions for Glance pool

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22282 from rjfd/wip-dashboard-api-docs

mgr/dashboard: Swagger-UI based Dashboard REST API page

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22387 from votdev/missing_space_value_unit

mgr/dashboard: Print a blank space between value and unit

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-volume tests/functional destroy filestore osds on the monitors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Swagger-UI based Dashboard REST API page

Fixes: http://tracker.ceph.com/issues/23898
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21279 from sungjunyoung/master

doc: erasure-code-jerasure: removed default section of crush-device-class

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: find liboath using the correct name

we should reference liboath by the $name in Find${name}.cmake, also the
$name should be consistent when calling find_package_handle_standard_args().
in this change
* rename Findliboath.cmake to FindOATH.cmake to be consistent with other
  find_package() moduless.
* use "OATH" in find_package_handle_standard_args() instead of "oath"
* set the interface properties for OATH::OATH, so the target linking
  against it can reference its header directories and libraries automatically.
* remove the stale comment for find_package_handle_standard_args()
* set OATH_INCLUDE_DIRS and OATH_LIBRARIES to follow the convention of
  find_package(), even they are not used directly in this project.

Reported-by: Erwan Velu <erwan@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests/functional destroy bluestore osds on the monitors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Print a blank space between value and unit.

When displaying values with their unit a blank space must be shown between them, see #15 at https://physics.nist.gov/cuu/Units/checklist.html.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Login failure should return HTTP 400

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21644 from p-na/grafana-proxy

mgr/dashboard: Grafana proxy backend

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #21880 from votdev/upgrade_swimlane_datatable_angular_6

mgr/dashboard: Upgrade Swimlane's data-table

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22386 from tspmelo/fix-24281

mgr/dashboard: Changed default value of decimal point to 1

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22404 from ricardoasmarques/wip-optimize-rbd-edit

mgr/dashboard: Do not fetch pool list on RBD edit

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22409 from wido/mgr-dashboard-port

mgr/dashboard: Listen on port 8443 by default and not 8080

Reviewed-by: Kai Wagner <kwagner@suse.com>

mgr/dashboard: Add Grafana proxy

Add the ability to connect the dashboard with a Grafana instance to
display graphs from Grafana incorporated into the dashboard.

Possible configuration options:

$ ceph dashboard set-grafana-api-host <host> # Default: 'localhost'
$ ceph dashboard set-grafana-api-port <port> # Default: 3000
$ ceph dashboard set-grafana-api-scheme <scheme> # Default: 'http'

The configuration can be checked using a GET request to the API endpoint
`/api/grafana/status`. It returns a JSON response in the following
format:

    {
"available": False,
"message": "<error message>",
    }

The message is not returned if Grafana is available.

The Grafana proxy is reachable through
`http(s)://<host>/api/grafana/proxy`.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

doc: erasure-code-jerasure: remove crush-device-class option's unused default

Signed-off-by: Junyoung Sung <junyoung.sung@navercorp.com>

Merge pull request #22365 from tone-zhang/doc_spdk

doc: update the description for SPDK in bluestore-config-ref.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22356 from tone-zhang/wip-bug-24371

spdk: fix ceph-osd crash when activate SPDK

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22329 from bstillwell/transaction_code_comments_corrections

os: Minor fixes in comments describing a transaction

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22371 into master

* refs/pull/22371/head:
doc/conf.py: fix man page build vs governance.rst
doc/governance: adjust title
doc/governance: fix link
doc/governance: edits and add user committee
doc/governance.rst: a few notes on ceph project governance

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

doc/conf.py: fix man page build vs governance.rst

Signed-off-by: Sage Weil <sage@redhat.com>

doc/governance: adjust title

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22408 from wumingqiao/wip-pgscan-cleanup

messages/MOSDPGScan: fix initialization of query_epoch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: Use a file that should be on all OSes

Also, create temporary files in test specific dir and remove

Caused by: 154330fd68d952a4e1b972891f02b6dc9c355424

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #22420 from b-ranto/wip-mgr-fix-avg

mgr: expose avg data for long running avgs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22396 from liewegas/wip-24396

osd/PrimaryLogPG: fix on_local_recover crash on stray clone

Reviewed-by: David Zafman <dzafman@redhat.com>

osd/PrimaryLogPG: fix on_local_recover crash on stray clone

If there is a stray clone (one that does not appear in the SnapSet) and
we do any sort of recovery on it the OSD will crash.  Log an error instead
but continue.

This addresses a problem where a cluster has both (1) an unexpected clone
and (2) the clone is not present on all replicas.  Doing repair on that
PG will both not fix the unexpected clone and also cause the remaining
OSDs to crash trying to recover it.

Include a test.

Fixes: https://tracker.ceph.com/issues/24396
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21999 from myoungwon/wip-manifest-unset-op

osd, librados: add unset-manifest op

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22395 from liewegas/wip-24408

osd,mds: make 'config rm ...' idempotent

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #21775 from Synesis-LLC/fix-23964

librados: fix buffer overflow for aio_exec python binding

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume api.lvm fallback to 1 part if none are define when creating lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify create_lvs behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm default to null tags on lvs created

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-deploy api.lvm allow multiple lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm add a sizing method for VGs to calculate LV creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume exceptions create an error for size allocations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify vg sizing behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm force vgs to display units in 'g'

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22417 from tspmelo/fix-lint-error

mgr/dashboard: fix lint error

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Listen on port 7000 by default and not 8080

Port 8080 is a common alternative HTTP port used for web traffic.

The dashboard however uses SSL (which can not be turned off) and for
that purpose we should either use 8443.

Historically the dashboard would run on port 7000 so to keep this backwards
compatibility the dashboard should bind again on TCP 7000.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #22402 from ricardoasmarques/wip-stacktrace-optional

mgr/dashboard: Stacktrace is optional on 'js-error' endpoint

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc/governance: fix link

Signed-off-by: Sage Weil <sage@redhat.com>

doc/governance: edits and add user committee

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Upgrade Swimlane's data-table

Upgrade to latest version 13.0.1 which now supports Angular 6. Check https://github.com/swimlane/ngx-datatable/blob/13.0.1/docs/changelog.md for more information.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #22399 from alfredodeza/wip-rm24417

ceph-volume tests do not include admin keyring in OSD nodes

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr: expose avg data for long running avgs

While exposing the long running avgs, the order of regular and long
running avg data got mixed up resulting in exposing empty data for both
since there is no regular data for long running avgs and vice versa.
This commit fixes this issue by changing the order of the if/else
statement exposing regular data for regular counters and avg data for
long running avgs.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #22401 from ricardoasmarques/wip-prettify-package-json

mgr/dashboard: Prettify package.json

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: fix lint error

Error was introduced in 78dc50443cf64168098f1fb5bc43b7d6125621b8.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21826 from tspmelo/rxjs6

mgr/dashboard: Update RxJS to version 6

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22122 from tspmelo/wip-osd-scrub

mgr/dashboard: Add scrub action to the OSDs table

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Changed default value of decimal point to 1

Changed the default decimals point to 1 in 'formater_number' method, this
is used in the 'dimlessPipe' and 'dimlessBinaryPipe'.
This value can still be changed when needed, using one of the parameters.

I have also removed the 'truncate' method.

Fixes: https://tracker.ceph.com/issues/24281
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22392 from tchaikov/wip-cmake-boost-1.67

cmake: bump up the required boost version to 1.67

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Hide `required` for read-only fields on RBD edit

Signed-off-by: Ricardo Marques <rimarques@suse.com>

messages/MOSDPGScan: fix initialization of query_epoch

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

ceph-mds: Fix usage

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22405 from neha-ojha/wip-24365

qa/tasks/cbt.py: changes to run on bionic

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/cbt.py: changes to run on bionic

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Do not fetch pool list on RBD edit

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Stacktrace is optional on 'js-error' endpoint

Signed-off-by: Ricardo Marques <rimarques@suse.com>

osd,mds,common: make 'config rm ...' idempotent

Fixes: http://tracker.ceph.com/issues/24408
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prettify package.json

Signed-off-by: Ricardo Marques <rimarques@suse.com>

ceph-volume tests do not include admin keyring in OSD nodes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22393 from wido/ceph-volume-bluestore-tool

ceph-volume lvm.activate Do not search for a MON configuration

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

common/config: fix rm_val error codes

EINVAL if the config option is invalid.
ENOENT if the config option wasn't set.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22359 into master

* refs/pull/22359/head:
osdc/objecter: remove unused method finish_op

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: bump up the required boost version to 1.67

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Add scrub action to the OSD table

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add OSD scrub modal component

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add scrub methods to OSD controller and service

Signed-off-by: Tiago Melo <tmelo@suse.com>