Merge pull request #21331 from liewegas/wip-slow-ops

qa/suites/rados/monthrash: whitelist SLOW_OPS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21317 from liewegas/wip-23622

common/Formatter: fix string_view usage for {json,xml}_stream_escaper

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21312 from wjwithagen/wjw-PG-public

osd: make PG::*Force* event structs public

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21293 from tchaikov/wip-no-venv-for-dashboard

test: only test enabled python bindings

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21287 from yanghonggang/master

os/kstore: fix statfs problem and add vstart.sh support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21174 from guzhongyan/crush-log-cleanup

crush: cleanup update_device_class() log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21110 from majianpeng/fix-objecter

osdc: dec num_in_flight for pool_dne case.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #21029 from rzarzynski/wip-osd-close_op_ctx-nohobjcpy

osd: don't memcpy hobject_t in PrimaryLogPG::close_op_ctx().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21018 from ashitakasam/wip-bug-23440

osd: remove duplicated "commit_queued_for_journal_write" in OpTracker

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20670 from chardan/jfw-wip-rand_migration-core

Extends random.h: numeric types relaxed to compatible types (with

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #21350 from tspmelo/notifications-improvement

mgr/dashboard: Notification improvements

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge pull request #20996 from rzarzynski/wip-bs-bug23246

os/bluestore: fix exceeding the max IO queue depth in KernelDevice.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21345 from ceph/fix_e2fsprogs

specs: require of e2fsprogs

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #20658 from majianpeng/cmake-link-error

os/CMakeLists: fix link errro when enable WITH_PMEM=ON.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21332 from cbodley/wip-cmake-lz4

cmake: enable LZ4 by default

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #21181 into master

* refs/pull/21181/head:
fuse: wire up fuse_ll_access

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21276 into master

* refs/pull/21276/head:
MDSMonitor: fix mds health printed in bad format

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #21323 into master

* refs/pull/21323/head:
mds: properly dirty sessions opened by journal replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21362 from dzafman/wip-hex-digest

osd: Change shard digests to hex like object info digests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21026 from ivancich/wip-rgw-unordered-listing

rgw: ability to list bucket contents in unsorted order for efficiency

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #21291 from smithfarm/wip-23596

RGWCivetWeb::read_data: fix arguments to mg_read() call

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21285 from joke-lee/wip-max_keys_convert

rgw: return EINVAL if max_keys can not convert correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21129 from oritwas/wip-rgw-admin-no-cache

rgw: radosgw-admin should not use metadata cache for readonly commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20834 from linuxbox2/wip-linked

rgw_file: conditionally unlink handles when direct deleted

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20926 from yehudasa/wip-22963-truncation

rgw: fix for usage truncated flag

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20814 from niupengju/master

rgw: Fix multisite Synchronization failed when read and write delete …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: Change shard digests to hex like object info digests

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #21140 from ceph/cv

ceph-volume: remove extra space

install-deps: add liblz4 to freebsd build dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian: add build dependency for lz4

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: add build dependency for lz4

Signed-off-by: Casey Bodley <cbodley@redhat.com>

lz4: adjusted release notes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Enable LZ4 if required library is available.

Signed-off-by: Grant Slater <grant.slater@wearefriday.com>

Merge pull request #20962 from ricardoasmarques/wip-task-manager-frontend

mgr/dashboard: Asynchronous tasks (frontend)

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Notification improvements

Notification list is now using the new CdDate pipe for formating the date and
they are now sorted desc by date.

Fixed the class name of the info icon and changed the max-height so it can work
in smaller screens.

Decreased the size of the icons.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21099 from ricardoasmarques/wip-table-selection-refresh

mgr/dashboard: Update selected items on table refresh

Reviewed-by: Volker Theile vtheile@suse.com
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21268 from smithfarm/wip-doc-config-ceph

doc: some improvements to ceph-conf.rst

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

mgr/dashboard: Asynchronous tasks front-end

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21320 from tchaikov/wip-rocksdb

rocksdb: sync with upstream

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21329 from liewegas/wip-23622-b

mon/ConfigKeyService: dump: print placeholder value for binary blobs

Reviewed-by: John Spray <jspray@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #21357 from gregsfortytwo/wip-doc-remove-naked-repo

doc: remove references to unversioned repository addresses

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21325 from xiexingguo/wip-fix-up-seg-error

 crush/CrushWrapper: fix potential invalid use of iterator

Reviewed-by: Sage Weil <sage@redhat.com>

doc: remove references to unversioned repository addresses

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon/ConfigKeyService: dump: print placeholder value for binary blobs

JSON cannot express arbitrary binary blobs.  Instead of outputting invalid
and unparseable JSON, represent the value of blobs as something like
'<<< binary blob of length 12 >>>'.

Fixes: http://tracker.ceph.com/issues/23622
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21336 from ceph/wip-yuriw-crontab

qa/tests: Changed rhel7.4 to rhel7.5

Merge pull request #21311 from liewegas/wip-validate-caps

mon: expand cap validity check for mgr, osd, mds

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #21265 into master

* refs/pull/21265/head:
doc: outline steps to take CephFS fs down

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

specs: require of e2fsprogs

in ceph/ceph-container we've realized that `e2fsprogs` isn't installed in
centos container image because ceph hasn't a dependency for it.
It has for consequence to fail when deploying a containerized cluster
with dmcrypt when using centos image.

Typical error encountered:

typical error:
```
......
get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sda5 uuid path is /sys/dev/block/8:5/dm/uuid
populate: Creating lockbox fs on %s: mkfs -t ext4 /dev/sda5
command_check_call: Running command: /usr/sbin/mkfs -t ext4 /dev/sda5
mkfs.ext4: No such file or directory
Traceback (most recent call last):
  File "/usr/sbin/ceph-disk", line 9, in <module>
    load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
......
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

rgw: ability to list bucket contents in unsorted order for efficiency

Add the ability to list the contents of a bucket without the results
being sorted. This can have performance benefits since all bucket
shards do not need to be queried. This is done by adding a parameter
to the REST query string (i.e., "allow_unordered" [note underscore]
for swift or "allow-unordered" [note hyphen] for s3) to true.

This works with other request parameters such as marker, end_marker,
and prefix. But it does not work when delimeter is used. Also, because
the results are not in order, a subsequent marker may precede an
earlier marker.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #21236 from jecluis/wip-mon-auth-bootstrap

mon/AuthMonitor: create bootstrap keys on create_initial()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21102 from trociny/wip-23487

mon: add  'ceph osd pool get erasure allow_ec_overwrites' command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21300 from dillaman/wip-rbd-mirror-active-active

rbd-mirror: integrate image map policy as incremental step to active-active

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21314 from badone/wip-ms_inject_socket_failures-SIGFPE

msg/async/AsyncConnection: Fix FPE in process_connection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21308 from dillaman/wip-23597

librbd: disable ENOENT tracking within the object cacher

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21318 from badone/wip-qa-mon-misc-add-osdmap-prune-tests

qa/standalone/mon/misc.sh: Add osdmap-prune tests

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21306 from rzarzynski/wip-bs-bug23333-allow-eio

os/bluestore: alter the allow_eio policy regarding kernel's error list.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21288 from jcsp/wip-23584

mgr/prometheus: fix PG state names

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21281 from zealoussnow/wip-leozhang-0407

common: retry_sys_call no need take address of a function pointer

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21297 from tchaikov/wip-ceph-w-utf-8

ceph: print output of "status" as string not as bytes

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #21275 from smithfarm/wip-23296

osdc/Objecter: add ignore overlay flag if got redirect reply

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21259 from rzarzynski/wip-common-hint-dout

common: hint the main branch of dout() accordingly to default verbosity.

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: silence lint warnings

this silences following errors:

9: ./awsauth.py:42:1: E722 do not use bare except'
9: ./awsauth.py:116:13: E722 do not use bare except'

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: no need to setup venv for dashboard

unlike ceph-disk and ceph-detect-init, dashboard is not a standalone
python application, it is a python application hosted by ceph-mgr,
so no need to create a venv in which it is deployed. the python env
created by tox would suffice.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/dashboard: only test enabled python bindings

we should test the python version of MGR_PYTHON_VERSION_MAJOR, if we are
testing both py2.7 and py3, we are preparing for distributing dashboard
as a separated package independent of ceph-mgr.

restructure the dashboard tests as 2 matrices, so we have 2*2 tests for
coverage and lint respectively.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20186 from jecluis/wip-codingstyle

CodingStyle: add python and javascript/typescript

CodingStyle: add python and javascript/typescript

As we are rapidly expanding our code base beyond C++ and C, it is a good
idea to enforce coding styles for other languages.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21334 from batrick/i23643

qa: fix typo in test_full

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21349 from wjwithagen/wjw-freebsd-deps

FreeBSD: add new required packages to be installed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21258 from votdev/rgw_proxy

mgr/dashboard: Implement a RGW proxy

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #21347 from SUSE/wip-doc-mgr-plugin-limitations

doc/mgr: Add limitations section to plugin guide

Reviewed-by: Lenz Grimmer <lenz@grimmer.com>
Reviewed-by: John Spray <john.spray@redhat.com>

FreeBSD: add new required packages to be installed

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/dashboard: Update selected items on table refresh

Signed-off-by: Ricardo Marques <rimarques@suse.com>

doc/mgr: Add limitations section to plugin guide

Signed-off-by: Tim Serong <tserong@suse.com>

mgr/dashboard: Takeover PR #20799 and adapt comments.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Implement RGW proxy

This implementation is basically a Rados Gateway reverse proxy.  It
additionally takes care of the authentication to the Rados Gateway, but to use
it you will have to be authenticated against the dashboards RESTful API.

The corresponding credentials can be configured using the following commands:

    dashboard set-rgw-api-secret-key <secret-key>

    dashboard set-rgw-api-access-key <access-key>

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: RGW proxy: Include `python-requests-aws`

This commit will include the `python-requests-aws` [1] library.  This library
extends the `requests` library by an authentication mechanism to S3 gateways.

[1] https://github.com/tax/python-requests-aws

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #21022 from sebastian-philipp/dashboard_get_range

mgr/dashboard: Refactor multiple duplicates of `get_rate()`

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #20920 from sebastian-philipp/dashboard_v2-erasure-code-profile

mgr/dashboard: Add ErasureCodeProfile controller

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21086 from ricardoasmarques/fix-table-without-fetch-data

mgr/dashboard: Fix table without fetchData

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge pull request #21041 from votdev/custom_validators

mgr/dashboard: Add custom validators

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21078 from tspmelo/wip-notifications

mgr/dashboard: Add notification service/component

Reviewed-by: Volker Theile vtheile@suse.com
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Loading indicator only if fechData provided

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Hide refresh button if no fetchData provided

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add custom validators.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #19220 into master

* refs/pull/19220/head:
mds: limit run time of load balancer
mds: add list to track recently used sub-directories
mds: calculate other mds' last_epoch_under locally
mds: cleanup mds_load map access/update
mds: check export pin when choosing dirfrags for exporting
mds: optimize MDBalancer::find_exports()
mds: avoid creating unnecessary subtrees during load balance
mds: mds: optimize MDBalancer::try_rebalance()
mds: don't try exporting subdir if dirfrag is already being exported
mds: don't try exporting dirfrags under mds's own mdsdir
mds: cleanup MDBalancer::try_rebalance()
mds: always pass current time to MDBalancer::{hit_inode,hit_dir}
mds: remove unused MDBalancer::export_empties()
mds: don't decay load counters when printing load
mds: adjust pop_nested when importing subtree
mds: adjust subtree popularity when import fails
mds: adjust subtree popularity after rename
mds: fix request rate calculation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21310 from tchaikov/wip-23610

osd: resend osd_pgtemp if it's not acked

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20947 from dzafman/wip-23364

Special scrub handling of hinfo_key errors

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common: hint the main branch of dout() accordingly to default verbosity.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: Changed rhel7.4 to rhel7.5

Note: This is a temp fix till we get $ implementaion

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: Add release notes for the output format changes

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Be smarter about when jsonschema can be used

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Show snapset in list-inconsistent-snapset

Add SnapSet bufferlist to inconsistent_snapset_t

Partial fix for http://tracker.ceph.com/issues/23428

Signed-off-by: David Zafman <dzafman@redhat.com>

rbd-mirror: image map shut down should reset task context

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: wire image map between pool watcher and instance watcher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: wire instance notifications through to pool replayer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: integrate image map init/shut down into pool replayer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial policy instance update should detect dead instances

At initial promotion to the leader position, it can be nearly always
assumed that at least one instance has died and needs to have its images
shuffled to another instance (i.e. the old leader).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simplify the interface between image mapper and policy

The context callbacks between the image map and its policy required
too many hooks between the two classes. Additionally, the original
implementation didn't support re-sending of acquire messages after
initializing (to close a potential race condition).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image map listener should provide a callback context

The context wraps the necessary data and provides a pass-through chain
to the instance notification helpers.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved policy inner-types to common types header

Signed-off-by: Jason Dillaman <dillaman@redhat.com>