doc: Improved dashboard documentation

Moved content (RGW access configuration) from the dashboard `README.rst`
into the reference manual. Made various textual improvements to both the
README and the documentation.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21402 from tspmelo/wip-module-cleanup

mgr/dashboard: Angular modules cleanup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge pull request #21398 from ivancich/wip-revert-dmclock-feature-bit-plus

osd: Revert use of dmclock message feature bit since not yet finalized

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21403 from dillaman/wip-rbd-mirror-dead-instances

test/rbd-mirror: improve coverage for dead instance handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #21322 into master

* refs/pull/21322/head:
osdc: cleanup code.
osdc: Remove useless condition.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge PR #21341 into master

* refs/pull/21341/head:
client: Fix the gid_count check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21370 into master

* refs/pull/21370/head:
client: Take reference for operator= function parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21391 into master

* refs/pull/21391/head:
qa: fix log typo

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #21396 into master

* refs/pull/21396/head:
ceph-fuse: exit with failure on failed mount
common: ignore errors during preforker exit
client: do not overload system errnos

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #20398 from wjwithagen/wip-posix_fallocate

common: posix_fallocate on ZFS returns EINVAL

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20752 from jcsp/wip-dashboard-build-tweak

mgr/dashboard: build tweaks

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21108 from xiexingguo/wip-kill-subset-of

interval_set: kill subset_of()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21369 from majianpeng/test-objectstore_bench-usage

test/objectstore_bench: Don't forget judging whether call usage.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20100 from mogeb/iostat-plugin

mgr/iostat: implement 'ceph iostat' as a mgr plugin

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21373 from ovh/bp-hb-divbyzero

osd: fix crash caused by divide by zero in heartbeat code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21395 from liewegas/wip-log-channels

mon/LogMonitor: separate out summary by channel

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19935 from myoungwon/wip-manifest-ref-count

osd: refcount for manifest object (redirect, chunked)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19117 from jan--f/jan-object-counts-decimal

include/types: format decimal numbers with decimal factor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21427 from liewegas/wip-balancer-roots

mgr/balancer: improve error message

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #21401 from liewegas/wip-mgr-help

ceph-mgr: exit after usage

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #21430 from tchaikov/wip-osdmap-prune

mon/OSDMonitor: txsize should be greater or eq to prune_interval - 1

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

mon/OSDMonitor: txsize should be greater or eq to prune_interval - 1

so we can prune 1 or more intervals in a single iteration. and in each
interval, we prune (prune_interval - 1) versions of osdmap.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21404 from tchaikov/wip-cmake-boost

cmake: changes of BuildBoost.cmake to ready seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21410 from badone/wip-test_large_omap_detection-scrub-pgs-instead-of-osds

qa/workunits/rados/test_large_omap_detection: Scrub pgs instead of OSDs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/balancer: identify roots that osds overlap in

This error message isn't very informative; tell the user which osds
appear in mulitple roots, and which roots those are!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20853 from theanalyst/rgw/recalc-stats

rgw: add an option to recalculate user stats

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21421 from liewegas/wip-doc-balancer

doc/mgr/balancer: document

Reviewed-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #21425 from cbodley/wip-rgw-fix-set_threaded

rgw: fix sync status conflict with cloud sync

rgw: fix conflict with cloud sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20027 from Leeshine/wip-rgw-sync-bucket-shards

rgw: improve sync status: display behind bucket shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21123 from theanalyst/wip-civetweb-1.1

rgw: civetweb fixes for v1.1 upgrade

doc/mgr/balancer: document

Not sure how we missed this for luminous!

Signed-off-by: Sage Weil <sage@redhat.com>

cls/refcount: add chunk_refcount_client()

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

src/test: add a unit test for manifest refcount

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: decrease refcount when object is deleted (has manifest) if the object is referenced

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: increase refcount if redirect or chunked object is enabled

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd: add refcount_manifest() for the object that has manifest

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

osd/osd_types.h: add HAS_REFERENCE flag

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

librados: add with_reference flag when redirect or chunk is enabled

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

cls/refcount: add chunk_refcount

refcount() for chunked object is added (based on source id,
pool id and offset)

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

rgw: update civetweb submodule sha1 to track ceph-master

Updating the sha1 to track ceph-master which currently updates civetweb to 1.1

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: set OPENSSL_API_1_1 in civetweb conditionally

While not ideal, we set the civetweb openssl 1.1 conditional compile based on
the openssl version that cmake reports. In future we should make civetweb itself
do this based on OPENSSL_VERSION_COMPAT

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa: drop rfc2616 checking on civetweb

Since newer versions of civetweb are also strict on rfc2616 checks let's enforce
strict rfc2616 checks in s3tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: update civetweb to 1.1

Introduces the following additions in rgw:
- allow_unicode_in_urls introduced with a corresponding downstream commit in
civetweb, as the newer version of civetweb validates that urls are url encoded
which swifttests do not follow, so introduce this as a configurable which we set
as true
- mg header struct changes in civetweb update, use auto here
- drop info->uri and use local_uri instead as the former is deprecated

wip: rgw: civetweb fixes for v1.1 upgrade

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

common/options: change mon_data_size_warn type to TYPE_SIZE

As the option represents a byte count, TYPE_SIZE is appropriate and the
correct IEC unit prefixes will be parsed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/workunits/cephtool/test.sh: fix SI unit test, add IEC unit test

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21416 from tchaikov/wip-tox-pwd

test/dashboard: specify workdir using tox.ini

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #18932 from yehudasa/wip-rgw-cloud-sync

rgw cloud sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: Revert use of dmclock message feature bit since not yet finalized

Because the updated use of dmclock is not yet finalized and the
unfinished functionality uses a valuable messaging feature bit that we
don't want to waste, we're removing it for the mimic release and from
master.

Reverts most of the following commits for mimic:

  de8110ebb39401d3e09cb871f843776421ccd56f
  739d7e462f3a03c4e188fcfa48eb05c0063970c9
  2361e9c790306e789393323ef349d8b4a20bc70d
  9b702cf8fc2ff881ca16edc78874f2118a2e4641

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #21356 from smithfarm/wip-23657

rpm,debian: Ensure all ceph-disk runtime dependencies are declared for ceph-base

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

test/dashboard: specify workdir using tox.ini

--workdir was introduced in tox v2.4.0, but the tox shipped by
ubuntu/xenial is 2.3.1. also because i removed the step to prepare the
tox using "pip install -r requirements.txt", what we have is the tox
v2.3.1 . so, instead of passing workdir in the command line, we specify
this setting in tox.ini.

also drop the tox in requirements.txt, it's just misleading. and we are
not using the tox installed in the venv.

Fixes: http://tracker.ceph.com/issues/23709
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21342 from SUSE/wip-argparse-fix-validate_command-error

pybind/ceph_argparse: Fix UnboundLocalError if command doesn't validate

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21331 from liewegas/wip-slow-ops

qa/suites/rados/monthrash: whitelist SLOW_OPS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21317 from liewegas/wip-23622

common/Formatter: fix string_view usage for {json,xml}_stream_escaper

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21312 from wjwithagen/wjw-PG-public

osd: make PG::*Force* event structs public

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21293 from tchaikov/wip-no-venv-for-dashboard

test: only test enabled python bindings

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21287 from yanghonggang/master

os/kstore: fix statfs problem and add vstart.sh support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21174 from guzhongyan/crush-log-cleanup

crush: cleanup update_device_class() log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21110 from majianpeng/fix-objecter

osdc: dec num_in_flight for pool_dne case.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #21029 from rzarzynski/wip-osd-close_op_ctx-nohobjcpy

osd: don't memcpy hobject_t in PrimaryLogPG::close_op_ctx().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21018 from ashitakasam/wip-bug-23440

osd: remove duplicated "commit_queued_for_journal_write" in OpTracker

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20670 from chardan/jfw-wip-rand_migration-core

Extends random.h: numeric types relaxed to compatible types (with

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #21350 from tspmelo/notifications-improvement

mgr/dashboard: Notification improvements

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

qa/workunits/rados/test_large_omap_detection: Scrub pgs instead of OSDs

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20996 from rzarzynski/wip-bs-bug23246

os/bluestore: fix exceeding the max IO queue depth in KernelDevice.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21345 from ceph/fix_e2fsprogs

specs: require of e2fsprogs

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #20658 from majianpeng/cmake-link-error

os/CMakeLists: fix link errro when enable WITH_PMEM=ON.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21332 from cbodley/wip-cmake-lz4

cmake: enable LZ4 by default

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

common: posix_fallocate on ZFS returns EINVAL

But even still it would not work on any COW FS.
So reorganised the code to have a common routine
that in the end will allocate a file on disk if needed

FileStore would not build when there was no HAVE_POSIX_FALLOCATE
other than on Apple. With ceph_posix_fallocate FileStore will also
fallback to manually allocating the required file.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: expose Boost_VERSION

to appease the consumers of FindBoost.cmake, they are expecting this
variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename the boost header target to Boost::boost

because FindBoost.cmake defines Boost::boost as the target for header
only dependencies.

due to the limit of cmake < 3.1, this only applies to cmake > 3.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/tests_http_manager: fix initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

json_formattable: fix out of bounds array entity removal

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #21181 into master

* refs/pull/21181/head:
fuse: wire up fuse_ll_access

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21276 into master

* refs/pull/21276/head:
MDSMonitor: fix mds health printed in bad format

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #21323 into master

* refs/pull/21323/head:
mds: properly dirty sessions opened by journal replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Angular modules cleanup

Removed the import of modules that were already being imported through the
Shared module.

Removed duplicated providers, those services were already provided in the api
module.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: force last writer wins on marker writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/radosgw: cloud sync docs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #21362 from dzafman/wip-hex-digest

osd: Change shard digests to hex like object info digests

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: rename aws tier type to 'cloud'

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: etag fixes

Use string instead of bufferlist to avoid potential issues.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following code review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw/test_multi: fix a few tests to only iterate over rw zones

Some of the tests require at least two read-write (regular rgw) zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: zone_cloud: deal with key representation and other fixes

Needed to present a key to the tests that reflected its original name
and version_id (and etag), so that the callers don't need to be modified.
However, this can only be achieved if we get the key, which doesn't work
if the caller was just listing the bucket objects. Created a new CloudKey
class to deal with the different issues there.
Also, other test related fixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: streaming put also stores content_type and other fields

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix parse_tier_config_param function

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: don't call http data callbacks under lock

There is no need to hold req_data->lock when calling into client
callbacks. This removes an unneeded lock dependency (that is a
problem when cancelliing coroutines stack).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutines: cancel stacks on teardown

If we don't cancel stacks, ops might not be destructed, so ops callbacks
could still be active.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't store etag with extra null character at the end

head objects etag attr doesn't need to store an extra null char.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync: store source object info in destination object

store extra meta params on target object (original name, version_id, etag,
etc.)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync: store versioned epoch in target object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on cloud sync test

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework cloud sync configuration

Change the configuration structure to separate between
connections and acl mappings, and other changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use different io id when waking up interval wait

Can't use a zero channel id as a wildcard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation warning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: fixes for mstart

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: apply acl mappings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>