test: Use jq in a compatible way and for easier diff analysis

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20853 from theanalyst/rgw/recalc-stats

rgw: add an option to recalculate user stats

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21421 from liewegas/wip-doc-balancer

doc/mgr/balancer: document

Reviewed-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #21425 from cbodley/wip-rgw-fix-set_threaded

rgw: fix sync status conflict with cloud sync

rgw: fix conflict with cloud sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20027 from Leeshine/wip-rgw-sync-bucket-shards

rgw: improve sync status: display behind bucket shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21123 from theanalyst/wip-civetweb-1.1

rgw: civetweb fixes for v1.1 upgrade

doc/mgr/balancer: document

Not sure how we missed this for luminous!

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: update civetweb submodule sha1 to track ceph-master

Updating the sha1 to track ceph-master which currently updates civetweb to 1.1

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: set OPENSSL_API_1_1 in civetweb conditionally

While not ideal, we set the civetweb openssl 1.1 conditional compile based on
the openssl version that cmake reports. In future we should make civetweb itself
do this based on OPENSSL_VERSION_COMPAT

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa: drop rfc2616 checking on civetweb

Since newer versions of civetweb are also strict on rfc2616 checks let's enforce
strict rfc2616 checks in s3tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: update civetweb to 1.1

Introduces the following additions in rgw:
- allow_unicode_in_urls introduced with a corresponding downstream commit in
civetweb, as the newer version of civetweb validates that urls are url encoded
which swifttests do not follow, so introduce this as a configurable which we set
as true
- mg header struct changes in civetweb update, use auto here
- drop info->uri and use local_uri instead as the former is deprecated

wip: rgw: civetweb fixes for v1.1 upgrade

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #21416 from tchaikov/wip-tox-pwd

test/dashboard: specify workdir using tox.ini

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #18932 from yehudasa/wip-rgw-cloud-sync

rgw cloud sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21356 from smithfarm/wip-23657

rpm,debian: Ensure all ceph-disk runtime dependencies are declared for ceph-base

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

test/dashboard: specify workdir using tox.ini

--workdir was introduced in tox v2.4.0, but the tox shipped by
ubuntu/xenial is 2.3.1. also because i removed the step to prepare the
tox using "pip install -r requirements.txt", what we have is the tox
v2.3.1 . so, instead of passing workdir in the command line, we specify
this setting in tox.ini.

also drop the tox in requirements.txt, it's just misleading. and we are
not using the tox installed in the venv.

Fixes: http://tracker.ceph.com/issues/23709
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21342 from SUSE/wip-argparse-fix-validate_command-error

pybind/ceph_argparse: Fix UnboundLocalError if command doesn't validate

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21331 from liewegas/wip-slow-ops

qa/suites/rados/monthrash: whitelist SLOW_OPS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21317 from liewegas/wip-23622

common/Formatter: fix string_view usage for {json,xml}_stream_escaper

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21312 from wjwithagen/wjw-PG-public

osd: make PG::*Force* event structs public

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21293 from tchaikov/wip-no-venv-for-dashboard

test: only test enabled python bindings

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21287 from yanghonggang/master

os/kstore: fix statfs problem and add vstart.sh support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21174 from guzhongyan/crush-log-cleanup

crush: cleanup update_device_class() log messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21110 from majianpeng/fix-objecter

osdc: dec num_in_flight for pool_dne case.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #21029 from rzarzynski/wip-osd-close_op_ctx-nohobjcpy

osd: don't memcpy hobject_t in PrimaryLogPG::close_op_ctx().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21018 from ashitakasam/wip-bug-23440

osd: remove duplicated "commit_queued_for_journal_write" in OpTracker

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20670 from chardan/jfw-wip-rand_migration-core

Extends random.h: numeric types relaxed to compatible types (with

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #21350 from tspmelo/notifications-improvement

mgr/dashboard: Notification improvements

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile vtheile@suse.com

Merge pull request #20996 from rzarzynski/wip-bs-bug23246

os/bluestore: fix exceeding the max IO queue depth in KernelDevice.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #21345 from ceph/fix_e2fsprogs

specs: require of e2fsprogs

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #20658 from majianpeng/cmake-link-error

os/CMakeLists: fix link errro when enable WITH_PMEM=ON.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21332 from cbodley/wip-cmake-lz4

cmake: enable LZ4 by default

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw/tests_http_manager: fix initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

json_formattable: fix out of bounds array entity removal

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #21181 into master

* refs/pull/21181/head:
fuse: wire up fuse_ll_access

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21276 into master

* refs/pull/21276/head:
MDSMonitor: fix mds health printed in bad format

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #21323 into master

* refs/pull/21323/head:
mds: properly dirty sessions opened by journal replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: force last writer wins on marker writes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/radosgw: cloud sync docs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #21362 from dzafman/wip-hex-digest

osd: Change shard digests to hex like object info digests

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: rename aws tier type to 'cloud'

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: etag fixes

Use string instead of bufferlist to avoid potential issues.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following code review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw/test_multi: fix a few tests to only iterate over rw zones

Some of the tests require at least two read-write (regular rgw) zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: zone_cloud: deal with key representation and other fixes

Needed to present a key to the tests that reflected its original name
and version_id (and etag), so that the callers don't need to be modified.
However, this can only be achieved if we get the key, which doesn't work
if the caller was just listing the bucket objects. Created a new CloudKey
class to deal with the different issues there.
Also, other test related fixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: streaming put also stores content_type and other fields

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix parse_tier_config_param function

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: don't call http data callbacks under lock

There is no need to hold req_data->lock when calling into client
callbacks. This removes an unneeded lock dependency (that is a
problem when cancelliing coroutines stack).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: coroutines: cancel stacks on teardown

If we don't cancel stacks, ops might not be destructed, so ops callbacks
could still be active.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't store etag with extra null character at the end

head objects etag attr doesn't need to store an extra null char.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync: store source object info in destination object

store extra meta params on target object (original name, version_id, etag,
etc.)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync: store versioned epoch in target object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: initial work on cloud sync test

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustment following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework cloud sync configuration

Change the configuration structure to separate between
connections and acl mappings, and other changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use different io id when waking up interval wait

Can't use a zero channel id as a wildcard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation warning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: fixes for mstart

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: apply acl mappings

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: sync acls

still not transforming acls according to configuration

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: acl_mappings config handling

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: remove bucket suffix configurable

No need to have a special suffix, target path can be used instead.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: use configurable target path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: dump config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formatter: add two utility classes

For automatic section scoping

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: configure different targets

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: aws sync: new config structure, support multiple connections

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formattable: more fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync module tier config fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formattable: fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cloud sync, sync object meta

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

formattable: adjust array access interface

appending new entry by providing empty index, setting existing entries
by providing negative index.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: formattable, set, erase, unitest

Extend the formattable api to provide a mechanism to set and erase
entities by a string key that references them. E.g., "foo.bar[123].xyz"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle the situation that get a 409 response from S3 correctly
pass the http body and use it when creating a bucket

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: avoid use Chunked transfer encoding in RGW's request

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add a bucket-suffix tier-config

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: virtual hosted-style support

add host-style field in tier-config to specify the related
zone's hosted-style used in request from RGW, if this config
is not specified, path hosted-style will be used as default.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: RGWRESTStreamRWRequest::send_prepare(), url encode dest

encode the resource in send_prepare(), as a result, all the callers
to do_send_prepare() have do url_encode before calling it.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add encode_slash param in url_encode

when sent request to S3, we should not encode the forward
slash character('/') in the object key name, so we need add
a encode_slash param in url_encode to decide whether to encode
the slash or not.

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: use GMT time format defined by RFC1123 in http request header

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: sync modules: adapt to formattable based config

still need to fix rgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/ceph_json_formattable: add unitest

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: formattable fix, api changes

add implicit casts

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: refine Formattable api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync modules, use JSONFormattable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_str_to_bool() should also accept "true"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/formattable: initial work

initial work on a dict-like api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #21026 from ivancich/wip-rgw-unordered-listing

rgw: ability to list bucket contents in unsorted order for efficiency

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #21291 from smithfarm/wip-23596

RGWCivetWeb::read_data: fix arguments to mg_read() call

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21285 from joke-lee/wip-max_keys_convert

rgw: return EINVAL if max_keys can not convert correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21129 from oritwas/wip-rgw-admin-no-cache

rgw: radosgw-admin should not use metadata cache for readonly commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20834 from linuxbox2/wip-linked

rgw_file: conditionally unlink handles when direct deleted

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20926 from yehudasa/wip-22963-truncation

rgw: fix for usage truncated flag

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20814 from niupengju/master

rgw: Fix multisite Synchronization failed when read and write delete …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: Change shard digests to hex like object info digests

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #21140 from ceph/cv

ceph-volume: remove extra space

install-deps: add liblz4 to freebsd build dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

debian: add build dependency for lz4

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rpm: add build dependency for lz4

Signed-off-by: Casey Bodley <cbodley@redhat.com>