rgw: add cors header rule check in cors option request

fix http://tracker.ceph.com/issues/22002

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #18460 from liewegas/wip-straw2-command

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18428 from liewegas/wip-21878

os/bluestore/BlueFS: fix race with log flush during async log compaction

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #18470 from yuyuyu101/wip-tracker-lock

osd/OpRequest: reduce overhead when disabling tracking

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18472 from yuyuyu101/wip-kill-asynccompressor

compressor: kill AsyncCompressor which is broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #18255 into master

* refs/pull/18255/head:
os/bluestore: handle compressed extents in blob unsharing checks

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17722 from amitkumar50/cov-rgw-6

rgw: Initialization of epoch,len

Merge pull request #17305 from theanalyst/rgw-lc-tagging

RGW: support for tagging in lifecycle policies

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18291 from ZVampirEM77/wip-copy-errormsg

rgw: copy object add response error messages

Reviewed-by: Amit Kumar amitkuma@redhat.com
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18194 from ivancich/wip-mclock-repops-bitmap

osd: replace mclock subop opclass w/ rep_op opclass; combine duplicated code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18487 from liewegas/wip-20557

/etc/sysconfig/ceph: remove jemalloc option

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #18448 from ceph/wip-bz1499840

ceph-volume lvm bluestore support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18403 from dillaman/wip-librbd-flush

 librbd: sync flush should re-use existing async flush logic

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rbd: bench should flush prior to starting a test

For the new read-based bench tests, flushing prior to the start of the test
will result in the exclusive lock being acquired and the object map being
utilized.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

/etc/sysconfig/ceph: remove jemalloc option

This breaks when used with rocksdb, which is now the default.

See http://tracker.ceph.com/issues/20557

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18486 from liewegas/wip-20557

etc/default/ceph: remove jemalloc option

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

etc/default/ceph: remove jemalloc option

This breaks when used with rocksdb, which is now the default.

See http://tracker.ceph.com/issues/20557

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18408 from liewegas/wip-buffer-mempool-abi

buffer: avoid changing bufferlist ABI by removing new _mempool field

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18356 from wido/docs-zabbix-mgr

doc: Updated documentation for Zabbix Mgr module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18474 from tchaikov/wip-debian-fix-upgrade

debian: fix package relationships after 40caf6a6

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.activate only prime-osd-dir when directory is empty

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: replace mclock subop opclass w/ rep_op opclass; combine duplicated code

The OSD subops were replaced with replication ops (i.e., MSG_OSD_REPOP
and MSG_OSD_REPOPREPLY) in jewel. mClock incorrectly used them as an
op class rather than the newer replication ops. This fixes that and
places eight ops into the repop class -- MSG_OSD_REPOP,
MSG_OSD_REPOPREPLY, MSG_OSD_PG_UPDATE_LOG_MISSING,
MSG_OSD_PG_UPDATE_LOG_MISSING_REPLY, MSG_OSD_EC_WRITE,
MSG_OSD_EC_WRITE_REPLY, MSG_OSD_EC_READ, and MSG_OSD_EC_READ_REPLY.

This alo removes dupliated code related to op classes in
mClockOpClassQueue and mClockClientQueue and combines into
mClockOpClassSupport.

The three configuration options
osd_op_queue_mclock_osd_subop_{res,wgt,lim} have also been replaced
with osd_op_queue_mclock_osd_rep_op_{res,wgt,lim}.

This also removes OpQueueItem::OpQueueable::op_type_t::osd_subop since
it's not a category core ceph recognizes and is instead it's part of
an mclock view of ops.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-volume lvm.activate safeguard against auto detect objectstore flag not being set

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18423 from ashishkumsingh/wip-doc-fix-iscsi-chap-config

doc: Added CHAP configuration instructions for iSCSI

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

This is a common and recommended step that users should take once they
have clients that support straw2 (hammer!).  Let's make it easy.

This may result in a small amount of data movement, but usually it is
very little.

Signed-off-by: Sage Weil <sage@redhat.com>

debian: fix package relationships after d3ac8d18

d3ac8d18 moves ceph-client-debug from ceph-test to ceph-base without
updating the package relationships between the two involved packages.
which results in:

dpkg: error processing archive /var/cache/apt/archives/ceph-test_12.2.1-241-g43e027b-1trusty_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/ceph-client-debug', which is also in package ceph-base 10.2.10-14-gcbaddae-1trusty
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: fix package relationships after 40caf6a6

we have issues when running upgrade tests:

dpkg: error processing archive /var/cache/apt/archives/ceph-osd_13.0.0-2201-g6cc0b41-1trusty_amd64.deb (--unpack):
trying to overwrite '/usr/bin/ceph-osdomap-tool', which is also in package ceph-test 10.2.10-14-gcbaddae-1trusty

in 40caf6a6, we moves some tools from ceph-test out into ceph-osd,
ceph-mon and ceph-base respectively. but didn't update the relationships
between these packages accordingly. this causes the upgrade failure.

see https://www.debian.org/doc/debian-policy/#document-ch-relationships
for more details on "Breaks" and "Conflicts".

the reason why the package version to be replaced/conflicted is 12.2.2
is that: i assume that this change will be backported to luminous, and
the next release of it will be 12.2.2 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18478 from trociny/wip-valgrid-trim-unittest

test/librbd: memory leak in recently added test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15811 from x-ion-de/wip-20309

doc: Update manual deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librbd: memory leak in recently added test

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

compressor: kill AsyncCompressor which is broken

Signed-off-by: Haomai Wang <haomai@xsky.com>

osd/OpRequest: don't mark_event when disable tracker

Signed-off-by: Haomai Wang <haomai@xsky.com>

common/TrackedOp: make tracking_enabled atomic without lock protect

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge PR #18372 into master

* refs/pull/18372/head:
ptl-tool: Fix more readme examples

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: sync flush should re-use existing async flush logic

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18368 from kungf/scrub_week_day_bound

osd: add scrub week day constraint

Reviewed-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: add bucket_set_alg()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18400 from dillaman/wip-21844

osdc/Objecter: skip sparse-read result decode if bufferlist is empty

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18418 from dillaman/wip-21860

common/common_init: disable ms subsystem log gathering for clients

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18427 from dillaman/wip-21845

osdc/Objecter: delay initialization of hobject_t in _send_op

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18450 from liewegas/wip-osd-removing

osd: add numpg_removing metric

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18452 from liewegas/wip-21880

ceph_test_objectstore: disable filestore_fiemap for tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18454 from liewegas/wip-21882

messages/MOSDMap: do compat reencode of crush map, too

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18457 from liewegas/wip-restful-health

qa/suites/rados/rest/mgr-restful: whitelist more health

qa/suites/rados/rest/mgr-restful: whitelist more health

The test is fiddling with OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDMap: do compat reencode of crush map, too

If we are reencoding an incremental, and it embeds a crush map, we need
to reencode that in a compatible way too.  This is especially true now
because we have the compat crush weight-sets.  Otherwise, a client may
learn the crush map through an incremental but not understand choose_args,
and not see/understand the alternate weight set.  It will send requests
to the wrong OSDs where they will just get dropped.

Fixes: http://tracker.ceph.com/issues/21882
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18406 from adamemerson/wip-insecticide

test,osd,kvstore_tool: silence warnings and prepare test buffer in the right way

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18379 from liewegas/wip-kill-replay-interval

osd/osd_types: pg_pool_t: remove crash_replay_interval member

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18407 from adamemerson/wip-upgrade-zstd

zstd: Upgrade to v1.3.2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18357 from asomers/have_libaio

Build: create a proper WITH_BLUESTORE option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18354 from kungf/no_scurb_deadline

osd: make scrub no deadline when max interval is zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18396 from xiexingguo/wip-scrub-mismatch

osd/osd_types.cc: copy extents map too while making clone

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: disable filestore_fiemap for tests

This option was enabled in 87f33376d977962ab7438c46873ea9b6292390d1 but
causes ObjectStore/StoreTest.Synthetic/1 (filestore) to fail.  Revert that
bit for now until we fix fiemap properly.

See http://tracker.ceph.com/issues/21880

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16958 from mikulely/fix-full-uri

rgw: fix opslog uri as per Amazon s3

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17546 from mikulely/rgw-cleanup-min

rgw: cleanup MIN macro with std::min

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17934 from joke-lee/rgw_list_objecst_wrong_result_with_marker_when_enable_versioning

rgw:fix list objects with marker wrong result  when bucket is enable versioning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17673 from mogeb/blkin

blkin: link against lttng-ust-fork

ceph-volume util.system use strings for comparison in py3

Signed-off-by: Alfredo Deza <adeza@redhat.com>

f ceph-volume tests update changes from the help menu in lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18414 from majianpeng/pmem-work

libmpem: Revert "submodule: make libmpem as a submodule."

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: add numpg_removing metric

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Added CHAP cofiguration instructions for iSCSI

Added details to specify the CHAP username and password while
discovering/login the iSCSI target.

Signed-off-by: Ashish Singh <assingh@redhat.com>

ceph-volume lvm.common --journal-size doesn't accept a volume group

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare filestore does not require a volume group

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common volume groups are no longer required

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate workflow mention tmpfs in osd dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update activation to include bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update create to indicate bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update prepare with bluestore workflow

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update filestore to indicate it is not the only/default objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume elaborate on env vars for interval and tries in activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common update cli flags help for bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk st_mode is needed for stat()

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume decorators always log to the file with exceptions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify behavior of get_lv_from_argument

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm return result of get_lv when using arguments

If the argument is an absolute path it is fine to just return whatever
get_lv finds since it is a "safe" call, it will return a None if nothing
is found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing support bluestore, not only filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm allow full paths to detect lv from arguments

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add support for bluestore list verification

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests activate with fsid works for bluestore and filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests arg validator can accept absolute paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update changes from the help menu in lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update to get_lv from get_journal_lv

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util new path/device mount interface

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update api.lvm tests

The create_lv signature changed to require full size description and
tags need to be an actual dictionary (vs. keyword args)

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18335 from linuxbox2/wip-rgw-readdir2

rgw_file: implement variant offset readdir processing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/common_init: disable ms subsystem log gathering for clients

The log gathering causes large performance degradation to clients
with high message throughputs. This is hopefully a short-term
workaround until per-message logging can be replaced with an
efficient data recording system for post-incident analysis
use-cases.

Fixes: http://tracker.ceph.com/issues/21860
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: fix race with log flush during async log compaction

During async log compaction we rely on _flush-and_sync_log to update the
log_writer to jump_to.  However, if racing threads are also trying to flush
the log and manage to flush our new log events for us, then our flush will
turn into a no-op, and we won't update jump_to correctly at all.  This
results in a corrupted log size a bit later one.

Fix by ensuring that there are no in-progress flushes before we add our
log entries.  Also, add asserts to _flush_and_sync_log to make sure we
never bail out early if jump_to is set (which would indicate this or
another similar bug is still present).

Fixes: http://tracker.ceph.com/issues/21878
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18392 from tchaikov/wip-21842

os/bluestore: implement BlueRocksEnv::AreFilesSame()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osdc/Objecter: delay initialization of hobject_t in _send_op

Fixes: http://tracker.ceph.com/issues/21845
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: copy object add response error message

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

buffer: remove list _mempool member

This broke the C++ ABI by changing the list structure size.  Also, it's
not necessary as we can infer the mempool by looking at the other list
contents.  We don't (currently) have a need to map an empty list to a
particular mempool and have that state stick.

Fixes: http://tracker.ceph.com/issues/21573
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18251 from jan--f/mgr-py-module-constants

pybind/mgr_module: move PRIO_* and PERFCOUNTER_* to MgrModule class

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18021 from jcsp/wip-prometheus-disk-occupation

mgr/prometheus: add ceph_disk_occupation series

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Revert "submodule: make libmpem as a submodule."

This reverts commit 6a14159ee4b910bf83b38316f98ef07edf5f9a01.
Using ExternalProject_Add::GIT to download nvml source .

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18250 from dillaman/wip-21763

doc/rbd: tweaks for the LIO iSCSI gateway

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

os/bluestore: implement BlueRocksEnv::AreFilesSame()

it is used by the "repair" feature to dedup the files to be searched for
MANIFEST-* files. the default implementation is the POSIX one, which
tries to look at the local fs, but we should be looking for the files in
the bluefs. in this very use case, wal and db do not share the same device,
so we can just compare the paths. actually, it should aways return
"false". as the files being compared are always "db" and "db.wal".

Fixes: http://tracker.ceph.com/issues/21842
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18387 from tchaikov/wip-dmclock-client-info-ptr

osd,dmclock: use pointer to ClientInfo instead of a copy of it

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #18350 from ovh/bp-ceph-conf-dump-all

tools/ceph-conf: dump parsed config in plain text or as json

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18367 from zealoussnow/wip-10180

ceph-bluestore-tool: create out_dir before create full path  of kvdb

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>