rgw: Fix swift object expiry not deleting objects

In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check.
This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard,
resulting in a lost removal hint and a leaked object.

Fixes: http://tracker.ceph.com/issues/22084
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #18770 from tchaikov/wip-doc-log-file

doc: fix desc of option "mon cluster log file"

Reviewed-by: John Spray <john.spray@redhat.com>

doc: fix desc of option "mon cluster log file"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #18807 into master

* refs/pull/18807/head:
ptl-tool: put common use-cases up front in README
ptl-tool: only create tag if new branch
ptl-tool: hint how to skip contributor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: put common use-cases up front in README

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: only create tag if new branch

It doesn't make sense to create tags for branches that already exist.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: hint how to skip contributor

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18709 from ivancich/wip-bktidx-race-bugfix

rgw: fix BZ 1500904, stale bucket index entry remains after obj delete

Merge pull request #18476 from majianpeng/objecter-misc-cleanup

osdc/Objecter: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18768 from wumingqiao/wip-WaitActingChange

PG: fix name of WaitActingChange

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18789 from jcsp/wip-doc-config-set

doc: replace injectargs usage with "config set"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18769 from yaozongyou/wip-dashboard-format-label

mgr/dashboard: format tooltip's label as user friendly string

Reviewed-by: John Spray <john.spray@redhat.com>

doc: replace injectargs usage with "config set"

Cleaner and easier.  Also implicitly documents
the config set command, which hadn't been explicitly
called out in the docs before.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge PR #18689 into master

* refs/pull/18689/head:
cephfs: Do not check auth gid when not specified

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18386 into master

* refs/pull/18386/head:
ptl-tool: allow setting project/repo for GH req

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18658 from linuxbox2/wip-s3-post-20201

RGW: update S3 POST policy handling of Content-Type

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/dashboard: format chart's tooltip's label as friendly string

The default tooltip's label is a long number string, so format
the label as a user friendly string.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #16238 from trociny/wip-copy-deep

librbd: generalized deep copy function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

PG: fix name of WaitActingChange

the parent state of WaitActingChange is Primary, not Peering.

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

qa/suites/rbd: enable fsx deep_copy test case

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

test/librbd: tweak fsx to optionally use deep-copy instead of clone

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: add deep_copy function to API

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

rbd-mirror: use generalized deep copy for image sync

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: generalized deep copy function

(based on rbd-mirror image sync)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: export ImageOptions print operator

(so it could be used for logging image options)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

test/librbd: set rand seed for all tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18635 from majianpeng/bluestore-omap-flush

os/bluestore: remove unused parameters.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove unused paramerter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18541 from trociny/wip-mstart-bashizm

mstart.sh: remove bashizm in /bin/sh script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18729 from iliul/fix-vstart-typo

vstart.sh: fix a typo

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18693 from XinzeChi/wip-dup-async-read-stat

osd/PrimaryLogPG: fix dup stat for async read

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-By: Kefu Chai <kchai@redhat.com>

vstart.sh: fix a spelling mistake

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #18724 from iliul/add-missing-blank-line

doc: add missing blank line

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18708 from dillaman/wip-22036

librbd: added preprocessor macro for detecting compare-and-write support

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18704 from dillaman/wip-rbd-iscsi-win

doc/rbd: tweaks to the Windows iSCSI initiator directions

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18712 from liu-chunmei/rm_redundant_dpdksource

dpdk: remove redundant dpdk submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18711 from adamemerson/wip-system-includes

cmake: System Includes to silence warnings from submodules and libraries!

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18701 from tchaikov/wip-denc

denc: support enum with underlying type

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18597 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc: Add missing blank line

Signed-off-by: iliul <liul124@chinaunicom.cn>

build: Merge wip-system-includes from upstream

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Squashed 'src/dmclock/' changes from 1bbd42b8fe..d1ff03b667

d1ff03b667 Merge pull request #44 from adamemerson/wip-system-includes
4cc4b949ca build: Mark dependency includes as SYSTEM in dmclock
05096c1756 Merge pull request #43 from TaewoongKim/anticipation_conf
f356c45461 Add missing anticipation_timeout argument for PullPriorityQueue constructor
9896448ec5 Merge pull request #42 from tchaikov/wip-cmake
979899ef86 add travis CI on gnu/linux
8a3dabdbee cmake: the built archives are located in ${binary_dir}
ee15ced3e9 cmake: check for include in /usr/include also

git-subtree-dir: src/dmclock
git-subtree-split: d1ff03b667d9551478b2803ea533fc356ca441a9

build: Mark dependency includes as SYSTEM

It is not really our business to debug python, boost, or our other
dependencies. Mark them as system includes.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18672 from jcsp/wip-21999

mgr: emit cluster log message on serve() exception

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18691 from liewegas/wip-fix-balancer

mgr/balancer: fix upmap; default balancer module enabled

Merge PR #18538 into master

* refs/pull/18538/head:
mds: clean up clog damage message when repairing
mds: flush after scrub repairs
mds: don't report repaired backtraces in damagetable
mds: cleaner scrub complete log msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18555 into master

* refs/pull/18555/head:
mds: fix inode count check in MDCache::check_memory_usage()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18612 into master

* refs/pull/18612/head:
mds: fix race in PurgeQueue::wait_for_recovery()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18624 into master

* refs/pull/18624/head:
mds: trim 'N' log segments according to how many log segments are there

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18646 into master

* refs/pull/18646/head:
mds: fix MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition
mds: remove useless incompat feature definination in FSMap.h

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18654 into master

* refs/pull/18654/head:
ceph.in: pass RADOS inst to LibCephFS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Tested-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18686 from xiexingguo/wip-bf-retry-interval

osd/PG: use osd_backfill_retry_interval for schedule_backfill_retry()

Reviewed-by: Sage Weil <sage@redhat.com>

remove redundant dpdk source folder, this one is not maintained since last Nov.2016,
the other one is located in src/spdk/dpdk, which is alwasy maintained and used by spdk.
so remove this src/dpdk folder, we only need maintain one copy.

Signed-off-by: chunmei <chunmei.liu@intel.com>

rgw: fix BZ 1500904, Stale bucket index entry remains after object deletion

We have a race condition:

 1. RGW client #1: requests an object be deleted.
 2. RGW client #1: sends a prepare op to bucket index OSD #1.
 3. OSD #1:        prepares the op, adding pending ops to the bucket dir entry
 4. RGW client #2: sends a list bucket to OSD #1
 5. RGW client #2: sees that there are pending operations on bucket
                   dir entry, and calls check_disk_state
 6. RGW client #2: check_disk_state sees that the object still exists, so it
                   sends CEPH_RGW_UPDATE to bucket index OSD (#1)
 7. RGW client #1: sends a delete object to object OSD (#2)
 8. OSD #2:        deletes the object
 9. RGW client #2: sends a complete op to bucket index OSD (#1)
10. OSD #1:        completes the op
11. OSD #1:        receives the CEPH_RGW_UPDATE and updates the bucket index
                   entry, thereby **RECREATING** it

Solution implemented:

At step #5 the object's dir entry exists. If we get to beginning of
step #11 and the object's dir entry no longer exists, we know that the
dir entry was just actively being modified, and ignore the
CEPH_RGW_UPDATE operation, thereby NOT recreating it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

cephfs: Do not check auth gid when not specified

For auth caps that omit the gid, do not check for a gid match.

Fixes: http://tracker.ceph.com/issues/22009
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

librbd: added preprocessor macro for detecting compare-and-write support

Fixes: http://tracker.ceph.com/issues/22036
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17923 from cbodley/wip-rgw-beast-update

rgw: pull up beast submodule and update frontend

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc/rbd: tweaks to the Windows iSCSI initiator directions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18522 from C2python/cleanup_to_string

rgw: Delete to_string functions. stringify defined in include/stringify.h can provide the same feature.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18536 from ZVampirEM77/wip-fix-loadgen

rgw: loadgen fix generate random object name rgw crash issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18526 from tchaikov/wip-17454

mgr: In plugins 'module' classes need not to be called "Module" anymore.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18678 from liu-chunmei/fix_dpdk_coremask

msg/async: fix ms_dpdk_coremask and ms_dpdk_coremask conflict

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18670 from liewegas/wip-min-in-4

qa/tasks/thrashosds: set min_in default to 4

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18671 from ifed01/wip-ifed-bluefs-empty-reclaiming-fix

os/bluestore: do not release empty bluefs_extents_reclaiming.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18653 from dillaman/wip-21391

mgr/dashboard: added iSCSI IOPS/throughput metrics

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

denc: support enum with underlying type

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,deb: remove the last trace of atomic_ops

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unused definitions

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ${XIO_LIBRARY} if not necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

ms_dpdk_coremask and ms_async_op_threads has some relationship,
dpdk library will create threasds as many as bit 1 number in
ms_dpdk_coremask and the thredas num should be equal ms_async_op_threads+1,
by ceph default confiuration,ms_async_op_threads =3,ms_dpdk_coremask =1,
so dpdk library can't create enough threads for dpdk workers. the sytem
will stuck at NetworkStack::start().so set ms_dpdk_coremask default 0xF.
and add assert to comapre the two configurations.

Signed-off-by: chunmei <chunmei.liu@intel.com>

Merge pull request #18695 from cooboos/wip-doc-wrongspell

doc/release-notes: fix typo 'psd' to 'osd'

Reviewed-By: Kefu Chai <kchai@redhat.com>

doc: correct wrong spell 'psd' to 'osd'

Signed-off-by: wangsongbo wangsongbo@cloudin.cn

osd/PrimaryLogPG: fix dup stat for async read

The async read would call do_osd_ops twice.

Signed-off-by: Xinze Chi <xinze@xsky.com>

pybind/mgr/mgr_module: fix calc_pg_upmaps

This was fixed in the object cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: enable module by default

It will still be "off".

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: loadgen fix rgw crash issue

When generating random bucket name and object name during
loadgen processing, RGW will crash. The reason is calling
"gen_rand_alphanumeric" with "cct = NULL".

Fixes: http://tracker.ceph.com/issues/22006
Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #13992 from dreamhost/wip-19264

rgw: fix Swift container naming rules.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18606 from adamemerson/wip-op-perm-refactor

rgw: Small refactor and two bug fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #14955 from aclamk/fix-civetweb-cryptopp

radosgw: Make compilation with CryptoPP possible.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18664 from oritwas/wip-rgw-21996

rgw: init oldest period after setting run_sync_thread
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18680 from tchaikov/wip-doc-mgr

doc/mgr: document facilities methods using `automethod` directive

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18685 from jcsp/wip-mgr-load-module

qa: fix mgr _load_module helper

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: use osd_backfill_retry_interval for schedule_backfill_retry()

These two options currently share the same value and I have no particular
preference, but using osd_backfill_retry_interval for the specific scenario
gives us more flexible choices.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa: fix mgr _load_module helper

I inadvertently broke this with the latest change
to the module ls output.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18656 from ceph/ceph-volume-bluestore-tests

ceph-volume: adds functional testing for bluestore

Merge pull request #18399 from jcsp/wip-20950

mon: don't blow away bootstrap-mgr on upgrades

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18681 from tchaikov/wip-msg-async-popcnt

msg/async: use bitset<> to do the popcnt

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18652 from idryomov/wip-qa-krbd-blkroset

qa/suites/krbd: add krbd BLKROSET test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async: use bitset<> to do the popcnt

libstdc++ uses popcnt to count the bits. it's faster and simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr: document mgr/plugin using automethod

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: sync the docstring with the doc/mgr

also update the docstring to confirm to
http://www.sphinx-doc.org/en/stable/domains.html#info-field-lists.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: In plugins 'module' classes need not to be called "Module" anymore.

Fixes: http://tracker.ceph.com/issues/17454
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: bhavishyagopesh <bhavishyagopesh@gmail.com>

Merge pull request #18619 from trociny/wip-qa-mirror-remove

qa/workunits/rbd: unnecessary sleep after failed remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17983 from liewegas/wip-balancer

mgr/balancer: make crush-compat mode work

Reviewed-by: John Spray <john.spray@redhat.com>

ptl-tool: allow setting project/repo for GH req

This allows using ptl-tool for other projects like ceph-linode.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: emit cluster log message on serve() exception

Fixes: http://tracker.ceph.com/issues/21999
Signed-off-by: John Spray <john.spray@redhat.com>

rgw: add missing current_history initialization

Fixes: http://tracker.ceph.com/issues/21996
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: init oldest period after setting run_sync_thread

Fixes: http://tracker.ceph.com/issues/21996
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18657 from cbodley/wip-21990

rgw: remove placement_rule from rgw_link_bucket()