mds: don't create snap inode during trans-authority rename

MDCache::cow_inode() may add auth pin to frozen inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: trim non-auth snap dentry during cache rejoin

Because corresponding dentry in auth mds may have been pruned (snap deleted)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: detach unlinked inode's snaprealm from its parent

When replaying slave update, inode may gets unlinked, but preserved
in memory in case of rollback. In this case, we need to detach the
inode's snaprealm from its parent.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix CInode::find_snaprealm()

If inode has no parent dentry (newly created), CInode::find_snaprealm()
should try the oldest projected parent.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix snap dentry discover

bloom filter does not track snap dentries. pass proper snapid to
CDir::add_null_dentry

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: introduce MDSMap::get_mds_set_lower_bound()

return mds set whose states are larger than or equal to a given state

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly distinguish snaptable TABLE_OP_CREATE from TABLE_OP_UPDATE

SnapInfo::encode() does not encode long_name, operation name stored
in long_name gets lost after mds restarts

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly set reply buffer for table request 'agree' message

Origin code does not set reply buffer for 'agree' messages that
are re-sent.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: make change to mds table after corresponding event get logged

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #18729 from iliul/fix-vstart-typo

vstart.sh: fix a typo

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18693 from XinzeChi/wip-dup-async-read-stat

osd/PrimaryLogPG: fix dup stat for async read

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-By: Kefu Chai <kchai@redhat.com>

vstart.sh: fix a spelling mistake

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #18724 from iliul/add-missing-blank-line

doc: add missing blank line

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18708 from dillaman/wip-22036

librbd: added preprocessor macro for detecting compare-and-write support

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18704 from dillaman/wip-rbd-iscsi-win

doc/rbd: tweaks to the Windows iSCSI initiator directions

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18712 from liu-chunmei/rm_redundant_dpdksource

dpdk: remove redundant dpdk submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18711 from adamemerson/wip-system-includes

cmake: System Includes to silence warnings from submodules and libraries!

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18701 from tchaikov/wip-denc

denc: support enum with underlying type

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18597 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc: Add missing blank line

Signed-off-by: iliul <liul124@chinaunicom.cn>

build: Merge wip-system-includes from upstream

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Squashed 'src/dmclock/' changes from 1bbd42b8fe..d1ff03b667

d1ff03b667 Merge pull request #44 from adamemerson/wip-system-includes
4cc4b949ca build: Mark dependency includes as SYSTEM in dmclock
05096c1756 Merge pull request #43 from TaewoongKim/anticipation_conf
f356c45461 Add missing anticipation_timeout argument for PullPriorityQueue constructor
9896448ec5 Merge pull request #42 from tchaikov/wip-cmake
979899ef86 add travis CI on gnu/linux
8a3dabdbee cmake: the built archives are located in ${binary_dir}
ee15ced3e9 cmake: check for include in /usr/include also

git-subtree-dir: src/dmclock
git-subtree-split: d1ff03b667d9551478b2803ea533fc356ca441a9

build: Mark dependency includes as SYSTEM

It is not really our business to debug python, boost, or our other
dependencies. Mark them as system includes.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18672 from jcsp/wip-21999

mgr: emit cluster log message on serve() exception

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18691 from liewegas/wip-fix-balancer

mgr/balancer: fix upmap; default balancer module enabled

Merge PR #18538 into master

* refs/pull/18538/head:
mds: clean up clog damage message when repairing
mds: flush after scrub repairs
mds: don't report repaired backtraces in damagetable
mds: cleaner scrub complete log msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18555 into master

* refs/pull/18555/head:
mds: fix inode count check in MDCache::check_memory_usage()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18612 into master

* refs/pull/18612/head:
mds: fix race in PurgeQueue::wait_for_recovery()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18624 into master

* refs/pull/18624/head:
mds: trim 'N' log segments according to how many log segments are there

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18646 into master

* refs/pull/18646/head:
mds: fix MDS_FEATURE_INCOMPAT_FILE_LAYOUT_V2 definition
mds: remove useless incompat feature definination in FSMap.h

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18654 into master

* refs/pull/18654/head:
ceph.in: pass RADOS inst to LibCephFS

Reviewed-by: Kefu Chai <kchai@redhat.com>
Tested-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18686 from xiexingguo/wip-bf-retry-interval

osd/PG: use osd_backfill_retry_interval for schedule_backfill_retry()

Reviewed-by: Sage Weil <sage@redhat.com>

remove redundant dpdk source folder, this one is not maintained since last Nov.2016,
the other one is located in src/spdk/dpdk, which is alwasy maintained and used by spdk.
so remove this src/dpdk folder, we only need maintain one copy.

Signed-off-by: chunmei <chunmei.liu@intel.com>

librbd: added preprocessor macro for detecting compare-and-write support

Fixes: http://tracker.ceph.com/issues/22036
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17923 from cbodley/wip-rgw-beast-update

rgw: pull up beast submodule and update frontend

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc/rbd: tweaks to the Windows iSCSI initiator directions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18522 from C2python/cleanup_to_string

rgw: Delete to_string functions. stringify defined in include/stringify.h can provide the same feature.

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18536 from ZVampirEM77/wip-fix-loadgen

rgw: loadgen fix generate random object name rgw crash issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18526 from tchaikov/wip-17454

mgr: In plugins 'module' classes need not to be called "Module" anymore.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18678 from liu-chunmei/fix_dpdk_coremask

msg/async: fix ms_dpdk_coremask and ms_dpdk_coremask conflict

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18670 from liewegas/wip-min-in-4

qa/tasks/thrashosds: set min_in default to 4

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18671 from ifed01/wip-ifed-bluefs-empty-reclaiming-fix

os/bluestore: do not release empty bluefs_extents_reclaiming.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18653 from dillaman/wip-21391

mgr/dashboard: added iSCSI IOPS/throughput metrics

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

denc: support enum with underlying type

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake,deb: remove the last trace of atomic_ops

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unused definitions

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ${XIO_LIBRARY} if not necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

ms_dpdk_coremask and ms_async_op_threads has some relationship,
dpdk library will create threasds as many as bit 1 number in
ms_dpdk_coremask and the thredas num should be equal ms_async_op_threads+1,
by ceph default confiuration,ms_async_op_threads =3,ms_dpdk_coremask =1,
so dpdk library can't create enough threads for dpdk workers. the sytem
will stuck at NetworkStack::start().so set ms_dpdk_coremask default 0xF.
and add assert to comapre the two configurations.

Signed-off-by: chunmei <chunmei.liu@intel.com>

Merge pull request #18695 from cooboos/wip-doc-wrongspell

doc/release-notes: fix typo 'psd' to 'osd'

Reviewed-By: Kefu Chai <kchai@redhat.com>

doc: correct wrong spell 'psd' to 'osd'

Signed-off-by: wangsongbo wangsongbo@cloudin.cn

osd/PrimaryLogPG: fix dup stat for async read

The async read would call do_osd_ops twice.

Signed-off-by: Xinze Chi <xinze@xsky.com>

pybind/mgr/mgr_module: fix calc_pg_upmaps

This was fixed in the object cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: enable module by default

It will still be "off".

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: loadgen fix rgw crash issue

When generating random bucket name and object name during
loadgen processing, RGW will crash. The reason is calling
"gen_rand_alphanumeric" with "cct = NULL".

Fixes: http://tracker.ceph.com/issues/22006
Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #13992 from dreamhost/wip-19264

rgw: fix Swift container naming rules.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18606 from adamemerson/wip-op-perm-refactor

rgw: Small refactor and two bug fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #14955 from aclamk/fix-civetweb-cryptopp

radosgw: Make compilation with CryptoPP possible.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18664 from oritwas/wip-rgw-21996

rgw: init oldest period after setting run_sync_thread
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18680 from tchaikov/wip-doc-mgr

doc/mgr: document facilities methods using `automethod` directive

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18685 from jcsp/wip-mgr-load-module

qa: fix mgr _load_module helper

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PG: use osd_backfill_retry_interval for schedule_backfill_retry()

These two options currently share the same value and I have no particular
preference, but using osd_backfill_retry_interval for the specific scenario
gives us more flexible choices.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa: fix mgr _load_module helper

I inadvertently broke this with the latest change
to the module ls output.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18656 from ceph/ceph-volume-bluestore-tests

ceph-volume: adds functional testing for bluestore

Merge pull request #18399 from jcsp/wip-20950

mon: don't blow away bootstrap-mgr on upgrades

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18681 from tchaikov/wip-msg-async-popcnt

msg/async: use bitset<> to do the popcnt

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18652 from idryomov/wip-qa-krbd-blkroset

qa/suites/krbd: add krbd BLKROSET test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

msg/async: use bitset<> to do the popcnt

libstdc++ uses popcnt to count the bits. it's faster and simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr: document mgr/plugin using automethod

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: sync the docstring with the doc/mgr

also update the docstring to confirm to
http://www.sphinx-doc.org/en/stable/domains.html#info-field-lists.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: In plugins 'module' classes need not to be called "Module" anymore.

Fixes: http://tracker.ceph.com/issues/17454
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: bhavishyagopesh <bhavishyagopesh@gmail.com>

Merge pull request #18619 from trociny/wip-qa-mirror-remove

qa/workunits/rbd: unnecessary sleep after failed remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17983 from liewegas/wip-balancer

mgr/balancer: make crush-compat mode work

Reviewed-by: John Spray <john.spray@redhat.com>

mgr: emit cluster log message on serve() exception

Fixes: http://tracker.ceph.com/issues/21999
Signed-off-by: John Spray <john.spray@redhat.com>

rgw: add missing current_history initialization

Fixes: http://tracker.ceph.com/issues/21996
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: init oldest period after setting run_sync_thread

Fixes: http://tracker.ceph.com/issues/21996
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18657 from cbodley/wip-21990

rgw: remove placement_rule from rgw_link_bucket()

Merge pull request #18534 from ZVampirEM77/wip-emlc-contentmd5

rgw: lc support Content-MD5 request header and fix a rgw crash bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #18610 from cbodley/wip-osd-subop-num

osd: add processed_subop_count for cls_cxx_subop_version()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18641 from joke-lee/rgw_modify_subuser_access_error

rgw: modify s3 type subuser access permissions fail through admin rest api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18659 from tchaikov/wip-21993

qa: stop testing deprecated "ceph osd create"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15193 from mynaramana/arraycode

osd: introduce sub-chunks to erasure code plugin interface

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not release empty bluefs_extents_reclaiming.
This saves us one mutex lock attempt in Stupid Allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18637 from yaozongyou/fix-dashboard-bug

osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18347 from Synesis-LLC/fix-21773

mgr: skip first non-zero incremental in PGMap::apply_incremental()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18453 from liewegas/wip-osd-devs

osd: enumerate device names in a simple way

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18509 from tchaikov/wip-osd-cleanups

osd,mgrclient: pass daemon_status by rvalue ref and other cleanups

Reviewed-by: John Spray <john.spray@redhat.com>

qa/tasks/thrashosds: set min_in default to 4

We have EC tests with k=2,m=2, so we need a min of 4.

Fixes: http://tracker.ceph.com/issues/21997
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: remove placement_rule from cls_user_bucket_entry

placement_rule is no longer needed in cls_user_bucket_entry, because the
only time that it's needed, we can read it from the bucket instance in
RGWRados::update_containers_stats()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18504 from badone/wip-GroupSnapshotNamespace-group_pool-uninitialised

cls/rbd: Silence gcc7 maybe-uninitialized warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: remove placement_rule from rgw_link_bucket()

with the fallback in RGWRados::update_containers_stats(), we no longer
need to pass placement_rule into the cls_user_bucket_entry

this removes the dependency between bucket entrypoint metadata and its
bucket instance metadata during multisite sync

Fixes: http://tracker.ceph.com/issues/21990
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: take placement_rule from bucket info in update_containers_stats

in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/balancer: simplify pool_info tracking

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: less verbose on 'eval' by default; add 'eval-verbose'

The verbose output is helpful for debugging and understanding what is
being measured, but most of the time all that you care about is the score.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: fix pg vs object terminology

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: restrict to time of day

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/module: adjust osd_weight min step to .005

That should be ~1 PG or less on average; no real sense it taking a
step smaller than that!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: if score regresses, take a few more steps

To avoid getting stuck in a local minima, take a few additional
steps if our score drops and see if it gets better.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: allow 5% misplaced

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: more aggressive steps

We are smart enough to back off if we misplace too much.

Signed-off-by: Sage Weil <sage@redhat.com>