qa/workunits/rbd: remove "trash purge --threshold" test

It is difficult to make it work reliably in different environments.

Fixes: http://tracker.ceph.com/issues/22803
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20090 from liewegas/wip-bluestore-label

os/bluestore: make bdev label parsing error more meaningful and less noisy

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #20214 from dillaman/wip-22819

librbd: potential crash if object map check encounters error

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20175 from tone-zhang/wip-fix-bug22828

cmake: fix libcephfs-test.jar build failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20174 from cbodley/wip-log-stream-clear

 log: clear thread-local stream's ios flags on reuse

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20166 from smithfarm/wip-fix-python2-global

build/ops: rpm: fix _defined_if_python2_absent conditional

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20183 from tchaikov/wip-crushtool

crushtool: add --add-bucket and --move options

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20180 from xiexingguo/wip-more-balancer-fixes

mgr/balancer: two more fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20154 from xiexingguo/wip-mgr-balancer-fixes

mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20128 from guzhongyan/master

common/pick_address: wrong prefix_len in pick_iface()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20080 from dillaman/wip-librados-client-compat

librados: getter for min compatible client versions

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

librbd: potential crash if object map check encounters error

Fixes: http://tracker.ceph.com/issues/22819
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20195 from luomuyao/bugfix

rgw: drop a repeated statement for encode_xml()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: drop a repeated statement for encode_xml()

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge PR #19760 into master

* refs/pull/19760/head:
mds: add dump flag for CInode
mds: add command "openfiles ls"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19825 into master

* refs/pull/19825/head:
client: properly choose target mds for rmsnap/renamesnap
mds: properly setup need_snapflush for snapped inode
mds: ignore MClientCaps::FLAG_NO_CAPSNAP

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19833 into master

* refs/pull/19833/head:
common: add C_SaferCond ctor with name argument
Client: use C_SaferCond for "_flush_range"
client: use C_SaferCond for _fallocate and ll_write_block
client: use C_SaferCond for _sync_fs and _fsync
client: use C_SaferCond in Client::_write
client: use C_SaferCond for _read_sync
client: use C_SaferCond in _read_async
client: use C_SaferCond for "Client::_read"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19836 into master

* refs/pull/19836/head:
Avoid the assert failure when the inode for the cap_export from other MDS happened not in MDCache

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19842 into master

* refs/pull/19842/head:
client: unify function name in debug log for Client.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19991 into master

* refs/pull/19991/head:
cephfs-journal-tool: move shutdown to the deconstructor of MDSUtility

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20001 into master

* refs/pull/20001/head:
mds: use the steady clock on tick

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20012 into master

* refs/pull/20012/head:
mds: fix scrub crash

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #20079 into master

* refs/pull/20079/head:
mds: set higher priority for some perf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20113 into master

* refs/pull/20113/head:
osdc/ObjectCacher: don't merge TX buffer heads
osdc/ObjectCacher: fix off-by-one error in bh_write_commit()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20000 from zhouyuan/rgw_beast_listen

rgw: allow beast frontend to listen on specific IP address

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20058 from cbodley/wip-doc-rgw-frontends

doc/rgw: add page for http frontend configuration

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge PR #20165 into master

* refs/pull/20165/head:
doc/client: uniform naming

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20164 from luomuyao/typo

rgw: fix a typo in comment

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

log: clear thread-local stream's ios flags on reuse

when reusing a thread-local std::ostream for a new log entry, clear any
ios flags (ie eofbit/failbit/badbit) that may have been set during its
previous use

without this, any stream errors (for example, passing a nullptr to
operator<< sets the badbit) will result in all log output on that thread
to be blank for the remainder of the process

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cli/crushtool: add test for --add-bucket and --move

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush/CrushWrapper: lower log level of check_item_loc()

it's not an error if the item is not at loc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crushtool: add --move

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20159 from shun-s/fix-rbd-log

librbd: clean up ManagedLock log prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crushtool: add --add-bucket

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: wrong prefix_len in pick_iface()
With prefix_len initialized as zero, mask in netmask_ipv4/6()
will always be zero, so find_ip_in_subnet() always return the
first interface.
Set prefix_len to the right value to fetch the right interface.

Signed-off-by: Gu Zhongyan guzhongyan@360.cn
Signed-off-by: yupeng chen chenyupeng-it@360.cn

pybind/mgr/balancer: fix sanity check against minimal pg numbers per pool

And rename some variables properly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: do not dirty compat_ws on error out

We will get a worse result if we execute the corresponding plan later.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

client: properly choose target mds for rmsnap/renamesnap

Old code does not set inode or dentry for rmsnap/renamesnap requests.
So rmsnap/renamesnap requests are sent to random mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly setup need_snapflush for snapped inode

MDCache::cow_inode() checks "cap->issued() & CEPH_CAP_ANY_WR" to decide
if it needs to setup need_snapflush for the new snapped inode.
Locker::handle_client_caps() should call Locker::_do_cap_update() first,
then update issued caps. Otherwise MDCache::cow_inode() will fail to
setup need_snapflush if cap messages flushes dirty caps and releases the
same caps.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #19770 from shinobu-x/mempool_lock

common: s/unique_lock/lock_guard/, if manual lock/unlock are not necessary

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cmake: fix libcephfs-test.jar build failure

When building Ceph Debian installation package, libcephfs-test.jar
build would fail if multi junit versions are installed on the host.

In src/java/CMakeLists.txt, the "junit" search order is "junit"
and then "junit4". In case multi junit versions are installed,
the lower version (like 3.8) is found firstly, but junit4 is
required only. The search result will cause the failure.

Fix the issue by change search order to firstly search junit4
by default.

Fixes: http://tracker.ceph.com/issues/22828
Signed-off-by: Tone Zhang <tone.zhang@arm.com>

rbd: clean up ManagedLock log prefix

  clean up ManagedLock log prefix

Signed-off-by: shun-s <song.shun3@zte.com.cn>

pybind/mgr/balancer: re-initialize everything on instantiating a new Eval

Otherwise there might be obsolete pool/root infos etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/log: add failing unit test for reuse of bad stream

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20052 from ceph/revert-19253-wip-rgw-22124

Revert "rgw: reshard should not update stats when linking new bucket instance"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20171 from linuxbox2/rgw_rvrt_disable_exact

Revert "rgw_file: disable FLAG_EXACT_MATCH enforcement"

Revert "rgw_file: disable FLAG_EXACT_MATCH enforcement"

This reverts commit 5e0bea71b52f652a5cb6323cf56391ada215c80b.
Fixes: http://tracker.ceph.com/issues/22827
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #20167 from cernceph/dvanders_withhold

osd: change log level when withholding pg creation

Reviewed-by: Sage Weil <sage@redhat.com>

osd: change log level when withholding pg creation

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

build/ops: rpm: fix _defined_if_python2_absent conditional

The expectation is that _defined_if_python2_absent will be defined
if (and only if) the python2 bcond is not set. Before this patch, if
somebody were to build the package on suse_version 1500 with
--with-python2, the _defined_if_python2_absent would be defined,
which is a bug.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge pull request #19758 from nrdmn/fix-dashboard-wip

pybind/mgr/dashboard: fix reverse proxy support

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20142 from shangzhong/wip-java-native

java/native: s/jni: lstat/jni: stat in native_ceph_stat

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20130 from k0ste/newfeature

mgr: balancer: fixed mistype "AttributeError: 'Logger' object has no attribute 'err'"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20127 from majianpeng/issue_op_cleanup

osd/ReplicatedBackend: clean up code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20119 from Liuchang0812/remove-health-service

mon: remove health service

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20092 from tangwenjun3/wip-fix-write-label

os/bluestore: no need to fsync when failed to write label

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20078 from renhwztetecs/renhw-wip-mon-paxos-cleanup

mon: paxos cleanup

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

doc/client: uniform naming

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #20126 from tchaikov/wip-doc-mgr-auth

doc: update mgr related auth settings

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/balancer: pool-specific optimization support

This currently only works for upmap.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #19990 from dillaman/wip-group-cleanup

librbd: group and snapshot cleanup

Reviewed-by: Mykola Golub <mgolub@suse.com>

rbd: normalize group command aliases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: dump op features in image info

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: normalize image group API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

java/native: s/jni: lstat/jni: stat in native_ceph_stat

Signed-off-by: Shangzhong Zhu <zhu.shangzhong@zte.com.cn>

Merge pull request #19828 from xxhdx1985126/wip_xxh_trackedop_tracking_start

common: mark events of TrackedOp outside its constructor

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20022 from badone/wip-OSDMonitor-error-message-outputs

mon/OSDMonitor: Fix OSDMonitor error message outputs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20115 from liewegas/wip-features

mon: fix structure of 'features' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20096 from myoungwon/wip-manifest-two-pool-test

src/test/osd: add two pool test for manifest objects

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20018 from smithfarm/wip-python3-only

build/ops: rpm: conditionalize Python 2 availability to enable Ceph build on Python 3-only system

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #20134 from tchaikov/wip-spdk-aarch64

spdk: update SPDK to fix the build failure on aarch64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/ReplicatedBackend: clean up code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: normalize image group C/C++ API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: shorten the generated group image snapshot name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: properly handle group snap add failure case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplified retrieving snapshots from image header

Added a new cls 'snapshot_get' method that retrieves all snapshot
specific details to avoid the need to add a new cls method to
retrieve each individual snapshot data field.

The cls 'get_snapshot_namespace' method has been dropped since Luminous
clients will treat that as a user-generated snapshot (and that's all
that is supported by Luminous clients). The cls 'get_snapshot_timestamp'
method has been deprecated and can be removed once <=Luminous clients
are no longer supported.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: automatically manage group op feature on images

When an image is added to a group, set the group op feature
bit and when removed, clear the bit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: renamed image group add/remove/get operations

The methods are used by the luminous release of librbd but are essentially
no-ops since group support was removed from the API. However, we need to
prevent odd interactions once a new client creates a group and uses an
older client to manipulate the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: add new op feature bits for groups and snap trash

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: removed unused snapshot feature bitmask

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: cleanup helper method visibility

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: group image add/remove ops should notify update

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove 'consistency' verbiage from group feature

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: treat empty features environment variable as old format

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

src/test/osd/TestRados: add a comment

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

pybind/mgr: fix typo

Signed-off-by: Nick Erdmann <n@nirf.de>

pybind/mgr/dashboard: fix reverse proxy support

This fixes http redirection for reverse http proxies

Fixes: http://tracker.ceph.com/issues/22557
Signed-off-by: Nick Erdmann <n@nirf.de>

Merge pull request #20064 from tchaikov/wip-cmake-py3

cmake: enabled py3 only build

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

spdk: enable SPDK on AArch64 by default

Enable SPDK in Ceph on AArch64 by default.

Change-Id: I81ad8e0b860968d62fd2740ae04923f477dab75a
Signed-off-by: Tone Zhang <tone.zhang@arm.com>

spdk: update SPDK to fix the build failure on aarch64

to include 29be88fab6bb026df27c93ee89c61965b3c43133 which addresses a
build failure on aarch64.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: be compatible with py3

so ceph-mgr can be compiled using py3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build ceph-mgr with specified version of python

* add an option named "MGR_PYTHON_VERSION", so we can build ceph-mgr
  which use py3 for running plugins
* also drop the line to specify the "Python_ADDITIONAL_VERSIONS", because
  2.7 is listed by all the the FindPythonInterp and FindPythonLibs in
  cmake 2.8.12 and up.
* use ${MGR_PYTHON_EXECUTABLE} for holding the path to the python
  interpreter used by mgr. because this variable might be overwritten by
  "find_package(PythonInterp 2 REQUIRED)" when checking for building env
  of pybinding for python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add WITH_PYTHON2 option

* add WITH_PYTHON2 option, so we can build python3 bindings only.
* change the default value of WITH_PYTHON3 option to "OFF", as the
option() command in cmake only allow the initial value to be "ON" or
"OFF". we could use a cached string for this option, but i think it
would be more explicit to continue using the option() command.
* fix the installation dir of "ceph_rest_api.py". please note, we still
have a *default* python version, which is specified by the last element
of ${py_vers}. for instance, ${PYTHON_VERSION} will be 3 if ${py_vers}
is 2;3. in this change, 2 is still the default python version, if both
WITH_PYTHON2 and WITH_PYTHON3 are enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: balancer: fixed mistype "AttributeError: 'Logger' object has no attribute 'err'"

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

Merge pull request #19922 from Rubab-Syed/performance_counter_browser

mgr/dashboard: performance counter browsers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20009 from Liuchang0812/fix-22727

mon: do not use per_pool_sum_delta to show recovery summary

Reviewed-by: Varada Kari <varada.kari@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20048 from jcsp/wip-22096

mon: fix mgr using auth_client_required policy

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update mgr related auth settings

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash/workloads: add paramters to support two pools

Signed-off-by: Myoungwon Oh <omwmw@sk.com>