mon/OSDMonitor: ignore and/or remove pgs that shouldn't exist

The update_creating_pgs() change is there mainly for clusters that already
have an invalid pg in the map; going forward they hopefully shouldn't
appear!

Fixes: https://tracker.ceph.com/issues/22847
Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: reject force-create-pg on pgids that shouldn't exist

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19902 from linuxbox2/wip-rgw-mploc

rgw: return valid Location element, CompleteMultipartUpload

Merge pull request #20199 from Songweibin/wip-group-cleanup

rbd: group misc cleanup and update rbd man page

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20002 from malc0lm/fix-copypart

rgw: fix 'copy part' without 'x-amz-copy-source-range'

Merge pull request #20222 from trociny/wip-deep-copy-default-format

librbd: by default use new format for deep copy destination

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20170 from trociny/wip-22803

qa/workunits/rbd: remove "trash purge --threshold" test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20099 from Songweibin/wip-fix-deep-cp

librbd: fix deep copy a child-image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20178 from xiexingguo/wip-map-version

crush, mon: bump up map version only if we truly created a weight-set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20197 from xiexingguo/wip-load-ws-from-ms

pybind/mgr/balancer: load weight-set from ms

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20196 from yaozongyou/improve-straw2-readability

crush: improve straw2 algorithm's readability

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20194 from xiexingguo/wip-badsteps

pybind/mgr/balancer: increase bad_steps properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20189 from neha-ojha/wip-fix-update_calc_stats

PG: primary should not be in the peer_info, skip if it is

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20151 from badone/wip-run-tox-ceph-disk-deps-failure

tests: ceph-disk: Remove sitepackages=True

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20140 from smithfarm/wip-python3-mopup

build/ops: rpm: Python 3-only ceph-disk and ceph-volume

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19953 from badone/wip-c_read_operations-tautological-compare-warning

c_read_operations.cc: Silence tautological-compare compiler warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20071 from joscollin/wip-ceph-fuse-doc

doc: Specify mount details in ceph-fuse

Reviewed-by: Zheng Yan <zyan@redhat.com>

doc: Specify mount details in ceph-fuse

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #19856 into master

* refs/pull/19856/head:
qa: synchronize kcephfs suites with fs/multimds

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #20203 from ceph/wip-rm22642

ceph-volume: fix usage of the --osd-id flag

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd: remove "trash purge --threshold" test

It is difficult to make it work reliably in different environments.

Fixes: http://tracker.ceph.com/issues/22803
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20090 from liewegas/wip-bluestore-label

os/bluestore: make bdev label parsing error more meaningful and less noisy

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #20214 from dillaman/wip-22819

librbd: potential crash if object map check encounters error

Reviewed-by: Mykola Golub <mgolub@suse.com>

librbd: by default use new format for deep copy destination

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20175 from tone-zhang/wip-fix-bug22828

cmake: fix libcephfs-test.jar build failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20174 from cbodley/wip-log-stream-clear

 log: clear thread-local stream's ios flags on reuse

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20166 from smithfarm/wip-fix-python2-global

build/ops: rpm: fix _defined_if_python2_absent conditional

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20183 from tchaikov/wip-crushtool

crushtool: add --add-bucket and --move options

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20180 from xiexingguo/wip-more-balancer-fixes

mgr/balancer: two more fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20154 from xiexingguo/wip-mgr-balancer-fixes

mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20128 from guzhongyan/master

common/pick_address: wrong prefix_len in pick_iface()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20080 from dillaman/wip-librados-client-compat

librados: getter for min compatible client versions

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

crush: improve straw2 algorithm's readability

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

librbd: group misc cleanup

* Add some error message when returning a negative error code.
* Validate dest name when renaming a group snap.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: fix deep copy a child-image

* tweak create a cloned image when the source image is
  a clone (or at least one of its snapshots is a clone).

Signed-off-by: songweibin <song.weibin@zte.com.cn>

ceph-volume: adds tests for zap --destroy and --osd-id

These are not added for dmcrypt right now because there is a bug related
to zapping encrypted disks that needs fixed first.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

librbd: potential crash if object map check encounters error

Fixes: http://tracker.ceph.com/issues/22819
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20195 from luomuyao/bugfix

rgw: drop a repeated statement for encode_xml()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: no need to return osd_id from util.prepare.check_id

Now that osd_id is passed to util.prepare.create_id it doesn't
make any sense to return osd_id from check_id anymore as it's
not being used.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

pybind/mgr/balancer: load weight-set from ms

See https://github.com/ceph/ceph/pull/20178, so we won't
try to call 'ceph osd crush weight-set create-compat' multiple times.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd: add open/close section when listing group images

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: translate pool id into pool name in error msg

Signed-off-by: songweibin <song.weibin@zte.com.cn>

pybind/rbd: fix some typo

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rgw: If use 'copy part' without 'x-amz-copy-source-range',
it will copy an entire source object

Signed-off-by: Malcolm Lee <fengxueyu35@126.com>

doc/rbd: update rbd man page

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: add missing period at the end of mesg

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: s/group-spec/group-snap-spec for group snapshot

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rgw: drop a repeated statement for encode_xml()

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

pybind/mgr/balancer: increase bad_steps properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush, mon: bump up map version only if we truly created a weight-set

In crush-compat mode mgr/balancer will do 'ceph osd crush weight-set create-compat'
on each call to 'ceph balancer optimize <plan>', which turns out to be
bumping up osdmap version without modifying anything (we haven't executed the
plan yet).

E.g.:
./bin/ceph balancer reset && ./bin/ceph balancer optimize crush test && ./bin/ceph balancer show crush
// starting osdmap epoch 993
// starting crush version 962
// mode crush-compat

./bin/ceph balancer reset && ./bin/ceph balancer optimize crush test && ./bin/ceph balancer show crush
// starting osdmap epoch 994
// starting crush version 963
// mode crush-compat

Fix the above problem by checking whether a new weight-set is truly created or not
on the OSDMonitor-side.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

PG: primary should not be in the peer_info, skip if it is

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #19760 into master

* refs/pull/19760/head:
mds: add dump flag for CInode
mds: add command "openfiles ls"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19825 into master

* refs/pull/19825/head:
client: properly choose target mds for rmsnap/renamesnap
mds: properly setup need_snapflush for snapped inode
mds: ignore MClientCaps::FLAG_NO_CAPSNAP

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19833 into master

* refs/pull/19833/head:
common: add C_SaferCond ctor with name argument
Client: use C_SaferCond for "_flush_range"
client: use C_SaferCond for _fallocate and ll_write_block
client: use C_SaferCond for _sync_fs and _fsync
client: use C_SaferCond in Client::_write
client: use C_SaferCond for _read_sync
client: use C_SaferCond in _read_async
client: use C_SaferCond for "Client::_read"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19836 into master

* refs/pull/19836/head:
Avoid the assert failure when the inode for the cap_export from other MDS happened not in MDCache

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19842 into master

* refs/pull/19842/head:
client: unify function name in debug log for Client.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19991 into master

* refs/pull/19991/head:
cephfs-journal-tool: move shutdown to the deconstructor of MDSUtility

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20001 into master

* refs/pull/20001/head:
mds: use the steady clock on tick

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20012 into master

* refs/pull/20012/head:
mds: fix scrub crash

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #20079 into master

* refs/pull/20079/head:
mds: set higher priority for some perf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20113 into master

* refs/pull/20113/head:
osdc/ObjectCacher: don't merge TX buffer heads
osdc/ObjectCacher: fix off-by-one error in bh_write_commit()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: when reusing an osd ID you must give that id to 'osd new'

If you do not then auth will not be created for the new OSD and the
daemon will not be able to start.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: prepare_* methods do not need to recreate fsid or osd_id

The prepare_filestore and prepare_bluestore methods do not need to
attempt to recreate osd_id or fsid because this happens in prepare
already.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph_volume: modify util.prepare.check_id to handle stdout as a list

process.call returns stdout as a list, not a string

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20000 from zhouyuan/rgw_beast_listen

rgw: allow beast frontend to listen on specific IP address

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20058 from cbodley/wip-doc-rgw-frontends

doc/rgw: add page for http frontend configuration

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge PR #20165 into master

* refs/pull/20165/head:
doc/client: uniform naming

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20164 from luomuyao/typo

rgw: fix a typo in comment

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

ceph-volume: tests for util.prepare.check_id

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: return osd_id from util.prepare.check_id if it exists

This also changes this so the osd_id is returned as a string so
an ID of 0 would evaluate to True.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

log: clear thread-local stream's ios flags on reuse

when reusing a thread-local std::ostream for a new log entry, clear any
ios flags (ie eofbit/failbit/badbit) that may have been set during its
previous use

without this, any stream errors (for example, passing a nullptr to
operator<< sets the badbit) will result in all log output on that thread
to be blank for the remainder of the process

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cli/crushtool: add test for --add-bucket and --move

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush/CrushWrapper: lower log level of check_item_loc()

it's not an error if the item is not at loc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crushtool: add --move

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20159 from shun-s/fix-rbd-log

librbd: clean up ManagedLock log prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crushtool: add --add-bucket

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: wrong prefix_len in pick_iface()
With prefix_len initialized as zero, mask in netmask_ipv4/6()
will always be zero, so find_ip_in_subnet() always return the
first interface.
Set prefix_len to the right value to fetch the right interface.

Signed-off-by: Gu Zhongyan guzhongyan@360.cn
Signed-off-by: yupeng chen chenyupeng-it@360.cn

pybind/mgr/balancer: fix sanity check against minimal pg numbers per pool

And rename some variables properly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: do not dirty compat_ws on error out

We will get a worse result if we execute the corresponding plan later.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

client: properly choose target mds for rmsnap/renamesnap

Old code does not set inode or dentry for rmsnap/renamesnap requests.
So rmsnap/renamesnap requests are sent to random mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: properly setup need_snapflush for snapped inode

MDCache::cow_inode() checks "cap->issued() & CEPH_CAP_ANY_WR" to decide
if it needs to setup need_snapflush for the new snapped inode.
Locker::handle_client_caps() should call Locker::_do_cap_update() first,
then update issued caps. Otherwise MDCache::cow_inode() will fail to
setup need_snapflush if cap messages flushes dirty caps and releases the
same caps.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

c_read_operations.cc: Silence tautological-compare compiler warning

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19770 from shinobu-x/mempool_lock

common: s/unique_lock/lock_guard/, if manual lock/unlock are not necessary

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cmake: fix libcephfs-test.jar build failure

When building Ceph Debian installation package, libcephfs-test.jar
build would fail if multi junit versions are installed on the host.

In src/java/CMakeLists.txt, the "junit" search order is "junit"
and then "junit4". In case multi junit versions are installed,
the lower version (like 3.8) is found firstly, but junit4 is
required only. The search result will cause the failure.

Fix the issue by change search order to firstly search junit4
by default.

Fixes: http://tracker.ceph.com/issues/22828
Signed-off-by: Tone Zhang <tone.zhang@arm.com>

rbd: clean up ManagedLock log prefix

  clean up ManagedLock log prefix

Signed-off-by: shun-s <song.shun3@zte.com.cn>

pybind/mgr/balancer: re-initialize everything on instantiating a new Eval

Otherwise there might be obsolete pool/root infos etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/log: add failing unit test for reuse of bad stream

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20052 from ceph/revert-19253-wip-rgw-22124

Revert "rgw: reshard should not update stats when linking new bucket instance"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20171 from linuxbox2/rgw_rvrt_disable_exact

Revert "rgw_file: disable FLAG_EXACT_MATCH enforcement"

ceph-volume: check to see if an OSD id exists before reusing it

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a prepare util for checking OSD ID existance

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Revert "rgw_file: disable FLAG_EXACT_MATCH enforcement"

This reverts commit 5e0bea71b52f652a5cb6323cf56391ada215c80b.
Fixes: http://tracker.ceph.com/issues/22827
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #20167 from cernceph/dvanders_withhold

osd: change log level when withholding pg creation

Reviewed-by: Sage Weil <sage@redhat.com>

osd: change log level when withholding pg creation

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

build/ops: rpm: fix _defined_if_python2_absent conditional

The expectation is that _defined_if_python2_absent will be defined
if (and only if) the python2 bcond is not set. Before this patch, if
somebody were to build the package on suse_version 1500 with
--with-python2, the _defined_if_python2_absent would be defined,
which is a bug.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge pull request #19758 from nrdmn/fix-dashboard-wip

pybind/mgr/dashboard: fix reverse proxy support

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20142 from shangzhong/wip-java-native

java/native: s/jni: lstat/jni: stat in native_ceph_stat

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20130 from k0ste/newfeature

mgr: balancer: fixed mistype "AttributeError: 'Logger' object has no attribute 'err'"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20127 from majianpeng/issue_op_cleanup

osd/ReplicatedBackend: clean up code.

Reviewed-by: Kefu Chai <kchai@redhat.com>