rgw: reuse CURL* for keystone.

When using keystone with https (the recommended setting),
it is inefficient to start up a new SSL connection for
each and every operation.  Keeping a CURL* structure around
should reduce the cost of doing this.  This logic tries
to do so, but it also tries to free them fairly aggressively
(5-10 seconds).  This should still greatly reduce load on
keystone at peak times while not tying up excess resources.

Fixes: https://tracker.ceph.com/issues/23162
Signed-off-by: Marcus Watts <mwatts@redhat.com>

build/link against curl w/ openssl not nss or gnutls. (debian)

curl + gnutls has some big significant performance hits at least
when doing keystone validation.  nss has long-term memory growth
issues.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #20016 from cbodley/wip-18885

rgw: log unlink_instance mtime as object's mtime

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20145 from linuxbox2/rgwf-deref-readdir-off

rgw_file: avoid evaluating nullptr for readdir offset

Merge pull request #20188 from cbodley/wip-rgw-frontend-multimap

rgw: beast frontend can listen on multiple endpoints

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20013 from cbodley/wip-rgw-auth-skew

rgw: share time skew check between v2 and v4 auth

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20017 from cbodley/wip-22248

rgw: unlink deleted bucket from bucket's owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20120 from cbodley/wip-22797

rgw: fix for empty query string in beast frontend

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20038 from oritwas/wip-rgw-22742

rgw: bucket resharding should not update bucket ACL or user stats
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20295 from trociny/wip-test_deep_copy_clone

test/pybind/rbd: skip test_deep_copy_clone if layering not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20309 from pritha-srivastava/wip-rgw-roles-error-codes

rgw: Reinstating error codes mapping for Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20316 from smithfarm/wip-doc-log

doc: log-and-debug: fix default value of "log max recent"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: log-and-debug: fix default value of "log max recent"

Doc says 1000000, but the real figure is two orders of magnitude less:

    Option("log_max_recent", Option::TYPE_INT, Option::LEVEL_ADVANCED)
      .set_default(500)
      .set_daemon_default(10000)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20229 from zhsj/fix-dashboard-prefix

pybind/mgr/dashboard: fix duplicated slash in html href

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: Reinstating error codes mapping for Roles.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #20114 from cethikdata/ceph-disk-lock

ceph-disk: reduce the scope of activate_lock.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20270 from ivancich/wip-fix-bad-dmclock-sim-change

osd,dmclock: fix dmclock test simulator change

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20258 from tchaikov/wip-balancer-cleanup

 mgr/balancer: replace magic value of -1 for DEFAULT_CHOOSE_ARGS

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20230 from ifed01/wip-ifed-store-test-improve

test/store_test: update Many4KWritesTest* test cases to finalize with…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20221 from fangyuxiangGL/max

osd/ReplicatedBackend: 'osd_deep_scrub_keys' doesn't work

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20217 from rzarzynski/wip-osd-legacy-conf-for-perf

osd: avoid the config's get_val() overhead on the read path.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20169 from liewegas/wip-22668-revert

os/bluestore: disable on_applied sync_complete

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: bucket resharding should not update bucket ACL or user stats

bucket link op resets bucket ACL using rgw_link_bucket instead

Fixes: http://tracker.ceph.com/issues/22124
Fixes: http://tracker.ceph.com/issues/22742
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

pybind/mgr/dashboard: fix duplicated slash in html href

The url_prefix val in html templates is expected not containing '/'
in the end. Otherwise the href will be rendered as '//static/', which
causes browser treating it as http://static/.

Fixes: http://tracker.ceph.com/issues/22851
Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>

test/pybind/rbd: skip test_deep_copy_clone if layering not enabled

(otherwise it will fail on protect snap)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20184 from shun-s/wip-add-osdmap-epoch-show

osd/OSDMap: add osdmap epoch info when printing info summary

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20225 from xiexingguo/wip-balancer-pool-options

mgr/balancer: more pool-specific enhancements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20224 from xiexingguo/wip-still-mgr-fixes

pybind/mgr/balancer: remove optimization plan properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20278 from xiexingguo/wip-null-ws-check

pybind/mgr/balancer: fix sanity check against empty weight-set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20261 into master

* refs/pull/20261/head:
doc: Fix path in ceph-fuse.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/mgr_module: do not offer empty default value for choose_args

as it should be always available. this also silence a pep8 warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/balancer: introduce CRUSHMap.DEFAULT_CHOOSE_ARGS and use it

avoid using '-1' as a magic variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: fix sanity check against empty weight-set

There are paths get_compat_weight_set_weights() will return a
empty dict, not None.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert change from commit 8c5fea68cb9e0c2e6cc2352e64c4f3f46e29c58f
that introduced a compilation error that was not noticed since the
dmclock support/testing simulators are not built by ceph.

Also, changes below the src/dmclock tree should be made in the dmclock
library and then pulled into ceph.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #19711 from trociny/wip-20762

rbd: unified way to map images using different drivers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19902 from linuxbox2/wip-rgw-mploc

rgw: return valid Location element, CompleteMultipartUpload

Merge pull request #20199 from Songweibin/wip-group-cleanup

rbd: group misc cleanup and update rbd man page

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20002 from malc0lm/fix-copypart

rgw: fix 'copy part' without 'x-amz-copy-source-range'

Merge pull request #20222 from trociny/wip-deep-copy-default-format

librbd: by default use new format for deep copy destination

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20170 from trociny/wip-22803

qa/workunits/rbd: remove "trash purge --threshold" test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20099 from Songweibin/wip-fix-deep-cp

librbd: fix deep copy a child-image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix path in ceph-fuse.rst

This mistake was introduced in https://github.com/ceph/ceph/commit/9b6f9da3ce935ba1dcc28dfd6723ca46883ccb5e.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20178 from xiexingguo/wip-map-version

crush, mon: bump up map version only if we truly created a weight-set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20197 from xiexingguo/wip-load-ws-from-ms

pybind/mgr/balancer: load weight-set from ms

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20196 from yaozongyou/improve-straw2-readability

crush: improve straw2 algorithm's readability

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20194 from xiexingguo/wip-badsteps

pybind/mgr/balancer: increase bad_steps properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20189 from neha-ojha/wip-fix-update_calc_stats

PG: primary should not be in the peer_info, skip if it is

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20151 from badone/wip-run-tox-ceph-disk-deps-failure

tests: ceph-disk: Remove sitepackages=True

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20140 from smithfarm/wip-python3-mopup

build/ops: rpm: Python 3-only ceph-disk and ceph-volume

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19953 from badone/wip-c_read_operations-tautological-compare-warning

c_read_operations.cc: Silence tautological-compare compiler warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20071 from joscollin/wip-ceph-fuse-doc

doc: Specify mount details in ceph-fuse

Reviewed-by: Zheng Yan <zyan@redhat.com>

doc: Specify mount details in ceph-fuse

Signed-off-by: Jos Collin <jcollin@redhat.com>

OSDMap: add osdmap epoch info when printing info summary

  add osdmap epoch info when printing info summary

Signed-off-by: shun-s <song.shun3@zte.com.cn>

Merge PR #19856 into master

* refs/pull/19856/head:
qa: synchronize kcephfs suites with fs/multimds

Reviewed-by: Zheng Yan <zyan@redhat.com>

rgw: beast frontend can listen on multiple endpoints

rgw_frontends options port= and endpoint= can be given multiple times,
and the beast frontend will listen on each endpoint

Fixes: http://tracker.ceph.com/issues/22779
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20203 from ceph/wip-rm22642

ceph-volume: fix usage of the --osd-id flag

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd: remove "trash purge --threshold" test

It is difficult to make it work reliably in different environments.

Fixes: http://tracker.ceph.com/issues/22803
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20090 from liewegas/wip-bluestore-label

os/bluestore: make bdev label parsing error more meaningful and less noisy

Reviewed-by: Varada Kari <varada.kari@gmail.com>

test/store_test: update Many4KWritesTest* test cases to finalize with fsck

Doing that when statfs report doesn't match our expectations will
provide more insight on the mismatch and helps to avoid races if any.
Maybe Fixes: https://tracker.ceph.com/issues/21312

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20214 from dillaman/wip-22819

librbd: potential crash if object map check encounters error

Reviewed-by: Mykola Golub <mgolub@suse.com>

osd: avoid the config's get_val() overhead on the read path.

Profiling shows the overhead of the md_config_t::get_val<T>
can be significant. Unfortunately, it is being used in two
critical places across the read path.

The patch resorts to the legacy config infrastructure to
mitigate the performance penalty. It is expected it will
be superseded with a solution that allows to use the new,
intended routines without hurting performance.
`
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

pybind/mgr/balancer: make 'eval' command can evaluate specific pool

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/balancer: pool-specific optimization for crush-compat

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: remove optimization plan properly

Should pass in plan name instead of the plan itself.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librbd: by default use new format for deep copy destination

Signed-off-by: Mykola Golub <mgolub@suse.com>

osd/ReplicatedBackend: 'osd_deep_scrub_keys' doesn't work

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #20175 from tone-zhang/wip-fix-bug22828

cmake: fix libcephfs-test.jar build failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20174 from cbodley/wip-log-stream-clear

 log: clear thread-local stream's ios flags on reuse

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20166 from smithfarm/wip-fix-python2-global

build/ops: rpm: fix _defined_if_python2_absent conditional

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20183 from tchaikov/wip-crushtool

crushtool: add --add-bucket and --move options

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20180 from xiexingguo/wip-more-balancer-fixes

mgr/balancer: two more fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20154 from xiexingguo/wip-mgr-balancer-fixes

mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20128 from guzhongyan/master

common/pick_address: wrong prefix_len in pick_iface()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20080 from dillaman/wip-librados-client-compat

librados: getter for min compatible client versions

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

crush: improve straw2 algorithm's readability

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

librbd: group misc cleanup

* Add some error message when returning a negative error code.
* Validate dest name when renaming a group snap.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: fix deep copy a child-image

* tweak create a cloned image when the source image is
  a clone (or at least one of its snapshots is a clone).

Signed-off-by: songweibin <song.weibin@zte.com.cn>

ceph-volume: adds tests for zap --destroy and --osd-id

These are not added for dmcrypt right now because there is a bug related
to zapping encrypted disks that needs fixed first.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

librbd: potential crash if object map check encounters error

Fixes: http://tracker.ceph.com/issues/22819
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20195 from luomuyao/bugfix

rgw: drop a repeated statement for encode_xml()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: no need to return osd_id from util.prepare.check_id

Now that osd_id is passed to util.prepare.create_id it doesn't
make any sense to return osd_id from check_id anymore as it's
not being used.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

pybind/mgr/balancer: load weight-set from ms

See https://github.com/ceph/ceph/pull/20178, so we won't
try to call 'ceph osd crush weight-set create-compat' multiple times.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd: add open/close section when listing group images

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: translate pool id into pool name in error msg

Signed-off-by: songweibin <song.weibin@zte.com.cn>

pybind/rbd: fix some typo

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rgw: If use 'copy part' without 'x-amz-copy-source-range',
it will copy an entire source object

Signed-off-by: Malcolm Lee <fengxueyu35@126.com>

doc/rbd: update rbd man page

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: add missing period at the end of mesg

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: s/group-spec/group-snap-spec for group snapshot

Signed-off-by: songweibin <song.weibin@zte.com.cn>

rgw: drop a repeated statement for encode_xml()

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

pybind/mgr/balancer: increase bad_steps properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush, mon: bump up map version only if we truly created a weight-set

In crush-compat mode mgr/balancer will do 'ceph osd crush weight-set create-compat'
on each call to 'ceph balancer optimize <plan>', which turns out to be
bumping up osdmap version without modifying anything (we haven't executed the
plan yet).

E.g.:
./bin/ceph balancer reset && ./bin/ceph balancer optimize crush test && ./bin/ceph balancer show crush
// starting osdmap epoch 993
// starting crush version 962
// mode crush-compat

./bin/ceph balancer reset && ./bin/ceph balancer optimize crush test && ./bin/ceph balancer show crush
// starting osdmap epoch 994
// starting crush version 963
// mode crush-compat

Fix the above problem by checking whether a new weight-set is truly created or not
on the OSDMonitor-side.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

PG: primary should not be in the peer_info, skip if it is

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #19760 into master

* refs/pull/19760/head:
mds: add dump flag for CInode
mds: add command "openfiles ls"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19825 into master

* refs/pull/19825/head:
client: properly choose target mds for rmsnap/renamesnap
mds: properly setup need_snapflush for snapped inode
mds: ignore MClientCaps::FLAG_NO_CAPSNAP

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19833 into master

* refs/pull/19833/head:
common: add C_SaferCond ctor with name argument
Client: use C_SaferCond for "_flush_range"
client: use C_SaferCond for _fallocate and ll_write_block
client: use C_SaferCond for _sync_fs and _fsync
client: use C_SaferCond in Client::_write
client: use C_SaferCond for _read_sync
client: use C_SaferCond in _read_async
client: use C_SaferCond for "Client::_read"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19836 into master

* refs/pull/19836/head:
Avoid the assert failure when the inode for the cap_export from other MDS happened not in MDCache

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #19842 into master

* refs/pull/19842/head:
client: unify function name in debug log for Client.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19991 into master

* refs/pull/19991/head:
cephfs-journal-tool: move shutdown to the deconstructor of MDSUtility

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20001 into master

* refs/pull/20001/head:
mds: use the steady clock on tick

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>