ptl-tool: allow setting project/repo for GH req

This allows using ptl-tool for other projects like ceph-linode.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18657 from cbodley/wip-21990

rgw: remove placement_rule from rgw_link_bucket()

Merge pull request #18534 from ZVampirEM77/wip-emlc-contentmd5

rgw: lc support Content-MD5 request header and fix a rgw crash bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #18610 from cbodley/wip-osd-subop-num

osd: add processed_subop_count for cls_cxx_subop_version()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18641 from joke-lee/rgw_modify_subuser_access_error

rgw: modify s3 type subuser access permissions fail through admin rest api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18659 from tchaikov/wip-21993

qa: stop testing deprecated "ceph osd create"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15193 from mynaramana/arraycode

osd: introduce sub-chunks to erasure code plugin interface

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18637 from yaozongyou/fix-dashboard-bug

osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18347 from Synesis-LLC/fix-21773

mgr: skip first non-zero incremental in PGMap::apply_incremental()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18453 from liewegas/wip-osd-devs

osd: enumerate device names in a simple way

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18509 from tchaikov/wip-osd-cleanups

osd,mgrclient: pass daemon_status by rvalue ref and other cleanups

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: remove placement_rule from cls_user_bucket_entry

placement_rule is no longer needed in cls_user_bucket_entry, because the
only time that it's needed, we can read it from the bucket instance in
RGWRados::update_containers_stats()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18504 from badone/wip-GroupSnapshotNamespace-group_pool-uninitialised

cls/rbd: Silence gcc7 maybe-uninitialized warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: remove placement_rule from rgw_link_bucket()

with the fallback in RGWRados::update_containers_stats(), we no longer
need to pass placement_rule into the cls_user_bucket_entry

this removes the dependency between bucket entrypoint metadata and its
bucket instance metadata during multisite sync

Fixes: http://tracker.ceph.com/issues/21990
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: take placement_rule from bucket info in update_containers_stats

in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16651 from jcsp/wip-mgr-module-interface

mgr: python interface rework + enable modules to run in standby mode

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: include disabled modules in `mgr module ls`

Otherwise, when someone wants to see what's possible
to do with `mgr module enable` they have to trawl
through the whole mgr map dump.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: describe using `mgr module ...` commands

...including the new "mgr services" command.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: describe how to implement standby modules

Signed-off-by: John Spray <john.spray@redhat.com>

qa: fix mgr caps

This was still using Kraken era settings

Signed-off-by: John Spray <john.spray@redhat.com>

qa: expand mgr testing

Some extra coverage of the dashboard, including its standby
redirect mode and the publishing of URIs.

Also invoking the command_spam mode of the selftest module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/selftest: extend test and add background spam mode

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: drop GIL around set_uri, set_health_checks

These didn't need to keep the GIL to go and do their
pure C++ parts, and by keeping it they could deadlock
while trying to take ActiveMgrModules::lock.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix ~MonCommandCompletion

This was doing a Py_DECREF outside of the Gil.

Fixes: http://tracker.ceph.com/issues/21593
Signed-off-by: John Spray <john.spray@redhat.com>

mgr: update for SafeThreadState

A bunch of the previous commits were done
before this class existed, so updating in
one go instead of trying to edit history
in fine detail.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor PyOSDMap etc implementation

Implement real python classes from the C side,
rather than exposing only module methods.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/PyOSDMap: add CRUSH get_item_weight

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: fix py_module_registry shutdown

Was calling way too early, which did a
Py_Finalize before the modules had been
joined.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix thread naming

Was passing a reference to a local stringstream into
Thread::create, not realising that it was taking a char*
reference instead of a copy.  Result was garbage (or usually,
all threads having the name of the last one created)

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: cut down duplication between active+standby

...by using PyModuleRunner class from ActivePyModule too.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix os._exit overrides

These would throw an exception when passed
a status code.

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMonitor: reset services map on drop_active

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: implement standby mode

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: add MgrStandbyModule

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: standby modules come up and run now

...they still don't have access to any config though.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: enable running modules in standby mode

Modules can implement a second, separate class
that has access to very little state about the
system and can't implement commands.

They have just enough information to redirect
or forward incoming requests/traffic to the
active instance of the module on the active mgr.

This enables module authors to create modules
that end users can access via any (running) mgr node
at any time, rather than having to first work out
which mgr node is active.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: clean up python source file naming

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor python module management

Separate out the *loading* of modules from
the *running* of modules.

This is a precursor to enabling modules to run
in standby mode.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: use set_uri hook from dashboard+restful modules

No more guessing the URL!

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: enable python modules to advertise their service URI

Fixes: http://tracker.ceph.com/issues/17460
Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMonitor: store services in map and expose with command

Signed-off-by: John Spray <john.spray@redhat.com>

messages: `services` in MMgrBeacon

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMap: store list of services

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: carry PyModules ref in MonCommandCompletion

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: update MgrModule for ceph_state->ceph_module

& tidy up the places where ceph_state was getting
used outside of MgrModule.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor python interface

Expose a python class instead of a module,
so that we have a place to carry our reference
to our MgrPyModule* and to PyModules*, rather than
passing a handle for the former and using
a global pointer for the latter.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove blue highlight on scrubbing pg states

This was kind of unnecessary, highlighting a completely normal
and healthy situation in a different colour.  The blue was
also really hard to read against a grey background.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: clean up fs standby list when empty

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18243 from jan--f/mgr-prom-osd-status

pybind/mgr/prometheus: add osd_in/out metric; make osd_weight a metric

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18651 from aermolaev/patch-1

doc: fix typo in radosgw/dynamicresharding.rst

Reviewed-By: Kefu Chai <kchai@redhat.com>

doc: fix typo in dynamicresharding.rst: admon => admin

Signed-off-by: Alexander Ermolaev <ave@integros.com>

mgr: skip first non-zero incremental in PGMap::apply_incremental()

After initialization of PGMap instance PGMap::stamp is zero
and this cause huge first delta.
Also after mgr restart first non-zero value coming to PGMap::apply_incremental()
is current pg_sum value so it produces unreasonably huge pg_sum_delta.
This patch introduces a workaround to save pg_sum and not update pg_sum_delta
by first non-zero incremental.

Signed-off-by: Aleksei Gutikov <aleksey.gutikov@synesis.ru>
Fixes: http://tracker.ceph.com/issues/21773

Merge pull request #18660 from ownedu/wip-fix-async-rdma-typo

msg/async/rdma: a tiny typo fix.

Reviewed-by: Kefu Chai <kefu@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18268 from jcsp/wip-20629

mon: handle monitor lag when killing mgrs

Reviewed-By: Kefu Chai <kchai@redhat.com>

cls_rbd: Silence gcc7 maybe-uninitialized warning

gcc7 warns that GroupSnapshotNamespace::group_pool may be used
uninitialized in SnapshotNamespaceOnDisk::decode. Reordering the member
variables silences the spurious warning.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

msg/async/rdma: a tiny typo fix.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18329 from ifed01/wip-fix-mempool-dump

mempool: fix lack of pool names in mempool:dump output for JSON format

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18575 from majianpeng/msg-cleanup-2

msg/async:  misc cleanup.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18581 from wzliao/wip-drop-unused-parameter

osd/PrimaryLogPG: drop unused parameters

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18588 from mogeb/rados-bench-clock

tools/rados: use the monotonic clock in rados bench

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18595 from batrick/mon-clock

mon: a few conversions to monotonic clock

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18636 from tchaikov/wip-osdmap-cleanup

osd/OSDMap: remove the unnecessary checks for null

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18647 from tchaikov/wip-pgmap-cleanup

mon/PGMap: use new-style options and cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18648 from tchaikov/wip-vstart-drop-num-opts

vstart.sh: drop --{mon,osd,mds,rgw,mgr}_num options

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa: stop testing deprecated "ceph osd create"

"ceph osd create" is not idempotent, and is considered deprecated.

Fixes: http://tracker.ceph.com/issues/21993
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #18618 into master

* refs/pull/18618/head:
mds: pass a reference of function to sanitize() not a copy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #18572 from linuxbox2/wip-rgwfile-env

rgw_log, rgw_file: account for new required envvars

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18327 from PCzhangPC/clonelostmeta

librbd: cannot clone all image-metas if we have more than 64 key/value pairs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18546 from liewegas/wip-deprecate-features

include/ceph_features: deprecate a bunch of features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18616 from xiexingguo/wip-hunt-scrub-mismatch

osd/PrimaryLogPG: more oi.extents fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18634 from tchaikov/wip-qa-add-restful

qa: add "restful" to ceph_mgr_modules in ceph-ansible suite

os/filestore: implement get_devices()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement get_devices()

Signed-off-by: Sage Weil <sage@redhat.com>

osd: return rolled-up 'devices' list as part of metadata

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: add get_devname(), get_devices()

Enumerate device(s) that we are backed by.  It may be plural because of
DM, which might mask an underlying device, or because the dm device is
combining multiple other devices.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18520 from alram/kube-helm-doc

doc: Kube + Helm installation

os/ObjectStore: add get_devices() method

The intent here is to return the set of hardware storage devices, as
identified by the "devname" (i.e., the directory name for the device in
/sys/block/).  For example, "sda", "nvme0n1", etc.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18639 from yaozongyou/make-readme-properly-align

doc: make the commands in README.md properly aligned

Reviewed-by: Amit Kumar <amitkuma@redhat.com>

cmake/cls: add install() for ceph_test_cls_log

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18499 from batrick/ptl-tool-testing-time

ptl-tool: make branch name configurable

Reviewed-by: Jos Collin <jcollin@redhat.com>

ptl-tool: make branch name configurable

Also, include hours/minutes in branch so multiple branches in a day have
different names.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18642 from zhangsw/cleanup-rgwputobj-getdata

rgw: add logs if get_data returns error in RGWPutObj::execute.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18645 from tchaikov/wip-cmake-Findgperftools

cmake: should use the value of GPERFTOOLS_LIBRARIES as REQUIRED_VARS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Changed ECUtil::decode to handle the case where there is more helper data than needed.
Made all the helper information uniform across all helper nodes.
Authors: Myna, Elita.
Signed-off-by: Myna Vajha <mynaramana@gmail.com>

Merge pull request #18283 from tchaikov/wip-laggy-mgr

 qa: reduce mon-client-hunt-interval-max-multiple to 2

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

vstart.sh: drop --{mon,osd,mds,rgw,mgr}_num options

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: drop unused PGMap::update_global_delta()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: use new-style options

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should check the availability of aligned_alloc() by running the test

this mimics the failure of rocksdb without a working
aligned_alloc()/free().

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should use the value of GPERFTOOLS_LIBRARIES as REQUIRED_VARS

otherwise it's always true, even if the libaries are not found. also
the cmake output will be wrong without this change:

without this change, it is:

-- Found gperftools: GPERFTOOLS_TCMALLOC_LIBRARY;GPERFTOOLS_TCMALLOC_MINIMAL_LIBRARY;GPERFTOOLS_PROFILER_LIBRARY (found version "2.5.93")

with this change, it would be:

-- Found gperftools: /tmp/gperf/lib/libtcmalloc.so (found version "2.5.93")

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add logs if get_data returns error in RGWPutObj::execute.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

rgw: modify_s3_type_subuser_access_permissions_fail_through_admin_rest_api

fix: http://tracker.ceph.com/issues/21983

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

doc: make the commands in README.md properly aligned

The commands in README.md are aligned using one tab or eight spaces,
the github markdown engine not handle this properly, so using one tab
universely.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

These two counters should be reported to mgr because the dashboard
plugin needing them to display osd detailed information.

Fixes: http://tracker.ceph.com/issues/21981
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

osd/PrimaryLogPG: drop unused parameters

Signed-off-by: Liao Weizhong <liaoweizhong@unionpay.com>

osd/OSDMap: remove the unnecessary checks for null

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18547 from batrick/ptl-tool-trackers

ptl-tool: print bzs/tickets cited in commit msgs/comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18495 from ceph/wip-da-SCA-20171013

osd,rgw,librbd: SCA fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18535 from yanghonggang/master

os/bluestore/bluestore_tool: add log-dump command to dump bluefs's log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18574 from shashalu/fix-strict-strtol

common: strict_strtol INT_MAX and INT_MIN is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>