pybind/ceph_argparse: disambiguate mgr tell and CLI commands

The mgr tell commands are somewhat special in that you can tell the mgr
with an empty id ('ceph tell mgr' or target ('mgr', '')) to get the
currently active mgr.  This makes it hard to disabiguate between a tell
command and a CLI command.

Fix that by explicitly setting the target to 'mon-mgr' when a CLI command
is flagged as a mgr command.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: make 'ceph tell mgr.*' send to the active mgr

We can only send tell commands to the active mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: send 'ceph tell mgr.X' to the right mgr

Previously this would go to the active mgr, whoever that was.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add rados_mgr_command_target

This variant lets you specify which mgr to talk to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: add start_command variant that takes a target

Note that the initial implementation can only target the active mgr!

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: drop unregister_command(); use per-hook variant

There's never a need to unregister individual commands; doing it by hook
is sufficient for all users.  Simpler and faster.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: drop explicit prefix arg to register_command

This is always the cmddesc prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: simplify command routing

Back in e30e937c8962249af283a7571eb106ef444b79e3 we made it possible to
route a command via any prefix. This worked when we wanted to pass
arguments but were just dealing with a vector<string>.  These days we have
an actual prefix followed by named arguments, so we don't need this
ad hoc routing.

Derive the prefix from the cmddesc at registration time, and match that
explicitly against the prefix at execution time.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: add ability to process MCommand via asok queue

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: pass cmdvec to execute_command

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: use pipe for general wakeup

Use the pipe to wake up the thread.  Use a separate bool to signal a
shutdown.

Signed-off-by: Sage Weil <sage@redhat.com>

include/compat: add flags arg to pipe_cloexec

This matches pipe(2).

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: drop unused args

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28569 from alimaredia/wip-rgw-ioctx-operate-replace

rgw: replace direct calls to ioctx.operate()

Merge pull request #30655 from tchaikov/wip-cmake/crimson/pass-args

cmake: pass arguments to crimson tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30613 from joke-lee/rgw-ceph-dencoder-add-RGWPeriodLatestEpochInfo

ceph-dencoder: add RGWPeriodLatestEpochInfo support.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: pass arguments to crimson tests

otherwise they will take all CPU cores and all memory

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30587 from sebastian-philipp/python-common-readme

python-common: Add small Readme

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #30341 into master

* refs/pull/30341/head:
cephfs-shell: better complain info, when deleting non-empty directory

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #30559 into master

* refs/pull/30559/head:
mds: Reorganize class members in MDBalancer header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30580 into master

* refs/pull/30580/head:
client: don't ceph_abort on bad llseek whence value
client: remove Inode dir_contacts field

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30407 from joscollin/wip-B41841-yes-really-mean-it

mgr/volumes: protection for `fs volume rm` command

Merge PR #30552 into master

* refs/pull/30552/head:
cmake: fix libtsan detection
test: librados startup/shutdown racer test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30630 from tchaikov/wip-build-on-centos8

install-deps,rpm,do_cmake: build on RHEL/CentOS 8

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #30531 from alimaredia/wip-rgw-maven-version-update

qa: bump maven repo version in s3a_hadoop.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30615 from xiexingguo/wip-progress-time-remaining

mgr/progress: estimated remaining time for events

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30591 from xiexingguo/wip-balancer-throttler

mgr/balancer: upmap_max_iterations -> upmap_max_optimizations; behave as it is per pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29674 from athanatos/sjust/wip-fio-trace

os/bluestore: expand lttng tracepoints, improve fio_ceph_objectstore backend

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cls: CLS_CLIET_HIDE_IOCTX changes in cls headers

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: various ioctx.operate refactoring changes

- remove unused #include for cls_client headers
- refactor cls_timeindex_trim call in
rgw_object_expirer_core.cc
- add more CLS_CLIENT_HIDE_IOCTX in header files
- move target_compile_definitions to rgw_a target
- make repeat cls helper functions static

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: overload cls_rgw functions that do io_ctx.operate()

Call the new overloaded functions and call
rgw_rados_operate after them.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cls: remove unused cls_rgw_reshard_get_head()

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ioctx.operate() with rgw_rados_operate()

For all operate calls in:
rgw_op.cc
rgw_orphan.cc
rgw_reshard.cc
rgw_rados.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: fix libtsan detection

This variable is case-sensitive.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30620 from votdev/issue_42069_validate_bucket_name

mgr/dashboard: Bucket names cannot be formatted as IP address

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>

Merge pull request #30647 from s0nea/wip-dashboard-fix-typo

mgr/dashboard: fix small typos in description message

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29925 from sebastian-philipp/run_tox-overwrite-build-dir

script/run_tox.sh: Don't overwrite the build dir

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix two small typos

Remove unnecessary blanks.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

script/run_tox.sh: Don't overwrite the Ceph's bulid dir

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #30639 from tchaikov/wip-crimson-readable

crimson/osd: implement readable/lease related methods

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30644 from majianpeng/osd-remove-unused-func

osd: remove unused function

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30632 from xiexingguo/wip-kick-recovery-pri

osd/PrimaryLogPG: always use strict priority ordering for kicked recovery ops

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30642 from Rjerk/wip-fix-doc

doc/radosgw: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: add closing ")" in log

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: implement schedule_renew_lease()

see also d883db70282c406edaceb72a90ecddccd97a0161

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement ShardServices::get_hb_stamps()

PeeringState::proc_lease() requires a valid hb_stamps[0], which is in
turned maintained by `pl`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: implement queue_check_readable() and recheck_readable()

see also 7aec060e0aff4588ff51f744dd194d1c0f7793c2

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_time: do not handle negative case

this change partially reverts 353a0e5f, unlike `signedspan`,
`timespan::rep` is unsigned. so no need to handle negative case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Bucket names cannot be formatted as IP address

In general, bucket names should follow domain name constraints:
- Bucket names must be unique.
- Bucket names cannot be formatted as IP address.
- Bucket names can be between 3 and 63 characters long.
- Bucket names must not contain uppercase characters or underscores.
- Bucket names must start with a lowercase letter or number.
- Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.

Fixes: https://tracker.ceph.com/issues/42069
Signed-off-by: Volker Theile <vtheile@suse.com>

osd: remove unused function

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #30396 into master

* refs/pull/30396/head:
doc: document mds journaling

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30463 into master

* refs/pull/30463/head:
vstart_runner: allow the use of it with kernel mounts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/radosgw: fix typo

Signed-off-by: Liu Lan <liulan@umcloud.com>

mgr/progress: estimated remaining time for events

Fixes: https://tracker.ceph.com/issues/40419
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30638 from uweigand/cmake-atomic-16byte

cmake: Test for 16-byte atomic support on IBM Z

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Test for 16-byte atomic support on IBM Z

On IBM Z the Boost tagged pointer implementation cannot use
"pointer compression" as there are no unused bits in an address;
the whole 64-bit address space is available to user space code.

Instead, Boost uses 16-byte atomics.  This is always supported
on IBM Z, but depending on the particular compiler (version)
it may require linking against libatomic.  The existing checks
in CheckCxxAtomic.cmake do not catch this, however, as they only
test for (up to) 8-byte atomic support.

Fixed by adding a test for 16-byte atomic support on IBM Z.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge pull request #30617 from xiexingguo/wip-weird-daemon-key

mgr: fix weird health-alert daemon key

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: fix weird health-alert daemon key

Was:
  19 slow ops, oldest one blocked for 34 sec, daemons [osd,2,osd,4] have slow ops.

Now:
  153 slow ops, oldest one blocked for 38 sec, daemons [osd.3,osd.4,osd.5] have slow ops.

Fixes: https://tracker.ceph.com/issues/42079
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29236 from liewegas/wip-read-hole-bypg

osd: implement per-pg leases to avoid stale reads

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: always use strict priority ordering for kicked recovery ops

See 590c4979d40ea2073da8180140303ac067d87cc4 for the background of
this change.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30527 from uweigand/bloom-filter-endian

common/bloom_filter: Fix endian issues

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30616 from ifed01/wip-ifed-fix-fsck

os/bluestore: fix invalid stray shared blob detection in fsck.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30625 from tchaikov/wip-42087

mgr/BaseMgrStandbyModule: drop GIL in ceph_get_module_option()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "osd: give recovery ops initialized by client op a higher priority"

This reverts commit c0f87e0f91539d4d12dad5bddf95a97457a21c67.

The 'osd_op_queue_cut_off' config option determines which level of
high priority ops should use strict priority ordering and may change
from time to time. Since the main strategy of 'osd_kick_recovery_op_priority'
is to simply follow up 'osd_op_queue_cut_off', we can instead make a direct
use of 'osd_op_queue_cut_off' to achieve the same thing explicitly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-dencoder: add RGWPeriodLatestEpochInfo support.

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

crimson/osd: fix osdpg build

These are stubs; the reschule one (at minimum) probably needs a meaningful
implementation in order for the PG to peer in some cases.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document new 'laggy' and 'wait' pg states

Signed-off-by: Sage Weil <sage@redhat.com>

doc: stale reads notes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: skip wait state if osd set is empty

If there are no down OSDs from prior intervals, then the normal peering
process will end up contacting all of the prior OSDs and ensuring that
their prior interval is terminated during peering.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: make proc_lease, recalc_readable_until more verbose

Signed-off-by: Sage Weil <sage@redhat.com>

osd: is_replica() -> is_nonprimary()

The 'replica' term does not map well onto EC pools.  More importantly,
the implementation is often wrong for EC pools, where role may be 0 or 1
for EC pools independent of whether the OSD is the primary or not.

Introduce 'nonprimary' to mean an acting osd that is not the primary.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: piggyback pg_lease on MOSDPGLog

We only do this for primary -> replica, so we only need to proc_lease()
from the replica states.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: piggyback lease and ack on activation messages

The lease goes out with the MOSDPGLog or info, and the ack comes back with
the info.

We no longer need to renew the lease explicitly in
all_activated_and_committed() because we *just* piggybacked on activation.
We can just wait for the normal renew event to fire.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: renew before activate messages; send after activated

We want to renew before we prepeare or send activate messages so that we
have the opportunity to include leases in them (coming soon!).

And we do not want to send explicit lease messages until we know that the
peers have activate.  In particular, we want to avoid queueing a notify
(via pending_activators) and then sending a lease that will arrive before
it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: notice 'dead' prior_readable OSDs

If we see that a prior_readable_down_osd is known to be dead, we can
remove it from the set.  And if the set is empty, we can skip the rest of
our waiting period and leave the WAIT state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: track down OSDs relevant to prior_readable_until_ub

Keep track of which OSDs from the prior set we care about that affect
the prior_readable_until_ub.  Note that it is only the *down* OSDs that
we have to track here, since everything in the *probe* set we will already
contact during peering (they are still up), guaranteeing that those PGs
are aware of the interval change and are no longer readable in the prior
interval.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsg/osd_stale_reads.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

do_cmake.sh: disable amqp and rdma on RHEL/CentOS 8

RHEL/CentOS 8 does not offer librabbitmq-devel, libibverbs-devel and
librdmacm-devel at this moment. so disable these features now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: set WITH_RADOSGW_AMQP_ENDPOINT using ARGS

so we can extend ARGS in a more unified way

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: disable "make check" for rhel8

following packages are not available on RHEL/CentOS 8, they are for
dashboard unit tests. so disable all python packages for dashboard
tests for now:

python-cherrypy
python-coverage
python-pecan
python-routes
python-tox
xmlsec1-devel

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: disable cephfs_java and amqp_endpoint if RHEL8

RHEL/CentOS 8 does not offer junit and sharutils yet, so disable
cephfs_java on rhel8 at this moment.

RHEL/CentOS 8 does not offer librabbitmq-devel yet, so disable
amqp_endpoint on rhel8 at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: only install python-srpm-macros for required macros

the reason why we need to install these macros is to solve the
egg-chicken problem -- to set `_python_buildid` and `python3_pkgversion`
so that we can prepare the build dependencies and install them. in which,
`_python_buildid` is defined using `python3_pkgversion`. this macro is
offered by python-srpm-macros.

the other macros, like `python3_sitelib` and `__python3` are offered by
`python3-rpm-macros`.

this change also avoid the issue if we install `*rpm-macros` on CentOS8:

Error:
 Problem: package R-rpm-macros-1.1.0-2.el8.noarch requires /usr/bin/Rscript, but none of the providers can be installed
  - package R-rpm-macros-1.1.0-2.el8.noarch requires R-core, but none of the providers can be installed
  - conflicting requests
  - nothing provides libRblas.so()(64bit) needed by R-core-3.6.1-1.el8.x86_64
  - nothing provides openblas-Rblas needed by R-core-3.6.1-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30441 from anthonyeleven/master

osd: Change osd op queue cut off default to high

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30581 from liewegas/wip-write-returns

osd: allow rados write ops to return data and error codes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30454 from trociny/wip-41891

osd: fix possible crash on sending dynamic perf stats report

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30544 from tchaikov/wip-check-getentropy-at-runtime

auth/Crypto: fallback to /dev/urandom if getentropy() fails

Reviewed-by: Chang Liu <liuchang0812@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/BaseMgrStandbyModule: drop GIL in ceph_get_module_option()

always drop GIL when performing block ops

Fixes: https://tracker.ceph.com/issues/42087
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/balancer: add per pool concurrent optimization limit

the main point is that in general we have physical isolated storage
pools in our products, so the cluster-wide target_max_misplaced_ratio
option would not make much sense in such case.
Introducing a fine-grained alternate throttling mechanism instead make
things much better now!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/balancer: upmap_max_iterations -> upmap_max_optimizations

With osd_calc_pg_upmaps_aggressively on we might have to iterate
hundreds or thousands of times to figure out an optimization,
but upmap_max_optimizations will remain a hard limit for the
total optimizations that can be returned.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

fio_ceph_objectstore: add option to cycle throttle values during test

Signed-off-by: Samuel Just <sjust@redhat.com>

fio_ceph_objectstore: add check_files option to validate object existence/size

Signed-off-by: Samuel Just <sjust@redhat.com>

bluestore: revamp tracepoints, add sampling

This patch adds per-io bluestore specific tracepoints detailing the
throttle state at queue time as well as state latencies during
execution.  Additionally, bluestore_throttle_trace_rate will limit
the rate at which ios have tracepoints emitted.

Signed-off-by: Samuel Just <sjust@redhat.com>

bluestore: consolidate state transitions and throttles into BlueStoreThrottle

This will make it easier to have a single place to emit trace information.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/kv: add support for fetching implementation specific properties

Only implemented for rocksdb for now.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/Throttle: maintain count even if disabled

Otherwise, it will behave incorrectly if enabled dynamically.

Signed-off-by: Samuel Just <sjust@redhat.com>

msg/async: fix timestamp members specific to EVENTTRACE

Signed-off-by: Samuel Just <sjust@redhat.com>

common/EventTrace: default to log level 30 for text logging

By nature, these tend to be extremely high volume.

Signed-off-by: Samuel Just <sjust@redhat.com>

*/CMakeLists: add EVENTTRACE as cmakedefine, require LTTNG

Because EVENTTRACE now implies LTTNG, also cleanup EVENTTRACE
conditions to assume LTTNG.

Also add missing eventtrace dependencies to rbd and
test/objectstore.

Signed-off-by: Samuel Just <sjust@redhat.com>

fio_ceph_objectstore: clean up formatted output, add option to write to file

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #30526 from uweigand/endian-private

include,os: Make ceph_le member private

Reviewed-by: Kefu Chai <kchai@redhat.com>