qa/suites/rados/singleton/all/admin-socket: fix test

We can't set the filestore setting because filestore isn't active and so
the option isn't observed, so it isn't changeable.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: EPERM setting config option after startup

Not sure why ENOSYS was chosen before...

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: fix tell output error check

It goes to stderr instead of stdout now.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: pass Formatter from generic infrastructure

The implementation can choose to either use the provided Formatter, or
put something directly into outbl.  The implementation may choose to
flush the formatter to the output buffer|stream, or let the caller do it
for them (usually the latter).

Lots of fiddling/cleanup in the implementations to make this build,
including dropping the (seeminlyg unused?) ostream& output mode for
the librbd asok implementations.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: pass ostream to call() for error output

This separates the error stream from the output stream for the synchronous
hook.

This patch includes misc cleanup in the various implementations to make
use of the new stream for errors.

Add a test to unittest_context to ensure we're getting the error stream.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix asok hook return value

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: fix asok return value

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: return error code from asok commands

Several of these commands (config set, injectargs, etc.) can return errors.
Do so.

Signed-off-by: Sage Weil <sage@redhat.com>

test/pybind/test_rados: fix accidental mon tell test

This test assumed 'mon dump' (a CLI command) would work when targetted at
a specific mon (i.e., a tell command).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: print entity_name along with caps to debug log

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: notes about asok changes

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: empty target string for 'tell' means active mgr

This lets the caller say "tell the active mgr", whoever it may be.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: report error code as part of output string

This isn't ideal, but it's better than nothing.

Net behavior change here is that calling an non-existant command now
succeeds with an error message instead of exiting with an error code
and no informative output.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change trigger_[deep_]scrub tommands to a pg tell command

This is cleaner.  All users are currently standalone tests; updated.

It also means that *all* commands that have a name=pgid arg are pg tell
commands.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove old command workqueue, threadpool

These are now sent to the asok handler.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop MMonCommand handling

Nothing sends these.. or has in a very very long time, AFAICS.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: resend OSD tell commands on EAGAIN

Request map *and* resend.  We don't have map epoch info about when the
reply was sent, and the OSD isn't ordering with respect to epochs anyway.
So, resend now, just in case we already saw a map change, or because we
were suffering from a peering vs command race on the OSD side, and then
also request a new map from the mon, in case we are missing a map update.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: route tell commands to asok; migrate commands

- move items from _do_command to asok_command in OSD.cc
- update PG::do_command to take a std::function on_finish
- sprinkle in some osd_lock locking (_do_command implicitly locks osd_lock,
  asok_command() does not; most commands don't need it)

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use unique_ptr<Formatter> for asok_command

Simpify lifecycle.

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: add generic asok 'injectargs'

Lots of scripts and users expect this, even though 'config set' is
generally nicer to use.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: allow dup prefixes

Allow multiple commands to register with the same prefix (but different
commands description strings).

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: refactor with sync and async execute_command variants

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: pass input bufferlist

Pass this to the async handler only for now, since the sync implementations
don't currently use it.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: transition to call_async() for asok

And some variable renames, error path fixup.

No other significant functional change.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: support alternative call_async()

Some tell commands reply asynchronously.  Support that through the generic
infrastructure.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: send tell commands out of band via MCommand

The current tell mon command handling is pretty fragile and semi-broken:
we force the client to (exclusively) connect to the target mon, disrupting
other monclient business, and the retry logic is fragile.

Instead, use entirely independent connections for each tell command, and
tear them down when we get a reply. Implement independent and simple
error handling and timeouts.

Keep most of the old behavior alive so that we can still use tell against
pre-octopus mons.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: accept tell commands via MCommand and send them to asok handler

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: return int from hook call()

Previously, call() returned a bool.  Return an int instead so we can
wire this up to tell command return values.

The admin socket 'ceph daemon ...' unix domain socket protocol does not
pass a return code, only data, so we cannot pass these errors that way.
We have two choices: make error codes silently succeed when accessed via
asok (so that we get an error string etc), or make them fail without any
specific error code or string.

Unfortunately, there are several cases where an exception was caught and
what() returned as a string, or where error strings are returned.  These
would "blindly" fail if we took the latter approach.

So, for the asok interface, -ENOSYS means a "hard" error that gives the
user no data and makes the 'ceph daemon ...' command return an error code.
Other error codes are interpreted as a success.  This is ONLY for the
asok interface; the tell interface has full fidelity with error codes and
error strings.

Note that this means that 'net new' tell-style commands that we move over
to this handler will also appear to succeed via the 'ceph daemon'
interface when they return error codes.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/DaemonServer: route MCommand (for octopus+) to asok commands

Send mgr 'tell' commands (if they originate from a octopus+ client that
knows the difference between MCommand and MMgrCommand) to the asok
comand queue.

Signed-off-by: Sage Weil <sage@redhat.com>

do not use 'ceph tell mgr'

This is now for a daemon/asok command, not a module command.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/ceph_argparse: disambiguate mgr tell and CLI commands

The mgr tell commands are somewhat special in that you can tell the mgr
with an empty id ('ceph tell mgr' or target ('mgr', '')) to get the
currently active mgr.  This makes it hard to disabiguate between a tell
command and a CLI command.

Fix that by explicitly setting the target to 'mon-mgr' when a CLI command
is flagged as a mgr command.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: make 'ceph tell mgr.*' send to the active mgr

We can only send tell commands to the active mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: send 'ceph tell mgr.X' to the right mgr

Previously this would go to the active mgr, whoever that was.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add rados_mgr_command_target

This variant lets you specify which mgr to talk to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: add start_command variant that takes a target

Note that the initial implementation can only target the active mgr!

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: drop unregister_command(); use per-hook variant

There's never a need to unregister individual commands; doing it by hook
is sufficient for all users.  Simpler and faster.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: drop explicit prefix arg to register_command

This is always the cmddesc prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: simplify command routing

Back in e30e937c8962249af283a7571eb106ef444b79e3 we made it possible to
route a command via any prefix. This worked when we wanted to pass
arguments but were just dealing with a vector<string>.  These days we have
an actual prefix followed by named arguments, so we don't need this
ad hoc routing.

Derive the prefix from the cmddesc at registration time, and match that
explicitly against the prefix at execution time.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: add ability to process MCommand via asok queue

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: pass cmdvec to execute_command

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: use pipe for general wakeup

Use the pipe to wake up the thread.  Use a separate bool to signal a
shutdown.

Signed-off-by: Sage Weil <sage@redhat.com>

include/compat: add flags arg to pipe_cloexec

This matches pipe(2).

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: drop unused args

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28569 from alimaredia/wip-rgw-ioctx-operate-replace

rgw: replace direct calls to ioctx.operate()

Merge pull request #30655 from tchaikov/wip-cmake/crimson/pass-args

cmake: pass arguments to crimson tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30613 from joke-lee/rgw-ceph-dencoder-add-RGWPeriodLatestEpochInfo

ceph-dencoder: add RGWPeriodLatestEpochInfo support.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: pass arguments to crimson tests

otherwise they will take all CPU cores and all memory

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30587 from sebastian-philipp/python-common-readme

python-common: Add small Readme

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #30341 into master

* refs/pull/30341/head:
cephfs-shell: better complain info, when deleting non-empty directory

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #30559 into master

* refs/pull/30559/head:
mds: Reorganize class members in MDBalancer header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30580 into master

* refs/pull/30580/head:
client: don't ceph_abort on bad llseek whence value
client: remove Inode dir_contacts field

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30407 from joscollin/wip-B41841-yes-really-mean-it

mgr/volumes: protection for `fs volume rm` command

Merge PR #30552 into master

* refs/pull/30552/head:
cmake: fix libtsan detection
test: librados startup/shutdown racer test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30630 from tchaikov/wip-build-on-centos8

install-deps,rpm,do_cmake: build on RHEL/CentOS 8

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #30531 from alimaredia/wip-rgw-maven-version-update

qa: bump maven repo version in s3a_hadoop.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30615 from xiexingguo/wip-progress-time-remaining

mgr/progress: estimated remaining time for events

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30591 from xiexingguo/wip-balancer-throttler

mgr/balancer: upmap_max_iterations -> upmap_max_optimizations; behave as it is per pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29674 from athanatos/sjust/wip-fio-trace

os/bluestore: expand lttng tracepoints, improve fio_ceph_objectstore backend

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cls: CLS_CLIET_HIDE_IOCTX changes in cls headers

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: various ioctx.operate refactoring changes

- remove unused #include for cls_client headers
- refactor cls_timeindex_trim call in
rgw_object_expirer_core.cc
- add more CLS_CLIENT_HIDE_IOCTX in header files
- move target_compile_definitions to rgw_a target
- make repeat cls helper functions static

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: overload cls_rgw functions that do io_ctx.operate()

Call the new overloaded functions and call
rgw_rados_operate after them.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cls: remove unused cls_rgw_reshard_get_head()

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: replace ioctx.operate() with rgw_rados_operate()

For all operate calls in:
rgw_op.cc
rgw_orphan.cc
rgw_reshard.cc
rgw_rados.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: fix libtsan detection

This variable is case-sensitive.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30620 from votdev/issue_42069_validate_bucket_name

mgr/dashboard: Bucket names cannot be formatted as IP address

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>

Merge pull request #30647 from s0nea/wip-dashboard-fix-typo

mgr/dashboard: fix small typos in description message

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29925 from sebastian-philipp/run_tox-overwrite-build-dir

script/run_tox.sh: Don't overwrite the build dir

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix two small typos

Remove unnecessary blanks.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

script/run_tox.sh: Don't overwrite the Ceph's bulid dir

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #30639 from tchaikov/wip-crimson-readable

crimson/osd: implement readable/lease related methods

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30644 from majianpeng/osd-remove-unused-func

osd: remove unused function

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30632 from xiexingguo/wip-kick-recovery-pri

osd/PrimaryLogPG: always use strict priority ordering for kicked recovery ops

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30642 from Rjerk/wip-fix-doc

doc/radosgw: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: add closing ")" in log

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: implement schedule_renew_lease()

see also d883db70282c406edaceb72a90ecddccd97a0161

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement ShardServices::get_hb_stamps()

PeeringState::proc_lease() requires a valid hb_stamps[0], which is in
turned maintained by `pl`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg: implement queue_check_readable() and recheck_readable()

see also 7aec060e0aff4588ff51f744dd194d1c0f7793c2

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_time: do not handle negative case

this change partially reverts 353a0e5f, unlike `signedspan`,
`timespan::rep` is unsigned. so no need to handle negative case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Bucket names cannot be formatted as IP address

In general, bucket names should follow domain name constraints:
- Bucket names must be unique.
- Bucket names cannot be formatted as IP address.
- Bucket names can be between 3 and 63 characters long.
- Bucket names must not contain uppercase characters or underscores.
- Bucket names must start with a lowercase letter or number.
- Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number.

Fixes: https://tracker.ceph.com/issues/42069
Signed-off-by: Volker Theile <vtheile@suse.com>

osd: remove unused function

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #30396 into master

* refs/pull/30396/head:
doc: document mds journaling

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30463 into master

* refs/pull/30463/head:
vstart_runner: allow the use of it with kernel mounts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/radosgw: fix typo

Signed-off-by: Liu Lan <liulan@umcloud.com>

mgr/progress: estimated remaining time for events

Fixes: https://tracker.ceph.com/issues/40419
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30638 from uweigand/cmake-atomic-16byte

cmake: Test for 16-byte atomic support on IBM Z

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Test for 16-byte atomic support on IBM Z

On IBM Z the Boost tagged pointer implementation cannot use
"pointer compression" as there are no unused bits in an address;
the whole 64-bit address space is available to user space code.

Instead, Boost uses 16-byte atomics.  This is always supported
on IBM Z, but depending on the particular compiler (version)
it may require linking against libatomic.  The existing checks
in CheckCxxAtomic.cmake do not catch this, however, as they only
test for (up to) 8-byte atomic support.

Fixed by adding a test for 16-byte atomic support on IBM Z.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge pull request #30617 from xiexingguo/wip-weird-daemon-key

mgr: fix weird health-alert daemon key

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: fix weird health-alert daemon key

Was:
  19 slow ops, oldest one blocked for 34 sec, daemons [osd,2,osd,4] have slow ops.

Now:
  153 slow ops, oldest one blocked for 38 sec, daemons [osd.3,osd.4,osd.5] have slow ops.

Fixes: https://tracker.ceph.com/issues/42079
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29236 from liewegas/wip-read-hole-bypg

osd: implement per-pg leases to avoid stale reads

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: always use strict priority ordering for kicked recovery ops

See 590c4979d40ea2073da8180140303ac067d87cc4 for the background of
this change.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30527 from uweigand/bloom-filter-endian

common/bloom_filter: Fix endian issues

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30616 from ifed01/wip-ifed-fix-fsck

os/bluestore: fix invalid stray shared blob detection in fsck.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30625 from tchaikov/wip-42087

mgr/BaseMgrStandbyModule: drop GIL in ceph_get_module_option()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Revert "osd: give recovery ops initialized by client op a higher priority"

This reverts commit c0f87e0f91539d4d12dad5bddf95a97457a21c67.

The 'osd_op_queue_cut_off' config option determines which level of
high priority ops should use strict priority ordering and may change
from time to time. Since the main strategy of 'osd_kick_recovery_op_priority'
is to simply follow up 'osd_op_queue_cut_off', we can instead make a direct
use of 'osd_op_queue_cut_off' to achieve the same thing explicitly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-dencoder: add RGWPeriodLatestEpochInfo support.

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

crimson/osd: fix osdpg build

These are stubs; the reschule one (at minimum) probably needs a meaningful
implementation in order for the PG to peer in some cases.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document new 'laggy' and 'wait' pg states

Signed-off-by: Sage Weil <sage@redhat.com>

doc: stale reads notes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: skip wait state if osd set is empty

If there are no down OSDs from prior intervals, then the normal peering
process will end up contacting all of the prior OSDs and ensuring that
their prior interval is terminated during peering.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: make proc_lease, recalc_readable_until more verbose

Signed-off-by: Sage Weil <sage@redhat.com>