cephadm: make /sys/fs/selinux empty

When the following conditions are true:

  1) A host has selinux-policy-targeted,
  2) We mount the host's /sys into a privileged container,
  3) The container has SELINUXTYPE=targeted in /etc/selinux/config,
  4) The container does not have an selinux-policy-targeted package,

then SELinux-enabled applications like restorecon or DNF do not work inside
the container.

Resolve this by making /sys/fs/selinux an empty directory.

Fixes: https://tracker.ceph.com/issues/49239
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #38963 from rhcs-dashboard/refresh-crushmap

mgr/dashboard: Automatically refresh crush map metadata table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39355 from sebastian-philipp/doc-install-rm-deepsea

doc/install: cephadm is the successor of DeepSea

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38505 from sebastian-philipp/doc-dev-cephadm-scalability-notes

doc/dev/cephadm: scalability notes

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39287 from david-z/wip-fix-join-fscid

mon/MDSMonitor.cc: fix join fscid not applied with pending fsmap at boot

Merge PR #38318 into master

* refs/pull/38318/head:
common: Add Windows ETW support

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39387 from xxhdx1985126/wip-client-request-pg-ref

crimson/osd: capture pgref by value if it is to be referenced asynchronously

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39388 from ivzhh/master

cmake: don't pass ccache argument to RocksDB build

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: don't pass ccache argument to RocksDB build

RocksDB's cmake has a similar logic to turn on ccache as the main cmake.
Currently BuildRocksDB.cmake explicitly passes ccache to RocksDB cmake
arguments as CMAKE_CXX_COMPILER_LAUNCHER. This causes a compilation error
on ccache 4.0, especially in make -j mode.

ccache: error: Recursive invocation (the name of the ccache binary must
be "ccache")

This commit lets cmake from RocksDB decide RULE_LAUNCH_COMPILE by itself.

Signed-off-by: Sheng Mao <shngmao@gmail.com>

crimson/osd: capture pgref by value if it is to be referenced asynchronously

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39351 from zdover23/wip-doc-dev-getting-binaries-rewrite-08-Feb-2021

doc/dev: rewrite "getting binaries" section

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39383 from badone/wip-unmask-deep-scrub-stdout-lazy_omap_stats_test

test/lazy-omap-stats: Immediate deep-scrub output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39333 from cyx1231st/wip-seastore-onode-tree-empty-oid-ns

crimson/onode-staged-tree: support empty ns and oid

Reviewed-by: Samuel Just <sjust@redhat.com>

test/lazy-omap-stats: Immediate deep-scrub output

Show stdout from the child process as it happens.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38871 from cbodley/wip-qa-rgw-valgrind-on

qa/rgw: reenable valgrind in rgw/verify suite

Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc/dev: rewrite "getting binaries" section

This rewrites the "Getting Binaries" section
of the "Workflow" section of the "Tests:
Integration Tests" section of the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa/rgw: test_rgw_reshard.py just tries common ports

the 'sudo netstat | grep radosgw' command is causing failures:

  File "qa/workunits/rgw/test_rgw_reshard.py", line 53, in get_radosgw_port
    x = out.decode('utf8').split(" ")
AttributeError: 'bool' object has no attribute 'decode'

because when valgrind is enabled, the process name is 'valgrind' instead
of 'radosgw'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: re-enable valgrind in rgw verify suite

this only enables it for radosgw itself, not the other daemons

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: move valgrind config into valgrind.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Automatically refresh the crush map metadata table

If we make any change to the osd crush map like do an osd crush reweight from cli, for that change to be reflected on metadata table we need to reload the entire page. Instead this PR takes care of auto refreshing the tree view.

Fixes: https://tracker.ceph.com/issues/48922
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #39211 from cbodley/wip-rgw-admin-formatter

radosgw-admin: fix leaks with make_unique()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39301 from cbodley/wip-49169

rgw: fix snprintf() truncate warning in ETagVerifier

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

doc/dev/cephadm: scalability notes

Add a few notes about possible future scalability improvements

Co-authored-by: Joshua Schmid <jschmid@suse.de>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/install: cephadm is the successor of DeepSea

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

common: Add Windows ETW support

This patch adds generic ETW logging on top of syslog calls.

The events are described in the message compiler file `event_logging.mc`.

Using the cross platform utility `windmc`, we will generate the header file,
`event_logging.h`, needed by the ETW implementation, and the resource file
`event_logging.rc`.

Over the generated resource file we will run another utility called `windres`.
this tool is used to compile the binary objects needed by the OS utilities
(i.e. `Event Viewer`) to view the logged events.

On usage, a registry key needs to be added/removed on the target computer.
The registry looks like the following:
[HKLM\SYSTEM\CurrentControlSet\Services\EventLog\Application\<get_process_name>]
"EventMessageFile"="<Folder_location_to>\\event_logging.dll"

I.E.:
[HKLM\SYSTEM\CurrentControlSet\Services\EventLog\Application\rbd-wnbd]
"EventMessageFile"="C:\\Program Files\\Ceph\\bin\\event_logging.dll"

Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Co-authored-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #39345 from tchaikov/wip-crimson-move

crimson/osd: do not pass oop as a const ref and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39353 from lxbsz/minor

client: use scoped_lock instead of unique_lock

crimson/onode-staged-tree: be explicit about string_key markers

To be explicit that string_kew_view_t::MIN and MAX are not valid
string lengths. They are reserved as markers.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #39367 from runsisi/wip-fix-spec

ceph.spec.in: fix missing backslash for line continuation

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: support empty ns and oid

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os: disable code block only executed by Debug build

silences warning like:

warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
  ).safe_then([this, target_size] {
               ^~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: capture labels using new variables

structured binding does not define variables, let's capture the bound
"variables" using variables defined in the capture list.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: mark ExitBarrier final

to silence -Wfinal-dtor-non-final-class warnings like

operation.h:461:20: warning: class with destructor marked 'final' cannot
be inherited from [-Wfinal-dtor-non-final-class]
    ~ExitBarrier() final {
                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor ClientRequest::do_process()

instead of using a local lambda which tries to have a single point
for handling both the happy path and failure path, it'd be simpler
to just early return if we fail to handle a request,

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not pass oop as a const ref

before this change, std::move(oop) will create a copy of oop.
after this change, std::move(oop) actually moves oop.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not pass ops to submit_transaction() separately

the order of evaluation of function arguments is unspecified with some
exceptions. but in this case, the reason why ops is still valid is that
the parameter type of `osd_op_p` is `const osd_op_params_t&`. so a copy
is passed to it.

to enable us to actually move `osd_op_p`, in this change, the
osd_op_p.req->ops is used instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract methods out of ClientRequest::process_op()

* do_recover_missing()
* do_process()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not move op_params_p in fill_op_params_bump_pg_version()

it's confusing to move the parameter for updating it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not add parentheses unless necessary

do not use the notation of

[]() -> return_type {

unless it helps with readability or we have to do so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove no-op func passed to OpsExecuter::flush_changes()

no need to create an empty func for an request which creates no
transactions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: rename PG::rep_repair_primary_object() to repair_object()

as we should expose the internals using the name of interface, and there
is no need to differentiate rep_repair_primary_object() from another
repair method.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39361 from sebastian-philipp/orch-fix-flake8

mgr/orch: Fix flake8 error

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix missing backslash for line continuation

it errors while building without lua_packages:

+ -DBOOST_J=102 -DWITH_GRAFANA=ON
/var/tmp/rpm-tmp.aq2X3J: line 91: -DBOOST_J=102: command not found

Signed-off-by: luo.runbing <luo.runbing@zte.com.cn>

Merge pull request #39332 from cyx1231st/wip-seastore-range-query

crimson/onode-staged-tree: implement Cursor::get_next() and comparators for range query

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/orch: Fix flake8 error

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37729 from mgfritch/cephadm-podman-logdriver

cephadm: configure journald as the logdriver

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #37764 from mgfritch/cephadm-no-container-init

cephadm: run containers using `--init` by default

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39153 from mgfritch/cephadm-flake8

mgr/cephadm: introduce flake8

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39331 from ceph/revert-37954-wip-F46226-log-dq-throttle-limit-new

Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39359 from tchaikov/wip-crimson-clang-build

crimson/os: fix build with clang

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: capture labels using a new variable

structured-binding does not define variables, so we cannot use "[=]" to
implicitly capture them. define variables in the capture list instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark unused variable [[maybe_unused]]

to silence warning like

key_layout.h:516:10: warning: unused variable 'nspace_masked'
[-Wunused-variable]
    auto nspace_masked = string_view_masked_t::decode(nspace, delta);
         ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not define assignment operator for Value

we cannot assign a reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: mark ExitBarrier final

to silence warnings like:

crimson/common/operation.h:394:20: warning: class with destructor marked
'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
    ~ExitBarrier() final {
                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: use scoped_lock instead of unique_lock

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39350 from tchaikov/wip-cephadm-unicode

cephadm: do not cast subnet to unicode

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39266 from sebastian-philipp/cephadm-rm-keyring-on-failed-deploy

mgr/cephadm: cleanup daemon keyring on failed daemon deploy

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38695 from pcuzner/orch-status-changes

mgr/orchestrator: minor improvements to orch status

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39347 from xxhdx1985126/wip-crimson-pg-bug-fix

crimson/osd: make sure ox be removed after all continuations are finished in PG::do_osd_ops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: do not cast subnet to unicode

this change addresses a regression introduced by
fe4f4402fbcd87667613640f2808d5d0e07e749d, which was tested before
abd9287db0e4f4f7873864119f5ce62519af1d48 got merged. and the former was
merged after abd9287db0e4f4f7873864119f5ce62519af1d48. hence the
regression.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make sure ox be removed after all continuations are finished in PG::do_osd_ops

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39193 from tchaikov/wip-cephadm-py3

cephadm: drop support to python<3.3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38897 from Daniel-Pivonka/mgroktostop

mgr/cephadm: ok-to-stop for mgr

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39082 from ifed01/wip-ifed-meta-in-objectstore-tool

tools/ceph-objectstore-tool: allow object operations on meta pgid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39273 from tchaikov/wip-timer-mono

common/Timer: use mono_clock for clock_t

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39295 from tchaikov/wip-mgr-balancer-ann

pybind/mgr/balancer: add type annotations

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #39164 from tchaikov/wip-crimson-push-mov

crimson/osd: consume the push payload

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39204 from TRYTOBE8TME/wip-rgw-create-bucket-dpp

src/rgw: Dpp addition to create bucket log

rgw: Dpp addition to create bucket log

This commit adds dpp to the create bucket log.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

pybind/mgr/balancer: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define Plan.{dump,show}()

as they are called by the commands

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/alerts: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define options using Option

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: add type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define commands using CLICommand

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39336 from AlexWangD/patch-4

doc: fix the spelling mistake of “Topicd”

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the spelling mistake of “Topicd”

“Topicd” is a spelling mistake of "Topics".

Merge pull request #37566 from AmnonHanuhov/wip-ObjectStore_EIO_Handling

crimson/osd: Recover missing object on read error

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: cleanup -- switch to get_*_slot() interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #39303 from dillaman/wip-49173

librbd/io: conditionally disable move optimization

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39312 from ivancich/wip-orphan-list-binary-data

rgw: allow rgw-orphan-list to handle intermediate files w/ binary data

Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

mgr/cephadm: Redefinition of unused name from line n (F811)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Module imported but unused (F401)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Local variable name is assigned to but never used (F841)

add error handling when `CompletedProcess` fails during rgw relam,
zongroup create et al.

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Local variable name is assigned to but never used (F841)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Line break occurred after a binary operator (W504)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Trailing whitespace (W291)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: undefined name 'channel' (F821)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use ==/!= to compare constant literals (F632)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: f-string is missing placeholders (F541)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not use variables named 'I', 'O', or 'l' (E741)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not assign a lambda expression, use a def (E731)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not use bare except, specify exception instead (E722)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Comparison to true should be 'if cond is true:' or 'if cond:' (E712)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Multiple statements on one line (def) (E704)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: statement ends with a semicolon (E703)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Expected 1 blank line before a nested definition (E306)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Expected 2 blank lines, found 0 (E302)

Signed-off-by: Michael Fritch <mfritch@suse.com>