doc/cephfs/nfs: add user id, fs name and key to FSAL block

Fixes: https://tracker.ceph.com/issues/50161
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #40582 from a16bitsysop/32bit

src/common/buffer.cc: change cast to static_cast

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40578 from tchaikov/wip-cmake-pmem

cmake: require libpmem 1.7 and cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #40572 from athanatos/sjust/wip-48613

osd/PeeringState: fix acting_set_writeable min_size check

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40475 from tchaikov/wip-qa-focal

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #40591 from tchaikov/wip-dencoder

tools/ceph-dencoder: link against libtcmalloc

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Sage Weil <sage@redhat.com>

osd/PeeringState: fix acting_set_writeable min_size check

acting.size() >= pool.info.min_size is meant to check min_size against
acting set participants, but acting is a vector with placeholders.
actingset is the representation with placeholders removed.

The upshot of this bug is that the activation process will basically
ignore min_size for an ec pool allowing writes in cases where it
shouldn't.  PastIntervals::check_new_interval, however, performs
the check correctly, and will therefore discount intervals in which
we really did serve writes as not writeable.  This can trigger many
different problem conditions including but not limited to:
  - Unfound objects due to accepting a last_update with insufficient
    osds
  - Lost writes
  - Crashes due to peering rules being violated

This bug was originally introduced with recovery below min_size in
e5a96fd, and then preserved through refactors in 749a13d and 95bec9.

7cb818a exposed it with with expansion of recovery below min_size
to include ec pools (acting.size() is sufficient for replicated
pools).

Fixes: https://tracker.ceph.com/issues/48613
Fixes: https://tracker.ceph.com/issues/48417
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #40599 into master

* refs/pull/40599/head:
rpm: add missing % in %dir directive

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40598 from TRYTOBE8TME/wip-rgw-kafka-tests-fix

src/rgw: Issue #50138 fix

Merge PR #40025 into master

* refs/pull/40025/head:
cephadm: use split cgroup strategy for podman
cephadm: use class to represent container engine

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40459 into master

* refs/pull/40459/head:
mgr/cephadm: don't cleanup the daemon keyring on failed redeploy

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40508 into master

* refs/pull/40508/head:
mgr/cephadm: fix orch host add with multiple labels and no addr

Reviewed-by: Sage Weil <sage@redhat.com>

rgw/test: stop kafka consumer if tests fails

This PR fixes https://tracker.ceph.com/issues/50138 by stopping the kafka consumer if the test fails.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rpm: add missing % in %dir directive

de6c8250a6d91403e6d334aeb901bf9720ba40eb added an explicit %dir directive for
a new directory added to the ceph-common package, but -- due to a typo --
neglected to include the "%". As a result, RPM builds started to fail with:

Processing files: ceph-common-17.0.0-2787.gde6c8250.el8.x86_64
error: File must begin with "/": {_libdir}/ceph/denc/

RPM build errors:
    File must begin with "/": {_libdir}/ceph/denc/

Fixes: de6c8250a6d91403e6d334aeb901bf9720ba40eb
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #40584 from smithfarm/wip-denc-fix

rpm: make ceph-common own the new denc directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40585 from yuvalif/wip-yuval-fix-50135

rgw/multisite: handle case when empty marker is provided

Merge pull request #39738 from SMIL-Infra/journald

common: enable log to journald

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40579 from tchaikov/wip-mgr-typing

pybind/mgr/mgr_util: fix typing annotation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40500 into master

* refs/pull/40500/head:
doc/cephadm: remove keepalived_user from haproxy docs

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

cmake: dedup components when finding pmem

otherwise we always find libpmem twice if libpmem is explicitly
specified when find_package()

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: find pmem once instead twice

less noise in cmake output

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph-dencoder: print human-readable error message

if $CEPH_LIB is not specified.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph-dencoder: link against libtcmalloc

to address the failure like

src/tcmalloc.cc:332] Attempt to free invalid pointer 0x5627da496130
Aborted

we need to link and load libtcmalloc before it is loaded by a shared
library, otherwise the memory chunk allocated by libc will be freed
by tcmalloc after it is loaded.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/multisite: handle case when empty marker is provided

marker is potional, however, it may also be provided empty

Fixes: https://tracker.ceph.com/issues/50135
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

resurrect 835f2f5f511e7363f4056e5137382982aac1bfbf, which was reverted by
1f885b6c2fdde1f3ab249a56f7b4b763556f4373

so all tests for `pacific` will be tested on the latest ubuntu version

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: make ceph-common own the new denc directory

2d3c6561b4ac1473a728e81c232d7dfe6fc0188c introduced a new library directory
"%{_libdir}/ceph/denc/" in ceph-common but did not explicitly state that it
should be owned by the package. This caused OBS builds to fail as follows:

[ 5515s] ceph-common-17.0.0-2786.1.x86_64.rpm: directories not owned by a package:
[ 5515s]  - /usr/lib64/ceph/denc

Fixes: 2d3c6561b4ac1473a728e81c232d7dfe6fc0188c
Signed-off-by: Nathan Cutler <ncutler@suse.com>

src/common/buffer.cc: change cast to static_cast in ceph::buffer::list::page_aligned_appender::_refill

part of fixing compilation on 32bit platforms

Signed-off-by: Duncan Bellamy <dunk@denkimushi.com>

pybind/mgr/mgr_util: fix typing annotation

and refactor lock_timeout_log() a little bit to drop `locked`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: require libpmem 1.7

detect the libpmem libraries' version using pkg-config

please note, the version is not identical to the ones defined in
libpmem.h or libpmemobj/base.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: consolidate the find logic in Findpmem.cmake

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40124 from adk3798/format-help

pybind/ceph_argparse: print --format flag name in help descs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #40316 from batrick/i49605

pybind/mgr/volumes: avoid deadlock in ceph-mgr Finisher thread

Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #40555 from jecluis/wip-cephadm-fix-apparmor.master

cephadm: ignore apparmor if profiles file is empty

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38500 from pdvian/wip-print-weight

crush/CrushCompiler: print weight with uniform precision

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38258 from badone/wip-tracker-48336-monmaptool-enforce-ip-better

monmaptool: Don't call set_port on an invalid address

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40518 from Rethan/fix-client-message-throttle-max

osd: allow osd_client_message_size throttle set to zero

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40444 from TheJJ/ensure-fsck-transactions

os/bluestore: ensure fsck repair transactions succeed

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #40457 from lihuibng/crush-bug

crush/CrushLocation: do not print logging message in constructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40488 from lxbsz/typo

mgr: fix the desc typo for mds PyCFunction methods

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40168 from amathuria/wip-amathuria-fix-mon-trimming

mon: Modifying trim logic to change paxos_service_trim_max dynamically

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40498 from guits/guits-quick_fix

cephadm: fix a typo

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40538 from tchaikov/wip-msg-type-cleanup

msg/msg_types: let ceph_entity_name::parse() accept string_view

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #40556 from liewegas/mon-no-config-indent

mon/ConfigMonitor: do not indent 'config dump' output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40574 from badone/wip-submanager-contatiner-failure-tracker-50118

install-deps: Use dnf config-manager

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: use .pc for hints for finding pmem libraries

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps: Use dnf config-manager

Using subscription-manager will fail in a container so use dnf
config-manager which should work on bare metal as well as in a
container.

Fixes: https://tracker.ceph.com/issues/50118
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

msg/msg_types: remove entity_name_t::parse(const char*...)

it can be replaced with entity_name_t::parse(string_view)

also refactor entity_name_t::parse(string_view) a little bit, to
embed the logic of `entity_name_t::parse(const char*...)` in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40435 from tchaikov/wip-48893

common/pick_address: Allow binding on loopback iface

Reviewed-by: Sage Weil <sage@redhat.com>

common/pick_address: Allow binding on loopback iface

in 6147c0917157efd2d35610e759685656a4989abb, "lo" is also skipped when
daemon is trying to find an address to bind. but that change reverts the
fix of 201b59204374ebdab91bb554b986577a97b19c36, to address the problem.

an option named "ms_bind_exclude_lo_iface" is added, it defaults to
"true". but it can be changed to false to allow daemon to bind on "lo".

Fixes: https://tracker.ceph.com/issues/50012
Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: pass string by reference

to silence warnings from clang-tidy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: use scope_guard for freeifaddrs()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_addr: refactor pick_address.cc and ipaddr.cc

* do not replicate the same logic in IPv4 and IPv6 paths
* use helpers returning bool for filtering the candidate addresses
  for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: document find_ip_in_subnet_list()

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PeeringState: fix get_backfill_priority min_size comparison

acting has placeholders for ec, need to use actingset.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40531 from sseshasa/wip-mclock-config-ref-doc

doc: Add mclock configuration reference documentation

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40456 from wjwithagen/wjw-fix-boost-version

cmake: bump the required boost version to 1.73

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40565 from tchaikov/wip-crimson-header-cleanup

crimson/osd: include headers at where they are used

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: include headers at where they are used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40564 from tchaikov/wip-crimson-internal_client_request-lite

crimson/osd: cleanups preparing for InternalClientRequest infrastructure.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pg_backend.h doesn't depend on pg.h anymore.

It's not only unnecessary but it also causes a depedency hell.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: use obc->get_oid() instead of passing hobject_t around.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: let error handlers to return errorated futures.

Before the change an error handler (a callable passed as second
or further parameters to `safe_then()`) was supposed to return
`seastar::future<..>` if it handled the errors or one of allowed
error types otherwise (like `crimson::ct_error::enoent`). Now
they may return also an erroratored future (`errator<ErrorT>::future<...>`)
and its error-set will fused with other sources of errors to be
included in errorator `safe_then()` returns.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: OpsExecuter::get_target() returns const reference now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: replace UTF-8 with ASCII in errorator's stateful_error_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: stateful_error_t allows for void-taking handlers.

Before this commit there was an inconsistencty between
`unthrowable_wrapper::handle()` and `stateful_error_t::handle()`
in the regard of parameters the provided handler had to accept.
The former was fine with void-taking handlers while the latter
was not.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop the never used ctors of osd_op_params_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: std::move func in pg::with_*_obc().

Well, likely all the commit does is just appending a couple
of extra bytes. Though, strictly juding, a func should be
`std::move()`d before calling in those cases. Otherwise we
can miss e.g. the `operator() &&` overload.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: use the alias for OpsExecuter in PG::do_osd_ops().

It's already defined there. We've just missed this fact in
one place.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix a thinko in the comment about OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop ObjectContext::loaded as it's never read.

The `PG::with_obc*` machinery uses the `existed` marker
provided by the LRU cache subsystem.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #40533 from Seong-yeop/master

src/rgw/rgw_client_io.h: fix a typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: don't cleanup the daemon keyring on failed redeploy

Fixes: https://tracker.ceph.com/issues/49872
Signed-off-by: Melissa Li <li.melissa.kun@gmail.com>

Merge pull request #40540 from tchaikov/wip-stop.sh

stop.sh: restructure stop logic

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40454 from zdover23/wip-doc-cephadm-osd-remove-main-text-rewrite-2021-Mar-26

doc/cephadm: rewrite osd.rst "Remove an OSD"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/ConfigMonitor: do not indent 'config dump' output

The indentation is misleading.  For example,

  mgr                                  advanced  mgr/telemetry/last_opt_revision       3                                                                                  *
  osd                                  advanced  osd_max_backfills                     10
  osd                                  basic     osd_memory_target                     6442450944
  osd                                  advanced  osd_scrub_auto_repair                 true
    mds.ar                             basic     mds_join_fs                           ar
    client.rgw.foo                     basic     rgw_frontends                         beast port=8080                                                                    *

The indentation kind of implies that mds.ar (or client.whatever) is
beneath 'osd', which it is not, and there rarely a 'client' scoped
option to make it look right.  Same problem for osd.foo etc.

Also, it is IMO easier to see the nesting structure when things align
naturally. e.g.

global
osd
osd.1
osd.2
mds
mds.foo
mds.foo.bar
client.foo.bar
...

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39092 from adamemerson/wip-datalog-genbacking

rgw: Fixes and Improvements for Datalog

Reviewed-By: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

cephadm: ignore apparmor if profiles file is empty

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #40548 from tchaikov/wip-doc-tweaks

doc: tweak .admonition.important and command line block formatting tweaks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/rbd: tweak the format of command line blocks

to use the "prompt" directive for better rendering.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_themes: tweak .admonition.important

do not render the links in "important" block in gray, it's not
readable in light red background. so use the blue and purple colors
specified by sphinx_rtd_theme.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #40524 into master

* refs/pull/40524/head:
doc: document CephFS Snapshot Mirroring

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40535 from tchaikov/wip-dashboard-pytest

pybind/mgr/dashboard: move pytest into requirements.txt

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #40504 into master

* refs/pull/40504/head:
doc/security: no gpg keys yet
doc/security: restructure a bit
doc/security: add security section

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40542 from tchaikov/wip-radosgw-admin-realm-rm

rgw/rgw_admin: s/realm delete/realm rm/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40541 from tchaikov/wip-executable-without-shebang

os/bluestore: remove executable bit of source file

Reviewed-by: Sage Weil <sage@redhat.com>

msg/msg_types: move ceph_entity_name::parse() into .cc

for faster compilation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/msg_types: let ceph_entity_name::parse() accept string_view

instead of accepting string, use string_view for the parsed string.

so the caller does not have to construct a temporary std::string object
for passing the string to be parsed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40173 from rhcs-dashboard/fix-40676-master

mgr/dashboard: Warn the user when creating/editing pools without application tags

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/security: no gpg keys yet

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40397 from rhcs-dashboard/update-texts

mgr/dashboard: Improve descriptions in some parts of the dashboard

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/security: restructure a bit

Signed-off-by: Sage Weil <sage@newdream.net>

doc/security: add security section

- how to report an issue
- describe process for handling vulnerability
- list past issues

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #40477 into master

* refs/pull/40477/head:
cephadm: fix failure when using --apply-spec and --shh-user

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40463 into master

* refs/pull/40463/head:
cephadm: specify addr on bootstrap's host add

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #40478 into master

* refs/pull/40478/head:
mgr/cephadm: don't have upgrade fail if "." in patch section of version

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

doc: document CephFS Snapshot Mirroring

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: Add mclock configuration reference documentation

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge PR #40530 into master

* refs/pull/40530/head:
doc/releases: reorg
doc/releases/releases.yaml: add missing releases
admin/build-doc: do not rebuild venv every time
doc/releases/releases.yaml: add 16.2.0
doc/releases: move 'active-releases' tag
doc/releases: add major headings for releases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw/rgw_admin: s/realm delete/realm rm/

it was an oversight of fa5398d31f9467b58a8b4eda55d1b5e2bbdb6500

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40451 from tchaikov/wip-dencoder-plugins

tools/ceph-dencoder: build dencoders as plugins

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>