doc/man/8/cephadm: add --log-to-file (and --single-host-defaults)

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: add bootstrap --log-to-file option

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41953 from SMIL-Infra/fix-dashboard-osd-count

mgr/dashboard: fix OSD out count

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42116 from votdev/issue_51443_migrate_userdb

mgr/dashboard: User database migration has been cut out

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #42107 into master

* refs/pull/42107/head:
mgr/ActivePyModules: streamline mgr_ips a bit
mgr/ActivePyModules: don't take unnecessary lock
pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()
mgr: add get() for standby modules

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #42130 from tchaikov/wip-doc-install

doc/install/get-packages: s/el7/el8/ and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42108 from zdover23/wip-doc-upgrading-ceph-cancelling-the-upgrade-2021-06-30

doc/cephadm: improve "Canceling an Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/install/get-packages: do not mention additional 3rd party packages

since we are able to build and run on ubuntu focal without extra
dependencies of 3rd party packages, there is no need to mention them.
actually all of the removed packages are available in ubuntu focal.
some of them have different package names though.

the same applies to el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/install/get-packages: s/el7/el8/

since we don't build for el7 anymore. replace all occurrences of el7 with
el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42080 from cyx1231st/wip-seastore-fix-cache

crimson/os/seastore/cache: misc fixes and cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

crimson/os/seastore/cache: rename try_construct_record to prepare_record

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: wait io when read from Cache::extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add the missing extent invalid check

see 2bc257beb2ef1253e38c254745f42b3ac5db52de

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add missing add_to_read_set()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup get_extent_if_cached()

And remove the query_cache_for_extent(t).

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: fix get_extent_by_type(t) to use Transaction::get_extent()

The original query_cache_for_extent() doesn't call wait_io() if the
extent is pending. And the following get_extent_by_type() already covers
the lookup to Cache::extents and also implements wait_io() correctly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add absent extent to read-set before retire

So that transaction can be invalidated if the retiring extent has
conflict.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: remove retired extents from cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: fix print_detail() of RetiredExtentPlaceholder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup try_construct_record() optional return

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: check invalid transaction only

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: retire PRESENT extents and add to retired-set

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: invalidate transaction referencing retired extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42103 from athanatos/sjust/wip-seastore-collection-ordering

crimson/seastore: enforce per-collection transaction ordering

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41932 from tchaikov/wip-mgr-mock-osdmap

mgr,pybind/mgr: add OSDMap.build_simple() method

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41958 from ceph/wip-yuriw-notes-14.2.22_2

doc: 14.2.22 Release Notes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus,octopus: add a note for msgr2 interoperability fix

octopus 15.2.13 already shipped, let's add it retroactively.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus: add notable changes for RADOS

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 14.2.22 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42118 from sandrobonazzola/patch-3

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Adding missing parameter in multipath configuration.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1769135.

Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>

Merge pull request #42121 from ceph/wip-md

ceph-release-notes: Support for markdown for new ceph.io site

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-release-notes: Support for markdown for new ceph.io site

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/ActivePyModules: streamline mgr_ips a bit

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/ActivePyModules: don't take unnecessary lock

This isn't needed.

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()

This is necessary to allow the standby mgr to bind to the right IP.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: add get() for standby modules

Fixes: https://tracker.ceph.com/issues/51446
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42026 from rhcs-dashboard/bucket-name-regex-fix

mgr/dashboard: Fix bucket name input allowing space in the value

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41353 from a16bitsysop/mgr-node_modules

pybind/mgr/CMakeLists.txt: exclude files not used at runtime

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: User database migration has been cut out

This PR will revert changes done by https://tracker.ceph.com/issues/49645 to auto-migrate user database v1 to v2.

Fixes: https://tracker.ceph.com/issues/51443
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #41954 from pkalever/notrim

rbd-nbd: support notrim option with map command

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42089 from tchaikov/wip-xattr-cleanup

osd, crimson/osd: do_cmp_xattr() related cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42054 from tchaikov/wip-os-xattrs

os: use transparent comparator in ObjectStore::getattrs()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #42043 from tchaikov/wip-ceph-argparse-ann

pybind/ceph_argparse: do not set self.typeargs in ctor of CephArgtype

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42087 from tchaikov/wip-common-buffers

common/buffers: check _num directly in list::c_str()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42084 from tchaikov/wip-49638

qa: run e2e test on centos only

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #42047 from tchaikov/wip-autoscaler-ann

pybind/mgr/pg_autoscaler: add typing annotations and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42082 from fengchunsong/master

common: Use double instead of long double to improve performance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37866 from pritha-srivastava/wip-sts-47809

rgw/sts: fix for copy object operation using sts

qa/workunits/rbd-nbd: add notrim test

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

mgr/dashboard: Fix bucket name input allows space in the value

Fixes: https://tracker.ceph.com/issues/51368
Signed-off-by: Nizamudeen A <nia@redhat.com>

pybind/mgr/pg_autoscaler: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: do not index pool_root with pool_name

pool_root is indexed by pool_id, and we never index it with pool_name.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: do not create a tuple long_desc

long_desc is supposed to be a str, not a tuple of str.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: extract CrushSubtreeResourceStatus out

as it also serves as part of interface of get_subtree_resource_status(),
not only its internals. to ease adding the type annotations, this class
is promoted out of the class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: define commands using CLICommand

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: define options using Option

more consistent and less error-prune this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: improve "Canceling an Upgrade"

This PR improves the section "Canceling an Upgrade"
in the "Upgrading Ceph" chapter of the cephadm
documentation.

I removed an extraneous prompt and rewrote a sentence
so that it was congruent with other sentences in similar
places elsewhere in the documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41919 from agayev/zoned-more-cleaning-support

os/bluestore: More support for cleaning zones.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42100 from rzarzynski/wip-crimson-assert_moveable

crimson: introduce assert_moveable().

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: add more type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41976 from tchaikov/wip-crimson-alienstore-lockless

crimson/os: use lockfree queue for sharded queue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #42101 into master

* refs/pull/42101/head:
cephsqlite: add comment on atexit

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add collection ordering

Adds a mutex to SeastoreCollection which ensures that transactions are
ordered on a per-collection basis.  Future optimizations could enable
the transaction construction phase to be pipelined for a single collection,
but would require correctly handling the case where a more recently
submitted transaction encounters a conflict.

Fixes: https://tracker.ceph.com/issues/51358
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore: preserve Transaction::handle on reset for writes

We're going to need to preserve the pipeline phase locking on conflict.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/ordering_handle: fix phase descriptions

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: reorder members, make handle private

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: remove unused initiated_after

retired_gate_token has this information, Transaction::initiated_after
was left by accident.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/alienstore: use semaphore to manage tasks in thread pool

* implement std::counting_semaphore in C++17
* use the homebrew counting_semaphore as the synchronization primitive
  to access the pending tasks, for better performance than the
  implementation using mutex and condition_variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephsqlite: add comment on atexit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson: introduce assert_moveable().

In C++ `std::moving` a `const`-qualified value yields a constant
r-value reference (`const T&&`) which won't be matched with a callable
taking non-constant r-value reference (like move constructors) but
can play with one taking a constant l-value reference (like copy
constructors do). This behaviour is surprising especially in lambas
where adding or removing the `mutable` specifier may lead to different
behaviour. The problem isn't obvious and it's easy to wrongly drop
the `mutable` druing a clean-up. Therefore introducing a tool for
developers to fail at compile-time if that happens seems desired.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42002 from tchaikov/wip-common-armor

common/armor: mark dst_end a const pointer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42035 from batrick/i51372

libcephsqlite: shutdown RADOS in destructor

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42094 from zdover23/wip-doc-upgrading-ceph-monitoring-the-upgrade-2021-06-29

doc/cephadm: improving "Monitoring the Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42079 from zdover23/wip-doc-upgrading-ceph-first-section-2021-06-29

doc/cephadm: improve "Upgrading Ceph" (main)

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: improving "Monitoring the Upgrade"

This PR improves the section "Monitoring the Upgrade"
in the "Upgrading Ceph" chapter of the cephadm documentation.

This PR introduces a couple of section breaks with signposting
information in their titles, and rewrites some sentences in order
to reduce the cognitive load of the reader.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42042 from tchaikov/wip-51375

common/options: convert a millisecs opt to a chrono::milliseconds and cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/alienstore: return attrs retrieved from bluestore directly

since ObjectStore returns a map<...,less<>> already, there is not point to
convert the returned map to the type expected by AlienStore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os: use transparent comparator in ObjectStore::getattrs()

for two reasons:

- better performance when looking on in the return map if the key
  is not a string, as we don't need to create a temporary string
  as the key
- improve the performance of crimson::AlienStore, as the latter
  uses the transparent comparator. as, without this change, we'd
  have to perform a deep copy to fill up the returned map with
  its non-transparent-comparator version.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use lockfree queue for sharded queue

each sharded queue has multiple producers and a single consumer queue,
but the producer side is in seastar world, so ideally, we should not
guard the queue using a POSIX lock.

in this change, the lockfree multiple-writer/multiple-reader queue is
used to replace the std::queue to drop the lock guarding it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr,mon: s/boost::optional/std::optional/

since the encoding schemes of boost::optional and std::optional are
identical, they can be used interchangeably.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: s/boost::variant/std::variant/

we should use standard library for more well defined behavior, and
less dependencies on 3rd party libraries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: s/boost::variant/std::variant/

we should use standard library for more well defined behavior, and
less dependencies on 3rd party libraries.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42013 from ronen-fr/wip-ronenf-scrubs-config

qa/suites/rados: add simultaneous scrubs to the thrasher

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42031 from sebastian-philipp/normalize_image_digest_local-registry

cephadm: Fix normalize_image_digest for local registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41933 from liewegas/fix-48311

mgr: set debug_mgr=2/5 (so INFO goes to mgr log by default)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41999 from tchaikov/wip-rpm-deb

rpm,debian: cleanups related to python3-setuptools dependencies

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #42076 from liu-chunmei/crimson-pgnls

crimson: fix pgnls exception

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: reorder #includes

so the oder of includes complies to https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: use std::from_chars() to convert str to integer

* no need to create a temporary string for using strtoull()
* use std::from_chars() so it's consistent with its counterpart in
  in crimson.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: check invalid input in do_xattr_cmp_u64()

we should return -EINVAL if the string does not represent a decimal
integer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: include used header

for using std::from_chars()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: extract do_cmp_xattr()

to consolidate the logic to dispatch by op.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: extract do_cmp_xattr()

to consolidate the logic to dispatch by op.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42039 from liu-chunmei/crimson-cmpxattr

crimson: add cmp_xatt support

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/buffers: check _num directly in list::c_str()

no need to create temporary iterator for comparing it with cend().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: fix pgnls exception

has_pg_op is always false, since m->ops is empty at that time.
so pgnls operation will go to process_op and report unknown operations.
move m->finish_decode ahead to fill m->ops.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

qa: run e2e test on centos only

it's a regression introduced by the restrcuture of the test suites,
let's pin the test to CentOS8.

See-also: https://tracker.ceph.com/issues/49638
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: add cmp_xatt support

fix Exceptional future ignored: std::runtime_error (op 'cmpxattr' not supported)

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

common: Use double instead of long double to improve performance

To convert namoseconds to seconds, the precision needs to be 10,
and the precision of double is 15, which is enough to use.
On aarch64, double division uses the div instruction, while long
double uses the gcc buildin _divtf3, which has poor performance.
Therefore, use double instead of long double for better performance.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>