crimson/os/seastore: use map::emplace(key, val) instead of map::emplace(pair<>)

simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: read_set_t<> should be an alias of set<>

we don't extend it by inheriting from this class. what we need is to
have a convenient template alias here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42143 from athanatos/sjust/wip-cm-omap-int-future

crimson/os/seastore: convert omap and collection_manager to interruptible future

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42138 from rzarzynski/wip-crimson-alienstore-indent

crimson/os: fix indentation in alien_store.cc.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42128 from tchaikov/wip-crimson-asock-cleanup

crimson/asock: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/seastore/omap_manager: convert to use interruptible future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/collection_manager: convert to use interruptible_future

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: clean up transaction_manager based tests

Permits using either vanilla TransactionManager and
InterruptedTransactionManager, updates users to use submit_transaction
helpers.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41980 from yuvalif/wip-yuval-fix-51320

rgw/notification: support version-id for all event types

Merge pull request #42127 from ivancich/wip-little-rgw-cleanup

rgw: a small set of clean-ups

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42012 from yuvalif/wip-yuval-fix-50039

rgw/notifications: support v4 auth for topics and notifications

rgw/notification: support version-id for all event types

including: object copy, multipart upload, delete marker on versioned bucket

Fixes: https://tracker.ceph.com/issues/51320
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #42134 from zdover23/wip-doc-cephadm-operations-2021-07-01

doc/cephadm: improve "Ceph Daemon Logs" (1 of x)

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42023 from TRYTOBE8TME/wip-rgw-notification-tests

rgw: Modification in the usage of force-branch

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #41945 from yuvalif/wip-yuval-fix-51261

rgw/notifications: support metadata filter in CompleteMultipartUpload and Copy events

Merge pull request #41953 from SMIL-Infra/fix-dashboard-osd-count

mgr/dashboard: fix OSD out count

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42116 from votdev/issue_51443_migrate_userdb

mgr/dashboard: User database migration has been cut out

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/os: fix indentation in alien_store.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #42107 into master

* refs/pull/42107/head:
mgr/ActivePyModules: streamline mgr_ips a bit
mgr/ActivePyModules: don't take unnecessary lock
pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()
mgr: add get() for standby modules

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/cephadm: improve "Ceph Daemon Logs" (1 of x)

This PR turned out to be a 3-in-1:

(1) improves syntax and formatting of "Logging to stdout"
(2) improves syntax and formatting of "Logging to files"
(3) replaces all carets with tildes in 3rd-level section
    headers in operations.rst (./build-doc was crying
    about inconsistency when I fed it tildes, but tildes
    and not carets are the RST standard according to
    https://docutils.sourceforge.io/ \
    docs/user/rst/quickstart.html#sections
    so the carets had to go.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42130 from tchaikov/wip-doc-install

doc/install/get-packages: s/el7/el8/ and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42108 from zdover23/wip-doc-upgrading-ceph-cancelling-the-upgrade-2021-06-30

doc/cephadm: improve "Canceling an Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/install/get-packages: do not mention additional 3rd party packages

since we are able to build and run on ubuntu focal without extra
dependencies of 3rd party packages, there is no need to mention them.
actually all of the removed packages are available in ubuntu focal.
some of them have different package names though.

the same applies to el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/install/get-packages: s/el7/el8/

since we don't build for el7 anymore. replace all occurrences of el7 with
el8.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: Modification in the usage of force-branch

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

crimson/admin: let AdminSocket::register_command() return void

instead of returning a seastar::future<>, it can just return void,
as it does not involve asynchronous operations.

also drop servers_tbl_rwlock, as we don't register hooks after OSD
starts, there is no need to protect hooks with a rw lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: remove unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42080 from cyx1231st/wip-seastore-fix-cache

crimson/os/seastore/cache: misc fixes and cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

crimson/os/seastore/cache: rename try_construct_record to prepare_record

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: wait io when read from Cache::extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add the missing extent invalid check

see 2bc257beb2ef1253e38c254745f42b3ac5db52de

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add missing add_to_read_set()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup get_extent_if_cached()

And remove the query_cache_for_extent(t).

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: fix get_extent_by_type(t) to use Transaction::get_extent()

The original query_cache_for_extent() doesn't call wait_io() if the
extent is pending. And the following get_extent_by_type() already covers
the lookup to Cache::extents and also implements wait_io() correctly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: add absent extent to read-set before retire

So that transaction can be invalidated if the retiring extent has
conflict.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: remove retired extents from cache

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: fix print_detail() of RetiredExtentPlaceholder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup try_construct_record() optional return

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: check invalid transaction only

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: retire PRESENT extents and add to retired-set

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: invalidate transaction referencing retired extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42103 from athanatos/sjust/wip-seastore-collection-ordering

crimson/seastore: enforce per-collection transaction ordering

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41932 from tchaikov/wip-mgr-mock-osdmap

mgr,pybind/mgr: add OSDMap.build_simple() method

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41958 from ceph/wip-yuriw-notes-14.2.22_2

doc: 14.2.22 Release Notes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus,octopus: add a note for msgr2 interoperability fix

octopus 15.2.13 already shipped, let's add it retroactively.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/releases/nautilus: add notable changes for RADOS

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: 14.2.22 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42118 from sandrobonazzola/patch-3

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd/iscsi-initiator-linux: set "product" in multipath.conf

Adding missing parameter in multipath configuration.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1769135.

Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>

rgw: a small set of clean-ups

Small items noticed along the way and addressed collectively in this
small "omnibus" PR.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #42121 from ceph/wip-md

ceph-release-notes: Support for markdown for new ceph.io site

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-release-notes: Support for markdown for new ceph.io site

Signed-off-by: David Galloway <dgallowa@redhat.com>

mgr/ActivePyModules: streamline mgr_ips a bit

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/ActivePyModules: don't take unnecessary lock

This isn't needed.

Signed-off-by: Sage Weil <sage@newdream.net>

pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()

This is necessary to allow the standby mgr to bind to the right IP.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr: add get() for standby modules

Fixes: https://tracker.ceph.com/issues/51446
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42026 from rhcs-dashboard/bucket-name-regex-fix

mgr/dashboard: Fix bucket name input allowing space in the value

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41353 from a16bitsysop/mgr-node_modules

pybind/mgr/CMakeLists.txt: exclude files not used at runtime

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: User database migration has been cut out

This PR will revert changes done by https://tracker.ceph.com/issues/49645 to auto-migrate user database v1 to v2.

Fixes: https://tracker.ceph.com/issues/51443
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #41954 from pkalever/notrim

rbd-nbd: support notrim option with map command

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42089 from tchaikov/wip-xattr-cleanup

osd, crimson/osd: do_cmp_xattr() related cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42054 from tchaikov/wip-os-xattrs

os: use transparent comparator in ObjectStore::getattrs()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #42043 from tchaikov/wip-ceph-argparse-ann

pybind/ceph_argparse: do not set self.typeargs in ctor of CephArgtype

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42087 from tchaikov/wip-common-buffers

common/buffers: check _num directly in list::c_str()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42084 from tchaikov/wip-49638

qa: run e2e test on centos only

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #42047 from tchaikov/wip-autoscaler-ann

pybind/mgr/pg_autoscaler: add typing annotations and cleanups

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42082 from fengchunsong/master

common: Use double instead of long double to improve performance

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37866 from pritha-srivastava/wip-sts-47809

rgw/sts: fix for copy object operation using sts

qa/workunits/rbd-nbd: add notrim test

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

mgr/dashboard: Fix bucket name input allows space in the value

Fixes: https://tracker.ceph.com/issues/51368
Signed-off-by: Nizamudeen A <nia@redhat.com>

pybind/mgr/pg_autoscaler: add typing annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: do not index pool_root with pool_name

pool_root is indexed by pool_id, and we never index it with pool_name.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: do not create a tuple long_desc

long_desc is supposed to be a str, not a tuple of str.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: extract CrushSubtreeResourceStatus out

as it also serves as part of interface of get_subtree_resource_status(),
not only its internals. to ease adding the type annotations, this class
is promoted out of the class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: define commands using CLICommand

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: define options using Option

more consistent and less error-prune this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: improve "Canceling an Upgrade"

This PR improves the section "Canceling an Upgrade"
in the "Upgrading Ceph" chapter of the cephadm
documentation.

I removed an extraneous prompt and rewrote a sentence
so that it was congruent with other sentences in similar
places elsewhere in the documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41919 from agayev/zoned-more-cleaning-support

os/bluestore: More support for cleaning zones.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42100 from rzarzynski/wip-crimson-assert_moveable

crimson: introduce assert_moveable().

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: add more type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41976 from tchaikov/wip-crimson-alienstore-lockless

crimson/os: use lockfree queue for sharded queue

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #42101 into master

* refs/pull/42101/head:
cephsqlite: add comment on atexit

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add collection ordering

Adds a mutex to SeastoreCollection which ensures that transactions are
ordered on a per-collection basis.  Future optimizations could enable
the transaction construction phase to be pipelined for a single collection,
but would require correctly handling the case where a more recently
submitted transaction encounters a conflict.

Fixes: https://tracker.ceph.com/issues/51358
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore: preserve Transaction::handle on reset for writes

We're going to need to preserve the pipeline phase locking on conflict.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/ordering_handle: fix phase descriptions

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: reorder members, make handle private

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: remove unused initiated_after

retired_gate_token has this information, Transaction::initiated_after
was left by accident.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/alienstore: use semaphore to manage tasks in thread pool

* implement std::counting_semaphore in C++17
* use the homebrew counting_semaphore as the synchronization primitive
  to access the pending tasks, for better performance than the
  implementation using mutex and condition_variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephsqlite: add comment on atexit

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson: introduce assert_moveable().

In C++ `std::moving` a `const`-qualified value yields a constant
r-value reference (`const T&&`) which won't be matched with a callable
taking non-constant r-value reference (like move constructors) but
can play with one taking a constant l-value reference (like copy
constructors do). This behaviour is surprising especially in lambas
where adding or removing the `mutable` specifier may lead to different
behaviour. The problem isn't obvious and it's easy to wrongly drop
the `mutable` druing a clean-up. Therefore introducing a tool for
developers to fail at compile-time if that happens seems desired.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42002 from tchaikov/wip-common-armor

common/armor: mark dst_end a const pointer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42035 from batrick/i51372

libcephsqlite: shutdown RADOS in destructor

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42094 from zdover23/wip-doc-upgrading-ceph-monitoring-the-upgrade-2021-06-29

doc/cephadm: improving "Monitoring the Upgrade"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42079 from zdover23/wip-doc-upgrading-ceph-first-section-2021-06-29

doc/cephadm: improve "Upgrading Ceph" (main)

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: improving "Monitoring the Upgrade"

This PR improves the section "Monitoring the Upgrade"
in the "Upgrading Ceph" chapter of the cephadm documentation.

This PR introduces a couple of section breaks with signposting
information in their titles, and rewrites some sentences in order
to reduce the cognitive load of the reader.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42042 from tchaikov/wip-51375

common/options: convert a millisecs opt to a chrono::milliseconds and cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os/alienstore: return attrs retrieved from bluestore directly

since ObjectStore returns a map<...,less<>> already, there is not point to
convert the returned map to the type expected by AlienStore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os: use transparent comparator in ObjectStore::getattrs()

for two reasons:

- better performance when looking on in the return map if the key
  is not a string, as we don't need to create a temporary string
  as the key
- improve the performance of crimson::AlienStore, as the latter
  uses the transparent comparator. as, without this change, we'd
  have to perform a deep copy to fill up the returned map with
  its non-transparent-comparator version.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use lockfree queue for sharded queue

each sharded queue has multiple producers and a single consumer queue,
but the producer side is in seastar world, so ideally, we should not
guard the queue using a POSIX lock.

in this change, the lockfree multiple-writer/multiple-reader queue is
used to replace the std::queue to drop the lock guarding it.

Signed-off-by: Kefu Chai <kchai@redhat.com>