seastore/rbm: remove changes related to transaction/cache

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: add protected_read interface to nvmedevice

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore/rbm: use const

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: return std::list in get_data_health()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: remove unused bits in nvmedevice.h

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: replace blk_paddr_t with size_t in free_extent()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: relocate file layout like SegmentManager

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: fix wrong loop condition

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: fix wrong statement

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: remove newline before function name

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: add check_bitmap_blocks when open()

This commits adds check_bitmap_blocks to check free_block_count
is correct whenever open() is called

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: fix wrong statement

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: use move()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/nvmedevice: use the necessary capture value and fix typo

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: move implementation of TestMemory to nvmeblockdevice.cc

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: utilize errorator as a return type of discard()

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: utilize unique_ptr as a return value of create()

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: change class name NormalNBD to PosixNVMeDevice

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: replace POSIX call to seastar::file in NormalNBD

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: add abort_allocation

when aborting uncommitted transaction, abort_allocation
should be called to cancel in-flight allocations in
in-memory allocator.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test: add more tests

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test: modify existing unit test to cover range based operations

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: fix wrong operator

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: make multiple bitmap block initialization to work correctly

when initializing bitmap area, this commit corrects multiple
block initialization.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: continous range based free_extent()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: continous range based complete_allocation()

To work with a range of allocated blocks as a input,
this commit modify complete_allocation().

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: introduce rbm_sync_block_bitmap_by_range()

To set continous range of block id at once, this commit adds
rbm_sync_block_bitmap_by_range().

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: make alloc_extent to allocate continuous blocks

If there is no continous blocks, alloc_extent() returns nospc

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: use deltas to deliver allocated info

alloc_extent() will allocates continous blocks (e.g., block 1 ~ 4).
To deliver such informations to journal efficiently,
this commit adds a member variable regarding only modified range in the deltas

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: add NormalNBD

NormalNBD - NVMe Block device with seastar file
 1. Support preffered write granularity/alignment which is reported
 from NVMe SSD. User should follow this write guidance for enhancing write
 performance.
 2. Support multi-stream IO in NVMe SSD.

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: randomblockmanager inital commit

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge PR #41072 into master

* refs/pull/41072/head:
doc/releases/nautilus: fix a braino in 14.2.20 release note

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40924 into master

* refs/pull/40924/head:
mgr/cephadm: check hostname resolution before adding host
cephadm: provide a way to checkhost connection without /etc/hosts passed the shell
doc/cephadm: remove /etc/hosts from list of hostname resoltion methods

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40593 from ideepika/wip-new-testing-params

qa/config/rados: add dispatch delay testing params

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge PR #40941 into master

* refs/pull/40941/head:
qa/suites/rados/cephadm/smoke-roleless: test client-keyring
qa/tasks/cephadm.py: adjust client.admin key mode; place on all hosts
cephadm: distribute client.admin keyring+conf to label:_admin on bootstrap
doc/cephadm: document the default 'admin' label
mgr/cephadm: 'ceph orch client-keyring ...' commands to manage keyring files
mgr/cephadm: reimplement ceph.conf pushing
mgr/cephadm: use _write_remote_file for ceph.conf
mgr/cephadm: _write_remote_file helper
mgr/cephadm: add placementspec for which hosts get ceph.conf

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40943 into master

* refs/pull/40943/head:
mgr/cephadm: skip ok-to-stop for mons in upgrade if < 3 mons
mgr/cephadm: don't allow upgrade start with less than 2 mgrs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #41049 into master

* refs/pull/41049/head:
cephadm: re-assimilate user provided conf after mgr created

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41055 into master

* refs/pull/41055/head:
cephadm: allow several public networks be matched

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41059 from tchaikov/wip-crimson-clang-cleanup

crimson: clean up issues spotted by the clang compiler

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/releases/nautilus: fix a braino in 14.2.20 release note

Setting auth_expose_insecure_global_id_reclaim to false disables the
"disconnect right after authentication" behavior, not the warning.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41063 from tchaikov/wip-rgw-move

rgw/rgw_rest_sts: return local variable w/o std::move()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41067 from zdover23/wip-doc-crismon-misspelling-fix-2021-Apr-28

doc/crimson: s/Crismon/Crimson/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41048 from zdover23/wip-doc-rados-config-cluster-names-2021-Apr-27

doc/rados: rename "Running Multiple Clusters"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/crimson: s/Crismon/Crimson/

This misspelling occurred twice in this
document.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: do not capture unused variable

this change silences the warning from clang, like:

 ../src/crimson/common/interruptible_future.h:276:8: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
       [this, func=std::move(func),
        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: mark final class "final"

this change silences warning like:

 ../src/crimson/os/seastore/onode_manager/staged-fltree/fltree_onode_manager.h:80:18: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
   ~FLTreeOnode() final {}
                  ^
 ../src/crimson/os/seastore/onode_manager/staged-fltree/fltree_onode_manager.h:13:8: note: mark 'crimson::os::seastore::onode::FLTreeOnode' as 'final' to silence this warning
 struct FLTreeOnode : Onode, Value {
        ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore: do not put braces around scalar initializer

this change silences the warning from clang like:

src/test/crimson/seastore/onode_tree/test_value.h:152:12: warning: braces around scalar initializer [-Wbraced-scalar-init]
     return {static_cast<value_size_t>(size - sizeof(value_header_t))};
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not reference non-template with template

this change addresses the build failure with clang:

src/crimson/os/seastore/onode_manager/staged-fltree/node_extent_accessor.h:352:29: error: 'encode_split' following the 'template' keyword does not refer to a template
    test_recorder->template encode_split(split_at, read().p_start());
                   ~~~~~~~~ ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture label using &

structured binding does not define variables. unlike GCC, Clang does
not allow this. this change address following build failures with clang:

../src/crimson/os/seastore/omap_manager/btree/omap_btree_node_impl.cc:210:24:
note: 'result' declared here
      auto &[complete, result] = ret;
                       ^
../src/crimson/os/seastore/omap_manager/btree/omap_btree_node_impl.cc:214:6:
error: reference to local binding 'complete' declared in enclosing
lambda expression
            complete = biter == eiter;
            ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados: rename "Running Multiple Clusters"

This PR corrects a title so that the title
more accurately describes the content in the
section it names.

Fixes: https://tracker.ceph.com/issues/50517 (partially)
Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw/rgw_rest_sts: return local variable w/o std::move()

in C++17, the C++ standard ensures that the return value in this case
can be optimized with copy elision, adding std::move() is not necessary,
and prevents copy elision from happening.

this change also silences the warning of:

../src/rgw/rgw_rest_sts.cc: In member function 'std::unique_ptr<rgw::sal::RGWOIDCProvider> rgw::auth::sts::WebTokenEngine::get_provider(const DoutPrefixProvider*, const string&, const string&) const':
../src/rgw/rgw_rest_sts.cc:92:19: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
   92 |   return std::move(provider);
      |          ~~~~~~~~~^~~~~~~~~~
../src/rgw/rgw_rest_sts.cc:92:19: note: remove 'std::move' call

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: allow several public networks be matched

Fixes: https://tracker.ceph.com/issues/50548
Signed-off-by: Stanislav Datskevych <me@nek0.net>

Merge pull request #41032 from leseb/raw-list-json

ceph-volume: fix "device" output

Merge pull request #41002 from strenuous-life/wip-cephadm-daemon-restart-mgr-loop

mgr/cephadm: The command of 'ceph orch daemon restart mgr.xxx' may ca…

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41009 from tchaikov/wip-mon-cleanup

mon/OSDMonitor: let OSDMonitor::_get_pending_crush() return

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/smoke-roleless: test client-keyring

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm.py: adjust client.admin key mode; place on all hosts

Except during upgrades, since it is not supported there.

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: distribute client.admin keyring+conf to label:_admin on bootstrap

If we are placing ceph.conf in /etc/ceph (the default), tell the cluster
to continue doing this going forward to hosts with the '_admin' label.

This doesn't induce the user to add the admin label to other hosts too,
unfortunately--e probably want them to add the admin label to other mons,
for instance--but it is a start.

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: re-assimilate user provided conf after mgr created

Fixes: https://tracker.ceph.com/issues/50544
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #41046 from rzarzynski/wip-crimson-monc-auth_reply

crimson/monc: fix use-after-free around Connection::do_auth_single().

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40936 from rhcs-dashboard/support-ingress-service-creation-UI

mgr/dashboard: fix HAProxy (now called ingress)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #40426 from rhcs-dashboard/host-maintenance-followups

mgr/dashboard: Host Maintenance Follow ups

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

crimson/monc: fix use-after-free around Connection::do_auth_single().

This bug caused following failure in teuthology testing [1]:

```
WARN  2021-04-23 07:08:29,449 [shard 0] osd - ms_handle_reset
WARN  2021-04-23 07:08:29,449 [shard 0] monc - active conn reset v2:172.21.15.100:3300/0
INFO  2021-04-23 07:08:29,449 [shard 0] monc - reopen_session to mon.-1
INFO  2021-04-23 07:08:29,449 [shard 0] monc - close
INFO  2021-04-23 07:08:29,449 [shard 0] monc - connecting to mon.0
...
ERROR 2021-04-23 07:08:29,450 [shard 0] none - /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-3454-gddedc2e1/rpm/el8/BUILD/ceph-17.0.0-3454-gddedc2e1/src/crimson/mon/MonClient.cc:228 : In function 'crimson::mon::Connection::do_auth_single(crimson::mon::Connection::request_t)::<lambda(Ref<MAuthReply>)>', ceph_assert(%s)
closed
```

[1]: http://pulpito.front.sepia.ceph.com/rzarzynski-2021-04-22_00:20:19-rados-master-distro-basic-smithi/6063316/

The root cause is in freeing the `active_con` in `Client::reopen_session()`
while there still could be the second, ongoing part of `do_auth_single()`.
This fix rectifies the issue by switching `std::unique_ptr` to `seastar::shared_ptr`
and extending the life-time with the help of `seastar::enable_shared_from_this<T>`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #40810 into master

* refs/pull/40810/head:
test: tests for *at() libcephfs APIs
test: cleanup files/dirs before finishing test
client: non-at APIs call at-APIs with CEPHFS_AT_FDCWD as file descriptor
libcephfs: introduce basic *at() calls
client: remove redundant caps_issued_mask() call in Client::fstatx()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephadm: document the default 'admin' label

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: 'ceph orch client-keyring ...' commands to manage keyring files

Teach cephadm to manage keyring files on cluster hosts.  These keys must
already exist in the mon auth database--cephadm does not create them if
they don't exist (and will issue warnings to the log if they do not).

A ceph.conf is pushed implicitly along with the keyring file.

Each keyring added will be pushed to the hosts described by the placement
spec with the appropriate ownership and mode.  If the ownership, mode, or
path are modified, the files are rewritten or removed as need.

If the client-keyring entry is removed, the keyring files are removed.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: reimplement ceph.conf pushing

Use a more generic inventory map of paths to digests to track what we've
pushed.

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/monc: assert Connection::handle_auth_reply() is truly provided with a msg.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: rename Connection::reply to auth_reply.

Easier to distinguish from local variables with the same name
we have a few in `MonClient.cc`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #40157 from leseb/do-not-fail-when-nosnap

mgr/pybind/snap_schedule: do not fail when no fs snapshots are available

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: Host Maintenance Follow ups

Added the ability to create a host in maintenance mode

Fixes: https://tracker.ceph.com/issues/49998
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41035 from ktdreyer/rbd-cache-err-spelling

librbd: fix spelling in immutable-object-cache conn error

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41037 from tchaikov/wip-npm-audit

mgr/dashboard: upgrade package-lock.json

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #40933 from vshankar/wip-50266

cephfs-mirror: sanitize `daemons status` JSON

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41013 from tchaikov/wip-doc-mds-confval

doc/cephfs/mds-config-ref: render options using confval directive

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41022 from tchaikov/wip-cmake-findpmem

cmake/modules/Findpmem: always set pmem_VERSION_STRING

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: upgrade package-lock.json

diminish "npm run fix:audit" divergences

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: The command of 'ceph orch daemon restart mgr.xxx' may case mgr daemon loop to restart
Scene:
The mgr daemon is active. After execing restart command, it may be save "scheduled_daemon_actions": {"mgr.xxx": "restart"}}" to config-key.
So the mgr daemon will restart before call rm_scheduled_daemon_action which case mgr daemon will load restart forever.

Fix mgr infinite restart issue refering to the same solution as 'ceph orch daemon redeploy'.

Signed-off-by: jianglong01 <jianglong01@qianxin.com>

Merge pull request #41010 from rzarzynski/wip-crimson-backtrace-fix-assert

crimson/common: fix broken assertion on FatalSignal setup.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: fix spelling in immutable-object-cache conn error

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #41030 from zdover23/wip-doc-adding-one-word-to-cephadm-nfs-2021-Apr-26

doc/cephadm: add a single word

Reviewed-by: Sage Weil <sweil@redhat.com>

ceph-volume: fix "device" output

During
https://github.com/ceph/ceph/commit/ec0f5f3b22d24754c16131a1996e42b787e4255f
I changed "device" by "dev", let's revert that!

Fixes: https://tracker.ceph.com/issues/50478
Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #38368 from rishabh-d-dave/cv-list-by-osd-id

ceph-volume: allow listing devices by OSD ID

mgr/dashboard: fix HAProxy (now called ingress)

Fixes: https://tracker.ceph.com/issues/50319
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Support from Dashboard UI to create Ingress service type.

Merge pull request #40992 from zdover23/wip-doc-cephadm-osd-advanced-adding-a-word-2021-apr-22

doc/cephadm: adding "device" to a sentence

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/cephadm: add a single word

s/If the nfs-ganesha doesn't/If the
nfs-ganesha pool doesn't/

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40724 from rhcs-dashboard/upgrade-caniuse-lite

mgr/dashboard: upgrade Browserslist: caniuse-lite

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41029 from zdover23/wip-doc-cephadm-nfs-service-corrections-2021-Apr-26

doc/cephadm: rewrite "nfs.rst"

Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #40594 from mkogan1/wip-rgw-multipart-complete

rgw: return OK on consecutive complete-multipart reqs

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

test: tests for *at() libcephfs APIs

Fixes: http://tracker.ceph.com/issues/50298
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: cleanup files/dirs before finishing test

Other tests might use the same file names and expect libcephfs
calls to succeed.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: non-at APIs call at-APIs with CEPHFS_AT_FDCWD as file descriptor

Signed-off-by: Venky Shankar <vshankar@redhat.com>

libcephfs: introduce basic *at() calls

Signed-off-by: Venky Shankar <vshankar@redhat.com>

client: remove redundant caps_issued_mask() call in Client::fstatx()

No need to include this check since _getattr() checks this anyway.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc/cephadm: rewrite "nfs.rst"

This PR makes a few (very small) stylistic
corrections, mostly typographical, to the text.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev/osd_internals/mclock_wpq_cmp_study: fix a typo

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs/mds-config-ref: render options using confval directive

less repeating, better maintanability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/operations/pools: define pool settings using "describe"

better readability and more structured this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration: reference options using :confval:

Signed-off-by: Kefu Chai <kchai@redhat.com>