Merge pull request #23044 from tchaikov/wip-crimson-connection

crimson/connection: misc changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23057 from liewegas/wip-12.2.7-notes

doc/release/luminous: v12.2.6 and v12.2.7 release notes

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23085 from Songweibin/wip-rbd-unused-var

test/librbd: drop unused variable ‘num_aios’

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #22681 into master

* refs/pull/22681/head:
common: use ceph_pthread_getname

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23052 into master

* refs/pull/23052/head:
osd: fix deep scrub with osd_skip_data_digest=true (default) and bluestore

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release/luminous: v12.2.7 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22899 from smithfarm/wip-doc-argonaut

doc: cleanup: prune Argonaut-specific verbiage

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/start/hardware-recommendations.rst: link to Ceph blog

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: fix up install/upgrading-ceph.rst

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: cleanup: prune Argonaut-specific verbiage

Also drop all release-specific upgrading instructions (they only go up to
Firefly, anyway - none of the current releases are covered).

Note that all of this verbiage I am removing here can still be accessed
on docs.ceph.com via e.g. http://docs.ceph.com/docs/firefly/

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23079 from smithfarm/wip-with-seastar-fix

build/ops: install-deps.sh: set with_seastar

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librbd: drop unused variable ‘num_aios’

Signed-off-by: songweibin <song.weibin@zte.com.cn>

build/ops: install-deps.sh: set with_seastar

25d5122b5538062e8282251c3d7bae0f9792d596 introduced code into
install-deps.sh which was intended to set a variable "with_seastar" to the
string "true" or "false" depending on whether or not an environment
variable WITH_SEASTAR is defined.

As originally implemented, the conditional is always true, so with_seastar
is set to "true" unconditionally. Fix this by putting $WITH_SEASTAR in
double quotes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23068 into master

* refs/pull/23068/head:
doc: fix some errors in the cephfs-journal-tool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22737 from trociny/wip-24675

librbd: validate data pool for self-managed snapshot support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: fix some errors in the cephfs-journal-tool

Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge PR #23059 into master

* refs/pull/23059/head:
common/ceph_strings: add request_flush_mdlog op names to ceph_session_op_name()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22211 into master

* refs/pull/22211/head:
cephfs-journal-tool: get and check layout info during dump/undump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22705 into master

* refs/pull/22705/head:
Changed Lua library support from all to partial
mds/Mantle: Fixed Mantle's obsolete Lua C++ API

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22751 into master

* refs/pull/22751/head:
test: add tests for ceph_utime() and friends
libcephfs: extend support for changing file access/modification time

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22801 into master

* refs/pull/22801/head:
client: add inst to asok status output

Reviewed-by: Venky Shankar <vshankar@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22804 into master

* refs/pull/22804/head:
mds: remove useless dout_subsys macro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: validate data pool for self-managed snapshot support

Fixes: https://tracker.ceph.com/issues/24675
Signed-off-by: Mykola Golub <mgolub@suse.com>

doc: releases: release notes for v12.2.6 luminous

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #22945 from cxytt/master

librbd:optionally support FUA (force unit access) on write requests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22637 from zy751713126/resize_equal_size

rbd: error if new size is equal to original size

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20112 from liyichao/aio_fail

librbd: handle aio failure in ManagedLock and PreReleaseRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: fix deep scrub with osd_skip_data_digest=true (default) and bluestore

The osd_skip_data_digest is about skipping the storing and checking of a
full-object data digest.  Here it is also disabling crc checks during
deep-scrub, which prevents us from detecting crc differences across
replicas.  Fix!

Fixes: http://tracker.ceph.com/issues/24922
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23046 from tchaikov/wip-ftbfs-rgw-kefu

rgw: fix FTBFS introduced by abca9805

rgw: fix FTBFS introduced by abca9805

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22809 from tchaikov/wip-with-seastar

run-make-check: enable --with-seastar option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22964 from tchaikov/wip-import-pub-key-without-hkp-server

install-deps.sh: import ubuntu-toolchain-r's key without keyserver

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

crimson/net: use boost::thread_unsafe_counter for refcounting Connection

seastar is basically single threaded.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: add Connection::get_peer_type()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22971 into master

* refs/pull/22971/head:
qa: optionally use old fs name during cephfs remount
qa: remove dead code
qa: run asok command on correct machine

qa: optionally use old fs name during cephfs remount

This fixes errors caused by remount done by some tests (test_recovery_pool.py)
where the fs name is not given.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove dead code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: run asok command on correct machine

The MDS may not be on the same machine where the cluster command is run.

Fixes: http://tracker.ceph.com/issues/24858
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21885 into master

* refs/pull/21885/head:
qa: update cluster log health warning message
qa: add tests for client features
mds: evict clients that lack required features
mds: cleanup MDSRank::evict_client
mds: infer client version by client metadata and connection's features
mds: introduce "ceph fs set <fs_name> min_compat_client <release_name>"
mds: tell client why it's rejected
mds: introduce cephfs' own feature bits
mds: make Server::prepare_force_open_sessions() update client metadata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23018 into master

* refs/pull/23018/head:
qa: use 4 mds cluster for multifs tests

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: use 4 mds cluster for multifs tests

Fixes: http://tracker.ceph.com/issues/24899
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update cluster log health warning message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23033 from linuxbox2/wip-rgw-409

rgw: do not ignore EEXIST in RGWPutObj::execute

Merge pull request #23020 from tchaikov/wip-config-value-friend

common/config_values: friend md_config_impl<>

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: do not ignore EEXIST in RGWPutObj::execute

The existing logic appears able to cause propagation of a failed
exclusive create to the client, when it should instead have been
retried, due to disagreement about the logical write offset. (The
value of ofs here could be > 0 due to the operation of a stacked
write filter [e.g., compressor], when the RADOS write offset was
0 and hence an exclusive write that should be retried).

Rationale for fix by Casey.

http://tracker.ceph.com/issues/22790

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rbd-ggate: filter out ceph options from args

we can just filter out ceph options from command line args instead of
parsing config in env variables, conf files and argvs.

the config-parsing steps were modeled after the ones in rbd-nbd.cc, but
the whole purpose of reading configurations from env/conf-file is to get
the "rbd_default_pool" for setting cfg->poolname. apparently, this does
not apply to rbd-ggate, where global_init() will take care of all the
trivial work to populate the settings. moreover, `config` is not
read after being initialized in main.cc. so drop it.

Signed-off-by: Mykola Golub <mgolub@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config_values: friend md_config_impl<>

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22995 into master

* refs/pull/22995/head:
qa: use root to find client admin socket

Reviewed-by: Nathan Cutler <ncutler@suse.com>

common/ceph_strings: add request_flush_mdlog op names to ceph_session_op_name()

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

qa: add tests for client features

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: evict clients that lack required features

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #22938 from tchaikov/wip-bundled-dpdk

cmake: do not depend on ${DPDK_LIBRARIES} if not using bundled dpdk

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: cleanup MDSRank::evict_client

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: infer client version by client metadata and connection's features

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: introduce "ceph fs set <fs_name> min_compat_client <release_name>"

The command set the oldest version of client that is allowed to connect
to mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: tell client why it's rejected

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: introduce cephfs' own feature bits

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/14456

mds: make Server::prepare_force_open_sessions() update client metadata

The function is used for opening sessions for importing caps. It didn't
set client metadata for new sessions.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #22594 from wido/mgr-pg-states

mgr: Centralize PG_STATES to MgrModule

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reveiwed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22952 from tchaikov/wip-cmake-atomic

cmake: should link against libatomic if libcxx/libstdc++ does not off…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: link libcrimson against libceph-common

as libcrimson uses symbols exposed by the later, among other things, the
Authorization related stuff.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unittest_seastar_{messenger,echo} from tests

unittest_seastar_echo cannot be run as unittest w/o starting two
instances.
unittest_seastar_messenger times out. need to RCA it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: extract SubProcess impl into .cc

cleanup the header dependencies and speed-up the compiling

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up fixes to address build failures

see also https://groups.google.com/forum/#!topic/seastar-dev/uEMYGgYJKOY

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: enable WITH_SEASTAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,install-deps.sh: add --with-seastar option

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include seastar headers with 'seastar/' prefix

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd:optionally support FUA (force unit access) on write requests

Fixes: http://tracker.ceph.com/issues/19366
Signed-off-by: ningtao <63358@sangfor.com>

Merge PR #22877 into master

* refs/pull/22877/head:
common: fix races when visiting a TrackedOp.

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #22946 into master

* refs/pull/22946/head:
os/bluestore: batch handle get_fragmentation.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #22950 into master

* refs/pull/22950/head:
rpm: move make check-specific build deps to appropriate conditional block
rpm: remove universal requires from distro-specific conditional blocks

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22972 into master

* refs/pull/22972/head:
os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22974 into master

* refs/pull/22974/head:
qa/standalone/osd/ec-error-rollforward: reproduce bug 24597
qa/suites/rados/thrash-erasure-code: add many deletes workload
qa/standalone/osd/repro_long_log.sh: fix test
osd/PG: do not blindly roll forward to log.head

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/standalone/osd/ec-error-rollforward: reproduce bug 24597

This reproduces http://tracker.ceph.com/issues/24597

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code: add many deletes workload

Having lots of deletes will mean deletes on objects that don't exist,
which will in turn mean error log entries and more coverage of the
append_log_entries_update_missing code.  Hopefully this will trigger
http://tracker.ceph.com/issues/24597

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/repro_long_log.sh: fix test

The log trimming case wasn't quite right.  Before HEAD^ we were
rolling forward too aggressively and miscalculating the can_rollforward_to,
which affected the trim_to calculation.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: do not blindly roll forward to log.head

If we are told we can roll forward by the primary, we should only roll
forward as far as the primary says we can.

This probably came out of the similar case in append_log(), but notably
that roll_forward() only happens if !transaction_applied (i.e., backfill
target), and that condition is not checked here.

Fixes: http://tracker.ceph.com/issues/24597
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22989 from theanalyst/doc/releases/v10.2.11

doc: releases: release notes for v10.2.11 Jewel

Reviewed-By: Nathan Cutler <ncutler@suse.com>

qa: use root to find client admin socket

Permissions on /var/run/ceph changed post-Mimic.

Fixes: http://tracker.ceph.com/issues/24872
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22908 into master

* refs/pull/22908/head:
common: move ConfigValues out of md_config_t
common,rbd,rgw,osd: extract config values into ConfigValues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22953 into master

* refs/pull/22953/head:
ceph-bluestore-tool: remove path_* support from prime-osd-dir

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #22968 into master

* refs/pull/22968/head:
messages/MMDSMap: fix reencoding for older clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22969 into master

* refs/pull/22969/head:
qa/mgr: delete devicehealth pool after selftest

Reviewed-by: Sage Weil <sage@redhat.com>

doc: releases: release notes for v10.2.11

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

messages/MMDSMap: fix reencoding for older clients

This was broken by ea1481d08d561cdf0229e440b70b88fe336d3f89.

Fixes: http://tracker.ceph.com/issues/24819
Signed-off-by: Sage Weil <sage@redhat.com>

rpm: move make check-specific build deps to appropriate conditional block

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: remove universal requires from distro-specific conditional blocks

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22593 from wido/mgr-dashboard-ssl

mgr/dashboard: Add option to disable SSL

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add option to disable SSL

Although is preferred and should be enabled by default users might
want to disable SSL as the dashboard might be running behind a proxy
which terminates the SSL.

Fixes: https://tracker.ceph.com/issues/24674
Signed-off-by: Wido den Hollander <wido@42on.com>

os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Example scenario:
  ```
  A: BlueStore::Collection::split_cache(src, dest)
  A: std::lock(src->cache->lock, dest->cache->lock)

  B: SharedBlob::~SharedBlob
  B: waits coll->cache->lock // coll == src

  A: sb->coll := dest
  A: unlocks both src and dest's cache mutexes

  C: // any locked operation on dest's cache
  C: acquires dest->cache->lock
  C: begins the op

  B: // with the src's cache mutex acquired
  B: BufferSpace::_clear(coll->cache) // coll == dest
  B: // oops, B operates on cache already locked by C
  ```

Fixes: http://tracker.ceph.com/issues/24859
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #22784 into master

* refs/pull/22784/head:
doc: fix Ceph FS
doc: new label and glossary url
doc: drop linking to itself

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22144 into master

* refs/pull/22144/head:
cephfs-journal-tool: Fix purging when importing an zero-length journal.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22455 into master

* refs/pull/22455/head:
qa/ceph-volume: add a test for put_object_versioned()
ceph-volume-client: allow atomic updates for RADOS objects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22543 into master

* refs/pull/22543/head:
qa/tasks/cephfs: add test for ceph_abort_conn()
libcephfs: introduce ceph_get_instance_id()
client: introduce ceph_abort_conn()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22961 into master

* refs/pull/22961/head:
doc/cephfs: make scary DR bits less prominent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/mgr: delete devicehealth pool after selftest

This prevents tests getting hung up on the health
warnings from its very low pg count.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22921 from ceph/wip-default-ceph-volume

qa/tests: remove ceph-disk tests from ceph-deploy and default all tests to use ceph-volume

common: fix races when visiting a TrackedOp.

Before the commit following race can happen:
  ```
  A           : OpTracker::visit_ops_in_flight(..., callable leaking TrackedOpRef outside)
  A           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)
  A with lock : (nref > 0) == true

  B           : TrackedOp::put(), nref := 0 // updating the counter is done without the lock
  B           : OpTracker::unregister_inflight_op()
  B           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)

  A with lock : visit() -> TrackedOp::get(), nref := 1
  A with lock : Mutex::Locker::~Locker()

  B with lock : boost::intrusive::list::iterator_to(op)
  B with lock : boost::intrusive::list::erase(iter)
  B with lock : Mutex::Locker::~Locker()

  A           : TrackedOp::put(), nref := 0
  A           : OpTracker::unregister_inflight_op()
  A           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)
  A with lock : boost::intrusive::list::iterator_to(op) // oops as op doesn't belong to the list anymore
  ```

Fixes: https://tracker.ceph.com/issues/24664
Related-To: https://tracker.ceph.com/issues/24037
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: move ConfigValues out of md_config_t

by moving ConfigValues out of md_config_t, we are able to point
md_config_t to a new ConfigValues instance at runtime, without
destroying the old one.

also remove ConfigReader, because all md_config_obs_t will be updated
with ConfigProxy now. as md_config_t is not able to offer the *data*
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>