osd/PGLog: allow pg log trim when complete_to is less than trim_to

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: reset complete_to when trimming the log past it

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: allow trim() to proceed when there are missing items

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: handle trim() during backfill and async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: add flag to remember async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: print pg log length and trim_to

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd: make calc_trim_to() independent of min_last_complete_ondisk

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #23068 into master

* refs/pull/23068/head:
doc: fix some errors in the cephfs-journal-tool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22737 from trociny/wip-24675

librbd: validate data pool for self-managed snapshot support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: fix some errors in the cephfs-journal-tool

Signed-off-by: Zhu Shangzhong <zhu.shangzhong@zte.com.cn>

Merge PR #23059 into master

* refs/pull/23059/head:
common/ceph_strings: add request_flush_mdlog op names to ceph_session_op_name()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22211 into master

* refs/pull/22211/head:
cephfs-journal-tool: get and check layout info during dump/undump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22705 into master

* refs/pull/22705/head:
Changed Lua library support from all to partial
mds/Mantle: Fixed Mantle's obsolete Lua C++ API

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22751 into master

* refs/pull/22751/head:
test: add tests for ceph_utime() and friends
libcephfs: extend support for changing file access/modification time

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22801 into master

* refs/pull/22801/head:
client: add inst to asok status output

Reviewed-by: Venky Shankar <vshankar@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22804 into master

* refs/pull/22804/head:
mds: remove useless dout_subsys macro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: validate data pool for self-managed snapshot support

Fixes: https://tracker.ceph.com/issues/24675
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22945 from cxytt/master

librbd:optionally support FUA (force unit access) on write requests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22637 from zy751713126/resize_equal_size

rbd: error if new size is equal to original size

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20112 from liyichao/aio_fail

librbd: handle aio failure in ManagedLock and PreReleaseRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23046 from tchaikov/wip-ftbfs-rgw-kefu

rgw: fix FTBFS introduced by abca9805

rgw: fix FTBFS introduced by abca9805

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22809 from tchaikov/wip-with-seastar

run-make-check: enable --with-seastar option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22964 from tchaikov/wip-import-pub-key-without-hkp-server

install-deps.sh: import ubuntu-toolchain-r's key without keyserver

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #22971 into master

* refs/pull/22971/head:
qa: optionally use old fs name during cephfs remount
qa: remove dead code
qa: run asok command on correct machine

qa: optionally use old fs name during cephfs remount

This fixes errors caused by remount done by some tests (test_recovery_pool.py)
where the fs name is not given.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove dead code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: run asok command on correct machine

The MDS may not be on the same machine where the cluster command is run.

Fixes: http://tracker.ceph.com/issues/24858
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21885 into master

* refs/pull/21885/head:
qa: update cluster log health warning message
qa: add tests for client features
mds: evict clients that lack required features
mds: cleanup MDSRank::evict_client
mds: infer client version by client metadata and connection's features
mds: introduce "ceph fs set <fs_name> min_compat_client <release_name>"
mds: tell client why it's rejected
mds: introduce cephfs' own feature bits
mds: make Server::prepare_force_open_sessions() update client metadata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23018 into master

* refs/pull/23018/head:
qa: use 4 mds cluster for multifs tests

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: use 4 mds cluster for multifs tests

Fixes: http://tracker.ceph.com/issues/24899
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update cluster log health warning message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23033 from linuxbox2/wip-rgw-409

rgw: do not ignore EEXIST in RGWPutObj::execute

Merge pull request #23020 from tchaikov/wip-config-value-friend

common/config_values: friend md_config_impl<>

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: do not ignore EEXIST in RGWPutObj::execute

The existing logic appears able to cause propagation of a failed
exclusive create to the client, when it should instead have been
retried, due to disagreement about the logical write offset. (The
value of ofs here could be > 0 due to the operation of a stacked
write filter [e.g., compressor], when the RADOS write offset was
0 and hence an exclusive write that should be retried).

Rationale for fix by Casey.

http://tracker.ceph.com/issues/22790

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rbd-ggate: filter out ceph options from args

we can just filter out ceph options from command line args instead of
parsing config in env variables, conf files and argvs.

the config-parsing steps were modeled after the ones in rbd-nbd.cc, but
the whole purpose of reading configurations from env/conf-file is to get
the "rbd_default_pool" for setting cfg->poolname. apparently, this does
not apply to rbd-ggate, where global_init() will take care of all the
trivial work to populate the settings. moreover, `config` is not
read after being initialized in main.cc. so drop it.

Signed-off-by: Mykola Golub <mgolub@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config_values: friend md_config_impl<>

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22995 into master

* refs/pull/22995/head:
qa: use root to find client admin socket

Reviewed-by: Nathan Cutler <ncutler@suse.com>

common/ceph_strings: add request_flush_mdlog op names to ceph_session_op_name()

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

qa: add tests for client features

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: evict clients that lack required features

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #22938 from tchaikov/wip-bundled-dpdk

cmake: do not depend on ${DPDK_LIBRARIES} if not using bundled dpdk

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: cleanup MDSRank::evict_client

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: infer client version by client metadata and connection's features

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: introduce "ceph fs set <fs_name> min_compat_client <release_name>"

The command set the oldest version of client that is allowed to connect
to mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: tell client why it's rejected

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: introduce cephfs' own feature bits

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/14456

mds: make Server::prepare_force_open_sessions() update client metadata

The function is used for opening sessions for importing caps. It didn't
set client metadata for new sessions.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #22594 from wido/mgr-pg-states

mgr: Centralize PG_STATES to MgrModule

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reveiwed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22952 from tchaikov/wip-cmake-atomic

cmake: should link against libatomic if libcxx/libstdc++ does not off…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: link libcrimson against libceph-common

as libcrimson uses symbols exposed by the later, among other things, the
Authorization related stuff.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove unittest_seastar_{messenger,echo} from tests

unittest_seastar_echo cannot be run as unittest w/o starting two
instances.
unittest_seastar_messenger times out. need to RCA it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: extract SubProcess impl into .cc

cleanup the header dependencies and speed-up the compiling

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up fixes to address build failures

see also https://groups.google.com/forum/#!topic/seastar-dev/uEMYGgYJKOY

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: enable WITH_SEASTAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,install-deps.sh: add --with-seastar option

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: include seastar headers with 'seastar/' prefix

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd:optionally support FUA (force unit access) on write requests

Fixes: http://tracker.ceph.com/issues/19366
Signed-off-by: ningtao <63358@sangfor.com>

Merge PR #22877 into master

* refs/pull/22877/head:
common: fix races when visiting a TrackedOp.

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #22946 into master

* refs/pull/22946/head:
os/bluestore: batch handle get_fragmentation.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #22950 into master

* refs/pull/22950/head:
rpm: move make check-specific build deps to appropriate conditional block
rpm: remove universal requires from distro-specific conditional blocks

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22972 into master

* refs/pull/22972/head:
os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22974 into master

* refs/pull/22974/head:
qa/standalone/osd/ec-error-rollforward: reproduce bug 24597
qa/suites/rados/thrash-erasure-code: add many deletes workload
qa/standalone/osd/repro_long_log.sh: fix test
osd/PG: do not blindly roll forward to log.head

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/standalone/osd/ec-error-rollforward: reproduce bug 24597

This reproduces http://tracker.ceph.com/issues/24597

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-erasure-code: add many deletes workload

Having lots of deletes will mean deletes on objects that don't exist,
which will in turn mean error log entries and more coverage of the
append_log_entries_update_missing code.  Hopefully this will trigger
http://tracker.ceph.com/issues/24597

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/repro_long_log.sh: fix test

The log trimming case wasn't quite right.  Before HEAD^ we were
rolling forward too aggressively and miscalculating the can_rollforward_to,
which affected the trim_to calculation.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: do not blindly roll forward to log.head

If we are told we can roll forward by the primary, we should only roll
forward as far as the primary says we can.

This probably came out of the similar case in append_log(), but notably
that roll_forward() only happens if !transaction_applied (i.e., backfill
target), and that condition is not checked here.

Fixes: http://tracker.ceph.com/issues/24597
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22989 from theanalyst/doc/releases/v10.2.11

doc: releases: release notes for v10.2.11 Jewel

Reviewed-By: Nathan Cutler <ncutler@suse.com>

qa: use root to find client admin socket

Permissions on /var/run/ceph changed post-Mimic.

Fixes: http://tracker.ceph.com/issues/24872
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22908 into master

* refs/pull/22908/head:
common: move ConfigValues out of md_config_t
common,rbd,rgw,osd: extract config values into ConfigValues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22953 into master

* refs/pull/22953/head:
ceph-bluestore-tool: remove path_* support from prime-osd-dir

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #22968 into master

* refs/pull/22968/head:
messages/MMDSMap: fix reencoding for older clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22969 into master

* refs/pull/22969/head:
qa/mgr: delete devicehealth pool after selftest

Reviewed-by: Sage Weil <sage@redhat.com>

doc: releases: release notes for v10.2.11

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

messages/MMDSMap: fix reencoding for older clients

This was broken by ea1481d08d561cdf0229e440b70b88fe336d3f89.

Fixes: http://tracker.ceph.com/issues/24819
Signed-off-by: Sage Weil <sage@redhat.com>

rpm: move make check-specific build deps to appropriate conditional block

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: remove universal requires from distro-specific conditional blocks

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22593 from wido/mgr-dashboard-ssl

mgr/dashboard: Add option to disable SSL

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add option to disable SSL

Although is preferred and should be enabled by default users might
want to disable SSL as the dashboard might be running behind a proxy
which terminates the SSL.

Fixes: https://tracker.ceph.com/issues/24674
Signed-off-by: Wido den Hollander <wido@42on.com>

os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Example scenario:
  ```
  A: BlueStore::Collection::split_cache(src, dest)
  A: std::lock(src->cache->lock, dest->cache->lock)

  B: SharedBlob::~SharedBlob
  B: waits coll->cache->lock // coll == src

  A: sb->coll := dest
  A: unlocks both src and dest's cache mutexes

  C: // any locked operation on dest's cache
  C: acquires dest->cache->lock
  C: begins the op

  B: // with the src's cache mutex acquired
  B: BufferSpace::_clear(coll->cache) // coll == dest
  B: // oops, B operates on cache already locked by C
  ```

Fixes: http://tracker.ceph.com/issues/24859
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #22784 into master

* refs/pull/22784/head:
doc: fix Ceph FS
doc: new label and glossary url
doc: drop linking to itself

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22144 into master

* refs/pull/22144/head:
cephfs-journal-tool: Fix purging when importing an zero-length journal.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22455 into master

* refs/pull/22455/head:
qa/ceph-volume: add a test for put_object_versioned()
ceph-volume-client: allow atomic updates for RADOS objects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22543 into master

* refs/pull/22543/head:
qa/tasks/cephfs: add test for ceph_abort_conn()
libcephfs: introduce ceph_get_instance_id()
client: introduce ceph_abort_conn()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22961 into master

* refs/pull/22961/head:
doc/cephfs: make scary DR bits less prominent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/mgr: delete devicehealth pool after selftest

This prevents tests getting hung up on the health
warnings from its very low pg count.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22921 from ceph/wip-default-ceph-volume

qa/tests: remove ceph-disk tests from ceph-deploy and default all tests to use ceph-volume

common: fix races when visiting a TrackedOp.

Before the commit following race can happen:
  ```
  A           : OpTracker::visit_ops_in_flight(..., callable leaking TrackedOpRef outside)
  A           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)
  A with lock : (nref > 0) == true

  B           : TrackedOp::put(), nref := 0 // updating the counter is done without the lock
  B           : OpTracker::unregister_inflight_op()
  B           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)

  A with lock : visit() -> TrackedOp::get(), nref := 1
  A with lock : Mutex::Locker::~Locker()

  B with lock : boost::intrusive::list::iterator_to(op)
  B with lock : boost::intrusive::list::erase(iter)
  B with lock : Mutex::Locker::~Locker()

  A           : TrackedOp::put(), nref := 0
  A           : OpTracker::unregister_inflight_op()
  A           : Mutex::Locker::Locker(sdata->ops_in_flight_lock_sharded)
  A with lock : boost::intrusive::list::iterator_to(op) // oops as op doesn't belong to the list anymore
  ```

Fixes: https://tracker.ceph.com/issues/24664
Related-To: https://tracker.ceph.com/issues/24037
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: move ConfigValues out of md_config_t

by moving ConfigValues out of md_config_t, we are able to point
md_config_t to a new ConfigValues instance at runtime, without
destroying the old one.

also remove ConfigReader, because all md_config_obs_t will be updated
with ConfigProxy now. as md_config_t is not able to offer the *data*
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,rbd,rgw,osd: extract config values into ConfigValues

this change introduce three classes: ConfigValues, ConfigProxy and
ConfigReader. in seastar port of OSD, each CPU shard will hold its own
reference of configuration, and upon changes of settings, each
shard will be updated with the new setting in async. so this forces us
to be able to keep two set of configuration at the same time. so we
need to extract the changeable part of md_config_t out. so we can
replace the old one with new one on demand, and let different shards
share the same unchanged part, amon the other things, the Options map
and the lookup tables. that's why we need ConfigValues. we will add
a policy template for this class, so we can specialize for Seastar
implementation to allow different ConfigProxy instances to point
md_config_impl<> to different ConfigValues.

because the observer interface is still using md_config_t, to minimise
the impact of this change, handle_conf_change() and
handle_subsys_change() are not changed. but as it accepts a `const
md_config_t`, which cannot be used to create/reference the ConfigProxy
holding it, we need to introduce ConfigReader for reading the updated
setting from md_config_t in a simpler way, without exposing the
internal "values" member variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22940 into master

* refs/pull/22940/head:
mon/OSDMonitor: add 'osd repeer <pgid>' command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22949 into master

* refs/pull/22949/head:
cmake: cleanup FindCython.cmake

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #22954 into master

* refs/pull/22954/head:
global/global_init: fix stdout/stderr/stdin closing for daemonization

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22931 from tchaikov/wip-find-py-interp-for-gtest-parallel

cmake: find a python2 interpreter for gtest-parallel

Reviewed-by: Erwan Velu <erwan@redhat.com>

install-deps.sh: import ubuntu-toolchain-r's key without keyserver

we are running into following error recently

gpg: requesting key BA9EF27F from hkp server keyserver.ubuntu.com
gpgkeys: key 1E9377A2BA9EF27F can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: keyserver communications error: keyserver helper general error
gpg: keyserver communications error: unknown pubkey algorithm
gpg: keyserver receive failed: unknown pubkey algorithm

so use the local copy of the pubkey instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: make scary DR bits less prominent

I'm sure people will still find them, but let's at least
force people to click through one more time to get to the
commands that can damage your cluster.

Also, the ".. danger" directive at the top of the page
wasn't actually getting special formatting, so I changed
it to a ".. warning" which is red.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22944 from falcon78921/wip-doc-rbdcmds

doc: fixed grammar in restore rbd image section

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add 'osd repeer <pgid>' command

Selecting force peering on a single PG.  In reality this probably induces
*2* interval changes.

Note that in the case of a single OSD cluster we can't actually force a
repeer on a single PG because the pg_temp code is pretty robust about
filtering out redundant or meaningless changes, so we can't pg_temp our
way into a new interval if there are no other OSDs to switch to and the
code also prevents an empty pg_temp.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22711 from fghaas/wip-doc-mimic-upgrade-prereq

 doc: Mention PURGED_SNAPDIRS and RECOVERY_DELETES in Mimic release notes

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>