mgr/dashboard: Bug Fix Error on Mgr Dashboard NFS

This change fixes error on NFS Ganesha management dashboard that throws an error when request daemon list.

Fixes: https://tracker.ceph.com/issues/39709
Signed-off-by: Nur Faizin <nur.faizin91@gmail.com>

Merge pull request #27870 from theanalyst/rgw-objexp-fixes-cli

rgw: object expirer fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27926 from tchaikov/wip-38627

pybind/mgr: convert str to int using "int()"

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #27914 from dzafman/wip-39304

osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27961 from dzafman/wip-39582

osd: Output Base64 encoding of CRC header if binary data present

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Fix merge_log_dups() not remove dup corresponding to log.tail

The log tail is not inclusive of the log, so not an overlap
Fix unittest_pglog to verify proper operation

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Remove irrelevant config values

osd_min_pg_log_entries and osd_max_pg_log_entries are not involved
in the log handling code being tested.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Adding standalone test of log copy handling

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27782 into master

* refs/pull/27782/head:
os/bluestore: protect BlueFS::FileReader::buf
os/bluestore: log 'buffered' in KernelDevice::read_random
os/bluestore: support RocksDB prefetching at BlueFS
os/bluestore: call BlueFS::_init_logger earlier.
tools/ceph-objectstore-tool: dump perf counters on competion.
os/bluestore: optimize mapping offset to extent in BlueFS.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27928 into master

* refs/pull/27928/head:
mon/OSDMonitor: require nautilus to set pg_autoscale_mode
mon/OSDMonitor: allow pg_num to increase when require_osd_release < N

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27877 into master

* refs/pull/27877/head:
mon: paxos: introduce new reset_pending_committing_finishers for safety

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21722 from mslovy/wip-parial-recovery

osd: partial recovery strategy based on PGLog

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28060 from ceph/wip-bz1666822

ceph-volume: use the Device.rotational property instead of sys_api

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: skip missing interpreters when running tox tests

This will skip tests for versions of python that don't exist on the node
that's running the tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

mon/OSDMonitor: require nautilus to set pg_autoscale_mode

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: allow pg_num to increase when require_osd_release < N

Setting pg_num_target has no effect when we encode our OSDMap with mimic
or luminous features.  Fall back to the pre-nautilus behavior of directly
setting pg_num.

Fixes: http://tracker.ceph.com/issues/39570
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27795 from cbodley/wip-rgw-beast-async-body

rgw: beast frontend uses yield_context to read/write body

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Added Object Gateway menu tab tests (#27565)

mgr/dashboard: Added Object Gateway menu tab tests

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #28042 from smithfarm/wip-doc-ceph-backport-sh

doc: improve ceph-backport.sh comment block

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: improve ceph-backport.sh comment block

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28048 from athanatos/sjust/wip-peering-state-cleanup

More PeeringState and related cleanups to ease use in crimson

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #27811 into master

* refs/pull/27811/head:
commom: make cluster_network work.

Reviewed-by: Sage Weil <sage@redhat.com>

osd: Include dups in copy_after() and copy_up_to()

Client saw out of order results in a test with a very small
pg_log because a backfill/recovery sent a small number of
log entries for duplicate checks.

Since these copy operations are equivalent to both a copy
and trim (an argument controls how many log entries
transfer), we need to include any new dups with enough
existing dups up to the configured maximum.

Fixes: http://tracker.ceph.com/issues/39304
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #28050 from dillaman/wip-39448

msg/async: reduce verbosity of connection timeout failures

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/: Move PG::_prepare_write_info into osd_types prepare_info_keymap

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: convert boost::optional users to std::optional

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: switch remaining boost::optional users

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: Clean up PeeringCtx ownership semantics and pointer/ref usage

PeeringCtx was a bit of a mess in terms of tracking down where the
resources were cleaned up.  This patch:
- Reworks PeeringCtx to directly own its maps/transaction
- Introduces PeeringCtxWrapper to deal with the message buffering
  internally
- Cleans up users to avoid passing pointers through the PeeringState
  interface unnecessarily.

This'll allow the PeeringCtx destructor to do its job vastly
simplifying usage in crimson.

Signed-off-by: Samuel Just <sjust@redhat.com>

os: break Transaction of of Objectstore, use in crimson

Transaction is needed in crimson, but not ObjectStore.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

src/osd: refactor to use ostream_temp

The concept of emitting log lines to a central mon log is likely
to be useful for both crimson and classic, but the mechanism is
likely to be different.  Break out OstreamTemp for use in interface.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

PeeringState: use HBHandle

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: include MOSDPGLog.h

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: create interface for TPHandle, remove WorkQueue include from MissingLoc

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move perf counter constants and constructors

PeeringState requires the same counters in crimson, but we
need to avoid OSD.h include in PeeringState.

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState::ActivateCommitted: intialize epoch

Fixes: https://tracker.ceph.com/issues/39636
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #28053 from liu-chunmei/ceph_seastar_bugfix

crimson/osd: fix the build broken by df771861

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28041 from tchaikov/wip-crimson-v1-auth

crimson/mon: fix the v1 auth

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: can't ‘operator<’ ‘ceph_release_t’ and ‘int’

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge PR #27717 into master

* refs/pull/27717/head:
mds: output lock state in format dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28009 from ricardoasmarques/rbd-feature-migrating

Add 'RBD_FEATURE_MIGRATING' to rbd.pyx

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: add a note on reshard object expirer fixes in PendingReleaseNotes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add a troubleshooting note on resharding expirer fixes

Added a note about the commands added to support stale expired object deletion

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

msg/async: reduce verbosity of connection timeout failures

Long running clients connected to thrashing OSDs could result in a
"see no progress in more than <timeout>" message printed to stderr.
This is not an error but can result in test failures when console
output is compared against expected output.

Fixes: http://tracker.ceph.com/issues/39448
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27680 from theanalyst/aws-rest-fixes

rgw: rest client fixes for cloud sync XML outputs

Reviewed-By: Casey Bodley <cbodley@redhat.com>

os/bluestore: protect BlueFS::FileReader::buf

Now this buffer might be accessed from BlueRocksRandomAccessFile
which is intended for multi-threading access. Hence we need a
proper protection for the buffer.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: make max_num_intervals of ObjectCleanRegions a configuration option

Signed-off-by: lishuhao <lishuhao@unitedstack.com>

mgr/dashboard: Remove messages.xlf (#28020)

mgr/dashboard: Remove messages.xlf

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27976 from mykaul/patch-2

doc: Update link to Red Hat documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon: run tick() periodically

also send keepalive to monitor in tick()

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: log 'buffered' in KernelDevice::read_random

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: support RocksDB prefetching at BlueFS

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: call BlueFS::_init_logger earlier.

So one can use perf counters during log replay.

Signed-off-by: Igor Fedotov <ifedotov@susu.com>

tools/ceph-objectstore-tool: dump perf counters on competion.

Performed  if debug mode is enabled

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: optimize mapping offset to extent in BlueFS.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Remove messages.xlf

Removes the messages.xlf file from the git repository, due to a lot of merge conflicts.
It won't cause any problems, because the file is not important for building or the translation during runtime.

Fixes: https://tracker.ceph.com/issues/38003
Signed-off-by: Sebastian Krah <skrah@suse.com>

crimson/mon: fix the v1 auth

* initialize mon::Connection::global_id to `0` in the ctor.
  global_id is assigned by monitor, a zero global_id implies
  that "allocate me a new global_id please". so we should
  never use a random number on stack for the global_id.
* do not use a magic number for initializing MAuth::protocol,
  use `CEPH_AUTH_UNKNOWN` instead.
* do not try to dereference `auth` before creating it. `auth`
  is created by `create_auth()`. in which, the global_id
  is always assigned to `auth->global_id`, as it's a new session.
  so we should just call `create_auth()`.
* restructure the code to finish the authentication. to remove
  the find_if(), and use the `parallel_for_each()` loop for
  both set the active_con and closing the other pending connections.
* ask for the rotating keyring once gets authenticated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: output lock state in format dump

dump cache in plain text will print lock state. But in json format dump,
it won't. It is not convenient to debug some MDS lock issues without
such information.

Fixes: http://tracker.ceph.com/issues/39645
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #26654 from sebastian-philipp/orchestrator-progress

mgr/orchestrator: add progress events to all orchestrators

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc: Update link to Red Hat documentation

The previous link was to version 2, which is a bit older. Fix to version 3.

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Merge pull request #27855 from tchaikov/wip-ceph-release

src/: define ceph_release_t and use it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27689 from Jeegn-Chen/wip-39390-fix

os/filestore: assure sufficient leaves in pre-split

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27925 from tchaikov/wip-38628

mgr/restful: do not use filter() for list

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #27962 from xiexingguo/wip-jira-3756

mgr/DaemonServer: safe-to-destroy - do not consider irrelevant pgs

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28036 from neha-ojha/wip-fix-ar-doc

doc/dev/osd_internals/async_recovery: update cost calculation

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28034 from letterwuyu/forward_incompat_pg_missing

ceph-object-corpus: forward_incompat pg_missing_item and pg_missing_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-object-corpus: forward_incompat pg_missing_item and pg_missing_t

Signed-off-by: lishuhao <lishuhao@unitedstack.com>

doc/dev/osd_internals/async_recovery: update cost calculation

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #27688 into master

* refs/pull/27688/head:
pybind: added lseek()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: Remove unused pg_log_t::copy_range()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix off by 1 error in trim handling of osd_pg_log_dups_tracked

Move log message out of loop so it doesn't repeat for no reason

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27482 into master

* refs/pull/27482/head:
client: fix fuse client hang because its bad session PipeConnection

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27761 into master

* refs/pull/27761/head:
qa: use skipTest method instead of exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27480 into master

* refs/pull/27480/head:
mds: check dir fragment to split dir if mkdir makes it oversized.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #27765 from linuxbox2/wip-rgw-crytpexcept

rgw: crypto: throw DigestException from Digest and HMAC

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27843 into master

* refs/pull/27843/head:
os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27716 into master

* refs/pull/27716/head:
cephfs-shell: fix string decoding for ls command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28013 from cbodley/wip-39611

rgw: use explicit to_string() overload for boost::string_ref

Reviewed-by: Nathan Cutler <nculter@suse.com>

mgr/dashboard: fix help message for minimum blob size in pool form (#28017)

mgr/dashboard: fix help message for minimum blob size in pool form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #27813 from ricardoasmarques/do-not-log-iscsi-get-requests

mgr/dashboard: iSCSI GET requests should not be logged

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #27946 from ricardoasmarques/iscsi-form-ipv6

mgr/dashboard: Fix iSCSI form when using IPv6

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: iSCSI GET requests should not be logged

Fixes: https://tracker.ceph.com/issues/39492
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Add 'RBD_FEATURE_MIGRATING' to rbd.pyx

Fixes: http://tracker.ceph.com/issues/39609
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/orchestrator: add progress events to all orchestrators

Now, progress events are part of `WriteCompletion` istead of part of the orchestrator module.

It does not yet provide a way to just show orchestrator events.

Also fixes issue in the SSH orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #27873 from rzarzynski/wip-crimson-writes

crimson: add support for basic write path

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/thread: include reactor.hh to avoid linking error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add support for basic write path, part 2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make ObjectState mutable. Evict mutated state on error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: implement bits needed for basic write path.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27413 from liu-chunmei/ceph_seastar_io_read

crimson/osd: support write pid_file when osd start

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix help message for minimum blob size in pool form

A value less than maximum blob size is required.

Fixes: http://tracker.ceph.com/issues/39624
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

commom: make cluster_network work.

This temporary parameter make address is zero. So make cluster_addr is
equal public_addr and make cluster_network disable.

Fixes: https://tracker.ceph.com/issues/39623
Reported-by: Jiang Xu <xu.jiang@dccsh.icbc.com.cn>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #27665 from badone/wip-improve-dump_pgstate_history-json

osd: Improve dump_pgstate_history json output

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28015 from tchaikov/wip-install-deps.sh

install-deps.sh: install libboost-test for seastar

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore: fix missing discard in BlueStore::_kv_sync_thread

Discard comand should be issued before space released.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/39621

install-deps.sh: install libboost-test for seastar

see
https://github.com/scylladb/seastar/blob/5858fe30c16a1a059200c129d20180ceb0199d99/cmake/SeastarDependencies.cmake#L58-L68

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: support write pid_file when osd start

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #27979 from yuriw/wip-yuriw-crontab-master

qa/test: reduce over all number of runs

Merge pull request #27941 from dzafman/wip-39011

doc: Describe recovery and backfill prioritizations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>