mgr/dashboard: Ensure E2E tests can be run independently

Introduces the PageHelper.

- Reads class attributes from Helpers to reduce boilerplace code for
  navigation.
- The PageHelper is supposed to be the new class for code that's reused
  across all Helpers for page related code.
- The Helper class is by now meant to be used for non-page specific
  helper code, like used in the `checkConsole` method.
- The Helper class will act as central registry to enable all other
  tests to use helper functions of all derived PageHelper tests.

  Example:

  `Helper.pools.create('foobar', ...);`

Fixes: http://tracker.ceph.com/issues/40397
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Cleanup HACKING.rst

Rephrase explanations on how to write E2E tests with Protractor.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #28085 from rzarzynski/wip-crimson-no-atomics-in-refcntobj

crimson, common: RefCountedObj doesn't use atomics in Seastar builds.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27999 from trociny/wip-rbd-sparse_copyup

librbd: clone copy-on-write operations should preserve sparseness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28071 from tchaikov/wip-crimson-perf-regres

crimson,osd: performance fixes

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson, common: RefCountedObj doesn't use atomics in SeaStar builds.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #27270 from theanalyst/beast-fe-ipv6

rgw: normalize v6 endpoint behaviour for the beast frontend

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26281 from smithfarm/wip-git-describe-long

make-dist: set version number only once

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28045 from cbodley/wip-39635

cls/user: cls_user_set_buckets_info overwrites creation_time

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27980 from cbodley/wip-24991

rgw: delete_obj_index() takes mtime for bilog

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28031 from ivancich/wip-rgw-admin-unordered

rgw: allow radosgw-admin to list bucket w --allow-unordered

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27676 from mkogan1/wip-rgw-39380

rgw: fix default_placement containing "/" when storage_class is standard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27984 from cbodley/wip-test-rgw-multisite-sh

test/rgw: fix test-rgw-multisite.sh script for creating multisite clusters

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #27807 from cyb70289/utf8-encode

common: optimize encode_utf8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28051 from dillaman/wip-39521

librbd: fix issues with object-map/fast-diff feature interlock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28063 from dillaman/wip-39656

qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@suse.com>

cmake: define BOOST_SP_DISABLE_THREADS for crimson

because we are single-threaded

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/osd_internal_types: bring moving semantics to ObjectState.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: simplify write handling in PG::do_osd_ops.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: clone copy-on-write operations should preserve sparseness

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27870 from theanalyst/rgw-objexp-fixes-cli

rgw: object expirer fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27926 from tchaikov/wip-38627

pybind/mgr: convert str to int using "int()"

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #27914 from dzafman/wip-39304

osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27961 from dzafman/wip-39582

osd: Output Base64 encoding of CRC header if binary data present

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Fix merge_log_dups() not remove dup corresponding to log.tail

The log tail is not inclusive of the log, so not an overlap
Fix unittest_pglog to verify proper operation

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Additional test cases for trim, copy_up_to and copy_after

Signed-off-by: David Zafman <dzafman@redhat.com>

test: unittest_pglog: Remove irrelevant config values

osd_min_pg_log_entries and osd_max_pg_log_entries are not involved
in the log handling code being tested.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Adding standalone test of log copy handling

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27782 into master

* refs/pull/27782/head:
os/bluestore: protect BlueFS::FileReader::buf
os/bluestore: log 'buffered' in KernelDevice::read_random
os/bluestore: support RocksDB prefetching at BlueFS
os/bluestore: call BlueFS::_init_logger earlier.
tools/ceph-objectstore-tool: dump perf counters on competion.
os/bluestore: optimize mapping offset to extent in BlueFS.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27928 into master

* refs/pull/27928/head:
mon/OSDMonitor: require nautilus to set pg_autoscale_mode
mon/OSDMonitor: allow pg_num to increase when require_osd_release < N

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27877 into master

* refs/pull/27877/head:
mon: paxos: introduce new reset_pending_committing_finishers for safety

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21722 from mslovy/wip-parial-recovery

osd: partial recovery strategy based on PGLog

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28060 from ceph/wip-bz1666822

ceph-volume: use the Device.rotational property instead of sys_api

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: skip missing interpreters when running tox tests

This will skip tests for versions of python that don't exist on the node
that's running the tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

qa/workunits/rbd: use https protocol for devstack git operations

Fixes: http://tracker.ceph.com/issues/39656
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: require nautilus to set pg_autoscale_mode

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: allow pg_num to increase when require_osd_release < N

Setting pg_num_target has no effect when we encode our OSDMap with mimic
or luminous features.  Fall back to the pre-nautilus behavior of directly
setting pg_num.

Fixes: http://tracker.ceph.com/issues/39570
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27795 from cbodley/wip-rgw-beast-async-body

rgw: beast frontend uses yield_context to read/write body

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Added Object Gateway menu tab tests (#27565)

mgr/dashboard: Added Object Gateway menu tab tests

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #28042 from smithfarm/wip-doc-ceph-backport-sh

doc: improve ceph-backport.sh comment block

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: improve ceph-backport.sh comment block

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28048 from athanatos/sjust/wip-peering-state-cleanup

More PeeringState and related cleanups to ease use in crimson

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #27811 into master

* refs/pull/27811/head:
commom: make cluster_network work.

Reviewed-by: Sage Weil <sage@redhat.com>

osd: Include dups in copy_after() and copy_up_to()

Client saw out of order results in a test with a very small
pg_log because a backfill/recovery sent a small number of
log entries for duplicate checks.

Since these copy operations are equivalent to both a copy
and trim (an argument controls how many log entries
transfer), we need to include any new dups with enough
existing dups up to the configured maximum.

Fixes: http://tracker.ceph.com/issues/39304
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #28050 from dillaman/wip-39448

msg/async: reduce verbosity of connection timeout failures

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/: Move PG::_prepare_write_info into osd_types prepare_info_keymap

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: convert boost::optional users to std::optional

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState: switch remaining boost::optional users

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: Clean up PeeringCtx ownership semantics and pointer/ref usage

PeeringCtx was a bit of a mess in terms of tracking down where the
resources were cleaned up.  This patch:
- Reworks PeeringCtx to directly own its maps/transaction
- Introduces PeeringCtxWrapper to deal with the message buffering
  internally
- Cleans up users to avoid passing pointers through the PeeringState
  interface unnecessarily.

This'll allow the PeeringCtx destructor to do its job vastly
simplifying usage in crimson.

Signed-off-by: Samuel Just <sjust@redhat.com>

os: break Transaction of of Objectstore, use in crimson

Transaction is needed in crimson, but not ObjectStore.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

src/osd: refactor to use ostream_temp

The concept of emitting log lines to a central mon log is likely
to be useful for both crimson and classic, but the mechanism is
likely to be different.  Break out OstreamTemp for use in interface.

Signed-off-by: Samuel Just <rexludorum@gmail.com>

PeeringState: use HBHandle

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: include MOSDPGLog.h

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: create interface for TPHandle, remove WorkQueue include from MissingLoc

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move perf counter constants and constructors

PeeringState requires the same counters in crimson, but we
need to avoid OSD.h include in PeeringState.

Signed-off-by: Samuel Just <sjust@redhat.com>

PeeringState::ActivateCommitted: intialize epoch

Fixes: https://tracker.ceph.com/issues/39636
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #28053 from liu-chunmei/ceph_seastar_bugfix

crimson/osd: fix the build broken by df771861

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28041 from tchaikov/wip-crimson-v1-auth

crimson/mon: fix the v1 auth

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: can't ‘operator<’ ‘ceph_release_t’ and ‘int’

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

librbd: fix issues with object-map/fast-diff feature interlock

Enabling/disabling one should enable/disable the other.

Fixes: http://tracker.ceph.com/issues/39521
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #27717 into master

* refs/pull/27717/head:
mds: output lock state in format dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28009 from ricardoasmarques/rbd-feature-migrating

Add 'RBD_FEATURE_MIGRATING' to rbd.pyx

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: rgw: document v6 endpoint options

document that v6 endpoints only binds to ipv6

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: beast: bind both v4 and v6 ports when using a port argument

This commit adds binding to both v4 and v6 port when specifying a port argument,
endpoint *only* binds to v4 or v6 depending on the address specified. Failure to
bind when the protocol is not supported is not treated as an error and we warn
and continue then.

Fixes: http://tracker.ceph.com/issues/39038
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add a note on reshard object expirer fixes in PendingReleaseNotes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add a troubleshooting note on resharding expirer fixes

Added a note about the commands added to support stale expired object deletion

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

msg/async: reduce verbosity of connection timeout failures

Long running clients connected to thrashing OSDs could result in a
"see no progress in more than <timeout>" message printed to stderr.
This is not an error but can result in test failures when console
output is compared against expected output.

Fixes: http://tracker.ceph.com/issues/39448
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27680 from theanalyst/aws-rest-fixes

rgw: rest client fixes for cloud sync XML outputs

Reviewed-By: Casey Bodley <cbodley@redhat.com>

librados_test_stub: update get_min_compatible_osd to octopus

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: provide async version of copyup method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: allow to provide sparse read result

Signed-off-by: Mykola Golub <mgolub@suse.com>

osdc/striper: make possible to assemble sparse read result

Signed-off-by: Mykola Golub <mgolub@suse.com>

librados_test_stub: pass snap context to "create" method

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: add sparse_copyup method

Signed-off-by: Mykola Golub <mgolub@suse.com>

os/bluestore: protect BlueFS::FileReader::buf

Now this buffer might be accessed from BlueRocksRandomAccessFile
which is intended for multi-threading access. Hence we need a
proper protection for the buffer.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: make max_num_intervals of ObjectCleanRegions a configuration option

Signed-off-by: lishuhao <lishuhao@unitedstack.com>

mgr/dashboard: Remove messages.xlf (#28020)

mgr/dashboard: Remove messages.xlf

Reviewed-by: Tiago Melo <tmelo@suse.com>

test/rgw: add multisite test_bucket_creation_time()

does not create any buckets itself, but checks that all creation times
match on buckets created by other tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: allow radosgw-admin to list bucket w --allow-unordered

Presently the `radosgw-admin bucket list --bucket=<bucket>` lists the
objects in lexical order. This can be an expensive operation since
objects are not stored in bucket index shards in order and a selection
sort process is done across all bucket index shards.

By allowing the user to add the "--allow-unordered" command-line flag,
a more efficient bucket listing is enabled. This is particularly
important for buckets with a large number of objects.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27976 from mykaul/patch-2

doc: Update link to Red Hat documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon: run tick() periodically

also send keepalive to monitor in tick()

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: log 'buffered' in KernelDevice::read_random

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: support RocksDB prefetching at BlueFS

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: call BlueFS::_init_logger earlier.

So one can use perf counters during log replay.

Signed-off-by: Igor Fedotov <ifedotov@susu.com>

tools/ceph-objectstore-tool: dump perf counters on competion.

Performed  if debug mode is enabled

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: optimize mapping offset to extent in BlueFS.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Remove messages.xlf

Removes the messages.xlf file from the git repository, due to a lot of merge conflicts.
It won't cause any problems, because the file is not important for building or the translation during runtime.

Fixes: https://tracker.ceph.com/issues/38003
Signed-off-by: Sebastian Krah <skrah@suse.com>

crimson/mon: fix the v1 auth

* initialize mon::Connection::global_id to `0` in the ctor.
  global_id is assigned by monitor, a zero global_id implies
  that "allocate me a new global_id please". so we should
  never use a random number on stack for the global_id.
* do not use a magic number for initializing MAuth::protocol,
  use `CEPH_AUTH_UNKNOWN` instead.
* do not try to dereference `auth` before creating it. `auth`
  is created by `create_auth()`. in which, the global_id
  is always assigned to `auth->global_id`, as it's a new session.
  so we should just call `create_auth()`.
* restructure the code to finish the authentication. to remove
  the find_if(), and use the `parallel_for_each()` loop for
  both set the active_con and closing the other pending connections.
* ask for the rotating keyring once gets authenticated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: output lock state in format dump

dump cache in plain text will print lock state. But in json format dump,
it won't. It is not convenient to debug some MDS lock issues without
such information.

Fixes: http://tracker.ceph.com/issues/39645
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #26654 from sebastian-philipp/orchestrator-progress

mgr/orchestrator: add progress events to all orchestrators

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

doc: Update link to Red Hat documentation

The previous link was to version 2, which is a bit older. Fix to version 3.

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Merge pull request #27855 from tchaikov/wip-ceph-release

src/: define ceph_release_t and use it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27689 from Jeegn-Chen/wip-39390-fix

os/filestore: assure sufficient leaves in pre-split

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27925 from tchaikov/wip-38628

mgr/restful: do not use filter() for list

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #27962 from xiexingguo/wip-jira-3756

mgr/DaemonServer: safe-to-destroy - do not consider irrelevant pgs

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28036 from neha-ojha/wip-fix-ar-doc

doc/dev/osd_internals/async_recovery: update cost calculation

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28034 from letterwuyu/forward_incompat_pg_missing

ceph-object-corpus: forward_incompat pg_missing_item and pg_missing_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-object-corpus: forward_incompat pg_missing_item and pg_missing_t

Signed-off-by: lishuhao <lishuhao@unitedstack.com>

doc/dev/osd_internals/async_recovery: update cost calculation

Signed-off-by: Neha Ojha <nojha@redhat.com>