crimson/osd: recovery locking uses ceph_assert_always().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement MOSDPGBackfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement BackfillRecoveryPipeline.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: improve debugs in PrimaryLogPG::do_scan().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement BackfillListener::backfilled().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement BackfillListener::enqueue_push().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: short-circuit budget_available().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire-up pg scanning on replicas.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire-up BackfillState and pg scanning on primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: introduce PGRecovery::start_backfill_recovery().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: settle BackfillState in PGRecovery.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PGRecovery provides dummy BackfillListener.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: make BackgroundRecovery abstract for the sake of backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: decouple backfill from PG::start_recovery_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: bring the boost::statechart-based FSM for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: refactor PrimaryLogPG::recover_backfill() a little bit.

This is change is intended to only refactor the code.
The main theme is just declaring the `backfill_pos` closer
to its usage.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: refactor PeeringState::find_best_info() a little bit.

This is change is intended to only refactor the code. All
it does is just declaring the `min_last_update_acceptable`
closer to its usage.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: sketch the backfill machine and its interfaces.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: send MOSDPGTemp on peering and map advance.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: rework handling of MOSDAlive.

1. Migrate `MOSDAlive` crafting from `OSD` to `ShardServices`.
2. `send_alive()` takes the `want` epoch from callers
   instead of assuming it matches current OSDMap.
   Callers pass `pg_history_t::same_interval_since`.
   The passed epoch is handled similarly to `queue_want_up_thru()`
   in the classical.
   The message is sent once and when the desired epoch is greater
   than the one returned by `OSDMap::get_up_thru()`.
3. Send the message also when completing the `PeeringEvent`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: link up the documentation for last_epoch_started.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement PG scanning.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/object_context: implement maybe_get_cached_obc().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/intrusive_lru: implement plain, non-creational get().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd/PrimaryLogPG: move BackfillInterval to recovery_types.

It will be needed in crimson as well.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add the infra for backend-agnostic recovery ops.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: drop unnecessary inclusion of MOSDPGScan.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd, osd: switch request_remote_recovery_reservation() to uptr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd, osd: switch PG::request_local_background_io_reservation() to uptr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: stuff PG::on_activate_complete() with debugs.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35940 from tchaikov/wip-ceph-dencoder-struct-ver

tools/ceph-dencoder: add get_struct_v,get_struct_compat commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35914 from tchaikov/wip-crimson-clang

crimson: clang related cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35933 from tchaikov/wip-interval-set

include/interval_set: use template as the 2nd template parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35496 into master

* refs/pull/35496/head:
client: introduce timeout for client shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35942 from tchaikov/wip-build-rtd

admin: move python-common deps out of doc-requirements.txt

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35354 from ishanrai05/rfs

mgr/dashboard: use relative font sizes

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

tools/ceph-dencoder: add get_struct_v,get_struct_compat commands

for understanding the current encoding version and the lower bound of
supported decoders.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35913 from ideepika/wip-container-tag-change

docker/podman: tag crimson flavor container as 'sha-1-flavor'

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35920 from uweigand/entity-addrvec-fix

msg/msg_types: entity_addrvec_t: fix decode on big-endian hosts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35890 from matthewoliver/cephadm_ipv6_list_networks

cephadm: Make list_networks ipv6 enabled

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35546 from agayev/zoned-fm

os/bluestore: Introduce zoned freelist manager.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

admin: move python-common deps out of doc-requirements.txt

to address the build failure of

python -m pip install --exists-action=w --no-cache-dir -r admin/doc-requirements.txt
ERROR: ../src/python-common is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with svn+, git+, hg+, or bzr+).

Signed-off-by: Kefu Chai <kchai@redhat.com>

docker/podman: tag crimson flavor container as 'sha-1-flavor'

see also: ceph/ceph-container#1704

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #35936 from tchaikov/wip-cmake-seastar-cxxflags

cmake: append to Seastar_CXX_FLAGS

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35812 from tchaikov/wip-more-log-for-45717

mgr/dashboard: print osd log as well

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: assign to bufferlist not to bufferlist*

this change addresses a regression introduced by a23a1929f5

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: append to Seastar_CXX_FLAGS

instead of overwriting Seastar_CXX_FLAGS, append to it. this change
allows user to set Seastar_CXX_FLAGS in the command line.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35610 from pcuzner/wip-grafana-container

monitoring: add grafana container build file

Merge pull request #35935 from tchaikov/wip-crimson-no-more-claim

crimson/osd: use operator=(...) instead of claim()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: use operator=(...) instead of claim()

to silence warnings like

objclass.cc:300:26: warning: 'void ceph::buffer::v15_2_0::list::claim(ceph::buffer::v15_2_0::list&)' is deprecated: in favor of operator=(list&&) [-Wdeprecated-declarations]
  300 |   outbl->claim(op.outdata);
      |                          ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35921 from uweigand/test_cdc-endianfix

unittest_cdc: fix test on big-endian hosts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/common: link unittest_interval_set against GTest::Main

its main() is offered by GTet::Main library

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: use template as the 2nd template parameter

to avoid repeating the unit for the interval in the internal map's
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35932 from zdover23/wip-doc-cephadm-suites-typo

doc/cephadm: index.rst: s/suites/suits/ typo fixed.

doc/cephadm: index.rst: s/suites/suits/ typo fixed.

This commit changes the sentence "Choose the method that best suites
your needs." to "Choose the method that best suits your needs."

Fixes: https://tracker.ceph.com/issues/46359
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35275 from AmnonHanuhov/wip-objclass_impl

crimson/osd: Add support for cls_log

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35871 from tchaikov/wip-45717

mon/PGMap: do not consider changing pg stuck

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35864 from tchaikov/wip-osdc-claim

osdc: use operator=(...) instead of claim()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35863 from wuxingyi/removeunused

osd: Remove unused function declearation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35818 from majianpeng/osd-replicatedbackend-redundant-parameter

osd/ReplicatedBackend: remove redundant parameter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35817 from tchaikov/wip-drop-py2

qa,test,pybind: drop py2 support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

pybind/mgr/telegraf: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/selftest: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/influx: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/devicehealth: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: drop six dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_objectstore_tool: drop six dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/barbican: drop six dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/workunit: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/s3tests: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/rbd: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/radosgw_admin: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/osd_failsafe_enospc: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/{ceph,ceph_manager}: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: drop py2 support

the python bindings are compiled using python3-cython, and packaged as
python3-{rados,rbd,cephfs,rgw}, so there is no need and is wrong to
claim that they are python2 compatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: s/assert/ceph_abort_msg/ when hitting unreachable code

clang++ complains with
```
crimson/os/seastore/root_block.h:100:3: warning: control reaches end of non-void function [-Wreturn-type]
  }
  ^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: silence -Wunused-lambda-capture warnings

they are pointed out by clang++.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: make FixedKVNodeLayout::set_val() public

otherwise clang++ fails with
```
mson/os/seastore/lba_manager/btree/lba_btree_node_impl.cc:183:10: error: 'set_val' is a private member of 'crimson::common::FixedKVNodeLayout<255, unsigned long, ceph_le<unsigned long long>,
crimson::os::seastore::paddr_t, crimson::os::seastore::paddr_le_t, true>::iter_t<false>'
      i->set_val(updated);
         ^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-master/src/crimson/common/fixed_kv_node_layout.h:148:10: note: declared private here
    void set_val(V val) const {
         ^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35855 from dillaman/wip-45986

librbd: new 'write_zeroes' API methods to suppliment the `discard` APIs

Reviewed-by: Mykola Golub <mgolub@suse.com>

unittest_cdc: fix test on big-endian hosts

The specific_result test hardcodes a particular output.  However,
the input buffer used by the test is currently initialized to a
different byte string depending host the host byte order, which
makes the expected result not match on big-endian hosts.

Fixed by always initializing the input buffer to the same string.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

msg/msg_types: entity_addrvec_t: fix decode on big-endian hosts

When decoding an entity_addrvec_t with marker 1, we just have
a single (non-legacy) entity_addr_t.  This should be decoded
exactly the same as done by entity_addr_t::decode, but it
currently is not.  Specifically, the sa_family member of
the sockaddr is not converted from the on-wire little-endian
format to host byte order (as done by entity_addr_t::decode).

Fixed by using the same code as in entity_addr_t::decode.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge pull request #35859 from tspmelo/wip-remove-ngx-bootstrap

mgr/dashboard: Remove ngx-bootstrap

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/dev: Update zoned-storage.rst to indicate that OSDs can now be written to after restart on zoned devices.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

os/bluestore: Introduce zoned freelist manager.

Introduced ZonedFreelistManager, which tracks the write pointer and the number
of dead bytes per zone.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

Merge pull request #35875 from Devp00l/wip-46303

mgr/dashboard: Fixes regression in device selection modal

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35912 from tchaikov/wip-crimson-osd-flush-pg-stats

crimson/osd: implement flush_pg_stats command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>