crimson/os/seastore: fix internal relative btree addrs

Internal relative addrs must be block relative on disk and record
relative in deltas.  Use maybe_generate_relative as intended.

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_transaction_manager: add a test which forces an lba split

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_btree_lba_manager: ensure record and block addrs are different

Adds an extra extent to each transaction in the test to ensure that
record and block relative addresses differ as they do in real usage.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #35725 into master

* refs/pull/35725/head:
client: Fix setxattr with 0 size specified

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35861 from dillaman/wip-write-zeroes-tweaks

librbd: allow thick-zeroing of an image extent

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #35743 into master

* refs/pull/35743/head:
qa/tasks/test_nfs: Add test for cluster info
mgr/volumes/nfs: Add cluster show info command

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35834 from yuvalif/add-amqp-topic-doc

rgw: add a doc hint that the topic-name is used as AMQP routing-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35923 from votdev/fix_various_issues

mgr/dashboard: Fix bugs in a unit test and i18n translation

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35886 from yuvalif/wip-yuval-fix-46316

rgw: make RGWToken string ctor explicit

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35707 from Devp00l/wip-45650

mgr/dashboard: Fixes ICU selection regression

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35764 from rzarzynski/wip-bl-cache-in-release

common/bl: document and slightly optimize ptr::release().

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/bl: document and slightly optimize ptr::release().

Before the patch GCC was generating:
```
  10f9910:   mov    (%rdi),%rax
  10f9913:   test   %rax,%rax
  10f9916:   je     10f9951 <ceph::buffer::v15_2_0::ptr::release()+0x41>
  10f9918:   mov    0x2c(%rax),%eax
  10f991b:   cmp    $0x1,%eax
  10f991e:   jne    10f9940 <ceph::buffer::v15_2_0::ptr::release()+0x30>
  10f9920:   mov    (%rdi),%rax
  10f9923:   movq   $0x0,(%rdi)
  10f992a:   test   %rax,%rax
  10f992d:   je     10f9958 <ceph::buffer::v15_2_0::ptr::release()+0x48>
  10f992f:   mov    (%rax),%rdx
  10f9932:   mov    %rax,%rdi
  10f9935:   mov    0x8(%rdx),%rdx
  10f9939:   jmpq   *%rdx
  10f993b:   nopl   0x0(%rax,%rax,1)
  10f9940:   mov    (%rdi),%rax
  10f9943:   lock subl $0x1,0x2c(%rax)
  10f9948:   je     10f9920 <ceph::buffer::v15_2_0::ptr::release()+0x10>
  10f994a:   movq   $0x0,(%rdi)
  10f9951:   retq
  10f9952:   nopw   0x0(%rax,%rax,1)
  10f9958:   retq
  10f9959:   nop
  10f995a:   nopw   0x0(%rax,%rax,1)
```

after:
```
  11209b0:   mov    (%rdi),%rax
  11209b3:   movq   $0x0,(%rdi)
  11209ba:   test   %rax,%rax
  11209bd:   je     11209e6 <ceph::buffer::v15_2_0::ptr::release()+0x36>
  11209bf:   mov    0x2c(%rax),%edx
  11209c2:   lea    0x2c(%rax),%rcx
  11209c6:   cmp    $0x1,%edx
  11209c9:   jne    11209e0 <ceph::buffer::v15_2_0::ptr::release()+0x30>
  11209cb:   mov    (%rax),%rdx
  11209ce:   mov    %rax,%rdi
  11209d1:   mov    0x8(%rdx),%rdx
  11209d5:   jmpq   *%rdx
  11209d7:   nopw   0x0(%rax,%rax,1)
  11209de:
  11209e0:   lock subl $0x1,(%rcx)
  11209e4:   je     11209cb <ceph::buffer::v15_2_0::ptr::release()+0x1b>
  11209e6:   retq
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: add a doc hint that the topic-name is used as AMQP routing-key

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

qa/tasks/test_nfs: Add test for cluster info

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Add cluster show info command

Fixes: https://tracker.ceph.com/issues/45743
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35940 from tchaikov/wip-ceph-dencoder-struct-ver

tools/ceph-dencoder: add get_struct_v,get_struct_compat commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35914 from tchaikov/wip-crimson-clang

crimson: clang related cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35933 from tchaikov/wip-interval-set

include/interval_set: use template as the 2nd template parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35496 into master

* refs/pull/35496/head:
client: introduce timeout for client shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35942 from tchaikov/wip-build-rtd

admin: move python-common deps out of doc-requirements.txt

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35354 from ishanrai05/rfs

mgr/dashboard: use relative font sizes

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

tools/ceph-dencoder: add get_struct_v,get_struct_compat commands

for understanding the current encoding version and the lower bound of
supported decoders.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35913 from ideepika/wip-container-tag-change

docker/podman: tag crimson flavor container as 'sha-1-flavor'

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35920 from uweigand/entity-addrvec-fix

msg/msg_types: entity_addrvec_t: fix decode on big-endian hosts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35890 from matthewoliver/cephadm_ipv6_list_networks

cephadm: Make list_networks ipv6 enabled

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35546 from agayev/zoned-fm

os/bluestore: Introduce zoned freelist manager.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

admin: move python-common deps out of doc-requirements.txt

to address the build failure of

python -m pip install --exists-action=w --no-cache-dir -r admin/doc-requirements.txt
ERROR: ../src/python-common is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with svn+, git+, hg+, or bzr+).

Signed-off-by: Kefu Chai <kchai@redhat.com>

docker/podman: tag crimson flavor container as 'sha-1-flavor'

see also: ceph/ceph-container#1704

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

mgr/dashboard: Fix bugs in a unit test and i18n translation

* SKIP: Unknown argument type: 'BinaryExpression' NodeObject { ... text: 'You can activate the module on the Telemetry configuration '
* WARNING: /ceph/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.spec.ts:39:32 - get is deprecated: from v9.0.0 use TestBed.inject

Fixes: https://tracker.ceph.com/issues/46371
Signed-off-by: Volker Theile <vtheile@suse.com>

client: Fix setxattr with 0 size specified

When xattrs are set with 0 size, the value should be set to an empty string.
Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #35936 from tchaikov/wip-cmake-seastar-cxxflags

cmake: append to Seastar_CXX_FLAGS

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35812 from tchaikov/wip-more-log-for-45717

mgr/dashboard: print osd log as well

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: assign to bufferlist not to bufferlist*

this change addresses a regression introduced by a23a1929f5

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: append to Seastar_CXX_FLAGS

instead of overwriting Seastar_CXX_FLAGS, append to it. this change
allows user to set Seastar_CXX_FLAGS in the command line.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35610 from pcuzner/wip-grafana-container

monitoring: add grafana container build file

rbd: switch thick-provision image creation to use new write_zeroes API

Also fix some of the concurrency and chunk size issues when handling
fancy striped images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add flag to force thick-provisioning during write_zeroes API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35935 from tchaikov/wip-crimson-no-more-claim

crimson/osd: use operator=(...) instead of claim()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: use operator=(...) instead of claim()

to silence warnings like

objclass.cc:300:26: warning: 'void ceph::buffer::v15_2_0::list::claim(ceph::buffer::v15_2_0::list&)' is deprecated: in favor of operator=(list&&) [-Wdeprecated-declarations]
  300 |   outbl->claim(op.outdata);
      |                          ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35921 from uweigand/test_cdc-endianfix

unittest_cdc: fix test on big-endian hosts

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/common: link unittest_interval_set against GTest::Main

its main() is offered by GTet::Main library

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: use template as the 2nd template parameter

to avoid repeating the unit for the interval in the internal map's
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35932 from zdover23/wip-doc-cephadm-suites-typo

doc/cephadm: index.rst: s/suites/suits/ typo fixed.

doc/cephadm: index.rst: s/suites/suits/ typo fixed.

This commit changes the sentence "Choose the method that best suites
your needs." to "Choose the method that best suits your needs."

Fixes: https://tracker.ceph.com/issues/46359
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35275 from AmnonHanuhov/wip-objclass_impl

crimson/osd: Add support for cls_log

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35871 from tchaikov/wip-45717

mon/PGMap: do not consider changing pg stuck

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35864 from tchaikov/wip-osdc-claim

osdc: use operator=(...) instead of claim()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35863 from wuxingyi/removeunused

osd: Remove unused function declearation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35818 from majianpeng/osd-replicatedbackend-redundant-parameter

osd/ReplicatedBackend: remove redundant parameter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35817 from tchaikov/wip-drop-py2

qa,test,pybind: drop py2 support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

pybind/mgr/telegraf: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/status: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/selftest: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/pg_autoscaler: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/influx: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/devicehealth: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: drop six dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_objectstore_tool: drop six dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/barbican: drop six dependency

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/workunit: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/s3tests: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/rbd: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/radosgw_admin: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/osd_failsafe_enospc: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/{ceph,ceph_manager}: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: drop py2 support

the python bindings are compiled using python3-cython, and packaged as
python3-{rados,rbd,cephfs,rgw}, so there is no need and is wrong to
claim that they are python2 compatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: drop py2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: s/assert/ceph_abort_msg/ when hitting unreachable code

clang++ complains with
```
crimson/os/seastore/root_block.h:100:3: warning: control reaches end of non-void function [-Wreturn-type]
  }
  ^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: silence -Wunused-lambda-capture warnings

they are pointed out by clang++.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: make FixedKVNodeLayout::set_val() public

otherwise clang++ fails with
```
mson/os/seastore/lba_manager/btree/lba_btree_node_impl.cc:183:10: error: 'set_val' is a private member of 'crimson::common::FixedKVNodeLayout<255, unsigned long, ceph_le<unsigned long long>,
crimson::os::seastore::paddr_t, crimson::os::seastore::paddr_le_t, true>::iter_t<false>'
      i->set_val(updated);
         ^
/home/jenkins-build/build/workspace/ceph-perf-crimson/ceph-master/src/crimson/common/fixed_kv_node_layout.h:148:10: note: declared private here
    void set_val(V val) const {
         ^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35855 from dillaman/wip-45986

librbd: new 'write_zeroes' API methods to suppliment the `discard` APIs

Reviewed-by: Mykola Golub <mgolub@suse.com>

unittest_cdc: fix test on big-endian hosts

The specific_result test hardcodes a particular output.  However,
the input buffer used by the test is currently initialized to a
different byte string depending host the host byte order, which
makes the expected result not match on big-endian hosts.

Fixed by always initializing the input buffer to the same string.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

msg/msg_types: entity_addrvec_t: fix decode on big-endian hosts

When decoding an entity_addrvec_t with marker 1, we just have
a single (non-legacy) entity_addr_t.  This should be decoded
exactly the same as done by entity_addr_t::decode, but it
currently is not.  Specifically, the sa_family member of
the sockaddr is not converted from the on-wire little-endian
format to host byte order (as done by entity_addr_t::decode).

Fixed by using the same code as in entity_addr_t::decode.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

Merge pull request #35859 from tspmelo/wip-remove-ngx-bootstrap

mgr/dashboard: Remove ngx-bootstrap

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/dev: Update zoned-storage.rst to indicate that OSDs can now be written to after restart on zoned devices.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

os/bluestore: Introduce zoned freelist manager.

Introduced ZonedFreelistManager, which tracks the write pointer and the number
of dead bytes per zone.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

Merge pull request #35875 from Devp00l/wip-46303

mgr/dashboard: Fixes regression in device selection modal

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35912 from tchaikov/wip-crimson-osd-flush-pg-stats

crimson/osd: implement flush_pg_stats command

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35374 from jschmid1/activate_osdspec_affinity

mgr/cephadm: re-enable osdspec affinity passing

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: avoid `-Wunused-but-set-variable` warning

in release build, `found` is not used at all. but GCC complains with a
`-Wunused-but-set-variable` warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35704 from Devp00l/wip-46135

mgr/dashboard: Fixes typeahead regression in the silence matcher

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/osd: update the osd stats with hb peers

just to align with classic OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement flush_pg_stats command

this command is used by teuthology based tests for updating the mgr with
the lastes pg stats on OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/{mgr,osd}: add WithStats::update_stats()

because we need to implement a tell command which forces osd to
send the latest pg stats to mgr, and the command returns the user with
the sequence id of the report, and `mgr::Client::report()` does not
return a future, so we have to update the seq id before sending
the report. the solution is to update the seq id in a separated
method, so in this change:

* add `const` to `WithStats::get_stats()
* add a dedicated method of `WithStats::update_stats()` to update
  the stats to be collected by mgr::Client.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35887 from joscollin/wip-update-kyr

.githubmap: add Kyr

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #35845 from tchaikov/wip-rpm-crimson

rpm: package crimson-osd in ceph-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #35867 from jwilk-forks/spelling

doc/dev/osd_internals: fix typos

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephadm: Make list_networks ipv6 enabled

Currently the list_network command and methods in cephadm only run and
parse ipv4 output from `ip route`.

This patch extends the list_network command and internal methods to be
ipv6 enabled. It now also checks `ip -6 route` and `ip -6 addr` to
gather gather all networks from both protocol families.

Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #35894 from tchaikov/wip-crimson-os

crimson/os: pin the last cpu core for the alien worker threads

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>