qa/standalone/ceph-helpers: default pg autoscale mode off for standalone

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix objectstore_blackhole read-after-write

Instead of tossing out transactions from queue_transactions (which prevents
us from reading back the results), instead (1) stop doing any write IO, and
(2) stop trimming the cache.

Fixes: https://tracker.ceph.com/issues/40684
Signed-off-by: Sage Weil <sage@redhat.com>

test,misc: do not specify pg_num per pool

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: do not specify pg_num

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/ceph_volume_client: do not specify pg_num for new pools

Signed-off-by: Sage Weil <sage@redhat.com>

doc: remove all pg_num arguments to 'osd pool create'

Also, update the dicussion about pg_num and pool creation, with a reference
to the autoscaler.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: do not require pg_num to 'osd pool create'

Signed-off-by: Sage Weil <sage@redhat.com>

common: default pg_autoscale_mode=on for new pools

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30457 from dillaman/wip-41908

osd/PrimaryLogPG: do not append outdata to TMAPUP ops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #30427 into master

* refs/pull/30427/head:
doc/cephfs: correct a description mistake about mds states

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30432 from yanghonggang/wip-yhg-balancer-eval-maxmin

pybind/mgr/balancer/module.py: add max/min info in stats_by_root

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30479 from liewegas/wip-fix-osd-invalid-stats

osd/osd_types: fix {omap,hitset_bytes}_stats_invalid handling on split/merge

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29800 from Devp00l/wip-41372

mgr/dashboard: CephFS tab component

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30412 from tchaikov/wip-crimson-osd-make-message

osd,crimson: use make_message for creating message

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30408 from p-se/wip-pna-smartctl-report

osd: add common smartctl output to JSON output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30414 from neha-ojha/wip-fix-missing

osd/PeeringState.h: get_num_missing() should report num_missing()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30448 from tchaikov/wip-40871

mgr: set hostname in DeviceState::set_metadata()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30449 from tchaikov/wip-37884

mon,mgr: pass MessageRef to monc.send_mon_message()  …

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30413 from vumrao/wip-vumrao-41875

tools/rados: call pool_lookup() after rados is connected.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30409 from uweigand/cleanup-endian

src/: Clean up endian handling

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30383 from vumrao/wip-vumrao-tools-rados-add-pgid-in-help

tools/rados: add --pgid in help

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30211 from majianpeng/msg-clear-payload

msg/async: clear_payload when requeue_sent.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #30445 from tchaikov/wip-41320

mgr/dashboard: do not log tokens

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #29947 from iotcg/nosrq

support RDMA NIC without SRQ in msg/async/rdma

Reviewed-by: Roman Penyaev <rpenyaev@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30502 from idryomov/wip-krbd-avoid-hexdump-s

qa: avoid hexdump skip and length options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30489 from liuchang0812/kill-compile-warning

rgw: kill compile warnning in rgw_object_lock.h

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

qa: avoid hexdump skip and length options

On Debian-based distros, hexdump(1) comes from FreeBSD in bsdmainutils
package (other distros ship a Linux version from util-linux).  It turns
out that '-s skip' for anything but a regular file has been broken for
ages in FreeBSD -- instead of calling fseek(), it consumed the input
byte by byte with getchar().  This was fixed in commit 7cd5ae492557
"hexdump(1): Speed up -s flag on devices" last year, but it hasn't made
it to the Debian world yet.

bsdmainutils_9.0.6ubuntu3 in xenial had a custom patch for that, but
it's no longer there in bsdmainutils_11.1.2ubuntu1 in bionic.  hexdump
-s <large offset> is now a fancy infinite loop!

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: kill compile warnning in rgw_object_lock.h

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #30484 from trociny/wip-handle_start_replay_interrupted

rbd-mirror: potential nullptr dereference in ImageReplayer::handle_start_replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr: pass MessageRef to monc.send_mon_message()

less chance to leak memory

see also: https://tracker.ceph.com/issues/37884

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: include necessary #includes

ideally, a header file should be self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: add send_mon_message(MessageRef)

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MLog: pass rvalue ref instead const ref to ctor

easier to use this way. and so far, there is no caller of this function
yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard Adds multiple selection in osd table (#29662)

mgr/dashboard Adds multiple selection in osd table

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30438 from dillaman/wip-41833

cls/rbd: sanitize entity instance messenger version type

Reviewed-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: potential nullptr dereference in ImageReplayer::handle_start_replay

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #30474 from tchaikov/wip-discard-future

crimson: run in foreground if possible, silence warnings

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28971 from yuvalif/wip-yuval-pubsub-api2

rgw/pubsub: add S3 compliant API to master zone

Merge pull request #30101 from rishabh-d-dave/cv-minor

ceph-volume: api/lvm: check if list of LVs is empty

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19020 from zhangsw/feature-rgw-bucketsync-api

rgw: add admin rest api for bucket sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29637 from liuchang0812/rgw-es7

rgw: sync with elastic search v7

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29897 from tianshan/fix_list_start_with_null_marker

rgw: fix list versions starts with version_id=null

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29909 from cbodley/wip-40935

rgw: fix default storage class for get_compression_type

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30127 from zhangsw/rgw-fix-adminuser

rgw: add missing admin property when sync user info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30163 from zhangsw/feature-rgw-opslog-transid

rgw: add transaction id to ops log.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30175 from tianshan/improve-data-sync-retry

rgw: improve data sync restart after failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30189 from linuxbox2/wip-rgw-ss3lite

rgw: crypt: permit RGW-AUTO/default with SSE-S3 headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30194 from vumrao/wip-vumrao-41589

radosgw-admin: add --uid check in bucket list command

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #30332 from linuxbox2/wip-ldap-denied

rgw: ldap auth: S3 auth failure should return InvalidAccessKeyId

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30384 from ZVampirEM77/bucket-create-time

rgw: add creation time information into bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29680 from liuchang0812/s3-lock-bugfix

rgw: DefaultRetention requires either Days or Years

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

osd/osd_types: fix {omap,hitset_bytes}_stats_invalid handling on split/merge

Fixes: https://tracker.ceph.com/issues/41936
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30451 from ajarr/fix-41903

mgr/volumes: return string type to ceph-manager

mgr/dashboard: Add smartctl data as an details tab of an OSD (#30324)

mgr/dashboard: Add smartctl data as an details tab of an OSD

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

crimson/osd: suppress warnings about unused future<> return values

These are dropped on purpose, suppress warnings.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: ShardServices::send_pg_temp() returns future

it should not run in background unless necessary.

please note, we don't have a caller of this method yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: evict object state in foreground

when doing error handling, we need to make sure the stale object state
is evicted before we return.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add_peer() in parallel

we should not add peers in background, and we can do this in parallel!

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: Heartbeat::add_reporter_peers() returns future

the sole caller `Heartbeat::update_peers()` also returns future, so
there is no reason that `add_reporter_peers()` runs in background.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the order in OSD::update_heartbeat_peers()

before this chance, we do following things in background and in
parallel:

* add peers from pg map
* and peers from whole osdmap
* remove dead peers

but we should prioritize the peer candidates, apparently, the ones from
pgmap should have higher priority. as they are more likely to be used in
cluser and they are more important to current OSD, as they serve the
same set of PGs.

so we need to do these things in a proper order:

1. add peers from pg map
2. and peers from whole osdmap
3. remove dead peers

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/heartbeat: avoid racing when connecting to peer

before this change, we could initialize connections to newly added peers
in parallel repeatly, if an OSD is listed in up/acting of multiple PGs
served by current OSD.

after this change, we try to add the peer to `Heartbeat.peers` first,
and then try to connect to it with the hb front/back addresses, to avoid
potential repeatly connecting to peer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30270 from ricardoasmarques/wip-41727

mgr/dashboard: Check if `num_sessions` is available

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/volumes: return string type to ceph-manager

The 'volumes' ceph-mgr module returns a byte type instead of a string
type as the stdout result of commands, `fs subvolume getpath` and
`fs subvolumegroup getpath` to the ceph-mgr. This causes the commands
to hang and the ceph-mgr to utilize 100% CPU. Fix the 'volumes'
ceph-mgr module to return string type to ceph-mgr as the stdout
result of the commands.

Introduced-by: 3d63cd947f55a8508f972ee249ea6a04836429ae
Fixes: https://tracker.ceph.com/issues/41903
Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/osd: do not cluster_msgr.set_myaddrs() in background

we should do the initialization in foreground

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: avoid warning about signed comparison with enum classes

Signed-off-by: Samuel Just <sjust@redhat.com>

pybind/mgr/balancer/module.py: add max/min info in stats_by_root

$ ceph balancer eval-verbose hahapool
...
count_by_pool {'hahapool': {'objects': {0: 1L, 1: 1L, 2: 0L, 3: 0L, 4: 1L}, 'bytes': {0: 0L, 1: 0L, 2: 0L, 3: 0L, 4: 0L}, 'pgs': {0: 35, 1: 42, 2: 23, 3: 30, 4: 26}}}
...
stats_by_root {'default': {'objects': {'min': 0.0, 'max': 1.0, 'score': 0.2970089549436925, 'stddev': 0.5477225575051662, 'avg': 0.6, 'sum_weight': 0.6000000000000001}, 'bytes': {'min': 0, 'max': 0, 'score': 0, 'stddev': 0, 'avg': 0, 'sum_weight': 0}, 'pgs': {'min': 23.0, 'max': 42.0, 'score': 0.07354228523075643, 'stddev': 7.52994023880668, 'avg': 31.2, 'sum_weight': 0.4}}}

max/min can give us an intuitive feel of the pg/bytes/object's distribution.
In addition, sometimes we really want to know which osd hold the most
number of pgs/bytes/objects for specified pool.

Signed-off-by: Yang Honggang <yanghonggang@umcloud.com>

Merge pull request #30410 from tchaikov/wip-debian-g++

debian/control: Build-Depends on g++

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #30405 from p-se/wip-pna-ubuntu-disco-deps

install-deps.sh: add support for Ubuntu Disco Dingo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30453 from tchaikov/wip-doc-crimson

doc/dev/crimson: transpose options of compare.py

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #30440 from dzafman/wip-41900

Revert "common: default pg_autoscale_mode=on for new pools"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30458 from tchaikov/wip-crimson-cbt-test

test/crimson: lower the bar for cbt test

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard Adds multiple selection to osd table

Fixes: https://tracker.ceph.com/issues/38091
Signed-off-by: Pooja <pooja.gautam@ts.fujitsu.com>

Merge pull request #30433 from tchaikov/wip-crimson-pgls

crimson/osd: add pgls support

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add pgls_filter support

it's modeled after pgnls_filter. and use seastar::map_reduce() for
filtering the objects into the response.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: return hobject_t from pgls_filter()

so we can use it as the mapper function in seastar::map_reduce().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add pgls support

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle pg ops with do_pg_ops()

currently, OpsExecuter tries to load the object info for all ops to be
executed, but the PG ops do not require the existence of object
specified by op. so we need to specialize for them.

in this change,

* `OpsExecuter::execute_pg_op()` is added to handle pg ops without
  loading object info first.
* `pgnls` and `pgnls_filter` are handled by
  `OpsExecuter::execute_pg_op()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/OpsExecuter: pass nspace in msg to do_pg_op()

as pg ops should not require the object info, as there is chance that
the object specified in pg ops does not exist at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/OpsExecuter: s/do_osd_op/execute_osd_op/

will add execute_pg_op() for performing pg specific ops, which do not
need the object state.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: lower the bar for cbt test

sometimes we have

13:27:59 - WARNING  - cbt      -prefill/incerta08.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.05)):: 13.0/9.53939  => rejected

i think it's acceptable

Signed-off-by: Kefu Chai <kchai@redhat.com>

github: add "jenkins test crimson perf" to triggers doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rbd: sanitize entity instance messenger version type

The entity_addr_t::type field does not uniquely identify the connection
and might switch back and forth between LEGACY, V2, and ALL depending on
the client, the OSD, and the OSD map feature bits. Therefore, always
fix the entity address type to ANY prior to any comparisons.

Fixes: https://tracker.ceph.com/issues/41833
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: do not append outdata to TMAPUP ops

Write operations should not have data but TMAPUP incorrectly appends
the previously read TMAP object to the outdata buferlist.

Fixes: https://tracker.ceph.com/issues/41908
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30452 from idryomov/wip-krbd-discard-direct

qa: avoid page cache for krbd discard round off tests

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: add common smartctl output to JSON output

Enables consumers like the dashboard to provide the regular smartctl
output to users.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #30386 from tchaikov/wip-run_tox.sh-cleanup

script/run_tox.sh: remove unused code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29734 from hselin/rados_bench_rand_read

tools/rados: use num ops instead of num objs for tracking outstanding IO

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

headers: Remove __le16/__le32/__le64

These types should no longer be used; use ceph_le16/32/64 instead.

Also removes the cmake check whether those types are provided by
kernel headers (likewise for the __be types, which already were
not used anywhere).

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

headers: Remove le16/32/64_to_cpu

There are no remaining callers of these routines, which are no longer
required when using ceph_le16/32/64 (and were no-ops already).

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

tools: Clean up endian handling

Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation, or where objects are only used locally
and do not require endian swaps.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

crimson: Clean up endian handling

Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation.

Also, remove unnecessary calls to le16/32/64_to_cpu.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

osd: Clean up endian handling

Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation, or where objects are only used locally
and do not require endian swaps.

Also, remove unnecessary calls to le16/32/64_to_cpu.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

messages: Clean up endian handling

Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation.

Also, add missing calls to init_le32 in an initializer.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

msg: Clean up endian handling

Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation, or where objects are only used locally
and do not require endian swaps.

Also, remove unnecessary calls to le16/32/64_to_cpu.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

os: Clean up endian handling

Replace remaining uses of __le32 in Transaction.h with uint32_t.
These objects are only used locally and do not require endian swaps.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

mgr/dashboard: do not log tokens

sensitive information should not go to logging messages.

see https://tracker.ceph.com/issues/41320

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29767 from votdev/issue_41314

mgr/dashboard: Editing RGW bucket fails because of name is already in use

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/dev/crimson: transpose options of compare.py

they were `-a baseline -b test` before this change, and they should be
`-a test -b baseline`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30313 from cyx1231st/wip-seastar-msgr-test-racing-crimson

crimson/net: v2 racing tests, stall tests and bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: set hostname in DeviceState::set_metadata()

before this change, we don't update daemon's host if the daemon already
exists in the `daemon_state` registry when handling MMgrOpen even if the
hostname is updated and different from the existing one. and we set
daemon's metadata and update daemon's hostname when handling MMgrOpen or
MServiceMap.

after this change, daemon's hostname is always set whenever
`DaemonState::set_metadata()` is called.

Fixes: https://tracker.ceph.com/issues/40871
Signed-off-by: Kefu Chai <kchai@redhat.com>