osdc/Objecter: invalidate crcs on preallocated rx buffers

Both simple and async messengers use c_str() when copying the data from
the socket into the receive buffer, going behind bufferlist's back.  If
the receive buffer is preallocated, we need to invalidate its crc cache
by hand to avoid possible data crc mismatches on the client side.

Fixes: https://tracker.ceph.com/issues/38416
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26483 from idryomov/wip-send-copyup-cleanups

librbd: send_copyup() fixes and cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26465 from dzafman/wip-38344

osd: Prevent negative local num_bytes sent to peer for backfill reser…

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: CopyupRequest::m_pending_copyups doesn't need to be atomic

It's protected by CopyupRequest::m_lock mutex.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't clone IoCtx in send_copyup()

Calling aio_operate() with the explicit snap_seq and snaps doesn't
mutate IoCtx, so no reason to clone it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: send copyup and hint only once

In the deep-copy case, don't send copyup and hint twice (once with the
blank and once with the current snapshot context).

Preserve the workaround for compare-and-write added in commit
f6db9b8027b6 ("librbd: copyup state machine needs to handle empty write
ops").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26519 from idryomov/wip-cmpext-bl

librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26262 into master

* refs/pull/26262/head:
mgr/orch/ssh: add qa test
qa: install/exclude ceph-mgr-ssh package
debian: package ssh orchestrator
spec.ceph: package ssh orchestrator
ssh/orch: add ssh orchestrator

Merge pull request #26197 from wanghao72/master

rgw: bug in versioning concurrent, list and get have consistency issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #26494 from cbodley/wip-rgw-datalog-autotrim

rgw: fix rgw_data_sync_info::json_decode()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/orch/ssh: add qa test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

qa: install/exclude ceph-mgr-ssh package

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

debian: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

spec.ceph: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

ssh/orch: add ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #26471 from tnaganawa/patch-3

ceph-deploy would not support --cluster option anymore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

test/librbd: add test cases for compare-and-write with copyup

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: make get_temp_image_name() static

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librados_test_stub: reimplement cmpext()

cmpext is a read operation.  As such, it shouldn't be creating objects
or extending them and it should accept snapshots.

The compare code is completely broken, resulting in false positives
(i.e. failures to detect mismatches) in many cases.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26299 from yuvalif/add_pubsub_tests

rgw/pubsub: add pubsub tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #26484 into master

* refs/pull/26484/head:
msg/async: msgr2: don't smash client cookie on reconnect
doc/msgr2: session establishment failure scenarios
msg/async: msgr2: don't smash peer_name and features on reconnect
msg/async: msgr2: added client and server cookie to protocol

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26417 into master

* refs/pull/26417/head:
qa, doc: update ec fast read setting
qa/suites: remove mon kv backend options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26489 into master

* refs/pull/26489/head:
qa/standalone/osd/osd-markdown: disable CLI command dups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26490 into master

* refs/pull/26490/head:
osd: fix typo in numa node check; lower debug level

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26492 into master

* refs/pull/26492/head:
osd: fix pg split/merge prime during startup

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26408 from tspmelo/wip-pool-slash

mgr/dashboard: Allow "/" in pool name

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25866 from yuvalif/add_amqp_push_endpoint_new

rgw/pubsub: add amqp push endpoint

Merge pull request #26433 from tspmelo/wip-rbd-slash

mgr/dashboard: Improve RBD form

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26443 from sebastian-philipp/orchestrator_fix_readme.md

mgr/orchestrator_cli: Fix README.md

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #26509 from p-na/wip-pna-fix-ganesha-pylint

mgr/dashboard: Clean up pylint's `disable:no-else-return`

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26478 from xiexingguo/wip-calc-upmap-vec-reserve

osd/OSDMap: using std::vector::reserve to reduce memory reallocation

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #26505 from xiexingguo/wip-balancer-weekday-range

mgr/balancer: add min/max fields for weekday and be compatible with C

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25138 from stackhpc/rdma-cm

msgr/async/rdma: Return from poll system call with EINTR should be retried

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: retry if poll() returns EINTR

Fixes: https://tracker.ceph.com/issues/37292
Signed-off-by: Stig Telfer <stig@stackhpc.com>

Merge pull request #26328 from badone/wip-update-ceph_stable_release-to-mimic

qa/ceph-ansible: Set ceph_stable_release to mimic

mgr/dashboard: Allow create and edit of pools with "/" in the name

Fixes: http://tracker.ceph.com/issues/38302
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25945 from trociny/wip-migration-parent

librbd: make it possible to migrate parent images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librados: don't claim ObjectOperation::cmpext() buffer

This is counter-intuitive and doesn't match the behavior of either
other compare ops (cmpxattr, omap_cmp) or write ops.

Fixes: http://tracker.ceph.com/issues/38383
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: pubsub tests. use bukcet checkpoint. allow redundant events

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/dashboard: Improve RBD form

Remove pools with '/' in the name and add RBD name validation.

Fixes: http://tracker.ceph.com/issues/38303
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace encodeURI for encodeURIComponent

Fixes: http://tracker.ceph.com/issues/38303
Signed-off-by: Tiago Melo <tmelo@suse.com>

msg/async: msgr2: don't smash client cookie on reconnect

Signed-off-by: Ricardo Dias <rdias@suse.com>

osd/osd_types: show cmpext arguments

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #26469 into master

* refs/pull/26469/head:
common/ceph_timer: stop timer's thread when it is suspended

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26470 into master

* refs/pull/26470/head:
mgr/MgrStandby: include addrs in metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26479 into master

* refs/pull/26479/head:
pybind: move FLAGS into Flag class

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26485 into master

* refs/pull/26485/head:
qa/suites/upgrade/luminous-x: force clone v1 format for final rbd python test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: wakeup lease_cr after abort on bucket sync disable

the 'bucket sync disable' tests were causing data sync to lag behind
because these calls to lease_cr->abort() were still waiting for the
RGWContinuousLeaseCR to wake up from its sleep. added data sync
checkpoints to each 'bucket sync disable' test to make sure that data
sync catches up before running other tests, as this was causing failures
in the new 'datalog autotrim' test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26452 from tchaikov/wip-crimson-send-known-addresses

crimson/osd: send known addresses at boot

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/msgr2: session establishment failure scenarios

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: don't smash peer_name and features on reconnect

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: added client and server cookie to protocol

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26450 from s0nea/wip-dashboard-readonly-config-options-2

mgr/dashboard: disallow editing read-only config options (part 2)

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25233 from p-na/wip-pna-qos-pr

mgr/dashboard: Add support for managing RBD QoS

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/balancer: make weekday be compatible with C

The __desc__ field of **begin[end]_weekend** suggests
sunday is the prevailing start of the week, whereas the
Python implementation says no.

See https://stackoverflow.com/questions/21045569/why-is-pythons-weekday-different-from-tm-wday-in-c

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/balancer: add min/max fields for begin[end]_weekday

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/dashboard: Clean up pylint's `disable:no-else-return`

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Render error in pool edit dialog

Fixes: http://tracker.ceph.com/issues/38004
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Add support for managing RBD QoS

Fixes: http://tracker.ceph.com/issues/36191
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #26405 from jtlayton/wip-jlayton-orch

Allow rook to scale the mon count

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #23221 from baixueyu/wip-rgw-cleanup-regex

rgw: delete some unused code about std::regex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

objclass: fix cls_get_snapset_seq for cache tier case

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: add parent migration case

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: allow to migrate parent if force flag set

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: add option to list all descendant images

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: use 'namespace' instead of 'namespace-name' in specs

to make them fit the screen

Signed-off-by: Mykola Golub <mgolub@suse.com>

crimson/osd: send known addresses in MOSDBoot

otherwise, the peers will drop the connection at seeing unmatched
address in banner.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: populate crc settings

we should populate messenger related settings before starting
messengers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: listen on v1 addresses

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26254 from tchaikov/wip-crimson-cache

crimson: cache osdmap using LRU cache

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rbd: allow 'children' command work if no snap specified

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add list_descendants API method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make list_children work if parent snap is head

(returning children for all the image snaps in this case)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make it possible to migrate parent images

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: replace attach child call with new state machine

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: new attach child state machine

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make AttachParentRequest support reattach

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: extend parent_attach to support reattach

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26378 from votdev/issue_38267

mgr/dashboard: Module dashboard.services.ganesha has several lint issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rgw: delete some unused code about std::regex

Signed-off-by: Xueyu Bai <baixueyu@inspur.com>

cmake: add PGMap.cc to crimson-common

otherwise we have failures like:

/usr/bin/ld: ../../../lib/libcrimson-common.a(Message.cc.o): in function
`PGMapDigest::~PGMapDigest()':
/var/ssd/ceph/src/mon/PGMap.h:37: undefined reference to `vtable for
PGMapDigest'
/usr/bin/ld: ../../../lib/libcrimson-common.a(Message.cc.o): in function
`PGMapDigest::~PGMapDigest()':

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: set_callback in ctor

there is no need to do so in `start()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: lookup using unordered_map first

assuming the unordered_map in LRU cache is faster.

inspired by #25480 by Radoslaw Zarzynski

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cache osdmaps using LRU cache

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: bug in versioning concurrent, list and get have consistency issue

Fixes: https://tracker.ceph.com/issues/38060
Signed-off-by: Wang Hao <wanghao72@baidu.com>

osd: Improve use of atomic operations to add and subtract from local_num_bytes

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #25570 from cbodley/wip-rgw-cls-bi-list-log-level

cls/rgw: raise debug level of bi_log_iterate_entries output

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw_multi: add test_datalog_autotrim()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add 'datalog autotrim' admin command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix rgw_data_sync_info::json_decode()

this was trying to decode a 'uint64_t instance_id' into the wrong
variable 'uint32_t num_shards'

datalog trimming relies on this json format, and decode failures here
prevent any trimming from taking place

Fixes: http://tracker.ceph.com/issues/38373
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26491 from ceph/revert-25926-wip-rgw-lc-prefix

Revert "rgw: lifcycle: don't reject compound rules with empty prefix"

osd: fix pg split/merge prime during startup

The priming of split/merge targets may modify pg_slots, an unordered_map,
which means our iterator may become invalid.  Put PGs in a temporary set
and use that instead.

Fixes: http://tracker.ceph.com/issues/36304
Signed-off-by: Sage Weil <sage@redhat.com>

Revert "rgw: lifcycle: don't reject compound rules with empty prefix"

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

osd: fix typo in numa node check; lower debug level

- s/to/do/
- no need for this on stderr, since they don't match on *most* systems
  it seems.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/osd-markdown: disable CLI command dups

The markdown test is based on marking down a specific number of times, but
the duplicate commands from the CLI may not get absorbed/batched by the
mon, breaking the test.  Override the default qa/tasks/workunit.py
behavior of sending dups.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26399 from jan--f/c-v-fix-hashable-py3

ceph-volume: make Device hashable to allow set of Device list in py3

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26401 from jan--f/c-v-fix-siez-format-py3

ceph-volume: implement __format__ in Size to format sizes in py3

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26403 from jan--f/c-v-no-iteritems-py3

ceph-volume: remove iteritems instances

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #26463 into master

* refs/pull/26463/head:
script/ptl-tool: allow invocation within a subdirectory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26272 from ivancich/wip-log-excl-lock-busy-info

rgw: when exclusive lock fails due existing lock, log add'l info

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>