pybind: auto encode/decode cstr

assuming all string in the cythonized string use the legacy python2
string type, instead of using the unicode, and assuming all c strings
are ASCII, so we don't need encode/decode them when coerce from/to C
strings.

this helps when generating py3 bindings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: set language_level for cythonize explicitly

Compiling rbd.pyx because it changed.
[1/1] Cythonizing rbd.pyx
/usr/lib/python2.7/dist-packages/Cython/Compiler/Main.py:367:
FutureWarning: Cython directive 'language_level' not set, using 2 for
now (Py2). This will change in a later re
lease! File: /var/ssd/ceph/src/pybind/rbd/rbd.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

this warning is raised by cython 0.29.2

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26402 from tchaikov/wip-install-deps

 install-deps.sh: selectively install dependencies

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #26551 into master

* refs/pull/26551/head:
osd/PG: fix get debug refcount line

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26511 into master

* refs/pull/26511/head:
mgr: Selftest fails, dashboard/server_port not string

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #26513 into master

* refs/pull/26513/head:
osd/PG: fix comments in merge_from
osd/PG: align past_intervals and last_epoch_clean for fabricated merge target

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26526 into master

* refs/pull/26526/head:
rpm,deb: remove btrfs-tools as a build-dependency
rpm,deb: remove bc as a build-dependency

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #26536 into master

* refs/pull/26536/head:
src/tools: only build dedup tool if WITH_TESTS is set

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/PG: fix get debug refcount line

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26483 from idryomov/wip-send-copyup-cleanups

librbd: send_copyup() fixes and cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26465 from dzafman/wip-38344

osd: Prevent negative local num_bytes sent to peer for backfill reser…

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: CopyupRequest::m_pending_copyups doesn't need to be atomic

It's protected by CopyupRequest::m_lock mutex.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't clone IoCtx in send_copyup()

Calling aio_operate() with the explicit snap_seq and snaps doesn't
mutate IoCtx, so no reason to clone it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: send copyup and hint only once

In the deep-copy case, don't send copyup and hint twice (once with the
blank and once with the current snapshot context).

Preserve the workaround for compare-and-write added in commit
f6db9b8027b6 ("librbd: copyup state machine needs to handle empty write
ops").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26519 from idryomov/wip-cmpext-bl

librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26262 into master

* refs/pull/26262/head:
mgr/orch/ssh: add qa test
qa: install/exclude ceph-mgr-ssh package
debian: package ssh orchestrator
spec.ceph: package ssh orchestrator
ssh/orch: add ssh orchestrator

Merge pull request #26197 from wanghao72/master

rgw: bug in versioning concurrent, list and get have consistency issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #26494 from cbodley/wip-rgw-datalog-autotrim

rgw: fix rgw_data_sync_info::json_decode()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/orch/ssh: add qa test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

qa: install/exclude ceph-mgr-ssh package

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

debian: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

spec.ceph: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

ssh/orch: add ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #26471 from tnaganawa/patch-3

ceph-deploy would not support --cluster option anymore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd/PG: fix comments in merge_from

Some of these were stale.

Signed-off-by: Sage Weil <sage@redhat.com>

test/librbd: add test cases for compare-and-write with copyup

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: make get_temp_image_name() static

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librados_test_stub: reimplement cmpext()

cmpext is a read operation.  As such, it shouldn't be creating objects
or extending them and it should accept snapshots.

The compare code is completely broken, resulting in false positives
(i.e. failures to detect mismatches) in many cases.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26299 from yuvalif/add_pubsub_tests

rgw/pubsub: add pubsub tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #26484 into master

* refs/pull/26484/head:
msg/async: msgr2: don't smash client cookie on reconnect
doc/msgr2: session establishment failure scenarios
msg/async: msgr2: don't smash peer_name and features on reconnect
msg/async: msgr2: added client and server cookie to protocol

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26417 into master

* refs/pull/26417/head:
qa, doc: update ec fast read setting
qa/suites: remove mon kv backend options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26489 into master

* refs/pull/26489/head:
qa/standalone/osd/osd-markdown: disable CLI command dups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26490 into master

* refs/pull/26490/head:
osd: fix typo in numa node check; lower debug level

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26492 into master

* refs/pull/26492/head:
osd: fix pg split/merge prime during startup

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26408 from tspmelo/wip-pool-slash

mgr/dashboard: Allow "/" in pool name

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25866 from yuvalif/add_amqp_push_endpoint_new

rgw/pubsub: add amqp push endpoint

Merge pull request #26433 from tspmelo/wip-rbd-slash

mgr/dashboard: Improve RBD form

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26443 from sebastian-philipp/orchestrator_fix_readme.md

mgr/orchestrator_cli: Fix README.md

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #26509 from p-na/wip-pna-fix-ganesha-pylint

mgr/dashboard: Clean up pylint's `disable:no-else-return`

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

src/tools: only build dedup tool if WITH_TESTS is set

We need to only build the dedup tool if WITH_TESTS is set.
Otherwise, when building an RPM *without* tests enabled, the build
will fail with "error: Installed (but unpackaged) file(s) found:
/usr/bin/cephdeduptool".

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #26478 from xiexingguo/wip-calc-upmap-vec-reserve

osd/OSDMap: using std::vector::reserve to reduce memory reallocation

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #26505 from xiexingguo/wip-balancer-weekday-range

mgr/balancer: add min/max fields for weekday and be compatible with C

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25138 from stackhpc/rdma-cm

msgr/async/rdma: Return from poll system call with EINTR should be retried

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: retry if poll() returns EINTR

Fixes: https://tracker.ceph.com/issues/37292
Signed-off-by: Stig Telfer <stig@stackhpc.com>

rpm,deb: remove btrfs-tools as a build-dependency

this change reverts 7d48d21a

we are using bluestore for testing by default on GNU/Linux now. and on
FreeBSD, install-deps.sh does not look at spec.in or d/control.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: remove bc as a build-dependency

this change reverts 3ff4be6d, because in ce9c0298, we replaced bc
with awk.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26328 from badone/wip-update-ceph_stable_release-to-mimic

qa/ceph-ansible: Set ceph_stable_release to mimic

mgr/dashboard: Allow create and edit of pools with "/" in the name

Fixes: http://tracker.ceph.com/issues/38302
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25945 from trociny/wip-migration-parent

librbd: make it possible to migrate parent images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librados: don't claim ObjectOperation::cmpext() buffer

This is counter-intuitive and doesn't match the behavior of either
other compare ops (cmpxattr, omap_cmp) or write ops.

Fixes: http://tracker.ceph.com/issues/38383
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: pubsub tests. use bukcet checkpoint. allow redundant events

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/dashboard: Improve RBD form

Remove pools with '/' in the name and add RBD name validation.

Fixes: http://tracker.ceph.com/issues/38303
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace encodeURI for encodeURIComponent

Fixes: http://tracker.ceph.com/issues/38303
Signed-off-by: Tiago Melo <tmelo@suse.com>

msg/async: msgr2: don't smash client cookie on reconnect

Signed-off-by: Ricardo Dias <rdias@suse.com>

osd/PG: align past_intervals and last_epoch_clean for fabricated merge target

Fix fix from b79442efce479fde7f8bda8fdadf86e91003a327.  We need to
use last_epoch_clean *after* we set it when comparing it to the
past_intervals.

Fixes: http://tracker.ceph.com/issues/37511
Fixes: http://tracker.ceph.com/issues/24866
Signed-off-by: Sage Weil <sage@redhat.com>

mgr: Selftest fails, dashboard/server_port not string

Fixes: http://tracker.ceph.com/issues/38380
Signed-off-by: Volker Theile <vtheile@suse.com>

osd/osd_types: show cmpext arguments

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #26469 into master

* refs/pull/26469/head:
common/ceph_timer: stop timer's thread when it is suspended

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26470 into master

* refs/pull/26470/head:
mgr/MgrStandby: include addrs in metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26479 into master

* refs/pull/26479/head:
pybind: move FLAGS into Flag class

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26485 into master

* refs/pull/26485/head:
qa/suites/upgrade/luminous-x: force clone v1 format for final rbd python test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: wakeup lease_cr after abort on bucket sync disable

the 'bucket sync disable' tests were causing data sync to lag behind
because these calls to lease_cr->abort() were still waiting for the
RGWContinuousLeaseCR to wake up from its sleep. added data sync
checkpoints to each 'bucket sync disable' test to make sure that data
sync catches up before running other tests, as this was causing failures
in the new 'datalog autotrim' test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26452 from tchaikov/wip-crimson-send-known-addresses

crimson/osd: send known addresses at boot

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/msgr2: session establishment failure scenarios

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: don't smash peer_name and features on reconnect

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async: msgr2: added client and server cookie to protocol

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26450 from s0nea/wip-dashboard-readonly-config-options-2

mgr/dashboard: disallow editing read-only config options (part 2)

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25233 from p-na/wip-pna-qos-pr

mgr/dashboard: Add support for managing RBD QoS

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/balancer: make weekday be compatible with C

The __desc__ field of **begin[end]_weekend** suggests
sunday is the prevailing start of the week, whereas the
Python implementation says no.

See https://stackoverflow.com/questions/21045569/why-is-pythons-weekday-different-from-tm-wday-in-c

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/balancer: add min/max fields for begin[end]_weekday

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/dashboard: Clean up pylint's `disable:no-else-return`

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Render error in pool edit dialog

Fixes: http://tracker.ceph.com/issues/38004
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Add support for managing RBD QoS

Fixes: http://tracker.ceph.com/issues/36191
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #26405 from jtlayton/wip-jlayton-orch

Allow rook to scale the mon count

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #23221 from baixueyu/wip-rgw-cleanup-regex

rgw: delete some unused code about std::regex

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

objclass: fix cls_get_snapset_seq for cache tier case

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: add parent migration case

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: allow to migrate parent if force flag set

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: add option to list all descendant images

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: use 'namespace' instead of 'namespace-name' in specs

to make them fit the screen

Signed-off-by: Mykola Golub <mgolub@suse.com>

crimson/osd: send known addresses in MOSDBoot

otherwise, the peers will drop the connection at seeing unmatched
address in banner.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: populate crc settings

we should populate messenger related settings before starting
messengers.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: listen on v1 addresses

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26254 from tchaikov/wip-crimson-cache

crimson: cache osdmap using LRU cache

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rbd: allow 'children' command work if no snap specified

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add list_descendants API method

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make list_children work if parent snap is head

(returning children for all the image snaps in this case)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make it possible to migrate parent images

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: replace attach child call with new state machine

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: new attach child state machine

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make AttachParentRequest support reattach

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: extend parent_attach to support reattach

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26378 from votdev/issue_38267

mgr/dashboard: Module dashboard.services.ganesha has several lint issues

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

rgw: delete some unused code about std::regex

Signed-off-by: Xueyu Bai <baixueyu@inspur.com>

cmake: add PGMap.cc to crimson-common

otherwise we have failures like:

/usr/bin/ld: ../../../lib/libcrimson-common.a(Message.cc.o): in function
`PGMapDigest::~PGMapDigest()':
/var/ssd/ceph/src/mon/PGMap.h:37: undefined reference to `vtable for
PGMapDigest'
/usr/bin/ld: ../../../lib/libcrimson-common.a(Message.cc.o): in function
`PGMapDigest::~PGMapDigest()':

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: set_callback in ctor

there is no need to do so in `start()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: lookup using unordered_map first

assuming the unordered_map in LRU cache is faster.

inspired by #25480 by Radoslaw Zarzynski

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: cache osdmaps using LRU cache

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: bug in versioning concurrent, list and get have consistency issue

Fixes: https://tracker.ceph.com/issues/38060
Signed-off-by: Wang Hao <wanghao72@baidu.com>

osd: Improve use of atomic operations to add and subtract from local_num_bytes

Signed-off-by: David Zafman <dzafman@redhat.com>