rgw: Add DoutPrefixProvider to create_bucket logging

This commit hits all log messages for a create bucket
request and ensures each has a DoutPrefixProvider.

This commit gets dpp's all the way down to rgw/services
up way though to rgw_op.cc, rgw_rest.cc files,
rgw_admin.cc, etc.

This commit also removes store as dpp in rgw_sal.h

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #39086 from orozery/qa-rbd-delete-passphrase-file

qa/tasks/rbd: cleanup passphrase file

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39089 from orozery/librbd-luks-cli-test

test/librbd: fix luks encryption cli test on unsupported runs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39091 from tchaikov/wip-crimson-build

cmake: let alienstore link against zoned allocator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: let alienstore link against zoned allocator

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #39028 into master

* refs/pull/39028/head:
cephadm: retry for 30s instead of 10s
cephadm: try 'mgr stat' before 'mgr dump' to get the epoch
mon: add 'mgr stat' command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

test/librbd: fix luks encryption cli test on unsupported runs

This commit changes the luks encryption cli test to return success instead of failure when not supported.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39077 from tchaikov/wip-libzbd

install-deps.sh: set codename when installing libzbd

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #38935 from tchaikov/wip-mgr-ann

pybind/mgr/mgr_module: annotate more methods and add poll to CLICommand

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kamoltat <ksirivad@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/rbd: cleanup passphrase file

This commit adds a cleanup for the passphrase file used to create/mount encrypted rbd images.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39022 from petrutlucian94/rbd_fixes

Windows RBD fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38984 from petrutlucian94/dll

Windows DLL support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39073 from tchaikov/wip-rbd-bench

rbd/bench: include used headers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39049 from lixiaoy1/fix_spec

spec: enable RWL/SSD caches

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39044 from orozery/librbd-crypto-close-bug

librbd: do not share crypto image layers with ancestors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39006 from orozery/librbd-luks-cli-test

test/librbd: add luks encryption cli test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39005 from orozery/librbd-plain64-iv-fix

librbd: use 512 sector size for crypto IV

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39064 from idryomov/wip-doc-osdtimeout

doc: don't mention osdtimeout option in mount.ceph man page

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #39060 from idryomov/wip-krbd-msgr2

krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: set codename when installing libzbd

this was an oversight in e7e3b86762ff3fccdae21e713c64215ab3bb7bb4

also use updated build with sha1 of
1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.

and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: return -ENOENT if self.ssh_user is None

there is chance that self.ssh_user is None when handling "cephadm
get-user" command, so return -ENOENT and error messages in that case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use _get_localized_key for composing store key

Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: add assert to appease mypy

self.get_osd_perf_counters() might return None, add assert() to assure
mypy that it does not return None here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/serve: add assert to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/module: assure self.ssh_user is not None

to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/inventory: cast variable to expected type

to silence mypy warnings like:

cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: do not pass --use-feature=2020-resolver to pip

pip use the new resolver by default now. so no need to pass this option
to it anymore.

This reverts commit fa9e2bfd4b3648f08ed3a88ce737d432ab97cce1.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: annotate variable type

to silence following mypy error:

cephadm/module.py:1155: error: Incompatible types in assignment
(expression has type "None", variable has type "str")

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: rename variables to avoid name reuse

to applease mypy, otherwise, for instance, "err" would be reused for
holding the return value of self.mgr.check_mon_command(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: annotate OSDMap and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()

it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define HandleCommandResult using NamedTuple

simpler this way, so we don't need to define __new__

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: use numeric values of logging level for index

less magic values in code this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: use dict instead of defaultdict to silence mypy warning

otherwise we have:

1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: handle the case where given osd is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass str as the 2nd param of set_store()

MgrModule.set_store() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass a str as the value of option to set_module_option()

MgrModule.set_module_option() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass id to constructor of Event

to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:

progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: be PEP8 compliant

to applease flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: minor cleanups

* do not pass typename and self to super()
* do not add parentheses around condition expression

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: use correct types

for better readability and for the sake of correctness.

* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/iostat: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: annotate more methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38929 from jdurgin/wip-cot-log

qa/tasks/ceph_manager: let c-o-t log errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38931 from trociny/wip-48821

osd: fix potential null pointer dereference when sending ping

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38901 from gmayyyha/io_uring-48872

blk/kernel: fix io_uring got (4) Interrupted system call

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38932 from tchaikov/wip-48905

mgr: update mon metadata when monmap is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

spec: enable RWL/SSD caches

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

cmake,win32*.sh: Windows DLL support

At the moment, the Windows binaries can only be linked statically.
This is less than ideal since:

* the resulting binaries can be quite large, especially when
including debug symbols
* librados and librbd cannot be used directly by 3rd party apps
* the build duration is increased

In order to do a dynamic build, we'll:

* add an option to win32_build.sh
* fix link order
* dynamically link boost
* disable the "-pie" flag when using Mingw, which generates incorrect
  executable entry points
* by default, cmake generates import libs for executables. The issue
  is that for rados.exe, it generates librados.dll.a, thus overriding
  the librados.dll import library, which breaks the build process.
  We'll configure cmake to skip generating import libs for executables.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

rbd/bench: include used headers

should help with fixing following failure:

src/tools/rbd/action/Bench.cc:20:26: error: aggregate 'std::atomic<bool>
terminating' has incomplete type and cannot be defined

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37250 from cbodley/wip-qa-librgw

qa/rgw: add librgw_file unit tests to rgw/verify task

Merge pull request #38852 from adamemerson/wip-thou-shalt-not-unwind

Thou Shalt Not Unwind

Merge pull request #38905 from cbodley/wip-48874

rgw: fix trailing null in object names of multipart reuploads

Merge pull request #39063 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific branch to the mix

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39009 from mdw-at-linuxbox/wip-master-keystonebindep

qa/tasks/keystone.py: centos-8 bindep fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38962 from aclamk/wip-aclamk-bluestore-tool-reshard-fix

Fix ceph-bluestore-tool reshard command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38965 from aclamk/wip-aclamk-bluestore-tool-sharding-show

Add command 'show-sharding' to ceph-bluestore-tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

test/rgw_file: bypass one bad-deletes test due to intermittent failure

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

test/rgw_file: override keyring if we gen keys

Run with ceph.keyring not that of client.${CEPH_ID}.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

test/rgw_file: elaborate test cycle

Ensure that all delete phases are run so that script can be
re-run when desired.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
 Please enter the commit message for your changes. Lines starting

rgw_file: delete test bucket when do_delete

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: add librgw unit test arguments

Also adjust some unit test files to avoid trivial failures.
Fixes missing commit after rgw_write(), as well as conditional
cleanup.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa/rgw: librgw workunit creates a test user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: install the ceph_test_librgw_file_* targets

these need to be installed in order to be included in packages for
testing in teuthology

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa/rgw: add librgw_file unit tests to rgw/verify task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/krbd: add msgr2 modes to most subsuites

basic, rbd and rbd-nomount subsuites are expanded to run with each
of ms_mode=legacy, ms_mode=crc and ms_mode=secure.  This increases
the total number of jobs in the suite from 100 to 220.

fsx, singleton and thrash subsuites choose ms_mode at random (from
the above plus ms_mode=prefer-crc).

unmap and wac subsuites remain msgr1-only.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: deprecate [no]cephx_require_signatures map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: add support for msgr2

Recognize ms_mode map option and filter initial monitor addresses
accordingly: if ms_mode is not given or ms_mode=legacy, discard v2
addresses, otherwise discard v1 addresses.

Note that nothing was discarded (i.e. v2 addresses were passed to
the kernel) previously.  The intent was to preserve that behaviour
in case ms_mode is not given, allowing to change the kernel default
in the future.  However, it turns out that mount.ceph helper has
been misguidedly discarding v2 addresses since commit eae01275134e
("mount.ceph: fork a child to get info from local configuration"),
so that ship has sailed.

Fixes: https://tracker.ceph.com/issues/48976
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37694 from BryceCao/wip-add-MD5-in-forward_request

rgw : add MD5 in forward_request

Merge pull request #38977 from adamemerson/wip-48929

rgw: Fix spurious error on empty datalog shard

Merge pull request #38958 from badone/wip-osd-df-tree-accounting-error

An empty bucket or OSD is not an error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38992 from ifed01/wip-ifed-fix-48776-2

os/bluestore: be more protective in _deferred_queue

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: don't mention osdtimeout option in mount.ceph man page

It was deprecated over 8 years ago and will be entirely removed in
kernel 5.12.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #39051 from tchaikov/wip-install-deps-libzbd

install-deps.sh: install libzbd-dev when WITH_ZBD is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: added pacific branch to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

cephadm: retry for 30s instead of 10s

No real reason not to wait longer, and I'm worried about things not coming
up quickly enough on slower or loady machines.

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: try 'mgr stat' before 'mgr dump' to get the epoch

This is less noise in the debug log.  (Also, there seems to be a problem
when the mgr stat output gets too big.)

Signed-off-by: Sage Weil <sage@newdream.net>

mon: add 'mgr stat' command

This is analogous to 'osd stat' and 'mds stat'.

Signed-off-by: Sage Weil <sage@newdream.net>

install-deps.sh: install libzbd-dev when WITH_ZBD is set

to avoid zbd backend bitrot, and for build test it in our "make check"
builds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define CLUSTER_LOG_PRIO* using enum.IntEnum

so the clog type / prio can be typed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: add poll param to CLICommand

and add it to the output of dump_cmd() call, if "poll" is missing
in the output of dump_cmd() call, ceph CLI consider the it as False.

but it'd be better to be explicit. so this field is always set.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr,pybind/mgr: use bool for poll in COMMANDS

use correctly typed variable helps with readability and less
error-prone.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39040 from liewegas/cephadm-qa-targets

qa/suites/rados/cephadm: test against ubuntu 20.04 (and update ubuntu_latest)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: do not share crypto image layers with ancestors

This commit creates a unique crypto image dispatch per image,
instead of sharing a single object with all ancestors.
This should fix a seg-fault caused when closing an encrypted cloned image.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: add luks encryption cli test

This commit adds a cli test for rbd encryption verifying LUKS compatbility with cryptsetup

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa/workunits/cephadm/test_repos: adjust target version

15.2.7 is available for 20.04; 15.1.1 is not

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39042 from tchaikov/wip-cram

test/run-cli-tests: use cram from github

Reviewed-by: Sage Weil <sage@redhat.com>

test/downloads: drop cram-0.5 tar ball

since we install cram from github, drop the tarball from repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/run-cli-tests: use cram from github

to address the failure of

  WARNING: Built wheel for cram is invalid: Metadata 1.2 mandates PEP 440 version, but '0.5.0ceph.2011-01-14' is not
Failed to build cram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39035 from tchaikov/wip-cephadm-call

cephadm: rewrite call() with asyncio

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: rewrite call() with asyncio

for better readability, also return 124 when subprocess times out

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm: add 20.04 targets

Add 20.04 targets.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/cephadm: add 20.04 target

Link directly to the distro version... no need to use _latest here since
it obscures the podman vs docker difference.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/distros: change ubuntu_latest to 20.04

Signed-off-by: Sage Weil <sage@newdream.net>

qa/distros: add ubuntu 20.04 targets

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #38974 into master

* refs/pull/38974/head:
qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39032 from adamemerson/wip-layout-initialize

rgw: Initialize Layout Logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39029 from dillaman/wip-48964

ocf: add support for mapping images within an RBD namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>