librbd: avoid dereferencing an empty container during deep-copy

While updating the image copy progress, a 'while' loop will pop the
last item from the queue and then deference the empty queue.

Fixes: http://tracker.ceph.com/issues/40368
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27951 from ifed01/wip-ifed-fix-slowop-log

os/bluestore: fix origin reference in logging slow ops.

Merge pull request #28490 from xiexingguo/wip-40193

mon/OSDMonitor: report pg[pgp]_num_target instead of pg[pgp]_num

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28322 from liewegas/wip-preextend

os/bluestore: bluefs_preextend_wal_files=true

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28260 from s0nea/wip-dashboard-pool-delete-flag

mgr/dashboard: consider 'mon_allow_pool_delete' flag

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28534 from tchaikov/wip-crimson-mkkey

crimson/osd: add --mkkey support

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28464 from majianpeng/fio-ceph-messenger-multi-client

test/fio/fio_ceph_messenger: make exec multi client on the same host.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28404 from xiexingguo/wip-osd-noup-stall

osd/OSD: keep synchronizing with mon if stuck at booting

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/osd: ignore exception when stopping osd

OSD::stop() is a top-level method in the sense that it will be called
directly when engine stops. so it should not throw any exceptions.
instead, we will print out the error and ignore the exception if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add --mkkey support

see qa/tasks/ceph.py for how it is used, and why we need it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use buffer::write_file() helper

instead of using the blocking calls

so we can write to file without blocking the reactor

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: add buffer_io helpers

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28507 from tchaikov/wip-cmake-dpdk-boost

cmake: update SPDK to build with GCC-9

Reviewed-by: Casey Bodley <cbodley@redhat.com>

auth/KeyRing: remove KeyRing::create_empty()

it never fails unless out of memory. and it would be simpler to just
allocate it on stack.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28535 from tchaikov/wip-crimson-packaging

debian/: use ceph-osd for packaging crimson-osd

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #28456 into master

* refs/pull/28456/head:
mds: ignore sessionmap version mismatch if mds_wipe_sessions is set
mds: fix corner case of replaying open sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28521 from alimaredia/wip-qa-rgw-ssl

qa: use curl in wait_for_radosgw() in util/rgw.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28415 from linuxbox2/wip-full-odsock

rgw/OutputDataSocket: actually discard data on full buffer

qa: use curl in wait_for_radosgw() in util/rgw.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #25496 from mkogan1/wip-rgw-dynamic-resharding-docs

rgw: update the "radosgw-admin reshard status"

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

debian/: use ceph-osd for packaging crimson-osd

* debian/: remove ceph-crimson-osd package.
* debian/control: set `CEPH_OSD_BASENAME` env variable, which
  will be consumed by `ceph-osd.install`. alternatively, we could
  rename crimson-osd to ceph-osd in `override_dh_auto_install`,
  but let's go with this way at this moment, unless `mv` in
  `override_dh_auto_install` is proved to be better.
* ceph-osd.install: replace ceph-osd with crimson-osd if
  `CEPH_EXTRA_CMAKE_ARGS` has `WITH_SEASTAR` in it. this only
  happens when we are packaging the "crimson" flavor packages from
  jenkins.
* ceph-osd.install: `chmod +x` this file, as we need to use
  `/usr/bin/dh-exec` as the interpreter of it to perform variable
  substitution and install.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: silence -Waddress-of-packed-member warnings from SPDK

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: silence GCC-9 warnings from rocksdb

GCC-9 complains:

rocksdb/db/version_edit.h:86:8: warning: implicitly-declared ‘constexpr
rocksdb::FileDescriptor::FileDescriptor(const rocksdb::FileDescriptor&)’
is deprecated [-Wdeprecated-copy]
   86 | struct FileMetaData {
      |        ^~~~~~~~~~~~
/var/ssd/ceph/src/rocksdb/db/version_edit.h:55:19: note: because
‘rocksdb::FileDescriptor’ has user-provided ‘rocksdb::FileDescriptor&
rocksdb::FileDescriptor::operator=(const rocksdb::FileDescriptor&)’
   55 |   FileDescriptor& operator=(const FileDescriptor& fd) {
      |                   ^~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: set ccache for rocksdb

should use launcher to set ccache

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update built libs of SPDK and DPDK

we need to link against these new libs for fulfilling the dependencies
required by new SPDK.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable isal and vhost support of SPDK

* isal needs nasm, which is a new dependency. so let's disable it
  at this moment.
* vhost requires DPDK compiled with vhost enabled. but it's disabled
  when we compile DPDK.

Signed-off-by: Kefu Chai <kchai@redhat.com>

spdk: pick up latest spdk

to silence the warnings reported by GCC-9

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: provide admin friendly reshard status output

Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #28446 from tspmelo/wip-prepare-backport

script/ceph-backport.sh: Add prepare function

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28515 from hvhaugwitz/mondb-recovery-script

doc: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28473 into master

* refs/pull/28473/head:
cephfs-shell: Catch OSError exceptions in lcd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28475 into master

* refs/pull/28475/head:
cephfs-shell: Fix 'lls' command errors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28407 into master

* refs/pull/28407/head:
mgr/volumes: fix error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27856 into master

* refs/pull/27856/head:
test: remove fs/test-volume.sh workunit
test: port fs/volume related tests to python

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28196 into master

* refs/pull/28196/head:
ceph_volume_client: decode d_name before using it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #28221 into master

* refs/pull/28221/head:
qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #26737 into master

* refs/pull/26737/head:
qa/cephfs: update tests for stale session handling
mds: change how mds revoke stale caps
mds: don't mark unresponsive sessions holding no caps stale

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28518 from yuriw/wip-yuriw-crontab-master

qa/tests: removed upgrade/client-upgrade-hammer becasue ubuntu 14.04 …

Merge pull request #28466 from hoerup/update-releases

Doc: update release schedule

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Noah Watkins <nwatkins@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Doc: update release schedule

Added nautilus + the latest releases from mimic and luminous
in order to keep dumpling column, ceph_releases is modified to allow 13 columns

Signed-off-by: Torben Hørup <th@synack.fo>

Merge pull request #28470 from yuvalif/wip-yuval-sync-type-pubsub

rgw/pubsub: use incremental sync for pubsub module by default

test: remove fs/test-volume.sh workunit

.. since its ported to python now.

Fixes: http://tracker.ceph.com/issues/39949
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: port fs/volume related tests to python

... and add subvolume related tests.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/tests: removed upgrade/client-upgrade-hammer becasue ubuntu 14.04 is not supported

Also removed ubuntu 14.04 filters

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: update mondb recovery script

- be specific about stopped OSDs
- add missing '--no-mon-config' option
- fix indent of here script delimiting identifier
- use $host variable in for loop

Signed-off-by: Hannes von Haugwitz <hannes@vonhaugwitz.com>

Merge pull request #28508 from smithfarm/wip-40295

cmake: set empty-string RPATH for ceph-osd

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22446 into master

* refs/pull/22446/head:
relicense LGPL-2.1 code as LGPL-2.1 or LGPL-3.0
COPYING: add reference to a few Apache-2.0 licensed source files
debian/copyright: sync with COPYING
COPYING: LGPL2.1 -> LGPL-2.1
COPYING: extra bits from debian/copyright

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

script/ceph-backport.sh: Add prepare function

Add a new flag, --prepare, that when used will fetch all necessary info and
cherrypick all original commits automatically.
It will also prevent the main code to be executed.

After running with --prepare, and if there were no conflicts, you can simply
remove the flag from the command and run the script again.
This time it will run the main code and create the PR and update the backport
issue.

p.e.:
ceph-backport.sh 19206 jewel --prepare
ceph-backport.sh 19206 jewel

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28484 from tchaikov/wip-test_librados_build

qa/workunits/rados/test_librados_build.sh: install build deps

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28486 from tchaikov/wip-remove-digest-from-buffer.h

buffer,crypto,tools: extract digest methods out of bufferlist

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #28494 from ricardoasmarques/update-iscsi-dependencies-versions

doc: Update 'python-rtslib' and 'tcmu-runner' min versions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: set empty-string RPATH for ceph-osd

Fixes: http://tracker.ceph.com/issues/40295
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: support SUSE distros

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_librados_build.sh: install build deps

to build helloworld of librados, we need to install the build
dependencies.

Fixes: https://tracker.ceph.com/issues/40155
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: use helper script

instead of having its own copy of `install()`, try to leverage the
helper in ceph-helpers-root.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers-root: use /etc/os-release instead

as lsb_release does not work on SUSE distros

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28465 from tchaikov/wip-xio-cleanup

include/config-h.in.cmake: remove HAVE_XIO

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: move boost patch to cmake/modules

as it's used by cmake only

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/BuildDPDK.cmake: update to adapt upstream changes

* update execenv to remove "app", the targets ending with "app" changed
  to the ones without "app" now.
* pass CC by make variables not env variables, otherwise the default
  `cc` is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28495 from tchaikov/wip-dashboard-portal_ip_addresses

mgr/dashboard: take portal_ip_addresses as a list

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: s/portal_ip_address/portal_ip_addresses/

and take portal_ip_addresses as a list

it's a follow-up fix of #28084

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28290 from b-ranto/wip-selinux-rhel8

selinux: Update the policy for RHEL8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27507 from pritha-srivastava/wip-rgw-cross-tenant-doc

rgw: Improving doc for Cross Project(Tenant) access with Openstack Ke…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc: Update 'python-rtslib' and 'tcmu-runner' min versions

Signed-off-by: Ricardo Marques <rimarques@suse.com>

rgw: update the "radosgw-admin reshard status"
command documentation with expected output examples

Fixes: http://tracker.ceph.com/issues/37615
Signed-off-by: Mark Kogan <mkogan@redhat.com>

mon/OSDMonitor: report pg[pgp]_num_target instead of pg[pgp]_num

As by calling "ceph osd pool set <pool-name> pg_num", we set the
pg_num_target instead. Hence we should report pg_num_target as well
when invoked by the "ceph osd pool get <pool-name> pg_num" call.

I doubt an ideal fix would be exposing both the pg_num(*actual*) and
pg_num_target concepts to user, but for now I think we can stop here
and get some more feedback first.

Fixes: http://tracker.ceph.com/issues/40193
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28084 from ricardoasmarques/improve-portal-ips-calculation

mgr/dashboard: Optimize portal IPs calculation

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28454 from tchaikov/wip-gcc-9-on-bionic

install-deps.sh: use GCC-9 on bionic

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28246 from kungf/mgr_default_addr

mgr: use ipv4 default when ipv6 was disabled

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28481 from iotcg/common

include: buffer_raw.h: Copyright time fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

buffer,crypto,osd,tools: extract digest methods out of bufferlist

they were consumers of ceph::list::buffers(), and there is no need to
make them member functions of ceph::list, so move them into ceph::crypto
namespace. and update the unit test and ceph_dedup_tool.cc accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Fixes: http://tracker.ceph.com/issues/40184
Signed-off-by: Lianne <liyan.wang@xtaotech.com>

common/ceph_crypto: add digest_size to digest classes

prepare for extracting the digest methods from ceph::buffer::list, so we
can use them in a templated implementation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/fio/fio_ceph_messenger: make exec multi client on the same host.

When create Messenger, for client we should give different nonce to
make multi clients on the same host can work. Otherwise, server only
accept one connection.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #28443 from dzafman/wip-40198

osd: Don't randomize deep scrubs when noscrub set

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28431 from cbodley/wip-40188

common: OutputDataSocket retakes mutex on error path

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28427 from tchaikov/wip-40174

mgr/influx: try to call close()

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #28254 from myoungwon/wip-hashfuncs-dedup

common,osd: add hash algorithms for dedup fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28176 from p-na/wip-pna-fix-fixmod

mgr/dashboard: Fix `npm run fixmod` command

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26909 from vitalif/wal-sync-fix

os/bluestore: Actually wait until completion in write_sync

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28167 from dillaman/wip-39972

librados: move buffer free functions to inline namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28463 from tchaikov/wip-yaml.safe_load

admin,doc/_ext/ceph_releases.py: use yaml.safe_load()

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #28074 from falcon78921/wip-doc-39712

doc: fixed broken links in nautilus release page

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

include: buffer_raw.h: Copyright time fix

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #28479 from yuriw/wip-yuriw-crontab-master

qa.tests: added "smoke" suite to the schedule on "mimic","nautilus"

qa.tests: added "smoke" suite to the schedule on "mimic","nautilus"

Note: frequency can/will reduce later

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28451 from linuxbox2/wip-rgwfile-dirinv

rgw_file: all directories are virtual with respect to contents

Merge pull request #28434 from dillaman/wip-40110

rbd-mirror: link against the specified alloc library

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28414 from smithfarm/wip-40175

build/ops: make "patch" build dependency explicit

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs-shell: Fix 'lls' command errors

This patch fixes following:
* Not printing complete path when '..' is passed as argument.
* Printing of path for current working directory.
* No need to set default value for add_argument. As it returns empty list with
  nargs='*'.
* No need to check for absolute path. os.listdir takes both relative and
  absolute path.
* Use try-catch instead of if-else statement to catch exceptions.

Fixes: http://tracker.ceph.com/issues/40244
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Catch OSError exceptions in lcd

This patch adds try-except statement to catch the OSError exceptions.

Fixes: http://tracker.ceph.com/issues/40243
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #28388 from dillaman/wip-msg-async-1

msg/async: avoid unnecessary costly wakeups for outbound messages

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: make "patch" build dependency explicit

This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.

The "patch" command is used by the BuildBoost cmake module.

Fixes: http://tracker.ceph.com/issues/40175
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw/pubsub: make new PSZone parameters optional in test

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

include/config-h.in.cmake: remove HAVE_XIO

* remove the HAVE_XIO template
* remove README.xio

as XioMessenger was removed back in cc9a9142 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_volume_client: decode d_name before using it

d_name is received as byte array but the code intends to use it as
a string. Therefore, convert it before using.

Fixes: http://tracker.ceph.com/issues/39406
Signed-off-by: Rishabh Dave <ridave@redhat.com>

admin,doc/_ext/ceph_releases.py: use yaml.safe_load()

see
https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23312 from sylvia0409/master

doc: Fixes OSD node labels which based on the osd_devices name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28461 from tchaikov/wip-cmake-pg-meta-set

crimson/osd/pg_meta: use initializer list for passing set<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg_meta: use initializer list for passing set<>

we cannot assume that the data in string views are always nul
terminated.

this change partially reverts 82fedbd0089073cfe86640eaa7d73ed1e2545c31.

Signed-off-by: Kefu Chai <kchai@redhat.com>