qa/rbd: add cram-based snap diff test

Fixes: http://tracker.ceph.com/issues/39447
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #28456 into master

* refs/pull/28456/head:
mds: ignore sessionmap version mismatch if mds_wipe_sessions is set
mds: fix corner case of replaying open sessions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28521 from alimaredia/wip-qa-rgw-ssl

qa: use curl in wait_for_radosgw() in util/rgw.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28415 from linuxbox2/wip-full-odsock

rgw/OutputDataSocket: actually discard data on full buffer

qa: use curl in wait_for_radosgw() in util/rgw.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #25496 from mkogan1/wip-rgw-dynamic-resharding-docs

rgw: update the "radosgw-admin reshard status"

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: provide admin friendly reshard status output

Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #28446 from tspmelo/wip-prepare-backport

script/ceph-backport.sh: Add prepare function

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28515 from hvhaugwitz/mondb-recovery-script

doc: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28473 into master

* refs/pull/28473/head:
cephfs-shell: Catch OSError exceptions in lcd

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28475 into master

* refs/pull/28475/head:
cephfs-shell: Fix 'lls' command errors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28407 into master

* refs/pull/28407/head:
mgr/volumes: fix error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27856 into master

* refs/pull/27856/head:
test: remove fs/test-volume.sh workunit
test: port fs/volume related tests to python

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28196 into master

* refs/pull/28196/head:
ceph_volume_client: decode d_name before using it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #28221 into master

* refs/pull/28221/head:
qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #26737 into master

* refs/pull/26737/head:
qa/cephfs: update tests for stale session handling
mds: change how mds revoke stale caps
mds: don't mark unresponsive sessions holding no caps stale

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28518 from yuriw/wip-yuriw-crontab-master

qa/tests: removed upgrade/client-upgrade-hammer becasue ubuntu 14.04 …

Merge pull request #28466 from hoerup/update-releases

Doc: update release schedule

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Noah Watkins <nwatkins@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Doc: update release schedule

Added nautilus + the latest releases from mimic and luminous
in order to keep dumpling column, ceph_releases is modified to allow 13 columns

Signed-off-by: Torben Hørup <th@synack.fo>

Merge pull request #28470 from yuvalif/wip-yuval-sync-type-pubsub

rgw/pubsub: use incremental sync for pubsub module by default

test: remove fs/test-volume.sh workunit

.. since its ported to python now.

Fixes: http://tracker.ceph.com/issues/39949
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: port fs/volume related tests to python

... and add subvolume related tests.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/tests: removed upgrade/client-upgrade-hammer becasue ubuntu 14.04 is not supported

Also removed ubuntu 14.04 filters

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: update mondb recovery script

- be specific about stopped OSDs
- add missing '--no-mon-config' option
- fix indent of here script delimiting identifier
- use $host variable in for loop

Signed-off-by: Hannes von Haugwitz <hannes@vonhaugwitz.com>

Merge pull request #28508 from smithfarm/wip-40295

cmake: set empty-string RPATH for ceph-osd

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22446 into master

* refs/pull/22446/head:
relicense LGPL-2.1 code as LGPL-2.1 or LGPL-3.0
COPYING: add reference to a few Apache-2.0 licensed source files
debian/copyright: sync with COPYING
COPYING: LGPL2.1 -> LGPL-2.1
COPYING: extra bits from debian/copyright

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

script/ceph-backport.sh: Add prepare function

Add a new flag, --prepare, that when used will fetch all necessary info and
cherrypick all original commits automatically.
It will also prevent the main code to be executed.

After running with --prepare, and if there were no conflicts, you can simply
remove the flag from the command and run the script again.
This time it will run the main code and create the PR and update the backport
issue.

p.e.:
ceph-backport.sh 19206 jewel --prepare
ceph-backport.sh 19206 jewel

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28484 from tchaikov/wip-test_librados_build

qa/workunits/rados/test_librados_build.sh: install build deps

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28486 from tchaikov/wip-remove-digest-from-buffer.h

buffer,crypto,tools: extract digest methods out of bufferlist

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #28494 from ricardoasmarques/update-iscsi-dependencies-versions

doc: Update 'python-rtslib' and 'tcmu-runner' min versions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: set empty-string RPATH for ceph-osd

Fixes: http://tracker.ceph.com/issues/40295
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: support SUSE distros

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_librados_build.sh: install build deps

to build helloworld of librados, we need to install the build
dependencies.

Fixes: https://tracker.ceph.com/issues/40155
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: use helper script

instead of having its own copy of `install()`, try to leverage the
helper in ceph-helpers-root.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers-root: use /etc/os-release instead

as lsb_release does not work on SUSE distros

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28465 from tchaikov/wip-xio-cleanup

include/config-h.in.cmake: remove HAVE_XIO

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28495 from tchaikov/wip-dashboard-portal_ip_addresses

mgr/dashboard: take portal_ip_addresses as a list

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: s/portal_ip_address/portal_ip_addresses/

and take portal_ip_addresses as a list

it's a follow-up fix of #28084

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28290 from b-ranto/wip-selinux-rhel8

selinux: Update the policy for RHEL8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27507 from pritha-srivastava/wip-rgw-cross-tenant-doc

rgw: Improving doc for Cross Project(Tenant) access with Openstack Ke…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

doc: Update 'python-rtslib' and 'tcmu-runner' min versions

Signed-off-by: Ricardo Marques <rimarques@suse.com>

rgw: update the "radosgw-admin reshard status"
command documentation with expected output examples

Fixes: http://tracker.ceph.com/issues/37615
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #28084 from ricardoasmarques/improve-portal-ips-calculation

mgr/dashboard: Optimize portal IPs calculation

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28454 from tchaikov/wip-gcc-9-on-bionic

install-deps.sh: use GCC-9 on bionic

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28246 from kungf/mgr_default_addr

mgr: use ipv4 default when ipv6 was disabled

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28481 from iotcg/common

include: buffer_raw.h: Copyright time fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

buffer,crypto,osd,tools: extract digest methods out of bufferlist

they were consumers of ceph::list::buffers(), and there is no need to
make them member functions of ceph::list, so move them into ceph::crypto
namespace. and update the unit test and ceph_dedup_tool.cc accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Fixes: http://tracker.ceph.com/issues/40184
Signed-off-by: Lianne <liyan.wang@xtaotech.com>

common/ceph_crypto: add digest_size to digest classes

prepare for extracting the digest methods from ceph::buffer::list, so we
can use them in a templated implementation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28443 from dzafman/wip-40198

osd: Don't randomize deep scrubs when noscrub set

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28431 from cbodley/wip-40188

common: OutputDataSocket retakes mutex on error path

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28427 from tchaikov/wip-40174

mgr/influx: try to call close()

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #28254 from myoungwon/wip-hashfuncs-dedup

common,osd: add hash algorithms for dedup fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28176 from p-na/wip-pna-fix-fixmod

mgr/dashboard: Fix `npm run fixmod` command

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26909 from vitalif/wal-sync-fix

os/bluestore: Actually wait until completion in write_sync

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28167 from dillaman/wip-39972

librados: move buffer free functions to inline namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28463 from tchaikov/wip-yaml.safe_load

admin,doc/_ext/ceph_releases.py: use yaml.safe_load()

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #28074 from falcon78921/wip-doc-39712

doc: fixed broken links in nautilus release page

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

include: buffer_raw.h: Copyright time fix

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #28479 from yuriw/wip-yuriw-crontab-master

qa.tests: added "smoke" suite to the schedule on "mimic","nautilus"

qa.tests: added "smoke" suite to the schedule on "mimic","nautilus"

Note: frequency can/will reduce later

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28451 from linuxbox2/wip-rgwfile-dirinv

rgw_file: all directories are virtual with respect to contents

Merge pull request #28434 from dillaman/wip-40110

rbd-mirror: link against the specified alloc library

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28414 from smithfarm/wip-40175

build/ops: make "patch" build dependency explicit

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs-shell: Fix 'lls' command errors

This patch fixes following:
* Not printing complete path when '..' is passed as argument.
* Printing of path for current working directory.
* No need to set default value for add_argument. As it returns empty list with
  nargs='*'.
* No need to check for absolute path. os.listdir takes both relative and
  absolute path.
* Use try-catch instead of if-else statement to catch exceptions.

Fixes: http://tracker.ceph.com/issues/40244
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Catch OSError exceptions in lcd

This patch adds try-except statement to catch the OSError exceptions.

Fixes: http://tracker.ceph.com/issues/40243
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #28388 from dillaman/wip-msg-async-1

msg/async: avoid unnecessary costly wakeups for outbound messages

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: make "patch" build dependency explicit

This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.

The "patch" command is used by the BuildBoost cmake module.

Fixes: http://tracker.ceph.com/issues/40175
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw/pubsub: make new PSZone parameters optional in test

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

include/config-h.in.cmake: remove HAVE_XIO

* remove the HAVE_XIO template
* remove README.xio

as XioMessenger was removed back in cc9a9142 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_volume_client: decode d_name before using it

d_name is received as byte array but the code intends to use it as
a string. Therefore, convert it before using.

Fixes: http://tracker.ceph.com/issues/39406
Signed-off-by: Rishabh Dave <ridave@redhat.com>

admin,doc/_ext/ceph_releases.py: use yaml.safe_load()

see
https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23312 from sylvia0409/master

doc: Fixes OSD node labels which based on the osd_devices name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28461 from tchaikov/wip-cmake-pg-meta-set

crimson/osd/pg_meta: use initializer list for passing set<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg_meta: use initializer list for passing set<>

we cannot assume that the data in string views are always nul
terminated.

this change partially reverts 82fedbd0089073cfe86640eaa7d73ed1e2545c31.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28397 from tchaikov/wip-38403

osdc/Objecter: use unique_ptr<OSDMap> for Objecter::osdmap

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28403 from xiexingguo/wip-unify-is-noup

osd: unify sources of no{up,down,in,out} flags into singleton helpers

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: move Objecter::Objecter() into .cc

to speed up compilation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: use unique_ptr<OSDMap> for Objecter::osdmap

Fixes: http://tracker.ceph.com/issues/38403
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28379 from jtlayton/wip-jlayton-40114

msg: fix addr2 encoding for sockaddrs

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28368 from kamoltat/wip-progress-module-pg-osdmap-comparison

mgr/progress: Look at PG state when PG epoch >= OSDMap epoch

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28352 from smithfarm/wip-rpm-obsolete-python

rpm: have pybind RPMs provide/obsolete their python2 predecessors

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27512 from jmolmo/tls_auth

mgr/ansible: TLS Mutual Authentication

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mds: ignore sessionmap version mismatch if mds_wipe_sessions is set

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: fix corner case of replaying open sessions

Marking a session dirty may flush all existing dirty sessions. MDS
calls Server::finish_force_open_sessions() for log event that opens
multiple sessions. The function marks sessions dirty one by one. So
sessions opened by a log event may get flushed partially.

When replaying a log event that opens multiple sessions, mds need to
check if some of these sessions have already been flushed.

Fixes: https://tracker.ceph.com/issues/40211
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/cephfs: update tests for stale session handling

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: change how mds revoke stale caps

- Only revokes conflicting caps from stale client.
- If stale client holds conflicting CEPH_CAP_ANY_WR,
  blacklist and kill it.

Fixes: https://tracker.ceph.com/issues/38326
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't mark unresponsive sessions holding no caps stale

When an unresponsive MDS session holds no caps, do not mark it stale
even after session_timeout; at session_autoclose, evict it directly.

Fixes: http://tracker.ceph.com/issues/17854
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #28171 into master

* refs/pull/28171/head:
test_volume_client: simplify test_get_authorized_ids()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28190 into master

* refs/pull/28190/head:
mds: cleanup unneeded client_snap_caps when splitting snap inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28293 into master

* refs/pull/28293/head:
mds: avoid passing null to SessionmMap::hit_session()
qa/cephfs: add test for blacklisted client eviction
mds: fix 'is session in blacklist' check in Server::apply_blacklist()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28376 into master

* refs/pull/28376/head:
mds: count purge queue items left in journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28406 into master

* refs/pull/28406/head:
mds: reset heartbeat inside big loop

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28411 into master

* refs/pull/28411/head:
qa/cephfs: fix test_evict_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28433 into master

* refs/pull/28433/head:
cephfs-shell: Remove unnecessary empty line
cephfs-shell: Fix flake8 bare 'except' warnings
cephfs-shell: Remove unused variable 'e'
cephfs-shell: Fix flake8 use != to compare str literals warning
cephfs-shell: Fixe flake8 line too long error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: Don't randomize deep scrubs when noscrub set

Fixes: http://tracker.ceph.com/issues/40198
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #28455 from yuriw/wip-yuriw-40208-master

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

Fixes: http://tracker.ceph.com/issues/40207
Fixes: http://tracker.ceph.com/issues/40208
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw_file: all directories are virtual with respect to contents

This change causes directory handles to always report an mtime of
"now."  This is not an invalidate per se--it interacts with the
nfs implementation to produce that result when the implementation
updates its cached attributes.  Hence, it can be modulated by timers
or other rules governing attribute caching at the upper layer.

Fixes: http://tracker.ceph.com/issues/40204
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28440 from linuxbox2/wip-rgwfile-invbckt

rgw_file: permit lookup_handle to lookup root_fh