cmake: disable isal and vhost support of SPDK

* isal needs nasm, which is a new dependency. so let's disable it
  at this moment.
* vhost requires DPDK compiled with vhost enabled. but it's disabled
  when we compile DPDK.

Signed-off-by: Kefu Chai <kchai@redhat.com>

spdk: pick up latest spdk

to silence the warnings reported by GCC-9

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move boost patch to cmake/modules

as it's used by cmake only

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/BuildDPDK.cmake: update to adapt upstream changes

* update execenv to remove "app", the targets ending with "app" changed
  to the ones without "app" now.
* pass CC by make variables not env variables, otherwise the default
  `cc` is used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28290 from b-ranto/wip-selinux-rhel8

selinux: Update the policy for RHEL8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27507 from pritha-srivastava/wip-rgw-cross-tenant-doc

rgw: Improving doc for Cross Project(Tenant) access with Openstack Ke…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28084 from ricardoasmarques/improve-portal-ips-calculation

mgr/dashboard: Optimize portal IPs calculation

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28454 from tchaikov/wip-gcc-9-on-bionic

install-deps.sh: use GCC-9 on bionic

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28246 from kungf/mgr_default_addr

mgr: use ipv4 default when ipv6 was disabled

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28481 from iotcg/common

include: buffer_raw.h: Copyright time fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28443 from dzafman/wip-40198

osd: Don't randomize deep scrubs when noscrub set

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28431 from cbodley/wip-40188

common: OutputDataSocket retakes mutex on error path

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28427 from tchaikov/wip-40174

mgr/influx: try to call close()

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #28254 from myoungwon/wip-hashfuncs-dedup

common,osd: add hash algorithms for dedup fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28176 from p-na/wip-pna-fix-fixmod

mgr/dashboard: Fix `npm run fixmod` command

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26909 from vitalif/wal-sync-fix

os/bluestore: Actually wait until completion in write_sync

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28167 from dillaman/wip-39972

librados: move buffer free functions to inline namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28463 from tchaikov/wip-yaml.safe_load

admin,doc/_ext/ceph_releases.py: use yaml.safe_load()

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #28074 from falcon78921/wip-doc-39712

doc: fixed broken links in nautilus release page

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

include: buffer_raw.h: Copyright time fix

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #28479 from yuriw/wip-yuriw-crontab-master

qa.tests: added "smoke" suite to the schedule on "mimic","nautilus"

qa.tests: added "smoke" suite to the schedule on "mimic","nautilus"

Note: frequency can/will reduce later

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28451 from linuxbox2/wip-rgwfile-dirinv

rgw_file: all directories are virtual with respect to contents

Merge pull request #28434 from dillaman/wip-40110

rbd-mirror: link against the specified alloc library

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28414 from smithfarm/wip-40175

build/ops: make "patch" build dependency explicit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28388 from dillaman/wip-msg-async-1

msg/async: avoid unnecessary costly wakeups for outbound messages

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build/ops: make "patch" build dependency explicit

This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.

The "patch" command is used by the BuildBoost cmake module.

Fixes: http://tracker.ceph.com/issues/40175
Signed-off-by: Nathan Cutler <ncutler@suse.com>

admin,doc/_ext/ceph_releases.py: use yaml.safe_load()

see
https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23312 from sylvia0409/master

doc: Fixes OSD node labels which based on the osd_devices name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28461 from tchaikov/wip-cmake-pg-meta-set

crimson/osd/pg_meta: use initializer list for passing set<>

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd/pg_meta: use initializer list for passing set<>

we cannot assume that the data in string views are always nul
terminated.

this change partially reverts 82fedbd0089073cfe86640eaa7d73ed1e2545c31.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28397 from tchaikov/wip-38403

osdc/Objecter: use unique_ptr<OSDMap> for Objecter::osdmap

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28403 from xiexingguo/wip-unify-is-noup

osd: unify sources of no{up,down,in,out} flags into singleton helpers

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: move Objecter::Objecter() into .cc

to speed up compilation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: use unique_ptr<OSDMap> for Objecter::osdmap

Fixes: http://tracker.ceph.com/issues/38403
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28379 from jtlayton/wip-jlayton-40114

msg: fix addr2 encoding for sockaddrs

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28368 from kamoltat/wip-progress-module-pg-osdmap-comparison

mgr/progress: Look at PG state when PG epoch >= OSDMap epoch

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28352 from smithfarm/wip-rpm-obsolete-python

rpm: have pybind RPMs provide/obsolete their python2 predecessors

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27512 from jmolmo/tls_auth

mgr/ansible: TLS Mutual Authentication

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #28171 into master

* refs/pull/28171/head:
test_volume_client: simplify test_get_authorized_ids()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28190 into master

* refs/pull/28190/head:
mds: cleanup unneeded client_snap_caps when splitting snap inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28293 into master

* refs/pull/28293/head:
mds: avoid passing null to SessionmMap::hit_session()
qa/cephfs: add test for blacklisted client eviction
mds: fix 'is session in blacklist' check in Server::apply_blacklist()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28376 into master

* refs/pull/28376/head:
mds: count purge queue items left in journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28406 into master

* refs/pull/28406/head:
mds: reset heartbeat inside big loop

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28411 into master

* refs/pull/28411/head:
qa/cephfs: fix test_evict_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28433 into master

* refs/pull/28433/head:
cephfs-shell: Remove unnecessary empty line
cephfs-shell: Fix flake8 bare 'except' warnings
cephfs-shell: Remove unused variable 'e'
cephfs-shell: Fix flake8 use != to compare str literals warning
cephfs-shell: Fixe flake8 line too long error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: Don't randomize deep scrubs when noscrub set

Fixes: http://tracker.ceph.com/issues/40198
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #28455 from yuriw/wip-yuriw-40208-master

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

qa/tests: removed `1node` and `systemd` tests as ceph-deploy is not actively developed

Fixes: http://tracker.ceph.com/issues/40207
Fixes: http://tracker.ceph.com/issues/40208
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw_file: all directories are virtual with respect to contents

This change causes directory handles to always report an mtime of
"now."  This is not an invalidate per se--it interacts with the
nfs implementation to produce that result when the implementation
updates its cached attributes.  Hence, it can be modulated by timers
or other rules governing attribute caching at the upper layer.

Fixes: http://tracker.ceph.com/issues/40204
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28440 from linuxbox2/wip-rgwfile-invbckt

rgw_file: permit lookup_handle to lookup root_fh

Merge pull request #28442 from neha-ojha/wip-40120-port

qa/tasks/cbt.py: change port to work with client_endpoints

Reviewed-by: Mark Nelson <mnelson@redhat.com>

install-deps.sh: update ubuntu-toolchain-r mirrors

the removed two do not have GCC-9, but the newly added mirros do.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: use GCC-9 on bionic

two reasons

* GCC-9 features more checks. so let's use it!
* crimson targets the hardware + toolchain + kernel after 1+ years,
  so it would be great if we can compile and test crimson using
  the toolchain which is ubiquitous on most mainstream distos.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28435 from yuriw/wip-yuriw-smoke-master

qa/tests: reduced distro to run to be random

Merge pull request #28428 from tchaikov/wip-package-crimson-osd

rpm,debian,install-deps: package crimson-osd

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

ceph.spec.in: align "Requires" of ceph-osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: package crimson-osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/: add ceph-crimson-osd package

this package is commented out using `# Crimson`. and it will be enabled
only if we are building "crimson" flavor builds.

this package depends on ceph-osd, because it'll basically be a drop-in
replacement of ceph-osd. and it shares all the dependencies of ceph-osd.
so let's just reuse ceph-osd package for now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install crimson-osd if WITH_SEASTAR

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: populate WITH_SEASTAR in the same way as for_make_check

because `install-deps.sh` is executed using `source`, we have to pass
these options using env variables. but before this change, `WITH_SEASTAR` is used directly,
while `FOR_MAKE_CHECK` is checked and translated to a local variable
`for_make_check`. which, in my opinion, has better readability.

so, in this change, `WITH_SEASTAR` is translated to `with_seastar`
variable in `install-deps.sh`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cbt.py: change port to work with client_endpoints

hack to work with https://github.com/ceph/cbt/pull/160/

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28439 from yuriw/wip-yuriw-crontab-master

qa/tests:  filtered in only "trusty"

Merge PR #27731 into master

* refs/pull/27731/head:
qa: use mimic-O upgrade process

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests:  filtered in only "trusty"

Fixes: http://tracker.ceph.com/issues/40195
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw_file: permit lookup_handle to lookup root_fh

Originally, root_fh was reachable from RGWLibFS::fh_cache, but
this was problematic as it has infinite lifetime and so is not
present in RGWLibFS::fs_lru.

To fix the above, RGWLibFS::root_fh came to live unhooked from the
handle caches which works because nfs-ganesha/the application
always has an instance handle.  However, this silently breaks
READDIR event invalidation.

Fixes: https://tracker.ceph.com/issues/40196
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa: use mimic-O upgrade process

Fixes: https://tracker.ceph.com/issues/39436
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27015 into master

* refs/pull/27015/head:
OSD: rollforward may need to mark pglog dirty

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28242 into master

* refs/pull/28242/head:
osdc/Objecter: fix OSDMap leak in handle_osd_map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #28363 into master

* refs/pull/28363/head:
osd/ClassHandler: use std::variant for storing func
osd/ClassHandler: no need to add `struct` before struct type
osd/ClassHandler: use ceph::mutex instead of Mutex
osd/ClassHandler: remove unused function
osd/ClassHandler: s/string/std::string/ in header

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: reduced distro to run to be random

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rbd-mirror: link against the specified alloc library

As a daemon, use the build-time specified alloc library
(tcmalloc/jemalloc) when linking the application.

Fixes: http://tracker.ceph.com/issues/40110
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephfs-shell: Remove unnecessary empty line

This patch fixes flake8 too many blank lines error.

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 bare 'except' warnings

Instead of bare 'except', catch OSError and libcephfs exceptions.

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Remove unused variable 'e'

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 use != to compare str literals warning

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fixe flake8 line too long error

Break the line into multiple lines and add new variable wherever necessary, to
fix the line too long error.

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #28313 from Songweibin/wip-fix-get-args

rbd/action: fix error getting positional argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28253 from Songweibin/wip-39973-rbd

osd/OSDCap: rbd profile permits use of rbd_info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27902 from mikechristie/rbd-nbd-netlink-support

rbd-nbd: add netlink map/unmap support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common: OutputDataSocket retakes mutex on error path

the list of buffers is protected by mutex m_lock. when dump_data() fails
and puts unwritten buffers back into the list, it needs to reaquire the
lock

Fixes: http://tracker.ceph.com/issues/40188
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: add unittests

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

mgr/influx: try to call close()

let's at least try to call influxclient.close() which in turn closes the
requests.Session

see
https://github.com/influxdata/influxdb-python/blob/v4.1.1/influxdb/client.py
and
https://github.com/influxdata/influxdb-python/blob/v5.0.0/influxdb/client.py

Fixes: http://tracker.ceph.com/issues/40174
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28370 from linuxbox2/wip-rgw-tntbucket

rgw_file: include tenant when hashing bucket names

common,osd: add hash algorithms for dedup (sha256, sha512)

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

d/control,install-deps: use d/control for installing seastar deps

prepare for building ceph-crimson-osd deb package

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: show degraded/misplaced/unfound objects. (#28104)

mgr/dashboard: show degraded/misplaced/unfound objects.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28039 from majianpeng/make-tcmalloc/jemalloc-work-well

make perf_async_msgr link jemalloc/tcmalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28373 from xiexingguo/wip-40104

mon, osd: parallel clean_pg_upmaps

Reviewed-by: Sage Weil <sage@redhat.com>

mds: count purge queue items left in journal

MDS purge queue didn't have a perf counter to record how many items
still left in journal. Even when MDS restarted, there was no any hint to
know how many inodes haven't been really deleted from disks.

Fixes: http://tracker.ceph.com/issues/40121
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

test/crimson: make perf_async_msgr link tcmalloc/jemalloc.

For 1M performance, perf_async_msgr w/ tcmalloc/jemalloc is better than
libc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: add crimson-osd as a dependency of "tests" target

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: add parallel clean_pg_upmaps test

With parallel clean_pg_upmaps feature on, the total time cost
of the performance test which now can utilize up to 8 threads for
parallel upmap validating decreased from:

maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:104s

to:

clean_pg_upmaps (~10000 pg_upmap_items) latency:7s

Note that by default the mon uses only 4 worker threads for
CPU intensive background work, you could further increase
the "mon_cpu_threads" option value if you decided the
time-consuming of clean_pg_upmaps still matters.

Fixes: http://tracker.ceph.com/issues/40104
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: do clean_pg_upmaps the parallel way if necessary

There could definitely be some certain cases we could reliably
skip this kind of checking, but there is no easy way to separate
those out.
However, this is clearly the general way to do the massive pg
upmap clean-up job more efficiently and hence should make sense
in all cases.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: split clean_pg_upmaps into smaller helpers

- it's good to read.
- the updating pending_inc part should be made independent
  since it is going to be racy while running in parallel.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28402 from tchaikov/wip-crimson-cleanup

crimson/osd: some cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28302 from dzafman/wip-40078

test: Make sure that extra scheduled scrubs don't confuse test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

selinux: Update the policy for RHEL8

We hit a couple more SELinux denials when running ceph on RHEL8. The
dac_read_search change is related to a kernel change where it checks
dac_read_search before dac_override, now.

Signed-off-by: Boris Ranto <branto@redhat.com>

rgw_file: pretty-print fh_key

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: include tenant in hashes of object

Because bucket names are taken as object names in the top
of an export.  Make hashing by tenant general to avoid disjoint
hashing of bucket.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>