Merge PR #19007 into master

* refs/pull/19007/head:
src/libcephfs.cc: Cleanup nullptr initializations against deleted objects, etc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #19014 into master

* refs/pull/19014/head:
mds: properly execute scrub finish context

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19018 into master

* refs/pull/19018/head:
mds: don't delay processing completed requests in replay queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19060 into master

* refs/pull/19060/head:
client: anchor Inode while trimming caps

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #19102 from jecluis/wip-qa-find-used-ports

qa: add simple and dirty script to find ports being used

Merge pull request #19044 from idryomov/wip-resurrect-msg-auth

msg: resurrect support for !CEPH_FEATURE_MSG_AUTH

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa: add simple and dirty script to find ports being used

Seriously, having this is way better than realizing that one needs to
find an unused port each time a new test is created.

It's far from a perfect script. It's downright not elegant.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #19082 from trociny/wip-qa-demote-fixup

qa/workunits/rbd: wait for status propagated only if daemon started

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19067 from tchaikov/wip-nvme-mgr-fixes

bluestore/NVMEDevice: do not deference a dangling pointer

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19035 from ifed01/wip-ifed-rep-backend-speedup

osd: make operations on ReplicatedBackend::in_progress_ops more effective

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17799 from amitkumar50/cov-misc-7

osd,messages: Initialize read_length,options,send_reply

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17792 from amitkumar50/cov-misc-5

messages,journal: Initialization of stats_period,m_active_set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17772 from amitkumar50/cov-misc-3

client,osd,test: Initialize fuse_req_key,snap,who,seq

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17767 from amitkumar50/cov-misc-2

messages,test,msg: initialize h,reply_type,owner

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17766 from amitkumar50/cov-misc-1

common,os: initialize commit_data,cmount,iocb

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17443 from JiYou/restapi-default-port

ceph-rest-api: when port=0 use the DEFAULT_PORT instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17085 from yaozongyou/remove_unused_using

librgw_file: remove unused `using` statement.

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19017 from tchaikov/wip-bluestore-without-aio

bluestore,cmake: enable building bluestore without aio

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19029 from tchaikov/wip-cmake-sysctl-cleanup

cmake: only create sysctl file on linux

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19072 from smithfarm/wip-22207

tests: ceph-disk: ignore E722 in flake8 test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #17678 into master

* refs/pull/17678/head:
mon/AuthMonitor: improve error message
mon/OSDMonitor: disallow "all" as a key or value name
cephfs, mon/AuthMonitor, OSD/osdcap: make 'all' a synonym for '*'
vstart.sh: Create an admin user for each CephFS
mon/AuthMonitor: Allow * wildcard for filesystem name
OSD/OSDCap: Allow namespace and pool tag to be combined
OSD/OSDCap: Namespace globbing
mon/AuthMonitor: Use new osd auth caps for ceph fs authorize
OSD/auth caps: Add OSD auth caps based on pool tag
mon/FSCommands: Tag pools used for cephfs by default
mon/OSDMonitor: Add key/value arguments for pool tagging

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/rbd: wait for status propagated only if daemon started

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19000 from dillaman/wip-15322

rbd-mirror: removed dedicated thread from image deleter

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17072 from rzarzynski/wip-rgw-21015

rgw: incorporate the Transfer-Encoding fix for CivetWeb.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19073 from trociny/wip-qa-demote

qa/workunits/rbd: wait for demote status is propagated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19075 from tchaikov/wip-22154

qa/workunits: silence py warnings for ceph-disk tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rbd-mirror: fix possible race condition in error path tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19064 from qrGitHub/wip-rgw-typo-refity

rgw: correct typo refity to refit

Reviewed-by: Amit Kumar <amitkuma@redhat.com>

rbd-mirror: removed dedicated thread from image deleter

Fixes: http://tracker.ceph.com/issues/15322
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image deleter now takes a librados::IoCtx shared pointer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: move image deletion to standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: move snap purge to standalone state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure deep-copy can remove an unprotected snapshot

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: added async version of get_snapcontext

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18790 from amitkumar50/cov-rbd-6

rbd: null check before pool_name use

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: null check before pool_name use

Fixes the coverity issue:

CID 1394846 (#1 of 1): Dereference after null check (FORWARD_NULL)
15. var_deref_model: Passing null pointer pool_name to regex_match,
which dereferences it.

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

qa/workunits: silence py warnings for ceph-disk tests

ceph-disk now prints "depreacted" warning message when it starts. but
the tests parses its stdout and stderr for a json string. so we need to
silence the warnings for the tests.

Fixes: http://tracker.ceph.com/issues/22154
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: wait for demote status is propagated

(before promoting the image on another cluster)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #19045 from dillaman/wip-22200

rbd: disk usage on empty pool no longer returns an error message

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

tests: ceph-disk: ignore E722 in flake8 test

Very old, and very new, versions of flake8 treat E722 as an error:

flake8 runtests: commands[0] | flake8 --ignore=H105,H405,E127 ceph_disk tests
ceph_disk/main.py:1575:9: E722 do not use bare except'
ceph_disk/main.py:1582:9: E722 do not use bare except'
ceph_disk/main.py:3252:5: E722 do not use bare except'
ceph_disk/main.py:3288:21: E722 do not use bare except'
ceph_disk/main.py:3296:17: E722 do not use bare except'
ceph_disk/main.py:4358:5: E722 do not use bare except'
tests/test_main.py:26:1: E722 do not use bare except'
ERROR: InvocationError: '/opt/j/ws/mkck/src/ceph-disk/.tox/flake8/bin/flake8 --ignore=H105,H405,E127 ceph_disk tests'

Fixes: http://tracker.ceph.com/issues/22207
References: https://gitlab.com/pycqa/flake8/issues/361

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: BuildDPDK: use ${CMAKE_MAKE_PROGRAM} when appropriate

DPDK requires GNU Make. see
spdk/dpdk/doc/guides/freebsd_gsg/build_dpdk.rst. so, we should use the
configured make program to do the job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: BuildDPDK: use POSIX compliant syntax

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore/NVMEDevice: do not deference a dangled pointer

* pass coremask_arg to the working thread by value.
  after 81249ab9d, get_val<>() returns a temporary variables instead of a
  reference to the variant held by config. so calling string::c_str(), and
  passing the returned `const char*` to a thread is not advisable. instead
  we should pass a string by value. since spdk_env_init() will copy the
  settings passed in by opt. it's safe even to destruct the coremask_arg
  afterward.
* use ffsll() to find the LSB. it's not a bottleneck, but it's easier
  and simpler, and probably improves the readability.
* refactor the NVMEManager::try_get() method: to define the variables
  when they are used for the first time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #18274 into master

* refs/pull/18274/head:
mds: fold mds_revoke_cap_timeout into mds_session_timeout
client: add new delegation testcases
client: add delegation support for cephfs
common: remove data_dir_option from common_preinit and global_pre_init

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: correct typo refity to refit

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

client: anchor Inode while trimming caps

This prevents the Inode from being deleted until after cap trimming is
finished. In particular, this prevents remove_all_caps from being called which
screws up the traversal of caps in trim_caps.

Fixes: http://tracker.ceph.com/issues/22157
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18710 from com6056/wip-doc-fix-cephfs-snap-dir

doc: Fixes the name of the CephFS snapshot directory.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10992 from zhangsw/fix-rgw-cache-bug

rgw: fix a bug in rgw cache in delete_system_obj and get_system_obj.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

mon/AuthMonitor: improve error message

Improve the error message for 'ceph fs authorize' where the
given entity exists but has different auth caps.

Fixes: http://tracker.ceph.com/issues/21765
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #19011 from trociny/wip-mirror-missing-lock

rbd-mirror: missing lock when re-sending update_sync_point

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19016 from trociny/wip-valgrind-unittest_rbd_mirror

test/rbd_mirror: fix valgrind warnings in unittest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18799 from ivancich/wip-remove-StreamObjData

rgw clean-up: remove unreferenced pure virtual class StreamObjData

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #19022 into master

* refs/pull/19022/head:
githubmap: Add oritwas

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: disk usage on empty pool no longer returns an error message

Fixes: http://tracker.ceph.com/issues/22200
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

msg/simple/Pipe: resurrect support for !CEPH_FEATURE_MSG_AUTH

Revert CEPH_FEATURE_MSG_AUTH-related hunks of commit 7ba4d5324e2d
("msg/simple/Pipe: remove legacy feature case handle"), which
removed code that handled !CEPH_FEATURE_MSG_AUTH footers.  It's needed
for kernels prior to 3.19, where cephx signatures aren't supported.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #18987 from ivancich/wip-remove-sys-obj-get-state

rgw clean-up: remove unused var & func in RGWRados::SystemObject

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

msg/async/AsyncConnection: resurrect support for !CEPH_FEATURE_MSG_AUTH

Revert CEPH_FEATURE_MSG_AUTH-related hunks of commit 83f7e392a163
("msg/async/AsyncConnection: remove legacy feature case handle"), which
removed code that handled !CEPH_FEATURE_MSG_AUTH footers.  It's needed
for kernels prior to 3.19, where cephx signatures aren't supported.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #18173 from gmayyyha/osd-status-21707

mgr: fix "osd status" command exception if OSD not in pgmap stats

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19024 from tchaikov/wip-21474

qa/tasks: prolong revive_osd() timeout to 6 min

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18667 from Leeshine/wip-rgw-bug-21615

rgw: radosgw-admin zonegroup get and zone get return defaults when there is no realm

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18302 from joscollin/wip-update-radosgw-admin-usage

rgw: Various cleanups and options update in rgw_admin.cc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18979 from cfanz/dump-header-improve-wip

rgw: clean code with helper function dump_header_if_nonempty
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18985 from theanalyst/wip-keystone-logging

rgw: keystone: bump up logging when error is received
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: make operations on ReplicatedBackend::in_progress_ops a bit more effective.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

cmake: only create sysctl file on linux

and check 64bit platform by using the sizeof(void*)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18980 from shinobu-x/os_filestore_filestore_nit_cleanup

os/filestore/FileStore: Initialized by nullptr, NULL or 0 instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18982 from neha-ojha/wip-const-cleanup

osd/PG: const cleanup for recoverable/readable predicates

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18934 from tchaikov/wip-option-return-by-ref

Revert "common/config: return const reference instead of a copy"

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #18916 from tchaikov/wip-22113

osd: subscribe osdmaps if any pending pgs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18816 from amitkumar50/cov-common-2

common: Check this->data.op_size before use

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18810 from majianpeng/fix-21982

osdc/Objecter: record correctly value for l_osdc_op_send_bytes.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18777 from majianpeng/bluestore-collection-list

os/bluestore: optimize _collection_list.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18544 from ddiss/pid_max

sysctl.d: set kernel.pid_max=4194304

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18718 from shinobu-x/sk-ceph_osd

ceph_osd.cc: Drop legacy or redundant code

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks: prolong revive_osd() timeout to 6 min

see also #17902

Fixes: http://tracker.ceph.com/issues/21474
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: complain if spdk/pmem is enabled w/o bluestore

and complain if bluestore is enabled w/o any backend found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: assert(!num_running) before releasing running_aios

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: extract IOContext logic out of BlueStore

to avoid HAVE_LIBAIO appearing in BlueStore.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

githubmap: Add oritwas

Added oritwas to .githubmap.

Signed-off-by: Jos Collin <jcollin@redhat.com>

cmake: check libaio on linux before building SPDK

SPDK requires libaio on linux.

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: enable building bluestore w/o libaio

KernelDevice is tightly coupled with libaio. more work is needed to
decouple aio from it. but by guarding KernelDevice with HAVE_LIBAIO, we
can enable bluestore on platforms w/o libaio.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: don't delay processing completed requests in replay queue

Completed requests can also be in replay queue, they need to be processed
(treated as lookup request) in clientreplay stage. MDS will be stuck at
clientreplay stage if it delays processing requests of this type.

the bug was introduced by commit 0afbc033
(mds: don't rdlock locks in replica object while auth mds is recovering)

Fixes: http://tracker.ceph.com/issues/22163
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

test/rbd_mirror: fix valgrind warnings in unittest

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18776 from JianyuLi/joa-mul-mds

MDS: make popular counter decay at proper rate

Merge pull request #18784 from taodd/client-bugfix

Client:Fix readdir bug

Merge pull request #18786 from taodd/master

Client: setattr should drop "Fs" rather than "As" for mtime and size

Merge pull request #18836 from taodd/cephfs-hadoop

MDS: correct the error msg when init mon client

Merge pull request #18892 from taodd/client-bugfix2

MDS: add null check before we push_back "onfinish"

Merge pull request #18893 from taodd/mds-inotable-cleanup

mds: simplify the code logic in replay_alloc_ids

Merge pull request #18904 from taodd/mds-sesseionmap-cleanup

mds: get rid of the "if" check which is unnecessary inside a loop

mds: properly execute scrub finish context

Bug was introduced by commit 7e52729699 (mds: flush after scrub repairs)

Fixes: http://tracker.ceph.com/issues/22058
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

rbd-mirror: missing lock when re-sending update_sync_point

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

src/libcephfs.cc: Cleanup nullptr initializations against deleted objects, etc

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #19003 from dillaman/wip-22158

librbd: set deleted parent pointer to null

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

librbd: set deleted parent pointer to null

Fixes: http://tracker.ceph.com/issues/22158
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18963 from amitkumar50/test-cov-test-9

test: Incorrect conversion to double

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18901 from jan--f/jan-crushtool-decompile-missing-ids

tools/crushtool: skip device id if no name exists

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18939 from majianpeng/bluestore-reap-collection

os/bluestore: fix the wrong usage for map_any

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18913 from yaozongyou/wip-rados-add-touch-command

tools/rados: add touch command to change object modification time

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18848 from jcsp/wip-22098

mgr/dashboard: fix audit log loading

Reviewed-by: Kefu Chai <kchai@redhat.com>