mds: fix hang issue when accessing a file under a lost parent directory

When a directory metadata is lost, if we looked up a file under this lost directory,
this request will hang forever.

Fixes: https://tracker.ceph.com/issues/46129
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #35658 from dillaman/wip-46083

librbd: race when disabling object map with overlapping in-flight writes

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35588 from jschmid1/osdspec_affinity_to_metadata

osd: expose osdspec_affinity to osd_metadata

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35569 from sungjunyoung/master

install_deps.sh: Add debian based elementary os

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35294 from agayev/punch-hole

os/bluestore: Add missing punch_hole call so that object offsets are …

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34189 from changchengx/bufferlist

common/buffer: refine bufferlist implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35592 from tchaikov/wip-ceph-in-de2

ceph.in: fallback to install path of "ceph" when "ceph-conf" is not found

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35078 from idryomov/wip-msgr21

msg/async: New msgr2 crc and secure modes (msgr2.1)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35607 from wuxingyi/schedulerlogger

osd/scheduler: add white space for better readability

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35584 from aclamk/wip-mempool-bufferlist-c_str

common/mempool: copies bufferlist assignment to mempool through rebuild

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #32271 from toliu/disable-restful-basic-auth

mgr/restful: allow users to disable http authentication

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: race when disabling object map with overlapping in-flight writes

The block guard that protects against overlapping updates to the object
map needs to be flushed prior to closing the object map instance.

Fixes: https://tracker.ceph.com/issues/46083
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33823 from zdover23/wip-doc-ceph-objectstore-tool-2020-mar

doc/man: add manpage for ceph-objectstore-tool

Merge pull request #35681 from yuriw/wip-yuriw-crontab-master

qa/tests: removed mimic runs as EOL

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: removed mimic runs as EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35673 from tspmelo/wip-chromium

mgr/dashboard: Fix e2e chromium binary validation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Fix e2e chrome and chromium binaries verification

Fixes: https://tracker.ceph.com/issues/46110
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35223 from smithfarm/wip-releases-ordering

doc/releases: list archived releases in correct order

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35586 from tspmelo/wip-service-root

mgr/dashboard: Provide all services in root

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30670 from trociny/wip-41736

mgr: fix race between module load and notify

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #35141 from matthewoliver/cephadm_iscsi_lib_module_mount

cephadm: Give iscsci a RO /lib/modules bind mounted

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35190 from cybozu/ceph-volume-fix-rejected-reason-in-device-type-mismatch

ceph-volume: report correct rejected reason in inventory if device type is invalid

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: show correct rejected reason in inventory if device type is not acceptable

If device type is not acceptable in `c-v inventory`, its rejected reason
becomes "Insufficient space (<5GB)" by mistake. It's because sys_api is
empty due to skipping devices that are neither `disk` nor `device`. We
should report the target device is not acceptable in this case.

Fixes: https://tracker.ceph.com/issues/46102
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>

ceph-volume: cleanup code

Simplify the logic and fix a typo.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>

incorporating dzafman's suggestions

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35652 from rzarzynski/wip-crimson-thread-cleanup

crimson: clean-up the crimson/thread directory

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35632 from dzafman/wip-46064

tools: Add statfs operation to ceph-objecstore-tool

Reviewed-by: Neha Ojha <nojha@redhat.com>

tools: Add statfs operation to ceph-objecstore-tool

Fixes: https://tracker.ceph.com/issues/46064
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #35329 from trociny/wip-quiesce_complete

librbd: quiesce/unquiesce API improvements

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #35649 into master

* refs/pull/35649/head:
vstart: allow specifying a different ganesha binary
vstart: handle multiple ganesha's more sanely

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

vstart: allow specifying a different ganesha binary

Signed-off-by: Jeff Layton <jlayton@redhat.com>

vstart: handle multiple ganesha's more sanely

Currently, if you have more than one ganesha daemon, it clobbers the
log, config and pidfiles from the previous daemons. Give them each
their own files.

While we're in here, move the log and pidfile to the out/ directory
with all of the other logfiles/pidfiles.

Fixes: https://tracker.ceph.com/issues/46079
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35650 from trociny/wip-46076

test/rbd_mirror: fix race on test shut down

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35631 from tchaikov/wip-cmake-cleanup

cmake: use add_compile_options() and add_definitions()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35630 from tchaikov/wip-systemd-subvars

cmake: move include(GNUInstallDirs) up

Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: move Throttle to src/crimson/common.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make ThreadPool specific to AlienStore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/rbd_mirror: fix race on test shut down

in TestMockImageReplayerSnapshotReplayer.UnlinkRemoteSnapshot

Fixes: https://tracker.ceph.com/issues/46076
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35637 from bk201/wip-46067

stop.sh: do not block script when there is no running cluster

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #35603 from dillaman/wip-46034

mgr/prometheus: automatically discover RBD pools for stats gathering

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Provide all services in root

Fixes: https://tracker.ceph.com/issues/46030
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35574 from tspmelo/wip-tslint

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35634 from mgfritch/cephadm-manpage-shell-mount

doc/man: add missing cephadm shell `--mount` option

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: add definitions using add_definitions()

also, add definitions as a list instead of a single string for better
readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use add_compile_options()

instead of appending compile flags to CMAKE_C_FLAGS, use
add_compile_options(), as COMPILE_OPTIONS is a list, it'd simpler to
append options to it and to access it in a structured way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use string(APPEND...) for appending cflags

* use string(APPEND...) for less repeatings
* `-rdynamic` is a linker option, so it should be added to
  `CMAKE_EXE_LINKER_FLAGS` for adding symbols to dynamic
  symbol tables so they can be consumed by plugins.

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: do not block script when there is no running cluster

A query for current fsid is called inside `do_killcephadm`. This blocks
the script when there is no running cluster. The fix avoids entering the
function if cephadm command fails or returns no daemons.

The change also hides the following output for non-cephadm environments:

```
Unable to locate any of ['podman', 'docker']
```

Fixes: https://tracker.ceph.com/issues/46067
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35469 from xxhdx1985126/wip-crimson-block-io-for-recovery

crimson: block io for recovery ---- recovery part 2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35626 into master

* refs/pull/35626/head:
mds: fix whitespace

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/man: add missing cephadm shell `--mount` option

also clean-up `--mount` help usage

Signed-off-by: Michael Fritch <mfritch@suse.com>

crimson/osd: share map with clients when osdmap epoch doesn't match

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: do not assert in start_backfill_ops

osd_recovery_max_single_start is not necessarily used up in start_recovery_ops, and
start_backfill_ops could be invoked even there's no need to backfill, so do not assert
here, just return.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: fix client op version issue

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/monc: take care of connection lost when sending messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: send "mark me down" messages to monitor when shutting down

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: restart ongoing client_request after peering

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: block io for object recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

Merge pull request #35507 from aclamk/wip-bluestore-avl-description

BlueStore, added descriptions for AVL allocator parameters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: move include(GNUInstallDirs) up

so systemd/CMakeLists.txt can have access to the variables defined by
it. quote from https://cmake.org/cmake/help/latest/command/include.html.

> Variable reads and writes access the scope of the caller (dynamic
> scoping).

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35623 from liewegas/wip-bloomberg

doc/foundation: add bloomberg

doc/dev/msgr2: fix inconsistencies and update for msgr2.1

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

msg/async/ProtocolV2: add msgr2.1 feature bit

Use msgr2.1 if the peer supports it and fall back to msgr2.0
otherwise.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

msg/async/ProtocolV2: store supported features instead of required

We aren't interested in peer_required_features anywhere outside
_handle_peer_banner_payload() -- once we know there is no mismatch,
it's all about peer_supported_features.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

msg/async/ProtocolV2: short circuit empty segments and epilogue

In both msgr2.0 and msgr2.1, segments can be empty.  In msgr2.1,
epilogue can be empty as well.

Handle both by calling the respective handler function directly
instead of allocating a buffer::ptr_node for an empty buffer and
passing that through READ[_RXBUF].

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

msg/async/frames_v2: add initial unit tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

msg/async/crypto_onwire: implement msgr2.1 nonce format

Move to a 64-bit counter to avoid wrapping and having to reset
the session before the counter repeats.  This is in line with NIST
Recommendation for GCM [1]:

  "... this Recommendation suggests, but does not require, that
  the leading (i.e., leftmost) 32 bits of the IV hold the fixed
  field; and that the trailing (i.e., rightmost) 64 bits hold the
  invocation field."

See commit bb61e6a5adc3 ("msg/async/ProtocolV2: avoid AES-GCM nonce
reuse vulnerabilities").

[1] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

msg/async/frames_v2: implement msgr2.1 wire format

Implement msgr2.1-crc and msgr2.1-secure modes.

Issues with existing msgr2.0-crc and msgr2.0-secure modes and
their resolution will be described in doc/dev/msgr2.rst.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/foundation: add bloomberg

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: Fix tslint warnings

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve typescript lint

Fixes: https://tracker.ceph.com/issues/46000
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35514 from dzafman/wip-41564

mon: Warn when too many reads are repaired on an OSD

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34933 from tspmelo/wip-remove-entry

mgr/dashboard: Remove entryComponents

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35505 from tspmelo/wip-ng-dropdown

mgr/dashboard: Use ng-bootstrap for Dropdown

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #32111 into master

* refs/pull/32111/head:
test_cephfs_shell: run cephfs-shell with conf file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #35539 into master

* refs/pull/35539/head:
vstart_runner: set omit_sudo to True by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35608 into master

* refs/pull/35608/head:
cephfs_features: add metric collection feature support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35578 from mgfritch/cephadm-infer-fsid-ukn

cephadm: skip any invalid fsid(s) during infer

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35567 from ceph/cephadm_annotate_scheduler

mgr/cephadm: annotate scheduling code with comments

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35492 from liewegas/wip-bocloud

doc/foundation: drop BoCloud

Merge pull request #35581 from bk201/ganesha-doc-pool-existence

mgr/cephadm: document pool requirement for NFS-Ganesha service

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Alexandra Settle <asettle@suse.com>

Merge pull request #35225 from ishanrai05/bootstrap-variables

mgr/dashboard: use bootstrap variables for colors

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/prometheus: automatically discover RBD pools for stats gathering

To simplify configuration, the existing 'rbd_stats_pools' config option
now accepts '*' as a wildcard for the pool and/or namespace portion of
the <pool>[/<namespace>] spec.

Fixes: https://tracker.ceph.com/issues/46034
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35559 from xiexingguo/wip-split-child-stat

osd/PeeringState: fix info.stats.up[acting] of splitting child

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: yanjun <yan.jun8@zte.com.cn>

test_cephfs_shell: run cephfs-shell with conf file

And set colors to False and debug to True in this conf file.

stdout captured by Python code in this testsuite can contain the colour
codes along with the expected string output. Setting colors to False in
cephfs-shell.conf should fix this.

Setting debug to False should help with analyzing test failures since
it enables printing messages that helps debugging (stacktrace for
cephfs-shell code, for example).

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35395 from kshtsk/wip-registry-conf

qa/tasks/cephadm: setup site based container registry

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35277 from aclamk/wip-limit-wal-sharding

kv/RocksDBStore: Set size for WALs in sharding mode.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #35547 from adk3798/cephadm-45097

cephadm: UX: Change error message when 'orch host add <host>' fails

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #35552 from jan--f/systemd-osd-libexec

systemd/ceph-osd: ceph-osd-prestart.sh now lives in /usr/libexec

Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35558 from xiexingguo/wip-45991

osd/PG: fix history.same_interval_since of merge target again

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35497 from ifed01/wip-ifed-remove-wal-preext

os/bluestore: remove preextended WAL support.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35549 from mgfritch/cephadm-iscsi-strip-version

cephadm: strip whitespace from iscsi version

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #34643 from aclamk/wip-fix-bluefs-log-dump-v2

os/bluestore/BlueFS: Make log_dump possible without opening bluefs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35146 from wjwithagen/wjw-fix-shufflerandom

rgw: replace deleted std function: randomshuffle

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35609 from tchaikov/wip-mgr-volumes-nfs-fstring

mgr/volumes/nfs: use f-string for creating Exception

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35466 from ishanrai05/quotes

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35614 from tchaikov/wip-install-deps-enable-required-crb-repo

install-deps.sh: do not enable unavailable repos

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #35295 from rhcs-dashboard/fix-snapshot-dropdown

mgr/dashboard: cropped actions menu in nested details

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35493 from mgfritch/orch-service-spec-validate

python-common: fix ServiceSpec validation

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

install-deps.sh: do not enable unavailable repos

should be specific when enabling codeready-builder repos, there is
chance that some repos are just not available, while the required one
is. for instance, "codeready-builder-for-rhel-8-x86_64-eus-debug-rpms"
might not be available. and in that case, `install-deps.sh` just fails.

so in this change, only the required one is enabled. see also
https://fedoraproject.org/wiki/EPEL

Signed-off-by: Kefu Chai <kchai@redhat.com>