qa/distros: configure podman to pull from docker-mirror

This also requires a change to cephadm.py to search for
registry entries for docker.io by 'prefix' or 'location',
and to only add the mirror if a mirror isn't already present

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #38638 from idryomov/wip-krbd-stable-writes-attr

qa: krbd_stable_pages_required.sh: move to stable_writes attribute

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38635 from ronen-fr/wip-ronenf-crimsclang-tree

crimson/seastore: fixing Clang errors and warnings

Merge pull request #38602 from dillaman/wip-48486

librbd: explicitly disable readahead for writearound cache

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38645 from cyx1231st/wip-seastar-onode-tree-fix-clang-link

crimson/onode-staged-tree: fix explicit template instantiation for clang

crimson/onode-staged-tree: fix explicit template instantiation for clang

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38618 from neha-ojha/wip-14-2-16

doc/releases/nautilus.rst: add release notes for 14.2.16

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38622 from neha-ojha/wip-15-2-8

doc/releases/octopus.rst: add release notes for 15.2.8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38351 from yuvalif/wip-yuval-fix-46128

rgw/notifications: automatically delete topics associated with a bucket

Merge pull request #38246 from yuvalif/wip-yuval-fix-48321

rgw/notifications: fix tag based filtering to work on all ops

Merge pull request #38190 from yuvalif/wip-yuvali-fix-48293

rgw/amqp/kafka: prevent concurrent shutdowns from happening

Merge pull request #37924 from yuvalif/allow_packages_in_lua

rgw/lua: support packages via luarocks

Merge pull request #38284 from rhcs-dashboard/48355-read-pwd-from-file

mgr/dashboard: CLI commands: read passwords from file

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: knortema <knortema@redhat.com>

Merge pull request #38473 from votdev/issue_48474_health_check

mgr/dashboard: Display a warning message in Dashboard when debug mode is enabled

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38599 from tchaikov/wip-47977

ceph.in: try harder if asok path does not exist

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38447 into master

* refs/pull/38447/head:
ceph-volume: disable cache for blkid calls

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38636 from ceph/fix-labeler-master

.github/labeler: fix yaml syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38634 from tchaikov/wip-install-deps

install-deps.sh: do not install libboost-all-dev and nlohmann-json-dev

Reviewed-by: Josh Durgin <jdurgin@redat.com>

Merge pull request #38601 from linuxbox2/more-librgw-zipper9

More librgw zipper9

Merge pull request #23396 from linuxbox2/wip-rgw-xattrs-2

rgw_file:  expose RGW user-defined attributes

Merge pull request #37138 from lxbsz/iscsi_qa

qa: add ceph iscsi test suite support

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: add ceph iscsi test suite support

Currently this only supports the gwcli tests on CentOS/RHEL 8 with
2 ceph iscsi gateways.

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa: krbd_stable_pages_required.sh: move to stable_writes attribute

bdi/stable_pages_required attribute was deprecated in 5.10 and now
always returns 0.  The replacement is queue/stable_writes.  (It is
also writeable, so we can simplify these test cases somewhat in the
future.)

Fixes: https://tracker.ceph.com/issues/48232
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks: get the correct clone_dir for each client

The clone_dir will be overrided in the loop.

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks: allow none client prefixed node to run the cram

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks: fix not enough values to unpack value error

Fixes: https://tracker.ceph.com/issues/47439
Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/seastore: fixing Clang errors and warnings

Errors: capturing structured-binding "special ref"

Warnings: unused captures

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

.github/labeler: fix yaml syntax

Introduced by https://github.com/ceph/ceph/pull/38617

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

install-deps.sh: do not install nlohmann-json{3}?-dev

"mk-build-deps --install" should be able to take care of this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: do not install libboost-all-dev for old distros

ubuntu disco and ubuntu focal do not ship libboost 1.72 and up, and
we depend on libboost 1.72 or up, so it does not help to install
liboost 1.67 or libboost 1.71 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/BaseMgrModule: add optional inbuf param to ceph_send_command.

BaseMgrModule.cc: ceph_send_command: add inbuf param for handling input file content.

Fixes: https://tracker.ceph.com/issues/48355
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: CLI commands: read passwords from file

Fixes: https://tracker.ceph.com/issues/48355
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38617 from batrick/cmake-labeler

.github/labeler: add build/ops label

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38629 from changchengx/sudo_fix

install-deps.sh: install debian package with proper right

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38464 into master

* refs/pull/38464/head:
mds/OpenFileTable: make the data length as long as possible for each op

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38363 into master

* refs/pull/38363/head:
mds/OpenFileTable: enhance the log info
mds/OpenFileTable: add _read_omap_values() to simplify the code

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38355 into master

* refs/pull/38355/head:
client: when STATX_NLINK is requested request FsLs caps
client: ensure we take Fs caps when querying dirstat vxattrs

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38236 into master

* refs/pull/38236/head:
qa: forward scrubbing background task

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38222 into master

* refs/pull/38222/head:
client: set CEPH_STAT_RSTAT mask for dir in readdir_r_cb

Reviewed-by: Jeff Layton <jlayton@redhat.com>

install-deps.sh: install debian package with proper right

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

.github/labeler: add build/ops label

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38624 from rzarzynski/wip-crimson-nopack-in-blocking_future

crimson/osd: drop arg packs from blocking_future.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38450 from zdover23/wip-doc-dev-guide-basic-workflow-auto-pr-valid-dec-2020-1

doc/dev: update auto pr validation section

Reviewed-by: Josh Durgin <jdurgin@redat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38600 from agayev/master

os/bluestore: Fix code rot in ZonedAllocator.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.in: try harder if asok path does not exist

when handling "ceph daemon <name> ...", we need to figure out the asok
path, if the "<name>" option is not a path but a "name" as it claims to
be. but if the "admin_socket" option includes substitution/meta variable of
"$pid", it will be expanded using its own pid, instead of the process
with the specified "name". but "ceph" cli does not offer a dedicated
option for overriding the pid for ceph-conf which is used "ceph" cli to
retrieve the "admin_socket" option with specified "name".

in this change, "<name>" option is interpreted as "${name}.${pid}" if
the returned admin_socket path does not point to a valid UNIX domain
socket file. and "ceph" cli will try harder by passing "--pid ${pid}"
and "--name ${name}" instead of "--name ${name}.${pid}" to ceph-conf.

Fixes: https://tracker.ceph.com/issues/47977
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38609 from ideepika/fix-yaml-failure

cmake: fix build failure due to yaml-cpp not available

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38608 from tchaikov/wip-crimson-build

rpm,deb: conditionally package crimson-store-nbd in ceph-test

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38578 from cyx1231st/wip-seastar-onode-tree-fix-stage-type

crimson/onode-staged-tree: fix match_stage_t related compile issue

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38625 from batrick/tox-fix

qa: fix tox failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: fix tox failures

Fixes: c6121e48ecb7224e5addf4b5fdeb64907a8d8683
Fixes: https://tracker.ceph.com/issues/48633
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: drop arg packs from blocking_future.

This feature is not necessary as the `seastar::feature` lost
the support for `T...`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/releases/octopus.rst: add release notes for 15.2.8

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/releases/nautilus.rst: add release notes for 14.2.16

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38548 from rhcs-dashboard/wip-48570-master

github: add action to manage needs-rebase label

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge branch 'CVE-2020-27781' into master

* CVE-2020-27781:
  tasks/cephfs/test_volume_client: Add tests for authorize/deauthorize
  pybind/ceph_volume_client: Optionally authorize existing auth-ids
  pybind/ceph_volume_client: Preserve existing caps while authorize/deauthorize auth-id
  pybind/ceph_volume_client: Disallow authorize auth_id

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

rgw_file: correct initialization of RGWWriteRequest

The legacy call to init_from_header() in RGWWriteRequest()
conflicted with initialization via ::init()

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: fix librgw_file_gp unit test and driver

Fixes missing commit after rgw_write(), as well as conditional
cleanup.

Reorder tests, and split out librgw_file_gp --delete to avoid an
issue w/sysobj cache (temporary).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: fix zipper crash(es) in RGWCopyObj

Instantiate zipper object and bucket handles when called from
rgw_rename().

The current change appears potentially valid, but also suggests
topics for further discussion:

1. redundant object and bucket vars (in req_state and RGWOp)
2. semantic implications std::unique_ptr (e.g., redundant handle
   forks)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38580 from anthonyeleven/anthonyeleven/48301-underscores

doc: standardize config option names with interstitial underscores

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/onode-staged-tree: comment out excessive range checks

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/dev: update auto pr validation section

This commit updates the "Automated PR Validation"
section of the "Basic Workflow" page in the
Developer Guide. This is part of a project that
aims to clean all of the sentences currently extant
in the Dev Guide, prior to a slight reorganization
of that guide for the sake of clarity and simplicity.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

os/bluestore: Fix code rot in ZonedAllocator.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

mgr/dashboard: Display a warning message in Dashboard when debug mode is enabled

Set a health check warning if debug mode is enabled.

Fixes: https://tracker.ceph.com/issues/48475
Signed-off-by: Volker Theile <vtheile@suse.com>

install-deps.sh: add with_jaeger options for installing it's dependencies

  * if WITH_JAEGER flag is specified, install-deps should mangle and update
  debian/control + ceph.spec the way we do for adding crimson dependencies
  with WITH_SEASTAR flag.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

debian/control: update yaml-cpp version if building with Jaeger

jaeger specifically requires yaml-dev +0.6 but since all supported
distro(bionic) still doesn't have this version, yaml-cpp will be build
from source if enabled.
This fixes the build failure:
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)
```

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #38316 from rhcs-dashboard/temporary-lockout

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38534 from votdev/issue_48542_rgw_status

mgr/dashboard: The /rgw/status endpoint does not check for running service

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38536 from dillaman/wip-48553

librbd/mirror: unlink the third snapshot when the limit is reached

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Updating the docs regarding the user lockout

Fixes: https://tracker.ceph.com/issues/40914
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Temporary User Lockout if 10 Invalid Login attempts

Implemented a user lockout mechanism if the user enters 10 invalid attempts. The attempt count gets resetted to 0 once the user succesfully logins before getting disabled. Once the user gets disabled administrator has to manually enable the user which will also resets the number of attempts.

Fixes: https://tracker.ceph.com/issues/40914
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38606 from rhcs-dashboard/log-e2e-fix

mgr/dashboard: Fixing dashboard logs e2e test

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Fixing dashboard logs e2e test

Yesterday there were some changes went in which disables the logging of config set and config-key set to to be logged in mgr audit logs. Dashboard has an e2e test which checks for this config set and broke the current e2e jenkins job. This commit removes that certain test to fix the jenkins job.

Fixes: https://tracker.ceph.com/issues/48623
Signed-off-by: Nizamudeen A <nia@redhat.com>

debian: install crimson-store-nbd with pkg.ceph.crimson profile

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: package crimson-store-nbd in ceph-test package

when building with seastar enabled, package crimson-store-ndb in
"ceph-test" package. "crimson-store-nbd" is a new tool for testing
seastore using nbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38581 from cyx1231st/wip-seastar-msgr-fix-test-preemptive-shutdown

crimson/net: fix test_preemptive_shutdown() failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31358 from ideepika/wip-jaegertracing-in-ceph

jaegertracing build/ops integration

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: standardize config option names to use interstitial underscores

Ceph config option names may use spaces, underscores, or by one reference hyphens
as interstitial separators. Most usage within the doc tree uses underscores,
though example conf files and especially structured lists of options mostly
use spaces. Mostly. Underscores help differentiate the config names from
surrounding text, and moreover facilitate scripting, grep, awk, etc and match
their form in src/common/options.cc.

This PR conforms these occurrences of option names to use interstitial underscores instead of spaces.

Fixes: https://tracker.ceph.com/issues/48301
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

librbd: explicitly disable readahead for writearound cache

The readahead IOs would be wasted effort since the writearound
cache does not service reads.

Fixes: https://tracker.ceph.com/issues/48486
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38479 from neha-ojha/wip-37503-2

Do not add sensitive information in Ceph log files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38514 from lixiaoy1/rbd_types_fix

librbd/cmake: Add missed PWL related types

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38546 from rhcs-dashboard/round-button

mgr/dashboard: Making the edges of not rounded button rounded

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38269 from rhcs-dashboard/fix-44960-master

mgr/dashboard: replace cdCopy2ClipboardButton Directive with a component

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37937 from p-se/cd-alternative-grafana-fe-url

mgr/dashboard: enable different URL for users of browser to Grafana

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

github: add action to manage needs-rebase label

Fixes: https://tracker.ceph.com/issues/48570
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

ceph-conf: add --pid option

this change partially reverts 1f09b196657d21cec8fca9bc868bae307200aae8,
and add "--pid" option, so user can specify the pid to override the $pid
meta variable when expanding options.

Fixes: https://tracker.ceph.com/issues/47977
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.in: extract get_admin_socket() out

will add retry support to this function, so we can use an alternative
name for the admin_socket path if the path is not a valid file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36715 from rhcs-dashboard/fix-39128-master

mgr/dashboard: new generic HTTP error page component

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38456 from SMIL-Infra/fix-crush-map-viewer

mgr/dashboard: Fix CRUSH map viewer VirtualScroll

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: replace cdCopy2ClipboardButton Directive with a component
Replaces Directive-Based copy2ClipboardButton with a Component-based version.
Further replaces all instances of Directive use with Component version.
Similar toastr logic used.
Fixes: https://tracker.ceph.com/issues/44960
Signed-off-by: Courtney Caldwell <ccaldwel@redhat.com>

mon, messages: use TOPNSPC::common for crimson compatibility

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/ActivePyModules.cc: don't log config value in get_typed_config

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon/ConfigMonitor.cc: don't log config values in refresh_config

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr: don't add config values to mgr log in config_callback and module_config

The original code has been commented out and left for future debugging
purposes.

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon/ConfigMonitor.cc: don't dump all keys and values in config map

Signed-off-by: Neha Ojha <nojha@redhat.com>

mon: don't log "config set" and "config-key set" dispatch and finished messages

Signed-off-by: Neha Ojha <nojha@redhat.com>

messages/MMonCommand, MMonCommandAck: don't log values for "config set" and "config-key set"

This acts like a big hammer to avoid adding sensitive information, like passwords
into mon/mgr/cluster logs when using "config set" and "config-key set" to set keys
whose values should be secure.

Fixes: https://tracker.ceph.com/issues/37503
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38010 from cbodley/wip-rgw-doc-reshard-syncstop

doc/rgw: update reshard design regarding 'bucket sync disable'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38585 from tchaikov/wip-doc-dev-errorator

doc/dev/crimson: highlight code blocks using C++ lexer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38582 from tchaikov/wip-crimson-cleanup

crimson/osd: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev/crimson: quote inline codeblock using "``"

backquote is used for quoting italic typed text.

Signed-off-by: Kefu Chai <kchai@redhat.com>