qa/suites/rados/cephadm[-smoke]: test podman on ubuntu 18.04

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32946 into master

* refs/pull/32946/head:
qa/suites/rados: improve valgrind leak check
common/ceph_context: add an asok command to deliberately leak memory

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33055 into master

* refs/pull/33055/head:
qa/tasks/mgr/test_orchestrator_cli: support multiple DriveGroups

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33056 into master

* refs/pull/33056/head:
common: fix clang compile errors from cython_modules

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33058 into master

* refs/pull/33058/head:
mgr/cephadm: enforce that a host is a valid DNS name
mgr/cephadm: verify host's hostname matches our host name
cephadm: check-host: add optional --expect-hostname

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #33069 into master

* refs/pull/33069/head:
cephadm: use appropriate default image for non-ceph components

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33039 into master

* refs/pull/33039/head:
osd/OSD: prevent down osds from immediately rejoining the culster
osd/OSD: trim osd_markdown_log in tick() thread

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: enforce that a host is a valid DNS name

This combines the hostname restrictions

 * 1-63 chars
 * a-z, A-Z, 0-9, -

and the DNS name restrictions

 * .-delimited
 * no empty components (or leading or trailing .)
 * 250 chars total max

Note that this allows bare IPv4 addresses (which are indistinguishable from
a valid DNS name, AFAICS), but disallows bare IPv6 addresses.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33085 from rzarzynski/wip-client-bl_iter_advance

client: fix FTBFS due to bl::iterator::advance().

Reviewed-by: Kefu Chai <kchai@redhat.com>

client: fix FTBFS due to bl::iterator::advance().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33026 from liewegas/wip-el81

qa/distros: add rhel/centos 8.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32996 from cbodley/wip-rgw-put-multipart-stripe

rgw: MultipartObjectProcessor supports stripe size > chunk size

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32811 from multi-arch/master

test/rgw: fix test_rgw_reshard_wait with -DHAVE_BOOST_CONTEXT=OFF

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33051 into master

* refs/pull/33051/head:
mgr/telemetry: check get_metadata return val

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33018 from mgfritch/cephadm-docker-disabled

qa/workunits/cephadm/test_cephadm.sh: skip docker when service is disabled

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32982 from krig/cephadm-fixes

cephadm: Read ceph version from io.ceph.version label if set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33025 from neha-ojha/wip-no-mgr

mon/MgrMonitor.cc: warn about missing mgr in a cluster with osds

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33077 from guits/guits-cv_fix_listing

ceph-volume: fix lvm list

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33004 from matthewoliver/argparge_matchcnt_kwargs

ceph_argparse: increment matchcnt on kwargs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33074 from guits/guits-cv_quick_fix

ceph-volume: fix has_bluestore_label() function

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33015 from rouming/double-unlock-p1-fix

msg/async: open() should be called with connection locked

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #33029 from yaarith/wip-telemetry-show-device

mgr/telemetry: anonymizing smartctl report itself

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33003 from tchaikov/wip-buffer-list-advance

include/buffer: add operator+=() for list::iterator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33034 from rzarzynski/wip-include-ceph_atomic_in_mempool

include: switch mempool.h to ceph::atomic.

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: remove stderr in has_bluestore_label()

We don't want to generate this log when a call to
`has_bluestore_label()` fails.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix has_bluestore_label() function

When using vg/lv, this function throws an error like following:

```
 stderr: unable to read label for test_group/data-lv2: (2) No such file or directory
 stderr: 2020-02-04T21:03:32.153+0000 7fe091af4200 -1 bluestore(test_group/data-lv2) _read_bdev_label failed to open test_group/data-lv2: (2) No such file or directory
```

using `self.abspath` fixes this error.

Fixes: https://tracker.ceph.com/issues/43970
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix lvm list

17957d9beb42a04b8f180ccb7ba07d43179a41d3 introduced a regression in `lvm
list`.

When passing a vg/lv path for generating a single report, it fails
because the filter used in the `lvs` command isn't right. It uses the lv
name instead of the vg name because `os.path.basename(device)` is used
while it should be `os.path.dirname(device)`

Fixes: https://tracker.ceph.com/issues/43969
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #33076 from liu-chunmei/fix_crimson_pg_coll

crimson: fix crimson pg coll usage error

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33057 from wjwithagen/wjw-fix-buffer.copy

rbd-ggate: fix fallout from bufferlist.copy() change

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix crimson pg coll usage error

coll in pg isn't initialized, should use coll_ref->get_cid() insead.

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

cephadm: use appropriate default image for non-ceph components

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33066 from gregsfortytwo/wip-specfile

rpm: fix up a specfile syntax error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30033 from theanalyst/rgw/public-buckets

rgw: add PublicAccessBlock set of APIs on buckets

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rpm: fix up a specfile syntax error

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33049 from yuvalif/wip-yuval-fix-43768

qa/rgw/pubsub: fix tests to sync from master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: verify host's hostname matches our host name

This ensures that we have a consistent view between the hosts that cephadm
is using and the hostname being reported as metadata by the ceph daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: check-host: add optional --expect-hostname

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add Pending Release Notes entry on public access config

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #32119 from joke-lee/rgw-sts-crash-duration-invalid

rgw: fix rgw crash when duration is invalid in sts request

Merge pull request #31987 from linuxbox2/rgw-putacls-no-body

rgw: s3: don't require a body in S3 put-object-acl

Merge pull request #30684 from theanalyst/rgw/qa/rgw-admin-user-stats

qa: radosgw_admin: validate a simple user stats output

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge PR #33048 into master

* refs/pull/33048/head:
cephadm: fix error handling in `command_check_host()`
cephadm: increase default retry_max value

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32995 into master

* refs/pull/32995/head:
cephadm: add group 'disk' to privileged container
cephadm: adopt: disable ceph-volume unit on host
cephadm: refactor privileged arg handling
cephadm: chown block symlink targets
cephadm-adoption-corpus: add stud-mon.tgz
cephadm: adopt: rename leveldb *.ldb -> *.sst
cephadm: adopt: by default, pull first
cephadm: adopt: chown data content

Reviewed-by: Michael Fritch <mfritch@suse.com>

rbd-ggate: fix fallout from bufferlist.copy() change

fixes: #3281
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common: fix clang compile errors from cython_modules

Signed-off-by: Mark Kogan <mkogan@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: support multiple DriveGroups

create_osds interface in Orchestrator supports multiple named DriveGroups
since https://github.com/ceph/ceph/pull/32972. Adapt the changes in
the test.

Fixes: https://tracker.ceph.com/issues/43945
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33020 into master

* refs/pull/33020/head:
osdc/Objecter: inline pool full check
osdc/Objecter: remove duplicated pause check code
osdc/Objecter: only pause if respects_full()
osdc/Objecter: move respects_full() to op_target_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32831 into master

* refs/pull/32831/head:
common, include: drop the copy{_in} from bufferlist entirely.
os/bluestore: switch copy_in() users to bufferlist::iterator.
osdc: switch users of bufferlist::copy{_in} to iterators.
osd: switch users of bufferlist::copy{_in} to iterators.
rgw: switch copy{_in} users to bufferlist::iterator.
ec: switch users of bufferlist::copy{_in} to iterators.
cls/queue: switch users of bufferlist::copy{_in} to iterators.
client: switch users of bufferlist::copy{_in} to iterators.
*: switch trivial users of bufferlist::copy{_in} to iterators.
test/bl: switch copy{_in} users to bufferlist::iterator.
common, include: kill the bl::last_p member.
common: encode for std::list<T> doesn't use bl::copy_in() anymore.

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_argparse: increment matchcnt on kwargs

Currently when you pass a param in on the ceph cli as a kwarg
(--<param_name>) the matchcnt isn't incremented in the validate method
which is used to choose the right command signature.

This means currently things fail like:

  ceph orchestrator rgw rm --realm_name=default --zone_name=default
  no valid command found; 1 closest matches:
  orchestrator rgw rm <realm_name> <zone_name>
  Error EINVAL: invalid command

The '--realm_name' and '--zone_name' isn't counted to the matchcnt, so
'orchestrator rgw rm' isn't picked as the valid command.

This patch simply corrects this by incrementing matchcnt on the kwarg
validate path before calling shortcircuiting the loop.

Fixes: https://tracker.ceph.com/issues/43803
Signed-off-by: Matthew Oliver <moliver@suse.com>

mgr/telemetry: check get_metadata return val

get_metada() returns 'None' when requesting a missing service, hence
trying to access its content fails. Added a check for osd and mgr
get_metadata() calls.

Fixes: https://tracker.ceph.com/issues/43642
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>

Merge pull request #32531 from zdover23/wip-doc-landing-page-update

doc: Added the crisp getting started guide to index.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: add group 'disk' to privileged container

This lets the osd read block devs that are group rw disk even after they
drop root privs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32977 into master

* refs/pull/32977/head:
qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests
cephadm: simplify Monitoring.components structure
cephadm: add proper tox type for monitoring components

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33012 into master

* refs/pull/33012/head:
mgr/cephadm: prefix daemon ids with hostname
cephadm: bootstrap: name mgr with $hostname.$random

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: adopt: disable ceph-volume unit on host

This might be a simple or lvm unit.  Disable it so that the host doesn't
try to start this OSD after a reboot.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: refactor privileged arg handling

Pass a bool if we want a privileged container instead of explicitly
passing --privileged.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix error handling in `command_check_host()`

`find_program()` raises `ValueError` when the executable hasn't been
found. It means we need to catch `ValueError` exception in
`command_check_host()` and raise `Error` instead of `RuntimeError` since
only `Error` is caught at the end.

Typical failure:

```
INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: New host mon1 failed check: ['INFO:cephadm:podman|docker (/bin/podman) is present', 'INFO:cephadm:systemctl is present', 'Traceback (most recent call last):', '  File "<stdin>", line 2820, in <module>', '  File "<stdin>", line 2434, in command_check_host', '  File "<stdin>", line 796, in find_program', 'ValueError: lvcreate not found']
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

rgw: move public access conf to perm_state_base

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

test: rgw_iam_policy update tests with new Get Actions

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: iam_policy: add all the actions to actpairs map

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: public access: drop unused function

Also cleanup the comment to mention why we've deviated from the spec here

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: rename ACL & policy IsPublic to is_public

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: PublicAccessConfiguration -> PublicAccessBlockConfiguration

also drop iam namespace

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_op: get_public_access_from_attr indent fixes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_rest_s3: use formatter->dump_bool

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

public_access: reuse formatter->dump_bool

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: Fix IgnorePublicACLs for bucket ACLs

Currently Bucket ACLs with IgnorePublicACLs were broken this should fix that

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

drop redundant bucket policy status in rgw_common

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: implement IgnorePublicACLs

This allows for ignoring bucket/object acls that are configured to be public

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: move PublicAccessConfiguration to req_state

This allows for evaluation of more complex use cases where IgnorePublicACLs and
the like are set which need to be evaluated for GET/HEAD requests as well

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: op: Get Policy Status checks if there is a policy first

before evaluating its public nature

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_op: reuse function to get public access conf

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: honor PublicAccessBlockConfiguration for put object

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: block public access for Policies

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: enforce BlockPublicPolicy on put bucket policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: public_access: rename the getters to reflect what the conf actually does

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw_op: use ldpp_dout version wherever logs are required

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: move IsPublic to RGWAccessControlPolicy class

This helps reusing when evaluating for PutACLs and Put Policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: implement get/put/delete public access block for buckets

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add RGWPublicAccess for configuring public access settings for a resource

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: both princ and nonprinc will not coexist in a single statement

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: initial implementation of a public policy tester

doesn't cover all the cases involving a nonprinc user yet

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: s3: implement GetBucketPolicyStatus API

This API returns whether the Bucket Policies/ACLs are public. There are a couple
of caveats:
- AWS currently returns PolicyNotFound error in case a bucket policy doesn't
exist, though a non existant bucket policy would mean the default ACLs apply
where the bucket is private, so error return here seems like an error
- the API spec mentions TRUE and FALSE as the response IsPublic element value,
however in practice both boto/aws clients and AWS S3 return/expect a lowercase
response.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Conflicts:
src/rgw/rgw_rest_s3.h
merge conflict after zipper rework, dropped a spurious newline in rgw_rest_s3.h
after get_obj_op decl.
src/rgw/rgw_common.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h:
merge conflict after bucket replication merge, trivial conflicts

rgw: acl: drop unused function & make get_group_perm const

Drop the unused RGWAccessControlPolicy::get_group_perm, make the ACL
get_group_perm as a const member function

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/suites/rados: improve valgrind leak check

Verify we can detect leak in the osd, mon, and mgr independently.  Also
include a negative test (no leaks).

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: add an asok command to deliberately leak memory

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: increase default retry_max value

When playing with cephadm, at multiple times, I've reached the max
number of attempt in `is_available()`

Increasing the `retry_max` helps to avoid failure like following:

```
INFO:cephadm:mgr not available, waiting (1/5)...
INFO:cephadm:mgr not available, waiting (2/5)...
INFO:cephadm:mgr not available, waiting (3/5)...
INFO:cephadm:mgr not available, waiting (4/5)...
INFO:cephadm:mgr not available, waiting (5/5)...
ERROR: mgr not available after 5 tries

```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

qa/rgw: test with non-default rgw-obj-stripe-size

each job will select one of the striping strategies at random

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32155 from yuvalif/wip-yuval-multidelete-notifications

rgw/pubsub: send notifications from multi-delete op

mgr/cephadm: prefix daemon ids with hostname

This is friendlier to a human operator since they can immediately see
where an instance is located, as with the legacy scheme, while still
keeping the unique random suffix.  Use a . to separate so that we can
set per-host options.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSD: prevent down osds from immediately rejoining the culster

In 114c65fc I posted a work-around to fix a heartbeat brain-split case
but it really looks to me now like I am missing some other cases where
an immediate attempt to rejoin is bad, like when the network actually
isn't working properly rather than being predictably manipulated by an
admin.
This patch instead slows the unconditionally rejoining attempt down,
especially make sure that we don't try to immediately rejoin the culster
when an osd has just been marked down by mon.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #32859 from liewegas/wip-cephadm-services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33030 from smithfarm/wip-luminous-12.2.13-release-notes

doc: release notes for Luminous v12.2.13

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33031 from smithfarm/wip-nautilus-14.2.7-release-notes

doc: release notes for Nautilus 14.2.7

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

osd/OSD: trim osd_markdown_log in tick() thread

so we don't have to do it in multiple places. Note that
we can't do it in the tick_without_osd_lock thread instead
because we we can not access it safely without the protection
of osd_lock.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #33038 from tchaikov/wip-librados-reinterpret-cast

librados: prefer reinterpret_cast over c-style cast

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #32933 from DimStar77/obs-mini

ceph.spec.in: Use pkgconfig() style BuildRequires for udev/libudev-devel

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32553 from tspmelo/wip-ts-strict

mgr/dashboard: Enable compiler options used by Angular --strict flag

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

librados: prefer reinterpret_cast over c-style cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33016 from tchaikov/wip-do-cmake

do_cmake.sh: bail out if something goes wrong

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>