Casey Bodley [Mon, 3 Feb 2020 20:59:58 +0000 (15:59 -0500)]
rgw: use new Stopped state for special handling of 'bucket sync disable'
some sync modules like pubsub can disable full sync, and only run
incremental sync. without full sync, 'bucket sync disable/enable' will
continuously restart incremental sync from the beginning of the bilog
this adds a new Stopped state for 'bucket sync disable', which leaves the
incremental marker position where it is (just past the SYNCSTOP entry).
this means that when sync is reenabled, it can continue sync from where
it left off
Casey Bodley [Wed, 5 Feb 2020 17:50:54 +0000 (12:50 -0500)]
rgw: fix bilog processing of SYNCSTOP entries
when BucketShardIncrementalSync encounters SYNCSTOP, it needs to call
try_update_high_marker() to advance past its marker. for sync modules
that don't support full sync, this allows them to resume incremental
sync from this position in the log when sync is reenabled
Casey Bodley [Wed, 5 Feb 2020 17:47:01 +0000 (12:47 -0500)]
rgw: flush datalog entries on shutdown
pending datalog entries need to be issued before shutdown. a race with
sshutdown is especially easy to reproduce with radosgw-admin, ie for
'bucket sync enable/disable'
When using vg/lv, this function throws an error like following:
```
stderr: unable to read label for test_group/data-lv2: (2) No such file or directory
stderr: 2020-02-04T21:03:32.153+0000 7fe091af4200 -1 bluestore(test_group/data-lv2) _read_bdev_label failed to open test_group/data-lv2: (2) No such file or directory
```
When passing a vg/lv path for generating a single report, it fails
because the filter used in the `lvs` command isn't right. It uses the lv
name instead of the vg name because `os.path.basename(device)` is used
while it should be `os.path.dirname(device)`
Sage Weil [Tue, 4 Feb 2020 03:28:40 +0000 (21:28 -0600)]
Merge PR #33020 into master
* refs/pull/33020/head:
osdc/Objecter: inline pool full check
osdc/Objecter: remove duplicated pause check code
osdc/Objecter: only pause if respects_full()
osdc/Objecter: move respects_full() to op_target_t
Sage Weil [Tue, 4 Feb 2020 03:28:19 +0000 (21:28 -0600)]
Merge PR #32831 into master
* refs/pull/32831/head:
common, include: drop the copy{_in} from bufferlist entirely.
os/bluestore: switch copy_in() users to bufferlist::iterator.
osdc: switch users of bufferlist::copy{_in} to iterators.
osd: switch users of bufferlist::copy{_in} to iterators.
rgw: switch copy{_in} users to bufferlist::iterator.
ec: switch users of bufferlist::copy{_in} to iterators.
cls/queue: switch users of bufferlist::copy{_in} to iterators.
client: switch users of bufferlist::copy{_in} to iterators.
*: switch trivial users of bufferlist::copy{_in} to iterators.
test/bl: switch copy{_in} users to bufferlist::iterator.
common, include: kill the bl::last_p member.
common: encode for std::list<T> doesn't use bl::copy_in() anymore.
Matthew Oliver [Tue, 4 Feb 2020 02:29:48 +0000 (13:29 +1100)]
ceph_argparse: increment matchcnt on kwargs
Currently when you pass a param in on the ceph cli as a kwarg
(--<param_name>) the matchcnt isn't incremented in the validate method
which is used to choose the right command signature.
Yaarit Hatuka [Mon, 3 Feb 2020 19:19:39 +0000 (14:19 -0500)]
mgr/telemetry: check get_metadata return val
get_metada() returns 'None' when requesting a missing service, hence
trying to access its content fails. Added a check for osd and mgr
get_metadata() calls.
cephadm: fix error handling in `command_check_host()`
`find_program()` raises `ValueError` when the executable hasn't been
found. It means we need to catch `ValueError` exception in
`command_check_host()` and raise `Error` instead of `RuntimeError` since
only `Error` is caught at the end.
Typical failure:
```
INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: New host mon1 failed check: ['INFO:cephadm:podman|docker (/bin/podman) is present', 'INFO:cephadm:systemctl is present', 'Traceback (most recent call last):', ' File "<stdin>", line 2820, in <module>', ' File "<stdin>", line 2434, in command_check_host', ' File "<stdin>", line 796, in find_program', 'ValueError: lvcreate not found']
```
This allows for evaluation of more complex use cases where IgnorePublicACLs and
the like are set which need to be evaluated for GET/HEAD requests as well
This API returns whether the Bucket Policies/ACLs are public. There are a couple
of caveats:
- AWS currently returns PolicyNotFound error in case a bucket policy doesn't
exist, though a non existant bucket policy would mean the default ACLs apply
where the bucket is private, so error return here seems like an error
- the API spec mentions TRUE and FALSE as the response IsPublic element value,
however in practice both boto/aws clients and AWS S3 return/expect a lowercase
response.
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
Conflicts:
src/rgw/rgw_rest_s3.h
merge conflict after zipper rework, dropped a spurious newline in rgw_rest_s3.h
after get_obj_op decl.
src/rgw/rgw_common.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h:
merge conflict after bucket replication merge, trivial conflicts
When playing with cephadm, at multiple times, I've reached the max
number of attempt in `is_available()`
Increasing the `retry_max` helps to avoid failure like following:
```
INFO:cephadm:mgr not available, waiting (1/5)...
INFO:cephadm:mgr not available, waiting (2/5)...
INFO:cephadm:mgr not available, waiting (3/5)...
INFO:cephadm:mgr not available, waiting (4/5)...
INFO:cephadm:mgr not available, waiting (5/5)...
ERROR: mgr not available after 5 tries
Sage Weil [Fri, 31 Jan 2020 14:35:26 +0000 (08:35 -0600)]
mgr/cephadm: prefix daemon ids with hostname
This is friendlier to a human operator since they can immediately see
where an instance is located, as with the legacy scheme, while still
keeping the unique random suffix. Use a . to separate so that we can
set per-host options.
* refs/pull/31633/head:
cephfs-shell: Instead of assert use stat for tests in rmdir
cephfs-shell: Add function for common rmdir test code
cephfs-shell: Add rmdir test for non empty directory
cephfs-shell: Add rmdir -p test for non empty directory
cephfs-shell: Add rmdir -p test for non existing dir
cephfs-shell: Add rmdir -p test to delete all dirs in given path
cephfs-shell: Add rmdir -p test for root directory with empty directories
cephfs-shell: Add rmdir test for valid file
cephfs-shell: Add rmdir test for invalid directory
cephfs-shell: Add rmdir test for valid directory
cephfs-shell: Fix rmdir '-p' issues
Reviewed-by: Rishabh Dave <ridave@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Kefu Chai [Fri, 31 Jan 2020 05:02:28 +0000 (13:02 +0800)]
include/buffer: add operator+=() for iterators
less type than p.advance(). please note, this does not imply that
list::iterator is a RandomAccessIterator. as the parameter type of
p.adavance() is not `ptrdiff_t`, which is a signed type.
projects / ceph.git / log