Kefu Chai [Sun, 24 Mar 2019 09:11:31 +0000 (17:11 +0800)]
ceph-volume: terminal: encode unicode when writing to stdout
python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.
in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.
for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .
Jan Fajerski [Tue, 6 Aug 2019 13:34:12 +0000 (15:34 +0200)]
mimic: ceph-volume broken assertion errors after pytest changes
This fixes a backported test that was not fixed on a backport. Original
fix: https://github.com/ceph/ceph/pull/28948. Backport:
https://github.com/ceph/ceph/pull/29243.
Jan Fajerski [Tue, 30 Jul 2019 15:32:06 +0000 (17:32 +0200)]
ceph-volume: fall back to PARTTYPE if PARTLABEL is empty
In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.
Fixes: https://tracker.ceph.com/issues/40917 Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 82d2ae7ffdf62ca93c7848fb62e506db4cda893c)
Jan Fajerski [Tue, 30 Jul 2019 15:29:09 +0000 (17:29 +0200)]
ceph-volume: adjust tests for empty PARTLABEL fields
It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.
Jan Fajerski [Tue, 30 Jul 2019 11:29:22 +0000 (13:29 +0200)]
ceph-volume: refactor ceph-disk_member unittests
This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.
Tianshan Qu [Sun, 11 Nov 2018 11:56:51 +0000 (19:56 +0800)]
rgw: set null version object acl issues
1.set null version object acl will create empty index
RGWRados::set_attrs did not clear instance, so index prepare, complete got instance=null,
which lead to empty index 1000_<obj>_i_null.
there is no harm to create empty index, but listomapkeys to find that key.
2.if object is exist with versioned key, we can set none exists null version object
order:
1) enable bucket version
2) put obj
3) disable bucket version
4) set versoned_id=null acl will succeed which should not
Matt Benjamin [Wed, 5 Jun 2019 17:25:32 +0000 (13:25 -0400)]
rgw/OutputDataSocket: actually discard data on full buffer
A dout message in OutputDataSocket::append_output() states that
data will be dropped when appending would cause data_max_backlog
to be exceeded--but the method appends it anyway.
Log output discards at level 0, as messages will be lost. Suppress
repeated warnings mod 100. Switch to vector.
Fixes: http://tracker.ceph.com/issues/40178 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c806b825dae649829de8847d36cb21ffd2bbee8e)
Matt Benjamin [Wed, 5 Jun 2019 11:25:35 +0000 (07:25 -0400)]
rgw_file: include tenant in hashes of object
Because bucket names are taken as object names in the top
of an export. Make hashing by tenant general to avoid disjoint
hashing of bucket.
Fixes: http://tracker.ceph.com/issues/40118 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 8e0fd5fbfa7c770f6b668e79b772179946027bce)
Matt Benjamin [Wed, 15 May 2019 19:53:32 +0000 (15:53 -0400)]
rgw_file: include tenant when hashing bucket names
Prevent identical paths from distinct tenants from colliding in
RGW NFS handle cache.
Fixes: http://tracker.ceph.com/issues/40118 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit b800a9de83dff23a150ed7d236cb61c8b7d971ae)
Casey Bodley [Fri, 24 May 2019 20:59:34 +0000 (16:59 -0400)]
radosgw-admin: fix syncs_from in 'bucket sync status'
If a zone is set up with sync_from_all=false, the bucket sync status
command will incorrectly report "not in sync_from" because it's calling
'zone.syncs_from(source.id)', where syncs_from() expects a zone name
instead of id
Casey Bodley [Mon, 6 May 2019 19:01:07 +0000 (15:01 -0400)]
rgw: delete_obj_index() takes mtime for bilog
writing an empty timestamp to the bilog prevents other zones from
applying the delete. this means that the --bypass-gc flag for
'radosgw-admin bucket rm' doesn't work in multisite
rgw: RGWPeriodPusher uses zone system key for inter-zonegroup messages
RGWPeriodPusher was using an empty RGWAccessKey for inter-zonegroup
messages, which were rejected as an anonymous user with 403 Forbidden.
this prevented multi-zonegroup configurations from converging on the
same period configuration
Sage Weil [Thu, 2 May 2019 16:39:31 +0000 (11:39 -0500)]
os/bluestore: be verbose about objects that existing on rmcoll
This is always a bug (OSD doesn't try to remove a collection unless it
thinks it is empty), and not seeing it at default debug levels makes it
hard to track down.
Sage Weil [Thu, 18 Jul 2019 16:09:47 +0000 (11:09 -0500)]
osd/PG: fix cleanup of pgmeta-like objects on PG deletion
If an object has an empty 'name' field, it "looks" like a pgmeta object,
and the PG cleanup code was skipping it. However, we were letting these
objects get created.
Fix by only skipping *our* pgmeta object. If there are other pgmeta-like
objects in the PG collection, clean them up.
liyichao [Wed, 5 Dec 2018 04:22:20 +0000 (12:22 +0800)]
tools: Add clear-data-digest command to objectstore tool.
There may be a situation where data digest in object info is
inconsistent with that computed from object data, then deep-scrub
will fail even though all three repicas have the same object data.
Fixes: https://tracker.ceph.com/issues/37935 Signed-off-by: Li Yichao <liyichao.good@gmail.com>
(cherry picked from commit da5832b2b49be4ab23f7ffc5ba1ba630168b886f)
Conflicts:
src/tools/ceph_objectstore_tool.cc
- change cbegin() to begin() to get it to build
Patrick Donnelly [Wed, 23 Jan 2019 02:26:38 +0000 (18:26 -0800)]
mds: remove cache drop asok command
`cache drop` is a long running command that will block the asok interface
(while the tell version does not). Attempting to abort the command with ^C or
equivalents will simply cause the `ceph` command to exit but won't stop the
asok command handler from waiting for the cache drop operation to complete.
Instead, just allow the tell version.
Fixes: http://tracker.ceph.com/issues/38020 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7fa1e3c37f8c7fb709ae9e070a3154a7084c0584)
client: fix fuse client hang because its bad session PipeConnection
to mds.
There still has a risk client will hang all the time and which
happened in my enviroment a few days ago. Well konw, the start
of mds will go through four stages at least(replay, reconnect,
rejoin, active). According to current logic, we recived
replay mdsmap and marked down pipe will return directly and if
we miss the next reconnect mdsmap the client may hang all the
time becuase we lost the opportunity due to old_inc equal to
new_inc. So we shuold judge if we have lost MDSMap::reconnect
by comparing newstate and oldstate.
Conflicts:
src/client/Client.cc
- mimic has "kick_maxsize_requests(session); wake_inode_waiters(session);"
where master has just "wake_up_session_caps(session, true)
Casey Bodley [Fri, 4 Jan 2019 20:53:34 +0000 (15:53 -0500)]
rgw: fetch_remote_obj filters out olh attrs
each zone manages its own olh log, so we don't want any of the source
zone's olh attributes. in one case (sync of an object overwrite in a
versioning-suspended bucket), these olh attributes were being written
directly to the head object and leading to link_olh errors due to the
olh_tag mismatch
Nathan Cutler [Wed, 5 Jun 2019 15:30:30 +0000 (17:30 +0200)]
build/ops: make "patch" build dependency explicit
This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.
The "patch" command is used by the BuildBoost cmake module.
rgw: fix race b/w bucket reshard and ops waiting on reshard completion
A previous commit (f84f70d4) added functionality to clean up old
bucket instances and bucket shards once sharding completed
successfully. However the existing code that allowed a bucket
operation to wait until resharding completed was relying on a field in
the old bucket instance to know the bucket instance id of the new
bucket instance. This created a race condition as to whether the
clean-up or read of the bucket instance id would occur first.
This solution rereads the bucket entry point object when resharding
completes to determine the bucket instance id of the new bucket
thereby avoiding the race.
projects / ceph.git / log