]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
6 years agoMerge pull request #29197 from smithfarm/wip-38963-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:10:43 +0000 (09:10 -0700)]
Merge pull request #29197 from smithfarm/wip-38963-mimic

mimic: mgr: DaemonServer::handle_conf_change - broken locking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
6 years agoMerge pull request #29201 from smithfarm/wip-40268-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:09:58 +0000 (09:09 -0700)]
Merge pull request #29201 from smithfarm/wip-40268-mimic

mimic: common: data race in OutputDataSocket

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29217 from smithfarm/wip-39692-mimic
Yuri Weinstein [Tue, 30 Jul 2019 16:09:25 +0000 (09:09 -0700)]
Merge pull request #29217 from smithfarm/wip-39692-mimic

mimic: bluestore: _txc_add_transaction error (39) Directory not empty not handled on operation 21 (op 1, counting from 0)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
6 years agoMerge pull request #29243 from jan--f/mimic-c-v-extend-batch
Jan Fajerski [Mon, 29 Jul 2019 15:57:41 +0000 (17:57 +0200)]
Merge pull request #29243 from jan--f/mimic-c-v-extend-batch

mimic: c-v extend batch

6 years agoceph-volume: create two NVME drives for testing 29243/head
Andrew Schoen [Tue, 27 Nov 2018 23:34:10 +0000 (17:34 -0600)]
ceph-volume: create two NVME drives for testing

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
Resolves: bz#1650306
(cherry picked from commit 893b61b256bcff142dbd90b777b2c9df9e032e9d)

6 years agoceph-volume: set block_db_size to 9GB in centos7 mixed type test
Andrew Schoen [Wed, 28 Nov 2018 16:10:24 +0000 (10:10 -0600)]
ceph-volume: set block_db_size to 9GB in centos7 mixed type test

This is to enable testing of the --block-db-size flag

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit eab962c6ef68c588674bdac6fc4a86c84714f300)

6 years agoceph-volume: use two NVME drives in centos7 mixed_type tests
Andrew Schoen [Wed, 28 Nov 2018 16:05:45 +0000 (10:05 -0600)]
ceph-volume: use two NVME drives in centos7 mixed_type tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 934c32acb318aac9aff3c800cdbf99bbdee04d06)

6 years agoMerge pull request #29350 from jan--f/wip-40977-mimic
Alfredo Deza [Fri, 26 Jul 2019 18:33:21 +0000 (14:33 -0400)]
Merge pull request #29350 from jan--f/wip-40977-mimic

mimic: ceph-volume: missing string substitution when reporting mounts

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoceph-volume:util: Use proper param substition 29350/head
Shyukri Shyukriev [Fri, 7 Jun 2019 12:09:19 +0000 (12:09 +0000)]
ceph-volume:util: Use proper param substition

Fixes: https://tracker.ceph.com/issues/25030
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
(cherry picked from commit 654660cd89fe9d93087d390805651c860bf4e862)

6 years agoMerge pull request #26991 from ceph/backport-mimic-26957
Alfredo Deza [Thu, 25 Jul 2019 15:54:37 +0000 (11:54 -0400)]
Merge pull request #26991 from ceph/backport-mimic-26957

mimic: ceph-volume: look for rotational data in lsblk

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoceph-volume: look for rotational data in lsblk 26991/head
Andrew Schoen [Thu, 14 Mar 2019 20:14:40 +0000 (15:14 -0500)]
ceph-volume: look for rotational data in lsblk

Also ask lsblk if a device is rotational if no information
is found in /sys/block, default to True if nothing is found.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit de748f9aef894b247622ee5dc49c1fbc31097ad0)

6 years agoMerge pull request #28918 from smithfarm/wip-40222-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:37:25 +0000 (07:37 -0700)]
Merge pull request #28918 from smithfarm/wip-40222-mimic

mimic: mds: reset heartbeat during long-running loops in recovery

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
6 years agoMerge pull request #29187 from xiaoxichen/wip-40875-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:36:55 +0000 (07:36 -0700)]
Merge pull request #29187 from xiaoxichen/wip-40875-mimic

mimic: cephfs: client: bump ll_ref from int32 to uint64_t

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #29200 from smithfarm/wip-39685-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:36:25 +0000 (07:36 -0700)]
Merge pull request #29200 from smithfarm/wip-39685-mimic

mimic: cephfs: ceph-fuse: client hang because its bad session PipeConnection to mds

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
6 years agoMerge pull request #29208 from smithfarm/wip-40317-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:35:41 +0000 (07:35 -0700)]
Merge pull request #29208 from smithfarm/wip-40317-mimic

mimic: tests: test_volume_client: print python version correctly

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #29210 from smithfarm/wip-38099-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:35:08 +0000 (07:35 -0700)]
Merge pull request #29210 from smithfarm/wip-38099-mimic

mimic: mds: remove cache drop admin socket command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #29211 from smithfarm/wip-38687-mimic
Yuri Weinstein [Thu, 25 Jul 2019 14:34:17 +0000 (07:34 -0700)]
Merge pull request #29211 from smithfarm/wip-38687-mimic

mimic: kcephfs TestClientLimits.test_client_pin fails with client caps fell below min

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
6 years agoMerge pull request #27028 from mrandybu/mimic-alt
Yuri Weinstein [Wed, 24 Jul 2019 23:06:01 +0000 (16:06 -0700)]
Merge pull request #27028 from mrandybu/mimic-alt

mimic: ceph_detect_init: Add support for ALT Linux

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28210 from votdev/fix_rgw_issue
Yuri Weinstein [Wed, 24 Jul 2019 23:04:23 +0000 (16:04 -0700)]
Merge pull request #28210 from votdev/fix_rgw_issue

mimic: mgr/dashboard: Fix various RGW issues

Reviewed-by:  Tiago Melo <tmelo@suse.com>

6 years agoMerge pull request #28911 from bk201/wip-40660-mimic
Yuri Weinstein [Wed, 24 Jul 2019 23:03:24 +0000 (16:03 -0700)]
Merge pull request #28911 from bk201/wip-40660-mimic

mimic: mgr/dashboard: cephfs multimds graphs stack together

Reviewed-by: Ricardo Dias <rdias@suse.com>
6 years agoMerge pull request #29024 from ifed01/wip-ifed-fix-len0-mimic
Yuri Weinstein [Wed, 24 Jul 2019 23:02:19 +0000 (16:02 -0700)]
Merge pull request #29024 from ifed01/wip-ifed-fix-len0-mimic

mimic: os/bluestore: avoid length overflow in extents returned by Stupid Allocator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
6 years agoMerge pull request #29174 from neha-ojha/wip-40583-mimic
Yuri Weinstein [Wed, 24 Jul 2019 23:00:55 +0000 (16:00 -0700)]
Merge pull request #29174 from neha-ojha/wip-40583-mimic

mimic: common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #29189 from badone/wip-mimic-lazy-omap-stats-backport-tracker...
Yuri Weinstein [Wed, 24 Jul 2019 22:59:50 +0000 (15:59 -0700)]
Merge pull request #29189 from badone/wip-mimic-lazy-omap-stats-backport-tracker-38552

mimic: core: lazy omap stat collection

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #29196 from smithfarm/wip-37749-mimic
Yuri Weinstein [Wed, 24 Jul 2019 22:58:23 +0000 (15:58 -0700)]
Merge pull request #29196 from smithfarm/wip-37749-mimic

mimic: tools: Add clear-data-digest command to objectstore tool

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agoMerge pull request #29016 from ivancich/mimic-abort-multipart-fix
Yuri Weinstein [Wed, 24 Jul 2019 15:26:33 +0000 (08:26 -0700)]
Merge pull request #29016 from ivancich/mimic-abort-multipart-fix

mimic: rgw: abort multipart fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29021 from smithfarm/wip-39074-mimic
Yuri Weinstein [Wed, 24 Jul 2019 15:26:07 +0000 (08:26 -0700)]
Merge pull request #29021 from smithfarm/wip-39074-mimic

mimic: rgw: multisite: data sync loops back to the start of the datalog after reaching the end

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29139 from ivancich/mimic-fix-resharding-race
Yuri Weinstein [Wed, 24 Jul 2019 15:25:38 +0000 (08:25 -0700)]
Merge pull request #29139 from ivancich/mimic-fix-resharding-race

mimic: rgw: fix race b/w bucket reshard and ops waiting on reshard completion

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #29017 from smithfarm/wip-38080-mimic
Yuri Weinstein [Wed, 24 Jul 2019 15:22:59 +0000 (08:22 -0700)]
Merge pull request #29017 from smithfarm/wip-38080-mimic

mimic: multisite: overwrites in versioning-suspended buckets fail to sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoceph-volume: add explicit mixed-type functional tests
Jan Fajerski [Wed, 16 Jan 2019 12:48:25 +0000 (13:48 +0100)]
ceph-volume: add explicit mixed-type functional tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bea35c63c83fc2ab87fc983774ed8181cbe74e5b)

6 years agoceph-volume: add tests for Device.lvm_size
Andrew Schoen [Thu, 6 Dec 2018 18:45:10 +0000 (12:45 -0600)]
ceph-volume: add tests for Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 01185fb32479f271beeb14dcf94e32b1401ad2d9)

6 years agoceph-volume: adds a lvm_size property to util.Device
Andrew Schoen [Wed, 5 Dec 2018 15:43:28 +0000 (09:43 -0600)]
ceph-volume: adds a lvm_size property to util.Device

This will give us the size lvm should report because it takes into
account the 1GB physical extent size we set.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5197c16b44fcf344862201549777dc2935fc19a3)

6 years agoceph-volume: functional tests, add libvirt customization
Jan Fajerski [Thu, 20 Dec 2018 12:53:41 +0000 (13:53 +0100)]
ceph-volume: functional tests, add libvirt customization

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c330a4758755e3815b30e4f00a4a00e8319db399)

6 years agoos/bluestore: be verbose about objects that existing on rmcoll 29217/head
Sage Weil [Thu, 2 May 2019 16:39:31 +0000 (11:39 -0500)]
os/bluestore: be verbose about objects that existing on rmcoll

This is always a bug (OSD doesn't try to remove a collection unless it
thinks it is empty), and not seeing it at default debug levels makes it
hard to track down.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e387cbc45037db78db5dfbb01483b8d76e1c276f)

Conflicts:
src/os/bluestore/BlueStore.cc
- mimic does not have 87e8231d3fa61bc95ea0cda4d731e30f3d4de9d5

6 years agoosd/PrimaryLogPG: disallow ops on objects with an empty name
Sage Weil [Thu, 2 May 2019 16:30:09 +0000 (11:30 -0500)]
osd/PrimaryLogPG: disallow ops on objects with an empty name

These may conflict with pgmeta objects.  And are going to cause other
problems later (e.g., https://tracker.ceph.com/issues/38724).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bf6e8a4a4db8b7d7101ca80b844ee953aa4580b2)

6 years agoosd/PG: fix cleanup of pgmeta-like objects on PG deletion
Sage Weil [Thu, 18 Jul 2019 16:09:47 +0000 (11:09 -0500)]
osd/PG: fix cleanup of pgmeta-like objects on PG deletion

If an object has an empty 'name' field, it "looks" like a pgmeta object,
and the PG cleanup code was skipping it.  However, we were letting these
objects get created.

Fix by only skipping *our* pgmeta object.  If there are other pgmeta-like
objects in the PG collection, clean them up.

Fixes: https://tracker.ceph.com/issues/38724
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a0f48db5cbfc48761eeac0daa259fecbaa573969)

# Conflicts:
# src/osd/PG.cc
- location of this code is do_delete_work in master, _delete_some in n

6 years agocommon: OutputDataSocket retakes mutex on error path 29201/head
Casey Bodley [Thu, 6 Jun 2019 13:48:42 +0000 (09:48 -0400)]
common: OutputDataSocket retakes mutex on error path

the list of buffers is protected by mutex m_lock. when dump_data() fails
and puts unwritten buffers back into the list, it needs to reaquire the
lock

Fixes: http://tracker.ceph.com/issues/40188
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8645e24dd4fb3ecc9f02b34a1be916620b5f71b1)

Conflicts:
    src/common/OutputDataSocket.cc
- use Mutex::Locker instead of std::scoped_lock

6 years agomgr/DaemonServer: handle_conf_change - fix broken locking 29197/head
xie xingguo [Tue, 26 Mar 2019 08:08:43 +0000 (16:08 +0800)]
mgr/DaemonServer: handle_conf_change - fix broken locking

The is_locked_by_me() is only meant to be safe if you know
you hold the lock. It's meant to be used in an assertion.

Fixes: http://tracker.ceph.com/issues/38899
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a929a5a01508e661fc7f3cea52822320d39983d2)

Conflicts:
    src/mgr/DaemonServer.cc
- use Mutex::Locker instead of std::lock_guard

6 years agotools: Add clear-data-digest command to objectstore tool. 29196/head
liyichao [Wed, 5 Dec 2018 04:22:20 +0000 (12:22 +0800)]
tools: Add clear-data-digest command to objectstore tool.

There may be a situation where data digest in object info is
inconsistent with that computed from object data, then deep-scrub
will fail even though all three repicas have the same object data.

Fixes: https://tracker.ceph.com/issues/37935
Signed-off-by: Li Yichao <liyichao.good@gmail.com>
(cherry picked from commit da5832b2b49be4ab23f7ffc5ba1ba630168b886f)

Conflicts:
    src/tools/ceph_objectstore_tool.cc
- change cbegin() to begin() to get it to build

6 years agoqa/cephfs: relax min_caps_per_client check 29211/head
Yan, Zheng [Thu, 7 Mar 2019 13:12:28 +0000 (21:12 +0800)]
qa/cephfs: relax min_caps_per_client check

new kernel client proactively release caps. caps count can go below
mds_min_caps_per_client

Fixes: http://tracker.ceph.com/issues/38270
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 8e81bd74c59259a06f587bb425d58bf2b2176341)

Conflicts:
qa/tasks/cephfs/test_client_limits.py
- mimic TestClientLimits does not have expected_caps method at all

6 years agomds: remove cache drop asok command 29210/head
Patrick Donnelly [Wed, 23 Jan 2019 02:26:38 +0000 (18:26 -0800)]
mds: remove cache drop asok command

`cache drop` is a long running command that will block the asok interface
(while the tell version does not). Attempting to abort the command with ^C or
equivalents will simply cause the `ceph` command to exit but won't stop the
asok command handler from waiting for the cache drop operation to complete.

Instead, just allow the tell version.

Fixes: http://tracker.ceph.com/issues/38020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7fa1e3c37f8c7fb709ae9e070a3154a7084c0584)

Conflicts:
PendingReleaseNotes
qa/tasks/cephfs/test_misc.py
src/mds/MDSDaemon.cc

6 years agoqa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs 29208/head
Lianne [Tue, 11 Jun 2019 10:27:06 +0000 (18:27 +0800)]
qa/tasks/cephfs/test_volume_client: print py2 or py3 which the test case runs

Fixes: http://tracker.ceph.com/issues/40184
Signed-off-by: Lianne <liyan.wang@xtaotech.com>
(cherry picked from commit 7c7c7870d38902a0df83a0fdecaa56baad556d82)

6 years agoclient: fix fuse client hang because its bad session PipeConnection 29200/head
IvanGuan [Wed, 17 Apr 2019 02:21:54 +0000 (10:21 +0800)]
client: fix fuse client hang because its bad session PipeConnection
to mds.

There still has a risk client will hang all the time and which
happened in my enviroment a few days ago. Well konw, the start
of mds will go through four stages at least(replay, reconnect,
rejoin, active). According to current logic, we recived
replay mdsmap and marked down pipe will return directly and if
we miss the next reconnect mdsmap the client may hang all the
time becuase we lost the opportunity due to old_inc equal to
new_inc. So we shuold judge if we have lost MDSMap::reconnect
by comparing newstate and oldstate.

Fixes: http://tracker.ceph.com/issues/39305
Signed-off-by: Guan yunfei <yunfei.guan@xtaotech.com>
(cherry picked from commit 51c9f5cf9af61fe317259ac79a503ef8a7139622)

Conflicts:
src/client/Client.cc
- mimic has "kick_maxsize_requests(session); wake_inode_waiters(session);"
  where master has just "wake_up_session_caps(session, true)

6 years agomimic: osd: Modify lazy omap stats to only show one copy 29189/head
Brad Hubbard [Tue, 9 Jul 2019 05:03:24 +0000 (15:03 +1000)]
mimic: osd: Modify lazy omap stats to only show one copy

Only show stats from the primary to get more in line with other pg stats
representations.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 490829c8eaf14f41de978ac0f93ecda3df8bcdef)

6 years agomimic: tests: Add test for lazy omap stat collection
Brad Hubbard [Mon, 4 Mar 2019 08:12:27 +0000 (18:12 +1000)]
mimic: tests: Add test for lazy omap stat collection

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 88e9ca58a06ba572ff4c6f6125a8ba7f8e0a7795)

6 years agomimic: osd: Implement lazy omap usage statistics
Brad Hubbard [Tue, 11 Dec 2018 04:00:35 +0000 (14:00 +1000)]
mimic: osd: Implement lazy omap usage statistics

Opportunistic gathering of omap statistics during deep scrub.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 9e21ef0e40ff89b10dfdf857c703336496b3171f)

 Conflicts:
doc/dev/placement-group.rst: Trivial
src/osd/osd_types.h: Trivial

6 years agoClient: bump ll_ref from int32 to uint64_t 29187/head
Xiaoxi CHEN [Fri, 19 Jul 2019 15:31:24 +0000 (08:31 -0700)]
Client: bump ll_ref from int32 to uint64_t

ll_ref can surge up if application repeatly lookup same dentry
and finally cause overflow.

https://tracker.ceph.com/issues/40775 is an example.

in kernel fuse driver and libfuse, uint64_t is used, update
ceph-fuse side to match.

Fixes: https://tracker.ceph.com/issues/40775
Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>
(cherry picked from commit bb46de885336a8cd2adc9c0507e543c6832346e7)

 Conflicts:
src/client/Client.h
        use new code, skip all non-toched changes.
src/client/Inode.h
        use new code.

6 years agocommon/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_th... 29174/head
Neha Ojha [Fri, 28 Jun 2019 16:31:35 +0000 (09:31 -0700)]
common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Warning when there are already 2M omap keys does not help, reduce this
threshold so that preemptive measures can be taken based on this.

Fixes: https://tracker.ceph.com/issues/40583
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 1ec948409ab547effbd5c656af8f0c9565f1717a)

6 years agorgw: add RGWRados::repair_olh() to reconstruct olh attrs 29017/head
Casey Bodley [Fri, 25 Jan 2019 21:44:03 +0000 (16:44 -0500)]
rgw: add RGWRados::repair_olh() to reconstruct olh attrs

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1309e2d50ee8118afe1138cd6677d84438edb55a)

Conflicts:
    src/rgw/rgw_rados.cc
- no rgw_bl_str() in mimic
- mimic does not have a4d3977cfacc5eb0de16618f2c957bb417505597

6 years agorgw: add RGWRados::bi_get_olh() to read/decode olh entry
Casey Bodley [Fri, 25 Jan 2019 21:43:36 +0000 (16:43 -0500)]
rgw: add RGWRados::bi_get_olh() to read/decode olh entry

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b8d7f0ad1f882b999e0fd7f3e45f8579599714cf)

Conflicts:
    src/rgw/rgw_rados.cc
- bufferlist::cbegin()

6 years agorgw: bi_get() takes bucket_info
Casey Bodley [Fri, 25 Jan 2019 21:42:50 +0000 (16:42 -0500)]
rgw: bi_get() takes bucket_info

and calls the bucket_info overload of BucketShard::init() to avoid
reading the bucket info again

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a15e8736340c916120fef1f3a5ff1b982cf7c837)

6 years agorgw: add BucketShard::init overload for bucket_info,obj
Casey Bodley [Fri, 25 Jan 2019 21:17:19 +0000 (16:17 -0500)]
rgw: add BucketShard::init overload for bucket_info,obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c6c445f6074df113a28adcae89248b909d031f8e)

6 years agotest/rgw: add multisite test_version_suspended_incremental_sync
Casey Bodley [Wed, 2 Jan 2019 16:24:45 +0000 (11:24 -0500)]
test/rgw: add multisite test_version_suspended_incremental_sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 81fa354c1dadf4f957592f20fe9e02acd70671e0)

Conflicts:
src/test/rgw/rgw_multi/tests.py

6 years agorgw: fetch_remote_obj filters out olh attrs
Casey Bodley [Fri, 4 Jan 2019 20:53:34 +0000 (15:53 -0500)]
rgw: fetch_remote_obj filters out olh attrs

each zone manages its own olh log, so we don't want any of the source
zone's olh attributes. in one case (sync of an object overwrite in a
versioning-suspended bucket), these olh attributes were being written
directly to the head object and leading to link_olh errors due to the
olh_tag mismatch

Fixes: http://tracker.ceph.com/issues/37792
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a4195fc55fb76f0a2576d5ff0b2c99ad12fd7a49)

6 years agoMerge pull request #29020 from smithfarm/wip-39047-mimic
Nathan Cutler [Mon, 22 Jul 2019 09:19:00 +0000 (11:19 +0200)]
Merge pull request #29020 from smithfarm/wip-39047-mimic

mimic: rgw: update resharding documentation

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
6 years agoMerge pull request #29014 from smithfarm/wip-37891-mimic
Nathan Cutler [Mon, 22 Jul 2019 08:53:54 +0000 (10:53 +0200)]
Merge pull request #29014 from smithfarm/wip-37891-mimic

mimic: doc: wrong value of usage log default in logging section

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agorgw: fix race b/w bucket reshard and ops waiting on reshard completion 29139/head
J. Eric Ivancich [Mon, 1 Apr 2019 14:59:51 +0000 (10:59 -0400)]
rgw: fix race b/w bucket reshard and ops waiting on reshard completion

A previous commit (f84f70d4) added functionality to clean up old
bucket instances and bucket shards once sharding completed
successfully. However the existing code that allowed a bucket
operation to wait until resharding completed was relying on a field in
the old bucket instance to know the bucket instance id of the new
bucket instance. This created a race condition as to whether the
clean-up or read of the bucket instance id would occur first.

This solution rereads the bucket entry point object when resharding
completes to determine the bucket instance id of the new bucket
thereby avoiding the race.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 26b793680805721cac86f64cdd234b273bf33be4)

Conflicts:
        src/rgw/rgw_rados.cc: equivalent changes made to rgw_reshard.cc

6 years agorgw: correct comment describing function try_refresh_bucket_info
J. Eric Ivancich [Wed, 27 Mar 2019 20:32:17 +0000 (16:32 -0400)]
rgw: correct comment describing function try_refresh_bucket_info

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit d0894e6f6c062907796d10b524088c2abf7e4252)

6 years agorgw: fix log messages using "proceeding" to "processing"
J. Eric Ivancich [Tue, 12 Mar 2019 19:38:54 +0000 (15:38 -0400)]
rgw: fix log messages using "proceeding" to "processing"

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 1c2aa902cb2bec690387ace8eb6b8afb5678373f)

6 years agomgr/dashboard: Fix various RGW issues 28210/head
Volker Theile [Wed, 22 May 2019 09:30:50 +0000 (11:30 +0200)]
mgr/dashboard: Fix various RGW issues

* Fix handling of tenanted users
* Better exception handling if RGW backend is not available
* RGW user/bucket "max. size" should be hidden when "user/bucket quota" is not enabled
* Fix bug in loading user quota.

Fixes: https://tracker.ceph.com/issues/38800
partial manual backport of 61995970916cb11f66c3800bcf869755f8dae32c
partial manual backport of e6f130d47023c24f8a9c742f145abd34d7320cd8
partial manual backport of 945e790cab57785e1bc2dbf708d5ddafec618aba
partial manual backport of a98bca6a2f161140f78f34cfa5f5ebec07af2f82
partial manual backport of 975736aba546adb09ce4f38a1ae061621f7240cd

This fix is not cherry-picked because the code was completely re-worked for Nautilus (codebase is too far apart).

Signed-off-by: Volker Theile <vtheile@suse.com>
6 years agoceph-volume: add space between words
Sébastien Han [Fri, 1 Feb 2019 17:45:45 +0000 (18:45 +0100)]
ceph-volume: add space between words

The current help of batch was showing:

rotational drivesare passed in DEVICES

This commit fixes it and now we have:

rotational drives are passed in DEVICES

Tremendous fix, I know.

Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit a151e111a2710e864bd66a44db38685459cf55c8)

6 years agoceph-volume: use the Device.rotational property instead of sys_api
Andrew Schoen [Fri, 10 May 2019 14:02:08 +0000 (09:02 -0500)]
ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8ebff475dfa136c88101ef7364820fe3c5cee547)
(cherry picked from commit aa6da416cb14ba521796c950fba9667e7b581b27)

6 years agoceph-volume: set a lvm_size property on the fakedevice fixture
Andrew Schoen [Thu, 6 Dec 2018 18:37:56 +0000 (12:37 -0600)]
ceph-volume: set a lvm_size property on the fakedevice fixture

This is so tests can continue to set sys_api['size'] and the code
can retrieve that as Device.lvm_size

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9e2175c4001434f7b0ed8ce6f723e063cf001440)

6 years agoceph-volume: don't crash on unusable data devices
Jan Fajerski [Sat, 26 Jan 2019 10:08:01 +0000 (11:08 +0100)]
ceph-volume: don't crash on unusable data devices

Also add test case for a MixedStrategy with unusable data devices.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_filestore.py
        Pick test addition and error-value change

6 years agoceph-volume: add a wal device specific unit test and size validator
Jan Fajerski [Thu, 24 Jan 2019 17:10:45 +0000 (18:10 +0100)]
ceph-volume: add a wal device specific unit test and size validator

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_bluestore.py
        Pick test addition and error.value change

6 years agoceph-volume: add osd_ids argument
Jan Fajerski [Wed, 23 Jan 2019 14:04:29 +0000 (15:04 +0100)]
ceph-volume: add osd_ids argument

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
6 years agoceph-volume: strategies allow for external wal lv's
Jan Fajerski [Thu, 17 Jan 2019 14:13:05 +0000 (15:13 +0100)]
ceph-volume: strategies allow for external wal lv's

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
6 years agoceph-volume: add default argument values to strategy
Jan Fajerski [Mon, 7 Jan 2019 13:43:12 +0000 (14:43 +0100)]
ceph-volume: add default argument values to strategy

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 5465712e4d71315ff8daac9345be6ba96d1f44c4)

 Conflicts:
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_bluestore.py
src/ceph-volume/ceph_volume/tests/devices/lvm/strategies/test_filestore.py
        pick with_auto_devices and error.value changes

6 years agoAdd a unit test that tests the ceph_volume.devices.lvm.batch.Batch::get_devices method.
Volker Theile [Thu, 29 Nov 2018 12:04:14 +0000 (13:04 +0100)]
Add a unit test that tests the ceph_volume.devices.lvm.batch.Batch::get_devices method.

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 19a38189fd15f57907449631a0f6e9faf908dc66)

6 years agoceph-volume strategies: rename db_devs to db_or_journal_devs
Jan Fajerski [Fri, 4 Jan 2019 10:30:30 +0000 (11:30 +0100)]
ceph-volume strategies: rename db_devs to db_or_journal_devs

This aims at reducing confusion when reading the code that is shared
between the bluestore and filestore strategies.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 029414e89b773896fddebe0a5869b57ec13f527e)

6 years agoceph-volume batch: rename block_devs to data_devs for consistency
Jan Fajerski [Thu, 13 Dec 2018 14:42:49 +0000 (15:42 +0100)]
ceph-volume batch: rename block_devs to data_devs for consistency

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 6dbf43e53ccd3ff06e3cdfc8e6de97475cc0c510)

6 years agoceph-volume batch: refactor filtered_devices and strategy retrieval
Jan Fajerski [Wed, 28 Nov 2018 11:20:20 +0000 (12:20 +0100)]
ceph-volume batch: refactor filtered_devices and strategy retrieval

Keep filtered_devices as a member variable of the batch object and pass
this to the report functionality instead of having the report functions
retrieve filtered_devices. Also keep selected strategy as a batch member
variable to avoid re-retrival in e.g. the report methods.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit dc2c4c597249d61ea6f48d37dbe02d8b97e3d55d)

6 years agoceph-volume: refactor strategies for explicit device arguments
Jan Fajerski [Tue, 27 Nov 2018 14:49:01 +0000 (15:49 +0100)]
ceph-volume: refactor strategies for explicit device arguments

This commit refactors lvm strategies to:
a) change the constructor to receive usage specific device lists
b) add classmethod with_auto_devices of automatic splitting by the
rotational flag (keep backwards compatible behavior)
c) rename ssds and hdds member variables to wal_devs and block_devs
d) add db_devs member variable

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 60dec8b1dfdd056662ba6a030de291f93b2f3650)

6 years agoceph-volume: move arg parsing to __init__ and add explicit device lists
Jan Fajerski [Fri, 21 Dec 2018 10:01:27 +0000 (11:01 +0100)]
ceph-volume: move arg parsing to __init__ and add explicit device lists

Moving the arg parsing to __init__ allows to get the help message with
running as root.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit a7717818ab3790b226067da3dee8f8c216ab8f4b)

6 years agoceph-volume: use Device.lvm_size in batch strategy validators
Andrew Schoen [Thu, 6 Dec 2018 16:49:03 +0000 (10:49 -0600)]
ceph-volume: use Device.lvm_size in batch strategy validators

This will validate the devices on the size that lvm will allow to be
used, not the raw physical size of the device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aaa915cfe6678d5326b2c15730febfff79e40463)

6 years agoceph-volume: batch mixed type scenarios have no need to calulate data size
Andrew Schoen [Tue, 4 Dec 2018 20:23:47 +0000 (14:23 -0600)]
ceph-volume: batch mixed type scenarios have no need to calulate data size

We know with a mixed type scenario the device used for data will be used
at 100% capacity. This means we do not need to be explict when asking
for the size of the data lvs, which avoids rounding errors with very
small device sizes.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0023961397d19659200eaf420f2789db17a71b34)

6 years agoceph-volume: set use_large_block_db in validate, not compute
Andrew Schoen [Fri, 30 Nov 2018 17:55:27 +0000 (11:55 -0600)]
ceph-volume: set use_large_block_db in validate, not compute

The self.use_large_block_db property was never getting set because
the block in compute was never called as block_db_size was reset in
validate if it was 0. We needed to set self.use_large_block_db in
validate instead of compute.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit fdfb79b91c52f82877d08190cb626ed00b45f3e8)

6 years agoceph-volume: use Device.lvm_size in batch strategies
Andrew Schoen [Thu, 6 Dec 2018 15:56:58 +0000 (09:56 -0600)]
ceph-volume: use Device.lvm_size in batch strategies

We should show the user what the size of the device will be after lvm
creates a pv out of it. This way there isn't a discrepency between the
sizes that are reported to the user and what is actually created.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 071e7ce41a58702c0eb84c35bd9388e2ec24ac13)

6 years agoceph-volume: use extents instead of size when creating block.db lvs
Andrew Schoen [Tue, 27 Nov 2018 23:39:38 +0000 (17:39 -0600)]
ceph-volume: use extents instead of size when creating block.db lvs

In the batch command use extents instead of size when creating lvs. This
gives a more precise size and avoids rounding errors.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit add9f88dc800913adf387c90cbf4352e392b0a75)

 Conflicts:
src/ceph-volume/ceph_volume/devices/lvm/strategies/bluestore.py
        remove json import; pick ours and add self.use_large_block_db = False

(cherry picked from commit 83e71842d8cd77b7d45efdd344bd8fac31078745)

6 years agoMerge pull request #28206 from pdvian/wip-39475-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:52:11 +0000 (12:52 -0700)]
Merge pull request #28206 from pdvian/wip-39475-mimic

mimic: common/util: handle long lines in /proc/cpuinfo

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28217 from smithfarm/wip-40005-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:51:39 +0000 (12:51 -0700)]
Merge pull request #28217 from smithfarm/wip-40005-mimic

mimic: build/ops: do_cmake.sh: source not found

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
6 years agoMerge pull request #28741 from smithfarm/wip-40547-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:50:33 +0000 (12:50 -0700)]
Merge pull request #28741 from smithfarm/wip-40547-mimic

mimic: common: Keyrings created by ceph auth get are not suitable for ceph auth import

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28894 from ifed01/wip-ifed-fix-no-compress-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:49:29 +0000 (12:49 -0700)]
Merge pull request #28894 from ifed01/wip-ifed-fix-no-compress-mimic

mimic: os/bluestore: load OSD all compression settings unconditionally.

Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28967 from liewegas/wip-set-aio-write-max-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:49:00 +0000 (12:49 -0700)]
Merge pull request #28967 from liewegas/wip-set-aio-write-max-mimic

mimic: os/bluestore: fix >2GB bluefs writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28970 from neha-ojha/wip-40720-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:47:58 +0000 (12:47 -0700)]
Merge pull request #28970 from neha-ojha/wip-40720-mimic

mimic: os/bluestore: default to bitmap allocator for bluestore/bluefs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #28995 from smithfarm/wip-38751-mimic
Yuri Weinstein [Thu, 18 Jul 2019 19:47:14 +0000 (12:47 -0700)]
Merge pull request #28995 from smithfarm/wip-38751-mimic

mimic: core: should report EINVAL in ErasureCode::parse() if m<=0

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28220 from pdvian/wip-39513-mimic
Yuri Weinstein [Wed, 17 Jul 2019 23:01:04 +0000 (16:01 -0700)]
Merge pull request #28220 from pdvian/wip-39513-mimic

mimic: osd: take heartbeat_lock when calling heartbeat()

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agoMerge pull request #26499 from k0ste/mimic_backports2
Yuri Weinstein [Wed, 17 Jul 2019 22:19:26 +0000 (15:19 -0700)]
Merge pull request #26499 from k0ste/mimic_backports2

mimic: mgr/balancer: restrict automatic balancing to specific weekdays

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #27986 from pdvian/wip-39311-mimic
Yuri Weinstein [Wed, 17 Jul 2019 22:18:24 +0000 (15:18 -0700)]
Merge pull request #27986 from pdvian/wip-39311-mimic

mimic: crushtool crash on Fedora 28 and newer

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28089 from dzafman/wip-39720
Yuri Weinstein [Wed, 17 Jul 2019 22:17:30 +0000 (15:17 -0700)]
Merge pull request #28089 from dzafman/wip-39720

mimic: osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28097 from pdvian/wip-39374-mimic
Yuri Weinstein [Wed, 17 Jul 2019 22:17:00 +0000 (15:17 -0700)]
Merge pull request #28097 from pdvian/wip-39374-mimic

mimic: osd/OSD.cc: make osd bench description consistent with parameters

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #28142 from pdvian/wip-39422-mimic
Yuri Weinstein [Wed, 17 Jul 2019 22:14:53 +0000 (15:14 -0700)]
Merge pull request #28142 from pdvian/wip-39422-mimic

mimic: osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
6 years agoMerge pull request #28165 from pdvian/wip-39458-mimic
Yuri Weinstein [Wed, 17 Jul 2019 22:14:06 +0000 (15:14 -0700)]
Merge pull request #28165 from pdvian/wip-39458-mimic

mimic: mgr/prometheus: replace whitespaces in metrics' names

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
6 years agoMerge pull request #28201 from dzafman/wip-38341
Yuri Weinstein [Wed, 17 Jul 2019 22:13:21 +0000 (15:13 -0700)]
Merge pull request #28201 from dzafman/wip-38341

mimic: osd: pg stuck in backfill_wait with plenty of disk space

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #29074 from ceph/backport-mimic-26397
Alfredo Deza [Wed, 17 Jul 2019 20:44:53 +0000 (16:44 -0400)]
Merge pull request #29074 from ceph/backport-mimic-26397

mimic: ceph-volume: update testing playbook 'deploy.yml'

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoMerge pull request #27949 from smithfarm/wip-37613-mimic
Yuri Weinstein [Wed, 17 Jul 2019 19:46:02 +0000 (12:46 -0700)]
Merge pull request #27949 from smithfarm/wip-37613-mimic

mimic: rpm: missing dependency on python34-ceph-argparse from python34-cephfs (and others?)

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #28090 from pdvian/wip-39233-mimic
Yuri Weinstein [Wed, 17 Jul 2019 19:45:14 +0000 (12:45 -0700)]
Merge pull request #28090 from pdvian/wip-39233-mimic

mimic: cephfs: mount.ceph.c: do not pass nofail to the kernel

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge pull request #28186 from pdvian/wip-39472-mimic
Yuri Weinstein [Wed, 17 Jul 2019 19:44:45 +0000 (12:44 -0700)]
Merge pull request #28186 from pdvian/wip-39472-mimic

mimic: mds: fix SnapRealm::resolve_snapname for long name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
6 years agoMerge pull request #28274 from pdvian/wip-39669-mimic
Yuri Weinstein [Wed, 17 Jul 2019 19:44:23 +0000 (12:44 -0700)]
Merge pull request #28274 from pdvian/wip-39669-mimic

mimic: mds: output lock state in format dump

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge pull request #28337 from pdvian/wip-39679-mimic
Yuri Weinstein [Wed, 17 Jul 2019 19:43:59 +0000 (12:43 -0700)]
Merge pull request #28337 from pdvian/wip-39679-mimic

mimic: cephfs: pybind: added lseek()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
6 years agoMerge pull request #28381 from pdvian/wip-39689-mimic
Yuri Weinstein [Wed, 17 Jul 2019 19:43:35 +0000 (12:43 -0700)]
Merge pull request #28381 from pdvian/wip-39689-mimic

mimic: mds: check dir fragment to split dir if mkdir makes it oversized.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>