mon: mgr: move 'osd df' handling to manager

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit c416cb4dc47f1365f80a85f2817820ebc05dd743)

mon: move most PGMapStatService into PGMap; rename PGMon's to PGMonStatService

Most of this is independent of the PGMonitor, so move it into
PGMap and strip out those few bits. It'll come in handy shortly
when I move "ceph osd df" into the mgr.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 82a04ca579a314116632dcb38dee3baf3231ad04)

mon: move creating_pgs and reweight_by_utilization into new MonPGStatService

Use that throughout the monitor code, so that we can use the
PGStatService elsewhere.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 9812319616056a92750d3c89151445353f0fb501)

qa: deepsea: drop 2nodes health-mds test

Now that health-mds requires a client node, 2nodes is no longer an option.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa: deepsea: umount only existing OSD data partitions

This fixes the "end" (unwind) phase of the deepsea task so it unmounts OSD data
partitions only if they exist and are block devices (umount on a non-existent
partition can cause bogus test failure).

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #123 from SUSE/wip-deepsea-suite

Deepsea suite

qa: deepsea: add test yaml and tweak task

* adds test yaml in qa/deepsea/basic
* adds exec property
* adds documentation
* after cloning the DeepSea git repo, print the branch name and sha1
  so we have it in the log for verification.
* clone with --depth 1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa: add initial deepsea task

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: one License line to rule them all

The SUSE and Fedora packaging guidelines specify that subpackages _may_ have
their own License line if it is necessary. (Hopefully it's not, because
maintaining one License line is much easier.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit dcab19bc80545b7c623172fe967a4a407a6b65f7)

Conflicts:
ceph.spec.in (recover from bogousness in the previous commit)

build/ops: rpm: one License line to rule them all

The SUSE and Fedora packaging guidelines specify that subpackages _may_ have
their own License line if it is necessary. (Hopefully it's not, because
maintaining one License line is much easier.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 5943c512788e493b495386a95f19e75e757b8ef3)

build/ops: rpm: drop autoconf exception license from ceph-mgr

It has already been dropped from the main License line.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm, COPYING: drop GPL 2.0 with Autoconf Exception

This license is no longer used.

Fixes: http://tracker.ceph.com/issues/20091
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 1d53c28cbbf5672dfe9bc75247268dd7e912a647)

Conflicts:
ceph.spec.in (upstream doesn't use the SPDX 2.0 syntax)

Merge remote-tracking branch 'ceph/master' into wip-20170606

Merge pull request #15499 from jcsp/wip-doc-whitespace

doc: fix syntax on code snippets in cephfs/multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15492 from dillaman/wip-20185

rbd: properly decode features when using image name optional

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15483 from dillaman/wip-20182

librbd: filter expected error codes from is_exclusive_lock_owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15490 from ceph/wip-fix-selinux

selinux: Install ceph-base before ceph-selinux

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15415 from dillaman/wip-rbd-valgrind

librbd: fix valgrind errors and ensure tests detect future leaks

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15140 from dillaman/wip-18963

rbd-mirror: permit release of local image exclusive lock after force promotion

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15484 from Adirl/wip

msg/async/rdma: Add DSCP support

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15493 from yehudasa/wip-rgw-dynamic-sharding

rgw: dynamic resharding

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: add missing members initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14239 from liewegas/wip-bluestore-ec-clone

os/bluestore: try to unshare blobs for EC overwrite workload

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15299 from joscollin/wip-cleanup-redundant-headers-6

common: Remove redundant includes - 6

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15429 from jcsp/wip-nuke-rest

pybind/mgr: Delete `rest` module

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix a race

We referred to complete_op_data after aio call that could have released
it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/cli/radosgw-admin: update reference help.t

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix index completion

Was missing bucket complete op, only had the guard (bad merge?).
Pass zones_trace as pointer and treat it as optional param (which
it is).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: increase max shard prime, consolidate logic

Handle larger num of shards (max ~64k now). Use the same hash logic for shard ids
everywhere.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new num shards proportional to number of objects per shard

and also limit number of shards to not exceed the max.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle zones_trace in bi async completion handler

needed following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove debug helper code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ignore -ENOENT on bi_list()

just treat it as success with empty result

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run resharding thread only on master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add check to see if out is null

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw:  update entry in resharding log when resharding has started

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: start resharding theard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw-admin: radosgw-admin reshard execute -> reshard process

to be in line with other radosgw-admin commands.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket instance with resharding details

can be useful info for later in a cross zone sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: reshard execute output adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: remove unused field in reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard: renew lease and handle marker when listing reshard repo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix reshard_list truncation identification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: expose api to generate reshard indexing key

So that we could use it as a marker when doing reshard_list.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded and duplicate code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize bucket structure

after reading bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: revive reshard execute command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket_id does not include bucket name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bi_list() initialize is_truncated if -ENOENT

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: internal api cleanup

rename method, minor change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: reshard add command doesn't read existing status

No need to read existing status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix remove_reshard objclass op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: modify logshard key

The logshard key shouldn't include bucket id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes and adjustments related to resharding scheduler

still wip, but:
 - get rid of some unneeded rados locking
 - rename shards to logshards where relevant to avoid confusion
 - remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: actually reshard an entry from the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check if buckets needs resharding on all ops that add new objects

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for bucket resharding  when adding an object

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a function to add a bucket to the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a check_bucket_shards to the quota

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: moved reshard_bucket into RGWReshard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove cct from RGWReshard and BucketIndexLockGuard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add resharding thread

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: store new num shards in bucket index reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix deletion of index completion object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard complete handling, use multiple locks

to reduce contention. Also, fix completions cleanup.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean up index completions on shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update data log after index completion reshard handling

Also rearrange code a bit.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard resharding in async completions

Catch return value from async index completions, resend operations
synchronously on a separate worker thread if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard guard for versioned operations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: changes to support reshard guard on olh bi ops

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle reshard wait logic in cancel() too

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix update_bucket_id, reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: add rgw-admin reshard status command and initial blocking logic

A command that returns the reshard status field on the bucket index objects (all
shards).
Also, initial implementation of identifying and blocking while reshard in
progress.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more detailed resharding status kept in bucket header

So that we can differentiate between in-progress and complete status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark source bucket shards when starting reshard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move resharding code to BucketReshard class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: split per-bucket resharding logic out from general resharding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert some reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: create an objclass method to guard in-progress resharding

This method will be called on the bucket index shard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWReshard update to new cls API

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: adjust types and ops

 - don't keep data bufferlist in struct cls_rgw_bucket_instance_entry
 - remove rgw_get_bucket_instance

We need to keep the reshard info as part of the bucket index header (for
each source bucket shard), and not in a separate object. Otherwise it'll
be impossible to have the resharding check atomic as part of the regular
bucket index updates.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: minor changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: generate_reshard_key, change output param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: return default header if not initialized

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cls_reshard_get remove unneeded check for bufferlist length

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: get_bucket_header will return -ENOENT when there is no header

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add guard class for bucket index lock

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add reshard_pool_ctx

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for exisiting entries in cls_rgw_reshard_add

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use pointer for is_truncated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cls_rgw_reshard use helper function the generate key

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: limit op.max and add extra entry to handle tuncation in cls_rgw_reshard_list

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use .rgw.log:reshard pool for resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove RGWReshard::get_head

Signed-off-by: Orit Wasserman <owasserm@redhat.com>