Merge pull request #16227 from tchaikov/wip-kill-clang-analyzer-warnings

mgr,os,test: kill clang analyzer warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #16239 from xiexingguo/wip-crush-weight-overflow

mon: a few more upmap (and other) fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16249 from gregsfortytwo/wip-20562-cluster-stats

mon: maintain the "cluster" PerfCounters when using ceph-mgr

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #15989 from linuxbox2/wip-nfs-compress

rgw_file: add compression interop to RGW NFS

rgw_file: add large(ish) write test

allows for read-after-write verify

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  compression fix and cleanup from review

A fix and a cleanup from Adam Kupczyk <akupczyk@redhat.com> review.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  enable compression filter (stores)

Previously NFS clients could access compressed data, but the
RGWWriteDataRequest lacked glue to attach an inbound
compression filter.

Fixes: http://tracker.ceph.com/issues/20462
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #16241 from liewegas/wip-dashboard-default

doc/mgr/dashboard: update dashboard docs to reflect new defaults

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16034 from guihecheng/rgw_file-policy-read

rgw: rgw_file skip policy read for virtual components

Merge pull request #16229 from Liuchang0812/wip-misc-cleanup

ceph.in, mgr: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16230 from jecluis/wip-20443

mon/Monitor: recreate mon session if features changed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16233 from liewegas/wip-osd-create

qa/tasks/ceph.py: no osd id to 'osd create' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16235 from tchaikov/wip-timer

os/filestore: do not free event if not added

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16237 from YankunLi/dev-bluestore0710

os/blueStore: Failure retry for opening file

Reviewed-by: Jos Collin <jcollin@redhat.com>

mon: maintain the "cluster" PerfCounters when using ceph-mgr instead of PGMonitor

Fixes: http://tracker.ceph.com/issues/20562
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16246 from gregsfortytwo/wip-fix-mgr-cephx-link

doc: fix link for ceph-mgr cephx authorization

mon/OSDMonitor: dedup pg-upmap[-items]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: fix potential divided-by-zero error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: allow user to permanently delete pool

If mon_fake_pool_delete is enabled, "osd pool delete"
will unable to delete pool permanently, as we ask user
to pass in a "--yes-i-really-really-mean-it-no-faking"
string now.

The old "osd pool delete" command won't allow user to do this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: sanity check osd number before applying pg-temp

In case user did something stupid and causing weird results.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: fix link for ceph-mgr cephx authorization

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16177 from Synesis-LLC/fix_rgw_crash_ec_data_pool

rgw: fix not initialized vars which cause rgw crash with ec data pool

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15467 from joke-lee/fix-aws-v2-signature-X-Amz-Date-priority

rgw：fix s3 aws v2 signature priority between header['X-Amz-Date'] and header['Date']

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16033 from zhaochao/rgw-add-a-new-errcode-for-non-existed-user

rgw: add a new error code for non-existed user.

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16185 from smithfarm/wip-lease-stack-spawn

rgw: lease_stack: use reset method instead of assignment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16204 from jimifm/wip-rgw_frontend

rgw/rgw_frontend.h: Return negative value for empty uid in RGWLoadGenFrontend::init()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16016 from liewegas/wip-osd-df

mon: include device class in tree view; hide shadow hierarchy

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph.py: no osd id to 'osd create' command

This isn't recognized by hammer, and we don't need it for jewel.

Fixes: http://tracker.ceph.com/issues/20548
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15858 from liewegas/wip-mgr-servicemap

mgr,librados: service map

Reviewed-by: John Spray <john.spray@redhat.com>

doc/mgr/dashboard: update dashboard docs to reflect new defaults

Also add not about load balancer.

Signed-off-by: Sage Weil <sage@redhat.com>

test/mon/osd-crush-tree.rng: fix expected xml schema for osd tree

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool/tree.t: fix

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16049 from fullerdj/wip-djf-key-timeout

ceph-create-keys: add an argument to override default 10-minute timeout

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16236 from shashalu/clean-up-RGWRemoteDataLog-unused-function

rgw: drop unused function RGWRemoteDataLog::get_shard_info()

Reviewed-by: Jos Collin <jcollin@redhat.com>

BlueStore:Failure retry for opening file

Failure retry to open file

Signed-off-by: Yankun Li <lioveni99@gmail.com>

rgw:multisite: clean up RGWRemoteDataLog unused function

'RGWRemoteDataLog::get_shard_info' function not used, drop it

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

os/filestore: do not free event not added

if an event fails to be added, it's freed by the callee. so we should
not try to free it again. that address could be re-used by the
allocator, to we might be freeing an irrelevent event.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: return false if event not added

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: guard "osd crush rule create-replicated"

require Luminous before using the crush device classes.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: apply both pg_upmap and pg_upmap_items

This allows a more flexible upmap policy by combinating
both pg_upmap and pg_upmap_items.

For example:
./bin/ceph osd pg-upmap 1.0 1 2 3
set 1.0 pg_upmap mapping to [1,2,3]

./bin/ceph osd pg-upmap-items 1.0 1 5
set 1.0 pg_upmap_items mapping to [1->5]

 ./bin/ceph pg ls-by-pool cephfs_data_a
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP      UP_PRIMARY ACTING  ACTING_PRIMARY
1.0           0                  0        0         0       0     0   0        0 active+clean 2017-07-10 11:04:53.955681     0'0    22:13 [5,2,3]          5 [5,2,3]              5

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: improve "osd upmap" error message

by adding a explicit guide.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: fix potential weight overflow

E.g.:
./bin/ceph osd crush reweight osd.0 32768

ID WEIGHT      TYPE NAME                                            UP/DOWN REWEIGHT PRIMARY-AFFINITY
-4 32770.00000 root default~hdd
-3           -     host gitbuilder-ceph-rpm-centos7-amd64-basic~hdd
 0           -         osd.0                                             up  1.00000          1.00000
 1     1.00000         osd.1                                             up  1.00000          1.00000
 2     1.00000         osd.2                                             up  1.00000          1.00000
-1 32770.00000 root default
-2           -     host gitbuilder-ceph-rpm-centos7-amd64-basic
 0           -         osd.0                                             up  1.00000          1.00000
 1     1.00000         osd.1                                             up  1.00000          1.00000
 2     1.00000         osd.2                                             up  1.00000          1.00000

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16160 from xiexingguo/wip-misc2

mon, osd: misc fixes and cleanups

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16220 from dzafman/wip-20544

mon: Fix status output warning for mon_warn_osd_usage_min_max_delta

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16223 from liewegas/wip-dashboard-default

pybind/mgr/dashboard: bind to :: by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: register as a service

support dynamic reload, and also add frontend info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon: make service summary string customizable; simple default

Eventually the mgr can populate this field with something tailored to the
service.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: set hostname propery on ServiceMap services

Signed-off-by: John Spray <john.spray@redhat.com>

librados: include common daemon metadata in service_daemon_register

Aside from being generally useful, this uniformity enables ceph-mgr
to have some common information about all the hosts
it knows about; otherwise we would sometimes learn
about a host without e.g. knowing about the cpu/ram/kernel.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose daemon status to modules

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyModules: expose service_map to modules

Signed-off-by: Sage Weil <sage@redhat.com>

librados: allow service registrations

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: remove useless headers in MgrStandby.h

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

tool/ceph: print timeout tips iff verbose

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mon/Monitor: recreate mon session if features changed

Fixes: http://tracker.ceph.com/issues/20433
Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #16079 from smithfarm/wip-doc-make-check

doc: dev: add notes on PR make check validation test

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/Monitor: list sessions via admin socket

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/Paxos: fix annoying formatting spacing

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/Session: fix some annoying spacing

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon: shorten 'PRIMARY-AFFINITY' -> 'PRI-AFF' column header

Signed-off-by: Sage Weil <sage@redhat.com>

mon: include device class in 'osd tree' output

Signed-off-by: Sage Weil <sage@redhat.com>

mon: include class in 'osd df [tree]' output

Signed-off-by: Sage Weil <sage@redhat.com>

crush: hide device class shadow roots from tree dumper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16027 from liewegas/wip-crush-rule-class

mon,crush: create crush rules using device classes for replicated and ec pools via cli

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16228 from smithfarm/wip-rados-upgrade-2

tests: fix rados/upgrade/jewel-x-singleton and make workunit task handle repo URLs not ending in ".git"

Reviewed-by: Sage Weil <sage@redhat.com>

mgr: implement 'service dump' and 'service status'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: allow/track service registrations

Signed-off-by: Sage Weil <sage@redhat.com>

mon: include services in 'ceph -s'

Signed-off-by: Sage Weil <sage@redhat.com>

mon: persist ServiceMap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add ServiceMap

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: give rgw daemons letter names

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: index daemon state by string service type

If we use a string we can allow for other service names like
'rgw' and 'rbd-mirror' and 'iscsigw' and so on.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15973 from majianpeng/bluefs-async-compact-log

os/bluestore/BlueFS: clear current log entrys before dump all fnode,

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16214 from liupan1111/wip-fix-mutex

common/Mutex.cc: fixed the error in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa: workunit: handle repo ending in "/ceph-ci"

If the repo ends in "/ceph-ci" it's the same as if it ended in "/ceph-ci.git"

Before this change, the following command was broken if the workunit specified,
e.g., "branch: jewel":

teuthology-suite --ceph-repo https://github.com/ceph/ceph --ceph master
--suite-repo https://github.com/ceph/ceph-ci --suite-branch wip-foo . . .

Fixes: http://tracker.ceph.com/issues/20554
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/suites/rados/upgrade: upgrade client.0 node too

Fixes: http://tracker.ceph.com/issues/20368
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16017 from liewegas/wip-20454

os/bluestore/BlueFS: clean up log_writer aios from compaction

Merge pull request #16025 from liewegas/wip-19964

mon: do crushtool test with fork and timeout, but w/o exec of crushtool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16121 from shashalu/fix-kerneldevice-size

os/bluestore: clear up redundant size assignment in KerenelDevice

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16205 from javacruft/build-ec-isa-as-module

build: build erasure-code isa lib without versions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16212 from dillaman/wip-qa-ceph-task

qa/tasks/ceph: don't hard-code cluster name when copying fsid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16213 from liewegas/wip-radosbench-timeout

qa/tasks/radosbench: longer timeout

test/ceph_test_librgw_file_nfsns: always free allocated memory

use unique_ptr to manage allocated memory, so we can free it even if the
test fails. this silences the warnings like:

Potential leak of memory pointed to by 'buf2'

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/libradosstriper: always free allocated memory

use unique_ptr to manage allocated memory, so we can free it even if the
test fails. this silences the warnings like:

Potential leak of memory pointed to by 'buf2'

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign to never-read variable

this silences the clang analyzer warnings like:

Value stored to 'p' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/PyState: fix leak if `type` is unknown

this silences a clang analyzer warning:

Potential leak of memory pointed to by 'c'

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Mutex.cc: fixed the error in comment

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

pybind/mgr/dashboard: bind to :: by default

This is all available v4 and v6 addrs on the host.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16198 from tchaikov/wip-doc-fixes

doc: kill sphinx warnings

test/cli/crushtool: fix rules.t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16225 from liewegas/wip-mon-auth-caps

qa/suites/rados/singleton/all/mon-auth-caps: more osds so we can go clean

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16216 from ceph/wip-mgr-cd-updates

[qa/ceph-deploy]:  run create mgr nodes as well

Merge pull request #15689 from jdurgin/wip-filestore-rand-split

os/filestore/HashIndex: randomize split threshold by a configurable amount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16078 from xiexingguo/wip-misc

mon, osd: misc fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16218 from javacruft/ubuntu-fix-systemd-enable

build: execute dh_systemd_{enable,start} after dh_install

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15944 from xiexingguo/wip-bluestore-20170627

os/bluestore: reduce some overhead for _do_clone_range() and _do_remove()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16052 from dzafman/wip-20471

osd: Check for and automatically repair object info soid during scrub

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-create-keys: add an argument to override default 10-minute timeout

ceph-create-keys waits 10 minutes for mon quorum by default. Add an
option, -t, to override the timeout with a custom value in seconds.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #16147 from Yan-waller/wip-waller-0703radostool

tools/rados: some cleanups

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #16217 from liewegas/wip-class-luminous

mon/OSDMonitor: guard 'osd crush set-device-class'