mon/OSDMonitor: dedup pg-upmap[-items]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: fix potential divided-by-zero error

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: allow user to permanently delete pool

If mon_fake_pool_delete is enabled, "osd pool delete"
will unable to delete pool permanently, as we ask user
to pass in a "--yes-i-really-really-mean-it-no-faking"
string now.

The old "osd pool delete" command won't allow user to do this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: sanity check osd number before applying pg-temp

In case user did something stupid and causing weird results.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: guard "osd crush rule create-replicated"

require Luminous before using the crush device classes.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: apply both pg_upmap and pg_upmap_items

This allows a more flexible upmap policy by combinating
both pg_upmap and pg_upmap_items.

For example:
./bin/ceph osd pg-upmap 1.0 1 2 3
set 1.0 pg_upmap mapping to [1,2,3]

./bin/ceph osd pg-upmap-items 1.0 1 5
set 1.0 pg_upmap_items mapping to [1->5]

 ./bin/ceph pg ls-by-pool cephfs_data_a
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE        STATE_STAMP                VERSION REPORTED UP      UP_PRIMARY ACTING  ACTING_PRIMARY
1.0           0                  0        0         0       0     0   0        0 active+clean 2017-07-10 11:04:53.955681     0'0    22:13 [5,2,3]          5 [5,2,3]              5

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: improve "osd upmap" error message

by adding a explicit guide.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

crush: fix potential weight overflow

E.g.:
./bin/ceph osd crush reweight osd.0 32768

ID WEIGHT      TYPE NAME                                            UP/DOWN REWEIGHT PRIMARY-AFFINITY
-4 32770.00000 root default~hdd
-3           -     host gitbuilder-ceph-rpm-centos7-amd64-basic~hdd
 0           -         osd.0                                             up  1.00000          1.00000
 1     1.00000         osd.1                                             up  1.00000          1.00000
 2     1.00000         osd.2                                             up  1.00000          1.00000
-1 32770.00000 root default
-2           -     host gitbuilder-ceph-rpm-centos7-amd64-basic
 0           -         osd.0                                             up  1.00000          1.00000
 1     1.00000         osd.1                                             up  1.00000          1.00000
 2     1.00000         osd.2                                             up  1.00000          1.00000

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16079 from smithfarm/wip-doc-make-check

doc: dev: add notes on PR make check validation test

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16027 from liewegas/wip-crush-rule-class

mon,crush: create crush rules using device classes for replicated and ec pools via cli

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16228 from smithfarm/wip-rados-upgrade-2

tests: fix rados/upgrade/jewel-x-singleton and make workunit task handle repo URLs not ending in ".git"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15973 from majianpeng/bluefs-async-compact-log

os/bluestore/BlueFS: clear current log entrys before dump all fnode,

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16214 from liupan1111/wip-fix-mutex

common/Mutex.cc: fixed the error in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa: workunit: handle repo ending in "/ceph-ci"

If the repo ends in "/ceph-ci" it's the same as if it ended in "/ceph-ci.git"

Before this change, the following command was broken if the workunit specified,
e.g., "branch: jewel":

teuthology-suite --ceph-repo https://github.com/ceph/ceph --ceph master
--suite-repo https://github.com/ceph/ceph-ci --suite-branch wip-foo . . .

Fixes: http://tracker.ceph.com/issues/20554
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/suites/rados/upgrade: upgrade client.0 node too

Fixes: http://tracker.ceph.com/issues/20368
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16017 from liewegas/wip-20454

os/bluestore/BlueFS: clean up log_writer aios from compaction

Merge pull request #16025 from liewegas/wip-19964

mon: do crushtool test with fork and timeout, but w/o exec of crushtool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #16121 from shashalu/fix-kerneldevice-size

os/bluestore: clear up redundant size assignment in KerenelDevice

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16205 from javacruft/build-ec-isa-as-module

build: build erasure-code isa lib without versions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16212 from dillaman/wip-qa-ceph-task

qa/tasks/ceph: don't hard-code cluster name when copying fsid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16213 from liewegas/wip-radosbench-timeout

qa/tasks/radosbench: longer timeout

common/Mutex.cc: fixed the error in comment

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #16198 from tchaikov/wip-doc-fixes

doc: kill sphinx warnings

test/cli/crushtool: fix rules.t

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16225 from liewegas/wip-mon-auth-caps

qa/suites/rados/singleton/all/mon-auth-caps: more osds so we can go clean

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16216 from ceph/wip-mgr-cd-updates

[qa/ceph-deploy]:  run create mgr nodes as well

Merge pull request #15689 from jdurgin/wip-filestore-rand-split

os/filestore/HashIndex: randomize split threshold by a configurable amount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16078 from xiexingguo/wip-misc

mon, osd: misc fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16218 from javacruft/ubuntu-fix-systemd-enable

build: execute dh_systemd_{enable,start} after dh_install

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15944 from xiexingguo/wip-bluestore-20170627

os/bluestore: reduce some overhead for _do_clone_range() and _do_remove()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16052 from dzafman/wip-20471

osd: Check for and automatically repair object info soid during scrub

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16147 from Yan-waller/wip-waller-0703radostool

tools/rados: some cleanups

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #16217 from liewegas/wip-class-luminous

mon/OSDMonitor: guard 'osd crush set-device-class'

qa/suites/rados/singleton/all/mon-auth-caps: more osds so we can go clean

and scrub

Signed-off-by: Sage Weil <sage@redhat.com>

osd: Check for and automatically repair object info soid during scrub

Fixes: http://tracker.ceph.com/issues/20471
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge remote-tracking branch 'gh/luminous'

mon/OSDMonitor: clean up error messages

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15997 from pritha-srivastava/wip-rgw-assert-correction

rgw: Correcting the condition in ceph_assert while parsing an AWS Principal

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15703 from DHB-liuhong/website-lh

 rgw: fix the subdir without slash of s3 website url

Reviewed-by: Robin H. Johnson <robbat2@gentoo.org>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15916 from oritwas/wip-rgw-fix-prefix-list

Wip rgw fix prefix list

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15996 from mikulely/rgw-rest-complain

rgw: silence compile warning from -Wmaybe-uninitialized

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16062 from YangFanlinux/myceph1

rgw: Replace get_zonegroup().is_master_zonegroup() with is_meta_master() in RGWBulkDelete::Deleter::delete_single()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15998 from mdw-at-linuxbox/fixmarker

rgw: fix marker encoding problem.

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

create rbd pool since its not created by default anymore

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #16095 from zhaochao/rgw-add-a-new-errcode-for-non-existed-subuser

rgw: add a new error code for non-existed subuser.

Reviewed-by: Jos Collin <jcollin@redhat.com>

build: execute dh_systemd_{enable,start} after dh_install

Ensure that dh_systemd_* debhelpers are executed after dh_install
has installed the systemd unit and target definitions.

This ensures that targets are enabled by default once installed,
resolving issues with startup of ceph daemons on server reboot.

Fixes: http://tracker.ceph.com/issues/19585
Signed-off-by: James Page <james.page@ubuntu.com>

mon/OSDMonitor: guard 'osd crush set-device-class'

Requires luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16148 from liewegas/wip-20434

mon: collect mon metdata as part of the election

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

mon: collect mon metadata as part of election

Previously each peon would send a message to teh leader with its metadata
immediately after the election was won.  However, at that point paxos
usually wasn't writeable, which meant the old update_mon_metadata() method
didn't persist reliably, updates would race, and generally speaking
metadata wasn't reliably updated.

Fix this by including metadata as part of the election ack, and persisting
the whole quorum when the election is won.  This ensures it is up to date.

Fixes: http://tracker.ceph.com/issues/20434
Signed-off-by: Sage Weil <sage@redhat.com>

mon: keep mon metadata in ram

Signed-off-by: Sage Weil <sage@redhat.com>

mon: use single collect_metadata helper, consistently set 'addr'

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: supply cluster name to ceph CLI

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

create mgr nodes explicitly now, previously they where
started alongside monitor daemons due to systemd rules.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/workunits/cephtool/test.sh: remove two crushtool validation tests

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushTester: remove old test_with_crushtool helper

test_with_fork is superior in all ways :)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: test crush with fork but not crushtool spawn

We see timeouts here, but I very much suspect they are due to the overhead
of launching the crushtool process and not the test itself.  We have
perfectly code already in our process, though; we just want to isolate
failure and time out reliably.  So, fork and timeout, without executing
a new binary.

Hopefully-fixes: http://tracker.ceph.com/issues/19964
Signed-off-by: Sage Weil <sage@redhat.com>

common/fork_function: close all fds in children

Not strictly necessary, but a tidier.

Signed-off-by: Sage Weil <sage@redhat.com>

common/fork_function: helper to run a function, forked, with a timeout

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16128 from liewegas/wip-20475

mon: debug session feature tracking

Merge pull request #16157 from xiexingguo/wip-bluestore-cache

os/bluestore: differ default cache size for hdd/ssd backends

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16188 from liewegas/wip-device-class-guard

mon/OSDMonitor: do not allow crush device classes until luminous

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16194 from cbodley/wip-20534

msg: QueueStrategy::wait() joins all threads

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16196 from liewegas/wip-19518

ceph_test_rados_api_c_read_operations: do not assert per-op rval is correct

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/radosbench: longer timeout

Hit another timeout due to aggressive thrashing with size==min_size.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: don't hard-code cluster name when copying fsid

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16201 from tchaikov/wip-20432

common/Timer: do not add event if already shutdown

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16208 from ceph/wip-base-postun

rpm: Fix undefined FIRST_ARG

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rpm: Fix undefined FIRST_ARG

If FIRST_ARG is undefined, the rpms will show an error on upgrade
because the condition in the postun script gets broken.

This was a regression introduced by commit for issue 20077 that moved
ceph-disk unit files to ceph-base.

Fixes: http://tracker.ceph.com/issues/20077
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #16126 from ktdreyer/rpm-epoch-2

rpm: bump epoch ahead of RHEL base

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

build: build erasure-code isa lib without versions

As erasure coding modules are loaded dynamically by Ceph, there
is no need to build the backend modules with library versioning.

This change brings the isa library inline with other erasure-code
sub-modules.

Signed-off-by: James Page <james.page@ubuntu.com>

Merge pull request #16199 from tchaikov/wip-release-note-escape-stars

tools: ceph-release-notes: escape asterisks not for inline emphasis

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15895 from ganeshmaharaj/isal-v2.18

isa-l: update isa-l to v2.18

Reviewed-by: Kefu Chai <kchai@redhat.com>

common/Timer: do not add event if already shutdown

otherwise the callback is leaked.

Fixes: http://tracker.ceph.com/issues/20432
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools: ceph-release-notes: escape asterisks not for inline emphasis

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'upstream/pull/15844/head' into master

* commit '47ab651a1f':
  qa: reset journal before cephfs_journal_tool_smoke.sh exits
  qa: remove 'apply' from journal_tool_smoke.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge remote-tracking branch 'upstream/pull/15822/head' into master

* upstream/pull/15822/head:
  qa: add timeout/repeat to pool df

Reviewed-by: John Spray <jspray@redhat.com>

Merge remote-tracking branch 'upstream/pull/15950/head' into master

* upstream/pull/15950/head:
  qa: update thrash max mds testing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge remote-tracking branch 'upstream/pull/15758/head' into master

* upstream/pull/15758/head:
  mds: repeat dir->is_frozen in rdlock_path_xlock_dentry

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge remote-tracking branch 'upstream/pull/15510/head' into master

* upstream/pull/15510/head:
  mds: improve freeze deadlock detection

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge remote-tracking branch 'upstream/pull/15937/head' into master

* upstream/pull/15937/head:
  qa: remove unused quota config option

Reviewed-by: John Spray <jspray@redhat.com>

Merge remote-tracking branch 'upstream/pull/15825/head' into master

* upstream/pull/15825/head:
  mds: remove unused MDBalancer member
  Update last_epoch_(over|under) in MDBalaner if mds0 has failed

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge remote-tracking branch 'upstream/pull/15818/head' into master

* upstream/pull/15818/head:
  mds: assert straydn to silence coverity error

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge remote-tracking branch 'upstream/pull/15817/head' into master

* upstream/pull/15817/head:
  qa: wait for healthy cluster before testing pins

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge remote-tracking branch 'upstream/pull/13770/head' into master

* upstream/pull/13770/head:
  tasks/cephfs: add TestStrays.test_replicated_delete_speed

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: kill sphinx warnings

* doc/release-notes.rst: escape asterisks not used for inline emphasis
    fix bad hyper links
* doc/rados/troubleshooting/troubleshooting-osd.rst: escape asterisks
    not used for inline emphasis
* doc/radosgw/index.rst: add orphan docs to toc
* doc/dev/perf_histograms.rst: indent block quote
* doc/install/manual-freebsd-deployment.rst: fix block quotes
* doc/mgr/administrator.rst: escape asterisks not used for inline emphasis
* doc/start/quick-rbd.rst: add missing hyper link target

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: differ default cache size for hdd/ssd backends

This is a follow-up change of https://github.com/ceph/ceph/pull/15976
and makes the bluestore cache capacity being self-adaptive for
different backends.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15786 from markhpc/wip-bluestore-cache-behavior

os/bluestore: cap rocksdb cache size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #16120 from yanghonggang/master

os/filestore/FileJournal: FileJournal::open() close journal file before return error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16019 from wido/mgr-plugin-zabbix

mgr: Zabbix monitoring module

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_c_read_operations: do not assert per-op rval is correct

This is not included in the pg log and may be zeroed if there is an op
replayed.  Disable the assertion until the underlying bug is fixed.  See
http://tracker.ceph.com/issues/19518

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15647 from gmayyyha/format-config-show

common,config: OPT_FLOAT and OPT_DOUBLE output format in config show

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15743 from yuyuyu101/wip-20330

msg: make listen backlog an option, increase from 128 to 512

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15781 from kchiu/wip-filejournal-fix

os/filestore: call committed_thru when no journal entries are replayed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15810 from liupan1111/wip-precise

msg/async: make recv_stamp more precise

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #16159 from tchaikov/wip-20432

Context: C_ContextsBase: delete enclosed contexts in dtor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15878 from renhwztetecs/renhw-wip-common-dns

common: Update the error string when res_nsearch() or res_search() fails

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

Update isa-l to v2.18

This upgrade brings
-  Complete rewrite of DEFLATE optimizations resulting in 5X better
   throughput and compression ratios comapred to  zlib, lz4, lzo and 2X
   better decompression performance when compared to zlib.
-  AVX512 improvements to multi-buffer versions of MD5, SHA-1 and SHA-256
   cryptographic hashing functions resulting in 3X better in performance
   compared to the AVX2 generation.

This update improves bluestore compression and potential to improve
dedup performance.

Testing this patch with objectstore fio yeided a max of 37% increase in
compression performance.

Fio Params:
rw=randwrite,buffer_compress_percentage=50,nr_files=64,direct=1,buffered=0,size=4G,bs=64k

Test Avg BlueStore Compression Time
v2.16(us)  v2.18(us)
iodepth=1,jobs=1 241.658    175.476 27.39%
iodepth=2,jobs=2 184.174    145.861 20.80%
iodepth=4,jobs=4 143.617    104.392 27.31%
iodepth=8,jobs=8 146.984    116.505 20.74%
iodepth=16,jobs=8 180.167    112.769 37.41%

Signed-off-by: Ganesh Mahalingam <ganesh.mahalingam@intel.com>
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>

msg: QueueStrategy::wait() joins all threads

wait() was only looping over disp_threads, which is an intrusive list
that only contains threads that are waiting on a message from
ds_dispatch(). this means that some QSThreads could outlive the
QueueStrategy itself, causing a segfault in QueueStrategy::entry()

Fixes: http://tracker.ceph.com/issues/20534
Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/config_opts: favor bluestore kv cache before meta until max.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #16171 from wjwithagen/wip-wjw-freebsd-nobtrfs

os/filestore: Exclude BTRFS on FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>