Merge pull request #22065 from ricardoasmarques/wip-table-pagination-style

Improve table pagination style

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22149 from b-ranto/wip-fix-metadata

prometheus: Fix order of occupation values

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #22031 from tspmelo/wip-jest

mgr/dashboard: Add Jest Runner

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22088 from tspmelo/fix-rbd-task-metadata

mgr/dashboard: Fix RBD task metadata

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

prometheus: Fix order of occupation values

When we merged the PR to unify the metadata labels, we forgot to switch
the order of hostname and disk in ceph_disk_occupation metric.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #22069 from tchaikov/wip-mgr-cleanup

mgr: fix typo in variable name and cleanups

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #22077 from majianpeng/config-w/o-mgr

cmake: make -DWITH_MGR=OFF work.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22089 from jcsp/wip-24175

mgr/status: output to stdout, not stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22074 from dzafman/wip-parens

test: Need to escape parens in log-whitelist for grep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22096 from yuriw/wip-yuriw-crontab

qa/tests: added filters to support distro tests for client-upgrade tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - reverted clients upgrades for luminous runs
removed ~/rhel_only_on_ovh.yaml to be able to run on ovh
forced runs on smithi only

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #22036 from linuxbox2/wip-warn-orphans

rgw: require --yes-i-really-mean-it to run radosgw-admin orphans find

test: Add CACHE_POOL_NO_HIT_SET to whitelist for mon/pool_ops.sh

Ignore
  cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Need to escape parens in log-whitelist for grep

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #22064 into master

* refs/pull/22064/head:
os/bluestore: fix signed/unsigned mismatch warning during building

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #22091 into master

* refs/pull/22091/head:
crush: update choose_args on bucket removal
crush: update choose_args on bucket removal, resize, or position mismatch
crush: create weight-set on demand when doing a choose-args reweight
test/cli/crushtool: use straw2 buckets for choose-args test
crush: weight_set_size -> weight_set_positions

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #21540 into wip-sage-testing-20180521.120735

* refs/pull/21540/head:
tests/crypto: print compile warning when NSS is unavailable.
tests/crypto: add tests for the no-bl encrypt/decrypt, part 2.
tests/crypto: add tests for the no-bl encrypt/decrypt.
auth: use OpenSSL for CryptoAESKeyHandler's no-bl encrypt/decrypt.
auth: extend CryptoKey with no-bl encrypt/decrypt.
auth: CryptoAESKeyHandler switches from NSS to OpenSSL.
auth: the outbuf of AES should be multiple of block size
auth: cache the PK11Context for CryptoAESKeyHandler

Merge PR #21973 into wip-sage-testing-20180521.120735

* refs/pull/21973/head:
pybind/ceph_argparse.py: show usage on fully matched cmd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #21985 into wip-sage-testing-20180521.120735

* refs/pull/21985/head:
common: partially revert 95fc248 to make get_process_name work

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #21994 into wip-sage-testing-20180521.120735

* refs/pull/21994/head:
os/bluefs/BlueFS: cleanup code.

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #22103 from tchaikov/wip-qa-test_envlibrados_for_rocksdb

qa/workunits/rados/test_envlibrados_for_rocksdb: install g++ not g++-4.7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: make -DWITH_MGR=OFF work.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #22084 from liewegas/wip-fix-fop

osd/PrimaryLogPG: fix try_flush_mark_clean write contention case

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22072 from dillaman/wip-24165

librbd: prevent the use of internal feature bits from external users

Reviewed-by: Mykola Golub <mgolub@suse.com>

buffer.h: avoid using c++14 features

so we dont't need to force librados client to move to C++14 or up.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: use cmake not make

* so we just rely on a single build system instead of two of them, the
  other place we use cmake is cmake/modules/BuildRockDB.cmake.
* disable gflags when building rocksdb, it's optional and does not help
  in the sense of testing librados support.
* disable prompts when installing on debian, to silence warnings like:
  debconf: unable to initialize frontend: Dialog
* drop --force-yes option, as it is deprecated, and is replaced with
  --allow-downgrades, --allow-remove-essential,
  --allow-change-held-packages,  but none of them apply in our case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21982 from tchaikov/mgr-brag

mgr/telemetry: Add Ceph Telemetry module to send reports back to project

Merge PR #21847 into wip-sage-testing-20180521.023125

* refs/pull/21847/head:
log: disk write coalescing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22063 from liewegas/wip-unit-spaces

include/types: space between number and units

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #22042 from liewegas/wip-rm-pool-race

ceph_test_rados_api_misc: remove obsolete LibRadosMiscPool.PoolCreationRace

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #22106 from liewegas/wip-rados-distros

qa/suites/rados/thrash-old-clients: only centos and 16.04

Merge PR #21997 into master

* refs/pull/21997/head:
mds: broadcast quota to relevant clients when quota is explicitly set

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22001 into master

* refs/pull/22001/head:
client: avoid freeing inode when it contains TX buffer heads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22023 into master

* refs/pull/22023/head:
messages: print quota info in MClientQuota msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22045 into master

* refs/pull/22045/head:
client: cleanup Cap init

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22080 into master

* refs/pull/22080/head:
client: fix issue of revoking non-auth caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crush: update choose_args on bucket removal

The specific bug I see is that a bucket no longer exists but its
choose_args still does.  However, I'm also taking the opportunity to
verify that the choose_args agrees with the bucket sizes and position
counts everywhere else, too.  Check for

- ids or weight_sets for buckets that don't exist or aren't straw2
- weight_set_positions that don't match the choose_args
  - don't fix this, just warn.  i'm not sure how it would happen.  :/
- weight_set sizes that don't match the bucket size

Fixes: http://tracker.ceph.com/issues/24167
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: only centos and 16.04

We don't have old client builds for the newer distros.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21961 from ceph/wip-cd-ovh-vol

qa/tests: fix volume size when running in ovh

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22098 from liewegas/wip-24180

mon/LogMonitor: call no_reply() on ignored log message

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: install g++ not g++-4.7

since fog now deploys ubuntu 18.04 where g++-4.7 is not available
anymore, so g++ is good enough.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22092 from liewegas/wip-mimic-upgrades

qa/suites/upgrade/luminous-x: a few fixes

Merge pull request #21941 from ukernel/wip-24053

mds: reply session reject for open request from blacklisted client

qa/tasks/cephfs: add timeout parameter to kclient umount_wait

Just make caller happy. there is no easy way to support timeout.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24053

mds: reply session reject for open request from blacklisted client

Kernel client and old version libcephfs do not check if themselves
are blacklisted. They can be stuck at opening session after getting
blacklisted. The session reject message can avoid this.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24054

client: fix issue of revoking non-auth caps

when non-auth mds revokes caps, Fcb caps can still be issued by auth
auth mds. It's wrong to flush buffer or invalidate cache when non-auth
mds revokes other caps. This bug can cause client to not respond the
revoke.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24172

Merge PR #22073 into master

* refs/pull/22073/head:
client: delay dentry trimming until after cap traversal
qa: test for trim_caps segfault for trimmed dentries

Reviewed-by: Zheng Yan <zyan@redhat.com>

client: delay dentry trimming until after cap traversal

Fixes: http://tracker.ceph.com/issues/24137
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test for trim_caps segfault for trimmed dentries

Test case by Yan Zheng [1].

[1] http://tracker.ceph.com/issues/24137#note-6

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests:  Removed kraken tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: Removed hammer client tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

crush: update choose_args on bucket removal, resize, or position mismatch

The specific bug I see is that a bucket no longer exists but its
choose_args still does.  However, I'm also taking the opportunity to
verify that the choose_args agrees with the bucket sizes and position
counts everywhere else, too.  Check for

- ids or weight_sets for buckets that don't exist or aren't straw2
- weight_set_positions that don't match the choose_args
- weight_set sizes that don't match the bucket size

Fixes: http://tracker.ceph.com/issues/24167
Signed-off-by: Sage Weil <sage@redhat.com>

crush: create weight-set on demand when doing a choose-args reweight

If we try to reweight an item in a choose-args set for a given item
but the bucket doesn't have a weight_set yet, create it on the fly, and
populate it with the canonical bucket weights.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool: use straw2 buckets for choose-args test

Choose-args only works for straw2 so it is misleading to test using a
map with straw buckets.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tests: added filters to support distro tests for upgrade/client-upgrade-* tests

ref:
https://github.com/ceph/ceph/pull/22087
https://github.com/ceph/ceph/pull/22090
https://github.com/ceph/ceph/pull/22094
https://github.com/ceph/ceph/pull/22095

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #21548 into master

* refs/pull/21548/head:
osdc/Objecter: apply throttle on enumerate_objects.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #21963 into master

* refs/pull/21963/head:
mds: properly setup client_need_snapflush for snap inode
Revert "mds: properly setup need_snapflush for snapped inode"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21972 into master

* refs/pull/21972/head:
client: increase verbosity level for log messages in helper methods

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22066 from liewegas/wip-pgmap-digest-decode

mon/PGMap: fix PGMapDigest decode

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/LogMonitor: call no_reply() on ignored log message

If we're dropping it on the floor, we need to tell the mon that, so that
it can tell the forwarding mon to discard its state.

Fixes: https://tracker.ceph.com/issues/24180
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/thrashosds-health: whitelist REQUEST_SLOW

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/luminous-x: whitelist SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/luminous-x: more cinder volumes

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22040 from alfredodeza/wip-rm24147

common/options: add description of osd objectstore backends

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <cutler@suse.cz>

crush: weight_set_size -> weight_set_positions

This naming was confusing!  This is the number of positions we
have weight_sets for.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix signed/unsigned mismatch warning during building

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/status: output to stdout, not stderr

I think this was just me forgetting the order
of the arguments when writing the return statements.

Fixes: http://tracker.ceph.com/issues/24175
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: Fix RBD task metadata

Error message template for RBD copy was trying to read
an unexistent property of the returned metada.

Metadata for RBD edit was missing the new image name.
The new name should be displayed, instead of the old one,
when the user tries to use an existent image name.

Fixes: https://tracker.ceph.com/issues/24171
Signed-off-by: Tiago Melo <tmelo@suse.com>

tests/crypto: print compile warning when NSS is unavailable.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tests/crypto: add tests for the no-bl encrypt/decrypt, part 2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tests/crypto: add tests for the no-bl encrypt/decrypt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: use OpenSSL for CryptoAESKeyHandler's no-bl encrypt/decrypt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: extend CryptoKey with no-bl encrypt/decrypt.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: CryptoAESKeyHandler switches from NSS to OpenSSL.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: the outbuf of AES should be multiple of block size

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3cfcd02af6381022081d51367f7338bde1fcb1d2)

auth: cache the PK11Context for CryptoAESKeyHandler

in the flame graph, 0.50% of total time is used by
CephxSessionHandler::check_message_signature(), in which 0.27% is used
by PK11_CreateContextBySymKey(). so we should cache the PK11Context.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit f8c2036f7663be54030e13f1f9c1511cb40070f6)

cls/rbd: prevent creating image with internal feature bits enabled

Fixes: http://tracker.ceph.com/issues/24165
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: fsx should use the default image features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: prevent creation of images with internal feature bits

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: fix try_flush_mark_clean write contention case

Release ctx *after* we dereference it to queue up the other flush waiters.
This was a dumb mistake in the recent fix from 41609ba1b24e5384494c7959494e19302b5a2429

Fixes: http://tracker.ceph.com/issues/24174
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: mask off implicitly enabled feature bits for new images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: re-use image create option logic for clones

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22078 from tchaikov/wip-cls-rbd-fix-FTBFS

cls/rbd: fix build

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: fix build

3cbe0ced was merged after d03ff14, so update accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21915 from dillaman/wip-rbd-mirror-policy

rbd-mirror: optionally support active/active replication

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21651 from ceph/wip-da-dmclock-submodule

move dmclock subtree into submodule

Reviewed-by: Eric Ivancich <ivancich@redhat.com>

mgr/dashboard: Use Jest in package.json scripts

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22015 from tchaikov/wip-denc

buffer,denc: use ptr::const_iterator for decode

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Adapt unit tests to Jest

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add Jest Runner

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22059 from LenzGr/mgr-dashboard-documentation

doc: Updated dashboard documentation (features, SSL config)

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr: fix typo in variable name and cleanups

* s/sercive/service/
* also fix the logging message. before this change, we have

send_report send_report osd,2.0x557ea1c16550 sent me an unknown health
metric:

after this change,

send_report send_report osd.2 sent me an unknown health metric: 0x01

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd/import_export: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd/cli_generic: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli-integration/rbd: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: space between number and units

not

    client:   504B/s rd, 10.6MiB/s wr, 0op/s rd, 30op/s wr

but

    client:   504 B/s rd, 10.6 MiB/s wr, 0 op/s rd, 30 op/s wr

This was changed as part of d3cecebacdcebcf475808a6204de22dfa94d729d.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix PGMapDigest decode

The compat path wasn't clearing the map before filling in new entries.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22061 from tspmelo/wip-npmrc

mgr/dashboard: Disable package-lock.json creation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: Updated dashboard documentation (features, SSL config)

Updated Ceph manager dashboard documentation: added SSL to the
feature overview, added new section about configuring SSL certificates.
Rearranged installation instructions for setting host name and port.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>