Merge pull request #24594 from theanalyst/wip-reshard-enoent

rgw: reshard add: fail correctly on a non existant bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #24638 from linuxbox2/wip-reshard-st

radosgw-admin: translate reshard status codes (trivial)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24816 from linuxbox2/wip-rgw-keystone-secret

rgw: move keystone secrets from ceph.conf to files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24869 from joke-lee/wip-op-mask-radosgw-admin-rest

rgw: admin rest api support op-mask

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24887 from cbodley/wip-36662

rgw: beast frontend parses ipv6 addrs

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23906 from dongbula/nf-add-latency-after-finish-processing

rgw: add latency info in the log of req done

Merge pull request #24947 from votdev/bug_36708

mgr/dashboard: tasks.mgr.dashboard.test_osd.OsdTest failures

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #24966 from daixiang0/correct-rbytes-description

doc: correct rbytes description

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: correct rbytes description

The default was changed from on to off in 4.6 because the way rbytes behaves can be surprising and sometimes confuses tools like rsync.

Signed-off-by: Xiang Dai <764524258@qq.com>

Merge pull request #24817 from tone-zhang/wip-64-assert

bluestore/NVMEDevice.cc: fix ceph_assert() when enable SPDK with 64KB kernel page size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24964 from Songweibin/wip-doc-rados

doc/rados: correct osd path in troubleshooting-mon.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23940 from robbat2/wip-12713-chunked-encoding

rgw: use chunked encoding to get partial results out faster

rgw: add latency info in the log of req done

Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #24749 from dzafman/wip-36474

Add support for osd_delete_sleep configuration value

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24977 from alfredodeza/wip-bz1644265

ceph-volume util.encryption robust blkid+lsblk detection of lockbox

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume util.encryption robust blkid+lsblk detection of lockbox

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: backend api tests: tasks.mgr.dashboard.test_osd.OsdTest failures

- Fix bug in Dashboard QA unit test framework. Don't set the application type header manually, this is done by the requests library if required.
- Enhance QA unit test helper: Print the response of the API request if it fails. This should help to identify the problem more easily.
- Fix bug in the OSD controller. A parameter needs to be converted to integer.
- Take care that the params of the request object are not modified.

The issue was introduced by PR https://github.com/ceph/ceph/pull/24475. The CherryPy json_in plugin disclosed the errorneous unit test helper implementation.

Fixes: https://tracker.ceph.com/issues/36708
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #24847 from pritha-srivastava/wip-rgw-assume-role-doc

rgw: Improvements to STS Lite documentation.

Merge pull request #24841 from b-ranto/wip-deep-scrub

mgr/restful: Fix deep-scrub typo

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/restful: Fix deep-scrub typo

There is a typo in implemented commands for OSD. It reads deep_scrub
instead of deep-scrub which is the true valid command.

Fixes: http://tracker.ceph.com/issues/36720
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #24681 from tspmelo/wip-npm-update

mgr/dashboard: Update npm packages

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/rados: correct osd path in troubleshooting-mon.rst

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24937 from alfredodeza/wip-rm36704

ceph-volume tests.systemd update imports for systemd module

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #15047 into master

* refs/pull/15047/head:
tool/ceph_objectstore_tool: add new op that reset last_complete to last_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24717 from trociny/wip-rbd-mirror-status-instance

rbd: show info about mirror daemon instance in image mirror status output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24815 from iridescent-rsy/remove-ceph_abort

rbd-nbd: do not ceph_abort() after print the usages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24919 from trociny/wip-tests-migration-namespace

test/librbd: migration supporting namespace tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests.systemd update imports for systemd module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd import main so console_scripts work for executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24824 from Devp00l/wip-issue-36616

mgr/dashboard: Simplify OSD disabled action test

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #14982 into master

* refs/pull/14982/head:
common/mempool: tolerate sum < 0
common/ceph_context: add perf counters for all mempools
common/ceph_common: add heartbeat perf counters unconditionally

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #20735 into master

* refs/pull/20735/head:
msg/async: fix variable type to make it same to EventCenter::init() incoming parameter type

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #21723 into master

* refs/pull/21723/head:
osd: flush dirty data if cache_target_dirty_ratio = 0.0

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #24835 into master

* refs/pull/24835/head:
include/buffer: make static_assert c++11-safe

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #24881 from alfredodeza/wip-rm36672

ceph-volume tests install ceph-ansible's requirements.txt dependencies

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

mgr/dashboard: Fix "merge" audit warning

Fixes: https://tracker.ceph.com/issues/36702
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove unit tests warnings

Recent Angular update is causing a warning to be shown when you call
"router.navigate" outside an angular zone.

This is not affecting the code itself, just the unit tests.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update remaining packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update Angular packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update rxjs and ngx-bootstrap packages

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update prettier package

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #24766 from familyuu/crush

mgr/dashboard: CRUSH map viewer

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add CRUSH map viewer

Fixes: http://tracker.ceph.com/issues/35684
Signed-off-by: familyuu <guodan1@lenovo.com>

Merge pull request #24920 from wjwithagen/wjw-fix-build-symbolic

cmake: do not pass -B{symbolic,symbolic-functions} to linker on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24814 into master

* refs/pull/24814/head:
Merge PR #24804 into nautilus
include/types: fixed compile warning for signed/unsigned comparison
osd/PrimaryLogPG: uncommitted dup ops should respond with logged return code
osd/PrimaryLogPG: propagate error return codes on object copy_get ops
osd/PGLog: optionally record error return codes for extra_reqids
osd/osd_types: include PG log return codes in object copy data

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: move keystone secrets from ceph.conf to files

Permits setting restrictive permissions on these secrets.

Fixes: http://tracker.ceph.com/issues/36621
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ceph-volume tests install ceph-ansible's requirements.txt dependencies

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rbd: show info about mirror instance in image mirror status output

It is particularly useful when running multiple rbd-mirror instances
in Active-Passive or Active-Active mode.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24891 from iridescent-rsy/fix-return-value

rbd: make sure the return-value 'r' will be returned

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24893 from iridescent-rsy/need-to-return

rbd: return error code when the source and distination namespace are different

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24823 from Devp00l/wip-issue-36615

[RFC] mgr/dashboard: Cleanup of OSD list methods

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Simplify OSD disabled action test

Fixes: https://tracker.ceph.com/issues/36616
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #24475 from votdev/feature_36193

mgr/dashboard: Audit REST API calls

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24856 from leseb/spec-rbd-mirror

spec: add missing rbd mirror bootstrap directory

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24892 from p-na/pna-cleanup-pool-form-simpl

mgr/dashboard: Simplification of PoolForm method

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cmake: do not pass -B{symbolic,symbolic-functions} to linker on FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

spec: add missing rbd mirror bootstrap directory

Currently /var/lib/ceph/bootstrap-rbd-mirror is absent, which means we
need to create it manually in order to pool the
client.bootstrap-rbd-mirror key.

Signed-off-by: Sébastien Han <seb@redhat.com>

mgr/dashboard: Audit REST API calls

Fixes: https://tracker.ceph.com/issues/36193
Enable API auditing with 'ceph dashboard set-audit-api-enabled true' (default is false). If you do not want to log the request payload, then disable it via 'set-audit-api-log-payload false' (default is true).

Example output:
2018-10-08 10:25:21.850994 mgr.x [INF] [DASHBOARD] from='https://[::1]:44410' path='/api/auth' method='POST' user='None' params='{"username": "admin", "password": "***", "stay_signed_in": false}'

Signed-off-by: Volker Theile <vtheile@suse.com>

test/librbd: migration supporting namespace tests

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #23190 from maxcuttins/master

doc: rewrote the iscsi-target-cli installation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: provide instance_id for `ceph service status`

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: API to get info which rbd-mirror daemon is mirroring image

Signed-off-by: Mykola Golub <mgolub@suse.com>

cls/rbd: new methods to get image mirror instance info

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #24870 into master

* refs/pull/24870/head:
client: request next osdmap for blacklisted client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: rewrote the iscsi-target-cli installation

Rewrote the documentation in a step-by-step style, reformatted using RST's
titles instead of bolds, added installation of GIT if needed, added
installation of compilers needed for tcmu-runner

Signed-off-by: Massimiliano Cuttini <massimiliano.cuttini@phoenixweb.it>

Merge pull request #24895 from votdev/feature_36681

rgw: Return tenant field in bucket_stats function

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24864 from dillaman/wip-36659

rbd-mirror: update mirror status when stopping

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24836 from dillaman/wip-26951

rbd: support namespaces for image migration

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24812 from gregsfortytwo/wip-readable-fails-on-empty

test: make readable.sh fail if it doesn't run anything

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22923 from ceph/wip-fix-latest-distro

qa/tests: update links for centos latest to point to 7.5

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #24897 from trociny/wip-36683

test/rbd_mirror: race in TestMockImageMap.AddInstancePingPongImageTest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/rbd_mirror: race in TestMockImageMap.AddInstancePingPongImageTest

Remove a context from peer_ack_ctxs before completing to avoid a race
with a listener inserting a new one.

Fixes: http://tracker.ceph.com/issues/36683
Signed-off-by: Mykola Golub <mgolub@suse.com>

rgw: Return tenant field in bucket_stats function

Fixes: https://tracker.ceph.com/issues/36681
Signed-off-by: Volker Theile <vtheile@suse.com>

rbd: return error code when the source and distination namespace are different.

Signed-off-by: Shiyang Ruan <ruansy.fnst@cn.fujitsu.com>

mgr/dashboard: Simplification of PoolForm method

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

rbd: make sure the return-value 'r' will be returned.

Signed-off-by: Shiyang Ruan <ruansy.fnst@cn.fujitsu.com>

Merge pull request #24616 from tchaikov/wip-wip-librados-3

librados: split C++ and C APIs into different source files

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #24555 into master

* refs/pull/24555/head:
doc: document optional timeout argument for `cache drop`
mds: make timeout parameter optional for "cache drop"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #24747 into master

* refs/pull/24747/head:
client: add new routine to get fscid from a ceph_mount

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24868 from liewegas/wip-doc-fix

doc/rados/operations/crush-map-edits: fix 'take' syntax

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: beast frontend parses ipv6 addrs

Fixes: http://tracker.ceph.com/issues/36662
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: beast frontend uses default port 443 for ssl_endpoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/tests: update links for centos latest to point to 7.5

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rpm,deb: package libceph-common.so.* not libceph-common.so*

be more explicit on what we are packaging. because only
libceph-common.so.${soversion} will be packaged, since libceph-common.so
won't be installed by cmake anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not create symlink for libceph-common.so.*

and bump up its soversion, so it won't conflict with libceph-common.so.0
offered by librados2.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: add librados3 to exclude list of pre-nautilus install tasks

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: s/librados2/librados3/

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: bump librados so version

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: bump version number

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: update test_envlibrados_for_rocksdb.sh for libradospp split

include a patch so rocksdb can use libradospp instead of librados. will
upstream the patch and make it work for both pre-nautilus librados and
nautilus libradospp

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: add libradospp* packages

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: add libradospp* packages

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: move C++ APIs into libradospp

the goal is to decouple C++ API from C API, and to version them
differently, as they are targeting different consumers.

this allows us to change the C++ API and bumping up its soversion
without requiring consumer to recompile the librados client for
using the new librados. in this way, C++ API can move faster than
C API. for example, if bufferlist interface is changed for better
performance, and this breaks existing API/ABI, we can bump up
the C++ library's soversion, and and the C library's version unchanged
but ship the new librados's C binding. so the librados client linked
against librados's C library will be able to take advantage of
the improvement in C++ library. while the librados client
linked against C++ library won't break at runtime due to unresolved
symbol or changed structure layout.

this is massive change, the genereal idea is to

* split librados.cc into two source files: librados_c.cc and
  librados_cxx.cc, the former for implementing C APIs, the later
  for C++ APIs.
* extract the C++ API in librados into librados-cxx, the library
  name will be libradospp. but we can change it before nautilus
  is released.
* link these librados libraries with static libraries which it
  depends on, so "-Wl,--exclude-libs,ALL" link flags can help
  hide the non-public symbols.
* extract the tests exercising librados' C++ API into a different
  source file named *_cxx.cc. for instance, to move the C++ tests
  in aio.cc into aio_cxx.cc
* extract the shared helper functions which do not use any librados
  or librados-cxx APIs into test_shared{.cc,h}. the "shared" here
  means, *shared* by C++ and C tests.
* extract the test fixtures, i.e., the subclasses of testing::Test,
  for testing C++ APIs into testcase_cxx.cc.
* update qa/workunits/rados/test.sh accordingly to add the splitted
  tests
* update the consumers of librados to link against librados-cxx
  instead, if they are using the C++ API.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: move get_inconsistent_pgs() into RadosClient

so we don't need parse the pg string and render it again for librados C
API.

the downside of this change is that, get_inconsistent_pgs() could be
implemented using RadosClient, instead be *in* it. but before we have
a place for the helper functions for these higher-level functions,
RadosClient is a good place for hosting them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24406 from ivancich/wip-rgw-renew-reshard-lock

rgw: renew resharding locks to prevent expiration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Cleanup of OSD list methods

Some test changes were required, too.

Fixes: https://tracker.ceph.com/issues/36615
Signed-off-by: Stephan Müller <smueller@suse.com>

rbd: support namespaces for image migration

Fixes: http://tracker.ceph.com/issues/26951
Signed-off-by: Jason Dillaman <dillaman@redhat.com>