ceph_volume_client: fix setting caps for IDs

... that have empty OSD and MDS caps. Don't add a ',' at the
start of OSD and MDS caps.

Fixes: http://tracker.ceph.com/issues/21501
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge PR #17697 into master

* refs/remotes/upstream/pull/17697/head:
pybind/ceph_volume_client: add get, put, and delete object interfaces
pybind/ceph_volume_client: remove 'compat_version'
pybind/ceph_volume_client: set the version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17604 into master

* refs/remotes/upstream/pull/17604/head:
client:fix a bug in lseek

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #16036 into HEAD

* refs/remotes/upstream/pull/16036/head:
mds: improve cap min/max ratio descriptions
mds: fix whitespace
mds: cap client recall to min caps per client
mds: fix conf types
mds: fix whitespace
doc/cephfs: add client min cache and max cache ratio describe
mds: adding tunable features for caps_per_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: improve cap min/max ratio descriptions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cap client recall to min caps per client

Fixes: http://tracker.ceph.com/issues/21575
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17774 from amitkumar50/cov-misc-4

include,messages,rbd: Initialize counter,group_pool

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17820 from zmedico/PGPool-update-optimize-with-subset-of

osd/PGPool::update: optimize with subset_of

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17884 from kungf/stats_invalid_scrub

osd: make scrub right now when pg stats_invalid is true

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17984 from adamemerson/wip-lists-of-pointers-are-bad-in-options

options: Do not use linked lists of pointers!

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17987 from xiexingguo/wip-kick-recovery-for-degraded-obj

osd/PrimaryLogPG: kick off recovery on backoffing a degraded object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17662 from shashalu/fix-set-bucket-versioning

rgw: set bucket versioninig donot change versioning status if missing status in xml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17775 from theanalyst/rgw-keystone-logging

rgw: avoid logging keystone revocation messages when not configured

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17916 from cbodley/wip-21506

rgw: dont reuse stale RGWObjectCtx for get_bucket_info()

Merge pull request #17742 from adamemerson/wip-21389

rgw: Check payment operations in policy

Reviewed-by: Enming Zhang <enming.zhang@umcloud.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18023 from liewegas/wip-changelog

debian/rules: no more ChangeLog

Merge pull request #18002 from adamemerson/wip-21582

rgw: Check bucket GetBucketLocation in policy

Merge pull request #18024 from adamemerson/wip-21597

rgw: Check bucket Website operations in policy

Merge pull request #17839 from liewegas/wip-recovery-preemption

osd: allow PG recovery scheduling preemption

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

debian/rules: no more ChangeLog

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket Website operations in policy

Add code to check s3:GetBucketWebsite and s3:PutBucketWebsite
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21597
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493896
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/rados/singleton/all/recovery-preemption: add test

This mirrors what I was testing locally.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18014 from theanalyst/doc-12.2.1

doc: 12.2.1 release notes

Merge pull request #17973 from ligne/patch-1

doc: Fix typos in placement-groups.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add 12.2.1 to releases table

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.2.1 release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #18000 from adamemerson/wip-21578

rgw: Check bucket CORS operations in policy

Merge pull request #17684 from majianpeng/bluefs-release-extents

os/bluestore/BlueFS: move release unused extents work in _flush_and_syn_log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18020 from theanalyst/doc-changelog-12.2.1

doc: add changelog for 12.2.1

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add changelog for 12.2.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #17769 from optimistyzy/97_new_opt

os/bluestore/NVMEDevice: Remove using dpdk thread

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18001 from liewegas/wip-21417

os/bluestore: put cached attrs in correct mempool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17755 from dillaman/wip-21353

doc/release-notes: ensure RBD users can blacklist prior to upgrade

Reviewed-By: Nathan Cutler <ncutler@suse.com>

os/bluestore: move several buffer{ptr,list}s into cache_other mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put new attrs in correct mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17966 from xiexingguo/wip-bluefs-kill-block-total

os/bluestore/BlueFS: several cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18013 from tchaikov/wip-dashboard-favicon

mgr/dashboard: use rel="icon" for favicon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17993 from jcsp/wip-21572

mgr/dashboard: sort servers and OSDs in OSD list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17937 from tchaikov/wip-automake-cleanup

automake: remove files required by automake

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mgr/dashboard: use rel="icon" for favicon

shortcut is not a standard compliant link type, see
https://www.w3.org/TR/html5/links.html#linkTypes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17991 from jcsp/wip-21570

mgr/dashboard: handle null in format_number

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17986 from tchaikov/wip-doc-rgw-config

doc: fix hyper link to radosgw/config-ref

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17926 from batrick/ptl-tool

scripts: add ptl-tool for scripting merges

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

rgw: Check bucket GetBucketLocation in policy

Add code to check s3:GetBucketLocation against bucket policy.

Fixes: http://tracker.ceph.com/issues/21582
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493934
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore: put attrs in mempool

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add ptr::[try_]reassign_to_mempool

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket CORS operations in policy

Add code to check s3:GetCORS and s3:PutCORS operations against bucket
policy.

Fixes: http://tracker.ceph.com/issues/21578
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/PG: handle racy preemption

If we finish recovery/backfill and go active, but also get
preempted at the same time, we can ignore the event.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow local recovery reservations to be preempted

If a PG has a higher recovery priority and a lower-priority item is in
progress, allow it to be preempted.  This triggers the RecoveryCancel
or BackfillCancel event with a 0 delay, which means it will immediately
re-request a reservation (and presumably wait).

Signed-off-by: Sage Weil <sage@redhat.com>

common/AsyncReserver: support preemption

If an (optional) preemption context is provided, use that to preempt
and existing reservation and grant a higher-priority one.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket versioning operations in policy

Add code to check s3:GetBucketVersioning and s3:PutBucketVersioning
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Check payment operations in policy

Add code to check s3:GetBucketRequestPayment and
s3:PutBucketRequestPayment operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17978 from alram/rbdmap-fix

rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix conf types

This correct an assertion failure.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning

test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: sort servers and OSDs in OSD list

Fixes: http://tracker.ceph.com/issues/21572
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: handle null in format_number

Fixes: http://tracker.ceph.com/issues/21570
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove ceph.com ref to favicon

We have a local copy referenced in the very next line.

Signed-off-by: John Spray <john.spray@redhat.com>

osd/osd_types: kill osd_peer_stat_t

Which currently has no more consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix typos in placement-groups.rst

Signed-off-by: Matt Boyle <matt.boyle@gmail.com>

Merge pull request #17988 from wumingqiao/wip-rados-update

include/rados: fix typo in librados.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: kick off recovery on backoffing a degraded object

As we are now blocking frontend ops from accessing that very object!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

include/rados: fix typo in librados.h

opeprate -> operate

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

doc: fix hyper link to radosgw/config-ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGPool::update: optimize with subset_of

Replace expensive inverval_set intersection_of and operator==
calls with a single subset_of call. I borrowed this idea from
Piotr Dałek's "osd/PGPool: don't use intermediate interval set"
patch. The following benchmark program demonstrates a 38%
performance increase:

#include <iostream>
#include <chrono>
#include "include/interval_set.h"

#define NANOSECONDS(d) \
    std::chrono::duration_cast<std::chrono::nanoseconds>(d).count()

typedef uint64_t snapid_t;
typedef std::chrono::steady_clock::duration duration;

duration PGPool_update_old(const interval_set<snapid_t> &rs) {
  std::chrono::steady_clock::time_point start, end;
  interval_set<snapid_t> newly_removed_snaps, cached_removed_snaps;

  // initialize state
  cached_removed_snaps = rs;

  // start timed simulation
  start = std::chrono::steady_clock::now();

  {
    newly_removed_snaps = cached_removed_snaps;
    interval_set<snapid_t> intersection;
    intersection.intersection_of(newly_removed_snaps, cached_removed_snaps);

    assert(intersection == cached_removed_snaps);
    cached_removed_snaps.swap(newly_removed_snaps);
    newly_removed_snaps = cached_removed_snaps;
    newly_removed_snaps.subtract(intersection);
  }

  // end timed simulation
  end = std::chrono::steady_clock::now();

  return end - start;
}

duration PGPool_update_new(const interval_set<snapid_t> &rs) {
  std::chrono::steady_clock::time_point start, end;
  interval_set<snapid_t> newly_removed_snaps, cached_removed_snaps;

  // initialize state
  cached_removed_snaps = rs;

  // start timed simulation
  start = std::chrono::steady_clock::now();

  {
    newly_removed_snaps = cached_removed_snaps;

    assert(cached_removed_snaps.subset_of(newly_removed_snaps));
    interval_set<snapid_t> removed_snaps = newly_removed_snaps;
    newly_removed_snaps.subtract(cached_removed_snaps);
    cached_removed_snaps.swap(removed_snaps);
  }

  // end timed simulation
  end = std::chrono::steady_clock::now();

  return end - start;
}

int main(int argc, char *argv[])
{
  assert(argc == 3);
  const int sample_count = std::stoi(argv[1]);
  const int interval_count = std::stoi(argv[2]);
  const int interval_distance = 4;
  const int interval_size = 2;
  const int max_offset = interval_count * interval_distance;
  interval_set<snapid_t> removed_snaps;

  for (int i = 0; i < max_offset; i += interval_distance)
    removed_snaps.insert(i, interval_size);

  duration old_delta(0), new_delta(0);

  for (int i = 0; i < sample_count; ++i) {
    old_delta += PGPool_update_old(removed_snaps);
    new_delta += PGPool_update_new(removed_snaps);
  }

  float ratio = float(NANOSECONDS(old_delta)) / NANOSECONDS(new_delta);

  std::cout << ratio << std::endl;
}

Suggested-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Signed-off-by: Zac Medico <zmedico@gmail.com>

inverval_set: optimize subset_of with sequential search

Optimize subset_of to use sequential search when it
performs better than the lower_bound method, for set
size ratios smaller than 10. This is analogous to
intersection_of behavior since commit 825470fcf919.

The subset_of method can be used in some cases as a
less-expensive alternative to the intersection_of
method, since subset_of can return early if any element
of the smaller set is not contained in the larger set,
and intersection_of has the added burden of storing
the intersecting elements.

Signed-off-by: Zac Medico <zmedico@gmail.com>

rbdmap: fix umount when multiple mounts use the same RBD

When a Kubernetes Pod consumes a RBD it is mounted two
times on the same host. When the host shutdown umount will
fail leading to a hung system

Signed-off-by: Alexandre Marangone <amarango@redhat.com>

Merge pull request #17063 from bspark8/wip-dynamic-cli-info-f

osd: use dmclock library client_info_f function dynamically

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17335 from Carudy/cxw

crush: safe check for 'ceph osd crush swap-bucket'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17410 from zmedico/PGPool-update-optimize-removed_snaps-comparison-when-possible

PGPool::update: optimize removed_snaps comparison when possible

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

common/options.h: Do not use linked lists of pointers

Especially since we're just pushing things in from the end. The linked
list has more overhead, is slower, and adds no capability.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17980 from liewegas/wip-boost-url

make-dist,cmake: move boost tarball location to download.ceph.com

Reviewed-by: Jeff Layton <jlayton@redhat.com>

make-dist,cmake: move boost tarball location to download.ceph.com

Sourceforge is down.  Also, we can hammer our servers instead of
theirs.

Signed-off-by: Sage Weil <sage@redhat.com>

include,messages,rbd: Initialize counter,group_pool

Fixes the coverity issues:

** 1396182 Uninitialized scalar field
CID 1396182 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member counter is not initialized
in this constructor nor in any functions that it calls.

** 1396194 Uninitialized scalar field
CID 1396194 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member group_pool is not initialized
in this constructor nor in any functions that it calls

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17589 from myENA/feature/20883-adminapi_sync_stats_master

rgw: admin api - add ability to sync user stats from admin api

Merge pull request #17433 from C2python/check_action

rgw: Return Error if Bucket Policy Contians Undefined Action

Merge pull request #17969 from ceph/rbd-mirror-reboot

systemd: rbd-mirror does not start on reboot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: check device id before using hint

As we might be allocating space from different devices (though
the chance is rare).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BlueFS: drop redundant merge logic of pextents

As Allocator will handle it automatically and efficiently!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BlueFS: kill block_total

As block_all will suffice for the same purpose.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17929 from renhwztetecs/renhw-wip-mgrmon-command_descs_prefix

mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17932 from jcsp/wip-21197

mgr: store declared_types in MgrSession

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17933 from jcsp/wip-17737

mgr: fix crashable DaemonStateIndex::get calls

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17944 from thmour/tmour_upmap

osd/OSDMap: upmap should respect the osd reweights

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17880 from amitkumar50/cov-rgw-9

rgw: Error check on return of read_line()

rbd-mirorr: does not start on reboot

The current systemd unit file misses 'PartOf=ceph-rbd-mirror.target',
which results in the unit not starting after reboot.
If you have ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0, it won't start
after reboot even if enabled.
Adding 'PartOf=ceph-rbd-mirror.target' will enable
ceph-rbd-mirror.target when ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0
gets enabled.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #17961 from tchaikov/wip-doc-local-pool-in-toc

doc/mgr: add "local pool" plugin to toc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17949 from asomers/nproc_freebsd

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17940 from jcsp/wip-21534

mon: update get_store_prefixes implementations

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: silence warnings from -Wsign-compare

Fixed the warnings:

ceph/src/test/bufferlist.cc:2150:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

ceph/src/test/bufferlist.cc:2151:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/mgr: add "local pool" plugin to toc

Signed-off-by: Kefu Chai <kchai@redhat.com>

automake: remove files required by automake

INSTALL, NEWS, README, AUTHORS, and ChangeLog are required by automake,
since we've switched to cmake, remove the empty ones.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17842 from MahatiC/fix_example

examples: fix link order in librados example Makefile

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17633 from yonghengdexin735/wip-zzz-doc

doc/rbd: add info for rbd group

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17803 from amitkumar50/cov-librbd-5

test: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17800 from amitkumar50/cov-misc-8

test/librbd: initialize on_finish,locker,force,snap_id

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Use sysctl(8) instead.  Also, there's no longer any need for /bin/bash
on FreeBSD.

Signed-off-by: Alan Somers <asomers@gmail.com>

Merge pull request #17813 from liewegas/wip-21410-note

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>