librbd: snapshots should be created/removed against data pool

Fixes: http://tracker.ceph.com/issues/21567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17920 from dzafman/wip-21382

Erasure code recovery should send additional reads if necessary

Fixes: http://tracker.ceph.com/issues/21382
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18036 from ownedu/wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under heavy RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17979 from dillaman/wip-21559

rbd-mirror: forced promotion can result in incorrect status

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18035 from tchaikov/wip-bit-cleanup

script/build-integration-branch: python3 compatible and pep8 clean

Merge pull request #17846 from jecluis/wip-21300

mon/MgrMonitor: read cmd descs if empty on update_from_paxos()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: For recovery get all possible shards to read on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Allow recovery to send additional reads

For now it doesn't include non-acting OSDs
Added test for this case

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Allow modified options to existing setup functions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use feature to get last array element

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge branch 'wip-fix-asyncrdma-coredump' of https://github.com/ownedu/ceph into wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under
heavy RDMA traffic, there are chances to access a current_chunk which can
be beyond the range of pre-allocated Tx buffer pool thus causes a coredump

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Fix a potential coredump when handling tx_buffers under heacy RDMA
traffic, there are chances to access a current_chunk which can be beyond the
range of pre-allocated Tx buffer pool thus causes a coredump.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

script/build-integration-branch: python3 compatible and pep8 clean

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17697 into master

* refs/remotes/upstream/pull/17697/head:
pybind/ceph_volume_client: add get, put, and delete object interfaces
pybind/ceph_volume_client: remove 'compat_version'
pybind/ceph_volume_client: set the version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17604 into master

* refs/remotes/upstream/pull/17604/head:
client:fix a bug in lseek

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #16036 into HEAD

* refs/remotes/upstream/pull/16036/head:
mds: improve cap min/max ratio descriptions
mds: fix whitespace
mds: cap client recall to min caps per client
mds: fix conf types
mds: fix whitespace
doc/cephfs: add client min cache and max cache ratio describe
mds: adding tunable features for caps_per_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: improve cap min/max ratio descriptions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cap client recall to min caps per client

Fixes: http://tracker.ceph.com/issues/21575
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17774 from amitkumar50/cov-misc-4

include,messages,rbd: Initialize counter,group_pool

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17820 from zmedico/PGPool-update-optimize-with-subset-of

osd/PGPool::update: optimize with subset_of

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17884 from kungf/stats_invalid_scrub

osd: make scrub right now when pg stats_invalid is true

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17984 from adamemerson/wip-lists-of-pointers-are-bad-in-options

options: Do not use linked lists of pointers!

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17987 from xiexingguo/wip-kick-recovery-for-degraded-obj

osd/PrimaryLogPG: kick off recovery on backoffing a degraded object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17662 from shashalu/fix-set-bucket-versioning

rgw: set bucket versioninig donot change versioning status if missing status in xml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17775 from theanalyst/rgw-keystone-logging

rgw: avoid logging keystone revocation messages when not configured

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17916 from cbodley/wip-21506

rgw: dont reuse stale RGWObjectCtx for get_bucket_info()

Merge pull request #17742 from adamemerson/wip-21389

rgw: Check payment operations in policy

Reviewed-by: Enming Zhang <enming.zhang@umcloud.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18023 from liewegas/wip-changelog

debian/rules: no more ChangeLog

Merge pull request #18002 from adamemerson/wip-21582

rgw: Check bucket GetBucketLocation in policy

Merge pull request #18024 from adamemerson/wip-21597

rgw: Check bucket Website operations in policy

Merge pull request #17839 from liewegas/wip-recovery-preemption

osd: allow PG recovery scheduling preemption

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

debian/rules: no more ChangeLog

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket Website operations in policy

Add code to check s3:GetBucketWebsite and s3:PutBucketWebsite
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21597
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493896
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/rados/singleton/all/recovery-preemption: add test

This mirrors what I was testing locally.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18014 from theanalyst/doc-12.2.1

doc: 12.2.1 release notes

Merge pull request #17973 from ligne/patch-1

doc: Fix typos in placement-groups.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add 12.2.1 to releases table

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.2.1 release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #18000 from adamemerson/wip-21578

rgw: Check bucket CORS operations in policy

Merge pull request #17684 from majianpeng/bluefs-release-extents

os/bluestore/BlueFS: move release unused extents work in _flush_and_syn_log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18020 from theanalyst/doc-changelog-12.2.1

doc: add changelog for 12.2.1

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add changelog for 12.2.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #17769 from optimistyzy/97_new_opt

os/bluestore/NVMEDevice: Remove using dpdk thread

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18001 from liewegas/wip-21417

os/bluestore: put cached attrs in correct mempool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17755 from dillaman/wip-21353

doc/release-notes: ensure RBD users can blacklist prior to upgrade

Reviewed-By: Nathan Cutler <ncutler@suse.com>

os/bluestore: move several buffer{ptr,list}s into cache_other mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put new attrs in correct mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17966 from xiexingguo/wip-bluefs-kill-block-total

os/bluestore/BlueFS: several cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18013 from tchaikov/wip-dashboard-favicon

mgr/dashboard: use rel="icon" for favicon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17993 from jcsp/wip-21572

mgr/dashboard: sort servers and OSDs in OSD list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17937 from tchaikov/wip-automake-cleanup

automake: remove files required by automake

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mgr/dashboard: use rel="icon" for favicon

shortcut is not a standard compliant link type, see
https://www.w3.org/TR/html5/links.html#linkTypes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17991 from jcsp/wip-21570

mgr/dashboard: handle null in format_number

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17986 from tchaikov/wip-doc-rgw-config

doc: fix hyper link to radosgw/config-ref

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17926 from batrick/ptl-tool

scripts: add ptl-tool for scripting merges

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

rgw: Check bucket GetBucketLocation in policy

Add code to check s3:GetBucketLocation against bucket policy.

Fixes: http://tracker.ceph.com/issues/21582
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493934
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore: put attrs in mempool

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add ptr::[try_]reassign_to_mempool

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket CORS operations in policy

Add code to check s3:GetCORS and s3:PutCORS operations against bucket
policy.

Fixes: http://tracker.ceph.com/issues/21578
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/PG: handle racy preemption

If we finish recovery/backfill and go active, but also get
preempted at the same time, we can ignore the event.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow local recovery reservations to be preempted

If a PG has a higher recovery priority and a lower-priority item is in
progress, allow it to be preempted.  This triggers the RecoveryCancel
or BackfillCancel event with a 0 delay, which means it will immediately
re-request a reservation (and presumably wait).

Signed-off-by: Sage Weil <sage@redhat.com>

common/AsyncReserver: support preemption

If an (optional) preemption context is provided, use that to preempt
and existing reservation and grant a higher-priority one.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket versioning operations in policy

Add code to check s3:GetBucketVersioning and s3:PutBucketVersioning
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Check payment operations in policy

Add code to check s3:GetBucketRequestPayment and
s3:PutBucketRequestPayment operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17978 from alram/rbdmap-fix

rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix conf types

This correct an assertion failure.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning

test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: sort servers and OSDs in OSD list

Fixes: http://tracker.ceph.com/issues/21572
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: handle null in format_number

Fixes: http://tracker.ceph.com/issues/21570
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove ceph.com ref to favicon

We have a local copy referenced in the very next line.

Signed-off-by: John Spray <john.spray@redhat.com>

rbd-mirror: ensure forced-failover cannot result in sync state

If the local image already exists and the local image isn't
registered in the remote journal, it should be assumed that
it's already in the replaying state so it can verify the
ancestry to detect split-brains.

Fixes: http://tracker.ceph.com/issues/21559
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: forced-promotion should interrupt replay delay to shut down

Fixes: http://tracker.ceph.com/issues/21559
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/osd_types: kill osd_peer_stat_t

Which currently has no more consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix typos in placement-groups.rst

Signed-off-by: Matt Boyle <matt.boyle@gmail.com>

Merge pull request #17988 from wumingqiao/wip-rados-update

include/rados: fix typo in librados.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: kick off recovery on backoffing a degraded object

As we are now blocking frontend ops from accessing that very object!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

include/rados: fix typo in librados.h

opeprate -> operate

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

doc: fix hyper link to radosgw/config-ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGPool::update: optimize with subset_of

Replace expensive inverval_set intersection_of and operator==
calls with a single subset_of call. I borrowed this idea from
Piotr Dałek's "osd/PGPool: don't use intermediate interval set"
patch. The following benchmark program demonstrates a 38%
performance increase:

#include <iostream>
#include <chrono>
#include "include/interval_set.h"

#define NANOSECONDS(d) \
    std::chrono::duration_cast<std::chrono::nanoseconds>(d).count()

typedef uint64_t snapid_t;
typedef std::chrono::steady_clock::duration duration;

duration PGPool_update_old(const interval_set<snapid_t> &rs) {
  std::chrono::steady_clock::time_point start, end;
  interval_set<snapid_t> newly_removed_snaps, cached_removed_snaps;

  // initialize state
  cached_removed_snaps = rs;

  // start timed simulation
  start = std::chrono::steady_clock::now();

  {
    newly_removed_snaps = cached_removed_snaps;
    interval_set<snapid_t> intersection;
    intersection.intersection_of(newly_removed_snaps, cached_removed_snaps);

    assert(intersection == cached_removed_snaps);
    cached_removed_snaps.swap(newly_removed_snaps);
    newly_removed_snaps = cached_removed_snaps;
    newly_removed_snaps.subtract(intersection);
  }

  // end timed simulation
  end = std::chrono::steady_clock::now();

  return end - start;
}

duration PGPool_update_new(const interval_set<snapid_t> &rs) {
  std::chrono::steady_clock::time_point start, end;
  interval_set<snapid_t> newly_removed_snaps, cached_removed_snaps;

  // initialize state
  cached_removed_snaps = rs;

  // start timed simulation
  start = std::chrono::steady_clock::now();

  {
    newly_removed_snaps = cached_removed_snaps;

    assert(cached_removed_snaps.subset_of(newly_removed_snaps));
    interval_set<snapid_t> removed_snaps = newly_removed_snaps;
    newly_removed_snaps.subtract(cached_removed_snaps);
    cached_removed_snaps.swap(removed_snaps);
  }

  // end timed simulation
  end = std::chrono::steady_clock::now();

  return end - start;
}

int main(int argc, char *argv[])
{
  assert(argc == 3);
  const int sample_count = std::stoi(argv[1]);
  const int interval_count = std::stoi(argv[2]);
  const int interval_distance = 4;
  const int interval_size = 2;
  const int max_offset = interval_count * interval_distance;
  interval_set<snapid_t> removed_snaps;

  for (int i = 0; i < max_offset; i += interval_distance)
    removed_snaps.insert(i, interval_size);

  duration old_delta(0), new_delta(0);

  for (int i = 0; i < sample_count; ++i) {
    old_delta += PGPool_update_old(removed_snaps);
    new_delta += PGPool_update_new(removed_snaps);
  }

  float ratio = float(NANOSECONDS(old_delta)) / NANOSECONDS(new_delta);

  std::cout << ratio << std::endl;
}

Suggested-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
Signed-off-by: Zac Medico <zmedico@gmail.com>

inverval_set: optimize subset_of with sequential search

Optimize subset_of to use sequential search when it
performs better than the lower_bound method, for set
size ratios smaller than 10. This is analogous to
intersection_of behavior since commit 825470fcf919.

The subset_of method can be used in some cases as a
less-expensive alternative to the intersection_of
method, since subset_of can return early if any element
of the smaller set is not contained in the larger set,
and intersection_of has the added burden of storing
the intersecting elements.

Signed-off-by: Zac Medico <zmedico@gmail.com>

rbdmap: fix umount when multiple mounts use the same RBD

When a Kubernetes Pod consumes a RBD it is mounted two
times on the same host. When the host shutdown umount will
fail leading to a hung system

Signed-off-by: Alexandre Marangone <amarango@redhat.com>

Merge pull request #17063 from bspark8/wip-dynamic-cli-info-f

osd: use dmclock library client_info_f function dynamically

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17335 from Carudy/cxw

crush: safe check for 'ceph osd crush swap-bucket'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17410 from zmedico/PGPool-update-optimize-removed_snaps-comparison-when-possible

PGPool::update: optimize removed_snaps comparison when possible

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

common/options.h: Do not use linked lists of pointers

Especially since we're just pushing things in from the end. The linked
list has more overhead, is slower, and adds no capability.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17980 from liewegas/wip-boost-url

make-dist,cmake: move boost tarball location to download.ceph.com

Reviewed-by: Jeff Layton <jlayton@redhat.com>

make-dist,cmake: move boost tarball location to download.ceph.com

Sourceforge is down.  Also, we can hammer our servers instead of
theirs.

Signed-off-by: Sage Weil <sage@redhat.com>

include,messages,rbd: Initialize counter,group_pool

Fixes the coverity issues:

** 1396182 Uninitialized scalar field
CID 1396182 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member counter is not initialized
in this constructor nor in any functions that it calls.

** 1396194 Uninitialized scalar field
CID 1396194 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member group_pool is not initialized
in this constructor nor in any functions that it calls

Signed-off-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17589 from myENA/feature/20883-adminapi_sync_stats_master

rgw: admin api - add ability to sync user stats from admin api

Merge pull request #17433 from C2python/check_action

rgw: Return Error if Bucket Policy Contians Undefined Action

Merge pull request #17969 from ceph/rbd-mirror-reboot

systemd: rbd-mirror does not start on reboot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: check device id before using hint

As we might be allocating space from different devices (though
the chance is rare).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BlueFS: drop redundant merge logic of pextents

As Allocator will handle it automatically and efficiently!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BlueFS: kill block_total

As block_all will suffice for the same purpose.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17929 from renhwztetecs/renhw-wip-mgrmon-command_descs_prefix

mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17932 from jcsp/wip-21197

mgr: store declared_types in MgrSession

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17933 from jcsp/wip-17737

mgr: fix crashable DaemonStateIndex::get calls

Reviewed-by: Kefu Chai <kchai@redhat.com>