os/bluestore: use normal Context for async deferred_try_submit

I'm not quite sure why the FunctionContext did not ever execute on the
finisher thread (perhaps the [&] captured some state on the stack that it
shouldn't have?).  In any case, using a traditional Context here appears
to resolve the problem (of the async deferred_try_submit() never executing,
leading to a bluestore stall/deadlock).

Fixes: http://tracker.ceph.com/issues/21470
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wake kv thread when blocking on deferred_bytes

We need to wake the kv thread whenever setting deferred_aggressive to
ensure that txns with deferred io that have committed but haven't submitted
their deferred writes get submitted.  This aligns us with the other
users of deferred_aggressive (e.g., _osr_drain_all).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18023 from liewegas/wip-changelog

debian/rules: no more ChangeLog

Merge pull request #18002 from adamemerson/wip-21582

rgw: Check bucket GetBucketLocation in policy

Merge pull request #18024 from adamemerson/wip-21597

rgw: Check bucket Website operations in policy

Merge pull request #17839 from liewegas/wip-recovery-preemption

osd: allow PG recovery scheduling preemption

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

debian/rules: no more ChangeLog

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket Website operations in policy

Add code to check s3:GetBucketWebsite and s3:PutBucketWebsite
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21597
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493896
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/rados/singleton/all/recovery-preemption: add test

This mirrors what I was testing locally.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18014 from theanalyst/doc-12.2.1

doc: 12.2.1 release notes

Merge pull request #17973 from ligne/patch-1

doc: Fix typos in placement-groups.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add 12.2.1 to releases table

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.2.1 release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #18000 from adamemerson/wip-21578

rgw: Check bucket CORS operations in policy

Merge pull request #17684 from majianpeng/bluefs-release-extents

os/bluestore/BlueFS: move release unused extents work in _flush_and_syn_log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18020 from theanalyst/doc-changelog-12.2.1

doc: add changelog for 12.2.1

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add changelog for 12.2.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #17769 from optimistyzy/97_new_opt

os/bluestore/NVMEDevice: Remove using dpdk thread

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18001 from liewegas/wip-21417

os/bluestore: put cached attrs in correct mempool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17755 from dillaman/wip-21353

doc/release-notes: ensure RBD users can blacklist prior to upgrade

Reviewed-By: Nathan Cutler <ncutler@suse.com>

os/bluestore: move several buffer{ptr,list}s into cache_other mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put new attrs in correct mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17966 from xiexingguo/wip-bluefs-kill-block-total

os/bluestore/BlueFS: several cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18013 from tchaikov/wip-dashboard-favicon

mgr/dashboard: use rel="icon" for favicon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17993 from jcsp/wip-21572

mgr/dashboard: sort servers and OSDs in OSD list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17937 from tchaikov/wip-automake-cleanup

automake: remove files required by automake

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mgr/dashboard: use rel="icon" for favicon

shortcut is not a standard compliant link type, see
https://www.w3.org/TR/html5/links.html#linkTypes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17991 from jcsp/wip-21570

mgr/dashboard: handle null in format_number

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17986 from tchaikov/wip-doc-rgw-config

doc: fix hyper link to radosgw/config-ref

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17926 from batrick/ptl-tool

scripts: add ptl-tool for scripting merges

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

rgw: Check bucket GetBucketLocation in policy

Add code to check s3:GetBucketLocation against bucket policy.

Fixes: http://tracker.ceph.com/issues/21582
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493934
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore: put attrs in mempool

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add ptr::[try_]reassign_to_mempool

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket CORS operations in policy

Add code to check s3:GetCORS and s3:PutCORS operations against bucket
policy.

Fixes: http://tracker.ceph.com/issues/21578
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/PG: handle racy preemption

If we finish recovery/backfill and go active, but also get
preempted at the same time, we can ignore the event.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow local recovery reservations to be preempted

If a PG has a higher recovery priority and a lower-priority item is in
progress, allow it to be preempted.  This triggers the RecoveryCancel
or BackfillCancel event with a 0 delay, which means it will immediately
re-request a reservation (and presumably wait).

Signed-off-by: Sage Weil <sage@redhat.com>

common/AsyncReserver: support preemption

If an (optional) preemption context is provided, use that to preempt
and existing reservation and grant a higher-priority one.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17978 from alram/rbdmap-fix

rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning

test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: sort servers and OSDs in OSD list

Fixes: http://tracker.ceph.com/issues/21572
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: handle null in format_number

Fixes: http://tracker.ceph.com/issues/21570
Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove ceph.com ref to favicon

We have a local copy referenced in the very next line.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: Fix typos in placement-groups.rst

Signed-off-by: Matt Boyle <matt.boyle@gmail.com>

Merge pull request #17988 from wumingqiao/wip-rados-update

include/rados: fix typo in librados.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

include/rados: fix typo in librados.h

opeprate -> operate

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

doc: fix hyper link to radosgw/config-ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbdmap: fix umount when multiple mounts use the same RBD

When a Kubernetes Pod consumes a RBD it is mounted two
times on the same host. When the host shutdown umount will
fail leading to a hung system

Signed-off-by: Alexandre Marangone <amarango@redhat.com>

Merge pull request #17063 from bspark8/wip-dynamic-cli-info-f

osd: use dmclock library client_info_f function dynamically

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17335 from Carudy/cxw

crush: safe check for 'ceph osd crush swap-bucket'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17410 from zmedico/PGPool-update-optimize-removed_snaps-comparison-when-possible

PGPool::update: optimize removed_snaps comparison when possible

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #17980 from liewegas/wip-boost-url

make-dist,cmake: move boost tarball location to download.ceph.com

Reviewed-by: Jeff Layton <jlayton@redhat.com>

make-dist,cmake: move boost tarball location to download.ceph.com

Sourceforge is down.  Also, we can hammer our servers instead of
theirs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17589 from myENA/feature/20883-adminapi_sync_stats_master

rgw: admin api - add ability to sync user stats from admin api

Merge pull request #17433 from C2python/check_action

rgw: Return Error if Bucket Policy Contians Undefined Action

Merge pull request #17969 from ceph/rbd-mirror-reboot

systemd: rbd-mirror does not start on reboot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/BlueFS: check device id before using hint

As we might be allocating space from different devices (though
the chance is rare).

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BlueFS: drop redundant merge logic of pextents

As Allocator will handle it automatically and efficiently!

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BlueFS: kill block_total

As block_all will suffice for the same purpose.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17929 from renhwztetecs/renhw-wip-mgrmon-command_descs_prefix

mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17932 from jcsp/wip-21197

mgr: store declared_types in MgrSession

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17933 from jcsp/wip-17737

mgr: fix crashable DaemonStateIndex::get calls

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17944 from thmour/tmour_upmap

osd/OSDMap: upmap should respect the osd reweights

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17880 from amitkumar50/cov-rgw-9

rgw: Error check on return of read_line()

rbd-mirorr: does not start on reboot

The current systemd unit file misses 'PartOf=ceph-rbd-mirror.target',
which results in the unit not starting after reboot.
If you have ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0, it won't start
after reboot even if enabled.
Adding 'PartOf=ceph-rbd-mirror.target' will enable
ceph-rbd-mirror.target when ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0
gets enabled.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #17961 from tchaikov/wip-doc-local-pool-in-toc

doc/mgr: add "local pool" plugin to toc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17949 from asomers/nproc_freebsd

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17940 from jcsp/wip-21534

mon: update get_store_prefixes implementations

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: silence warnings from -Wsign-compare

Fixed the warnings:

ceph/src/test/bufferlist.cc:2150:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

ceph/src/test/bufferlist.cc:2151:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/mgr: add "local pool" plugin to toc

Signed-off-by: Kefu Chai <kchai@redhat.com>

automake: remove files required by automake

INSTALL, NEWS, README, AUTHORS, and ChangeLog are required by automake,
since we've switched to cmake, remove the empty ones.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17842 from MahatiC/fix_example

examples: fix link order in librados example Makefile

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17633 from yonghengdexin735/wip-zzz-doc

doc/rbd: add info for rbd group

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17803 from amitkumar50/cov-librbd-5

test: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17800 from amitkumar50/cov-misc-8

test/librbd: initialize on_finish,locker,force,snap_id

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Use sysctl(8) instead.  Also, there's no longer any need for /bin/bash
on FreeBSD.

Signed-off-by: Alan Somers <asomers@gmail.com>

Merge pull request #17813 from liewegas/wip-21410-note

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17809 from yaozongyou/fix-rgw-file-write-error

rgw_file: fix write error when the write offset overlaps

Merge pull request #17947 from batrick/ceph-osd-core-dbg

ceph-debug-docker.sh: add ceph-osd-dbg package

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-debug-docker.sh: add ceph-osd-dbg package

So we can debug ceph-osd cores.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: upmap should respect osd reweights

Modify OSDMap::calc_pg_upmaps to take the osd reweight into account when
computing the size of each OSD.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
Fixes: http://tracker.ceph.com/issues/21538

test/cli/osdmaptool: test upmap with an out osd

Test upmap with one osd marked out.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #17810 from hjwsm1989/wip-21294

qa/ceph_manager: check pg state again before timedout

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17661 from yunfeiguan/master

librados: Fix a potential risk of buffer::list::claim_prepend(list& b…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17790 from joscollin/wip-test-warning-signed-unsigned

test: silence warning from -Wsign-compare

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17928 from xiexingguo/wip-more-interval-set-size

osd/PrimaryLogPG: calc clone_overlap size in a more efficient and concise way

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17528 from liewegas/wip-mgr-localpool

pybind/mgr/localpool: module to automagically create localized pools

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/mgr/test_localpool: simple test for localpool mode

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: add --mark-out option

Add a --mark-down <id> option which sets the reweight for a given
osd id to 0.0.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

examples: fix link order in librados example Makefile

The library link order in librados example Makefile is incorrect
and as a result 'make' throws an error. This change fixes the order.

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

mon/mgr: sync mgr_command_descs table and mgr_metadata table

sync mgr_command_descs table and mgr_metadata table to new mons

Fixes: http://tracker.ceph.com/issues/21527
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon/osd_metadata: sync osd_metadata table

sync osd_metadata table to new mons when add new mons

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mgr: fix crashable DaemonStateIndex::get calls

This function was recently fixed to return null
on missing entries: handle that properly.

Fixes: http://tracker.ceph.com/issues/17737
Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement MDSMonitor::get_store_prefixes

Fixes: http://tracker.ceph.com/issues/21534
Signed-off-by: John Spray <john.spray@redhat.com>

mon/OSDMonitor: tidy prefix definitions

We should define them in one place to make it easy
when updating get_store_prefixes.

Fixes: http://tracker.ceph.com/issues/21534
Signed-off-by: John Spray <john.spray@redhat.com>

mon/osd_metadata: sync osd_metadata table

sync osd_metadata table to new mons when add new mons

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #17831 from badone/wip-require-osd-release-with-no-up-osds

mon: osd feature checks with 0 up osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17848 from liewegas/wip-ruleset-errors

mon/OSDMonitor: error out if setting ruleset-* ec profile property

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #17902 from tchaikov/wip-21474

qa/tasks: prolong revive_osd() timeout to 6 min

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17903 from lixiaoy1/add_func_log

os/bluestore: Add missing __func__ in dout

Reviewed-by: Jos Collin <jcollin@redhat.com>