tools: Add pg log trim command to ceph-objectstore-tool

Add test script that verifies the command in qa/standalone/osd

Fixes: http://tracker.ceph.com/issues/23242
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20648 from trociny/wip-snap_set_diff-empty_snapset

librados/snap_set_diff: don't assert on empty snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20613 from Songweibin/wip-23038

rbd: import with option --export-format 2 fails to protect snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20608 from Songweibin/wip-rbd-snap-purge

rbd: allow remove all unprotected snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20578 from trociny/wip-pybind-group

librbd: small cleanup for recently merged code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20621 from cbodley/wip-rgw-data_lock

rgw: don't hold data_lock over frontend io

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20449 from cbodley/wip-rgw-beast-coroutine

rgw: switch beast frontend back to stackful coroutine

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #20546 from yehudasa/wip-rgw-gc-aio

rgw: gc use aio

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #20643 into master

* refs/pull/20643/head:
ceph-volume-client: fix command argument order

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #20650 from ceph/wip-rm22841

ceph-volume be resilient to $PATH issues

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20656 from yaozongyou/rbd-fix-return

librbd: fix missing return in NotifyMessage::get_notify_op

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: allow remove all unprotected snapshots

allow remove all unprotected snapshots when exiting
protected snapshots in the same image.
Fixes: http://tracker.ceph.com/issues/23126
Signed-off-by: songweibin <song.weibin@zte.com.cn>

rbd: import with option --export-format fails to protect snapshot

Fixes: http://tracker.ceph.com/issues/23038
Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20640 from tchaikov/wip-async-dpdk-temp-str

msg/async: avoid referencing the temporary string

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20531 from xiexingguo/wip-ec-scrub-sleep

osd/ECBackend: inject sleep during deep scrub

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20630 from joscollin/wip-doc-monitoring

doc: Update monitoring.rst

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

doc: Update monitoring.rst

Updated monitoring.rst with
* Latest command output
* misc doc fixes

Signed-off-by: Jos Collin <jcollin@redhat.com>

rbd: fix return is missing in NotifyMessage::get_notify_op

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-volume process always try to expand the executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure behavior of new which utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system add a helper to find system executables

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20646 from tchaikov/wip-qa/run-standalone.sh/python-path

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20634 from tchaikov/wip-delete-pool-in-standalone-test

qa/standalone: extract delete_pool()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20619 from theanalyst/doc/release-notes/12.2.4

doc: 12.2.4 release notes

Reviewed-By: Alfredo Deza <adeza@redhat.com>
Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>

librados/snap_set_diff: don't assert on empty snapset

Instead treat the diff as a full-object delta.

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

do not set PYTHONPATH=/usr/lib/python2.7/dist-packages anymore, it will
be overridden later on anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19015 from hansbogert/master

mon: Add `ceph osd get-require-min-compat-client` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #20607 from shangfufei/test_stack

msg/async:fix the incoming parameter type of EventCenter::process_events()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20509 from liewegas/wip-less-filestore-idempotent

qa/tasks/filestore_idempotent: shorter test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20498 from liewegas/wip-22616

os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20616 from tchaikov/wip-cmake

cmake/BuildBoost: fixes to ready seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20636 from tchaikov/wip-23169

Revert "core: hint the dout()'s message crafting as a cold code."

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #20631 from tchaikov/wip-cmake-systemd

cmake: s/sysconf/sysconfig/

Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-volume-client: fix command argument order

Fixes: http://tracker.ceph.com/issues/22038
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg/async: avoid referencing the temporary string

* get_val<std::string>(...) returns a temporary std::string, so we cannot
  keep a reference to it after evaluating this method. so convert it to
  an integer right away in the same expression.
* use std::stoull() with base = 16, so we can parse a hex string
  representing up to 64 bits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15849 from theanalyst/wip-doc-es

doc: rgw add some basic documentation for sync plugins & ES

Reviewed-By: Yehuda Sadeh <yehuda@redhat.com>

doc: 12.2.4 release notes

And updating the release table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Revert "core: hint the dout()'s message crafting as a cold code."

This reverts commit b265ed2955570c17b954270e6a1449637790e9b8.

with b265ed29, we cannot use `dout(10) << __func__ << ...` anymore for
logging the name of the inner most function. what we will have is
always `operator()`, because the lambda closure will be the inner most
function with the the hinting change.

Fixes: http://tracker.ceph.com/issues/23169
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: extract delete_pool()

some tests, like osd-backfill-stats.sh are using delete_pool(), but
they don't have this function defined. and this function is defined
in standalone tests separately, so would be simpler if we can
consolidate them in ceph-helper.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20411 from majianpeng/sync-nvml-submodule

cmake: sync nvml submodule to latest code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: make PMEMDevice work based on latest pmdk.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: sync nvml submodule

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: s/sysconf/sysconfig/

it's a regression caused by 638aadf

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20618 from tchaikov/wip-systemd-deb-env

cmake,deb: install system units using cmake

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20602 from gregsfortytwo/wip-23141-rotational

os/bluestore: correctly check all block devices to decide if journal is_rotational

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20595 from jcsp/wip-23037

mgr: die on bind() failure

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20593 from tchaikov/wip-23121

ceph-objectstore-tool: don't destroy SnapMapper until the txn is completed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20540 from ovh/bp-optracker-cleanup

common/OpHistory: move insert/cleanup into separate thread

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20539 from zeichenanonym/mgr-prometheus-fix-metadata-does-not-exist

mgr: prometheus: Don't crash on OSDs without metadata

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20530 from wido/issue-23078

common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20394 from liewegas/wip-pg-log-size

osd: min_pg_log_entries == max == pg_log_dups_tracked

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20290 from rzarzynski/wip-common-subsystemmap

core: make the main dout() paths faster and more maintanable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20273 from adamemerson/wip-any-singleton-will-do

common: Switch singletons to use immobile_any and cleanups

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20528 from tchaikov/wip-22462

mon: add MMonHealth back

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20415 from wumingqiao/wip-fallocate

filestore: include <linux/falloc.h>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20620 from ceph/wip-rm23140

ceph-volume: adds raw device support to 'lvm list'

Reviewed-by: Alfredo Deza <adeza@redhat.com>

librbd: normalization for group C API

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20599 from dillaman/wip-librbd-devstack-tempest

qa/workunits/rbd: switch devstack tempest to 17.2.0 tag

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: don't hold data_lock over frontend io

get_obj_data::data_lock protects access to the read_list, which the
librados finisher thread appends to on completions

locking isn't needed over calls to d->client_cb->handle_data() in
get_obj_iterate_cb(), because this callback is called synchronously
from the frontend/process_request() thread - no other thread will
be submitting io to the d->client_cb callback

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20612 from fangyuxiangGL/com

rgw: download object might fail for local invariable uninitialized

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19845 from fangyuxiangGL/escape

rgw: rework json/xml escape usage follow #19806

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20352 from yehudasa/wip-22928

rgw: use explicit index pool placement

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20440 from cbodley/wip-qa-rgw-beast

qa/rgw: remove some civetweb overrides for beast testing

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20471 from mdw-at-linuxbox/wip-master-swacldel

rgw: Allow swift acls to be deleted.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: download object might fail for local invariable uninitialized

This is an omission in #20226

Fixes: http://tracker.ceph.com/issues/23146
Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #20234 from smithfarm/wip-run-make-check

build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake,deb: set EnvironmentFile using cmake

this change also fix the EnvironmentFile specified in rbdmap.service.
without this change EnvironmentFile in rbdmap.service is always
/etc/sysconfig/ceph even on debian derived distros. after this change,
this variable is /etc/default/ceph in rbdmap.service shipped by the deb
packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: install system units using cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: if using lvm list on a device with many lvs, show them all

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20436 from smithfarm/wip-22999

install-deps.sh: set python2 %bcond by environment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

librbd: fix exit tracepoint leaks

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/librbd: add empty directory rbd_list test

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: leaking image close if remove failed due to operations disabled

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/rbd: pool id in rbd_group_image_info_t should be int64_t

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20611 from Leeshine/wip-rgw-22080-1

rgw: create sync-module instance when radosgw-admin sync run

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20397 from tdb/fix/perms-ocf-rbd

rbd: resource agent needs to be executable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20577 from Songweibin/wip-group-rename

rbd: add group rename methods

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: add implicit dependencies between boost libs

so the dependencies can be included when linking against the one which
depends on them. for example, libboost_filesystem depends on
libboost_system, if we don't link against the latter, linker will bail
out when linking an executable using Boost::filesystem.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add an alias for unit_test_framework

should pass --with-test for building Boost::unit_test_framework

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: skip boost dependency on ALIAS targets

the boost override for add_library() does not work for ALIAS targets:

CMake Error at cmake/modules/BuildBoost.cmake:227 (add_dependencies):
  Cannot add target-level dependencies to alias target "Seastar::seastar".

Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps.sh: set python2 %bcond by environment

If there is no /usr/bin/python2 we need to set %bcond_with python2.

Fixes: http://tracker.ceph.com/issues/22999
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20434 from smithfarm/wip-22998

build/ops: install-deps.sh: pass --no-recommends to zypper

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20404 from smithfarm/wip-cmake-with-systemd

build/ops: change WITH_SYSTEMD default to ON

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

build/ops: run-make-check.sh: fix SUSE support

Get /usr/bin/which from the right RPM.

Fixes: http://tracker.ceph.com/issues/22875
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20245 from smithfarm/wip-p2p-ripout

tests: cleanup: drop upgrade/jewel-x/point-to-point-x

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20138 from smithfarm/wip-recommend-chrony

build/ops: rpm: recommend chrony instead of ntp-daemon

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #20551 from shun-s/wip-add-notify-name-print

rbd: output notifyOp request name when watching

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

log: prohibit negative values for debug_* configurables.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

log: remove SubsystemMap::m_max_name_len.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph: switch to SubsystemMap::should_gather<SubV, LvlV>().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: resurrect the old check on dout's levels where possible.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: hint the dout()'s message crafting as a cold code.

The idea is to:
  1. Do not put the dout()'s crafting stuff on the hot, fall-through
     path. Cheapest branches are those that are forward and never taken.
  2. Move it to separated sections placed far away from the main path
     to be more friendly to ICache and ITLB. That is, dout_impl constructs
     a function now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: make SubsystemMap more statical and optimize should_gather().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: remove the PGLog::prefix_provider as it's always nullptr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: move subsys-dependant types to a dedicated header.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: dout checks ceph_subsys_* in compile time if possible.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #19471 into master

* refs/pull/19471/head:
cephfs-journal-tool: support purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20412 into master

* refs/pull/20412/head:
mds/CDir: cleanup in the lookup_exact_snap()
mds/cdir: add CDir missing state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20468 into master

* refs/pull/20468/head:
mds: fix check of underwater dentries

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20489 into master

* refs/pull/20489/head:
common: omit short option for id in help for clients

Reviewed-by: Jos Collin <jcollin@redhat.com>