Merge pull request #20034 from linuxbox2/wip-rgw-altlatchfix

rgw_file: alternate fix deadlock on lru eviction

Merge pull request #20030 from tchaikov/wip-doc-cxx11

doc/dev/cxx: add C++11 ABI related doc

Merge pull request #19708 from qrGitHub/wip-rgw-unused-variable4

rgw: drop unused variable bucket_instance_ids

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18717 from shinobu-x/sk_mgr_daemonserver_using_get_val

mgr/DaemonServer.cc: [Cleanup] Change to using  get_val template function

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19362 from myoungwon/wip-manifest-rados-op-promote

osd, librados: add a rados op (TIER_PROMOTE)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20045 from shangzhong/doc-cephfs-multimds

doc: redundant "cephfs" when set the "allow_multimds"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19744 from jan--f/mgr-prometheus-standby-mondown

pybind/mgr/prometheus: add StandbyModule and handle failed MON cluster

Reviewed-by: John Spray <john.spray@redhat.com>

qa/tasks/mgr: add test_prometheus; smoke tests for prometheus module

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19965 from ovh/bp-reorder-mimic-fields

osd_types.cc: reorder fields in serialized pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19963 from runsisi/wip-cmake

cmake: do not find bzip2/lz4 for rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19944 from badone/wip-better-prepare_command_pool_set-E2BIG-error-message

mon/OSDMonitor: Better prepare_command_pool_set E2BIG error message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19934 from david-z/wip-rocksdb-dump-mem

kv/RocksDB: get index and filter blocks memory usage

Reviewed-by: Sage Weil <sage@redhat.com>

doc: redundant "cephfs" when set the "allow_multimds"

Signed-off-by: Shangzhong Zhu <zhu.shangzhong@zte.com.cn>

Merge pull request #19948 from tchaikov/wip-standby-module-get-config

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_con…

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/mgr_module: add default param for MgrStandbyModule.get_config()

this matches its counterpart of MgrModule.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19863 from renhwztetecs/renhw-wip-common-time

common/time: add time.h for Alpine build

Reviewed-by: John Coyle <dx9err@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19899 from smithfarm/wip-rip-rcceph

build/ops: rpm: rip out rcceph script

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19998 from liewegas/wip-22715

fix broken use of streamstream::rdbuf()

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw_file: alternate fix deadlock on lru eviction

This change is an alternate fix for two problems found and fixed
by Yao Zongyou <yaozongyou@vip.qq.com>.

The deadlock can be avoided just by not taking it in the recycle
case, which invariantly holds the lock.

The invalidation of the insert iterator by the recyle-path unlink
we'd like to handle as a condition in order to preserve the cached
insertion point optimization we get in the common case.  (The
original behavior was, indeed, incorrect.)

Based on feedback from Yao, removed the RGWFileHandle dtor version
of the unlink check, which I think happened twice.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #19597 from joscollin/wip-lspools-output

mon: align lspools output

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #19936 from linuxbox2/wip-rgw-22121

rgw: URL-decode S3 and Swift object-copy URLs

Merge pull request #20019 from zhouyuan/fix_librbdpy_example

doc: fix librbdpy example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

doc/dev/cxx: add C++11 ABI related doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19577 from vshankar/wip-policy-state-machine-rollback

rbd-mirror: rollback state transitions in image policy

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19821 from trociny/wip-nbd-global-options

rbd: properly pass ceph global command line args to subprocess

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19912 from adamemerson/wip-minimax-rbd

librbd: prefer templates to macros

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19937 from trociny/wip-rbd-showmapped

rbd: adjusted "showmapped" JSON and XML formatted output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19972 from liewegas/wip-mgr-rados

pybind/mgr/mgr_module: make rados handle available to all modules

Reviewed-by: John Spray <john.spray@redhat.com>

doc: fix librbdpy example

need to connect cluster before opening ioctx

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

rgw: URL-decode S3 and Swift object-copy URLs

Fixes: http://tracker.ceph.com/issues/22121
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #20011 into master

* refs/pull/20011/head:
mds: fix some typo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18971 from liewegas/wip-pg-scrub-preempt

osd/PG: allow scrub preemption

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19476 from qrGitHub/wip-rgw-lack-timeSkewCheck

rgw: add time skew check in function parse_v4_auth_header

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19826 from joke-lee/rgw_fix_s3website_redirect_location_string_length

rgw: fix s3website redirect location string length

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19918 from fangyuxiangGL/lc-omap

rgw: abort multipart if upload meta object doesn't exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20003 from wjwithagen/fix-PG-protected

common: change routines to public access

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: squelch 'id' arg errors

Several commands use 'id' but for other types (e.g., vector<int64_t>).

Signed-off-by: Sage Weil <sage@redhat.com>

use stringsream::str(), not rdbuf()

Fixes: http://tracker.ceph.com/issues/22715
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19949 from ceph/wip-bz1498521

ceph-volume: adds --crush-device-class flag for lvm prepare and create

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19997 from Songweibin/wip-fix-typo

doc/rbd: fix typo s/morror/mirror

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19981 from k0ste/dashboard_iops

mgr: dashboard: show per pool IOPS on health page (#22495).

Reviewed-by: John Spray <john.spray@redhat.com>

mgr: dashboard: show per pool IOPS on health page (#22495).

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

common: change routines to public access

Clang complains tafter recent changes:
/home/jenkins/workspace/ceph-master/src/osd/OSD.cc:8787:8: error: 'DeleteStart' is a protected member of 'PG'
          PG::DeleteStart())));
              ^
/home/jenkins/workspace/ceph-master/src/osd/PG.h:1870:16: note: declared protected here
  TrivialEvent(DeleteStart)
               ^
/home/jenkins/workspace/ceph-master/src/osd/OSD.cc:9136:6: error: 'DeleteSome' is a protected member of 'PG'
        PG::DeleteSome())),
            ^
/home/jenkins/workspace/ceph-master/src/osd/PG.h:1871:16: note: declared protected here
  TrivialEvent(DeleteSome)
               ^
2 errors generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/rbd: fix typo s/morror/mirror

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #19931 from mikechristie/wip-doc-mnc-iscsi-updates

Misc iSCSI doc updates

Reviewed-by: John Wilkins <jowilkin@redhat.com>

ceph-volume: print the lvm activate success message correctly

Move the terminal.success to the methods that have access to the
correct osd_id so that they do not print None for the osd_id
in the log message.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17520 from joscollin/wip-cleanup-drop-return

rgw: Drop unnecessary return

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: properly pass ceph global command line args to subprocess

When initializing the global context, global_init parses command line
arguments and removes ceph global args from the list. As a result they
are not visible for rbd command line parser and global options were
ignored when passing them to subprocesses like nbd or ggate.

Fix this by keeping a list of ceph global init command line
arguments. To build the list we compare the list of initial command
line args and those that remain after global_init parsing.

After this fix it is possible to pass any ceph global option to a
subprocess.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #19803 into master

* refs/pull/19803/head:
mds: get rid of unnecessary if condition

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19837 into master

* refs/pull/19837/head:
client: avoid recursive lock in ll_get_vino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge PR #19860 into master

* refs/pull/19860/head:
cephfs-journal-tool: add usage help info for header set <field>
mds: add error log info
cephfs-journal-tool: add "set pool_id" option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19891 into master

* refs/pull/19891/head:
mds: handle 'inode gets queued for recovery multiple times'
mds: use elist to track recover queue items

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19911 into master

* refs/pull/19911/head:
cephfs: Switch MIN/MAX for std::min/max and use intarith templates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PG: drop 'seed' property from Scrubber

This has been -1 for many releases now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/divergent_priors*: unsquelch osd debug

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: debug ec scrub error paths

Signed-off-by: Sage Weil <sage@redhat.com>

osd: document scrub options

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow limited scrub preemption

If we receive a write within the scrub range, abort the scrub chunk and
shrink the chunk size.  If we do this too many times do not preempt and
allow the scrub to complete (to avoid scrub starvation due to client io).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19876 from theanalyst/build/do-cmake-args

build: do_cmake: allow ARGS to be overridden

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19433 from liewegas/wip-pg-removal

osd: put pg removal in op_wq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

pybind/mgr/mgr_module: make rados handle available to all modules

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: do on_shutdown on removal

This cleans up our reservations and misc other state in OSDService that
needs to be cleaned up.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track deleted pools' pg_nums for calculating split

This is needed to determine (quickly) whether PGs have split.  Calling
get_pg_num() on the latest map does not work when the pool has been
deleted from that map.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19850 from dzafman/wip-calc-stats

osd/PG: re-write of _update_calc_stats and improve pg degraded state

Fixes: http://tracker.ceph.com/issues/20059
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: recovery-unfound-found test needs to account for correct misplaced calculations

The test expected HEALTH_OK when in a state with misplaced objects therefore HEALTH_WARN

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: add rbd pool requirement for gwcli

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: add daemon-reload for rbd-target-api setup

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: add gwcli manual install instructions

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge pull request #19613 from qrGitHub/wip-rgw-optimize-bucketLCPrepare

rgw: optimize function bucket_lc_prepare

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19878 from cbodley/wip-rgw-get-omap-keys-cr

rgw: fix handling of ENOENT in RGWRadosGetOmapKeysCR

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19897 from cbodley/wip-rgw-put-system-obj

rgw: rgw_put_system_obj takes bufferlist

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume: docs for --crush-device-class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds crush_device_class to json metadata for 'osd new'

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds crush_device_class to the functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add crush_device_class to lvm metadata

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a --crush-device-class flag to lvm create and prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19945 from tchaikov/wip-deprecated-in-cxx17

librbd,librados: do not include stdbool.h in C++ headers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19261 from dillaman/wip-22286

mgr: disconnect unregistered service daemon when report received

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #19942 from shinobu-x/wip-sk-minmax-ms_async_op_threads

common: Add min/max of ms_async_op_threads

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19397 from xxhdx1985126/master

mon: update PaxosService::cached_first_committed in PaxosService::maybe_trim()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: piecewise scrub

Perform scrub in stages, with each unit of work requeuing an item in the
work queue.

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: do not find bzip2/lz4 for rocksdb

RocksDB has compression disabled by default[1] and it's never been enabled[2],
so do not link these compression libraries implicitly

[1] https://github.com/ceph/rocksdb/blob/master/CMakeLists.txt#L76
[2] https://github.com/ceph/ceph/blob/master/cmake/modules/BuildRocksDB.cmake

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd_types.cc: don't store 32 least siognificant bits of state twice

a25221e55172c326b48dea6f08c16d700921864e introduced a new, 64bit
pg state field, retaining encoding and decoding of least significant
32 bits for compatibility reasons.
This commit also adds encoding of entire 64bit state field, meaning
that the least significant bits are encoded and decoded twice.
Fix this by encoding each half of 64bit field separately.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

osd_types.cc: reorder encoding/decoding of fields in pg_stat_t

Fields state, purged_snaps and snaptrimq_len are new to Mimic.
Reorder them in a way that newest field (snaptrimq_len) is before
two others and uses other encoding version, so the pull request
https://github.com/ceph/ceph/pull/19520 can be backported to Luminous
without breaking Luminous -> Mimic upgrade.
This also changes encoding/decoding version back to 24 as both state
and purged snaps were added post-Luminous and pre-Mimic, so we can
push it into a single struct version and keep snaptrimq_len into other
version.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

doc: separate gwcli install from post install setup steps

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: add note to clarify trusted_ip_list use

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: update esx instructions

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge pull request #19822 from trociny/wip-ggate-ceph-globals

rbd-ggate: fix parsing ceph global options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19673 from zouaiguo/wip_erase_code

mon/OSDMonitor.cc : set erasure-code-profile to "" when create replicated pools.

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: mention rcceph drop in PendingReleaseNotes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: rip out rcceph script

"rcceph" is a SysVinit-style command-line interface for stopping, starting,
enabling, etc. all ceph-osd and ceph-mon systemd units on a machine, in one go.

Since the same functionality is provided by ceph-{osd,mon}.target, the script
is obsolete. It is also unmaintained. Judging from the absence of recent
mentions of the script online, I guess it is no longer used.

Leaving dead code in the tree can cause confusion, especially when the code is
packaged and shipped to customers. Therefore I propose to rip it out.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

common/time: add time.h for Alpine build

We create the Alpine apk package to view "https://github.com/ceph/ceph/blob/master/README.alpine.md",
at compile time found the following error:
"/src/common/ceph_time.h:209:4: error: invalid use of incomplete type 'struct ceph :: time_detail :: timeval'.

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #19896 from tchaikov/wip-vive-cxx-17

msg/simple: s/ceph::size/std::size/

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19870 from liewegas/wip-strip-ceph-base

debian/rules: strip ceph-base libraries

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18755 from aclamk/wip-reverted-approx-size

osd/PG: revert approx size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18191 from hjwsm1989/osd-mark-down

qa/standalone/osd/osd-mark-down: create pool to get updated osdmap faster

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19877 from adamemerson/wip-dencish

denc cleanups and other fixes

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Verify stat calculations during backfill

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Verify stat calculations during recovery

Signed-off-by: David Zafman <dzafman@redhat.com>