rgw: URL-decode S3 and Swift object-copy URLs

Fixes: http://tracker.ceph.com/issues/22121
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #20011 into master

* refs/pull/20011/head:
mds: fix some typo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18971 from liewegas/wip-pg-scrub-preempt

osd/PG: allow scrub preemption

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19476 from qrGitHub/wip-rgw-lack-timeSkewCheck

rgw: add time skew check in function parse_v4_auth_header

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19826 from joke-lee/rgw_fix_s3website_redirect_location_string_length

rgw: fix s3website redirect location string length

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19918 from fangyuxiangGL/lc-omap

rgw: abort multipart if upload meta object doesn't exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20003 from wjwithagen/fix-PG-protected

common: change routines to public access

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19949 from ceph/wip-bz1498521

ceph-volume: adds --crush-device-class flag for lvm prepare and create

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19997 from Songweibin/wip-fix-typo

doc/rbd: fix typo s/morror/mirror

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19981 from k0ste/dashboard_iops

mgr: dashboard: show per pool IOPS on health page (#22495).

Reviewed-by: John Spray <john.spray@redhat.com>

mgr: dashboard: show per pool IOPS on health page (#22495).

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

common: change routines to public access

Clang complains tafter recent changes:
/home/jenkins/workspace/ceph-master/src/osd/OSD.cc:8787:8: error: 'DeleteStart' is a protected member of 'PG'
          PG::DeleteStart())));
              ^
/home/jenkins/workspace/ceph-master/src/osd/PG.h:1870:16: note: declared protected here
  TrivialEvent(DeleteStart)
               ^
/home/jenkins/workspace/ceph-master/src/osd/OSD.cc:9136:6: error: 'DeleteSome' is a protected member of 'PG'
        PG::DeleteSome())),
            ^
/home/jenkins/workspace/ceph-master/src/osd/PG.h:1871:16: note: declared protected here
  TrivialEvent(DeleteSome)
               ^
2 errors generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/rbd: fix typo s/morror/mirror

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #19931 from mikechristie/wip-doc-mnc-iscsi-updates

Misc iSCSI doc updates

Reviewed-by: John Wilkins <jowilkin@redhat.com>

ceph-volume: print the lvm activate success message correctly

Move the terminal.success to the methods that have access to the
correct osd_id so that they do not print None for the osd_id
in the log message.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17520 from joscollin/wip-cleanup-drop-return

rgw: Drop unnecessary return

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #19803 into master

* refs/pull/19803/head:
mds: get rid of unnecessary if condition

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19837 into master

* refs/pull/19837/head:
client: avoid recursive lock in ll_get_vino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge PR #19860 into master

* refs/pull/19860/head:
cephfs-journal-tool: add usage help info for header set <field>
mds: add error log info
cephfs-journal-tool: add "set pool_id" option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19891 into master

* refs/pull/19891/head:
mds: handle 'inode gets queued for recovery multiple times'
mds: use elist to track recover queue items

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19911 into master

* refs/pull/19911/head:
cephfs: Switch MIN/MAX for std::min/max and use intarith templates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PG: drop 'seed' property from Scrubber

This has been -1 for many releases now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/divergent_priors*: unsquelch osd debug

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: debug ec scrub error paths

Signed-off-by: Sage Weil <sage@redhat.com>

osd: document scrub options

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow limited scrub preemption

If we receive a write within the scrub range, abort the scrub chunk and
shrink the chunk size.  If we do this too many times do not preempt and
allow the scrub to complete (to avoid scrub starvation due to client io).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19876 from theanalyst/build/do-cmake-args

build: do_cmake: allow ARGS to be overridden

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19433 from liewegas/wip-pg-removal

osd: put pg removal in op_wq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PrimaryLogPG: do on_shutdown on removal

This cleans up our reservations and misc other state in OSDService that
needs to be cleaned up.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track deleted pools' pg_nums for calculating split

This is needed to determine (quickly) whether PGs have split.  Calling
get_pg_num() on the latest map does not work when the pool has been
deleted from that map.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19850 from dzafman/wip-calc-stats

osd/PG: re-write of _update_calc_stats and improve pg degraded state

Fixes: http://tracker.ceph.com/issues/20059
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: recovery-unfound-found test needs to account for correct misplaced calculations

The test expected HEALTH_OK when in a state with misplaced objects therefore HEALTH_WARN

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: add rbd pool requirement for gwcli

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: add daemon-reload for rbd-target-api setup

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: add gwcli manual install instructions

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge pull request #19613 from qrGitHub/wip-rgw-optimize-bucketLCPrepare

rgw: optimize function bucket_lc_prepare

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19878 from cbodley/wip-rgw-get-omap-keys-cr

rgw: fix handling of ENOENT in RGWRadosGetOmapKeysCR

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19897 from cbodley/wip-rgw-put-system-obj

rgw: rgw_put_system_obj takes bufferlist

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume: docs for --crush-device-class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds crush_device_class to json metadata for 'osd new'

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds crush_device_class to the functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add crush_device_class to lvm metadata

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a --crush-device-class flag to lvm create and prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19945 from tchaikov/wip-deprecated-in-cxx17

librbd,librados: do not include stdbool.h in C++ headers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19261 from dillaman/wip-22286

mgr: disconnect unregistered service daemon when report received

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #19942 from shinobu-x/wip-sk-minmax-ms_async_op_threads

common: Add min/max of ms_async_op_threads

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19397 from xxhdx1985126/master

mon: update PaxosService::cached_first_committed in PaxosService::maybe_trim()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: piecewise scrub

Perform scrub in stages, with each unit of work requeuing an item in the
work queue.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: separate gwcli install from post install setup steps

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: add note to clarify trusted_ip_list use

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: update esx instructions

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge pull request #19822 from trociny/wip-ggate-ceph-globals

rbd-ggate: fix parsing ceph global options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19673 from zouaiguo/wip_erase_code

mon/OSDMonitor.cc : set erasure-code-profile to "" when create replicated pools.

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19896 from tchaikov/wip-vive-cxx-17

msg/simple: s/ceph::size/std::size/

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19870 from liewegas/wip-strip-ceph-base

debian/rules: strip ceph-base libraries

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18755 from aclamk/wip-reverted-approx-size

osd/PG: revert approx size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18191 from hjwsm1989/osd-mark-down

qa/standalone/osd/osd-mark-down: create pool to get updated osdmap faster

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19877 from adamemerson/wip-dencish

denc cleanups and other fixes

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Verify stat calculations during backfill

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Verify stat calculations during recovery

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-helpers.sh: Add flush_pg_stats() to wait_for_clean() to make it reliable

osd-scrub-repair.sh: Fixes for omap keys landing on different OSDs due to flush

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't start recovery for missing until active pg state set

I was seeing recovery hang when it is started before _activate_committed()
The state machine passes into "Active" but this transitions to activating
pg state and after commmitted into "active" pg state.

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Ignore degraded PGs when injecting random eio errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Improve the way insufficient targets is handled to be compatible with EC

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Improve pg degraded state setting based on _update_calc_stats() degraded count

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19946 from liewegas/wip-22677

qa/workunits/rados/test_rados_tool: fix stray |, race

osd: Handling when recovery sources have missing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Base pg degraded state on num_degraded_objects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Rewrite _update_calc_stats() to make it cleaner and more accurate

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: cleanup: Remove unused const vars

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: cleanup: Fix log message

Signed-off-by: David Zafman <dzafman@redhat.com>

osd/PG: flush osr before listing scrub objects

We need to make sure previous transactions are readable before we list
objects.  This avoids races like, say,

- ecbackend rolls forward and removes a rollback object
- bluestore delays the txn apply due to some prior op
- scrub lists objects, sees rollback object it doesn't expect

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: turn be_deep_scrub options into legacy

We don't have a lightweight mechanism for doing trivial config options
that is better than legacy_config_opts.h yet.  Until then,

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: disable osd_debug_deep_scrub_sleep in case it is set

Otherwise the final scrub may take too long.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/*Backend: debug: inject sleep during deep scrub

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop waiting_on, use waiting_on_whom

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_rados_tool: remove stray |, fix race

Fixes: http://tracker.ceph.com/issues/22676
Signed-off-by: Sage Weil <sage@redhat.com>

mds: fix some typo

Signed-off-by: dongdong tao <tdd21151186@gmail.com>

librbd,librados: do not include stdbool.h in C++ headers

stdbool.h is offered to provide C++ keywords for C source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12431 from VictorDenisov/snapshot_rename

rbd: group snapshot rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cephfs: Switch MIN/MAX for std::min/max and use intarith templates

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Add group snapshot rename commands

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add group snap rename method

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

qa/workunits/rados/test_rados_tool.sh: -x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19913 from adamemerson/wip-minimax-rados

rados: Prefer templates to macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19914 from tchaikov/wip-cmake-check-boost-in-src

cmake: check version of boost in src/boost

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19926 from batrick/i22527

mon: revert mds metadata argument name change

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19903 from dillaman/wip-librbd-op-features

librbd: add preliminary support for new operation feature bit

Reviewed-by: Mykola Golub <mgolub@suse.com>

common: Add min/max of ms_async_op_threads

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #19793 from badone/wip-blocking-deep-scrub

ceph.in: Add blocking mode for scrub and deep-scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19910 from jdurgin/wip-21897

config: lower default omap entries recovered at once

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19894 into master

* refs/pull/19894/head:
cmake: link against ceph-common instead of common

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19920 into master

* refs/pull/19920/head:
client: fix cap revoke race
client: release revoking Fc after invalidate cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19885 into master

* refs/pull/19885/head:
qa: increase osd count for ec testing
qa: add missing openstack configs

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: RGWRados::put_system_obj takes const bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rados: Switch MIN/MAX for std::min/max and use intarith templates

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19900 from dillaman/wip-rbd-lock-list

rbd: adjusted "lock list" JSON and XML formatted output

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: rgw_put_system_obj takes bufferlist

all callers were converting from bufferlist to const char*, then
RGWRados::put_system_obj() copied that back into a new bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19933 from qrGitHub/wip-rgw-useless-line4

rgw: drop redundant RGW_OP_STAT_OBJ check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: adjusted "lock list" JSON and XML formatted output

Signed-off-by: Jason Dillaman <dillaman@redhat.com>