rgw: align head object size with pool alignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: head obj contains data only if on the same pool as tail

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: putobj processor: fix to allow different head sizes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj(): set compression_type inside callback

The tail_rule is being determined at the callback, not before.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: modify locked check

Can get either -EBUSY or -EEXIST.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: check if default placement target is set correctly

Don't check if rule is empty. Rule might consist of non empty
storage class, but in this case we only care about the placement
id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRadosPutObj() fix need_to_process_attrs handling

Set need_to_process_attrs anyway, and not only if have extra_data. This
can be triggered also in cases where we get error response, in which case
there's no extra data. Make sure it's only getting called once.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: user info json encoding preserves storage_class state

This is needed for meta sync. Otherwise one zone would have storage_class
not set, and the other zone would have it set.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/rgw/tempest: either force os_type or select random distro

Fixes: https://tracker.ceph.com/issues/37885
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/rgw_multi: adjust structure for new zone config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sanitize header attributes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/rgw: support for storage classes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean up generic attribute val

There are cases where we keep/kept attributes value with a null termination,
which bufferlist.to_str() returns as part of the string and returned
headers are broken.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/cli: fix radosgw-admin cli test

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/ragweed: change the way we determine which ragweed branch to use

 - if force-branch, use that
 - otherwise:
    - read default-branch from client config
    - use suite branch or ceph branch if suite branch is not defined
    - if this branch is one of official releases (or master), prefix
      it with 'ceph-'

try to clone branch specified above, if failed (branch doesn't exist probably)
and not force-branch, use default-branch.
Also add an option to override ragweed repo.

Switched all force-branch from ragweed qa suite to default-branch.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get layout rest op: fix last tail object size

This triggered an issue in ragweed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix generic obj attrs to http headers

use bl.to_str() instead of bl.c_str() to avoid encoding of the null
termination.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix check for no storage_class specified in obj copy

S3 doesn't allow copying object into itself if it doesn't modify
attributes. Need to check that the object's storage class changes
but we only have that info later, so code rearranged a bit to
allow that.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: don't allow configuring zone placement if not configured in zonegroup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: rework storage class zone config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use dest placement for compression type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove a warning

No need for warning, rgw_placement_rule is being read as a string
and backward compatible.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() supports placement rule

placement rule is optional, and if not provided will use the storage_class of
the source object that is being fetched (if possible).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustments following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: do filter checks later

Filter checks (currently only tags checks) need to access the objects
themselves, while the action checks don't. Make action checks happen first
to avoid unnecessary rados operations if objects don't pass action checks.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: support for noncurrent transition

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: current obj transitions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: use unmod_since when deleting obj

To avoid a race. Also use the op object ctx to avoid reading object
head again (if read previously), and consolidate the different
removal calls.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: fixes, some useful logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tighter lifecycle processing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: consolidate versioned and non-versioned operation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: rework listing iteration

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle: some code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: lifecycle and xml parser fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework lifecycle parsing

Use new style xml parsing, instead of inheriting from XMLParser.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move check_date place

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

rgw: remove enum defination for storage class.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

rgw: support object transition in lifecycle config set.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

rgw: check that placement rule is valid if bucket exists

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: prevent creation of bucket with non-existent storage class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't try to read beyong object size

Otherwise even if we prefetch all the data, we'd try to issue another
request to read the rest of the object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tiering: encode standard storage_class in rule explicitly

If it's standard storage class, don't avoid encoding it,
otherwise we can't know if it was set or not.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tiering: add support for multipart uploads

Encode multipart info structure in the upload meta object,
store the storage_cass info there. Use the storage class
that was set on upload init when uploading all the parts,
and when completing the upload.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tighten storage_class assignment

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix storage_class in rgw_bucket_dir_entry_meta::dump

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

rgw, cls_rgw: propagate storage class to the index

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: copy_obj() inherit dest rule from default bucket rule

Otherwise it can be empty, in which case we'll always copy data, even
if rule doesn't change.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set dest placement rule when copying object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: blacklist x-amz-storage-class header

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data obj synchronous read, use correct ioctx

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: encode dest storage_class when copying object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use bl.to_str() and not bl.c_str()

can't rely on bufferlist to be null terminated

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: clean dumped attrs

don't show terminating null chars

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: tiering fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: api fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate storage class to manifest generator

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more storage class data structure changes

storage class also handles compression info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add support for storage class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: object write processors use tail placement rule

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25674 from alfredodeza/wip-rm37442

ceph-volume normalize comma to dot for string to int conversions

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #24635 into master

* refs/pull/24635/head:
test: add scrub control command tests
mds: scrub abort/pause/resume/status control commands
test: switch using "scrub start" tell interface to initiate scrub
mds: introduce "scrub start" tell interface to initiate scrub
mds: dump scrub formatted output when context completion
mds: generate random scrub tag when empty
mds: introduce C_ExecAndReply context completion class
mds: use CInode::scrub_is_in_progress() wherever required

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #25473 from theanalyst/rgw/version-copy-fix

rgw: fix versioned obj copy generating tags

Merge pull request #25761 from tchaikov/wip-qa-health-schema

mgr/dashboard: updated API test to reflect changes in ModuleInfo

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25736 into master

* refs/pull/25736/head:
common/options: document some osd/rados options
unittest_osdmap: feed options as defaults
mon/OSDMonitor: allow osd_pool_default_pgp_num to be 0

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25766 from tchaikov/wip-rgw-readdir-flags

pybind/rgw: pass the flags to callback function

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #25635 from rhcs-dashboard/36740-pool-stats-disabled-by-default

mgr/dashboard: pool stats not returned by default

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #25528 into master

* refs/pull/25528/head:
mds: log evicted clients to clog/dbg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #24845 from pritha-srivastava/wip-rgw-assume-role-fixes

rgw: Minor fixes to AssumeRole for boto compliance.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #25180 from pritha-srivastava/wip-rgw-user-policy-fixes

rgw: Fixes to permission evaluation related to user policies.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25659 from Linbing/wip-fixup-read-deadlock

os/bluestore: fixup access a destroy cond cause deadlock or undefine behavior

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25673 into master

* refs/pull/25673/head:
mon/Monitor: quorum_age in JSON output as int, not string

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/mgr: add "module_options" to module_info_schema

it was introduced in 280c23ba. so update the schema test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rgw: pass the flags to callback function

before this change, the `flags` parameter passed to `LibRGWFS.readdir()`
will be dropped on the floor and ignored.
after this change, it will be passed to the specified callback function.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25580 from cyx1231st/wip-crimson-msgr-logging

crimson/net: fix addresses during banner exchange

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: fix address learning during banner exchange

* Don't store my_addr in `Connection`, because my_addr can be learned
  and thus changed.
* Support nonce in SocketMessenger.
* Always set nonce when set_myaddr().
* Add learned_addr() for SocketMessenger.
* Add side_t and socket_port to show the real connecting
  ports of the SocketConnection.
* Fix bannder exchange logic for addresses, including nonce, type, ip,
  port, socket_port for my_addr and peer_addr.
* Add more detailed logging prefixes for SocketConnection.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: add logic_name to indicate the msgr role for debugging

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

crimson/net: add basic loggings for SocketConnection

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #25757 from gregsfortytwo/wip-doc-put-warning

doc: warn about how 'rados put' works in the manpage

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25706 from joscollin/wip-build-doc-librados

librados: fix admin/build-doc warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25734 from tchaikov/wip-mgr-default-options

mgr: define option defaults for MgrStandbyModule as well

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25735 from wjwithagen/wjw-fix-WRITE_LIFE

os/bluestore: Only use WRITE_LIFE_ when available

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25746 from tchaikov/wip-pg-autoscale-mode

osd/OSDMap: set pg_autoscale_mode with setting from conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25588 from tchaikov/wip-auth-silence-gcc-warning

cls/rbd: init local var with known value

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

qa/mgr: extract module_info_schema and reused it

both standby mgr and active mgr have this sub-schema. so extract it out
and reuse it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rbd: init local var with known value

DirectoryState does not have an "invalid" enum so far, since it's
defined using `enum class`, init a value of this type with a known value
could be a better choice even it is always initialized before being read.

this silences the GCC warning of:

src/cls/rbd/cls_rbd.cc:3147:3: warning: ‘on_disk_directory_state’ may be
used uninitialized in this function [-Wmaybe-uninitialized]
   if (directory_state != on_disk_directory_state) {
   ^~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25687 from tchaikov/wip-pg-stats

osd/PrimaryLogPG: do not count failed read in delta_stats

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: warn about how 'rados put' works in the manpage

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25705 from joscollin/wip-doc-common

doc: Fix Create a Cluster url in Running Multiple Clusters

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25450 from ivancich/wip-rgw-category-cleanup

rgw: clean-up -- use enum class for stats category

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

rgw: clean-up -- remove unnessary "struct"s

In C++ "struct" is unnecessary when referencing a defined
struct. Clean up all uses in src/cls/rgw source files.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean-up use enum class for stats category

The stats entries for rgw buckets has a category, which used a
combination of uint8_t and enum RGWObjClass. Clean this up by
converting RGWObjClass to an enum class and using that
throughout. This provides type safety and better code clarity. Also,
add some source code documentation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-volume tests.util ensure ints and strings with commas can be converted to ints

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume test verify strings with a comma can be converted to ints

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util normalize comma with dot for str-to-int conversion

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #24676 from TheJJ/cryptsetup-allow-discards

ceph-volume: enable device discards

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

common/options: document some osd/rados options

Many of these are actually on the mon.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_osdmap: feed options as defaults

We can't change some of these at runtime due to FLAG_CREATE.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: set pg_autoscale_mode with setting from conf

* update build_simple_optioned() to set pg_autoscale_mode with the setting
  read from conf, otherwise it will be a random value in heap.
* update cli test accordingly, otherwise we will have

/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/create-racks.t:
failed
---
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/create-racks.t
+++
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/create-racks.t.err
@@ -796,7 +796,7 @@
   nearfull_ratio 0
   min_compat_client jewel

-  pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 15296 pgp_num 15296 last_change 0 flags hashpspool
stripe_width 0 application rbd
+  pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 15296 pgp_num 15296 autoscale_mode ??? last_change 0
flags hashpspool stripe_width 0 application rbd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25441 from theanalyst/decode-refcount-types

cls: refcount: add obj_refcount to ceph-dencoder

Reviewed-By: Casey Bodley <cbodley@redhat.com>

ceph-volume: enable device discards

When using SSDs as encrypted OSD device, discards do not pass the
encryption layer. This option activates discard requests.

Signed-off-by: Jonas Jelten <jj@stusta.net>