Merge pull request #32215 from smithfarm/wip-43234-luminous

luminous: tests: radosgw-admin: remove dependency on bunch package

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #32135 from jecluis/wip-telemetry-luminous

luminous: telemetry module for mgr

Reviewed-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: add telemetry mgr module

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

mgr/telemetry: bump revision

We should have done this while cherry-picking from master, but we
didn't. And here we are now. It's simpler to apply this one-off patch
than going back to the cherry-picking maze to adjust this one thing.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

mgr/telemetry: add stats about crush map

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 713dbc9722888d3bf60d772dbca23e13b0cafc38)

 Conflicts:
src/pybind/mgr/telemetry/module.py
          Missing context due to missing patches.
        PendingReleaseNotes
          Dropped to prevent conflicts in the future

mgr/telemetry: add rgw metadata

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f62c6e8cba2e894f84ddabdea6db4ce56e02ea63)

 Conflicts:
PendingReleaseNotes
          Dropped to prevent conflicts in the future
src/pybind/mgr/telemetry/module.py
            Context issues due to missing patches

mgr/telemetry: mds cache stats

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f4c736699478f608bba77770a85f96a7bf8d24e5)

 Conflicts:
src/pybind/mgr/telemetry/module.py
            Due to missing context resulting from missing patches.
        PendingReleaseNotes
            Dropped to prevent conflicts in the future

mgr/telemetry: add more pool metadata

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 87670fdc3c227387068d527b4659b50bc3bb64a3)

 Conflicts:
src/pybind/mgr/telemetry/module.py
          Context issues
        PendingReleaseNotes:
          Dropped to prevent conflicts in the future

mgr/telemetry: remove crush rule name

This is a user-specified string and could contain identifying info.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 26b427356d920682b61cdf98fc2745e324c28baa)

 Conflicts:
src/pybind/mgr/telemetry/module.py
            Context issues

mgr/telemetry: include min_mon_release and msgr v1 vs v2 addr count

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3453930d438dc3ba9ba5addca59aec6786293bd4)

 Note:
    This commit was heavily modified. We wanted to provide the number of
    ipv4 and ipv6 monitors in the report, so we rewrote that part so we
    can report on it; but we had to drop everything else (msgr1 and
    msgr2), as well as 'min_mon_release'. Those do not exist in
    luminous. In the end, the commit message itself is misleading, but
    we are somehow (*shrug*) opting for leaving the commit as the original.
    Additionally, we removed PendingReleaseNotes changes to prevent
    conflicts in the future.

mgr/telemetry: add CephFS metadata

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7f6aad677b76847514f6f9b893827412dfb35a6b)

 Conflicts:
PendingReleaseNotes
            Dropped due to conflicts down the road
src/pybind/mgr/telemetry/module.py
            Context issues

Merge pull request #31857 from smithfarm/wip-40947-luminous

luminous:  osd: add hdd, ssd and hybrid variants for osd_snap_trim_sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31858 from smithfarm/wip-38205-luminous

luminous: osd: refuse to start if we're > N+2 from recorded require_osd_release

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31992 from dzafman/wip-balancer3-luminous

luminous: mgr: Release GIL and Balancer fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: radosgw-admin: remove dependency on bunch package

Fixes: https://tracker.ceph.com/issues/43184
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1bf21326aa7f8eaafd7049b44eb73aeb36bcc5d7)

Merge pull request #31846 from smithfarm/wip-42988-luminous

luminous: tests: kernel.sh: update for read-only changes

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

release note: Add pending release notes for already merged code

Follow on to https://github.com/ceph/ceph/pull/31774

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 65d03bae8b4f50cc3cbaa50640eaeab4cabd711f)

common/options.cc, doc: osd_snap_trim_sleep overrides other variants

A value > 0 for osd_snap_trim_sleep, will override the backend specific
variants of osd_snap_trim_sleep.

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 733df09fe5111e7beca75f8be0afb8669ef9a625)

doc/rados/configuration/osd-config-ref.rst: document snap trim sleep

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit accf95e9dea257c3aaa64b7a36d077468d7c86ec)

osd: add hdd, ssd and hybrid variants for osd_snap_trim_sleep

This is better than the earlier default, which was set to 0.

Fixes: https://tracker.ceph.com/issues/40528
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 560fca12e695a817e1b7e46d365838ed871b64bd)

Conflicts:
src/osd/OSD.cc
src/osd/OSD.h
src/osd/PrimaryLogPG.h
- no OSD::get_osd_delete_sleep() in luminous, no OSD::get_recovery_max_active()
  in luminous
- use cct->_conf->get_val instead of cct->_conf.get_val

osd: refuse to start if release > recorded min_osd_release + 2

If we try to start up the objectstore, we may make writeable changes to
(say) rocksdb that are not backwards compatible.  This happens, for
example, if you start a mimic osd.  Even if the compatset checks fail,
rocksdb may have written something that is not backwards compatible.

Fixes: http://tracker.ceph.com/issues/38076
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9f7713a905d67441b28371e4494e9447319d2129)

Conflicts:
src/ceph_osd.cc
- include common/version.h for ceph_release()
- use exit instead of forker.exit

osd: record require_osd_release in objectstore meta

Record the require_osd_release value from the OSDMap in the 'meta' portion
of the osd's metadata that can be accessed without actually mounting the
OSD.  This will be useful as a safety gate to prevent you from mounting
an osd thet is too new that may make incompatible changes to the store.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 482cdca55351ca85290f1b2adb3c0cdf78af411d)

Conflicts:
src/osd/OSD.cc
src/osd/OSD.h
- ignore differences in surrounding context, as they do not seem relevant to
  the fix

mgr/telemetry: clear the event after being awaken by it

otherwise telemetry will have a busy-loop once it's signaled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0dddb20685007d990dc30daddca6212d36e5e308)

mgr/telemetry: force re-opt-in if the report contents change

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 143e1f046909cb68d769ebbbaa80cb7106879997)

 Conflicts:
doc/rados/operations/health-checks.rst
            We don't have the crash module, hence neither its docs.
src/pybind/mgr/telemetry/module.py
            Issues due to context

mgr/telemetry: less noise in the log

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7f1897b238f5c55859229748b200bac715941cc3)

Conflicts:
src/pybind/mgr/telemetry/module.py
            Due to context resulting from previous cherry-pick amends

mgr/telemetry: track telemetry report revisions

Assign revisions to track changes to the content of the telemetry
reports.

Track the revision when the user opts in or out.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b93e4050261b962449c9721cea713616ce5e03bd)

Conflicts:
    pybind/mgr/telemetry/modules.py
        Due to set_module_option() -> set_config()

mgr/telemetry: specify license when opting in

Choosing not to include this in the docs so that the user is more likely
to see this interactively.  (That is...probably good?)

Choose sharing-1-0.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f1eac8ba4becfb0ad6a7e7a303fc45a9d6ab59ee)

Conflicts:
src/pybind/mgr/telemetry/module.py
          Slight conflicts due to past cherry-picks (or lack thereof)
          Using set_config() instead of set_module_option()

mgr/telemetry: move contact info to an 'ident' channel

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 35c273c9c3ebd00fbac8e1a9fd281333a24c5fe7)

Conflicts:
src/pybind/mgr/telemetry/module.py
          Due to lack of 'crash' and 'devicehealth' modules, and a bit
          on how we keep options (self.config[] vs class attributes)

mgr/telemetry: accept channel list to 'telemetry show'

Also include a 'channels_available' item so that a user can tell which
channels are available.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e9fdb219c1b946dc2b346b11c5e5d4b04786345d)

Conflicts:
src/pybind/mgr/telemetry/module.py
          Due to lack of 'crash' and 'devicehealth' modules

mgr/telemetry: always generate new report for 'telemetry show'

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f0762ce4470972ce8f555d8aab87362331a53afd)

mgr/telemetry: add separate channels

'basic' is the basic cluster stats (version, size, etc)
'crash' is the crash dumps.

By default these are both on, but they can be selectively enabled or
disabled.

New channels will follow.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f3a3ccb52fa5a7b24f8d11138b5b51345865547e)

Conflicts:
src/pybind/mgr/telemetry/module.py
          Don't backport code related to the 'crash' module, and adjust
          how we read option variables (luminous goes through a config
          map, instead of master's that goes through class attributes)

mgr/telemetry: use cluster-provided timestamp unmolested

The cluster stamp is now ISO 8601; just use that.

(The isoformat() puts a : in +hh:mm the timezone offset, which is slightly
different than what Ceph does; just pass Ceph's value through for
consistency.)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 383006a5cc2da3e0b643a9dc600e75e1ce088bd6)

Conflicts:
src/pybind/mgr/telemetry/module.py
          Due to missing scaffolding that exists in master but was not
          backported to luminous.

mgr/telemetry: default to reports every 24h; lower minimum

Allow more frequent telemetry reports.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 712987d53337e39ad871ee7abb38c2e2755fc75d)

Conflicts:
src/pybind/mgr/telemetry/module.py
          Past commit in master had introduced field types and a
          'minimum' value for the interval. We concluded that the field
          types commit does not affect the telemetry module in a
          significant way to force us to backport it, and the minimum
          value commit is introduced for the benefit of the dashboard
          (which, in luminous, does not have control over telemetry)

mgr/telemetry: add report_timestamp to sent reports

Received time may differ from report time, and correlating
to local cluster state events might be useful.

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit a42c8e327c9f7d53b8c13cf51837c294bc4c643d)

mgr/telemetry: fix 'telemetry {on,off}'

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 497e00c4dadd9e29a792413a425671c061fa44c6)

mgr/telemetry: make 'telemetry show' readable by a human

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ec78dcf516273b18ce6e92708da89466c9c0409d)

mgr/telemetry: check for errors when sending report

There was no error checking, and the server has been failing for
some time, but no one noticed.  Oops.

Signed-off-by: Dan Mick <dan.mick@redhat.com>
(cherry picked from commit de71f38a2c0b37a96970d6b2fd62ea19b20bdf46)

Conflicts:
src/pybind/mgr/telemetry/module.py
          mostly due to store_get/store_set not existing in luminous,
          and we relying instead on config_get/config_set.

mgr/telemetry: add 'telemetry on' and 'telemetry off' commands

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6ab90c9cb09627649ca31f27c35e1c1efd6a6f12)

Conflicts:
src/pybind/mgr/telemetry/module.py
          master no longer has 'telemetry selftest' due to some other
          major changes that we did not backport, as they would require
          too many changes that were not, in an obvious manner, relevant
          for us.

mgr/telemetry: off by default

This way a user can enable the module and look at the output before
deciding to send it to anyone.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9320bdb8ac06668719a9f910e12a9e9f4ed56405)

Conflicts:
src/pybind/mgr/telemetry/module.py
          We don't have some other scaffolding that exists on master,
          and we are not cherry-picking it because it changes
          significantly the module's code in a way that is not a clear
          advantage for the telemetry module (in 'luminous' context)

mgr/telemetry: fix total_objects

This field was removed from df output a while back in
342f309645df886fb96eb401634e38376553e6d9

Fixes: http://tracker.ceph.com/issues/37976
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c849d7dfcc7306b8945d7a697fb76558ed50f983)

mgr/telemetry: Check if boolean is False or not present

Now we disable the module if the boolean is True

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit 20f5696a735094cbaa0a698f41094cd5a5ac96f4)

mgr/telemetry: Add Ceph Telemetry module to send reports back to project

This Manager Module will send statistics and version information from
a Ceph cluster back to telemetry.ceph.com if the user has opted-in on sending
this information.

Additionally a user can tell that the information is allowed to be made
public which then allows other users to see this information.

Signed-off-by: Wido den Hollander <wido@42on.com>
(cherry picked from commit 8f6137d162d2505eaefe489e16f3adffc7977cf8)

Conflicts:
doc/mgr/index.rst
qa/tasks/mgr/test_module_selftest.py

Merge pull request #32019 from dzafman/wip-revert-verify-upmap-luminous

luminous: Revert "crush: remove invalid upmap items"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31966 from smithfarm/wip-43100-luminous

luminous: ceph-detect-init: run tox tests on Python 2 only

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

ceph-detect-init: run tox tests on Python 2 only

Luminous is EOL upstream and 12.2.13 will be the last point release, so it's
pretty much a given that luminous will not be migrated to py3.

Luminous runs on OS versions where py3 is not a "first-class citizen": compiling
the mgr with a py3 subinterpreter on such systems would be "interesting"...

Python 3 testing was dropped for ceph-disk by
28c545eff5f73e392b70ef4c0cf74f8a896b693b (over 3 years ago) and
ceph-detect-init is a component of ceph-disk.

This commit cannot be cherry-picked from master because ceph-detect-init was
removed in nautilus.

Fixes: https://tracker.ceph.com/issues/43100
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Revert "crush: remove invalid upmap items"

This reverts commit 09528ac4663cda9f206eb8d4dcc15926b8e4586f.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test to use -gt instead of creating an empty file "0"

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d0b260c272cf2732437c838c80ab380b4bb8d433)

test balancer: Backport specific fixes

Add "ceph balancer sleep" command to set balancer sleep_interval for testing
Remove unavailable "ceph balancer pool" part of testing
Remove setting of nonexistant osd_pool_default_pg_autoscale_mode
Improve balancer module log message
Fix log message test (no pg merging)
Pool balancing isn't grouped by rule, so results different here

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 61dedc249bb93a208e685c68a65a5c07d60870d1)
From mimic backport

qa/standalone/ceph-helpers.sh: fix mgr module path

callers of get_python_path were not passing in a $1 parameter, so
ceph_lib was an empty string resulting in an invalid path to the built
cython modules. assume this is called from the `lib` parent directory.

pass path to the manager modules when starting ceph-mgr.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>
(cherry picked from commit 7d3fa9bda3b0160c872e388c9f75cd1cabf9d4b5)

Conflicts:
qa/run-standalone.sh (No get_python_path() to modify)

test: Luminous specific change required due to output difference

Signed-off-by: David Zafman <dzafman@redhat.com>

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()
the use of `std::random_shuffle()` was introduced by
b946308 .

in this change, it is replaced using `std::shuffle()`.

Fixes: https://tracker.ceph.com/issues/43084
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 30dfef208d859a67c1f1b13c314735861c536e85)

Conflicts:
src/tools/osdmaptool.cc (randomizing pools here instead of rules)

Merge pull request #31934 from smithfarm/wip-tox-luminous

luminous: rpm: explicitly declare python-tox build dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

test: Add test case based on Xie script in commit comment

Other test fixes to reflect changes

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9d2e0267e1f025a634ee4925c919008cacdd3d38)

Conflict:
src/test/cli/osdmaptool/upmap-out.t (older output format)
src/test/cli/osdmaptool/upmap.t (older output format)

osd: ceph_pg_upmaps() use any overfull when there are still underfull

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f38d38398f33f64bd49f017b1c8e737452157c53)

osd: Create more_underfull with below target that aren't in underfull

Use more_underfull for finding remaps for overfull OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit fe03383e6691f6bd95189d3ba4d91fbb2ea9e9c8)

osd: calc_pg_upmaps() pick most overfull remap from try_pg_upmap()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 3f132061fd33c82dc131e18ae80cbec067b53cae)

osd: ceph_pg_upmaps() use max_deviation to determine perfect distribution

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 7f4ae9312314c329b63c850e09b403a95ad18bed)

Conflicts:
src/pybind/mgr/balancer/module.py (upmap_max_deviation config option
   handled differently)

tools: osdmaptool: Perform upmap calculation as ceph-mgr does

This is the backportable commit that works with older balancer module.py

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b946308f03f1798915fafe7878f34b4a234c2ae4)

osd: For balancer crush needs the rule passed to get_parent_of_type()

Fixes: https://tracker.ceph.com/issues/42718
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 3eb69fd0cb15974e33e675babdd5d232d69c3c6d)

osdmaptool: Match default max value of 10

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 91bab14ea0e2ec7c94eb758cd76f9a620688725f)

tools: odsmaptool truncate target upmap file

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a49324ceb9c322f792ea00720f438ea9ea09555f)

mgr: Improve balancer module status

Add balancer status fields so that slow optimizations can be detected

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit f04c505b9dc128bdfbcca49d82f8aac4abf8b56f)

test: Test balancer module commands

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 3a0e2c8ff1cce44bf44b139572f4d1bd9bbfb766)

mgr: Release GIL before calling OSDMap::calc_pg_upmaps()

Prevent optimize and execute commands from running with active balancer

Fixes: https://tracker.ceph.com/issues/42432
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit e2a35e8c8e9f381a635ad0ca01326a2c512590a9)

Conflicts:
src/pybind/mgr/balancer/module.py (trivial)
src/test/cli-integration/balancer/misplaced.t (doesn't exist)

rpm: explicitly declare python-tox build dependency

"make check" needs python-tox to run, but until 71ac2163831ffd764bf3da6a3efa76ef02e5e884
("ceph.spec: added dashboard_v2 development and runtime dependencies")
this was not explicitly declared in the spec file. It does not make sense to
backport that commit to luminous, though, because the dashboard itself was never
backported to luminous.

Fixes: https://tracker.ceph.com/issues/43082
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #31664 from hillpd/wip-38714-luminous

luminous: rgw: resolve bugs and clean up garbage collection code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31655 from smithfarm/wip-40496-luminous

luminous: doc: fixed --read-only argument value in multisite doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31647 from smithfarm/wip-39603-luminous

luminous: doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: kernel.sh: update for read-only changes

Snapshot existence validation code was removed from krbd.  It was racy
and relied on having watch established for snapshots.

Fixes: https://tracker.ceph.com/issues/42916
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 6fb544045e743c5dc73dac2f62aeeb964dcd62bc)

Merge pull request #31696 from k0ste/wip-42849-luminous

luminous: pybind/mgr: Cancel output color control

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31587 from k0ste/wip-42784-luminous

luminous: mgr/prometheus: assign a value to osd_dev_node when obj_store is not filestore or bluestore

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #31360 from idryomov/wip-krbd-udev-fixes-luminous

luminous: krbd: avoid udev netlink socket overrun and retry on transient errors from udev_enumerate_scan_devices()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #31654 from smithfarm/wip-40350-luminous

luminous: rgw/OutputDataSocket: actually discard data on full buffer

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31653 from smithfarm/wip-40359-luminous

luminous: rgw: set null version object acl issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31652 from smithfarm/wip-40132-luminous

luminous: rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31651 from smithfarm/wip-40127-luminous

luminous: rgw: parse_copy_location defers url-decode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31649 from smithfarm/wip-39747-luminous

luminous: rgw: delete_obj_index() takes mtime for bilog

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31648 from smithfarm/wip-39733-luminous

luminous: rgw: cls/user: cls_user_set_buckets_info overwrites creation_time

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31220 from less-is-morr/wip-39732-luminous

luminous: rgw: allow radosgw-admin to list bucket w --allow-unordered

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31218 from smithfarm/wip-39696-luminous

luminous: rgw: return ERR_NO_SUCH_BUCKET early while evaluating bucket policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30946 from smithfarm/wip-40852-luminous

luminous: rgw/multisite: Don't allow certain radosgw-admin commands to run on non-master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30984 from k0ste/luminous_backports

luminous: rgw: add list user admin OP API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31598 from smithfarm/wip-42796-luminous

luminous: osd/OSDMap.cc: don't output over/underfull messages to lderr

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29122 from taodd/barckport-26564

luminous: rgw: LC: handle resharded buckets

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #31674 from smithfarm/wip-42834-luminous

luminous: os/bluestore: fix improper setting of STATE_KV_SUBMITTED.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #31449 from batrick/i41031-luminous

luminous: qa: malformed job

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #31450 from batrick/i42672

luminous: qa: cfuse_workunit_kernel_untar_build fails on Ubuntu 18.04

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #31573 from ukernel/luminous-42774

luminous: mds: add command that config individual client session

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

rgw: resolve bugs and clean up garbage collection code

Does a number of things to clean up rgw gc code:

  * adds additional logging to make future debugging easier.
  * resolves bug where the truncated flag was not always set correctly
    in gc_iterate_entries
  * resolves bug where marker in RGWGC::process was not advanced
  * resolves bug in which gc entries with a zero-length chain
    were not trimmed
  * resolves bug where same gc entry tag was added to list for
    deletion multiple times

Fixes: http://tracker.ceph.com/issues/38454
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 73d7d36)

Conflicts:
src/rgw/rgw_gc.cc dout() vs ldpp_dout()

Note: This was a clean cherry-pick from Mimic, which already resolved
      dout() vs ldpp_dout() conficts.

Signed-off-by: Dan Hill <daniel.hill@canonical.com>
(cherry picked from commit a598ccce2e)

pybind/mgr: Cancel output color control

When I do "ceph osd status" command, the units are not visible
when using a black background, So cancel outpul color control

Fixes: https://tracker.ceph.com/issues/42517
Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>
(cherry picked from commit ca2675e215acc8fbfb2d730b9c1f9f3aafc3e67d)

Conflicts:
- path: rc/pybind/mgr/mgr_util.py
  comment: mgr_util.py in luminous is status/module.py

os/bluestore: fix improper setting of STATE_KV_SUBMITTED.

Fixes: https://tracker.ceph.com/issues/42209
The issue is Nautilus and earlier releases specific as master already has
some changes making the case even worse and then fixing the whole bunch.
See https://tracker.ceph.com/issues/42189

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f8ed28a812bd068d07d4fe5013a731100038bf17)

Conflicts:
src/os/bluestore/BlueStore.cc
- luminous assert became ceph_assert later

doc: fixed --read-only argument value in multisite doc

Fixes: https://tracker.ceph.com/issues/40458
Signed-off-by: Chenjiong Deng <dengchenjiong@umcloud.com>
(cherry picked from commit c65832c1ff03e802f98cb7f4bda620e1e515264f)

rgw/OutputDataSocket: actually discard data on full buffer

A dout message in OutputDataSocket::append_output() states that
data will be dropped when appending would cause data_max_backlog
to be exceeded--but the method appends it anyway.

Log output discards at level 0, as messages will be lost.  Suppress
repeated warnings mod 100.  Switch to vector.

Fixes: http://tracker.ceph.com/issues/40178
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit c806b825dae649829de8847d36cb21ffd2bbee8e)

Conflicts:
src/common/OutputDataSocket.cc
src/common/OutputDataSocket.h

rgw: set null version object acl issues

1.set null version object acl will create empty index
RGWRados::set_attrs did not clear instance, so index prepare, complete got instance=null,
which lead to empty index 1000_<obj>_i_null.
there is no harm to create empty index, but listomapkeys to find that key.

2.if object is exist with versioned key, we can set none exists null version object
order:
1) enable bucket version
2) put obj
3) disable bucket version
4) set versoned_id=null acl will succeed which should not

Fixes: http://tracker.ceph.com/issues/36763
Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit ca6e5c89e8d6da498da15f0eea4bcd4d271b44e9)

rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Fixes: http://tracker.ceph.com/issues/20797
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 40e602bc3866598952eb0dd68ecec947dd7b70d6)

rgw: parse_copy_location defers url-decode

don't url-decode until after we search for the ?, or we'll truncate
object names that contain a url-encoded ?

Fixes: http://tracker.ceph.com/issues/27217
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 804bb853f1c7b5ce546c100d61c6fc526838c134)

rgw: delete_obj_index() takes mtime for bilog

writing an empty timestamp to the bilog prevents other zones from
applying the delete. this means that the --bypass-gc flag for
'radosgw-admin bucket rm' doesn't work in multisite

Fixes: http://tracker.ceph.com/issues/24991
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0316a81875ae93a06841d532bf3548c81370296e)

test/rgw: add multisite test_bucket_creation_time()

does not create any buckets itself, but checks that all creation times
match on buckets created by other tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8df991c5cc48a6709185edf506d8bdec4d28e371)

cls/user: cls_user_set_buckets_info overwrites creation_time

metadata sync of a new bucket entrypoint may call rgw_link_bucket()
(which in turn calls into cls user) without deleting/unlinking the
previous bucket entrypoint. this prevented the new bucket entrypoint
from overwriting the creation_time of the old one

Fixes: http://tracker.ceph.com/issues/39635
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c469e5150a3e95b547a7d05d1ae08fa88fad7159)

doc/rgw: document CreateBucketConfiguration for s3 PUT Bucket api

Fixes: http://tracker.ceph.com/issues/39597
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e97668bb7a1333850a5791ae16b97496851ac854)