radosgw-admin: 'zonegroup add' prints updated zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d355208023fe08a7e16f1078c872286048c4bcc6)

radosgw-admin: 'zonegroup modify' prints updated zonegroup

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a033f40caf60083111886aa804fcdbf3ee60aef8)

radosgw-admin: 'zone modify' prints updated zone

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8a639014a6f2dd950062aa405bbee1762ca23e09)

Merge pull request #9006 from athanatos/wip-15655-jewel

OSDMonitor: avoid underflow in reweight-by-utilization if max_change=1

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Objecter: fix race condition for sortbitwise flag detection

During list_objects/list_nobjects(), we are checking osdmap's
sortbitwise flag setting without holding rwlock in hand, which
is a potential race condition.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 7fb6be706be2454cc2a831df29808997f0aa5ab3)

fix deb package /etc/default/ceph location

commit 7384a14f243519547a61534d22e21e6069aae016 have introduce
the /etc/default/ceph file at a wrong location : /etc/default/ceph/ceph

(cherry picked from commit 13804fafd4c33068d8ed6d0480b594daec0f9341)

mon/OSDMonitor: improve reweight_by_utilization() logic

By calling reweight_by_utilization() method, we are aiming at an evener result
of utilization among all osds. To achieve this, we shall decrease weights of
osds which are currently overloaded, and try to increase weights of osds which
are currently underloaded when it is possible.
However, we can't do this all at a time in order to avoid a massive pg migrations
between osds. Thus we introduce a max_osds limit to smooth the progress.

The problem here is that we have sorted the utilization of all osds in a descending
manner and we always try to decrease the weights of the most overloaded osds
since they are most likely to encounter a nearfull/full transition soon, but
we won't increase the weights from the most underloaded(least utilized by contrast)
at the same time, which I think is not quite reasonable.

Actually, the best thing would probably be to iterate over teh low and high osds
in parallel, and do the ones that are furthest from the average first.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit e7a32534ebc9e27f955ff2d7a8d1db511383301e)

mon/OSDMonitor: avoid potential expensive grace calculation

The grace calculation during check_failure() is now very complicated
and time-consuming. Therefore we shall skip this when it is possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3557903d5d57642179b2ae137bedc389974b1956)

mon/OSDMonitor: max_osds must be > 0 for reweight-by-*

Include a few test cases.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 84bd0dce18f6dd0c35d7e6cfee91ce8c5a0f4db7)

Merge pull request #8787 from xiaoxichen/wip-15631

jewel: CephFSVolumeClient should isolate volumes by RADOS namespace

Reviewed-by: John Spray <john.spray@redhat.com>

OSDMonitor: avoid underflow in reweight-by-utilization if max_change=1

Fixes: http://tracker.ceph.com/issues/15655
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit c121bc51a5a877a49094cfbd3bd16af1f3ae5090)

Merge pull request #8843 from SUSE/wip-msd-typo-fix-jewel

systemd: fix typo in preset file

Reviewed-by: Boris Ranto <branto@redhat.com>

systemd: fix typo in preset file

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 80be4a8cbf1d42232ff61494d035e7abd90c2c14)

systemd: fix typo in preset file

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 80be4a8cbf1d42232ff61494d035e7abd90c2c14)

Merge pull request #8801 from SUSE/wip-15633-jewel

jewel: ceph-{mds,mon,osd} packages need scriptlets with systemd code

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rpm: Add rpm scripts for ceph-rbd-mirror

We are currently missing the systemd scripts for the new ceph-rbd-mirror
daemons. This patch introduces them.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 14652f32165f72efde569a939179fc26584ab155)

rpm: Start all the targets in %post

We need to have the sub-targets active for the ceph.target to be able
to propagate its calls to the services. If the sub-target is inactive,
the main target won't propagate the stop/restart calls.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 73f457986753b504fd088c1fa064608aa9d340c9)

rpm: implement scriptlets for the post-split daemon packages

This patch gives each of the ceph-{mds,mon,osd,radosgw} packages its own
%post, %preun, and %postun scriptlets dealing with the package's unit files.

The scriptlets of ceph-base are adapted to handle the ceph.target unit file
only.

The scriptlets of ceph-mon handle ceph-create-keys services in addition to ceph-mon.

The scriptlets of ceph-osd handle ceph-disk services in addition to ceph-osd.

Fixes: http://tracker.ceph.com/issues/14941
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 644aba9270714e5e231ac7d7e3437477837531eb)

systemd: enable all the ceph .target services by default

Some distros, like Fedora and openSUSE, have a policy that all services are
disabled by default.

This patch changes that default for the ceph.target and
ceph-{mds,mon,osd,radosgw}.target services.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 53b1a6799c2ce08acc8382e4aa996d1bde700316)

python_cephfs: rule out empty/None volume_id

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
(cherry picked from commit 73368069dd466cdbde1ebeae97771d603ce431f5)

python-cephfs: use rados namespace for data isolation.

As cephfs dir layout supports rados namespace in jewel, it would be
good to use rados namespace instead of pool, since it is free,
we always enforce namespace isolation for volumes.

The meaning of "data_isolated" flag changed a bit, it's no longer used
for security isolation, referring to physical isolation
(i.e. potentially using different OSDs).

Also, (de)authorize based on namespace.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
(cherry picked from commit 0e1d013ea690c18f7fa6e2a19500f854a9949091)

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>
s/ceph-deploy/ceph-tests, hence removing ceph-deploy folder

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>
  renamed the workunit and test to reflect reality and for better clarity

(cherry picked from commit 0fd358b877c611bf67c94b4a2dacbe65e17deae9)

fix ceph init script

commit 65963739cd6815b8008282c8f64cd64365662e60 have introduce a bug

test variables need to be quoted, or -n always return true, even if variables don't exist

(cherry picked from commit b581dd56bea09520c154611ecea91ebe399058d3)

test: Fix ceph-objectstore-tool test to run manually from src non-cmake

Fix init-ceph to handle it when CEPH_* values set

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5e8e03b4d030db57c93047551d5b92db0b604063)

Drop --setuser/--setgroup from osd prestart

These are not supported by /usr/lib/ceph/ceph-osd-prestart.sh,
resulting in warnings:

 ceph-osd-prestart.sh[23367]: getopt: unrecognized option '--setuser'
 ceph-osd-prestart.sh[23367]: getopt: unrecognized option '--setgroup'

--setuser and --setgroup are only needed for the ceph-osd process.

Signed-off-by: James Page <james.page@ubuntu.com>
(cherry picked from commit 74977f78840fe95b67788f3aa135ac69a2353c43)

osd: fix backwards min/max osd utilization

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
(cherry picked from commit 7529851f2a186501664461916f785da8456ad612)

10.2.0

Merge pull request #8636 from ceph/wip-rgw-leak

rgw leaks fixes

Merge pull request #8653 from theanalyst/doc/fix-multisite-pools

doc: rgw multisite, add pools section & minor cosmetic improvements

rgw: drop rest op reference in error path

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: rgw multisite, add pools section & minor cosmetic improvements

`Pool Configuration` was mentioned previously but didn't contain any
info, adding this section and minor cosmetic changes to a few headings
after looking at the html view.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #8564 from theanalyst/fix/15251

rgw_admin: improve period update errors

rgw_admin: improve period update errors

If we do a period update on a realm with multiple masters, we fail
stating invalid argument, improving this to suggest that we have
actually errored out because of multiple master zg in a realm, and
printing both of these zonegroups. RGWPeriodMap::update takes
CephContext as an argument, so that we can use ldout. Also dropping the
error messages reporting invalid arguments on period.update() as these
are handled within the function itself whenever it manifests itself.

Fixes: http://tracker.ceph.com/issues/15251
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #8645 from tchaikov/wip-fix-test.sh

test: set a default $CEPH_ROOT env variable

test: set a default $CEPH_ROOT env variable

in cf24535, we use $CEPH_ROOT to specify the $top_srcdir to unify
cmake and autotools, but this breaks ceph-qa-suite/tasks/workunit.py,
as it only clones the necessary qa/workunits directory, and does not
pass $CEPH_ROOT to the test scripts. so we need to set a default
$CEPH_ROOT if it is not set.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8612 from xiexingguo/xxg-wip-rpg

osd/ReplicatedPG: fix implementation of register_on_success()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: clean async rest ops in calling cr

Need to drop a reference to the async rest op when done.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: drop async cr reference on simple cr completion

Added a new callback for simple cr that is called anyway when done. Then
drop the reference to the async operation there and not at the destructor.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRealmWatcher::watch_restart() also unwatches watch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: stop cr manager in destructor

callers don't need to call it explicitly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8590 from liewegas/wip-jewel

doc/release-notes: jewel updates

Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8526 from dreamhost/wip-rgw-xmlns

rgw: Ensure xmlns is consistent on S3 responses.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #8563 from linuxbox2/rgw-file-unix

rgw: file unix

Merge pull request #8592 from zaitcev/wip-5073-multen-1

rgw: Add a test for multi-tenancy

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #8597 from oritwas/wip-rgw-remove-bucket-instance

rgw: handle no current_period in is_syncing_bucket_meta

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8601 from jmunhoz/fix-aws4-unsigned-payload

rgw: aws4: handle UNSIGNED-PAYLOAD under header auth

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

osd: fix noisy debug line

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8628 from SUSE/wip-ceph-common-pre-scriptlet

ceph.spec.in: terminate if statement in %pre scriptlet

Reviewed-by: Sage Weil <sage@redhat.com>

doc/release-notes.rst: clarify ceph UID/GID assignment on SUSE

This patch attempts to describe the semantics by which the ceph UID/GID
is assigned on SUSE distros.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/release-notes.rst: minor grammar and style corrections

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/release-notes: expand distro/arch section to include arm64 info

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: more detail about the ext4 checks

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: include changes in rcs

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: clean up jewel notes

- add notes for the rc's
- move upgrade info up to 10.2.0 (draft)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: fix rados scrub summary

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: terminate if statement in %pre scriptlet

One of the if statements in the ceph-common %pre scriptlet was not properly
terminated.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8620 from yehudasa/wip-rgw-region-fixes

rgw : region to zonegroup related fixes
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8617 from liewegas/wip-15098

debian: start ceph-mon-all and ceph-osd-all on package install …

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Tested-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #8613 from liewegas/wip-debug-map-trim

osd: some debug output for old osdmap trimming

common: typo-fix for osd_check_max_object_name_len_on_startup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Add a test for multi-tenancy

This is a very rudimentary test for now. For one thing, it
only creates a tenantized user, but does not even try to run
the radosgw and access it with Boto. And it already has several
XXX tags. But baby steps.

Two areas of exising code had to be fixed up.

 - We forgot to patch an argument through in RGWcluster.rgw_admin()

 - We must not process the arguments of a command through eval(1)
   in test-rgw-call.sh. This causes a double-eval with the function
   x() in test-rgw-common.sh, and as a result arguments with spaces
   or dollars cannot be used. There is no way to escape or quote.
   But we need the dollar for the tenant!

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>

common: typo-fix for osd_check_max_object_name_len_on_startup

Fixes a typo introduced in 02f5885114eab8ce21dedb8c3337422d89cc0d9b

Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #8560 from liewegas/wip-lfn-startup

osd: refuse to start if object name limits are not possible

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7984 from rldleblanc/wpq-starvation

common: WeightedPriorityQueue Re-Add Round Robin for Classes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Karol Mroz <kmroz@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8505 from liewegas/wip-15391

osd/ReplicatedPG: make handle_watch_timeout no-op if !active

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8545 from xiexingguo/xxg-wip-lfn

os/filestore: fix return type mismatch for lfn parse

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8555 from avnerbh/fixes_no_3

msg/xio: simple fixes

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #8570 from xiexingguo/xxg-wip-scrub

osd: clear requeue_scrub_on_unblock flag after requeue scrub

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8586 from liewegas/wip-15441

osd: fix watch reconnect race

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8619 from yehudasa/wip-rgw-jewel-fixes

rgw: fix binfo_cache, RGWShardCollectCR

rgw: cross rgw request need to send 'zonegroup' http header

instead of 'region'.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix get_zonegroup()

was passing in the wrong param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8281 from theanalyst/doc/multisite-rgw

doc: rgw adding a very basic multisite doc

doc: add multisite to index

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: rgw multisite documentation

Adding basic documentation for new multisite features, with an example
showing 3 zones in a zonegroup.

Fixes: http://tracker.ceph.com/issues/15247
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: binfo_cache is now a RGWRados member

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWShardCollectCR does not need a cct member

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7912 from ceph/wip-cmake-make-check

Wip cmake make check

Merge pull request #8614 from liewegas/wip-fsmap-underscore

mds/FSMap: use _ in key name

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

debian: start ceph-mon-all and ceph-osd-all on package install

This matches ceph-mds-all and ceph-all behavior.

Fixes: http://tracker.ceph.com/issues/15098
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8615 from theanalyst/fix-mailmap-self

mailmap: Abhishek Lekshmanan affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Abhishek Lekshmanan affiliation

Adding an entry to normalize the personal email id and work email id
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mds/FSMap: use _ in key name

No space or else it isn't valid XML.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8608 from swamireddy/contrib_list_10.1.2

mailmap update

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: some debug output for old osdmap trimming

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add option to disable startup check

Signed-off-by: Sage Weil <sage@redhat.com>

osd: refuse to start if configure object limits don't work

Fail early if the configured max object name cannot be stored on
the (FileStore) backend.  This avoids rude surprises down the
line when some random rados op get ENAMETOOLONG.

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: --short to set ext4-friendly object name and namespace limits

So devs can work on ext4.  Note that this will also mean s3-tests
will fail under vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: fix implementation of register_on_success()

We shall push the callbacks into on_success instead, I think.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mailmap: Jenkins name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Abhishek Lekshmanan name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Wei Qiaomiao affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Luo Runbing affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

cmake: link ceph_test_rados_api_list against libglobal

see 769c0aff

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Add cython_modules to `make check`

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: remove repeat OSD.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: CMake environment variables added to automake

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Building ceph-detect-init from source code

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Building ceph-disk from source code

Signed-off-by: Ali Maredia <amaredia@redhat.com>