mgr/dashboard: Add option to disable SSL

Although is preferred and should be enabled by default users might
want to disable SSL as the dashboard might be running behind a proxy
which terminates the SSL.

Fixes: https://tracker.ceph.com/issues/24674
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge PR #22784 into master

* refs/pull/22784/head:
doc: fix Ceph FS
doc: new label and glossary url
doc: drop linking to itself

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22144 into master

* refs/pull/22144/head:
cephfs-journal-tool: Fix purging when importing an zero-length journal.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22455 into master

* refs/pull/22455/head:
qa/ceph-volume: add a test for put_object_versioned()
ceph-volume-client: allow atomic updates for RADOS objects

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22543 into master

* refs/pull/22543/head:
qa/tasks/cephfs: add test for ceph_abort_conn()
libcephfs: introduce ceph_get_instance_id()
client: introduce ceph_abort_conn()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22961 into master

* refs/pull/22961/head:
doc/cephfs: make scary DR bits less prominent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22921 from ceph/wip-default-ceph-volume

qa/tests: remove ceph-disk tests from ceph-deploy and default all tests to use ceph-volume

Merge PR #22940 into master

* refs/pull/22940/head:
mon/OSDMonitor: add 'osd repeer <pgid>' command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22949 into master

* refs/pull/22949/head:
cmake: cleanup FindCython.cmake

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #22954 into master

* refs/pull/22954/head:
global/global_init: fix stdout/stderr/stdin closing for daemonization

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #22931 from tchaikov/wip-find-py-interp-for-gtest-parallel

cmake: find a python2 interpreter for gtest-parallel

Reviewed-by: Erwan Velu <erwan@redhat.com>

doc/cephfs: make scary DR bits less prominent

I'm sure people will still find them, but let's at least
force people to click through one more time to get to the
commands that can damage your cluster.

Also, the ".. danger" directive at the top of the page
wasn't actually getting special formatting, so I changed
it to a ".. warning" which is red.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22944 from falcon78921/wip-doc-rbdcmds

doc: fixed grammar in restore rbd image section

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add 'osd repeer <pgid>' command

Selecting force peering on a single PG.  In reality this probably induces
*2* interval changes.

Note that in the case of a single OSD cluster we can't actually force a
repeer on a single PG because the pg_temp code is pretty robust about
filtering out redundant or meaningless changes, so we can't pg_temp our
way into a new interval if there are no other OSDs to switch to and the
code also prevents an empty pg_temp.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22711 from fghaas/wip-doc-mimic-upgrade-prereq

 doc: Mention PURGED_SNAPDIRS and RECOVERY_DELETES in Mimic release notes

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

global/global_init: fix stdout/stderr/stdin closing for daemonization

The global_init_postfork/prefork helpers close stdout/stdin/stderr on
fork and reopen /dev/null in their place.  This ensures that if later
code writes to those descriptors (e.g., a stray cout or cerr usage) the
output/input will go nowhere instead of interfering with some other open
fd.

However, with the use of preforker, there are other threads running when
these helpers are run, which means we can race with, say, filestore
opening an object file and end up sending log output there.

Fix by atomically replacing the fds with the dup2(2) syscall, which
will implicitly close and reopen the target fd in an atomic fashion.  This
behavior is present on both Linux and FreeBSD.

Fixes: http://tracker.ceph.com/issues/23492
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'private/wip-mon-snap-caps'

Merge pull request #22867 from smithfarm/wip-ccache-tweak

run-make-check.sh: ccache goodness for everyone

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge remote-tracking branch 'gh/master'

cmake: find python2 interpreter for gtest-parallel

gtest-parallel only supports python2 at this moment, so find it
a python2 interpreter if it is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'private/wip-cx'

 Conflicts:
src/include/ceph_features.h
src/msg/async/AsyncConnection.cc
src/msg/async/AsyncConnection.h

- SERVER_NAUTLIUS feature recently added to default
- msgr2 bool
- ceph::shared_ptr -> std::shared_ptr

Merge pull request #22778 from trociny/wip-mock-warnings

test/librbd: fix gmock warning in TestMockIoImageRequestWQ.AcquireLockError

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22903 from tspmelo/fix-e2e-script

mgr/dashboard: Fix e2e script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

run-make-check.sh: fill in missing DRY_RUN and add doc hint

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: reset trap after sourcing install-deps.sh

Since we source install-deps.sh, it replaces our trap with its own,
so we have to reinstate our trap after it finishes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cleanup: run-make-check.sh: no space between -D and name of cmake option

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: gtest-parallel only with Python 2

Fixes: http://tracker.ceph.com/issues/24817
Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: inform user of make -j setting

This can help the user figure out why the build is so slow.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: cleanup: wordsmith a comment in run-make-check.sh

Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: ccache goodness for everyone

Since run-make-check.sh already ensures that ccache is installed,
it makes sense to let everyone benefit from the ccache
tweaks introduced by 4cb5a590537a9caaf61db42ce8ea123d2ab961f3

Note 1: The previous solution using "date" would cause build tools to reset
their timestamps after 24 hours, on subsequent runs of run-make-check.sh.
In order to maximize ccache effectiveness, this commit sets SOURCE_DATE_EPOCH
to a fixed value: the number of seconds elapsed since the Unix epoch as at
January 1, 2000 (chosen to commemorate Y2K armageddon).

Note 2: this commit introduces "set -e". This was actually in effect
before, via "source install-deps.sh". Better to make it explicit.

Fixes: http://tracker.ceph.com/issues/24777
Signed-off-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: drop workaround for undeclared jq build dependency

jq build dependency is properly declared now.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22898 from majianpeng/bluestore-remove-redundant-fault-range

os/bluestore: remove redundant fault_range.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22893 from jcsp/wip-misc

mgr: Miscellaneous small mgr fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21962 from socketpair/perf

osd: Additional fields for osd "bench" command

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup FindCython.cmake

print out error message if cython module is not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #22836 into master

* refs/pull/22836/head:
qa: remove unused distro dir

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge PR #22741 into master

* refs/pull/22741/head:
qa: fix ffsb cd argument
qa: fix cd argument during iogen unpack

Reviewed-by: Neha Ojha <nojha@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22766 into master

* refs/pull/22766/head:
mds: refactor const ref return to avoid mem alloc

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22740 into master

* refs/pull/22740/head:
qa: create common conf for all cephfs suites
qa: remove wrongly created random distro conf

Reviewed-by: Zheng Yan <zyan@redhat.com>

doc: fixed grammar in restore rbd image section

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Merge pull request #22922 from smithfarm/wip-argparse-py3

tests: make test_ceph_argparse.py pass on py3-only systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22912 from tchaikov/wip-cmake-cleanup

cmake: remove workarounds for supporting cmake 2.x

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fix Ceph FS

s/Ceph FS/CephFS

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: new label and glossary url

Let the other docs link Ceph Filesystem glossary from outside.
If the user wants then let him visit the Ceph Filesystem doc(s)
from the glossary.

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: drop linking to itself

drop unnecessary linking to its own smaller definition.

Signed-off-by: Jos Collin <jcollin@redhat.com>

tests: make test_ceph_argparse.py pass on py3-only systems

On systems that do not have Python 2 installed, this test fails to
import StringIO.

Fixes: http://tracker.ceph.com/issues/24816
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tests: remove ceph-disk tests and default all tests to use ceph-volume

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

doc: Fix minor documentation issues in Mimic release notes

Small grammar glitches, typos, minor style issues.

Signed-off-by: Florian Haas <florian@citynetwork.eu>

doc: Mention PURGED_SNAPDIRS and RECOVERY_DELETES in Mimic release notes

71d1dcbb98bf1b3c7953c13a1165ba1cac7e7fd0 started enforcing the
PURGED_SNAPDIRS osdmap flag before am upgraded Mon could join the
quorum.

9b80b14783ef895390b4153320078661627f373d extended that check to also
include RECOVERY_DELETES. The Mimic release notes do not mention these
flags as prerequisites for an upgrade beyond Luminous. That creates an
obvious issue for users who skipped Luminous in production, and now
want to upgrade from Jewel to Mimic in, say, a weekend.

Update the release notes to include those flags as prerequisites for a
Luminous to Mimic upgrade, explain how users can make sure that they
are set, and also give users a one-liner to fix up their PGs in a
pinch, if they need to.

Signed-off-by: Florian Haas <florian@citynetwork.eu>

Merge PR #22863 into master

* refs/pull/22863/head:
rgw: fix gc may cause a large number of read traffic

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20779 from oritwas/wip-rgw-23257

rgw: radogw-admin reshard status command should print text for reshard status

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #22721 from linuxbox2/wip-rgw-23099

rgw: escape markers in RGWOp_Metadata_List::execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #22745 into master

* refs/pull/22745/head:
debian/control: require sudo
ceph.spec: require sudo
ceph.spec: install etc/sudoers.d/ceph-osd-smartctl
debian: install etc/sudoers.d/ceph-osd-smartctl
sudoers.d/ceph-osd-smartctl: allow user ceph to run smartctl as root

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge PR #22858 into master

* refs/pull/22858/head:
osd/PrimaryLogPG: rebuild attrs from clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #22870 into master

* refs/pull/22870/head:
mgr: replace iteritems with six.iteritems (mgr_module.py)
mgr: replace iteritems with six.iteritems (crash)
mgr: replace iteritems with six.iteritems (telegraf)
mgr: replace iteritems with six.iteritems (status)
mgr: replace iteritems with six.iteritems (restful)
mgr: replace iteritems with six.iteritems (influx)
mgr: replace iteritems with six.iteritems (balancer)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #22871 into master

* refs/pull/22871/head:
deb/rpm: add python-six as build-time and run-time dependency

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #22873 into master

* refs/pull/22873/head:
os/bluestore: fix incomplete faulty range marking when doing compression

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: remove workarounds for supporting cmake 2.x

and bump up the required cmake version to 3.5.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22896 from tchaikov/wip-cmake3

deb,rpm,do_cmake: switch to cmake3

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

os/bluestore: remove redundant fault_range.

In fact, _do_read already did fault_range for the same offset and
length.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #22619 from zhangsw/fix-rgw-deletemarker-bug

rgw: The delete markers generated by object expiration should have owner attribute.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #22145 from Jeegn-Chen/wip-issue-24223

rgw: reject invalid methods in validate_cors_rule_method

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22452 from theanalyst/wip-placement-target-warn

rgw: warn if zone doesn't contain all zg's placement targets

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22614 from tianshan/fix_24563

rgw: fix copy response header etag format not correct

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22623 from zhangsw/fix-rgw-lifecycle-rule-num

rgw: Limit the number of lifecycle rules on one bucket.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #22635 from zhangsw/fix-rgw-admin-objectexpire-bug

rgw: "radosgw-admin objects expire" always returns ok even if the process fails.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

mgr/dashboard: Fix e2e script

Angular 6 changed the cli and how you can configure the e2e cmd.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr: reduce log spam

Several things at level 1 that shouldn't have been, especially
the big dump of all possible commands every time we handle one.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: handle long-named mgr modules

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: improve logging on handle_command path

Signed-off-by: John Spray <john.spray@redhat.com>

deb,rpm,do_cmake: switch to cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/control: require sudo

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: require sudo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22878 from falcon78921/wip-falcon78921-docs-install

doc: fixed minor typo in Debian packages section

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: fixed minor typo in Debian packages section

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

mgr: replace iteritems with six.iteritems (mgr_module.py)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (crash)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (telegraf)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (status)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (restful)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (influx)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr: replace iteritems with six.iteritems (balancer)

Fixes: http://tracker.ceph.com/issues/24779
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22874 from ddiss/bluestore_docs

doc/bluestore: fix minor typos in compression section

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix incomplete faulty range marking when doing compression
GC.

Under some scenarios GC might process an extent range where some inner extents are left untouched by GC (as there is no need for that). Hence GC doesn't invaliate these inner extents with fault_range call. If untouched extents are mapped to unloaded shards it results in subsequent assertion on o->extent_map.dirty_range() call.
The solution is to invalidate the whole extent range when doing GC.

Fixes: https://tracker.ceph.com/issues/23540
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #22617 from tianshan/fix_http_err_return

rgw: set cr state if aio_read err return in RGWCloneMetaLogCoroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22793 from kshtsk/wip-add-opensuse-leap

Add new OpenSUSE Leap id for install-deps.sh

Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: radogw-admin reshard status command should print text for reshard status

Fixes: http://tracker.ceph.com/issues/23257
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

doc/bluestore: fix minor typos in compression section

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #22829 from tspmelo/fix-24757

mgr/dashboard: Fix regression on rbd form component

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22762 from ricardoasmarques/wip-toolbar-text

mgr/dashboard: Remove top-right actions text and add "About" page

Reviewed-by: Kanika Murarka <murarkakanika@gmail.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

deb/rpm: add python-six as build-time and run-time dependency

python-six is used by dashboard's controllers. please note,
${python:Depends} is able to fill the installation dependencies for
debian package. so python-six is not added to Requires of ceph-mgr.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20133 from smithfarm/wip-redundant-recommends

build/ops: deb: drop redundant ceph-common recommends

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: fix gc may cause a large number of read traffic

fix http://tracker.ceph.com/issues/24767

Signed-off-by: Xin Liao <liaoxin01@baidu.com>

Merge pull request #22847 from ErwanAliasr1/evelu-ccache2

run-make-check.sh: Adding ccache tuning for the CI

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: create common conf for all cephfs suites

This will be followed by removing common CephFS configurations in the
ceph.conf.template in teuthology.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22703 into master

* refs/pull/22703/head:
doc/mgr: add doc for crash mgr module
qa/suites/rados/mgr: Add test_crash
qa/tasks/mgr: add test_crash, call from test_module_selftest
qa/tasks/{ceph_manager.py,vstart_runner.py}: allow kwargs in raw_*
mgr/pybind/crash: handle crashdumps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #22704 into master

* refs/pull/22704/head:
osd/PG: do not send notify to empty peer
osd/PG: do not assume delete event means found_missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>