common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars

If a kubernetes pod spec specifies a limit of X, then the pod gets both
the limits.memory and requests.memory resource fields set, and rook passes
those as POD_MEMORY_LIMIT and POD_MEMORY_REQUEST environment variables.

This is a problem if only the limit is set, because we will end up
setting our osd_memory_target (and, in the future, other *_memory_targets)
to the hard limit, and the daemon will inevitably reach that threshold
and get killed.

Fix this by also looking at the POD_MEMORY_LIMIT value, and applying the
ratio (default: .8) to it, and setting our actual target to the min of
that and the POD_MEMORY_REQUEST.

Also, set the "default" target to ratio*limit, so that it will apply in
general when no request is specified.

When both request and limit are 10M, we then see

        "osd_memory_target": {
            "default": "800000000000",
            "env": "800000000000",
            "final": "800000000000"
        },

In a more "normal" situation where limit is 10M and request is 5M, we get

        "osd_memory_target": {
            "default": "800000000000",
            "env": "500000000000",
            "final": "500000000000"
        },

If only limit is specified (to 10M), we get

        "osd_memory_target": {
            "default": "800000000000",
            "final": "800000000000"
        },

Fixes: https://tracker.ceph.com/issues/41037
Signed-off-by: Sage Weil <sage@redhat.com>

common/config: let diff show non-build defaults

Notably, this includes things from set_val_default(), which may be set
at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: do no include multiple 'default' values

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27859 from jan--f/update-bluestore-cache-doc

doc: update bluestore cache settings and clarify data fraction

Merge pull request #29475 from tchaikov/wip-cmake3-for-xenial

install-deps.sh: use chacra for cmake repo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29337 into master

* refs/pull/29337/head:
mon: enable telemetry module by default
mgr/telemetry: force re-opt-in if the report contents change
mgr/telemetry: less noise in the log
mgr/telemetry: wake up serve on config change
mgr/telemetry: track telemetry report revisions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29457 from tchaikov/wip-consolidate-run-tox

cmake: one run_tox.sh to rule them all

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

script/run_tox.sh: add --help option

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29492 into master

* refs/pull/29492/head:
doc: Fix for new ceph-devel mailing list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

cmake: pass tox envs to run_tox.sh explicitly

* add two optional keyword parameters to `add_tox_test()`
* use the default `TOX_PATH` parameter instead passing it explicitly,
  as it is always `${CMAKE_CURRENT_SOURCE_DIR}`
* drop the code to guess the tox envs in `run_tox.sh`, and always pass
  them explicitly in CMake script using the `--tox-envs` argument.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29471 from tchaikov/wip-googletest

googletest: pick up change to suppress CMP0048 warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: Fix for new ceph-devel mailing list

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29458 from mikulely/rgw-drop-flush

rgw: drop dead flush_read_list declaration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27675 from jan--f/c-v-console-logging-to-stderr

ceph-volume: print most logging messages to stderr

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

cmake: move tests to where they are implemented

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not add gtest-parallel_ext if already added

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use script/run_tox.sh for running tox for dashboard

* adapt script/run_tox.sh to use different envs for dashboard's tox
  test.
* use script/run_tox.sh for running dashboard's tox test
* remove pybind/mgr/dashboard/run-tox.sh

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/AddCephTest.cmake: make ${name}-venv a target

so we can prepare the venv for tox with `make tests`. and it's optional,
as run_tox.sh will check for it and setup a venv if it's not around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

src: remove unused run-tox.sh

they are not used anymore after b9c9f74f

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29393 from ifed01/wip-ifed-bluestore-header-cleanup

os/bluestore: remove unused _tune_cache_size() method declaration

Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: update npm packages (#28343)

mgr/dashboard: update npm packages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29403 from rhcs-dashboard/wip-41013-master

install-deps.sh: remove failing error catching

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29476 from tchaikov/wip-test/journal-close-object

test/journal: always close object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/journal: always close object

otherwise when we destruct `TestObjectRecorder::Handler::lock` or
`journal::JournalRecorder::m_object_locks`, it/they would be still
being waited by some condition variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29474 from wjwithagen/wjw-fix-rbd_ggate_Server.cc

rbd-ggate: fix compile errors from ceph::mutex update

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-ggate: fix compile errors from ceph::mutex update

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

install-deps.sh: use chacra for cmake repo

two reasons:

- do not rely on kitware repo so we have better control of the
  cmake3: the `debian` directory is tracked by
  https://github.com/tchaikov/ceph-cmake
- chacra repo also offers aarch64 build of cmake3

Signed-off-by: Kefu Chai <kchai@redhat.com>

googletest: pick up change to suppress CMP0048 warning

newer cmake complains about googletest should have set CMP0048

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29472 from wjwithagen/wjw-fix-src/osdc/ObjectCacher.cc

osdc: Fix a missing ':' for the correct namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc: Fix a missing ':' for the correct namespace

Clang complains:
/home/jenkins/workspace/ceph-master/src/osdc/ObjectCacher.cc:1848:22: error: no viable constructor or deduction guide for deduction of template arguments of 'unique_lock'
    std::unique_lock l{lock, std:adopt_lock};

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29322 from rzarzynski/wip-objclass-cleanup

objclass, osd: clean up the cls-host interface. Turn ClassHandler into singleton

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29113 from tchaikov/wip-ceph-mutex

src/: s/Mutex/ceph::mutex/

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/Mutex: remove Mutex

as it's replaced by `ceph::mutex`

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Cond.h: remove Cond

as it's replaced by ceph::condition_variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: support is_locked() in non-debug mode

`PG` print details info in the prefix of logging messages if the PG is
being locked by current thread. but `ceph::mutex` is an alias of
`std::mutex` in non-debug mode, so neither `mutex::is_locked_by_me()` nor
`mutex::is_locked()` is supported in non-debug mode. to continue supporting
this feature, `PG::locked_by` is added to memorize the thread id of the owner
of the lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_context.cc: include necessary header

for using functions like `lockdep_unregister_ceph_context()`

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/testmsgr: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/compressor: do not include unused header

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/common/Throttle: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/obj_bencher: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: remove unused forwarde declaration

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librados_test_stub: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephfs: remove unused #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls_kvs: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: remove unused #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

fusetrace: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rbd_nbd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rbd_ggate: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rbd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph_dedup_tool: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/xattr_bench: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_snap_mapper: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/omap_bench: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/msgr: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/admin_socket: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

kv: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/immutable_object_cache: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/cephfs: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/os/filestore: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rbd_mirror: s/Mutex/ceph::Formatter/

* s/Formatter/ceph::Formatter/ in tools/rbd_mirror/ImageSyncThrottler,
  as we cannot rely on `using ceph` or `using ceph::Formatter` in
  some other included header files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/perf_local: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/osdc: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/objectstore: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/mon: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/journal: s/Mutex/ceph::mutex/

as we cannot create a `smart_pointer<ceph::mutex>` due to limitation of
`ceph::make_mutex()`, we need to refactor `TestObjectRecorder` to remove
its `TearDown()` method, as when `TearDown()` is called, all local
variables have been destroyed, including the `ceph::mutex` instances. so
we need to introduce a helper of `ObjectRecorderFlusher`, to flush the
objects before `mutex` instances are destroyed. to simplify the
interfaces, it flushes in its dtor. so its lifecycle should be shorter
than those of mutexes. that's why, mutexes are created before `flusher`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: s/Mutex/ceph::mutex/

* FutureImpl::m_lock is an exception. as, before this change, the lock
  was initialized like `m_lock("FutureImpl::m_lock", false, false)`, see
  the declaration of
  `Mutex(const std::string &n, bool r = false, bool ld=true, bool bt=false)`
  so `m_lock` is actually not using the extra features offered by
  `Mutex` like runtime lockdeps check. and `mutex_debugging_base` does
  not allow us to disable lockdeps individually. but it does use the `is_locked()`
  method. so instead of using `ceph::mutex` directly, a cutomized
  `ceph::mutex` is added for `CEPH_DEBUG_MUTEX` build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/mutex_debug: add lockdep parameter to mutex's ctor

so we can disable lockdep when constructing a ceph::mutex. otherwise
it's difficult to use the locks from standard library and to use
`ceph_assert(ceph_mutex_is_locked(lock))` at the same time.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/mutex_debug.h: pass "no_lockdep" to try_lock()

there is chance that we want to disable lockdep, so we shold pass
`no_lockdep` down to `try_lock()` in `lock()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/Context: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_mutex: add more ceph_mutex_is_*() helpers

we have caller sites like

ceph_assert(!client->client_lock.is_locked_by_me())

in `src/client/Client.cc` and `src/librados/RadosClient.cc`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ceph_mutex: improve ceph::shared_mutex

add helpers for supporting `ceph_mutex_is_locked()` and
`ceph_mutex_is_wlocked()` macros

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: add necessary #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/RWLock: add RWLock::{lock,unlock}_shared()

so RWLock can be used with std::shared_lock,

rgw is using RWLock'prioritize_write feature, which is not offered by
std::shared_mutex, but we want to unify the way we use mutex, so add
these helper functions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/HeartbeatMap.cc: include necessary header

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Cond.h: include necessary header

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/rgw: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

os: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/utime: add utime_t<>(duration)

to ease the conversion from chrono durations to utime_t

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/C_Lock: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/condition_variable_debug: fix wait_until()

to_timespec() is not a member function of time_point, it is a member
funcion of real_clock / coarse_real_clock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/timer: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: s/Mutex/ceph::mutex/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29291 from tchaikov/wip-cmake-cleanups

cmake: require CMake v3.10.2

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

install-deps.sh: remove failing error catching

Fixes: https://tracker.ceph.com/issues/41013
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #29408 from neha-ojha/wip-bs-fsck-mnt

common/options.cc: change default value of bluestore_fsck_on_mount_deep to false

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>