mgr: fix locking in DaemonMetadata

Locks are great but it helps if you actually
bother acquiring them...

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: api_docs.py: Update for use outside Calamari

The ceph_state module is implemented in C++, and isn't available at
build time, which causes import failures when api_docs.py tries to
import calamari_rest in order to introspect it (MgrModule is eventually
imported, which in turn tries to import ceph_state, which fails unless
we stub it out).

Additional changes:

* Search calamari_rest.urls (not calamari_web)

calamari_web isn't available - it largely held static content in
Calamari, and in turn included calamari_rest.urls.  Here in ceph-mgr we
only have calamari_rest.

* note that --list-urls does nothing, apparently

* Pass actions to old as_view method

django-rest-framework 3.x raises TypeError if the actions argument isn't
passed to as_view().

* Use view().get_view_name() instead of metadata

Attempting to access view().metadata(None)['name'] results in
"AttributeError: 'super' object has no attribute 'metadata'".

* Use somewhat unfriendly field class names

django-rest-framework 3.x seems to have done away with type_label for
fields, so instead use the field's class name, which is somewhat
unfriendly, but arguably better than nothing.

* import global_instance to fix ImportError

This is necessary to avoid what seems to be weird import loops (if this
isn't present, we later get "ImportError: cannot import name
UserRequest" or similar).

* Make api_examples.json optional

This way we at least get bare docs with no examples, rather than no docs
at all.

* Explain how to generate API docs

Signed-off-by: Tim Serong <tserong@suse.com>

pybind/mgr: add the `rest` module

This is derived from what used to be Calamari.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: create mgr python module folder

Signed-off-by: John Spray <john.spray@redhat.com>

cmake: update for ceph-mgr

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: expose mgr commands

This is "tell mgr" at the moment but it should
be a slicker syntax later (ceph.in is awkward
to refactor just now)

Signed-off-by: John Spray <john.spray@redhat.com>

librados: expose mgr_command

This involves giving RadosClient an embedded
MgrClient.

Signed-off-by: John Spray <john.spray@redhat.com>

osd: embed a MgrClient

Signed-off-by: John Spray <john.spray@redhat.com>

mgrc: create MgrClient

The ceph-mgr equivalent to monclient

Signed-off-by: John Spray <john.spray@redhat.com>

common: accessors for list of perf counters

...and store the list by a string path, for
consumption by the world outside of integer
perf counter/subsystem IDs.

This is for consumption by MgrClient.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: create ceph-mgr service

Signed-off-by: John Spray <john.spray@redhat.com>

auth: add mgr service type

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: set up and run ceph-mgr

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add MgrMonitor

This is responsible for learning the locations of
mgrs, picking an active one, and sharing that
info with MgrClient consumers who subscribe to 'mgrmap'

Signed-off-by: John Spray <john.spray@redhat.com>

msg: introduce mgr entity_type_t

Mgr will probably ultimately have different auth caps,
so let's make it a separate entity up front instead
of e.g. pretending to be a mon.

Signed-off-by: John Spray <john.spray@redhat.com>

messages: add ceph-mgr messages

Signed-off-by: John Spray <john.spray@redhat.com>

client: refactor command handling

Create a CommandTable structure for places
in Client, MgrClient, Objecter where we do
basically the same kind of thing for sending
and tracking MCommands.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: fix handling bad cmd json

The 'save_exception' local would otherwise
be referenced before assignment during
exception handling.

Signed-off-by: John Spray <john.spray@redhat.com>

common/cmdparse: additional helpers

For dumping a cmdmap to a Formatter, and for
checking the prefix of a cmddesc.

Used by ceph-mgr when routing commands to python
modules.

Signed-off-by: John Spray <john.spray@redhat.com>

common: add mgr config settings

Signed-off-by: John Spray <john.spray@redhat.com>

common: add 'mgr' and 'mgrc' subsystems for logging

Signed-off-by: John Spray <john.spray@redhat.com>

common: move ceph_version up into collect_sys_info

...so that we get this piece of metadata from every
service uniformly.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add with_monmap to MonClient

Analogous to Objecter::with_osdmap, for safe access
to the map.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: fix missing osd metadata (again)

The JSON output was getting broken by continuing
in the wrong place.

Signed-off-by: John Spray <john.spray@redhat.com>

monc: remove unused AuthAuthorizeHandlerRegistry

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: fix warning when no CEPH_MAX_MDS set

This was a typo in 29535caf

Signed-off-by: John Spray <john.spray@redhat.com>

mon: drop encoded buffer after decode

This bufferlist was being kept around as a
class member but it didn't need to be.

Signed-off-by: John Spray <john.spray@redhat.com>

mon,mds: include 'addr' in daemon metadata

This is useful for correlating daemon metadata
to cluster maps.

Signed-off-by: John Spray <john.spray@redhat.com>

msg: typedef uint8_t entity_type_t

...rather than having a mixture of explicit __u8
and implicit enums/ints.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11137 from jcsp/wip-17173

mds: catch duplicates in DamageTable

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #11244 from xiexingguo/xxg-wip-bluestore-0928

os/bluestore: upgrade compression settings to atomics

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: upgrade compression settings to atomics

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11185 from dillaman/wip-17355

rbd-mirror: quiesce in-flight event commits before shut down

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11250 from liewegas/wip-bluestore-leaks

os/bluestore: fix leaks in our use of rocksdb

os/bluestore: do not leak when using EnvMirror

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: include EnvMirror leak fixes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11180 from liewegas/wip-mon-forward

messages/MForward: fix encoding features

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

messages/MForward: fix encoding features

We were encoding the message with the sending client's
features, which makes no sense: we need to encode with
the recipient's features so that it can decode the
message.

The simplest way to fix this is to rip out the bizarre
msg_bl handling code and simply keep a decoded Message
reference, and encode it when we send.

We encode the encapsulated message with the intersection
of the target mon's features and the sending client's
features.  This probably doesn't matter, but it's
conceivable that there is some feature-dependent
behavior in the message encode/decode that is important.

Fixes: http://tracker.ceph.com/issues/17365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11243 from liewegas/wip-bluestore-deferred-csum

os/bluestore: remove deferred_csum machinery

Merge pull request #11113 from yuyuyu101/wip-fix-filestore-clone

os/filestore: disable use of splice by default

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9652 from stiopaa1/osdc_ObjectCacher_constMember

osdc/ObjectCacher.h: add const to member functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11248 from mikulely/cleanup-vstart-doc

doc: cleanup outdated radosgw description

Merge pull request #11217 from ceph/wip-17395-rbdmap-permissions

rpm: fix permissions for /etc/ceph/rbdmap

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11240 from liewegas/wip-bluestore-assert

os/bluestore: add assert to compress_extent_map

Merge pull request #11245 from xiexingguo/xxg-wip-bluestore-bsearch-shard

os/bluestore: binary search specified shard

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11235 from xiexingguo/xxg-wip-bluestore-0926

os/bluestore: more cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11247 from majianpeng/msg-async-cleanup

msg/async: remove unused code.

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: remove unused code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #10980 from trociny/wip-16213

rbd-mirror: replicate dynamic feature updates

Reviewed-by: Jason Dillaman <dillamana@redhat.com>

librbd: potential null pointer dereference when requesting exclusive lock

m_require_lock_on_read should be cleared when holding owner_lock.

For safety, also check that exclusive_lock is not null.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: interlock image state machine and update features operations

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: use counter to track exclusive lock block/unblock requests

It's possible the watch/notify message is duplicated resulting in two
concurrent block_requests() call.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: replicate dynamic feature updates

Fixes: http://tracker.ceph.com/issues/16213
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async state machine to enable/disable image features

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: templetize journal StandardPolicy

It will be needed for unit tests with a mocked ImageCtx.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: allow to call append_disabled for DisabledPolicy

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async methods to enable/disable mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async method to check journal tag owner

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async image set flags method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: async methods to create/remove object map

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/rbd: alow to call set_features asynchronously

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/journal: async client_list method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11120 from renhwztetecs/renhw-wip-mds-server

mds/server: clean up handle_client_open()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11078 from ukernel/wip-17271

mds: remove fail-safe queueing replay request

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11225 from ukernel/wip-17392

client: protect InodeRef with client_lock

Reviewed-by: John Spray <john.spray@redhat.com>

doc: cleanup outdated radosgw description

radosgw support is on by default, as built-in Civetweb, set up Apache
is no necessary any more.

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

os/bluestore: add a counter to trace rejected compress ops

This is useful for choosing a more proper bluestore_compression_required_ratio.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make allocated space calc a little faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: binary search specified shard

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use std namespace for hex/dec output formatting

Because we use std::hex/dec everywhere in bluestore except these two places.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove fadvise_flags from WriteContext

Currently fadvise_flags is only used to check whether
buffered write is necessary, so there is no need
to keep it in the WriteContext as we have already
pre-calculated and kept the buffered field instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove dead local variables from BlueStore::_wctx_finish()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move nid assignment into BlueStore::_zero()

So other callers, such as BlueStore::_clone_range(),
won't do it twice.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove deferred_csum machinery

When we added this way back in d4f4fa0312d943dd0ce3c27f5fc56c7a753bb471,
we did not have our own buffer cache, and were relying
on the cache at the BlockDevice layer.  In that case,
we would have the problem of a partial wal overwrite
followed by another partial write that needed to read
the rest of the chunk.

However, now we have our own cache, and any data we write
in the _do_write_small() wal path will go into the cache,
which means we will never read the old data off of
disk and need the old csum values.

Remove this now-unnecessary kludge!

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: init g_ceph_context

The methods we're testing use it for dout.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add assert to compress_extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11238 from theanalyst/changelog-v10.2.3

doc/changelog: add v10.2.3

Merge pull request #10814 from liewegas/wip-bluestore-precondition

os/bluestore: precondition rocksdb/bluefs during mkfs

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

doc/changelog: add v10.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

os/bluestore: precondition rocksdb on mkfs

Write N bytes of garbage to the kv store on startup.  With rocksdb,
this ensures that our log files are preallocated.

This option needs to match up with the rocksdb tunables so that it
is enough data to start recycling log files.  For now, start with
128MB.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11234 from theanalyst/jewel-release-notes

doc/release-notes: v10.2.3 jewel

Merge pull request #11216 from liewegas/wip-bluestore-compress-map

os/bluestore: prevent extent merging across shard boundaries

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

doc/release-notes: more fixes to 10.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10445 from rjfd/wip-15259

journal: increase concurrency/parallelism of journal recorder

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/release-notes: v10.2.3 jewel

Adding release notes for the next jewel release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: add v10.2.3 to releases table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11226 from zealoussnow/wip-0926

doc: missing "make vstart" in quick_guide.rst

Merge pull request #11228 from stiopaa1/osd_pg_removeUnneededCount1

osd/PG.cc: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11229 from liewegas/wip-compressor

cmake: remove more autotools hacks

Reviewed-by: Ali Maredia <amaredia@redhat.com>

ceph-rest-api: drop autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: drop autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

detect-build-env-vars.sh: drop autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: remove autotools detection

Signed-off-by: Sage Weil <sage@redhat.com>

doc: remove old .libs reference

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression_*: remove lib dir hackery

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10143 from majianpeng/bluefs-fix

os/bluestore/BlueFS: don't inc l_bluefs_files_written_wal if overwrite.

Merge pull request #11138 from jcsp/wip-17236

mds: handle blacklisting during journal recovery

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: missing "make vstart" in quick_guide.rst

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #11220 from stiopaa1/mds_mdsdaemon_CMDSTickclass

mds/MDSDaemon: move C_MDS_Tick class to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11224 from wjwithagen/patch-13

osd-fast-mark-down.sh: fix typo in variable assignments