rbd: recognize lock_on_read option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11218 from ceph/wip-getuid

Client: pass "UserPerm" struct everywhere for security checks

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11300 from liewegas/wip-bluestore-onode-add

os/bluestore: fix onode cache addition race

os/bluestore: fix onode cache addition race

Two threads may try to add the same onode to the cache.
This is rare, but allowed (in the case of the meta
collection).  If that happens, one of them will just
back off and use the winning onode ref.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11298 from liewegas/wip-mgr-fix

mgr/MgrClient: fix ms_handle_reset
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/MgrClient: fix ms_handle_reset

Return false because we don't handle the reset.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11292 from rzarzynski/yj-wip-swift-build_token-cleanup

rgw/swift: remove redundant assignment operation

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11019 from liewegas/wip-autotools-must-die

autotools: remove a few other remaining traces

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11294 from trociny/fixup-qa-rbd_mirror-metadata_set

qa/workunits/rbd: set image-meta on primary image and wait it is replicated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11288 from stiopaa1/osd_replicatedpg_moveCOSDOnAppliedClass

osd/ReplicatedPG:move C_OSD_OnApplied class to cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11289 from stiopaa1/mds_sessionmap_moveStatementUnderAssertPushPv

mds/SessionMap.h: change statement to assertion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/rbd: set image-meta on primary image and wait it is replicated

After recently added image metadata replication it is not possible any
more to update it on non-primary image.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

run-make-check.sh: use cmake exclusively

..and drop the separate cmake script

Signed-off-by: Sage Weil <sage@redhat.com>

make-debs.sh: use make-dist

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/swift: remove redundant assignment operation

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #11086 from bassamtabbara/wip-ec-simd-runtime-detection

erasure-code: Runtime detection of SIMD for jerasure and shec

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds/SessionMap.h: change statement to assertion

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

osd/ReplicatedPG:move C_OSD_OnApplied class to cc

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11285 from liewegas/wip-mgr-rpm

ceph.spec: fix ceph-mgr version requirement

Reviewed-by: Dan Mick <dmick@redhat.com>

client: in UserPerm, correctly use delete[] and don't leak gids on copy

gids is an array. We also need to clean it up when deep_copy()ing, which
requires initializing our member fields in the copy-constructor.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph.spec: fix ceph-mgr version requirement

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11282 from liewegas/wip-bluestore-gc-fix

os/bluestore: fix gc when blob extends past eof

os/bluestore: fix gc when blob extends past eof

This can happen if we aren't referencing the bob.  Don't try to
read what isn't logically referenced.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11279 from liewegas/wip-bluestore-gc

os/bluestore: compaction fixes

os/bluestore: fix compaction left-side length

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: more concise SharedBlob print

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cosmetic gc cleanups

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11271 from stiopaa1/osd_repicatedpg_addOverride

osd/ReplicatedPG.h: add override to virtual funct

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11239 from liewegas/wip-bluestore-max-blob

os/bluestore: add max blob size; fix compressed min blob size logic

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

os/bluestore: set min compressed blob size floor at 2x min_alloc_size

Otherwise there's not point in compressing--we won't be able to
allocation a smaller space to store it.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: introduce bluestore_max_blob_size

This is enforced for compressed and non-compressed blobs alike.
We keep the separate compressoin min and max tunables since these
may be adjusted on a per-pool basis as well.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: adjust compression blob sizes down

These were way too big.  64KB is enough to get good compression,
and 4MB is way too big.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob depth encoding

1) Fix the flag constnat
2) Redefine the meaning to be more intuitive.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11232 from Roushan81/wip-bluestore-blob-gc

os/bluestore: garbage collect partially overlapped blobs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11263 from liewegas/wip-bluestore-reshard-slop

os/bluestore: sloppy reshard boundaries to avoid spanning blobs

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #11275 from xiexingguo/xxg-wip-make-audit-buffer-option

os/bluestore: drop redundant call of get_blob

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: drop redundant call of get_blob

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11268 from liewegas/wip-bluefs-aio-leak

os/bluestore/BlueFS: release completed aios

Merge pull request #11005 from jazeltq/add-sparse-read

rbd-mirror: Add sparse read for sync image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11168 from trociny/wip-16212

rbd-mirror: replicate image metadata settings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11274 from xiexingguo/xxg-wip-bluestore-0930

os/bluestore: still more cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11136 from jcsp/wip-16016

mds: populate DamageTable from scrub and log more quietly

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #11270 from ceph/wip-17443

rgw: set correct instance on the object
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10328 from jcsp/wip-ceph-mgr

Create ceph-mgr

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: avoid comparing object key and name twice

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use id of shard_id_t for encoding

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill struct DeferredCsum

Which is dead, and kill some other dead lines too.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/ReplicatedPG.h: add override to virtual funct

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: add garbage collection

Signed-off-by: Roushan Ali <roushan.ali@sandisk.com>

Merge pull request #11267 from liewegas/wip-bluestore-ref-map

os/bluestore/bluestore_types: uint64_t for ref_map

rgw: set correct instance on the object

Fixes: http://tracker.ceph.com/issues/17443
This was broken by commit bc840afafdfe5e528e5c1b711e71420ac3cb5a67

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore/BlueFS: be less complainy about fallback allocations

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: use 1TB block device

Big enough to get into 32-bit LBAs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: uint64_t for ref_map

We use this to track raw extents on disk for SharedBlob.  We
also use it for the in-memory Blob with the blob namespace, but
using uint64_t intead of uint32_t doesn't hurt us there.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: release completed aios

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11266 from stiopaa1/osd_passSharedPtrByConstRef

osd: pass shared_ptr by const reference

Reviewed-by: Sage Weil <sage@redhat.com>

Fix test-erasure-code-plugins.sh test on ARM

The ISA plugin is not available on ARM, this test now excludes it.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

erasure-code: improve non-regression testing

Pickup new ec corpus with improved testing.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

doc: Updated PendingReleaseNotes.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

erasure-code: Backward compatibility with legacy EC plugins

Resurrected jerasure_generic, jerasure_sse3, jerasure_sse4, jerasure_neon,
shec_generic, shec_sse3, shec_sse4 and shec_neon. These all are exact
copies of the new jerasure and shec plugins that support SIMD detection.

Moved EC preload code in ceph-mon and ceph-osd to a central location, added
warning when preloading legacy plugins.

OSMonitor::get_erasure_code and OSDMonitor:normalize_profile will now check
if legacy EC plugins are used and log a warning.

Added tests to check that warnings make it to the log.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

erasure-code: Move the EC factory code into separate header files

This helps the EC factories be invokable directly for testing
and other uses.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

cmake: normalize use of SIMD flags

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

erasure-code: Remove SIMD flavors for jerasure and shec

By switching to a new gf-complete with SIMD runtime detection, we can now remove all the different flavors of jerasure and shec. This simplifies deployment and configuration of erasure coding, enables hetergenous OSDs, and enables us to take advantage of new performance improvements in jerasure without config/build changes.

This commit removes flavors from cmake, removes ErasureCodePluginSelect___, and fixes unit tests. There is now a single plugin for jerasure and a single plugin for shec.

SIMDExt.cmake was changed so that its a little more generic, and is not polluted with gf-complete specific CFLAG defines. The #define for SIMD instruction were based on gf-complete.

I also added a small init helper for jerasure that has code that was common between jerasure and shec.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

erasure-code: Pickup gf-complete v3 with runtime SIMD detection

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

build/ops: remove autoconf leftover

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

remove do_autogen.sh

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/rest: remove unused DebugJob

This dates back from the Calamari/Salt days.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr/rest: fix "sync_object" API endpoint

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: include cython mods in PYTHONPATH

This is useful for ceph-mgr because it means that
any in-mgr modules that want to use e.g. rbd
can find it when running in a vstart cluster.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: remove hardcoded site-packages

Signed-off-by: John Spray <john.spray@redhat.com>

packaging: add debian packaging for ceph-mgr

...at least, try to!  This is a copy-paste of the ceph-mds
packaging with a search and replace mds to mgr.

Signed-off-by: John Spray <john.spray@redhat.com>

systemd: autogenerate ceph-mgr key during daemon startup

This is a hack to inject a key for the mgr daemon, using whatever
key already exists on the mon on this node to gain sufficient
permissions to create the mgr key.  Failure is ignored at every
step (the '-' prefix) in case someone has already used some other
trick to set everything up manually.

Signed-off-by: Tim Serong <tserong@suse.com>

systemd: encourage ceph-mgr to start in sync with ceph-mon

This change introduces the following behaviour:

- When ceph-mon starts, it will try to start ceph-mgr with the same
  instance id (Wants=), but will *not* fail to start if ceph-mgr
  doesn't start (i.e. the mon still works as it always did).
- ceph-mgr will start After= ceph-mon, and will stop and start when
  ceph-mon stops and starts, because it's PartOf= ceph-mon.

If you don't want ceph-mgr to run on the mons, you need to mask the
service, i.e. `systemctl mask ceph-mgr@INSTANCE`.  Hostnames are
typically instance names, so `systemctl mask ceph-mgr@$(hostname)`
should suffice if you wish to disable ceph-mgr on the mons.

Signed-off-by: Tim Serong <tserong@suse.com>

ceph.spec.in: add ceph-mgr package

This adds ceph-mgr following the same pattern used for ceph-mon,
ceph-mds, ceph-osd.  I've removed the boost-python BuildRequire
for SUSE distros, because we don't ship that (the relevant bits
are already included in boost-devel).

NOTE: This gives an installable ceph-mgr, but not necessarily a
runnable ceph-mgr.  This is because ceph-mgr needs the following
python packages:

- CherryPy
- Django
- gevent
- python-dateutil
- djangorestframework

Not all of these are available as distro packages.  For example,
on openSUSE Tumbleweed, we don't have djangorestframework available
in the base distro, and our Django version is 1.9, which seems to
be incompatible with ceph-mgr.  I'm not sure of the situation on
other distros.

Signed-off-by: Tim Serong <tserong@suse.com>

systemd: add ceph-mgr service and target files

Signed-off-by: Tim Serong <tserong@suse.com>

cmake: install ceph-mgr and its python modules

Signed-off-by: Tim Serong <tserong@suse.com>

pybind/mgr: clean up unused code

...and change the "if False" django-rest-framework
conditionals to actual version checks, with a view
to genuinely supporting 2.x in future if needed.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fix OSDMap notifications

This broken when MgrStandby was separated out,
because it now eats the original OSDMap message
without calling maybe_request_map.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: fixup ClusterState with_*map fns

Signed-off-by: John Spray <john.spray@redhat.com>

mon: remove need for friending OSDMap to PGMapUpdater

This used to be the friend relationship to PGMonitor.  It
was never really needed anyway, it was only being used
to skip the getter on `pools`

Signed-off-by: John Spray <john.spray@redhat.com>

mon: refactor PGMap updating code for reuse in mgr

...and remove the copypasta from mgr.

mgr still doesn't do all the same logic (it doesn't
have osdmap history handy) but it is now much easier
to see which bits are used one place vs. the other.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: update for Dispatcher::ms_handle_refused

Signed-off-by: John Spray <john.spray@redhat.com>

common: refactor CommandTable

Avoid handling out raw pointers to the ops,
which are in fact owned by the table.

Signed-off-by: John Spray <john.spray@redhat.com>

packaging: add boost-python dependency

Although ceph-mgr isn't packaged yet, builders
use these files to cue installation of dependencies.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: add initial ceph-mgr docs

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: add fsstatus module

A demonstration of creating a simple
ASCII-art status display by fusing
the metadata, perf counters, and maps that
are available to python modules in ceph-mgr.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: skip past broken modules and load the rest

...instead of stopping at the first python module
that fails to load.

Signed-off-by: John Spray <john.spray@redhat.com>

mgrc: less vebose logging of messages

...no need to tell us about all the things you're *not* handling.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement MgrMonitor.preprocess_beacon

So that we're not throwing out new mgrmap epochs
every beacon.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose osd stats from pgmap

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: dispose of connections after handle_command

...to avoid leaking them and eventually running out of fds.

Signed-off-by: John Spray <john.spray@redhat.com>

mgrc: mark down connections with dead sessions

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: fix logging

Previously only the errors were getting through, because
other stuff was getting held up in the python logging
framework.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose "df" to python modules

Signed-off-by: John Spray <john.spray@redhat.com>

mon: move "df" dump code from PGMonitor to PGMap

...so that ceph-mgr can use it too.

Signed-off-by: John Spray <john.spray@redhat.com>

common/TextTable: add include guards

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: update rest module for pg_summary

...which is now generated properly from PGMap
instead of reconstituted from some JSON.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: handle PGStats with a PGMap

No longer need the mon to send us
the pg_summary json hack.

Signed-off-by: John Spray <john.spray@redhat.com>

osd: send PGStats to mgr as well as mon

This is a rough implementation that sends all the stats,
without the efficiency of the queue+acks that we use
when talking to the mon.

Signed-off-by: John Spray <john.spray@redhat.com>

mgrc: enable sending pgstats

Signed-off-by: John Spray <john.spray@redhat.com>