mon: MonmapMonitor: don't expose uncommitted state to client

During prepare_command(), we were returning to the user based on
pending_map's state. Even though this weren't causing any issues we are
aware of, we really shouldn't do that.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6724 from dachary/wip-13592-test-auto-repair

tests: fix race condition testing auto scrub

Reviewed-by: Xinze Chi <xinze@xsky.com>

tests: fix race condition testing auto scrub

When testing auto scrub, waiting 20 seconds for the scrub to complete is
sometimes not enough and creates false negatives.

Split wait_for_scrub out of the repair helper so that it can be used to
wait for the scrub to happen instead of using a timer.

The scrub timestamp is obtained after removing the object, therefore
there is a chance for the scrub to be finished already. But since auto
scrub is scheduled every 5 seconds, it will only make the test wait an
extra 5 seconds and not hang forever.

http://tracker.ceph.com/issues/13592

Signed-off-by: Xinze Chi <xinze@xsky.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6503 from XinzeChi/wip-repop-worker

osd: defer decoding of MOSDRepOp/MOSDRepOpReply

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6597 from ceph/wip-fix-kvdb-test

tests: ceph_test_keyvaluedb_iterators: fix broken test

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #6543 from efirs/optimize-stringify-call

common: reduce CPU usage by making stringstream in stringify function thread local

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6455 from 0003ydlom/perfctrwa

osd: enable perfcounters on sharded work queue mutexes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6441 from aclamk/wip-faster-log-fixed

common: optimize debug logging code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6276 from david-z/wip-systemd-finegrain-ceph-service

systemd: start/stop/restart ceph services by daemon type

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6198 from XinzeChi/wip-buffer-history-alloc

common: perf counter for bufferlist history total alloc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6081 from trociny/wip-pool_opts_t

osd: pg_pool_t: add dictionary for pool options

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: store per pool scrub intervals in pool options

  ceph osd pool set $POOL scrub_min_interval N
  ceph osd pool set $POOL scrub_max_interval N
  ceph osd pool set $POOL deep_scrub_interval N

If N > 0, this value is used for the pool instead of
the corresponding global parameter from the config
(osd_scrub_min_interval, osd_scrub_max_interval or
osd_deep_scrub_interval).

Fixes: #13077
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

tests: workunits should not have ./ (assume it in $PATH)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: pg_pool_t: add dictionary for pool options

It is going to be used to store per pool configuration parameters like
scrub intervals or recovery priority.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6710 from jdurgin/wip-13877

test: use sequential journal_tid for object cacher test

Reported-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6711 from jdurgin/wip-13880

pybind: decode empty string in conf_parse_argv() correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6653 from branch-predictor/bp-no-zero-reverse

common: don't reverse hobject_t hash bits when zero

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6475 from branch-predictor/bp-trim-n-reorder-bools

osd: reduce memory consumption of some structs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6456 from stiopaa1/removeConstr

common: buffer: remove unneeded list destructor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6263 from dx9/wip-12406-os-filestore

osd: FileStore: remove __SWORD_TYPE dependency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6262 from dx9/wip-12406-os-fs

osd: remove __SWORD_TYPE dependency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6175 from XinzeChi/wip-finisher-lat

common: add latency perf counter for finisher

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6102 from mslovy/wip-mon-leak-fix

mon: fix routed_request_tids leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6093 from tchaikov/wip-kvstore-tool-argv

ceph-kvstore-tool: handle bad out file on command line

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5793 from yuyuyu101/impl-perf

osd: add osd op queue latency perfcounter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5734 from cxwshawn/mon-fix

auth: make keyring without mon entity type return -EACCES

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6714 from aiicore/test_encoding_readable

test/encoding/readable.sh fix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/encoding/readable.sh fix

1. Fix formatting (mixed tabs and spaces)
2. Status of skipped types was showing always last version in directory

Signed-off-by: Igor Podoski <igor.podoski@ts.fujitsu.com>

Merge pull request #6705 from vanpire110/master

doc: Fixes a spelling error

Reviewed-by: Loic Dachary <ldachary@redhat.com>

pybind: decode empty string in conf_parse_argv() correctly

cretargs is a array of c_char_p, which means ctypes has already
converted it to python byte strings. decode_cstr() would misinterpret
the empty string as a NULL c_char_p(), and convert it to None by
accident, resulting in errors when running commands like
'ceph config-key put foo ""'.

Since this is the only place we use arrays of c_char_p, just decode
it directly in conf_parse_argv(). Tested with python 2 and 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6699 from dachary/wip-ceph-helpers

tests: ceph-helpers assert success getting backfills

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: use sequential journal_tid for object cacher test

This matches the real usage by librbd.

Fixes: #13877
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: Fixes a spelling error

Signed-off-by: Jeremy Qian <vanpire110@163.com>

Merge pull request #6533 from ghost/wip-fix-trivial-bug

osd: fix trivial scrub bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6612 from H3C/wip-yrf-destroy_collection

osd: fix FileStore::_destroy_collection error return code

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #6660 from chengyli/master

mon: fix ceph df pool available calculation for 0-weighted OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6473 from H3C/wip-osd-bugfix1

auth: fail if rotating key is missing (do not spam log)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6278 from XinzeChi/wip-failinfo-mon

osd: cancel failure reports if we fail to rebind network

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6675 from rohanmars/wip-aix-librados-port

aix gcc librados port

Reviewed-by: Sage Weil <sage@redhat.com>

aix shared library build

Signed-off-by: Rohan Mars <code@rohanmars.com>

Merge branch 'wip-13800' of git://github.com/ukernel/ceph

client: fix deadlock related to async pagecache invalidation

Conflicts:
src/client/Client.cc
Fixed a conflict with the earlier page cache invalidate
changes in 73beb7f9378182cc3901fe86c4f1f5d5d98169a6.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6454 from H3C/wip-mds

mds: repair the command option "--hot-standby"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6269 from jcsp/wip-client-mark-down

client: close mds sessions in shutdown()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6380 from ukernel/wip-client-keep-cache

client: don't invalidate page cache when inode is no longer used

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6253 from jcsp/wip-client-availability

client: a better check for MDS availability

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6051 from clever215/master

rgw: add an inspection to the field of type when assigning user caps

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add an inspection to the field of type when assigning user caps

Bug #13096

This modification adds the check to the field of type of a user's capality while previous versions set it any value. i.e. we limit the option of types to the 9 certain values, which are "users|buckets|metadata|usage|zone|bilog|mdlog|datalog|ops These 9 choosens are found in ceph documents and in source codes.

Signed-off-by: Kongming Wu <wu.kongming@h3c.com>

Merge pull request #6700 from dillaman/wip-librbd-32bit-support

librbd: simplify IO method signatures for 32bit environments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: simplify IO method signatures for 32bit environments

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Fix mon routed_request_tids leak

Signed-off-by: Ning Yao <zay11022@gmail.com>

tests: ceph-helpers assert success getting backfills

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #6625 from dillaman/wip-12698

librbd: integrate journaling for maintenance operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6687 from dillaman/wip-journal-replay-fixes

journal: support replaying beyond skipped splay objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6685 from dachary/wip-erasure-code-benchmark

qa: erasure-code benchmark plugin selection

Reviewed-by: Andreas Peters <andreas.joachim.peters@cern.ch>

Merge pull request #6292 from dx9/wip-12406-res_nquery

rgw/rgw_resolve: fallback to res_query when res_nquery not implemented

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6236 from guangyy/err-msg

osd: use pg id (without shard) when referring the PG

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6610 from ktdreyer/wip-build-doc-lxml

admin/build-doc: add lxml dependencies on debian

Reviewed-by: Sage Weil <sage@redhat.com>

client: s/close_sessions/_close_sessions/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6323 from dingshang/wip-cephfs-dingshang

pybind/cephfs: add symlink and its unit test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: close mds sessions in shutdown()

Usually this happens in unmount(), but when we
have instantiated Client without mounting (to
send MDS commands), we need to handle closing
any open sessions in shutdown as well.

This is the correct replacement for the mark_down()
call that was removed from handle_command_reply
in the last commit.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge tag 'v10.0.0'

v10.0.0

Merge pull request #6684 from jcsp/wip-fix-scrub

mds: fix scrub_path

journal: support replay passed skipped splay objects

It's possible for a splay object within a set to be skipped
if the set is closed due to a full object within the set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: verify that journal player can handle skipped journal objects

It's possible for a journal object to not exist if another journal object
within the same object set filled up before records were written.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6605 from yuyuyu101/wip-13797

ceph_test_msgr: Use send_message instead of keepalive to wakeup connection

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6495 from objoo/master

Mailmap updates for infernalis.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Jenkins affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Burkhard Linke affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Chen Dihao affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Wei Qian affiliation
Signed-off-by: Yann Dupont <yann@objoo.org>

qa: erasure-code-benchmark technique and plugin selection

Update the PLUGINS variable that was no longer used. Add the TECHNIQUES
variable to control which techniques are compared.

Signed-off-by: Loic Dachary <loic@dachary.org>

qa: erasure-code has --erasure-code-dir

It is used instead of the obsoleted --parameter directory= to specify
the location of the erasure code directory plugins.

Signed-off-by: Loic Dachary <loic@dachary.org>

add aix compile warning

Signed-off-by: Rohan Mars <code@rohanmars.com>

initialized backtrace variables

Signed-off-by: Rohan Mars <code@rohanmars.com>

mds: fix scrub_path

This was tripping up over calling
validate_disk_state with no ScrubHeader.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6679 from suckowbiz/patch-1

Fixed typos

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/release-notes: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final v10.0.0 notes

Signed-off-by: Sage Weil <sage@redhat.com>

mon: do not ignore a failure report cancellation form osd

do not ignore a failure report cancellation form osd even if it is
down.

Signed-off-by: Xinze Chi <xinze@xsky.com>

mon: fix osd failure info in mon

when the network adapter of node A run into error, osd in this node
would tell mon other osd's heartbeat is timeout also. So when rebind
fail after retry 3 times, the osd should cancel in-flight failure report
send to mon before.

Signed-off-by: Xinze Chi <xinze@xsky.com>

doc: fix message typos in systemd

Signed-off-by: Tobias Suckow <tobias@suckow.biz>

Merge branch 'master' of github.com:ceph/ceph

Merge pull request #6666 from dachary/wip-release-notes

release-notes: draft v10.0.0 release notes

Merge branch 'wip-bigbang'

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/mon/osd-crush.sh: escape ceph tell mon.*

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make some of the pg_temp methods/fields private

Reported-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: call notify completion only once

If we race with a reconnect we could get a second notify message
before the notify linger op is torn down.  Ensure we only ever
call the notify completion once to prevent a segfault.

Fixes: #13805
Signed-off-by: Sage Weil <sage@redhat.com>

mon: change mon_osd_min_down_reporters from 1 -> 2

This makes more sense to me.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: simplify failure reporters vs reports logic

Since each OSD only sends a failure report for a given peer once,
we don't need to count reports vs reporters separately.  (This was
probably a bad idea anyway.)  Remove this logic and the associated
config option.

Reported-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osd: simplify pg creation

We used to have a complicated pg creation process in which we
would query any previous mappings for the pg before we created the
new 'empty' pg locally.  The tracking of the prior mappings was
very simple (and broken), but it didn't really matter because the
mon would resend pg create messages periodically.  Now it doesn't,
so that broke.

However, none of this is necessary: the PG peering process does
all of the same things.  Namely, it

- enumerates past intervals
- determines which ones may have been rw
- queries OSDs from each one to gather any potential changes

This is a more robust version of what the creation code was (or
should have been doing).  So, let's rip it all out and let
peering handle it.  As long as the newly instantiated PG sets
last_epoch_started and _clean to the created epoch we will probe
and consider all of these prior mappings and find any previous
instance of the PG (if one existed).

Yay for removing unnecessary code!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: make _sub_got behave if we "got" old stuff

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: fix oldest_map in send_incremental

This should be the oldest map on the sender (like every other
place that generates an MOSDMap message).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: avoid useless pg gets when pool is deleted

If the .0 pg no longer exists, we know the entire pool was
deleted, and can avoid querying every other pg.  (This is a good
thing because leveldb and rocksdb can be very slow to query
missing keys.)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: revamp how pg creates are tracked

Previously we were calculating and managing in-core state that
wasn't committed as part of the pg_map, leading to all sorts of
ugliness that didn't really work.  Instead,

 * set mapping in all creating pgs in the committed pg_map
 * make all pg create message sending be based on committed state
 * update mappings for creating pgs every time we consume a new
   osdmap, so that we have a reliable/stable epoch to attach to
   it.

In particular, having that stable epoch means we have a reference
we can put in the pg create message that will also be used for
the subscription version.  That way OSDs get consistent creates
from any mon.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only send pg create messages to up osds

If the OSD is down it will ignore the message.  If it gets marked up, we
will eventually consume that map and call check_subs().

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: only churn mapping_epoch if the primary changes

This results is fewer resent pg create messages.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: a bunch of cosmetic cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMonitor: drop old creating_pgs_by_osd

Obsoleted by creating_pgs_by_osd_epoch.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: reduce mon_subscribe messages

1. MonClient remembers our subscriptions; only indicate we want
osd_pg_creates once, in init.

2. We don't need to re-request the latest osdmap each time we
reconnect.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: only send new subscriptions

Instead of resending all subscriptions, only send the new ones.  This
avoids races like

 - ask for 4+
 - mon sends maps 4-50
 - ask for 4+ and something else
 - mon has to resend same maps and the other thing

Signed-off-by: Sage Weil <sage@redhat.com>