Implemented log message size predictor. It tracks size of log messages.
Initial allocation size is derived from last log message from the same line of code.
Fixed bug in test.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Merge pull request #6742 from dillaman/wip-13913

librbd: automatically flush IO after blocking write operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

init-ceph: fix systemd-run cant't start ceph daemon sometimes

Fixes: #13474
Signed-off-by: Chuanhong Wang <wang.chuanhong@zte.com.cn>

Merge pull request #6749 from dachary/wip-doc-install

doc: INSTALL redirect to online documentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6729 from majianpeng/librbd-journal

librbd: only send signal when AIO completions queue empty

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6751 from ceph/wip-cmake-strerror_r

cmake: define STRERROR_R_CHAR_P for GNU-specific strerror_r

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5465 from yuyuyu101/wip-librbd-ap

librbd: support eventfd for AIO completion notifications

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: define STRERROR_R_CHAR_P for GNU-specific strerror_r

This is what autotools does with AC_FUNC_STRERROR_R bit.  It takes us
from

    $ monmaptool --print nonexistant
    monmaptool: monmap file nonexistant
    monmaptool: couldn't open nonexistant: (2)

to

    $ monmaptool --print nonexistant
    monmaptool: monmap file nonexistant
    monmaptool: couldn't open nonexistant: (2) No such file or directory

on cmake - otherwise we end up using GNU-specific version as if it was
the POSIX version.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #6745 from drunkard/master

doc: fix typo in cephfs/quota

Merge remote-tracking branch 'gh/jewel'

doc: INSTALL redirect to online documentation

The INSTALL file has obsolete and misleading instructions to compile
from sources (-pg actually fails at link time). Replace with links to
the online installation and build from source documentations.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: typo fix in cephfs/quota

librbd/JournalReplay: Only Send signal when aio completions queue empty.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #6697 from XinzeChi/wip-scrub-test

test: fix osd-scrub-snaps.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: fix osd-scrub-snaps.sh

when we run run-make-check.sh in hdd disk, it would spend much more
time to finish scrub.

Signed-off-by: Xinze Chi <xinze@xsky.com>

test: add pg_scrub for ceph-helpers.sh

Signed-off-by: Xinze Chi <xinze@xsky.com>

librbd: Fix rebase with new io flow

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: fix lttng tracing argument mismatch

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add set_event_notify to AioImageRequestWQ

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Remove unneeded set_event_notify

Since xlist clear method invoked below is enough to disassociate the xlist
item from the xlist, so the remove_myself() call in the destructor is safe
to invoke since it's already been removed from the list.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Makefile: Add noinst headr files

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Make rbd header file uses independent enum definition

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: Make AioCompletion complete doesn't unlock if callback

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: fix return code of EventSocket init and notify

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add ictx check to avoid AIO_TYPE_NONE completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: normalize notify return code

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Fix incorrect api declaration

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: check event_notify to avoid extra logic

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

EventSocket: Add new event type pipe support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test: Add tests for getting arg of completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add interface to let user can get private data from comp

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

tests: Add tests for user io event notify

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add event notify interfaces

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: automatically flush IO after blocking write operations

This simplifies other state machines that previously had to flush IO
after blocking write operations.

Fixes: #13913
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6741 from SUSE/wip-tyepdef-typo-fix

include/rados/librados.h: fix typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

include/rados/librados.h: fix typo

This typo causes a warning when building the docs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6740 from dillaman/wip-13911

librbd: ensure copy-on-read requests are complete prior to closing parent image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: utilize common flush helper when closing parent images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only enqueue flush completion if a flush is pending

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6737 from dillaman/wip-13912

librbd: check for presence of journal before attempting to remove

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6668 from zaitcev/cleanup-dead-code-2

rgw: Remove unused code in PutMetadataAccount:execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #6738 from ceph/wip-rbd-cli-misc

rbd: fixes for refactored CLI and related tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6715 from jcsp/wip-librbd-cmake

cmake: update for recent librbd changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6432 from ukernel/wip-11482

mds: fix client capabilities during reconnect (client.XXXX isn't responding to mclientcaps(revoke))

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #6736 from dillaman/wip-13914

librbd: not necessary to hold owner_lock while releasing snap id

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: check for presence of journal before attempting to remove

Fixes: #13912
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to determine if journal exists

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: not necessary to hold owner_lock while releasing snap id

Fixes: #13914
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: bail if too many arguments provided

The code has a catch clause for that, but it was being rendered useless
by the preceding

    if (command_spec.size() > matching_spec->size())
      positional_options.add(at::POSITIONAL_ARGUMENTS.c_str(), -1);

which names all (both expected and extraneous) positional arguments.

Change it to name only expected arguments, deriving the number of
expected arguments from the length of positional_opts vector, supplied
by each action.  This works for all actions except "feature enable" and
"feature disable" which are specified as multitoken, so keep on passing
in -1 for those.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: don't append an extra newline after some errors

Don't append an extra newline after program_options-generated errors,
like "unrecognised option" or "the argument for option is invalid".

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

tests: update unmap.t CLI test

Fixup the exit code - the old CLI tried to differentiate between CLI
errors and action errors by returning EXIT_FAILURE in the former case.
Also remove a test that relied on a special case check in the old CLI.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

cmake: librbd needs libjournal and libcls_journal_client

Commit 4719696cadd1 ("cmake: updates for refactored librbd IO path")
fixed file lists but missed the link dependency - librbd now needs
libjournal and libcls_journal_client.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #6706 from xiexingguo/xxg-wip-13711

rbd: output formatter may not be closed upon error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6734 from accelazh/master

Correct typo 'restared' to 'restarted'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Correct typo 'restared' to 'restarted'

There is a typo in document "Monitoring OSDs and PGs" as below

    An OSD was ``down``, was restared, and is now ``recovering``.

This patch corrects 'restared' to 'restarted'. The whole code base
is also searched to ensure there is no more 'restared'.

Signed-off-by: Yilong Zhao <accelazh@gmail.com>

Merge pull request #6733 from tchaikov/wip-fixcodingstyle

CodingStyle: fix broken URLs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

CodingStyle: fix broken URLs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6724 from dachary/wip-13592-test-auto-repair

tests: fix race condition testing auto scrub

Reviewed-by: Xinze Chi <xinze@xsky.com>

mon/PGMonitor: MAX AVAIL is 0 if some OSDs' weight is 0

In get_rule_avail(), even p->second is 0, it's possible to be used
as divisor and quotient is infinity, then is converted to an integer
which is negative value.
So we should check p->second value before calculation.

It fixes BUG #13840.

Signed-off-by: Chengyuan Li <chengyli@ebay.com>
(cherry picked from commit 18713e60edd1fe16ab571f7c83e6de026db483ca)

os: FileStore::_destroy_collection may hide the real mistake.

Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>
(cherry picked from commit 9e9770ca87720781264d2e283739fc9e197706c9)

Fix mon routed_request_tids leak

Signed-off-by: Ning Yao <zay11022@gmail.com>
(cherry picked from commit ba3c64ca705590f833806300461f3b98de0e62f8)

pybind: decode empty string in conf_parse_argv() correctly

cretargs is a array of c_char_p, which means ctypes has already
converted it to python byte strings. decode_cstr() would misinterpret
the empty string as a NULL c_char_p(), and convert it to None by
accident, resulting in errors when running commands like
'ceph config-key put foo ""'.

Since this is the only place we use arrays of c_char_p, just decode
it directly in conf_parse_argv(). Tested with python 2 and 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit f76d5d6fe6a9f92b1ec17191f5504e2f61e0ff28)

ceph_test_keyvaluedb_iterators: Fix broken test

Introduced by #6312
Signed-off-by: Haomai Wang <haomai@xsky.com>
(cherry picked from commit ce0369444558b6308e1e0ffa49ae68faaac2bd1d)

tests: fix race condition testing auto scrub

When testing auto scrub, waiting 20 seconds for the scrub to complete is
sometimes not enough and creates false negatives.

Split wait_for_scrub out of the repair helper so that it can be used to
wait for the scrub to happen instead of using a timer.

The scrub timestamp is obtained after removing the object, therefore
there is a chance for the scrub to be finished already. But since auto
scrub is scheduled every 5 seconds, it will only make the test wait an
extra 5 seconds and not hang forever.

http://tracker.ceph.com/issues/13592

Signed-off-by: Xinze Chi <xinze@xsky.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: update for recent librbd changes

Recent merge moved lots of things.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6503 from XinzeChi/wip-repop-worker

osd: defer decoding of MOSDRepOp/MOSDRepOpReply

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6597 from ceph/wip-fix-kvdb-test

tests: ceph_test_keyvaluedb_iterators: fix broken test

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #6543 from efirs/optimize-stringify-call

common: reduce CPU usage by making stringstream in stringify function thread local

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6455 from 0003ydlom/perfctrwa

osd: enable perfcounters on sharded work queue mutexes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6441 from aclamk/wip-faster-log-fixed

common: optimize debug logging code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6276 from david-z/wip-systemd-finegrain-ceph-service

systemd: start/stop/restart ceph services by daemon type

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6198 from XinzeChi/wip-buffer-history-alloc

common: perf counter for bufferlist history total alloc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6081 from trociny/wip-pool_opts_t

osd: pg_pool_t: add dictionary for pool options

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: store per pool scrub intervals in pool options

  ceph osd pool set $POOL scrub_min_interval N
  ceph osd pool set $POOL scrub_max_interval N
  ceph osd pool set $POOL deep_scrub_interval N

If N > 0, this value is used for the pool instead of
the corresponding global parameter from the config
(osd_scrub_min_interval, osd_scrub_max_interval or
osd_deep_scrub_interval).

Fixes: #13077
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

tests: workunits should not have ./ (assume it in $PATH)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: pg_pool_t: add dictionary for pool options

It is going to be used to store per pool configuration parameters like
scrub intervals or recovery priority.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6710 from jdurgin/wip-13877

test: use sequential journal_tid for object cacher test

Reported-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6711 from jdurgin/wip-13880

pybind: decode empty string in conf_parse_argv() correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6704 from liewegas/wip-up-thru

mon: block 'ceph osd pg-temp ...' if pg_temp update is already pending

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6653 from branch-predictor/bp-no-zero-reverse

common: don't reverse hobject_t hash bits when zero

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6475 from branch-predictor/bp-trim-n-reorder-bools

osd: reduce memory consumption of some structs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6456 from stiopaa1/removeConstr

common: buffer: remove unneeded list destructor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6263 from dx9/wip-12406-os-filestore

osd: FileStore: remove __SWORD_TYPE dependency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6262 from dx9/wip-12406-os-fs

osd: remove __SWORD_TYPE dependency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6175 from XinzeChi/wip-finisher-lat

common: add latency perf counter for finisher

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6102 from mslovy/wip-mon-leak-fix

mon: fix routed_request_tids leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6093 from tchaikov/wip-kvstore-tool-argv

ceph-kvstore-tool: handle bad out file on command line

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5793 from yuyuyu101/impl-perf

osd: add osd op queue latency perfcounter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5734 from cxwshawn/mon-fix

auth: make keyring without mon entity type return -EACCES

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6714 from aiicore/test_encoding_readable

test/encoding/readable.sh fix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/encoding/readable.sh fix

1. Fix formatting (mixed tabs and spaces)
2. Status of skipped types was showing always last version in directory

Signed-off-by: Igor Podoski <igor.podoski@ts.fujitsu.com>

Merge pull request #6705 from vanpire110/master

doc: Fixes a spelling error

Reviewed-by: Loic Dachary <ldachary@redhat.com>

pybind: decode empty string in conf_parse_argv() correctly

cretargs is a array of c_char_p, which means ctypes has already
converted it to python byte strings. decode_cstr() would misinterpret
the empty string as a NULL c_char_p(), and convert it to None by
accident, resulting in errors when running commands like
'ceph config-key put foo ""'.

Since this is the only place we use arrays of c_char_p, just decode
it directly in conf_parse_argv(). Tested with python 2 and 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6699 from dachary/wip-ceph-helpers

tests: ceph-helpers assert success getting backfills

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: use sequential journal_tid for object cacher test

This matches the real usage by librbd.

Fixes: #13877
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

EventSocket: Add EventSocket structure used for event notification

EventSocket will wrap different user event notification method like linux
eventfd, solaris port. Caller can user this to replace signal

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

List: close formatter session on error exit.

Fixes: #13711
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

DiskUsage: close formatter session on error exit

Fixes: #13711
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fixes a spelling error

Signed-off-by: Jeremy Qian <vanpire110@163.com>

Merge pull request #6533 from ghost/wip-fix-trivial-bug

osd: fix trivial scrub bug

Reviewed-by: Kefu Chai <kchai@redhat.com>