common: re-enable backtrace support

A recent porting effort disabled support for backtrace creation.  It
also didn't initialize all variables, which would lead to a crash when
deleting a backtrace instance.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6769 from aiicore/configure_fix

configure.ac: macro fix

Reviewed-by: Boris Ranto <branto@redhat.com>

configure.ac: macro fix

Configure fails with autoconf 2.63 on Centos 6.6 with:

./configure: line 34026: syntax error near unexpected token `newline'
./configure: line 34026: `  yes:no:'

Signed-off-by: Igor Podoski <igor.podoski@ts.fujitsu.com>

Merge pull request #5685 from ceph/wip-12133

cephfs-data-scan: scrub tag filtering (#12133 and #12145)

tools/cephfs: enable tag filter in DataScan

Fixes: #12133
Signed-off-by: John Spray <john.spray@redhat.com>

tools/cephfs: fix datascan kwarg parsing

Signed-off-by: John Spray <john.spray@redhat.com>

cls_cephfs: add PGLSCephFSFilter

Special purpose filter for use with CephFS disaster
recovery.

Fixes: #12145
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6719 from jcsp/wip-13898

tools: Fix layout handing in cephfs-data-scan (#13898)

Merge pull request #6661 from xiaoxichen/rbd_misc_fix1

cls/cls_rbd.cc: fix misused metadata_name_from_key

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cls/cls_rbd.cc: no need to skip key == after.

We already skip the pair where key == after in both LevelDBWholeSpaceIteratorImpl::upper_bound
and RocksDBWholeSpaceIteratorImpl::upper_bound.

Fixes: #13922
Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #6742 from dillaman/wip-13913

librbd: automatically flush IO after blocking write operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

init-ceph: fix systemd-run cant't start ceph daemon sometimes

Fixes: #13474
Signed-off-by: Chuanhong Wang <wang.chuanhong@zte.com.cn>

Merge pull request #6749 from dachary/wip-doc-install

doc: INSTALL redirect to online documentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools/cephfs: fix layout handling in injection

When the stashed-layout handling stuff was added,
the guessing part got lost from the path where
we only had a single object.  Broke test_data_scan.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6729 from majianpeng/librbd-journal

librbd: only send signal when AIO completions queue empty

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6751 from ceph/wip-cmake-strerror_r

cmake: define STRERROR_R_CHAR_P for GNU-specific strerror_r

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5465 from yuyuyu101/wip-librbd-ap

librbd: support eventfd for AIO completion notifications

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: define STRERROR_R_CHAR_P for GNU-specific strerror_r

This is what autotools does with AC_FUNC_STRERROR_R bit.  It takes us
from

    $ monmaptool --print nonexistant
    monmaptool: monmap file nonexistant
    monmaptool: couldn't open nonexistant: (2)

to

    $ monmaptool --print nonexistant
    monmaptool: monmap file nonexistant
    monmaptool: couldn't open nonexistant: (2) No such file or directory

on cmake - otherwise we end up using GNU-specific version as if it was
the POSIX version.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #6745 from drunkard/master

doc: fix typo in cephfs/quota

Merge remote-tracking branch 'gh/jewel'

doc: INSTALL redirect to online documentation

The INSTALL file has obsolete and misleading instructions to compile
from sources (-pg actually fails at link time). Replace with links to
the online installation and build from source documentations.

Signed-off-by: Loic Dachary <loic@dachary.org>

doc: typo fix in cephfs/quota

librbd/JournalReplay: Only Send signal when aio completions queue empty.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #6697 from XinzeChi/wip-scrub-test

test: fix osd-scrub-snaps.sh

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: fix osd-scrub-snaps.sh

when we run run-make-check.sh in hdd disk, it would spend much more
time to finish scrub.

Signed-off-by: Xinze Chi <xinze@xsky.com>

test: add pg_scrub for ceph-helpers.sh

Signed-off-by: Xinze Chi <xinze@xsky.com>

librbd: Fix rebase with new io flow

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: fix lttng tracing argument mismatch

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add set_event_notify to AioImageRequestWQ

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Remove unneeded set_event_notify

Since xlist clear method invoked below is enough to disassociate the xlist
item from the xlist, so the remove_myself() call in the destructor is safe
to invoke since it's already been removed from the list.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Makefile: Add noinst headr files

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Make rbd header file uses independent enum definition

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: Make AioCompletion complete doesn't unlock if callback

Signed-off-by: Haomai Wang <haomai@xsky.com>

Librbd: fix return code of EventSocket init and notify

Signed-off-by: Haomai Wang <haomai@xsky.com>

librbd: Add ictx check to avoid AIO_TYPE_NONE completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: normalize notify return code

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Fix incorrect api declaration

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: check event_notify to avoid extra logic

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

EventSocket: Add new event type pipe support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test: Add tests for getting arg of completion

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add interface to let user can get private data from comp

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

tests: Add tests for user io event notify

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: Add event notify interfaces

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: automatically flush IO after blocking write operations

This simplifies other state machines that previously had to flush IO
after blocking write operations.

Fixes: #13913
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6741 from SUSE/wip-tyepdef-typo-fix

include/rados/librados.h: fix typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

include/rados/librados.h: fix typo

This typo causes a warning when building the docs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #6740 from dillaman/wip-13911

librbd: ensure copy-on-read requests are complete prior to closing parent image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: utilize common flush helper when closing parent images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only enqueue flush completion if a flush is pending

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6737 from dillaman/wip-13912

librbd: check for presence of journal before attempting to remove

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6668 from zaitcev/cleanup-dead-code-2

rgw: Remove unused code in PutMetadataAccount:execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #6738 from ceph/wip-rbd-cli-misc

rbd: fixes for refactored CLI and related tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6715 from jcsp/wip-librbd-cmake

cmake: update for recent librbd changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6432 from ukernel/wip-11482

mds: fix client capabilities during reconnect (client.XXXX isn't responding to mclientcaps(revoke))

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #6736 from dillaman/wip-13914

librbd: not necessary to hold owner_lock while releasing snap id

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: check for presence of journal before attempting to remove

Fixes: #13912
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to determine if journal exists

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: not necessary to hold owner_lock while releasing snap id

Fixes: #13914
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: bail if too many arguments provided

The code has a catch clause for that, but it was being rendered useless
by the preceding

    if (command_spec.size() > matching_spec->size())
      positional_options.add(at::POSITIONAL_ARGUMENTS.c_str(), -1);

which names all (both expected and extraneous) positional arguments.

Change it to name only expected arguments, deriving the number of
expected arguments from the length of positional_opts vector, supplied
by each action.  This works for all actions except "feature enable" and
"feature disable" which are specified as multitoken, so keep on passing
in -1 for those.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: don't append an extra newline after some errors

Don't append an extra newline after program_options-generated errors,
like "unrecognised option" or "the argument for option is invalid".

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

tests: update unmap.t CLI test

Fixup the exit code - the old CLI tried to differentiate between CLI
errors and action errors by returning EXIT_FAILURE in the former case.
Also remove a test that relied on a special case check in the old CLI.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

cmake: librbd needs libjournal and libcls_journal_client

Commit 4719696cadd1 ("cmake: updates for refactored librbd IO path")
fixed file lists but missed the link dependency - librbd now needs
libjournal and libcls_journal_client.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #6706 from xiexingguo/xxg-wip-13711

rbd: output formatter may not be closed upon error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6734 from accelazh/master

Correct typo 'restared' to 'restarted'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Correct typo 'restared' to 'restarted'

There is a typo in document "Monitoring OSDs and PGs" as below

    An OSD was ``down``, was restared, and is now ``recovering``.

This patch corrects 'restared' to 'restarted'. The whole code base
is also searched to ensure there is no more 'restared'.

Signed-off-by: Yilong Zhao <accelazh@gmail.com>

Merge pull request #6733 from tchaikov/wip-fixcodingstyle

CodingStyle: fix broken URLs

Reviewed-by: Loic Dachary <ldachary@redhat.com>

CodingStyle: fix broken URLs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6724 from dachary/wip-13592-test-auto-repair

tests: fix race condition testing auto scrub

Reviewed-by: Xinze Chi <xinze@xsky.com>

mon/PGMonitor: MAX AVAIL is 0 if some OSDs' weight is 0

In get_rule_avail(), even p->second is 0, it's possible to be used
as divisor and quotient is infinity, then is converted to an integer
which is negative value.
So we should check p->second value before calculation.

It fixes BUG #13840.

Signed-off-by: Chengyuan Li <chengyli@ebay.com>
(cherry picked from commit 18713e60edd1fe16ab571f7c83e6de026db483ca)

os: FileStore::_destroy_collection may hide the real mistake.

Signed-off-by: Ruifeng Yang <yangruifeng.09209@h3c.com>
(cherry picked from commit 9e9770ca87720781264d2e283739fc9e197706c9)

Fix mon routed_request_tids leak

Signed-off-by: Ning Yao <zay11022@gmail.com>
(cherry picked from commit ba3c64ca705590f833806300461f3b98de0e62f8)

pybind: decode empty string in conf_parse_argv() correctly

cretargs is a array of c_char_p, which means ctypes has already
converted it to python byte strings. decode_cstr() would misinterpret
the empty string as a NULL c_char_p(), and convert it to None by
accident, resulting in errors when running commands like
'ceph config-key put foo ""'.

Since this is the only place we use arrays of c_char_p, just decode
it directly in conf_parse_argv(). Tested with python 2 and 3.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit f76d5d6fe6a9f92b1ec17191f5504e2f61e0ff28)

ceph_test_keyvaluedb_iterators: Fix broken test

Introduced by #6312
Signed-off-by: Haomai Wang <haomai@xsky.com>
(cherry picked from commit ce0369444558b6308e1e0ffa49ae68faaac2bd1d)

tests: fix race condition testing auto scrub

When testing auto scrub, waiting 20 seconds for the scrub to complete is
sometimes not enough and creates false negatives.

Split wait_for_scrub out of the repair helper so that it can be used to
wait for the scrub to happen instead of using a timer.

The scrub timestamp is obtained after removing the object, therefore
there is a chance for the scrub to be finished already. But since auto
scrub is scheduled every 5 seconds, it will only make the test wait an
extra 5 seconds and not hang forever.

http://tracker.ceph.com/issues/13592

Signed-off-by: Xinze Chi <xinze@xsky.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: update for recent librbd changes

Recent merge moved lots of things.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #6503 from XinzeChi/wip-repop-worker

osd: defer decoding of MOSDRepOp/MOSDRepOpReply

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6597 from ceph/wip-fix-kvdb-test

tests: ceph_test_keyvaluedb_iterators: fix broken test

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #6543 from efirs/optimize-stringify-call

common: reduce CPU usage by making stringstream in stringify function thread local

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6455 from 0003ydlom/perfctrwa

osd: enable perfcounters on sharded work queue mutexes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6441 from aclamk/wip-faster-log-fixed

common: optimize debug logging code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6276 from david-z/wip-systemd-finegrain-ceph-service

systemd: start/stop/restart ceph services by daemon type

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #6198 from XinzeChi/wip-buffer-history-alloc

common: perf counter for bufferlist history total alloc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6081 from trociny/wip-pool_opts_t

osd: pg_pool_t: add dictionary for pool options

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: store per pool scrub intervals in pool options

  ceph osd pool set $POOL scrub_min_interval N
  ceph osd pool set $POOL scrub_max_interval N
  ceph osd pool set $POOL deep_scrub_interval N

If N > 0, this value is used for the pool instead of
the corresponding global parameter from the config
(osd_scrub_min_interval, osd_scrub_max_interval or
osd_deep_scrub_interval).

Fixes: #13077
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

tests: workunits should not have ./ (assume it in $PATH)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd: pg_pool_t: add dictionary for pool options

It is going to be used to store per pool configuration parameters like
scrub intervals or recovery priority.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #6710 from jdurgin/wip-13877

test: use sequential journal_tid for object cacher test

Reported-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6711 from jdurgin/wip-13880

pybind: decode empty string in conf_parse_argv() correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6704 from liewegas/wip-up-thru

mon: block 'ceph osd pg-temp ...' if pg_temp update is already pending

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #6653 from branch-predictor/bp-no-zero-reverse

common: don't reverse hobject_t hash bits when zero

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6475 from branch-predictor/bp-trim-n-reorder-bools

osd: reduce memory consumption of some structs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6456 from stiopaa1/removeConstr

common: buffer: remove unneeded list destructor

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6263 from dx9/wip-12406-os-filestore

osd: FileStore: remove __SWORD_TYPE dependency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6262 from dx9/wip-12406-os-fs

osd: remove __SWORD_TYPE dependency

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6175 from XinzeChi/wip-finisher-lat

common: add latency perf counter for finisher

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6102 from mslovy/wip-mon-leak-fix

mon: fix routed_request_tids leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6093 from tchaikov/wip-kvstore-tool-argv

ceph-kvstore-tool: handle bad out file on command line

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5793 from yuyuyu101/impl-perf

osd: add osd op queue latency perfcounter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5734 from cxwshawn/mon-fix

auth: make keyring without mon entity type return -EACCES

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6714 from aiicore/test_encoding_readable

test/encoding/readable.sh fix

Reviewed-by: Loic Dachary <ldachary@redhat.com>