buffer: use alignof for raw_combined allocation arithmetic

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: clean up raw_combined construction

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: size append_buffer so that it fits into page-multiple allocations

We drop some unittest assertions about alloc buffer size.  Sorry!

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-replay: s/CEPH_BUFFER_APPEND_SIZE/CEPH_PAGE_SIZE/

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: alloc right-sized buffer from read_fd

This may as well fit the input; this doesn't relate to the
append buffer.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: use raw_combined for certain allocations

If the alignment is on a page boundary, or the allocation is big,
a separate buffer::raw goes faster.  The rest of the time,
a raw_combined does.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: align unspecified allocations to a word

This will let us put policy create_aligned.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: combine data and buffer::raw into single allocation

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add front(), back(), get_num_buffers() methods

These eliminate most callers of buffers(), which exposes the
internal list<ptr>.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: benchmark some allocations

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: fix append_bench

- fix source
- include larger sizes

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: fix ptr move test

Do not assume there is a trailing null the terminate the string.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7739 from xiexingguo/xxg-wip-14387

osd: fix race condition for heartbeat_need_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7703 from xiexingguo/xxg-wip-statfs

osd: filestore: fast abort if statfs encounters ENOENT

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7766 from adamemerson/wip-portable-mutex

common: Do not use non-portable constants in mutex_debug

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7799 from xiexingguo/xxg-wip-fixnvme

osd: bluestore: NVMEDevice: fix error handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7804 from jjhuo/fixes

osd: kstore: fix a race condition in _txc_finish()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7816 from marcan/build-fixes

configure.ac: boost_iostreams is required, not optional

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7858 from jdurgin/wip-rbd-formatted-output-progress

test/cli-integration/rbd: disable progress output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7745 from sileht/sileht/rados-cython

pybind: move cephfs to Cython

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7831 from yuyuyu101/wip-14912

AsyncMessenger: fix several bugs

Merge pull request #7852 from xiexingguo/xxg-wip-14928

msg: async: start over after failing to bind a port in specified range

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

msg: async: start over after failing to bind a port in specified range

The original report associated with this fix can be found at
http://tracker.ceph.com/issues/13002. I reopen one in case it
needs to be backported or something alike.

Fixes: #14928
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/cli-integration/rbd: disable progress output

This test is intended to check machine-readable output and usage, not
human-readable output.

Fixes: #14931
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7778 from marcan/pybind-fixes

pybind/rados: fix object lifetime issues and other bugs in aio

Reviewed-by: Mehdi Abaakouk <sileht@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7855 from tchaikov/wip-fix-ftbfs

test/TestPGLog: fix the FTBFS

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/TestPGLog: fix the FTBFS

the new pure virtual method was introduced by e7edf20.

Fixes: #14930
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7781 from ceph/wip-fix-rbd-cli-tests

rbd/run_cli_tests.sh: Reflect test failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7769 from somnathr/wip-make-shared

Use make_shared while creating shared_ptr

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7765 from athanatos/wip-lost-unfound-repop

Repop and lost-unfound overhaul

Reviewed-by: Sage Weil <sage@redhat.com>

AsyncConnection: log cleanup, remove redundant state display

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: complete all pending events

Fix #14925
Signed-off-by: Haomai Wang <haomai@xsky.com>

AsyncConnection: dispatch write handler when accept done

Fix #12912
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7824 from dillaman/wip-aio-watch-notify

librados: fix test failure with new aio watch/unwatch API

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7832 from trociny/wip-journal-register-async

journal: async methods to (un)register and update client

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7829 from xiexingguo/xxg-wip-doc

doc: fix typo, indention etc.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix typo, indention etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind: update ceph_volume_client for cython

No need to explicitly touch the (no-longer-existing)
load_libcephfs method during module load, as with
the cython version we already get an ImportError
if the C library is unavailable.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: fix error handling on getxattr

The ctypes bindings returned empty string
instead of raising exception.  This was a bug,
because it made it impossible to detect the
difference between missing xattr and empty
xattr.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: move cephfs to Cython

This change moves cephfs binding to Cython.

Closes-bug: #14818
Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #7747 from vsjulu/master

xio: fix incorrect ip being assigned in case of multiple RDMA ports

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7630 from rzarzynski/wip-rgw-admin-token-cache

rgw: add support for caching of Keystone admin token.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7825 from ceph/wip-rgw-openssl

rgw: link civetweb with openssl (Sage, Marcus Watts)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test: add tests to (un)register and update journal client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Revert "AsyncConnection: Close connection when unregistered connection met WAIT"

This reverts commit 9cc1055eb6e18d55084d5af2b2f93eb8da5ab7da.

Fix #14912
Signed-off-by: Haomai Wang <haomai@xsky.com>

journal: async methods to (un)register and update client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls::journal: async client_unregister

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cmake fix: build civetweb with the right include path.

In an earlier commit, civetweb.c got moved into a separate cmake 'object'
library civetweb_common_objs so that it could be built just once for use
with several different targets.  At the time, there was a separate global
"include_directory" to give the right include path for including "civetweb.h".
A later commit in master created an rgw_a library, and restricted civetweb.h's
include path for only building rgw_a objects.  So, as the product of
these these two commits, the target_include_directories command for the
civetweb include path now needs to be applied to civetweb_common_objs
and not rgw_a.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #7650 from avnerbh/fix1_rand

msg: add thread safety for "random" Messenger + fix wrong usage of random functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7651 from xiexingguo/xxg-wip-fixtypecheck

osd: bluestore: improve fs-type verification and tidy up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7654 from rldleblanc/wip-wpq-optimization

osd: WeightedPriorityQueue: move to intrusive containers

Merge pull request #7821 from rzarzynski/wip-rgw-14903

rgw: fix wrong handling of limit=0 during listing of Swift account.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: change naming of utility functions in rgw_keystone.cc.

Slightly improve code formatting and const-correctness as well.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Use fixed version of civetweb to avoid warning.

The warning about SSLv23_client_method is because when using dynamic
linking, this should always be non-zero.  That's inside of civetweb,
so this commit is to pull the submodule up to the fix.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Fix ssl link error.

When linking against -lssl, should also include -lcrypto.

// fixup merge with previous

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Use ld.so to link in ssl crypto (cmake changes).

Using dlopen() to load in ssl crypto makes me cringe.
Also it requires either installing the devel package,
or appending library versioning to the library name
civetweb tries to load, which is ugly.  This is only
a license problem for people whose system distribution
does not include openssl.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

rgw: fix wrong handling of limit=0 during listing of Swift account.

Fixes: #14903
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7777 from theanalyst/rgw/insecure-keystone

rgw: allow authentication keystone with self signed certs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7809 from cbodley/wip-rgw-includes

rgw: use pimpl pattern for RGWPeriodHistory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7822 from rzarzynski/wip-rgw-keystone-proper-parse-ordering

rgw: try to parse Keystone token in order appropriate to configuration.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7803 from oritwas/wip-14826

rgw: reset return code in when iterating over the bucket the objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test: remove doubled unwatch API call

This error was unmasked by the fix included in commit 8caa2e4.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: aio watch/unwatch/notify now uses completion objver

A local variable was used for an async callback operation,
resulting in the potential for memory corruption.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

XIO: Changes to fix incorrect ip being assigned in case of multiple RDMA ports.remove rdma local and return error instead to avoid issues with using rdma_local incase of multiple ports.

Signed-off-by: Subramanyam Varanasi <s.varanasi@ssi.samsung.com>

rgw: try to parse Keystone token in order appropriate to configuration.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add insecure option to the http client

This allows the http client to turn off ssl certificate peer checking,
which is turned on by default. This is useful in cases like when
Keystone is SSL terminated with a self signed certificate.
The option `rgw_keystone_verify_ssl` (default true) can be toggled if
self signed certs are used, so that swift and s3 apis using keystone
authentication can work.

Fixes: #14583
Reported-by: Karol Mroz <mroz.karol@gmail.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: use pimpl pattern for RGWPeriodHistory

this removes the dependency on RGWPeriod from rgw_rados.h, which breaks
a circular dependency between rgw_rados.h -> rgw_metadata.h ->
rgw_period_history.h -> rgw_rados.h

Reported-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: final v9.2.1 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7694 from Abhishekvrshny/wip-infernalis-release-notes

release-notes: draft v9.2.1 release notes

rbd/run_cli_tests.sh: Reflect test failures

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #7813 from xiexingguo/xxg-wip-doc

doc: small fixes

doc: batch small fixes, including typo, syntax etc.

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

doc: fix typo

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMEDevice: remove unused variables ref

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMEDevice: fix error handling for try_get()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMEDevice: fix error handling for open()

According to Linux man page, the system call read() may return zero to indicate
that end of file reached on success. So below here we need to convert the result
code to a customized one in this case as errno is not set and we actually want
let caller know something is going wrong.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

configure.ac: boost_iostreams is required, not optional

Signed-off-by: Hector Martin <marcan@marcan.st>

Merge pull request #7805 from ErwanAliasr1/evelu-check-btrfs

Fixing BTFS issue at 'make check'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7802 from branch-predictor/bp-increase-async-threads

config: increase default async op threads

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7808 from cbodley/wip-cmake-jerasure

cmake: add ErasureCode.cc to jerasure plugins

cmake: add ErasureCode.cc to jerasure plugins

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: link against system openssl (instead of dlopen at runtime)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/: Use MOSDPGUpdateLogMissing to implement mark_unfound_lost_delete safely

Using a MOSDPGLog was unsafe since it is not ordered with
respect to repops.  Instead, use a new message sent through
the same paths as repops.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/,messages/: add MOSDPGUpdateLogMissing[Reply]

Signed-off-by: Samuel Just <sjust@redhat.com>

os/: add try_rename

Similar to collection_move_rename, except we ignore ENOENT and
don't allow different collections.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #7748 from ceph/wip-cmake-2-22-16

cmake: fix paths to various EC source files

cmake: installing _tp libraries into lib

Signed-off-by: Ali Maredia <amaredia@redhat.com>

PGLog: factor out claim_log_entries_update_missing

Signed-off-by: Samuel Just <sjust@redhat.com>

PGLog::merge_log: update missing, after removing divergent

This should not change behavior at all.  It'll be useful in
the next few commits.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG,PGLog: convert to DoutPrefixProviders

Signed-off-by: Samuel Just <sjust@redhat.com>

dout.h: add a DoutPrefixProvider and macro to allow passing of prefixes

This way, we can get proper logging within static functions.

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: use CEPH_MSG_PRIO_HIGH for write replies

Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: send subop writes and reads at normal priority

We want these to be processed ahead of new client ops since
there are resources being held.

Fixes: 14313
Backport: hammer
Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: remove OpContext from Repop

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: s/release_op_ctx_locks/release_object_locks, don't use opcontext

This gets us one step closer to removing OpContext from Repop.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG: refactor OpContext to use ObcLockManager

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::close_op_ctx: the err argument is unused

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: add ObcLockManager

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: use unique_ptr for backend trasaction, move into submit_transaction

submit_transaction takes ownership of the transaction implicitely.  Make
this implicit using an rvalue ref to force usage of std::move and move
constructor.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedPG::RepGather: remove on_finish, no more users

Signed-off-by: Samuel Just <sjust@redhat.com>