os/bluestore: add an ability to reproduce issue #21040 and verify if
fsck repair fixes it.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: implement BlueStore repair

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: simplify SharedBlob::put_ref interface

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20443 from rzarzynski/wip-common-inlineable-ceph_clock_now

common: make ceph_clock_now() inlineable.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20439 from jcsp/wip-percent_used

mon: dump percent_used PGMap field as float

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20177 from liewegas/wip-kill-onreadable

os,osd: initial work to drop onreadable/onapplied callbacks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20499 from idryomov/wip-krbd-discard-tests

qa: additional krbd discard test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20490 from dillaman/wip-rbd-journal-workunit

qa/workunits/rbd: remove sanity check in journal.sh test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20093 from supriti/wip_rgw_ganesha_df

RGW-NFS: Use rados cluster_stat to report filesystem usage

Merge pull request #20395 from Songweibin/wip-misc-cleanup

librbd: validate if dst group snap name is the same with src

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20420 from trociny/wip-unwatch-race

librbd: potential race in RewatchRequest when resetting watch_handle

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20478 from trociny/wip-api-mirror

librbd: fix structure size check in rbd_mirror_image_get_info/status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: krbd_blkroset.t: attempt discards in addition to writes

The block layer used to not enforce ioctl(BLKROSET) and set_disk_ro().
This has been fixed in 4.16-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20491 from dillaman/wip-23043

qa/workunits/rbd: fix issues within permissions test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20481 from dillaman/wip-rbd-remove-error

rbd: don't overwrite the error code from the remove action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20486 from dillaman/wip-11502

test/librbd: utilize unique pool for cache tier testing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20450 from dzafman/wip-22996

Snapset inconsistency is detected with its own error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20492 from tchaikov/wip-boost-cmake-3.0

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

see https://cmake.org/cmake/help/v3.1/module/ExternalProject.html and
https://cmake.org/cmake/help/v3.0/module/ExternalProject.html. the
former has DOWNLOAD_NO_PROGRESS, while the latter does not.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: remove sanity check in journal.sh test

When the OSDs are being concurrently thrashed, this can result in
sporadic failures due to the admin socket disappearing.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: correct permission test handling for clone v2 images

Fixes: http://tracker.ceph.com/issues/23043
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: utilize RBD profiles in permissions.sh test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: krbd_fallocate.sh: add -ENOENT handling test case

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: utilize unique pool for cache tier testing

Fixes: http://tracker.ceph.com/issues/11502
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: don't overwrite the error code from the remove action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix structure size check in rbd_mirror_image_get_info/status

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20031 from tchaikov/wip-doc-macos

doc/dev/macos: add doc for building on MacOS

Merge pull request #20474 from wjwithagen/wjw-wip-pthread_set_name_2

include: compat.h, fix the return result of pthread_set_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20168 into master

* refs/pull/20168/head:
Handle dir->dentries.empty()
client: lookup_ino when i1 becomes NULL

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20424 into master

* refs/pull/20424/head:
qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20428 into master

* refs/pull/20428/head:
qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20470 into master

* refs/pull/20470/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

include: compat.h, fix the return result of pthread_set_name()

On FreeBSD the function type is void(). Make it compatible with return
value on other platforms.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa: ignore full cluster warnings for mds-full

Fixes: http://tracker.ceph.com/issues/23033
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/objectstore: increase open file limit

Signed-off-by: Sage Weil <sage@redhat.com>

os: do not inject read EIO on meta pool objects

Avoid EIO on, say, osdmaps until we fix
http://tracker.ceph.com/issues/23029

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20274 from adamemerson/wip-socket-to-me!

common/admin_socket: various cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test_filestore_idempotent_sequence: put txn object in each collection

The transactions are idependent in each collection/sequencer, so we
can't record to a single txn object with racing transactions.  Fix
it by doing one in each collection, and when reading the latest op,
use the highest txn value we see.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: Use unique_ptr instead of new/delete

Mostly for the default hooks, plus one Formatter.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Cleanup path cleanup

Don't use strdup/free explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config/admin_socket: Remove PFL_* stuff since nobody's using it

If we want to, we have other ways of getting data out of a thread.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use one map instead of three

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph_test_objectstore: fix split ordering

We need to flush between split.  This requirement unfortunately
doesn't quite go away with the FileStore tracking.

Also, flush for each batch.  This is just because the test environment
may have a low open file ulimit.  (The old code did apply_transaction,
so it's functionally equivalent to this.)

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: Use std::mutex/condition_variable/thread

And fix a bug where we exit in one code-path without releasing the
lock.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_named_thread

To allow us to make a std::thread with a name.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Copy fewer strings

Pass std::string_view to call, rather than passing strings by value.

Change cmdmap_t and the maps in admin_socket to use the std::less<>
transparent comparator, so we can compare string_views to strings. (Or
anything else with an equality operator with strings.)

Pass the cmdmap_t in to call by const reference, not reference.

The change to cmdmap_t requires changing some of the users of
cmdmap_t. (That is, make them actually use cmdmap_t explicitly, not
the open-coed map type.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17020 from theanalyst/lc-time-disable

qa: re enable LC tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16757 from rzarzynski/wip-rgw-20883

rgw: dump Last-Modified in Swift's responses for GET/HEAD on container.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15054 from rzarzynski/wip-rgw-fix-browser-upload-error-handling

rgw: fix error handling in Browser Uploads.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12427 from linuxbox2/wip-rgw-ldap-shutdown

rgw: don't leak S3 LDAPHelper

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19065 from tianshan/fix_nfs_init

rgw: fix RGWLibIO did not init RGWEnv

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17094 from theanalyst/rgw-policy-tagging-2

rgw: add support for tagging and other conditionals in policy

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20226 from fangyuxiangGL/offset

rgw: can't download object with range when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #20332 from pritha-srivastava/wip-rgw-role-admin-only

rgw: Correct permission evaluation to allow only admin users to work with Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20396 from tianshan/cancel_op_miss_update_header

rgw: fix index cancel op miss update header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20459 from ceph/wip-cv-lint-fix

ceph-volume tests remove unused import

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume tests remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

buffer: Append from std::string_view

Now that we have std::string_view.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20426 from batrick/i22991

mon/MgrStatMonitor: fix formatting of pending_digest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20409 from badone/wip-better-error-reporting-ceph_test_admin_socket_output

test/admin_socket_output: better error reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20370 from ovh/bp-no-normalizing-of-normalized

common/ConfUtils: check key before actually normalizing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20437 from tchaikov/wip-admin-socket

common/admin_socket: validate command json before feeding it to hook

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20408 from ceph/wip-rm22785

ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/admin_socket: consolidate dump_cmd_to_json() and validate_cmd()

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: validate command json before feeding it to hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20158 from liaoxin01/fix-list-multi-upload

rgw: fix the max-uploads parameter not work
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20362 from tchaikov/wip-mgr-py3

mgr: fix py3 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20430 from tchaikov/wip-bluefs-demotion

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20410 from dzafman/wip-21218

osd: Fix assert when checking missing version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19322 from theanalyst/usage-trim-2

rgw: add an option to clear all usage entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19573 from Leeshine/wip-rgw-sync-status

rgw: improve sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19606 from cfanz/wip-rgw-fix-collect-return

rgw: fix collect()'s return in coroutine

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19815 from qrGitHub/wip-rgw-memory-optimize2

rgw: drop unused vector elements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19921 from fangyuxiangGL/lc-ret

rgw: lifecycle omap entry was removed in abnormal situation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20006 from jcsp/wip-rgw-perfcounters

rgw: set priority on perf counters

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20021 from tianshan/fix_bilog_list

cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20023 from badone/wip-rgw-better-ERANGE-error

rgw: Better ERANGE error message

Reviewed-by: <mkogan@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20024 from luomuyao/dev

rgw: fix a typo in rgw_perms[]

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: Add new snapset_inconsistency error check

Includes new test case

Caused by: 5f58301a1364e948834dabe503200dda07fc2790
This changed attr consistency checking to exclude system keys,
which required snapset to be handled just like object info.

Fixes: http://tracker.ceph.com/issues/22996
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20191 from yehudasa/wip-22833

rgw: sync tracing fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: <mkogan@redhat.com>

Merge pull request #20063 from cethikdata/rgw_quota

rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20095 from tianshan/fix_reload

rgw: fix RGWCompletionManager get_next stuck after going down

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume util.prepare extend flags when found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.constants update mount flags to include rw, normalize on list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration normalize underscore vs. whitespace named options

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check for mount/format usage from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add ceph.conf stub for custom conf files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20423 from dillaman/wip-22945

librbd: new tag should use on-disk committed position

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20442 from JordanHus/master

doc: Fixes a spelling error and a broken hyperlink.

Reviewed-by: John Wilkins

common: make ceph_clock_now() inlineable.

The `ceph_clock_now()` is a widely spread but thin routine.
All it does is wrap `clock_gettime` or `gettimeofday` with
accompanying conversion to `utime_t`.

Unfortunately, as it is defined outside of header, compilers
are enforced to generate a full-blown function. The overhead
is related not only the well visible stack smashing protection
but also to enforcing callers to go through PLT each time.

Taking into account the time getters are usually *user-space
syscalls* (leveraging e.g. the VDSO mechanism), eradicating
even small boilerplate might be beneficial.

```
0000000000000000 <ceph_clock_now()>:
   0:   48 83 ec 28             sub    $0x28,%rsp
   4:   31 ff                   xor    %edi,%edi
   6:   48 89 e6                mov    %rsp,%rsi
   9:   64 48 8b 04 25 28 00    mov    %fs:0x28,%rax
  10:   00 00
  12:   48 89 44 24 18          mov    %rax,0x18(%rsp)
  17:   31 c0                   xor    %eax,%eax
  19:   e8 00 00 00 00          callq  1e <ceph_clock_now()+0x1e>
  1e:   8b 44 24 08             mov    0x8(%rsp),%eax
  22:   48 c1 e0 20             shl    $0x20,%rax
  26:   48 89 c2                mov    %rax,%rdx
  29:   8b 04 24                mov    (%rsp),%eax
  2c:   48 09 d0                or     %rdx,%rax
  2f:   48 8b 4c 24 18          mov    0x18(%rsp),%rcx
  34:   64 48 33 0c 25 28 00    xor    %fs:0x28,%rcx
  3b:   00 00
  3d:   75 05                   jne    44 <ceph_clock_now()+0x44>
  3f:   48 83 c4 28             add    $0x28,%rsp
  43:   c3                      retq
  44:   e8 00 00 00 00          callq  49 <SubProcess::spawn()::__PRETTY_FUNCTION__+0x9>
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: Fixes a spelling error and a broken hyperlink.

Signed-off-by: Jordan Hus <jordan.hus@dicecorp.com>

journal: flush commit positions should wait for refresh

Fixes: http://tracker.ceph.com/issues/22945
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support concurrent refresh requests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19859 from yaozongyou/rgw-remove-register-warning

rgw: silence not allow register storage class specifier warning

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: dump percent_used PGMap field as float

Formatting this as a string was awkward for anyone
consuming it from other code.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #20187 from wido/influxdb-connection-error

mgr/influx: Various fixes and improvements

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20265 from ceph/wip-rm22876

ceph-volume: lvm zap will unmount osd paths used by zapped devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: fix index cancel op miss update header

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #20406 from dillaman/wip-22979

librbd: fix issues discovered in clone v2 during upgrade tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

* we should update the _size field when demotion kicks in, otherwise the
  interval_set::m and interval_set::_size won't be consistent.
* also rename post_process to claim, as it "steals" the leftover(s) of
chopped segment.

Signed-off-by: Kefu Chai <kchai@redhat.com>