osd/PrimaryLogPG: cancel all objecter ops atomically

We want to avoid a situation like this:

- start proxy op A (epoch E)
- start proxy op B (epoch E)
- start proxy op C (epoch E)
- objecter sends none of these because target is down in epoch E
- osdmap update to E+1
- pg cancels requeues A, B
- objecter updates to E+1
- objecter sends C
- pg cancels/requeues C

Note that the key thing is that operations on each object are canceled
atomically.  On the interval change we do it all at once.  In the other
cases, we cancel everything on the given object together.

Fixes: http://tracker.ceph.com/issues/22123
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: allow multiple ops to be canceled atomically

This allows multiple cancellations without worrying about a race with an
(objecter) osdmap update.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20031 from tchaikov/wip-doc-macos

doc/dev/macos: add doc for building on MacOS

Merge pull request #20474 from wjwithagen/wjw-wip-pthread_set_name_2

include: compat.h, fix the return result of pthread_set_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20168 into master

* refs/pull/20168/head:
Handle dir->dentries.empty()
client: lookup_ino when i1 becomes NULL

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20424 into master

* refs/pull/20424/head:
qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20428 into master

* refs/pull/20428/head:
qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20470 into master

* refs/pull/20470/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

include: compat.h, fix the return result of pthread_set_name()

On FreeBSD the function type is void(). Make it compatible with return
value on other platforms.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa: ignore full cluster warnings for mds-full

Fixes: http://tracker.ceph.com/issues/23033
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20274 from adamemerson/wip-socket-to-me!

common/admin_socket: various cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: Use unique_ptr instead of new/delete

Mostly for the default hooks, plus one Formatter.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Cleanup path cleanup

Don't use strdup/free explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config/admin_socket: Remove PFL_* stuff since nobody's using it

If we want to, we have other ways of getting data out of a thread.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use one map instead of three

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use std::mutex/condition_variable/thread

And fix a bug where we exit in one code-path without releasing the
lock.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_named_thread

To allow us to make a std::thread with a name.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Copy fewer strings

Pass std::string_view to call, rather than passing strings by value.

Change cmdmap_t and the maps in admin_socket to use the std::less<>
transparent comparator, so we can compare string_views to strings. (Or
anything else with an equality operator with strings.)

Pass the cmdmap_t in to call by const reference, not reference.

The change to cmdmap_t requires changing some of the users of
cmdmap_t. (That is, make them actually use cmdmap_t explicitly, not
the open-coed map type.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17020 from theanalyst/lc-time-disable

qa: re enable LC tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16757 from rzarzynski/wip-rgw-20883

rgw: dump Last-Modified in Swift's responses for GET/HEAD on container.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15054 from rzarzynski/wip-rgw-fix-browser-upload-error-handling

rgw: fix error handling in Browser Uploads.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12427 from linuxbox2/wip-rgw-ldap-shutdown

rgw: don't leak S3 LDAPHelper

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19065 from tianshan/fix_nfs_init

rgw: fix RGWLibIO did not init RGWEnv

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17094 from theanalyst/rgw-policy-tagging-2

rgw: add support for tagging and other conditionals in policy

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20226 from fangyuxiangGL/offset

rgw: can't download object with range when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #20332 from pritha-srivastava/wip-rgw-role-admin-only

rgw: Correct permission evaluation to allow only admin users to work with Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20396 from tianshan/cancel_op_miss_update_header

rgw: fix index cancel op miss update header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20459 from ceph/wip-cv-lint-fix

ceph-volume tests remove unused import

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume tests remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

buffer: Append from std::string_view

Now that we have std::string_view.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20426 from batrick/i22991

mon/MgrStatMonitor: fix formatting of pending_digest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20409 from badone/wip-better-error-reporting-ceph_test_admin_socket_output

test/admin_socket_output: better error reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20370 from ovh/bp-no-normalizing-of-normalized

common/ConfUtils: check key before actually normalizing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20437 from tchaikov/wip-admin-socket

common/admin_socket: validate command json before feeding it to hook

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20408 from ceph/wip-rm22785

ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/admin_socket: consolidate dump_cmd_to_json() and validate_cmd()

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: validate command json before feeding it to hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20158 from liaoxin01/fix-list-multi-upload

rgw: fix the max-uploads parameter not work
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20362 from tchaikov/wip-mgr-py3

mgr: fix py3 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20430 from tchaikov/wip-bluefs-demotion

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20410 from dzafman/wip-21218

osd: Fix assert when checking missing version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19322 from theanalyst/usage-trim-2

rgw: add an option to clear all usage entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19573 from Leeshine/wip-rgw-sync-status

rgw: improve sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19606 from cfanz/wip-rgw-fix-collect-return

rgw: fix collect()'s return in coroutine

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19815 from qrGitHub/wip-rgw-memory-optimize2

rgw: drop unused vector elements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19921 from fangyuxiangGL/lc-ret

rgw: lifecycle omap entry was removed in abnormal situation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20006 from jcsp/wip-rgw-perfcounters

rgw: set priority on perf counters

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20021 from tianshan/fix_bilog_list

cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20023 from badone/wip-rgw-better-ERANGE-error

rgw: Better ERANGE error message

Reviewed-by: <mkogan@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20024 from luomuyao/dev

rgw: fix a typo in rgw_perms[]

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20191 from yehudasa/wip-22833

rgw: sync tracing fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: <mkogan@redhat.com>

Merge pull request #20063 from cethikdata/rgw_quota

rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20095 from tianshan/fix_reload

rgw: fix RGWCompletionManager get_next stuck after going down

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume util.prepare extend flags when found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.constants update mount flags to include rw, normalize on list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration normalize underscore vs. whitespace named options

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check for mount/format usage from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add ceph.conf stub for custom conf files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20423 from dillaman/wip-22945

librbd: new tag should use on-disk committed position

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20442 from JordanHus/master

doc: Fixes a spelling error and a broken hyperlink.

Reviewed-by: John Wilkins

doc: Fixes a spelling error and a broken hyperlink.

Signed-off-by: Jordan Hus <jordan.hus@dicecorp.com>

journal: flush commit positions should wait for refresh

Fixes: http://tracker.ceph.com/issues/22945
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support concurrent refresh requests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19859 from yaozongyou/rgw-remove-register-warning

rgw: silence not allow register storage class specifier warning

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20187 from wido/influxdb-connection-error

mgr/influx: Various fixes and improvements

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20265 from ceph/wip-rm22876

ceph-volume: lvm zap will unmount osd paths used by zapped devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: fix index cancel op miss update header

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #20406 from dillaman/wip-22979

librbd: fix issues discovered in clone v2 during upgrade tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

* we should update the _size field when demotion kicks in, otherwise the
  interval_set::m and interval_set::_size won't be consistent.
* also rename post_process to claim, as it "steals" the leftover(s) of
chopped segment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: use terminal.MultiLogger in `lvm zap`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use api helper methods for pvs and lvs in lvm zap

This is just a cleaner way of doing this instead of using PVolumes
and Volumes directly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix documentation typos for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when zapping unmount osd directories

If you zap an lv, device or partition and it's currently mounted as a
ceph osd directory then it will be unmounted so the zap can complete.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a util.system.unmount function

This function will unmount the given path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20427 from ceph/wip-rm22988

ceph-volume use realpath when checking mounts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #19263 into master

* refs/pull/19263/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore bad backtrace cluster wrn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests verify realpath usage in mounts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system optionally use realpath on devices

To check a mounted device it is needed to verify on a combination of
realpath and plain devices against realpath and plain paths. In LVM, two
different paths might refer to the same devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19957 into master

* refs/pull/19957/head:
client: fixup parallel calls to ceph_ll_lookup_inode() in NFS FASL

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #20336 into master

* refs/pull/20336/head:
client: re-use free snap tag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20373 into master

* refs/pull/20373/head:
client: clamp I/O sizes to INT_MAX when we can't return larger values
test: new testcase for ceph_ll_readv and ceph_ll_writev
client: hook up ceph_ll_readv and ceph_ll_writev
client: type safety cleanup for _read and _write codepaths

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20386 into master

* refs/pull/20386/head:
mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore MON_DOWN while thrashing mons

Fixes: https://tracker.ceph.com/issues/22993
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: Add tests to validate scrub functionality

Add tests to ensure the scrub operation is not adversly affected
by certain metadata pathologies.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

MgrStatMonitor: fix formatting of pending_digest

Fixes: http://tracker.ceph.com/issues/22991
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore io pause warnings in mds-full test

Fixes: http://tracker.ceph.com/issues/22990
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20147 from smithfarm/wip-15904

 tools: rados: make -f be --format for consistency with ceph tool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20422 from cbodley/wip-check-cls-rgw

cmake: 'make check' builds radosgw and its cls dependencies

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: 'make check' builds radosgw and its cls dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: flush journal commit position after stopping replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: disable clone v2 for negative test cases

Fixes: http://tracker.ceph.com/issues/22979
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: don't clear CLONE_CHILD op feature if still a child

After removing the last snapshot linked to a parent image,
don't clear the CLONE_CHILD op feature bit if the image HEAD
is still linked to the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Handle dir->dentries.empty()

Handle dir->dentries.empty() (during stale file handle error) situation appropriately.

Fixes: http://tracker.ceph.com/issues/21995
Signed-off-by: Jos Collin <jcollin@redhat.com>

osd: Fix assert when checking missing version

Fixes: http://tracker.ceph.com/issues/21218
Signed-off-by: David Zafman <dzafman@redhat.com>

test/admin_socket_output: better error reporting

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19939 from liewegas/wip-osd-new-class

mon,osd: do not use crush_device_class file to initalize class for new osds

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20363 from dongbula/e-rados-get-put-command

make rados get/put/append command help txt clear

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/macos: add doc for building on MacOS

Signed-off-by: Kefu Chai <kchai@redhat.com>

client: lookup_ino when i1 becomes NULL

Also set FUSE_CAP_EXPORT_SUPPORT

Fixes: http://tracker.ceph.com/issues/21995
Signed-off-by: Jos Collin <jcollin@redhat.com>