rgw: dont use RGWOmapAppend for error_repo

the error_repo writes need to be synchronous

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove per-bucket-shard sync leases

bucket sync now gets a const pointer to the DataSyncShard's lease to
check whether the lease has expired

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ContinuousLeaseCR::is_locked() is const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ContinuousLeaseCR doesn't need locks/atomics

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use objv_tracker to read/write bucket sync status

use cls_version on bucket sync status to detect racing writes - whether
from other gateways, or from radosgw-admin commands like 'bucket sync'
or 'bucket sync init'

classes that require a non-null version tracker take it by reference

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RadosRemoveCR takes objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SimpleRadosReadAttrsCR takes objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: SysObj service applies objv on set_attrs

like write(), we need to apply the writev back to readv

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: data sync no longer needs index_key_to_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry loops based on progress

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry does not duplicate bucket sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncSingleEntry takes cached state

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add unittest_rgw_bucket_sync_cache

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data sync cache for bucket shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: track last timestamp written for bucket sync

bucket sync remembers the latest timestamp that it successfully wrote to
the bucket sync status. data sync can use this to make future decisions
without having to reread its sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: store timestamp in InitBucketShardSyncStatus

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add struct data_sync_obligation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_single_entry() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use optional for DataSyncShardMarkerTrack

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: parse bucket-shard before DataSyncSingleEntry

it's easier for DataSyncShard to handle parsing failures before calling
MarkerTrack::start() and DataSyncSingleEntry

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: data sync tracks timestamps for error repo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add error repo primitives with timestamp comparisons

the sync error repo stores omap keys for each datalog entry that needs a
retry. this adds a new primitive for that, that also stores a timestamp
with each omap key, and will only allow overwrites/removals when a newer
timestamp is provided

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: DataSyncShardCR uses GetOmapVals

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add CR for omap_get_vals()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: incremental bucket sync status stores timestamp

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add [coarse_]real_time support to ceph_json.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: don't write to error repo for async notification failures

async notifications are just hints, and don't imply an obligation to
sync the bucket shard. if we fail to sync, don't write it to the error
repo for retry. we'll see the change later when processing the datalog

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWWaitCR

coroutines that want to sleep should just call RGWCoroutine::wait()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33982 from cbodley/wip-cls-cmpomap

cls/cmpomap: add cls module for CMPXATTR-like functionality in omap

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

options: add cmpomap to osd_class_load_list

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: add ceph_test_cls_cmpomap

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/cmpomap: add cls module for CMPXATTR-like functionality in omap

provides a cmp_vals() op similar to librados' omap_cmp(), but with full
support for the comparison modes and operations from cmpxattr(). if any
of the requested key/value comparisons are unsuccessful, the op fails
with -ECANCELED

cmp_vals() can be composed with other ops like set_omap_vals() to make
make them conditional on the successful comparison of all existing values

also provides cmp_set_vals() and cmp_rm_keys(), which apply a mutation
to only the keys that compare successfully. these enable batch
operations where, unlike cmp_vals() + set_omap_vals(), you don't want
one failed comparison to prevent operations on unrelated keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: add cls_cxx_map_get_vals_by_keys

add an objclass api for CEPH_OSD_OP_OMAPGETVALSBYKEYS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: expose cls_cxx_map_get_val in header

this was defined in osd/objclass.cc but not declared in objclass/objclass.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: add missing != operator for bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

buffer: comparison operators take const bufferlist&

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34455 from jtlayton/wip-44963

client: only set MClientCaps::FLAG_SYNC when flushing dirty auth caps

Merge pull request #33761 from jtlayton/wip-44441

doc: document wsync and nowsync mount options in mount.ceph(8)

Merge pull request #34231 from majianpeng/bluefs-fix-invalidate-cache-bug

os/bluestore: don't foreget invalidate cache of FileReaderBuffer.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #34272 into master

* refs/pull/34272/head:
mgr/PyModuleRegistry: ignore 'obsolete' modules
qa/suites/fs/upgrade: start with msgr2
qa/suites/fs/upgrade: s/mimic/nautilus/
qa/suites/upgrade: whitelist when orchestrator_cli can't be loaded
qa/standalone/mon/msgr-v2-transition: remove test
mon/MgrMonitor: disable orchestrator_cli if >= octopus
qa/suites/ugprade: remove mimic -> pacific upgrade tests
qa/standalone/mon/misc: update TEST_mon_features
mon/MgrMap: use always_on_modules from most recent release
mon/MonCommands: require-osd-release pacific
qa/suites/rados/thrash-old-clients: adjust octopus package list for el7
doc/dev/release-kickoff-checklist: include ceph-build change
qa/workunits/cephtool/test.sh: fix require-osd-release test
mon/MgrMonitor: define mgr modules for pacific
doc/dev/release-checklists: generalize
doc/dev/release-kickoff-checklist: quotes
qa/suites/rados/thrash-old-clients/1-install/octopus
qa/releases/pacific.yaml
mds/cephfs_features: CEPHFS_CURRENT_RELEASE
doc/dev/release-kickoff-checklist: a few other items
osd: remove redundant check
mon/OSDMonitor: make pacific changes
mon/mon_types: add mon pacific feature
include/ceph_features: define SERVER_PACIFIC
include/ceph_features: fix whitespace
mon: persistent INCOMPAT_PACIFIC item
ceph-volume: __release__ == pacific
script: update backport scripts for now-released octopus
common: pacific release constants
doc/dev/release-kickoff-checklist
src/ceph_release: pacific, 16, dev
CMakeLists: 16.0.0

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34525 from Dmole/patch-5

doc: Update index.rst

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32297 from varshar16/wip-mgr-nfs-ganesha-exports

Create and delete exports for nfs ganesha with mgr volume

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/PyModuleRegistry: ignore 'obsolete' modules

Old modules may be in the mgrmap (and always_on) but no longer exist.  Do
not try to load those or raise errors about them.

Signed-off-by: Sage Weil <sage@redhat.com>

Update index.rst

clean up "only only"

Signed-off-by: Tim <elatllat@gmail.com>

Merge pull request #34348 from ifed01/wip-ifed-fix-enforce

os/bluestore: apply bluestore_debug_enforce_settings when choosing mi…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34421 from ifed01/wip-ifed-kill-buf-in-read

os/bluestore: kill buf param in BlueFS::read

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34468 from tchaikov/wip-mgr-telegraph

mgr/telegraf: catch FileNotFoundError exception

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #34084 from sebastian-philipp/cephadm-host-ls-offline

mgr/cephadm: enhance `host ls`

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34382 from mgfritch/cephadm-nfs-grace

cephadm: add config required for ganesha grace

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #34488 from tchaikov/wip-dmclock

dmclock: pick up change to s/uint/unsigned/

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #34508 from tchaikov/wip-45022

selinux: add "type http_cache_port_t" in require section

Reviewed-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #34494 from idryomov/wip-krbd-unmap-spacing

tests: update unmap.t for table spacing changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

selinux: add "type http_cache_port_t" in require section

this addresses the regression introduced by
611a2a7ce1713cf92797b19d8a9a93a7dfc68fa5

Fixes: https://tracker.ceph.com/issues/45022
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34499 from ceph/wip-rtd

doc: enable Read the Docs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34487 from tchaikov/wip-test-cronjob-py3

qa/tests: pass "-t 2" to teuthology-suite when testing non-master bra…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

doc: add in-doc search from read the docs

readthedocs-sphinx-search features better user experience than the
builtin search offered by sphinx

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: use plantweb as fallback of sphinx-ditaa

RTD does not support installing system packages, the only ways to install
dependencies are setuptools and pip. while ditaa is a tool written in
Java. so we need to find a native python tool allowing us to render ditaa
images. plantweb is able to the web service for rendering the ditaa
diagram. so let's use it as a fallback if "ditaa" is not around.

also start a new line after the directive, otherwise planweb server will
return 500 at seeing the diagram.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add CACHEINODE and EXPORT_DEFAULTS sections

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add config required for ganesha grace

adds minimal config for ganesha-rados-grace

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add userid during grace add/remove

ganesha-rados-grace needs a rados userid to add/remove from the grace db

Mar 27 10:05:25 host1 bash[11684]: rados_connect: -13
Mar 27 10:05:25 host1 bash[11684]: Can't connect to cluster: -13

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34502 from ceph/kalebskeithley-patch-1-2

Update ceph.te

Merge pull request #34484 from cbodley/wip-sub-spawn

spawn: update submodule

Update ceph.te

Fixes: https://tracker.ceph.com/issues/45022
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

doc/conf.py: exclude pybindings docs from build for RTD

because it'd difficult to prepare (dummy) librados,libcephfs and librbd for
their python bindings in the building environment offered by Read the Docs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34314 from ishanrai05/html-cleanup

mgr/dashboard: Clean up HTML

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34223 from liewegas/cephadm-dashboard-grafana-ssl-verify

mgr/cephadm: disable dashboard's grafana cert ssl_verify if we generate it

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34454 from mgfritch/cephadm-osd-fsid-test

qa/workunits/cephadm/test_cephadm.sh: add fsid to ceph-volume cmds

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34249 from mgfritch/cephadm-nfs-common-config

mgr/cephadm: nfs common config

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #34485 from mgfritch/cephadm-no-func

cephadm: show error when no command is specified

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34411 from votdev/issue_44589

mgr/dashboard: lint error on plugins/debug.py

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

tests: update unmap.t for table spacing changes

Introduced in 4d42b4c5a0ed ("common/TextTable: default to 2 spaces
separating columns") and 41f003518a07 ("common/TextTable: only pad
between columns").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34452 from rhcs-dashboard/fix-44923-master

mgr/dashboard: use FQDN for failover redirection

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34471 from idryomov/wip-rbd-fio-rstrip

qa/tasks/rbd_fio: unbreak after the conversion from StringIO

Reviewed-by: Kefu Chai <kchai@redhat.com>

readthedocs: add .readthedocs.yml

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34408 from dillaman/wip-44727

rbd-mirror: improved replication statistics

Reviewed-by: Mykola Golub <mgolub@suse.com>

dmclock: pick up change to s/uint/unsigned/

to address the FTBFS with musl libc which does not define uint.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: pass "-t 2" to teuthology-suite when testing non-master branches

-t is an alias of "--teuthology-branch" which defaults to master. we
have a "py2" branch in teuthology. that branch is compatible with
python2, and the teuthology-worker is executed in a Python2 environment,
so all qa suites tested using "py2" branch of teuthology should work
fine as long as they are Python2 compatible. currently, all LTS branches
are compatible with Python2, and only master is also compatible with
Python3. but we plan to migrate master and octopus' qa suites to
python3 to ease the pain of maintainance in long term.

to achieve this goal, in this change, all non-master branches are now
tested using the "py2" teuthology branch.

once the python3 compatible changes are backported to octopus, "-t 2"
options will be removed from the command lines testing octopus.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34264 from tchaikov/wip-qa-py3

qa/tasks: be python3 compatible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34284 from liewegas/followon-34266-cleanup

mgr/DaemonServer: add missing ceph_abort

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs/upgrade: start with msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: show error when no command is specified

Traceback (most recent call last):
  File "./cephadm", line 4452, in <module>
    if args.func != command_check_host:
AttributeError: 'Namespace' object has no attribute 'func'

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: use fsid during dump_log

needed when running with multiple ceph clusters

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34414 from yuvalif/add_timeout_to_http_client

rgw/http: add timeout to http client

Reviewed-by: Casey Bodley <cbodley@redhat.com>

spawn: update submodule

pulls in fixes for exception handling, and support for multiple return
values with std::tuple<>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34463 from jan--f/c-v-batch-filter-check-lvs-before-access

ceph-volume/batch: check lvs list before access

Merge pull request #34472 from jan--f/c-v-noninteractive-batch-idempotency-all-filtered

ceph-volume/batch: return success when all devices are filtered

rbd-mirror: fixed race condition with snapshot sync and shutdown

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume/batch: return success when all devices are filtered

batch should only return an error if some (but not all) devices are
filtered. When only some devices are filtered the resulting osd layout
could look very different from what a user expects. If all devies are
filtered just return success.

Fixes: https://tracker.ceph.com/issues/44994
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/suites/fs/upgrade: s/mimic/nautilus/

Also rename 3-compat_client/mimic -> pacific to match the min_client
setting (the naming here confuses me...).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: whitelist when orchestrator_cli can't be loaded

This happens because mgrs may start before mons, or before mons have
published a new mgrmap that disables orchestrator_cli.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/msgr-v2-transition: remove test

v2 was introduced in nautilus, and we don't support mimic -> pacific
upgrades (only mimic -> octopus).  This test can be removed!

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: disable orchestrator_cli if >= octopus

Fixes: https://tracker.ceph.com/issues/44883
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/ugprade: remove mimic -> pacific upgrade tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/misc: update TEST_mon_features

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMap: use always_on_modules from most recent release

If our release isn't in the always_on_map, say because you just upgraded,
then start with the modules from the last/most recent release.  This
should be sufficient to allow an upgrade when the mon is still the old
release and the mgrmap doesn't tell us what modules we should be enabling.

(Arguably, the always_on_modules should be enshrined in the MgrMap itself
and not in the mon?)

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonCommands: require-osd-release pacific

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: adjust octopus package list for el7

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/release-kickoff-checklist: include ceph-build change

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: fix require-osd-release test

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: define mgr modules for pacific

Signed-off-by: Sage Weil <sage@redhat.com>