crimson/osd/osd.cc: add editor variables/options

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: s/ostream/std::ostream/

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark one of PG::do_peering_event()s private

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #28324 into master

* refs/pull/28324/head:
client: clean up error checking and return of _lookup_parent
client: drop Client::lookup_parent

Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #28116 into master

* refs/pull/28116/head:
client: fix "ceph.snap.btime" vxattr value
client: fix "ceph.dir.rctime" vxattr value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28371 into master

* refs/pull/28371/head:
qa: fix kclient symlink to use latest rhel7
qa: fix latest distros and add helper symlinks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28148 into master

* refs/pull/28148/head:
common/secret.c: fix key parsing when doing a remount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #26415 from zmc/wip-populate-grafana

mgr/dashboard: Push Grafana dashboards on startup

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #28294 from alfredodeza/wip-rm40063

ceph-volume: pass --ssh-config to pytest to resolve hosts when connecting

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #28228 from jan--f/spec-grafana-dashboards-require-grafana

ceph.spec.in: grafana-dashboards package depends on grafana

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

mgr/dashboard: Push Grafana dashboards on startup

Because we didn't have viable way to populate Grafana dashboards in a
fully containerized context, we needed a different method. This will
work in packaged and containerized deployments.

We provide three methods to push the dashboard JSON files to the Grafana
instance: a command (ceph dashboard update-grafana-dashboards); an API
endpoint (/api/grafana/update_dashboards); and an automatic on-startup
feature that is disabled by default (ceph dashboard
set-grafana-update-dashboards true). The on-startup method will also
retry periodically to avoid racing with the startup of Grafana itself.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #28385 from theanalyst/doc/releases/13.2.6

doc: add release notes for v13.2.6 mimic

Reviewed-By: Sage Weil <sage@redhat.com>

Merge PR #28236 into master

* refs/pull/28236/head:
doc: Fix incorrect indentations in cephfs-shell
doc: Add usage for stat command in cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28386 from cbodley/wip-vstart-rgw-run

vstart: fix run() invocation for rgw

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

vstart: fix run() invocation for rgw

add an extra argument to run() for rgw so that --redirect-output
sends stdout to rgw.{port}.stdout

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #28384 into master

* refs/pull/28384/head:
mgr/volumes: fix subvolume creation with quota

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: fix subvolume creation with quota

A quota is set on a subvolume by setting a quota virtual
xattr on a fs subdirectory. The value of the quota xattr
should be a bytes object. The quota value wasn't correctly
converted to a bytes object. Fix this.

Fixes: http://tracker.ceph.com/issues/40152
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #28252 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #28380 into master

* refs/pull/28380/head:
mgr/volumes: convert string to bytes object

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

doc: add release notes for v13.2.6 mimic

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/volumes: convert string to bytes object

mount method in libcephfs's cython modules expects filesystem_name to
be a bytes object. Therefore, convert string to bytes object
beforehand.

Fixes: http://tracker.ceph.com/issues/39750
Signed-off-by: Rishabh Dave <ridave@redhat.com>

cmake: extract rgw_*_scheduler.cc into rgw_schedulers

i was running into following issue:

../../../lib/libradosgw_a.a(rgw_loadgen_process.cc.o): In function
`RGWLoadGenProcess::run()':
/home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_loadgen_process.cc:102:
undefined reference to `signal_shutdown()'
collect2: error: ld returned 1 exit status

signal_shutdown() is only used by rgw_main.cc to notify the main thread,
but we don't need this facility in unit test exercising the schedulers,
so, in this change, rgw schedulers are extracted into librgw_schedulers,
so this library can be consumed by the unit test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28243 from cbodley/wip-40022

radosgw-admin: fix syncs_from in 'bucket sync status'

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #28156 from cbodley/wip-rgw-realm-list-api

rgw: add 'GET /admin/realm?list' api to list realms

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #28345 from tianshan/fix_http_return_check

rgw: fix miss handle curl error return

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28303 from cbodley/wip-39992

rgw: fetch_remote_obj() compares expected object size

Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>

Merge pull request #28188 from tianshan/fix_39984

rgw: fix bucket may redundantly list keys after BI_PREFIX_CHAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27993 from liuchang0812/rgw-bucket-tagging

rgw: bucket tagging

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: cleanup bucket tags code

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: show bucket tagging when user stats bucket

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

test/rgw: test truncated reads in RGWHTTPManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix miss handle curl error return

if meet tcp packet loss, curl return will be result=18(CURLE_PARTIAL_FILE)
and http_status=200, so sync will continue and cause content miss match.

Fixes: https://tracker.ceph.com/issues/39992
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #28362 from cyx1231st/wip-seastar-fix-random-failure

test/crimson: fix msgr test of ref counter racing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: fix msgr test of ref counter racing

Foreign-dispatch in crimson-messenger is not implemented yet, and no
need to test this feature now.

Fixes: http://tracker.ceph.com/issues/36405
Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

rgw: simplify RGWObjTags dump

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

doc: bucket tagging is supported now

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: forwards to master and uses gw_bucket_set_attrs

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: make max number of bucket/object tags configurble

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: iam policy about bucket tagging

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: impl put_bucket_tags and delete_bucket_tags

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: add set_bucket_tags api

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: uses 404 http code when there is NoSuchTagSetError

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: get bucket tagging

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: declares bucket tagging op classes

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw:new op type and error code about bucket tagging

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

qa: fix kclient symlink to use latest rhel7

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix latest distros and add helper symlinks

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: clean up error checking and return of _lookup_parent

ll_lookup_inode can end up getting back 0 from _lookup_parent,
without zeroing out the parent pointer, which ends up remaining
uninitialized.

Fix this by moving most of the sanity checks in _lookup_parent
into ll_lookup_inode, and only have it issue the call to the MDS.

This also allows us to do the checks in a more sane order.

Fixes: http://tracker.ceph.com/issues/40085
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge PR #26547 into master

* refs/pull/26547/head:
Added validation of zabbix_host to support hostnames, IPv4 and IPv6.
mgr/zabbix: Documentation added.
mgr/zabbix: Adds possibility to send data to multiple zabbix servers.

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge PR #27667 into master

* refs/pull/27667/head:
src/test: fix creating two different objects for testing chunked object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28200 into master

* refs/pull/28200/head:
pybind/mgr: add slash to prometheus illegal metric name characters
pybind/mgr: fix format for rbd-mirror prometheus metrics

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28180 from athanatos/sjust/wip-crimson-peering

Integrate PeeringState into crimson, fix related bugs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28068 from cbodley/wip-qa-rgw-s3a

qa/rgw: don't use ceph-ansible in s3a-hadoop suite

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #28300 into master

* refs/pull/28300/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

client: drop Client::lookup_parent

Nothing calls this method.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

cmake: link rbd_reply against rbd_replay_types

should make it `PUBLIC` because, for instance,

src/rbd_replay/actions.hpp uses
`rbd_replay::action::IoActionBase::dump()` and
`rbd_replay::action::ActionBase::dump()`

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link libosd against libos

apparently, libosd uses libos, but it does not use leveldb directly,
so s/leveldb/os/.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link rgw tests against ${rgw_libs}

if `WITH_RADOSGW_AMQP_ENDPOINT` is enabled, we should always link rgw
applications against `RabbitMQ::RabbitMQ` or `amqp_mock`, so, to
simplify the linkage dependencies, we should just use `${rgw_libs}`
instead of using `rgw_a` directly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/rgw: add dnsmasq back to s3a-hadoop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw task can override --rgw-dns-name on the command line

the value for rgw_dns_name isn't known until a machine is assigned, so
it can't be set via 'override: conf:'. add a per-client config option
to the rgw task so it can add the endpoint's hostname and/or s3website
hostname on the radosgw command line

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: remove ceph-ansible from s3a-hadoop suite

Fixes: http://tracker.ceph.com/issues/39706
Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: use default ports (80 or 443) unless overridden

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: allow rgw client config to override port

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28146 from smanjara/wip-39551

rgw/multisite:RGWListBucketIndexesCR for data full sync pagination

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26915 from dvanders/dvanders_doc_heap

doc: describe metadata_heap cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28359 from tchaikov/wip-split-ceph-dencoder

tools/ceph-dencoder: split types.h into smaller pieces

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: describe metadata_heap cleanup

Fixes: http://tracker.ceph.com/issues/18174
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #28305 from rzarzynski/wip-bug-38827

qa/valgrind.supp: generalize the whiterule for aes-128-gcm to help rgw suite.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25498 from cbodley/wip-27217

rgw: parse_copy_location defers url-decode

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #16659 from rzarzynski/wip-rgw-20797

rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12709 from rzarzynski/wip-rgw-18364-dbgs

rgw: improve debugs on the path of RGWRados::cls_bucket_head.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12470 from rzarzynski/wip-rgw-18241

rgw: dump s3_code as the Code response element in RGWDeleteMultiObj_ObjStore_S3

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: link libkv against libheap_profiler

PriorityCache.cc uses `ceph_heap_*` functions, and libkv contains
`PriorityCache.cc`, so it should link against libheap_profiler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link libkv against common_prioritycache_obj

as both `kv/rocksdb_cache/BinnedLRUCache.cc` and
`kv/rocksdb_cache/ShardedCache.cc` uses `PriorityCache` which is in turn
offered by common_prioritycache_obj, we need to link libkv against the
latter.

libos uses `PriorityCache` as well, but since libos links against
libkv, so it's safe.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove kv_objs target

libkv is better than kv_objs, as it is able to bring more
properties with with it, for instance, the public include
directories, the public/private libraries it is linked against.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28339 from tchaikov/wip-cmake-asan

cmake/modules: replace ";" with " " in compile flags

Reviewed-by: Yingxin Cheng <yingxincheng@gmail.com>

ceph.spec.in: install grafana dashboards world readable

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

tools/ceph-dencoder: split types.h into smaller pieces

instead of putting all types in a single compilation unit, split them
into smaller groups, in hope to reduce the memory footprint of compiler
when compiling ceph_dencoder.cc.

sometimes, GCC just fails to compile this source file.

Fixes: http://tracker.ceph.com/issues/39595
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #28354 into master

* refs/pull/28354/head:
doc: Add warning that the root directory cannot be fragmented

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd/: clarify rebuilt_missing_with_deletes and set in init

Updates PeeringState::init to set may_include_deletes_in_missing when
possible to avoid needlessly rebuilding the missing set on the next map.
The previous behavior was also subtly wrong as until the the next map
the osd would be possibly adding deletes to the missing set without the
flag having been set in the missing set on disk

This patch also renames the flag to reflect the fact that it simply
denotes wheter the may_include_deletes_in_missing omap entry needs to be
written out.  This may happen during rebuild_missing_set_with_deletes,
but it'll also happen more commonly during init now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: hook PeeringState into pg

This patch:
1) Adds partially implemented stubs for PeeringState::PeeringListener
2) Revamps the message handling to batch the peering events from
   a single message.
3) Replaces the message handling with hand-offs to PeeringState
4) Adds support for MOSDPGCreate2 enabling creation of pgs.
5) Introduces ShardServices for exposing osd machinery to the pgs.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/main: add --debug option

Really, what we need to do is examine the ceph conf and set
the right logger levels based on configured log levels, but
this'll suffice for now.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: clear exists in remove

Signed-off-by: Samuel Just <sjust@redhat.com>

cyan_store: map -ENOENT to 0 for _remove

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/ProtocolV1: close connection when replaced

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "cmake: define BOOST_SP_DISABLE_THREADS for crimson"

For whatever reason, this patch appears to cause the reactor/peering
to hang much of the time.  Reverting until we have time to debug it.

This reverts commit a09a4f342c0a0fddcae29fe6c8d97e3b353f8c0c.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/assert.cc: flush cout before abort

Seastar logger() instances don't flush the channel, so recent messages
won't be present otherwise.

Signed-off-by: Samuel Just <sjust@redhat.com>

dout: let <= 20 correspond to debug for crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: add --redirect-output option to redirect output for --nodaemon

Useful for crimson which doesn't currently support writing directly
to a log file.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: use configured ceph-osd uuid in mkfs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/Connection: add peer_is_* helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/net/SocketMessenger: wait for accepter thread to die on shutdown

Otherwise, it ends up trying to operator<< *this after it's been
deleted.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/mon/MonClient: handle mon client connection race

The first to establish a connection and close the others needs
to ensure that the others don't mess with broken state.  To that
end, leave a cleared pending_conns map while the conns are being
closed.  Also, take care to clean up outstanding promises and
fallout from conns disappearing.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/cyan_store: run t.get_on_* contexts

PeeringState makes liberal use of those callbacks, run them
upon completion of do_transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/cyan_store: add some omap machinery

Adds transaction methods for PeeringState.

Mainly omap_set, but also a stub for collection hint and the other omap
update methods.  Also adds omap_get and debugging to dump the
transaction upon encountering a problem.

Also fixes up std and ceph::bufferlist namespace qualifications.

Signed-off-by: Samuel Just <sjust@redhat.com>

os/cyan_collection: add pragma once

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: move pg/osd state machinery into osd_types for use in crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

PGLog: add cyan_store machinery for reading the log

Signed-off-by: Samuel Just <sjust@redhat.com>

doc: Add warning that the root directory cannot be fragmented

Documents: #39704

Signed-off-by: Nathan Fish <lordcirth@gmail.com>

Merge PR #28199 into master

* refs/pull/28199/head:
qa/tasks: upgrade the check for -c sudo option in vstart_runner.py

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>