os/bluestore: fix pmem osd build problem

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #28593 from smithfarm/wip-40398

cmake: set empty INSTALL_RPATH on crypto shared libs

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28623 from cyx1231st/wip-seastar-msgr-socket

test/crimson: add unit-test for ceph::net::Socket

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: added support for new errors

connection_aborted and invalid_argument.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: add unit-test for ceph::net::Socket

Added unittest_seastar_socket to make sure that seastar network layer
works as expected.

Added missing error code invalid_argument and address_in_use.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge pull request #28747 from tchaikov/wip-more-than-activate

crimson/osd: update peering_state in PG::on_activate_complete()

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: update peering_state in PG::on_activate_complete()

so that the PG can mark itself clean when it is activated and does not
need recovery.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use get_osdmap_epoch()

would be better to use the interfaces of the PG class, instead of
relying on its implementation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28634 from tchaikov/wip-22227

common, auth: use boost::spirit to parse ceph.conf, escape quotes in exported auths

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

common/ConfUtils: implement trim helpers using boost

remove unused trim_whitespace() and its tests. as it is not used anymore

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits: add auth caps allow command case

this case introduces multiple quotes in caps line
it will trigger the bug like http://tracker.ceph.com/issues/22227

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

auth/KeyRing: escape quotes around commands

so the exported auth can be re-imported.

Fixes: http://tracker.ceph.com/issues/22227
Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: parser returns up to one error

since config parser bails out at seeing the first error, there is no
need to offer the interfaces to report multiple errors.

Signed-off-by: Kefu Chai <kchai@redhat.com>

src: use spirit based ceph.conf parser

before this change, the ceph.conf parser was implemented using a
hand-written state machine. which works fine. but it's hard to
understand and hence difficult to maintain.

in this change, it's rewritten using boost::spirit. which well, is also
a state machine under the hood, but the declarative pattern matching
rules help the human readers to understand the machinary.

before this change, the state machine is able to skip the lines with
invalid syntax and spit a warning message for each of them. after this
change, the parser will just bail out when it fails to match the rules.

and the line continuation marker, i.e., "\" is simply skipped, so it's
allowed to have nothing after the backslash in the value of key-value
pair.

changes are listed as following:

* invalid UTF-8 key/value is now treated as an error
* backslash at the very end is now allowed
* repeated settings are tolerated, and no warnings will be printed.
  the last one wins. before this change, we will print out warning
  messages at seeing lines with duplicated keys.
* malformed section header is now an error
* a setting line which does not parse is now an error
* unmatched quotes in setting value is now an error
* "\" following an empty line is allowed now.
* the format of error message is changed, so the cli test is updated
  accordingly.
* [global] section is not added by default. unless there is a single
  line in the conf file. this is only for the convenience of testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28695 from kamoltat/wip-mgr-progress-fix-bug-osd-in-out

mgr/progress: Bug fix complete event when OSD marked in

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28689 from tchaikov/wip-crimson-add-me-to-crush

crimson/osd: add osd to crush when it boots

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28688 from ifed01/wip-ifed-fix-no-compress

os/bluestore: load OSD all compression settings unconditionally.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28660 from liewegas/wip-40451

osd/PG: do not queue scrub if PG is not active when unblock

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28601 from rzarzynski/wip-crimson-os-omit-block-write

os/memstore, crimson/os: introduce memstore_debug_omit_block_device_write

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28510 from liuchang0812/fix-arg-parser

common: sort best-matched commond by req argument count

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28018 from hjwsm1989/wip-39625

mon: take the mon lock in handle_conf_change

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27627 from cxytt/fix-add-omap-tail

os/bluestore: create the tail when first set FLAG_OMAP

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28718 from varshar16/wip-cephfs-shell-typo-mounting

cephfs-shell: Fix typo for mounting

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #28703 from tchaikov/wip-ceph.in-asan

ceph.in: do not preload asan even if not needed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28725 from cbodley/wip-cmake-ubsan

cmake: fix WITH_UBSAN

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28395 from athanatos/sjust/wip-crimson-op

Add structures for tracking in progress operations

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix state machine operations in advance_pg_to and pg creation

Both pg creation and advance_pg_to process statemachine events and
therefore need to be under the pg process pipeline stage.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #27285 from shangdehao1/wip-librbd-shared-read-only-cache-hook

librbd: shared read-only cache hook

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: refine re-connect implement and init method

- add new status to OpenImageRequest, including
  send_parent_cache and handle_parent_cache.
- refine re-connect to remove race condition.
- fixed read error bug

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

doc: fix example command on creating ceph user

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

librbd: change init method of Parent cache to asynchronous

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: add read_object method to ParentCache

Also remove SharedPersistentObjectCacher class

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: refine hard code and remove dead code

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: add lock to resovle race condition

possible race condition w/ multiple concurrent attempts to
re-create the cache session

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

doc: better format for rbd persistent cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

librbd: remove parent cache config from ImageCtx

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: cleanup parent cache

- remove compiler warning.
- add error log level.
- modify tool/immutable_obj_cache unit testing.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: modify Parent cache interface

follow the recent ObjectDispatchInterface to adjust parent cache hook

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

doc: updating doc for rbd shared parent cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

librbd: move SharedPersistentObjectCacher to ParentCacheObjectDispatch

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: add re-connect and infligh enable feature

- reconnect : when session don't work or RO daemon crash,
  parent cache will try to re-connect daemon.
- in-fligh enable RO daemon.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

tools: modify connect implement of RO cache

- In order to support re-connect, add async connect at CacheClient.
- sync connect is based on C_SaferCond + async_connect.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: fix double free issue for CacheGenContextURef

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: add TypeTrait class to librbd/cache

- add TypeTraits to librbd/cache
- remove the second template parameter of parent image dispatch
- remove m_cache_client and m_image_ctx to private selection,
  and add two public method to get these two data member.
- modify parent image cache uite testing

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: cleanup RO cache

- cleanup invalid comments
- cleanup useless method and data member
- fix spelling mistake
- change naming from SharedReadOnlyCache to ParentCache
- change file name from SharedReadOnlyObjectDispatch
  to ParentCacheObjectDispath

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: change RO hook naming.

change SharedReadOnlyCache to ParentCache

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

doc: adding docs for rbd shared read-only parent image cache

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

librbd: modify RO hook implements

modify read and handle_read_cache method, and cleanup.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: don't append zeros for RO caching reading

Just return actual contents.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: keep gen lambda context small

move all codes of lambda context to handle_read_cache.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: introduce new context type at RO

 - CacheGenContextURef is wrapper GenContextURef
 - use make_gen_lambda_context to create context.
 - smart pointer unique_ptr to auto release object.

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: modify flush interface of RO

Add to journal_tid parameter to flush interface
Delete LibrbdWriteback head file

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

test: add librbd/cache/ParentImageCache UT

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

test: add domain socket mock

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: template CacheClient at RO hook

Signed-off-by: Dehao Shang <dehao.shang@intel.com>

librbd: enable librbd hook for shared RO cache

Signed-off-by: Dehao Shang <dehao.shang@intel.com>
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge pull request #28665 from mikechristie/wip-doc-mikechristie-update-igw-ansible

doc: update ceph ansible iscsi info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28721 from yuriw/wip-yuriw-crontab-master

qa/tests: changed the TO email to 'ceph-qa@ceph.io'

doc: Add rbd-target-api ceph iscsi ansible vars

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Update rbd-target-gw behavior info

The lio setup and teardown functionality was moved from rbd-target-gw to
rbd-target-api in ceph-iscsi.

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Update ceph-ansible iscsigws settings

The ceph-ansible ceph-iscsi support does not allow setting up iscsi
objects. It must be done through gwcli like is done in the container
case. This drops the iscsi object related settings and admin info for
it.

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Fix ceph-ansible iscsigws command

Use the more standard --limit iscsigws argument instead of running the
playbook command on the iscsigws.yml file directly.

Signed-off-by: Mike Christie <mchristi@redhat.com>

doc: Fix igw ceph ansible host group name

The ceph-ansible iscsi group name is iscsigws and not
ceph-iscsi-gws.

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge PR #28624 into master

* refs/pull/28624/head:
mgr/zabbix: encode string for Python 3 compatibility

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28658 into master

* refs/pull/28658/head:
qa/suites/rados/thrash: force normal pg log length with cache tiering

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28662 into master

* refs/pull/28662/head:
osd: share curmap in handle_osd_ping

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28430 from cbodley/wip-40187

cls/rgw: keep issuing bilog trim ops after reset

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

cmake: fix WITH_UBSAN

CMake Error at cmake/modules/FindSanitizers.cmake:28 (message):
  Unsupported sanitizer: undefined-behavior

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28667 from majianpeng/msg-async-comments

msg/async: add comments for commit 294c41f18adada6ab.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: changed the TO email to 'ceph-qa@ceph.io'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28704 from joke-lee/sts_assume_policy

rgw: Policy should be url_decode when assume_role

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

msg/async: add comments for commit 294c41f18adada6ab.

Consider this case:
    send-thread                                 msg-work
                                            write_event()
                                             r = write_message()
connection->write_lock.lock()
if (.. && !write_in_progress) {
  write_in_progress = true;
  add external_event
}
connection->write_lock.unlock()
                                          connection->write_lock.lock()
                                          if (r > 0)
                                            break;
                                          } while ();
                                          write_in_progress = false;

For this case, we don't add external_event and in write_event we don't
check out_q whether empty rather than break.
This make msg-work never wake up again.
Fortunately, if write_message() > 0, in AsyncConnection::_try_send will
add an EVENT_WRITEABLE to wake up msg-work. So bug can't occur.

I add comment to descript this and hope get the better method to fix
this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: empty INSTALL_RPATH for libceph_crypto_openssl.so

See also 235448879e5e05808e10c2ad8e09f87a05d76179

Fixes: http://tracker.ceph.com/issues/40398
Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: empty INSTALL_RPATH for libceph_crypto_isal.so.1.0.0

Fixes: http://tracker.ceph.com/issues/40398
Signed-off-by: Nathan Cutler <ncutler@suse.com>

cephfs-shell: Fix typo for mounting

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #28709 from tchaikov/wip-seastar

seasetar: pick up the latest seastar

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

seasetar: pick up the latest seastar

couple noticeable changes

* use dpdk v19.05, please note the older version of dpdk might not
  work, for instance dpdk v18.11 does not compile with the latest
  seastar master HEAD
* future cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28707 from majianpeng/perf-async-msgr-remove-unsued-header

test/crimson/perf_async_msgr: remove unsued header file.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/perf_async_msgr: remove unsued header file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #28684 from tchaikov/wip-silence-warnings

src/: silence GCC warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #28677 from liewegas/wip-recovery-nvme

osd: add hdd and ssd variants for osd_recovery_max_active

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

rgw, Policy should be url_decode when assume_role

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #17619 from liuchang0812/wip-ec-below-min-size

osd: allow EC PGs to do recovery below min_size

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.in: do not preload asan even if not needed

if ceph python bindings are compiled with ASan enabled has nothing to do
with Seastar. so no need to check for Seastar related compiling
settings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28697 from cbodley/wip-qa-rgw-swift-skip-client

qa/rgw: fix swift warning message

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/rgw: fix swift warning message

  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/swift.py", line 245, in task
    log.warning('Swift tests cannot run on rhel 7.6+, skipping client {client}'.format(client))
KeyError: 'client'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/progress: Bug fix complete event when OSD marked in

Swap the order of checking for complete event
when OSD is marked in and creating a
PG Recovery event so that the event
doesn't cancel itself.

Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #28657 from dillaman/wip-40435

rbd: use the ordered throttle for the export action

Reviewed-by: Mykola Golub <mgolub@suse.com>

mon: take the mon lock in handle_conf_change

Let the finisher thread to do the job, it can hold the mon lock
and not hold the config::lock, so can avoid dead lock with mon::tick thread.

Fixes: http://tracker.ceph.com/issues/39625
Signed-off-by: huangjun <huangjun@xsky.com>

Merge PR #28567 into master

* refs/pull/28567/head:
Revert "mon/OSDMonitor: report pg[pgp]_num_target instead of pg[pgp]_num"

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

crimson/osd: add osd to crush when it boots

this maps the corresponding behavior of classic OSD, we could rely on
the start script to do this job. but statfs reported by object storage
is more accurate than what we have using `df`.

in future, we could extract this feature out of crimson-osd by querying
crimson-osd for the size of underlying storage. and prepare the mon
command using a script. but let's have this function builtin in
crimson-osd at this moment. so it's compatible with classic OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: add CyanStore::stat()

so crimson-osd can add itself to CRUSH map when booting up.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush,osd: add operator<<(ostream, const CrushLocation&)

* add operator<<(ostream&, const CrushLocation&) out from OSD.cc, so it can
  be reused by crimson-osd
* update OSD.cc to use operator<<(ostream&, const CrushLocation&)
* use ceph::mutex instead of std::mutex. so CrushLocation can be used by
  crimson-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: s/override/final/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28674 from tchaikov/wip-doc/crimson

doc/dev: add crimson.rst

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28462 from myoungwon/wip-doc-dedup

doc: add a deduplication document

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: load OSD all compression settings unconditionally.

Inititally we don't load them if OSD compression mode is set to none.
But we might need them when pool's compression is enabled but some
settings aren't set.

Fixes: http://tracker.ceph.com/issues/40480
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26251 from majianpeng/add-new-op-create

os/objectstore: add new op OP_CREATE for create a new object.

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add a deduplication document

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

doc/dev: add crimson.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>