Merge pull request #35709 from zdover23/wip-doc-preferring-yaml-to-cli-in-cephadm

doc/cephadm: add yaml info for orch apply mon

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29283 from myoungwon/wip-refcount-snap

osd: refcounting chunks for snapshotted manifest object

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com> (portions Sam wrote)

Merge pull request #35955 from jschmid1/deprecate_filstore

mgr/cephadm: do not accept filestore osds in osdspecs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/cephadm: add yaml infor for orch apply mon

This commit breaks one of the long sentences into
three shorter sentences, and adds a parenthetical
comment walking the reader through a series of commands,
explaining what each command does and why only the last
of the commands ends up having a lasting effect on the Ceph
environment.

Fixes: https://tracker.ceph.com/issues/46133
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36013 from Devp00l/wip-46376

cephadm: Make Vagrantfile more flexible

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #36072 from zdover23/wip-doc-dev-word-choice-guidance

doc/dev: remove extraneous word "guidance".

Merge pull request #35931 from tchaikov/wip-mds-denc

mds,denc: prepare for the enabling of denc support of EMetaBlob::fullbit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35755 into master

* refs/pull/35755/head:
mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Victoria Martinez de la Cruz <vkmc@redhat.com>
Reviewed-by: Goutham Pacha Ravi <gouthamr@redhat.com>

Merge pull request #35970 from athanatos/sjust/wip-fix-lba-addrs

crimson/seastore: fix lba btree internal references

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34246 into master

* refs/pull/34246/head:
mds: add request to batch_op before taking auth pins and locks
mds: move MDRequestImpl::batch_reqs into Batch_Getattr_Lookup
mds: track which map batch_op is in-use in MDRequest

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34785 into master

* refs/pull/34785/head:
ceph-fuse: show fuse helper options for libfuse >= 3.0
ceph-fuse: add splice read/write support to reduce the memory copy
ceph-fuse: add connection args parsing support for libfuse > 3.0
ceph-fuse: switch to fuse_reply_iov to reduce the memory copy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #35327 into master

* refs/pull/35327/head:
client: fix directory inode can not call release callback

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35690 into master

* refs/pull/35690/head:
mds: fix hang issue when accessing a file under a lost parent directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35848 into master

* refs/pull/35848/head:
mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35951 into master

* refs/pull/35951/head:
qa/cephfs: set omit_sudo to False for wait_until_mounted() in...

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: fix internal relative btree addrs

Internal relative addrs must be block relative on disk and record
relative in deltas.  Use maybe_generate_relative as intended.

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_transaction_manager: add a test which forces an lba split

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/test_btree_lba_manager: ensure record and block addrs are different

Adds an extra extent to each transaction in the test to ensure that
record and block relative addresses differ as they do in real usage.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #34920 from ofriedma/ofriedma-civetweb-timeout

rgw: increase civetweb request timeout to 65 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/dev: remove extraneous word "guidance".

This bug changes the sentence "If your patches emit
messages to the Ceph cluster log, please consult this
guidance:" to "If your patches emit messages to the
Ceph cluster log, please consult this:". This new sentence
is more natural and idiomatic.

Fixes: https://tracker.ceph.com/issues/46363
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #36055 from zli091/master

 rgw: lc: fix Segmentation Fault when the tag of the object was not f…

Reviewed-by: Or Friedmann <ofriedma@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

cephadm: Make Vagrantfile more flexible

Now you can use a JSON or pass multiple variable to vagrant in order to
configure the outcome of VMs you get. Similar to vstart.sh you can use
OSDS, MGRS and MONS as arguments to pass. As OSDS behave a bit different
in this scenario you can also specify the amount of extra disks and OSD
VM has.

Fixes: https://tracker.ceph.com/issues/46376
Signed-off-by: Stephan Müller <smueller@suse.com>

rgwlc: fix Segmentation Fault related to tags.

It is found that rgw crashes when putting a tag beyond the rule to the object, which can be solved by adding the return value judgment.

Fixes: https://tracker.ceph.com/issues/46485
Signed-off-by: zhuo li <lizhuo@chinatelecom.cn>
Signed-off-by: yupeng chen <chenyupeng@chinatelecom.cn>

Merge pull request #36008 from idryomov/wip-krbd-58-options

rbd: recognize crush_location, read_from_replica and compression_hint map options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: do not decode EMetaBlob::fullbit struct_v < 8

the struct_v of EMetaBlob::fullbit::encode was bumped from 7 to 8 in
d62c67ca2041797fc7aa212ae4a37ba91e5620e7, the oldest release containing
this change was hammer rc1.

`EMetaBlob::fullbit` is used to persist dentry + inode as part of
journal, after the journal is flushed to the metadata pool, and applied to
dirty metadata, the flushed segments are trimmed. also, it's not allowed
to upgrade a Ceph cluster across 2 releases. so, there is no need
to decode an on-disk struct which will be trimmmed as part of journal
recycling, and that struct was encoded in hammer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: add boost::container::small_vector support

i could use `_denc::container_base<>` and `_denc::pushback_details<>` to
implement the traits for boost::container::small_vector (bcs for short),
but the second template parameter of bcs is a value of type size_t, so
the value it not a type, forunately, C++17 allows us to declare non-type
template parameter using `auto`, but when instantiating
_denc::container_base<>, the compiler complains that it expects a
constatnt of `auto` but `T` is passed in. because i have to change the
signature of `container_base` to something like
```
template<template<auto...> class C, typename Details, auto ...Ts>
  struct container_base {
```

so, in this change, the traits for bsc is implemented from scratch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36041 from tchaikov/wip-mds-EMetaBlob

ceph-object-corpus: pick up changes to drop backward compatibility of…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-object-corpus: pick up changes to drop backward compatibility

to drop backward compatibility of EMetaBlob

see also https://github.com/ceph/ceph-object-corpus/pull/12

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35747 from adk3798/cephadm-44877

mgr/cephadm: allow custom dashboard grafana url

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #32852 from ifed01/wip-ifed-simplify-pin

os/bluestore: simplify Onode pin/unpin logic.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35973 from mlausch/mgr-progress-perf

mgr/progress: Skip pg_summary update if _events dict is empty.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35915 from jmolmo/fix_rook_create_osd

mgr/rook: Fix error creating OSDs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35968 from optimistyzy/73_debug

NVMEDevice: Fix the issue of multiple OSDs when using SPDK

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32707 from petrutlucian94/windows.4

Windows support [part 4]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35971 from tchaikov/wip-addr-parsing-strncpy

common/addr_parsing: refactor safe_cat()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35990 from votdev/fix_cephadm_py_code

cephadm: Fix Python formating issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35717 from dmick/wip-cephadm-doc

doc/dev/cephadm.rst: clarify, fix spelling nits

Merge PR #35813 into master

* refs/pull/35813/head:
vstart.sh: use output of hostname for cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #36009 into master

* refs/pull/36009/head:
doc: Update nfs-ganesha package requirements

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35930 from smithfarm/wip-luminous-eol

doc: remove luminous from active releases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/krbd: turn on balanced reads for the fsx subsuite

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd/fsx: respect rbd_default_map_options in krbd_open()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: recognize compression_hint map option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: recognize crush_location and read_from_replica map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/volumes: Deprecate protect/unprotect CLI calls for subvolume snapshots

Subvolume snapshots required to be protected, prior to cloning the same.

Also, protected snapshots were not allowed to be unprotected or removed,
if there were in-flight clones, whose source was the snapshot being
removed.

The protection of snapshots explicitly is not required, as these can be
prevented from being removed based only on the in-flight clones checks.

This commit hence deprecates the additional protect/unprotect requirements
prior to cloning a snapshot.

In addition to deprecating the above, support to query a subvolume for
supported features, via the info command, is added. The feature list
is set to "clone" and "auto-protect", where the latter is useful to
decide if protect/unprotect commands are required or not.

Fixes: https://tracker.ceph.com/issues/45371
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>

Merge pull request #36010 from votdev/issue_46450_fix_stateful_tab_unit_test

mgr/dashboard: StatefulTabDirective unit test fails

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #36003 from tserlin/private-tserlin-selinux-add-amqp-soundd-types

selinux: add amqp and soundd types to ceph.te

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: StatefulTabDirective unit test fails

Fixes: https://tracker.ceph.com/issues/46450
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #35967 from tspmelo/wip-language-selector

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc: Update nfs-ganesha package requirements

Fixes: https://tracker.ceph.com/issues/46449
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35939 from votdev/issue_46370_fix_edit_host_label

mgr/dashboard: Edit host label dialog is crashing

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34946 from votdev/issue_43120_stateful_tabs

mgr/dashboard: stay on active tab

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mds: fix hang issue when accessing a file under a lost parent directory

When a directory metadata is lost, if we looked up a file under this lost directory,
this request will hang forever.

Fixes: https://tracker.ceph.com/issues/46129
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

cephadm: Fix Python formating issues

Fix obvious Python formating errors reported by my IDE.

Signed-off-by: Volker Theile <vtheile@suse.com>

selinux: add amqp and soundd types to ceph.te

Signed-off-by: Thomas Serlin <tserlin@redhat.com>

Merge pull request #35904 from mkogan1/wip-fix-endpoint-not-connected-r2

rgw: fix boost::asio::async_write() does not return error...

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35853 from wjwithagen/wjw-fix-rgw-to-base64

rgw: include string header

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Prateek Sachan <psachan@cs.iitr.ac.in>

Merge pull request #35735 from ivancich/wip-radoslist-incomplete-multiparts-marker-maintenance

rgw: radoslist incomplete multipart uploads fix marker progression

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35525 from cbodley/wip-45969

rgw: stop realm reloader before store shutdown

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #35409 from pritha-srivastava/wip-iam-get-role-policy

rgw/iam: correcting the result of get role policy.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35189 from linuxbox2/wip-xxh-gc

rgw:gc: use XXHash for gc queue selection

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge PR #35999 into master

* refs/pull/35999/head:
doc/governance: add Patrick to CLTW
doc/governance: correct João's name

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/governance: add Patrick to CLTW

I guess I was missed!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/governance: correct João's name

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35963 from ricardoasmarques/enable-iscsi-edit

mgr/dashboard: Allow to edit iSCSI target with active session

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35983 from ceph/kalebskeithley-1854083-1

selinux: allow ceph_t amqp_port_t:tcp_socket

Merge PR #34861 into master

* refs/pull/34861/head:
test: adjust scrub control tests for optional scrub status
mgr: set `task_dirty_status` on reconnect
mds: send scrub status to ceph-mgr only when scrub is running

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: do not accept filestore osds in osdspecs

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35927 from gekios/wip-cephadm-image-fix

qa/tasks: fix cephadm config image check

Merge pull request #35919 from rzarzynski/wip-ec-clean-endecode_chunks

ec: drop stubs of {encode,decode}_chunks() from ErasureCode.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35465 from pritha-srivastava/wip-sts-token-fields-op-logs

rgw: adding user related web token claims to ops log

Merge pull request #35481 from pritha-srivastava/wip-rgw-keycloak-doc-correction

rgw/sts: Removing references to token introspection url,

Merge pull request #35858 from pritha-srivastava/wip-rgw-query-string

rgw: Normalizing X-Amz- headers for case in RGWHTTPArgs

test/store_test: add missing collection reopen

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: modify cache shutdown

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd_types: clarify comments in calc_refs_to_drop_on_removal

Signed-off-by: Samuel Just <sjust@redhat.com>

mds: do not submit omap_rm_keys if the dir is the basedir of merge.

Fixes: https://tracker.ceph.com/issues/46273
Signed-off-by: Chencan <chen.can2@zte.com.cn>

qa/tasks: fix cephadm config image check

Check container_image_name only if ceph cluster image is not pre-defined in config.
We shouldn't care about container_image_name if there cephadm or ceph already have image defined.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>

rgw: fix boost::asio::async_write() does not return error...

although remote has closed the connection

Fixes: https://tracker.ceph.com/issues/46332
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #34908 from tspmelo/wip-e2e-shared

mgr/dashboard: Add E2E tests for navigation, language and login

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/progress: Skip pg_summary update if _events dict is empty.

The get call of pg_stats and pg_ready are expensive and are not necessary in this case. This avoids blocking other modules like the prometheus module

Fixes: https://tracker.ceph.com/issues/46416
Signed-off-by: Manuel Lausch <manuel.lausch@1und1.de>

test: adjust scrub control tests for optional scrub status

Fixes: http://tracker.ceph.com/issues/45349
Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr: set `task_dirty_status` on reconnect

... so that upon reconnect the manager client updates manager
with whatever is available in ->task_status field.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: send scrub status to ceph-mgr only when scrub is running

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #35857 into master

* refs/pull/35857/head:
pybind/mgr/volumes: fix volume search for pools

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #35415 into master

* refs/pull/35415/head:
mds: make threshold for MDS_TRIM configurable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: fix volume search for pools

Loop logic would bail out if it first sees any file system that does not
match the volume it's looking for.

Fixes: https://tracker.ceph.com/issues/46277
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35868 into master

* refs/pull/35868/head:
mds: update codes of MSG_MDS_METRICS and MSG_MDS_PING

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35410 into master

* refs/pull/35410/head:
client: release the client_lock before copying data in read

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33827 into master

* refs/pull/33827/head:
mon: add 'ceph fs required_client_feature <fs_name> add/rm' command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33912 into master

* refs/pull/33912/head:
pybind: add special values for not reading conffile

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35934 into master

* refs/pull/35934/head:
pybind/cephfs: raise custom Exception with same arguments
pybind/cephfs: add DiskQuotaExceeded exception

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #35810 into master

* refs/pull/35810/head:
tool: convert str to num using strtoull

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #35361 into master

* refs/pull/35361/head:
mgr/volumes: Validate mon_allow_pool_delete before volume deletion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35979 from ivancich/wip-fix-orphan-list-teuthology

rgw: orphan list teuthology test & fully-qualified domain issue

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #35976 from idryomov/wip-make-check-msg

run-make-check.sh: fix the log message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35964 from dillaman/wip-neorados-compat

neorados: bridge support to librados cluster connections

Reviewed-by: Adam Emersen <aemerson@redhat.com>

selinux: allow ceph_t amqp_port_t:tcp_socket

allow ceph_t amqp_port_t:tcp_socket name_connect;
allow ceph_t soundd_port_t:tcp_socket name_connect;

Required for running RabbitMQ

(soundd_port_t) for running RabbitMQ on port 8000

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1854083
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #35949 from zdover23/wip-doc-dev-guide-essentials-2020-07-07

doc/devel: repair sentences in "Essentials" page

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/addr_parsing: refactor safe_cat()

* add a ROUND_UP_128() macro for calculating the round of to a multiple of
  128.
* instead of using a loop, use ROUND_UP_128() for the new size of *pstr
* rename str2 to src for better readability
* use memcpy() instead of strncpy(), as the size of src string is already known

this change also silences the warning of

ceph/src/common/addr_parsing.c: In function ‘safe_cat’:
ceph/src/common/addr_parsing.c:45:3: warning: ‘strncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Wstringop-truncation]
strncpy((*pstr)+pos, str2, len2);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ceph/src/common/addr_parsing.c:28:14: note: length computed here
int len2 = strlen(str2);
^~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>