Merge pull request #54058 from Matan-B/wip-51147-reef

reef: Crimson/osd: support multicore osd

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54069 from rhcs-dashboard/wip-63231-reef

reef: mgr/dashboard: disable hosts field while editing the filesystem

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>

Merge pull request #53921 from ljflores/wip-63168-reef

reef: qa/suites/upgrade: fix env indentation in stress-split upgrade tests

Merge pull request #54079 from zdover23/wip-doc-2023-10-19-backport-54051-to-reef

reef: doc: Update mClock QOS documentation to discard osd_mclock_cost_per_*

doc: Update mClock QOS documentation to discard osd_mclock_cost_per_*

The cost parameters (osd_mclock_cost_per_*) have been removed.
The cost of an operation is now determined using the random IOPS
and maximum sequential bandwidth capability of the OSD's underlying device.

Fixes: https://tracker.ceph.com/issues/58529
Signed-off-by: tanchangzhi <544463199@qq.com>
(cherry picked from commit a7502cb5125dda8d052dc3991ed5d7013606aca2)

Merge pull request #53654 from mcv21/wip-62970-reef

reef: Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #53823 from cfsnyder/wip-62944-reef

reef: rgw: add radosgw-admin bucket check olh/unlinked commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54071 from zdover23/wip-doc-2023-10-18-backport-54028-to-reef

reef: doc/start: Modernize and clarify hardware-recommendations.rst

Merge pull request #54073 from zdover23/wip-doc-2023-10-18-backport-54052-to-reef

reef: doc/rados: edit troubleshooting-pg.rst (1 of x)

Merge pull request #53229 from ifed01/wip-ifed-snapdiff-ref

reef: cephfs: implement snapdiff

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #52850 from lxbsz/wip-61165

reef: client: issue a cap release immediately if no cap exists

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52847 from lxbsz/wip-62192

reef: mds: do not send split_realms for CEPH_SNAP_OP_UPDATE msg

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: fix output formatting of bucket index check admin api

The bucket index check admin API was previously returning invalid
JSON.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 32fb6a1a68398a99324b2e64ebe3bcf3a9ccf02a)

Conflicts:
src/rgw/driver/rados/rgw_bucket.cc

Cherry-pick notes:
- Conflicts due to check_bad_index_multipart taking optional_yield param on main

rgw: fix radosgw-admin bucket check stat calculation bug

Fixes a regression with radosgw-admin bucket check stat
calculation and bucket reshard stat calculation when
there are objects that have transitioned from unversioned
to versioned. The bug was introduced in
152aadb71b61c53a4832a1c8cf82fce3d64b68d1.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 4728daa5557bfb79a608dd903b8630e2b15fcb2c)

rgw: add test case to reproduce bucket check stats bug for versioned bucket

Reproduces a regression where radosgw-admin bucket check incorrectly counts
objects that started as unversioned and later transitioned to versioned.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 340522f9aed50d65137568c1f9dcf4b1e7945a79)

rgw: radosgw-admin bucket check should only print index entries with --check-objects flag

Printing all index entries can be very time consuming for large
buckets and the inability to switch this behavior off makes it
cumbersome to use the command for fixing bucket stats. This was
also preventing the command from outputting recalculated bucket
stats when the --fix flag wasn't specified.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 6b057fe55413c0eaf9959f006584cba6cc4c192a)

rgw: prevent another leftover bucket index olh entry scenario

If a call to bucket_index_link_olh or bucket_index_unlink_instance
fails, its associated pending xattr may have prevented the olh object
from being removed by another thread. We should do a best effort
cleanup attempt for this case by calling update_olh before returning
an error to the caller.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 570adec5bb8142f5baf1f05f0040e8afdb11ec05)

rgw: fix rgw versioned bucket stat accounting during reshard and check index

Fixes: https://tracker.ceph.com/issues/62760
Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 152aadb71b61c53a4832a1c8cf82fce3d64b68d1)

qa/workunits/rgw: add tests that reproduce bucket stats inconsistency bugs

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit b79dcf640ac2cc3dacf1b87bbe351db823c445d0)

rgw: add radosgw-admin bucket check olh/unlinked commands

Adds commands to radosgw-admin for checking for and fixing leftover
entries in the bucket index (and associated RADOS objects).

Fixes: https://tracker.ceph.com/issues/62075
Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 9b2042a0750109f575b02c9c4b127dd941994662)

Conflicts:
src/rgw/driver/rados/rgw_rados.cc

Cherry-pick notes:
- Conflicts due to multiple methods takeing optional_yield param on main

Merge pull request #53329 from vshankar/tr-62583

reef: mds: blocklist clients with "bloated" session metadata

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #53271 from mchangir/wip-59408-reef

reef: cephfs_mirror: correctly set top level dir permissions

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #53241 from batrick/wip-62661-reef

reef: mds: drop locks and retry when lock set changes

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #53187 from batrick/wip-62609-reef

reef: mgr: register OSDs in ms_handle_accept

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #53183 from batrick/wip-62518-reef

reef: mds: acquire inode snaplock in open

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #53138 from mihalicyn/wip-62570-reef

reef: ceph_fs.h: add separate owner_{u,g}id fields

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #52975 from batrick/wip-62422-reef

reef: mds: dump locks when printing mutation ops

Reviewed-by: Xiubo Li <xiubli@redhat.com>

doc/rados: edit troubleshooting-pg.rst (1 of x)

Edit doc/rados/troubleshooting/troubleshooting-pg.rst (1 of x).

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit fbde0a441ee753c06afe3723584d6c9686ceef21)

doc/start: Modernize and clarify hardware-recommendations.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 83bd3a8dfb83c46a2ba84d9f309a6dbe72c64ea9)

Merge pull request #53954 from Matan-B/wip-52266-reef

reef: crimson/mgr: Fix config show command

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: disable hosts field while editing the filesystem

Even though the Placement field was disabled, the Host field was still
showing up in UI while Editing. That option is not possible in fs form.

Fixes: https://tracker.ceph.com/issues/63193
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit b18a1441bebdb5c997ed3970d0a53dd39cebd277)

crimson/osd: modify the interface to integrate with milticore-msg

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 00a5eaea4661e6fb23d174b2f367222541b24f21)

crimson/osd: remove seastar::now() to gurantee sequence in pipline

in debug build, if wait_barrier is seastar::now, will cause reactor
schedule the following continuation away, but can't schedule them back
in the same order as by which they are scheduled away, so the continuations
are out of order here.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit c7628de4d230cd08d536d7e0c35bf6822cc7146e)

crimson/osd: make OSDState and OSDMap_Gate sharded

put them into PerShardState to avoid cross core calling for normal cases.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 324191130ab77a06777f3a71b4c6bf995a4e5fe7)

crimson/osd: change OSD::ms_dispatch to take an ConnectionFRef

Shortly, we will need to be able to pass a ConnectionRef from another core.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit d657afc7a76d14882152d6f38da5adeaa39e55e1)

crimson/osd: relax requirement that PGShardManager methods be invoked on PRIMARY_CORE

Proxy uses of osd_singleton_state to PRIMARY_CORE as needed.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 6bb4f00a9d006f99b21104bf46b78dff269a116f)

crimson/osd: enable PGShardMapping access from all cores

Previously, all accesses (incuding loookups) had to occur on core 0.  Now that
we want to be able to dispatch from all cores, we need PGShardManager to be
accessible from all cores.  To that end, we now proxy updates to core 0, but maintain
local copies of the map so that cores can perform local lookups.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit a3052969bfcfd9329b2baf7883765642d09ff038)

crimson/osd: make osd sharded

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 5f82fbbc72f840fa8642015a120485648aaf507c)

Merge pull request #53956 from Matan-B/wip-52221-reef

reef: crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53942 from Matan-B/wip-51034-reef

reef: crimson/osd: fix ENOENT on accessing RadosGW user's index of buckets

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53168 from leonid-s-usov/bp/cap-throttle-event/reef

reef: mds/Server: mark a cap acquisition throttle event in the request

Merge pull request #52899 from leonid-s-usov/backport/bulk-data-pool/reef

reef: Consider setting "bulk" autoscale pool flag when automatically creating a data pool for CephFS

Merge pull request #53969 from Matan-B/wip-53432-reef

reef: crimson/vstart: default seastore_device_size will be out of space f…

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53968 from Matan-B/wip-53230-reef

reef: crimson/os/seastore/cbj: fix a potential overflow bug on segment_seq

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53967 from Matan-B/wip-53232-reef

reef: test/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53965 from Matan-B/wip-52204-reef

reef: crimson/osd/object_context: consider clones found as long as they're in SnapSet::clones

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53964 from Matan-B/wip-53082-reef

reef: crimson/admin/admin_socket: remove path file if it exists

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53963 from Matan-B/wip-53031-reef

reef: crimson/osd/pg: check if backfill_state exists when judging objects'

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53962 from Matan-B/wip-53051-reef

reef: crimson/os/seastore/omap_manager: fix the entry leak issue in BtreeOMapManager::omap_list()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53961 from Matan-B/wip-52128-reef

reef: crimson/os/seastore/journal/cbj: generalize scan_valid_records()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53960 from Matan-B/wip-53023-reef

reef: crimson/os/seastore: fix daggling reference of oid in SeaStore::Shard::stat()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53959 from Matan-B/wip-52757-reef

reef: crimson/os/seastore: retire_extent_addr clean up

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53958 from Matan-B/wip-52655-reef

reef: crimson/ertr: assert on invocability of func provided to safe_then()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53955 from Matan-B/wip-52446-reef

reef: crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53951 from Matan-B/wip-51179-reef

reef: crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53949 from Matan-B/wip-51916-reef

reef: crimson/net: support connections in multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53947 from Matan-B/wip-52217-reef

reef: crimson/os/seastore/omap_manager: correct editor settings

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53946 from Matan-B/wip-52090-reef

reef: crimson/os/seastore/btree: should add left's size when merging levels…

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53937 from Matan-B/wip-50598-reef

reef: crimson/osd: add embedded suppression ruleset for LSan

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53953 from Matan-B/wip-52369-reef

reef: crimson/os/seastore: add metrics about total invalidated transactions

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53952 from Matan-B/wip-51770-reef

reef: crimson/os/seastore/rbm: make rbm support multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53950 from Matan-B/wip-51852-reef

reef: crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53948 from Matan-B/wip-52032-reef

reef: crimson/osd: don't ignore start_pg_operation returned future

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53945 from Matan-B/wip-51980-reef

reef: crimson/os/seastore: fix in check_node

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53944 from Matan-B/wip-51961-reef

reef: Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53943 from Matan-B/wip-51945-reef

reef: crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53941 from Matan-B/wip-51948-reef

reef: crimson/os/seastore/cached_extent: add prepare_commit interface

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53940 from Matan-B/wip-51380-reef

reef: crimson/osd/heartbeat: Improve maybe_share_osdmap behavior

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53939 from Matan-B/wip-51750-reef

reef: crimson/os/seastore/backref_manager: scan backref entries by journal seq

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53938 from Matan-B/wip-51475-reef

reef: crimson/os/seastore: fix ceph_assert in segment_manager.h

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53935 from Matan-B/wip-51355-reef

reef: crimson/os/seastore: enable SMR HDD

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53797 from smanjara/ragweed-test-reef

reef: rgw/upgrade: point upgrade suites to ragweed ceph-reef branch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53773 from cbodley/wip-63057-reef

reef: rgw: improve buffer list utilization in the chunkupload scenario

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53770 from cbodley/wip-63054-reef

reef: rgw: fix SignatureDoesNotMatch when extra headers start with 'x-amz'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53767 from cbodley/wip-63051-reef

reef: rgw: s3website doesn't prefetch for web_dir() check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53765 from cbodley/wip-63048-reef

reef: rgw: fix rgw rate limiting RGWRateLimitInfo class decode_json max_rea…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53760 from cbodley/wip-63060-reef

reef: rgw: fix unwatch crash at radosgw startup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53756 from cbodley/wip-63042-reef

reef: [CVE-2023-43040] rgw: Fix bucket validation against POST policies

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53681 from mkogan1/wip-t62141-reef

reef: valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54026 from zdover23/wip-doc-2023-10-16-backport-54022-to-reef

reef: doc/rados: Edit troubleshooting-osd (3 of x)

doc/rados: Edit troubleshooting-osd (3 of x)

Edit doc/rados/troubleshooting/troubleshooting-osd.rst (3 of x).

Follows https://github.com/ceph/ceph/pull/53986.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 01b5aa5d27a97a02cef03b9eb0ee9dbcd55684f1)

Merge pull request #54008 from zdover23/wip-doc-2023-10-14-backport-54006-to-reef

doc/rados: remove FileStore material (in Reef)

doc/rados: remove FileStore material (in Reef)

Remove FileStore-related material because FileStore is not supported in
Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit eab0be5ca513ca2dfaeb31911c175e88bcf4f9e4)

Merge pull request #54007 from zdover23/wip-doc-2023-10-14-backport-54005-to-reef-2

reef: doc/dev: remove cache-pool

doc/dev: remove cache-pool

Remove cache-pool, which is deprecated in Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 073b0919ed9e413a04a7e12b7a0c23b5e058cd52)

Merge pull request #54000 from zdover23/wip-doc-2023-10-13-backport-53986-to-reef

reef: doc/rados: Edit troubleshooting-osd (2 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: Edit troubleshooting-osd (2 of x)

Edit doc/rados/troubleshooting/troubleshooting.rst (2 of x).

Follows https://github.com/ceph/ceph/pull/53936.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 37e7099267996a3075b4902a10a19d94fc738c08)

Merge pull request #53458 from rhcs-dashboard/wip-62602-reef

reef: mgr/dashboard: remove unnecessary failing hosts e2e

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #53982 from zdover23/wip-doc-2023-10-12-backport-53936-to-reef

reef: doc/rados: edit troubleshooting-osd (1 of x)

doc/rados: edit troubleshooting-osd (1 of x)

Edit doc/rados/troubleshooting/troubleshooting-osd.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0fc663d793fe34e42e19b251fcc79b5af52581c6)

crimson/vstart: add --seastore-device-size option in vstart.sh command line

default seastore_device_size will be out of space for smp >28

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit cf69c4f4946884266d6a42c92eef20e74771c3fa)

crimson/os/seastore/cbj: fix a potential overflow bug

CircularBoundedJournal is similar to circular queue, so segment_seq
increases after rolling. However, current implementation always
increases segment_seq_t when rolling occurs, resulting in
the overflow if segment_seq_t hits MAX_SEG_SEQ.

To mitigate this, this commit changes the type of the segment_seq_t
to uint64_t.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 36982b66574ff277e04a02d5b9a28e80d0790778)

crimson/os/seastore/object_data_handler: consider a RBM case when checking if write can be merged

RBM's paddr always indicates physical address, which means it doesn't have the dealayed.
So, this commit adds a condition that checks if given paddr is used for ongoing write.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 73de8937f65706c9c3dd2e36f6504196d0796b01)

test/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 670a19f27761c937ad8d07abf9bb7c94ed4a3b00)

crimson/osd/osd_operations/snaptrim_event: don't process snaptrim events
of the same PG concurrently

Concurrently triming snaps of the same PG may lead to the following
problem:

1. fiber 1 get objects to trim for PG 1.x and snapid 1.d, which involves
   object OBJ;
2. fiber 2 get objects to trim for PG 1.x and snapid 1.c, which also
   involes object OBJ;
3. fiber 1 removes snap 1.c since it has been removed in the osdmap
4. fiber 2 try to get obc for object OBJ and find that OBJ doesn't have
   snap for id 1.c

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 267babe435a54130eda5770a537420fb516e2a8f)

crimson/osd/object_context: consider clones found as long as they're in
SnapSet::clones

Fixes: https://tracker.ceph.com/issues/61825
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 957fd43e4b09a7607d4db52fe9674be905f76de8)

crimson/osd/object_context_loader: rename with_head_and_clone_obc to
with_clone_obc_direct

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit ec646eb22bfd30f6cf19c86fe24c233001272047)

crimson/admin/admin_socket: remove path file if it exists

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 374de2a7622bf3fc75eceb24e39cf36a27ed8d1b)

crimson/osd/pg: check if backfill_state exists when judging objects'
degradation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 04764838b5395a4e4d0c872f2d2a433fc4aec074)

crimson/os/seastore/omap_manager: fix the entry leak issue in
BtreeOMapManager::omap_list()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit cf105932a7fadf67ec494587b078bbfb4caf4d26)