Merge pull request #27287 from s0nea/wip-39052-nautilus

nautilus: mgr/dashboard add polish language

Reviewed-by: Sebastian Krah <skrah@suse.com>

Merge pull request #27382 from tchaikov/nautilus-multi-python3

nautilus: rpm,cmake: use specified python3 version if any

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

rpm: pass %{python3_version} to cmake

to force cmake to use the python3 and python3 modules for building
python3 bindings

on the debian side, it's okay to continue using "-DWITH_PYTHON3=ON", as

- cmake does normalize "ON" to 3
- debian's cmake extension lives on /usr/lib/python3/dist-packages/
  not in a specific /usr/lib/python3.x/dist-packages directory

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7681a38ce51743a19ec5679d49b362f04e271ddb)

cmake: check for libpython of the same version of interpreter

actually cython or python3-cython's dependencies should take care of it.
but we should at least get it right on our side if we check it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c961e002d22ab7ceec20f1019f8c558ec9fcce6c)

cmake: use specified python3 version if any

use might have multiple python3 installed, some of them has/have all
dependencies installed and is good enough for building Ceph. we should
not always use the latest python installed in the system and complain that
there is missing dependencies, even if user has installed all the
python3 dependencies for the older python3.

put in other words, if user only installs cython module for python3.4, but
she has both python3.6 and python3.4 in her system. we should not force
her to uninstall python3.6 for installing Ceph.

this change also aligns with MGR_PYTHON_VERSION. i am not applying the
same change to WITH_PYTHON2, because python2 is already stablized. and distros
are not likely to release new python2 releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d8c13e86048b7aac458f3eda101f9778e3d45dcf)

Merge pull request #27191 from theanalyst/nautilus-scheduler-cmake

nautilus: rgw: build async scheduler only when beast is built

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27296 from theanalyst/nautilus-sse-c-fixes

nautilus: sse c fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27314 into nautilus

* refs/pull/27314/head:
mon: quiet devname log noise

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27320 into nautilus

* refs/pull/27320/head:
valgrind.supp: add clauses for uninitialised gcm.Xi in OpenSSL.
src/valgrind.supp: replace with the teuthology's whitelist.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

valgrind.supp: add clauses for uninitialised gcm.Xi in OpenSSL.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 35274eba7a90ffed05561737cb935bb3dc049c2d)

src/valgrind.supp: replace with the teuthology's whitelist.

This change drops src/valgrind.supp file and transplates
the current teuthology's whitelist into qa/valgrind.supp.

This commit has sibiling in the teuthology repo dropping
the file there.

The rationale behind the entire change is duplication of these
files between two repos, their inconsistencies and confusion.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit d10ab9ebfdb4aa5ae285f1e2b89fd490584b0cb3)

Merge pull request #27261 from s0nea/wip-38969-nautilus

nautilus: mgr: Configure Py root logger for Mgr modules

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27267 from s0nea/wip-38988-nautilus

nautilus: mgr/dashboard: Add refresh interval to the dashboard landing page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mon: quiet devname log noise

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3221381d9fb924e397c3d031c7d4751490409bca)

Merge PR #27167 into nautilus

* refs/pull/27167/head:
mgr/diskprediction_cloud: Correct base64 encode translate table

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27216 into nautilus

* refs/pull/27216/head:
msg/async v2: make v2 work on rdma.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #27220 into nautilus

* refs/pull/27220/head:
osd/PG: move '}' to the proper place
doc: Document new pg state and changes to auto repair behavior
osd, test: Add num_shards_repaired to osd_stat_t for pushes with repair set 3(3)
osd: Track num_objects_repaired in pg stats 2(3)
test, osd: Improvements to auto_repair 1(3)
test: osd-scrub-repair.sh: use corrupt_and_repair_lrc for lrc tests
osd: Publish stats after all changes made
osd: Fixes for 64-bit PG state

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27240 into nautilus

* refs/pull/27240/head:
mgr/diskprediction_cloud: Correct base64 encode translate table

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27278 into nautilus

* refs/pull/27278/head:
log,global: do not start flusher thread until after we have our mon config
log: buffer log entries until flusher thread starts
log: open log file from flusher thread
common/ceph_context: fix log_to_file observer
common: add bool log_to_file option

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27237 from s0nea/issue_38265_backport

nautilus: mgr/dashboard/qa: Improve tasks.mgr.test_dashboard.TestDashboard.test_standby

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27246 from s0nea/wip-38974-nautilus

nautilus: ERROR: test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #27260 from pdvian/wip-38866-nautilus

nautilus: mgr/dashboard: fix sparkline component

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

rgw: fixup_range() clamps offsets to valid part range

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4b3bac20d550cae274c07a1d555751483dde463b)

test_rgw_crypto: add invalid ranges beyond obj boundary

This is just to assert we're not doing an invalid memory access

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b34a00de0f6ecea6d97fd80f65d2a2dca0876e6e)

tests: rgw crypto tests with partlen vector

4 tests are added when either obj_size for an aligned object, and when an object
is not aligned to a boundary (1 B last part), or part_size is not aligned to a
boundary. Comments on boundary case explains the logic on choosing whether to
align to a 4095B boundary or not

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 1bf59f68db608084a1e7a05aca4cca8a8ab76062)

test_rgw_crypt: BlockCryptNone takes a block_size param

So that block sizes like 4096 can be tested

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9e13a42daaba26751f56f2454f9ec314d7d0d460)

rgw_crypt: make decrypt's parts_len protected

This is so that testing classes can set this and use that to test the branches
in fixup_range tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 78b37a5ccd67c464f3c6e61568e9e827c327ad6d)

rgw_crypt: take care of start & end boundaries when parsing range

As http range header is inclusive of start and end ranges, this otherwise breaks
at ranges close to multipart boundaries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9c92b56d77932283ccc0336645dc3b76bd0fffa2)

rgw: decrypt filter does not cross multipart boundaries

multipart uploads with sse encrypts each part separately, using an
initialization vector based on the part offset

decryption must respect the same part boundaries, and start each part
with a fresh initialization vector. this means that the decrypt filter
must flush data up to part boundaries before starting the next

Fixes: http://tracker.ceph.com/issues/38700
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b782902954abd20e49c0ed1c14cebee4c8ab6f88)

Merge PR #27253 into nautilus

* refs/pull/27253/head:
qa/tests added mimic-x-singleton suite to rados as symlink

Reviewed-by: Sage Weil <sage@redhat.com>

osd/PG: move '}' to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 0fb951963ff9d03a592bad0d4442049603195e25)

doc: Document new pg state and changes to auto repair behavior

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 769cdc8294c74b33fe99dd2cdb071b0ebcc9ae69)

osd, test: Add num_shards_repaired to osd_stat_t for pushes with repair set 3(3)

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 57abdb11fae9f94417e1ee6c9427a33f729a9703)

osd: Track num_objects_repaired in pg stats 2(3)

Leave repair pg state on until recovery finishes or a new scrub starts

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d2ca3d2feb442f97ca89023c7d01178d96f517a6)

test, osd: Improvements to auto_repair 1(3)

Allow auto_repair for replicated bluestore pools
Regular scrub within auto repair parameters will trigger deep scrub
New state failed_repair if PG repair attempt could not fix everything
Set failed_repair if not possible to repair anything

Fixes: http://tracker.ceph.com/issues/38616
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 2202e5d0b107795837ce79ffce2a980e8c12fc62)

test: osd-scrub-repair.sh: use corrupt_and_repair_lrc for lrc tests

Fix for argument handling of create_ec_pool()
Always pass a value for allow_overwrites for consistency

Caused by: 3ca750d41dfe33c6efea4abc96d2bd426a9742b9

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 315d32488963705f28cd51a327281b8940d1e7cd)

osd: Publish stats after all changes made

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 06e6ebc7836d0bc65361a7857ba8cf4a31040bc2)

osd: Fixes for 64-bit PG state

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 12bff373d29f4cec8932aa1460da98dca711ec91)

mgr/dashboard: Add polish translation

Add polish translation.
Removed 5 unused trans-units.

Fixes: https://tracker.ceph.com/issues/38236
Signed-off-by: Sebastian Krah <skrah@suse.com>
(cherry picked from commit 53d84ad5d36f99a3d8c49ec1f8b60aa0ed4d248c)

Merge PR #27225 into nautilus

* refs/pull/27225/head:
osd/OSDMap: add log for better debugging
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only
crush: add root_bucket to identify underfull buckets

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27217 into nautilus

* refs/pull/27217/head:
common/config: parse --default-$option as a default value

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #27215 into nautilus

* refs/pull/27215/head:
mon/MonClient: do not dereference auth_supported.end()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27197 into nautilus

* refs/pull/27197/head:
msg: default to debug_ms=0

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #27139 into nautilus

* refs/pull/27139/head:
os/bluestore: unconditionally cap chunks returned by allocator to 2^31
os/bluestore: start using 64-bit intervals for bitmap allocator
os/bluestore: make bluestore interval base template.
tests/fastbmap_alloc: UT to reproduce 4G allocation bug
os/bluestore: os/bluestore: implement dump for bitmap allocator
os/bluestore be more tolerant to lack of space for bluefs.

Reviewed-by: Sage Weil <sage@redhat.com>

log,global: do not start flusher thread until after we have our mon config

We want to respect the settings in the mon config that affect logging before
we enable the flusher thread.  That allows us to set (via the monitor) things
like log_to_file=false or log_to_syslog=true.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cd6a5b9c40779956629803f222c365bdb291a169)

log: buffer log entries until flusher thread starts

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4af24e57aafd8774da4b1d2c7e3b17dcae1c18a0)

log: open log file from flusher thread

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 16c5d828fa93b1a5bdf9c396d7c0ce236b46df9b)

common/ceph_context: fix log_to_file observer

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d8810ab58241b80776e2564bce911299341f73e7)

common: add bool log_to_file option

This allows us to disable and reenable logging to a file while preservng
the default log_file location.  This is analogous to log_to_stderr,
log_to_syslog, log_to_graylog, etc.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 18df18aa4d69232f1e898c99522aad7444c9ff12)

qa/tests added mimic-x-singleton suite to rados as symlink

replaces https://github.com/ceph/ceph/pull/27131

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboar: Add refresh  nterval to the dashboard landing page

Fixes: http://tracker.ceph.com/issues/26872
Signed-off-by: guodan1 <guodan1@lenovo.com>
(cherry picked from commit 58e9d48259ad7ac562ffb85bc22b3d6fba6120e1)

mgr: Configure Py root logger for Mgr modules

Add the CPlusPlusHandler to the Python root logger, thus the Mgr module itself and all 3rd party libraries will log to the Ceph log. This may help to identify problems in used 3rd party libraries.

Fixes: https://tracker.ceph.com/issues/38888
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit b0be9475f1d9be9538183e636569ec851801dccb)

mgr/dashboard: fix sparkline component

Fixes: https://tracker.ceph.com/issues/38768
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 36997f93a9fe1b56646bb1f6574ac204467a74ff)

Merge pull request #27161 from cbodley/wip-qa-rgw-nautilus

nautilus: qa/rgw: point test repos at ceph-nautilus branch

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

mgr/dashboard/qa: Increase timeout for test_disable (tasks.mgr.dashboard.test_mgr_module.MgrModuleTelemetryTest)

Fixes: https://tracker.ceph.com/issues/38820
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 91fe2f07997d292e42005001e07dbf1dfb94d0c8)

mgr/diskprediction_cloud: Correct base64 encode translate table

Correct base64 encode/decode string translate table

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
(cherry picked from commit 9df1f3cfaaa3fb1de35ed729cc7548da19392255)

Merge pull request #27189 from smithfarm/wip-38903-nautilus

nautilus: doc: Minor rados related documentation fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27219 from LenzGr/wip-38871-nautilus

nautilus: mgr/dashboard: dashboard giving 401 unauthorized

mgr/dashboard/qa: Improve tasks.mgr.test_dashboard.TestDashboard.test_standby

The changes of this PR were done while trying to fix the failing test. The problem has been solved by another PR, but the changes are worth to be integrated because they help debugging and an additional test has been added (check if previously active manager is listed as standby).

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit c29e587007e461738d4f95938dc2eb44f2459964)

osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a89281ffbb50a4dfc700398e864138b5faaf00f5)

osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 01e8e9482ce7194d347e02ef41acfa6d8d14f614)

crush: add root_bucket to identify underfull buckets

All underfull buckets under root_buckets will be taken as target

For the crule rule:
    step take datacenter0
    step chooseleaf firstn 2 type host
    step emit
    step take datacenter1
    step chooseleaf firstn 2 type host
    step emit

If one host contains overfull osd but no underfull osd,
it will use other underfull buckets as target, which
maybe not in the same datacenter, that will
broke the rule.

Fixes: http://tracker.ceph.com/issues/38826
Signed-off-by: huangjun <huangjun@xsky.com>
(cherry picked from commit 3d5678d3561d90a10d9de3cb6e7e0405dbe8fdfe)

common/config: parse --default-$option as a default value

Sometimes it is useful to specify an alternative default value for an
option via the command line such that it has a lower priority than the
mon config database, config file, the rest of the command line, or the
environment.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fdd2000631b25fd964e94373488e7f94694d7020)

msg/async v2: make v2 work on rdma.

When exec " MON=1 OSD=1 RGW=0 MDS=0 MGR=0 ../src/vstart.sh -b -n -o "ms_type=async+rdma"  -o "ms_async_rdma_device_name=mlx5_1"  --msgr2 -o "debug ms=2/2" ", met the following error messages:

2019-03-18 10:04:17.074 7f6644dd8700  1 --2-  >> v2:10.239.48.9:40998/0 conn(0x7f66401691f0 0x7f6640169630 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_client_ident getsockname reveals I am : when talking to v2:10.239.48.9:40998/0
/home/ceph/jp-ceph/src/msg/msg_types.h: In function 'void entity_addr_t::set_port(int)' thread 7f6644dd8700 time 2019-03-18 10:04:17.079278
/home/ceph/jp-ceph/src/msg/msg_types.h: 363: abort()
 ceph version 14.1.1-196-g9c024bafa3 (9c024bafa3570dee401d3fcaf04754376f285eca) nautilus (rc)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xfe) [0x7f664cc6e896]
 2: (entity_addr_t::set_port(int)+0xaa) [0x7f664cc3a724]
 3: (ProtocolV2::send_client_ident()+0x3ba) [0x7f664d005270]
 4: (ProtocolV2::finish_client_auth()+0x4c) [0x7f664d004d8e]
 5: (ProtocolV2::handle_auth_signature(ceph::buffer::v14_2_0::list&)+0x612) [0x7f664d00ad70]
 6: (ProtocolV2::handle_frame_payload()+0x275) [0x7f664cffde79]
 7: (ProtocolV2::handle_read_frame_dispatch()+0x158) [0x7f664cffcbea]
 8: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr<ceph::buffer::v14_2_0::ptr_node, ceph::buffer::v14_2_0::ptr_node::disposer>&&, int)+0x80d) [0x7f664cffef77]
 9: (CtRxNode<ProtocolV2>::call(ProtocolV2*) const+0x8c) [0x7f664d02ac40]
 10: (ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x5e) [0x7f664cff144e]
 11: (()+0x175a34b) [0x7f664cff834b]
 12: (()+0x1774e00) [0x7f664d012e00]
 13: (std::function<void (char*, long)>::operator()(char*, long) const+0x61) [0x7f664cfb6269]
 14: (AsyncConnection::process()+0xc77) [0x7f664cfb16f7]
 15: (C_handle_read::do_request(unsigned long)+0x28) [0x7f664cfb5472]
 16: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x678) [0x7f664d02e4cc]
 17: (()+0x179dbf4) [0x7f664d03bbf4]
 18: (()+0x179f0c7) [0x7f664d03d0c7]
 19: (std::function<void ()>::operator()() const+0x32) [0x7f664d03a44a]
 20: (void std::__invoke_impl<void, std::function<void ()>>(std::__invoke_other, std::function<void ()>&&)+0x20) [0x7f664d039ce7]
 21: (std::__invoke_result<std::function<void ()>>::type std::__invoke<std::function<void ()>>(std::function<void ()>&&)+0x26) [0x7f664d039474]
 22: (decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::function<void ()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>)+0x28) [0x7f664d03b3aa]
 23: (std::thread::_Invoker<std::tuple<std::function<void ()> > >::operator()()+0x1d) [0x7f664d03b37b]
 24: (std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void ()> > > >::_M_run()+0x1c) [0x7f664d03b35a]
 25: (()+0xbd57f) [0x7f664af5257f]
 26: (()+0x76db) [0x7f6657d996db]
 27: (clone()+0x3f) [0x7f66580d288f]

This because getsockname can't work on connection->cs.fd(). So add new
api socket_fd() for rdma socket fd.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit de39c8534be644f760d7e7ccee7b782349237c1d)

mon/MonClient: do not dereference auth_supported.end()

if we are unable to find a supported method in allowed_methods, we will
dereference `auth_supported.end()` for searching it in desperation.

Reported-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 998b1925bd6d19b2b32141b92de66b77027130a0)

Merge pull request #27171 from yuriw/wip-yuriw-38819-nautilus_2

qa/tests: initial checkin for nautilus-p2p

qa/tests: initial checkin for nautilus-p2p
symlinked `objectstore` folder

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

msg: default to debug_ms=0

These have a measurable impact on performance and have a limited (although
not non-existent!) track record of providing debugging value.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3ebd59b872e3a77430b93bb51f9232868975b3bb)

mgr/dashboard: auth ttl expired error

Fixes: https://tracker.ceph.com/issues/38428
Signed-off-by: ming416 <geenature@163.com>
(cherry picked from commit 780bdf541ad7230347f2197651c616c217b1ed95)

librgw: expose dmclock linkage

As files that include librgw will fail to build with include errors otherwise

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit dc06e2186a9f258b8ce490489f7412556e29594a)

rgw: test_dmclock_scheduler: conditionalize tests with boost ctx

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b6e503d79cc625c1565ca5c5ae60881d16fefc72)

rgw: build async scheduler only when beast is built

As it depends on yield and other asio constructs don't build if beast frontend
isn't built

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit a51f51215f6fcf99ab0ba620fcdc92fae1d02389)

doc: Fix the pg states and auto repair config options

Fixes: http://tracker.ceph.com/issues/38896
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9fd4b062f1a22de61c37457a32d0d7cefc0e8fcf)

Merge PR #27168 into nautilus

* refs/pull/27168/head:
mon/Monitor.cc: print min_mon_release correctly

Reviewed-by: Sage Weil <sage@redhat.com>

mon/Monitor.cc: print min_mon_release correctly

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 21ca604a22eacbf5e1dbeffffa0dd38f84647505)

mgr/diskprediction_cloud: Correct base64 encode translate table

Correct base64 encode/decode string translate table

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
(cherry picked from commit 9df1f3cfaaa3fb1de35ed729cc7548da19392255)

Merge PR #27158 into nautilus

* refs/pull/27158/head:
common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/rgw: point test repos at ceph-nautilus branch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/blkdev: get_device_id: behave if model is lvm and id_model_enc isn't there

If that happens, leave ID_MODEL blank and we'll fall back to something
else, like ID_SERIAL (which is present in each case I've seen this).

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ec4434a65f2248fb823ccfb5d2e150bb4da71da1)

Merge pull request #27116 from liewegas/wip-38839-nautilus

nautilus: mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26972 into nautilus

* refs/pull/26972/head:
rgw: ldap: fix LDAPAuthEngine::init() when uri !empty()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27060 into nautilus

* refs/pull/27060/head:
mon/OSDMonitor: allow 'osd pool set pgp_num_actual'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27095 into nautilus

* refs/pull/27095/head:
mgr/orchestrator: Add error handling to interface

Merge PR #27096 into nautilus

* refs/pull/27096/head:
common/blkdev: handle devices with ID_MODEL as "LVM PV ..." but valid ID_MODEL_ENC

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27117 into nautilus

* refs/pull/27117/head:
osd/OSDMap: add 'zone' to default crush map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #27119 into nautilus

* refs/pull/27119/head:
crush/CrushWrapper: make update_choose_args less chatty
qa/standalone/crush/crush-choose-args: add weight-set tests
qa/standalone/crush/crush-choose-args: fix test
crush/CrushWrapper: move_item: do not clobber weight-set weights
crush/CrushWrapper: create_or_move: make weight-set update optional
mon/OSDMonitor: apply osd_crush_update_weight_set for reweight, create-or-move
crush/CrushWrapper: insert_item: make weight-set update optional (for leaves only)
crush/CrushWrapper: use adjust_item_weight_in_bucket for subtree reweight
crush/CrushWrapper: fix detach_bucket, remove_item[_under] vs weight-sets
crush/CrushWrapper: add update_weight_sets arg to adjust_item_weight_*
crush/CrushWrapper: refactor adjust_weight_* into per-bucket helper
crush/CrushWrapper: pass cct down into more places

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: unconditionally cap chunks returned by allocator to 2^31

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 1337443e37ffbfcaf494bdb2bb80db4cf41b97f9)

os/bluestore: start using 64-bit intervals for bitmap allocator

This helps to avoid 4Gb+ values wrapping which resulted in
allocation failures.

Fixes: https://tracker.ceph.com/issues/38761
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit d1466963a60ed2345a6fc159071ec09ce422b656)

os/bluestore: make bluestore interval base template.

We might need intervals at BlueStore with both 32-bit and 64-bit
lengths. Having base template eliminates the need for copy/paste

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f775d4227fb9432840c795115669766565c5e718)

tests/fastbmap_alloc: UT to reproduce 4G allocation bug

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ed8a93ba387458961444913a8695f538595efacf)

os/bluestore: os/bluestore: implement dump for bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 80310d98a7456afdb2002eaf2891853ea21eaf68)

os/bluestore be more tolerant to lack of space for bluefs.

'gift' space is just advisory for allocation, part of it actually requested
from BlueFS is mandatory only. Hence do not fail when unable to allocate
the whole space.

Fixes: https://tracker.ceph.com/issues/38760
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dbc1a78787baacd7bbc98ff8bbb72e609def2ad6)

crush/CrushWrapper: make update_choose_args less chatty

These were now exposed due to cct being passed down to more
places.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ef8b347fd0b74ca2da6d7c5cc70fa584fd3d36dc)

qa/standalone/crush/crush-choose-args: add weight-set tests

Verify we have the expected behavior for creates and moves that
maintain bucket summation, both with and without the
osd_crush_update_weight_set option enabled.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4c741c109df2eb8c35b7b895ba88b1d8e882a4e2)

qa/standalone/crush/crush-choose-args: fix test

- Make the initial weight-set actually consistent (summing)
- Fix the intermediate state so that it reflects a correctly
  maintained summation.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f20c736e990f967804768a3e75a05c077a591595)

crush/CrushWrapper: move_item: do not clobber weight-set weights

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c4eefb0b66be57bddbc888ed2b5d2e2a19452b37)

crush/CrushWrapper: create_or_move: make weight-set update optional

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 07bb2020376294067d3df1423c2a3e835870f9f3)

mon/OSDMonitor: apply osd_crush_update_weight_set for reweight, create-or-move

Since CrushWrapper no longer applies this setting at a low level,
where it can't tell what the real intention is, we instead apply
it at the top command level where we do.

Specifically, we use it to control whether the weight-set weights
are set for the commands

 osd crush reweight
 osd crush reweight-subtree
 osd crush move
 osd crush create-or-move

Note that this (indirectly) affects the way weight-set weights
are initialized for newly created OSDs, since those are added to
the crush map via the 'osd crush create-or-move' command.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 50e4b818eac7d67e3307fdde90420334ecad512e)

crush/CrushWrapper: insert_item: make weight-set update optional (for leaves only)

If it is a bucket, we should sum the weight-set values to weight
the bucket in the subtrees.  It only makes sense to reset the
weight-set weights for leaf items.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5579f388af18c55d03cbf32c244488ee38169fc2)

crush/CrushWrapper: use adjust_item_weight_in_bucket for subtree reweight

This avoids clobbering weight-sets when you reweight, if the
adjust_weight_sets=false.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 46b72611f3e59ec24d944ba907ece4659180c0c5)