Merge pull request #16356 from liewegas/wip-mgr-early-health

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #16359 from liewegas/wip-cli-stdinout

ceph: allow '-' with -i and -o for stdin/stdout

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #16364 from liupan1111/wip-fix-clean

os/bluestore: use reference to avoid string copy

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16366 from smithfarm/wip-babeltrace-typo

build/ops: rpm: fix typo WTIH_BABELTRACE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16261 from tchaikov/wip-ceph-release-notes-py3

tools: ceph-release-notes: port it to py3

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #16380 from mikulely/doc-index-typos

test,rgw: fix rgw placement rule pool config option

Reviewed-by: Jos Collin <jcollin@redhat.com>

test,rgw: fix rgw placement rule pool config option

actually we're checking '--index-pool' & `--max-buckets` during
the argument parsing.

same as 3af3f4e84ca226f63f7491eb64195d07419a3e1e

Signed-off-by: Jiaying Ren jiaying.ren@umcloud.com

Merge PR 16068 into master

* refs/remotes/upstream/pull/16068/head:
mds: wait auth pinned objects when deactivating mds
mds: fix "wait for stray manager to start"
ceph: kick purge queue when mds active
mds: properly do null snapflush
mds: force client to flush data when waiting on LOCK_SNAP_SYNC lock
mds: don't issue caps for frozen inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16202 into master

* refs/remotes/upstream/pull/16202/head:
doc: add some docs about 'cephfs-data-scan scan_links'
mds/FSMap.cc: remember stopped mds when resetting filesystem
tools/cephfs: handle removed dentries when replaying journal

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15969 into master

* refs/remotes/upstream/pull/15969/head:
mds/events: Initialize Non-static class members in ESession
mds/events: Initialize Non-static class members in fullbit
mds/events: Initialize Non-static class members in EImportStart
mds/events: Initialize Non-static class members in EFragment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16153 into master

* refs/remotes/upstream/pull/16153/head:
mds: fixup may be an unclaimed tick_event

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16041 into master

* refs/remotes/upstream/pull/16041/head:
mds: “*dir” print repeatedly
MDS: tick_event repeated assignment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 16039 into master

* refs/remotes/upstream/pull/16039/head:
common/config_opts: drop unused opt(mds_kill_journal_at)

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15982 into master

* refs/remotes/upstream/pull/15982/head:
mds/MDSMap: use find in is_data_pool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 15981 into master

* refs/remotes/upstream/pull/15981/head:
mds: reinstate daemonperf counters

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16200 into master

* refs/remotes/upstream/pull/16200/head:
qa: thrash max_mds and deactivate ranks

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR 15979 into master

* refs/remotes/upstream/pull/15979/head:
Ignore unmatched rstat errors from MDS during rebuild testing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR 16288 into master

* refs/remotes/upstream/pull/16288/head:
qa/cephfs: don't use int() to convert string of float point number

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16375 from batrick/docker-core-debug

script: add docker core dump debugger

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

script: add docker core dump debugger

This script sets up a docker container with all the packages used for your Ceph
(test) branch. From there, you can gdb the core dump if you have access to the
test cluster.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #16367 from Yan-waller/wip-walle-0715osdmon

mon: Fix output text and doc

Reviewed-by: Jos Collin <jcollin@redhat.com>

mon: cleanups

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #16203 from shashalu/raise-s3policy-debug-level

rgw: raise debug level of RGWPostObj_ObjStore_S3::get_policy

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #16373 from jcsp/wip-dashboard-osds

mgr/dashboard: add OSD list view

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15674 from chardan/jfw-wip-20240-printf-format

libradosstriper: fix format injection vulnerability

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add OSD list view

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15667 from rzarzynski/wip-rgw-fix-nullptrderef-in-rgw-admin

rgw: fix potential null pointer dereference in rgw_admin.

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16251 from linuxbox2/wip-mgr-nfs-svcmap

rgw_file:  add service map registration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16341 from liewegas/wip-20631

osd: restart boot process if waiting for luminous mons

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15807 from zhangsw/feature-lifecycle-date

rgw: S3 lifecycle now supports expiration date

Reviewed-by: Daniel Gryniewicz <dang@fprintf.net>

ceph: allow '-' with -i and -o for stdin/stdout

Signed-off-by: Sage Weil <sage@redhat.com>

build/ops: rpm: fix typo WTIH_BABELTRACE

Introduced by b331898ea9aefc547265b388dddbc388417184fe

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #16339 from majianpeng/test-fio-print-more-perfcounter

test/fio: print all perfcounters rather than objectstore itself.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use reference to void string copy

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #16346 from liewegas/wip-20602

mon: skip crush smoke test when running under valgrind

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16302 from liewegas/wip-mds-dup-alerts

mon/MDSMonitor: fix segv when multiple MDSs raise same alert

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16189 from bassam/pr-msgr-bind-addr

mon: add support public_bind_addr option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16349 from liewegas/wip-vstart-bind

vstart.sh: bind restful, dashboard to ::, not 127.0.0.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16329 from joscollin/wip-cleanup-crush-warning

crush: silence warning from -Woverflow

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16345 from jcsp/wip-watch-channel

ceph.in: filter out audit from ceph -w

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16315 from majianpeng/bluestore-misc-fix

os/bluestore: misc fix and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16351 from liewegas/wip-mgr-init-debug

mgr,mon: debug init and mgrdigest subscriptions

Merge pull request #16358 from liupan1111/wip-fix-client

test/msgr: fixed the hang issue for perf_msg_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

test/msgr: fixed the hang issue for perf_msg_client

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

qa/suites/rados/rest/mgr-restful: simplify

Use default port; don't bother setting bind addr.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: only induce mgr epoch shortly after mkfs

For early clusters, if there isn't an active manager, we eventually want
to trigger a health warning by rolling over the mgrmap epoch.  We don't
want to do that if we have no active/available manager after that.  Fix
by checking ever_had_active_mgr here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15587 from wjwithagen/wip-wjw-ceph-disk-is_diskdevice

ceph-disk/ceph_disk/main.py: Replace ST_ISBLK() test by is_diskdevice()

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16347 from tchaikov/wip-test-ceph-disk

tests: ceph-disk: use communicate() instead of wait() for output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16334 from wjwithagen/wjw-bug-stringyfy

core:" Stringify needs access to << before reference" src/include/stringify.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

qa/suites/ceph-disk: whitelist health warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: ceph-disk: use communicate() instead of wait() for output

to avoid possible deadlock. quote from doc of Popen.wait()

> This will deadlock when using stdout=PIPE and/or stderr=PIPE and the
child process generates enough output to a pipe such that it blocks
waiting for the OS pipe buffer to accept more data. Use communicate() to
avoid that.

and print out the stdout and stderr using LOG.warn() if the command
fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16045 from Liuchang0812/wip-compact-osd-feature

osd: compact osd feature

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/fio: print all perfcounters rather than objectstore itself.

Need bluefs,rocksdb perfcounters.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #16221 from liewegas/wip-20546

crush/CrushWrapper: make get_immediate_parent[_id] ignore per-class shadow hierarchy

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/Mgr: debug init a bit more

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrMonitor: debug mgrdigest subscriptions a bit

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: bind restful, dashboard to ::, not 127.0.0.1

Signed-off-by: Sage Weil <sage@redhat.com>

docs: add docs for public_bind_addr

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

test,qa/workunits: add tests for public_bind_addr

Add a set of new tests for the case when public_addr and public_bind_addr
are different for a mon. In order to test this properly I had to employ
port forwarding with socat. This helps simulate what would happen in a
environment like Kubernetes. socat is now a build dependency.

Also, moved jq_success to ceph-helpers.sh and refactored run_mon to enable
creating the mons without creating the rbd pool immediately.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

mon: add support public_bind_addr option

To support running in dynamic enviornments (like Kubernetes) the mon needs
to be able to advertise and ip address that is different from the ip address
that it listens on locally.

Added a new config option "public_bind_addr" which if set becomes the address
that the mon will bind to locally. If empty (the default) the public_addr
will be used to bind locally.

added a new function on Messenger to set_addr which is called by ceph-mon to set
the advertised address after doing the bind.

also relaxed the "wrong node!" errors in AsyncMessenger and SimpleMessenger as
its now valid to talk to a peer whose peer_addr_of_me is different from what
we expect.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12139 from tianshan/fix_zone_set_realm_id

rgw: fix zone did't update realm_id when added to zonegroup

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14915 from cbodley/wip-19817

rgw: add missing RGWPeriod::reflect() based on new atomic update_latest_epoch()

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

qa/suites: disable mon crush smoke test with valgrind

Valgrind runs itself on forked children, and does its cleanup when they
complete, and this is slow... slow enough that it frequently makes the
test time out.

Valgrind let's you ignore child *processes* that you exec, but I can't
find a way to skip forked children in the same address space.

Work around this by skip this validation when running under valgrind.

Fixes: http://tracker.ceph.com/issues/20602
Signed-off-by: Sage Weil <sage@redhat.com>

mon: add mon_osd_crush_smoke_test=true

This option allows us to disable the crush smoke test when creating pools,
injecting crush maps, or making other changes.  DANGER DANGER.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15953 from rzarzynski/wip-rgw-20418

rgw: reject request if decoded URI contains \0 in the middle.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph.in: filter out audit from ceph -w

...and add an optional --watch-channel argument.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #16340 from dillaman/wip-20630

qa/tasks: rbd-mirror daemon not properly run in foreground mode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16331 from zhangsw/cleanup-zonegroup-list

rgw: remove the useless output when listing zonegroups.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: restart boot process if waiting for luminous mons

If we start_boot and see that we don't have luminous mons, we will stop.
But we don't currently reliably notice when the luminous upgrade completes.
If we happen to be connected to the last mon we will start_boot() because
of the trigger in ms_handle_connect(), but if we are not connected to the
last mon we'll eventually get a monmap update but not restart booting.

Fix by setting a flag if we are waiting, and restart boot if the flag is
set, we are in preboot, and we see we now have luminous mons.

Fixes: http://tracker.ceph.com/issues/20631
Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks: rbd-mirror daemon not properly run in foreground mode

Fixes: http://tracker.ceph.com/issues/20630
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16043 from jcsp/wip-dashboard-updates

mgr: dashboard improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: update health display

This takes account of the new health format, also
expands and visually cleans up the frontpage
where we put the health information.

Dark backgrounds make it much easier to use
red/amber/green colours to grab attention.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: expose a MgrMap in PyModules

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #16020 from jcsp/wip-20383

mgr: clean up daemon start process

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16338 from scienceluo/wip-doc-branch

doc/release-notes: Luminous release notes typo fixes

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/release-notes: Luminous release notes typo fixes "systemctl ceph-osd.target"->"systemctl restart ceph-osd.target" and "systemctl ceph-mgr.target"->"systemctl restart ceph-mgr.target"

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

Merge pull request #16318 from smithfarm/wip-jewel-10-2-9

doc: Jewel v10.2.9 release notes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph_disk/main.py: fixed bugs from theutology report

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

ceph-disk/main.py: Replace ST_ISBLK() test by is_diskdevice()

 - FreeBSD does not have blockdevices any more (since 2002)
   So disk are just Character special devices, so test on ISCHR

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

core:" Stringify needs access to << before reference" src/include/stringify.h

Clang complains:

In file included from /home/jenkins/workspace/ceph-master/src/mon/HealthMonitor.cc:21:
/home/jenkins/workspace/ceph-master/src/include/stringify.h:15:6: error: call to function 'operator<<' that is neither visible in the template definition nor found by argument-dependent lookup
  ss << a;
     ^
/home/jenkins/workspace/ceph-master/src/mon/HealthMonitor.cc:129:32: note: in instantiation of function template specialization 'stringify<std::__1::set<std::__1::basic_string<char>, std::__1::less<std::__1::basic_string<char> >, std::__1::allocator<std::__1::basic_string<char> > > >' requested here
      boost::regex("%names%"), stringify(names[p.first]));
                               ^
/home/jenkins/workspace/ceph-master/src/include/types.h:160:17: note: 'operator<<' should be declared prior to the call site
inline ostream& operator<<(ostream& out, const set<A, Comp, Alloc>& iset) {

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16330 from scienceluo/master

doc/release-notes: Luminous release notes typo fixes  "ceph config-key ls"->"ceph config-key list"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc:fix release-notes

Signed-off-by: luo.kexue@zte.com.cn

crush: silence warning from -Woverflow

The following warning appears during build:
ceph/src/crush/CrushWrapper.cc: In member function ‘int32_t CrushWrapper::_alloc_class_id() const’:
ceph/src/crush/CrushWrapper.cc:1322:56: warning: integer overflow in expression [-Woverflow]
   uint32_t upperlimit = numeric_limits<int32_t>::max() + 1;
                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16300 from liewegas/wip-20600

mon: fix hang on deprecated/removed 'pg set_*full_ratio' commands

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #16321 from Yan-waller/wip-walle-0712cephosd

common: misc cleanups in common, global, os, osd submodules

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #16322 from liewegas/wip-20617

qa/tasks/ceph_manager: wait longer for pg stats to flush

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16323 from ceph/revert-15897-wip-20390

Revert "msg/async: increase worker reference with local listen table enabled backend"

Merge pull request #16319 from tchaikov/wip-ceph-helper-with-exp-features

qa/workunits/ceph-helpers: enable experimental features for osd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #16320 from tchaikov/wip-clang-analyzer-warnings

test,mon,msg: kill clang analyzer warnings

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

rgw: remove the useless output when listing zonegroups.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

doc: Jewel v10.2.9 changelog

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon/PGMonitor: EOPNOTSUPP for old pgmon commands

This includes 'pg set_full_ratio', which we have only for the upgrade, but
goes away afterwards.

Also, return true to either swallow the request or indicate it has been
processed.

Fixes: http://tracker.ceph.com/issues/20600
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: remove useless judgement.

In fact logger is never null.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mon: do not assign to never-read variable

this silences clang analyzer's warning of

Value stored to 'err' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: return stored errno on error

otherwise the errno would be overwritten, and we are returning 0 or the
errno set by ::close()

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: test_c_headers: silence clang analyzer warnings

this silences clang analyzer's warnings like:

Value stored to 'ret' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16262 from liewegas/wip-20208

mgr/ClusterState: do not mangle PGMap outside of Incremental

Merge pull request #9974 from weiqiaomiao/wqm-wip-copy_obj

rgw: fix memory leak in copy_obj_to_remote_dest

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore/BlueStore: no need to recalc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueStore: using buffprt::zero rather than memset.

Just for the code to look consistent

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>