rgw: req xml params size limitation error msg

Now rgw will return like following:

400
<?xml version="1.0" encoding="UTF-8"?><Error><Code>MalformedXML</Code><Message>The XML you provided was larger than the maximum 2048 bytes allowed.</Message><BucketName>333</BucketName><RequestId>tx000000000000000000009-00596a1331-101a-default</RequestId><HostId>101a-default-default</HostId></Error>

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #15305 from wjwithagen/wip-wjw-HAVE_BABELTRACE

cmake: Rewrite HAVE_BABELTRACE  option to WITH_

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16094 from zphj1987/master

pybind/mgr/dashboard: fix get kernel_version error

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-mgr:fix ceph-mgr web get kernel_version error

fix :
File "/usr/lib64/ceph/mgr/dashboard/module.py", line 523, in _clients
client['version'] = client['kernel_version']
----------

change client['kernel_version'] to
client['client_metadata']['kernel_version']

Signed-off-by: Peng Zhang zphj1987@gmail.com

Merge pull request #16093 from tchaikov/wip-20432

osd: clear_queued_recovery() in on_shutdown()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15823 from jdurgin/wip-omap-tp-heartbeat

osd/ReplicatedBackend: reset thread heartbeat after every omap entry …

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: clear_queued_recovery() in on_shutdown()

otherwise when we are kicking the PG waiting in OSD's awaiting_throttle
queue, the queue is still holding a strong reference of it. then we will
have warnings like "pgid 0.7 has ref count of 2" when shutting down OSD,
because pgmap is also holding a reference of it.

we could clear the "awaiting_throttle" directly in OSD::shutdown(), but
clear_queued_recovery() is more consistent this way, because it is PG
which registered itself to the queue in OSD, it'd be better if PG can
remove itself from the OSD when it is shutting down itself.

Fixes: http://tracker.ceph.com/issues/20432
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16081 from tsg-/tsg-affiliation

mailmap, organizationmap: add affiliation for Tushar Gohad

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge remote-tracking branch 'upstream/pull/16080/head' into HEAD

* upstream/pull/16080/head:
  doc/mon: clean up `mds fail`/`mds deactivate` descriptions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15833 from Linbing/wip-osd-delete-MPing

osd: osd does not using MPing Messages,do not include unused include

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/mon: clean up `mds fail`/`mds deactivate` descriptions

Fixes: http://tracker.ceph.com/issues/20424
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #16072 from tchaikov/wip-20453

debian: workaround the bug in dpkg-maintscript-helper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15835 from SUSE/wip-flatten-systemd-target-hierarchy-master

systemd: Add explicit Before=ceph.target

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #16067 from YangFanlinux/affi

 .mailmap, .organizationmap: Update Fan Yang information and affiliation

Reviewed By: Jos Collin <jcollin@redhat.com>

Merge pull request #16055 from dmick/master

ceph.in: allow developer mode from outside build tree

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: workaround the bug in dpkg-maintscript-helper

see https://bugs.launchpad.net/ubuntu/+source/snap-confine/+bug/1605052
and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=582819

Fixes: http://tracker.ceph.com/issues/20453
Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: Fan Yang information and affiliation

Signed-off-by: Fan Yang <yangf@neunn.com>

mailmap: add affiliation for Tushar Gohad

Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>

CMakeLists.txt: Rewrite 2 remaning options to WITH_

All options to en/disable inclusion of libraries or other software
are of the format WITH_ so that the Cmake commaind like ahs all
WITH_* options. The WITH_=ON option will result in a HAVE_ setting
in CMAKE so that tests can use that variable.

Last "abusers" to actually follow this format.

 - HAVE_BABELTRACE

 - HAVE_ZFSLIB is fixed in #15907

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16056 from xiexingguo/wip-kill-log-warn

common: fix log warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15986 from liewegas/wip-20435

mon/MgrStatMonitor: avoid dup health warnings during luminous upgrade

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #14178 from songbaisen/m10

mon: show the leader info on mon stat command

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #15585 from dachary/wip-20229-ceph-disk-timeout

ceph-disk: set the default systemd unit timeout to 3h

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15984 from idryomov/wip-chooseargs-encoding-fix

crush/CrushWrapper: chooseargs encoding fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16023 from liewegas/wip-19994

systemd/ceph-mgr: remove automagic mgr creation hack

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15956 from dillaman/wip-librbd-devstack

test: fix failing rbd devstack teuthology test

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15923 from dillaman/wip-qa-rbd-memcheck-python

qa/suites/rbd: restrict python memcheck validation to CentOS

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

common: fix log warnings

/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/LogEntry.cc:54:24: warning: suggest parentheses around comparison in operand of ‘|’ [-Wparentheses]
   } else if (level_str == "warn" | level_str == "warning") {
                        ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/LogEntry.cc:56:24: warning: suggest parentheses around comparison in operand of ‘|’ [-Wparentheses]
   } else if (level_str == "error" | level_str == "err") {

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph.in: allow developer mode from outside build tree

Use path of script to look for 'am I in a build tree' rather than
getcwd(), so that 'build tree' versions can be run from anywhere

Fixes: http://tracker.ceph.com/issues/20472
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #15911 from xiexingguo/wip-bluestore-20170626

os/bluestore: narrow cache lock range; make sure min_alloc_size p2 aligned

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #16044 from liewegas/wip-i-cant-believe-its-not-btrfs

qa: stop testing btrfs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #16046 from liewegas/wip-valgrind-on-centos

do all valgrind runs on centos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: recommend against btrfs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15963 from Songweibin/wip-release-notes

doc/release-notes: add Images creation timestamp note

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15924 from jcsp/wip-log-last-filters

mon: Filter `log last` output by severity and channel

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15946 from Yan-waller/wip-waller-dumpwatcher0627

osd: dump the field name of object watchers  and cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15959 from smithfarm/wip-rgw-singleton-cleanup

tests: rgw/singleton: drop duplicate filestore-xfs.yaml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15961 from majianpeng/filestore-remove-redundant-judgement

os/filestore/FileStore.cc: remove a redundant judgement when get max latency

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #15999 from Yan-waller/wip-waller-0629message

msg/MOSDOpReply: fix missing trace decode

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16037 from gmayyyha/monmapmonitor-use-func

mon/MonmapMonitor: use __func__ instead of hard code function name

Reviewed By: Joao Eduardo Luis <jecluis@gmail.com>

do all valgrind runs on centos

We are fighting two issues with valgrind on ubuntu (xenial, yakkety,
and z):

http://tracker.ceph.com/issues/18126
http://tracker.ceph.com/issues/20360

Revert this when it is fixed.

Signed-off-by: Sage Weil <sage@redhat.com>

qa: stop testing btrfs

This is too noisy and we don't recommend it anyway.

See http://tracker.ceph.com/issues/20169

Signed-off-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: add a note on choose_args encoding

It's unlikely that it's been used yet, but worth a note.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crush/CrushWrapper: crush_choose_arg::ids should be __s32

crush_choose_arg::ids array is encoded on the wire.  int is not fixed
size -- use __s32 instead (crush_bucket::id is __s32).

This was introduced in commit dbe36e08be00 ("crush: compile/decompile
crush_choose_arg_map") under SERVER_LUMINOUS bit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crush/CrushWrapper: encode choose_args.size() as __u32

choose_args.size() returns size_t, which is not fixed size.  Given that
choose_args code just open-codes our std::map encoder, do what we do for
std::map.

This was introduced in commit dbe36e08be00 ("crush: compile/decompile
crush_choose_arg_map") under SERVER_LUMINOUS bit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon/MonmapMonitor: use __func__ instead of hard code function name

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

systemd: Add explicit Before=ceph.target

The PartOf= and WantedBy= directives in the various systemd
unit files and targets create the following logical hierarchy:

- ceph.target
  - ceph-fuse.target
    - ceph-fuse@.service
  - ceph-mds.target
    - ceph-mds@.service
  - ceph-mgr.target
    - ceph-mgr@.service
  - ceph-mon.target
    - ceph-mon@.service
  - ceph-osd.target
    - ceph-osd@.service
  - ceph-radosgw.target
    - ceph-radosgw@.service
  - ceph-rbd-mirror.target
    - ceph-rbd-mirror@.service

Additionally, the ceph-{fuse,mds,mon,osd,radosgw,rbd-mirror}
targets have WantedBy=multi-user.target.  This gives the
following behaviour:

- `systemctl {start,stop,restart}` of any target will restart
  all dependent services (e.g.: `systemctl restart ceph.target`
  will restart all services; `systemctl restart ceph-mon.target`
  will restart all the mons, and so forth).
- `systemctl {enable,disable}` for the second level targets
  (ceph-mon.target etc.) will cause depenent services to come
  up on boot, or not (of course the individual services can
  be enabled or disabled as well - for a service to start
  on boot, both the service and its target must be enabled;
  disabling either will cause the service to be disabled).
- `systemctl {enable,disable} ceph.target` has no effect on
  whether or not services come up at boot; if the second level
  targets and services are enabled, they'll start regardless of
  whether ceph.target is enabled.  This is due to the second
  level targets all having WantedBy=multi-user.target.
- The OSDs will always start regardless of ceph-osd.target
  (unless they are explicitly masked), thanks to udev magic.

So far, so good.  Except, several users have encountered
services not starting with the following error:

  Failed to start ceph-osd@5.service: Transaction order is
  cyclic. See system logs for details.

I've not been able to reproduce this myself in such a way as to
cause OSDs to fail to start, but I *have* managed to get systemd
into that same confused state, as follows:

- Disable ceph.target, ceph-mon.target, ceph-osd.target,
  ceph-mon@$(hostname).service and all ceph-osd instances.
- Re-enable all of the above.

At this point, everything is fine, but if I then subseqently
disable ceph.target, *then* try `systemctl restart ceph.target`,
I get "Failed to restart ceph.target: Transaction order is cyclic.
See system logs for details."

Explicitly adding Before=ceph.target to each second level target
prevents systemd from becoming confused in this situation.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #15927 from jcsp/wip-mgr-log-on-load-failed

mgr: cluster log message on plugin load error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15951 from wido/issue-20430

osd: Execute crush_location_hook when configured in ceph.conf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15954 from Liuchang0812/wip-osd-compact

mon: fix kvstore type in mon compact command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15957 from liewegas/wip-rc

ceph_release: we are in the 'rc' phase (12.1.z)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15968 from xiexingguo/wip-poolopts-fix

mon/OSDMonitor: two pool opts related fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15974 from liewegas/wip-require-osd-release-print

osd/OSDMap: print require_osd_release

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15976 from liewegas/wip-bluestore-big-cache

os/bluestore: default cache size of 3gb

Merge pull request #15489 from xxhdx1985126/master

librados: redirect balanced reads to acting primary when targeting object isn't recovered

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14678 from joscollin/wip-cleanup-storeget-return-check

mon: handle cases where store->get() may return error

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15926 from liewegas/wip-log-channel

librados: add log channel to rados_monitor_log2 callback

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15925 from jcsp/wip-fsstatus-cleanup

mgr: clean up fsstatus module

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15932 from dillaman/wip-20421

pybind: restore original API for backwards compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14760 from dzafman/wip-19657

osd: On EIO from read recover the primary replica from another copy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15912 from cloudbau/wip-20107

rgw: Fix duplicate tag removal during GC

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Tested-by: Yuri Weinstein <yweinste@redhat.com>

systemd/ceph-mgr: remove automagic mgr creation hack

For kraken we auto-created mgr daemons next to mon daemons with some
systemd hackery.  This is awkward (you can't not get a new mgr daemon when
you deploy a mon), systemd-specific (not implemented for upstart on
trusty), and mostly unexpected.  Since ceph-mgr daemons are now first-class
citizens and required for every cluster, make their deployment explicit
and transparent to the administrator.  Major upgrades are a rare
opportunity to have the administrator's full attention so take advantage
of it.

This effectively reverts 61d779345e9efbe9a2e3f215af1f1dcf6630f04a and
082199f69dd0bd4c18a5f4baea67a88782586657 (and follow-on fixes).

Fixes/avoids: http://tracker.ceph.com/issues/19994
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15894 from liewegas/wip-no-default-pool

mon,osd: do not create rbd pool by default

mon: make `log` command send messages to default channel

...rather than leaving channel blank.  This means that
people watching the main "cluster" channel will see
the log messages sent using `log`.

Signed-off-by: John Spray <john.spray@redhat.com>

message: fix missing decode trace

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #15471 from tchaikov/wip-radostriper-leak-on-fail

libradosstriper: fix MultiAioCompletion leaks on failure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: Fix osd-scrub-repair for movement of shards

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Clean up extra json output from log

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Give time for OSDs to see map with flag change

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15812 from dzafman/wip-cot-cleanup

tools: Cleanup dead code in ceph-objectstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15983 from cbodley/wip-20386

rgw: fix infinite loop in rest api for log list

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15635 from dzafman/wip-13837

osd: Corrupt objects stop snaptrim and mark pg snaptrim_error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15987 from liewegas/wip-19994

doc/release-notes: fix ceph-deploy command

doc/release-notes: fix ceph-deploy command

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: avoid dup health warnings during luminous upgrade

PGMonitor is still providing warnings; do not expose the ones we are
getting from ceph-mgr (which also come from PGMap::get_health()) too or
else we'll get dups!

Fixes: http://tracker.ceph.com/issues/20435
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: rest handlers for mdlog and datalog list dont loop

the do-while loops are unnecessary, because cls_log_list() will already
give us the requested number of entries, unless truncated. and these
rest operations return the truncated flag and last_marker, which the
client side is already using to run the same loop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix RGWMetadataLog::list_entries() for null last_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWMetadataLog::list_entries() no longer splices

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15850 from rzarzynski/wip-rgw-20384

rgw: fix wrong error code for expired Swift TempURL's links.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15883 from rzarzynski/wip-rgw-20395

rgw: omit X-Account-Access-Control if there is no grant to serialize.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15884 from joscollin/wip-1412617-uninitialized-pointer-field

rgw: initialize non-static class members in ESQueryCompiler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15886 from joscollin/wip-CID1412616-uninitialized-pointer-field

rgw: initialize Non-static class member worker in RGWReshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15888 from joscollin/wip-1412615-uninitialized-scalar-field

rgw: initialize Non-static class member val in ESQueryNodeLeafVal_Int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15913 from smithfarm/wip-20417

tests: rgw.py: put client roles in a separate list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test,qa/workunits: fix a zillion tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not start mapping job on empty map

The Mapper doesn't do the completion.  Just avoid it in the caller instead
of adding a Finisher or something similarly annoying for the completion.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: create rbd pool after starting the osds

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers: shorten mon connect timeout

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: do not create default pool without --with-default-pool

Signed-off-by: Sage Weil <sage@redhat.com>

test: update TestPGLog's pool_id for the new world (no pool 0)

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: maintain a pool-creating build_simple() for use in unit tests

Change the standard build_simple() so it no longer takes pg[p]_num -- these
params were unused. Add a new build_simple_with_pool() that can be called
when we want the rbd pool to be created by default. These are implemented
via a simple switched function (marked private).

Update the existing callers so they choose the right one for their needs.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osdmap: reference the correct value in debug output

Looks like we've been printing out the (possibly-still-default-initialized)
num_osd member variable instead of the given nosd value! Luckily
I don't see any other misuse and we fill out

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cephtool/test: create an rbd pool since it's no longer automatic

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: provide more flexibility in the PrimaryAffinity check

Fix the margins for the primary affinity test.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

osdmap: do not automatically create an rbd pool on cluster creation; skip pool 0

We don't use pool ID 0 any more because CephFS can't handle it.
Update the user quick-start to include it, and add some links to support that process.
Fix the tests to handle the new state of life.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

TestOSDMap: actually loop through the pools as intended

Looks like this was meant to be a loop but (unlike the other tests)
just referred to a constant pool 0. Whoops!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

pgmap: extend mon_pg_warn_min_objects to cover the PG-per-OSD warnings too

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/jobs: remove these definitely-broken old testing scripts

They haven't been touched in 6 years and were considered broken then.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: default cache size of 3gb

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: print require_osd_release

Signed-off-by: Sage Weil <sage@redhat.com>