RadosStriperImpl.cc: fix double free

Some variables have been already free'd by calling
release(), no need to call delete() again on them.

CID 1396200 (#1 of 1): Double free (USE_AFTER_FREE)
 double_free: Calling operator delete frees pointer
 getxattr_completion which has already been freed.
CID 1396206 (#1 of 1): Double free (USE_AFTER_FREE)
 double_free: Calling operator delete frees pointer
 getxattr_completion which has already been freed.
CID 1396214 (#1 of 1): Double free (USE_AFTER_FREE)
 double_free: Calling operator delete frees pointer
 stat_completion which has already been freed.
CID 1396218 (#1 of 1): Double free (USE_AFTER_FREE)
 double_free: Calling operator delete frees pointer
 stat_completion which has already been freed.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osdc/ObjectCacher.cc: fix wrong self assignment

Fix for:

CID 1395468 (#1 of 1): Self assignment (NO_EFFECT)
self_assign: Assigning bh->last_write to itself has no effect.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Checksummer.h: close resource leak

CID 1395784 (#2-1 of 2): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable state going out of scope
 leaks the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_main.cc: close leak in error case

ID 1398895 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable config going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_rados.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_main.cc: reorder framework detection

Fix for:

CID 1398896: Resource leak (RESOURCE_LEAK)
 overwrite_var: Overwriting fe in fe = new
 RGWLoadGenFrontend(env, config) leaks the storage that fe
 points to.

Do not check again and again for framework if there is
already a match, make use of if/else. This should also
fix the RESOURCE_LEAK warning.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_compression.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_common.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_torrent.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

RDMAConnectedSocketImpl.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mon/OSDMonitor.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mds/Mantle.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_lc.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_token.h: remove redundant and slow call to c_str()

Do not call c_str() if the parameter is already of type string.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

kv/MemDB.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cls/rgw/cls_rgw.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #13278 from liewegas/wip-luminous

mon,osd: luminous feature bits, require flags, upgrade gates

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12829 from ovh/wip-perf-counters-histogram

common, osd, tools: Add histograms to performance counters

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: whu_liuchang <nfcj000@qq.com>
Reviewed-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #13296 from liewegas/wip-kill-sortbitwise

osd: remove sortbitwise thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13097 from ceph/wip-osd-after-mon

systemd: Start OSDs after MONs

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13319 from smithfarm/wip-buildpackages-py

tests: drop buildpackages.py

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: drop buildpackages.py

The buildpackages suite has been moved to teuthology. This cleans up a file
that was left behind by https://github.com/ceph/ceph/pull/13297

Fixes: http://tracker.ceph.com/issues/18846
Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/suites/rados/upgrade/jewel-x-singleton: upgrade mons first; box thrasher

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/luminous'

Merge pull request #13218 from guits/ceph_detect_init

ceph-detect-init: Add docker detection

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

12.0.0

ceph-detect-init: Add docker detection

if running `ceph-detect-init` from a container, the given output is the
reference from the hosting system. This patch add the capability of
detecting if the tool is being run inside a container.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #12586 from ktdreyer/wip-18313-rgw-upstart

upstart: start radosgw-all according to runlevel

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #12086 from wjwithagen/wip-wjw-freebsd-ceph-disk

ceph-disk: FreeBSD changes to get it working and passing tests

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13015 from chuanhong-wang/dev_change_osd_df_tree_output

mon: in output of "ceph osd df tree", display "-", not "0",  for pg amount of a bucket

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13159 from bassam/pr-fix-embedded-zstd

compressor/zstd: add zstd to embedded ceph

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13236 from adamemerson/wip-alarum

cleanup: resolve compiler warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13265 from tchaikov/wip-librados-hide-private-symbols

librados: do not expose non-public symbols

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13268 from tchaikov/wip-better-error-ping-mon

ceph: do not throw TypeError on connection failure

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13286 from Liuchang0812/remove-useless-close

msg/async: remove useless close function

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #13288 from dmick/master

ceph.in: add help for locally-handled commands

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #9976 from songbaisen/a13

mon: PGMonitor add check only concern our self cluster command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10150 from songbaisen/b

mon: avoid start election twice when quorum enter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10309 from songbaisen/b8

mon: OSDMonitor add check only concern our self cluster command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12273 from songbaisen/s4

mon: add warn info for osds were removed from osdmap but still kept in crushmap

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13121 from LiumxNL/wip-170125

osd: we know the definite epoch of marking down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13297 from dachary/wip-buildpackages

buildpackages: remove because it does not belong

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #13162 from liewegas/wip-bluestore-reshard

os/bluestore: partial reshard support

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

mon: prevent sortbitwise flag from being unset

Signed-off-by: Sage Weil <sage@redhat.com>

buildpackages: remove because it does not belong

It should live in teuthology, not in Ceph. And it is currently broken:
there is no need to keep it around.

Fixes: http://tracker.ceph.com/issues/18846
Signed-off-by: Loic Dachary <loic@dachary.org>

osd/PG: drop sortbitwise feature check assert

This feature bit will get deprecated shortly anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop osd_debug_randomize_hobj_sort_order

This is required post-jewel; no need to thrash it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: stop thrashing sortbitwise

This is required to upgrade past jewel; no need to keep testing it
on master.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: fix kraken feature requirement

Use the FEATUREMASK since this is a reused bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use HAVE_FEATURE for kraken, jewel feature tests

Signed-off-by: Sage Weil <sage@redhat.com>

mon: apply monmap features to quorum requirements

Previously this was indirect..

 monmap requirements -> compatset -> quorum requirements

However, how that we are consistently using the monmap features, we no
longer need to use the compatset features when the monmap already
communicates the same thing.  Rename the helper to reflect that
quorum requirements come from both the compatset and the monmap.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: fix kraken requirement to use FEATUREMASK, HAVE_FEATURE macro

The KRAKEN_SERVER bit was reused.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: require luminous mons before starting

Require that mons are upgraded to luminous first.  This will ensure that
the last_force_op_resend flag is set for pre-luminous clients so that
luminous OSDs can safely discard them.  (This will come in a future patch.)

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop redundant check for hammer OSDs

The OSDMAP_REQUIRE_JEWEL flag requirement implies that there are no
pre-0.94.4 (early hammer) OSDs running.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/mon_types: add sticky FEATURE_LUMINOUS feature bit

Once mons upgrade to luminous, you can't go back.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: all upgrades pass through luminous

Prevent post-luminous OSDs from starting before require_luminous_osds is
set.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add REQUIRE_LUMINOUS flag

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13035 from yehudasa/wip-18621

rgw: change default chunk size to 4MB

Merge pull request #13259 from jcsp/wip-jcsp-relnotes

PendingReleaseNotes: note the fuse fstab format change

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.in: add help for locally-handled commands

Signed-off-by: Dan Mick <dan.mick@redhat.com>

msg/async: remove useless close function

fix coverity cid 1395314: it's invalid to pass a negative value to close function, and we have closed socket in net.connect/nonblock_connect in error cases indeed

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #13279 from smithfarm/wip-fix-regression

tests: fix regression in qa/tasks/ceph_master.py

FreeBSD: changes to ceph-disk to get it working and passing tests

 - It currently only supports running with filestore
 - Testing is executed while running on a ZFS partition
 - All disktypes and naming is different on FreeBSD
 - Partitioning and tools are not workable on FreeBSD
 - add diagnostic timeout

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: don't update bucket index multiple times in overwrite

Instead of this for overwrites:
 prepare (index),
   write (head) [-EEXIST]
 cancel (index)
 read (head)
 prepare (index)
   write (head)
 complete (index)

We now do:
 prepare (index),
   write (head) [-EEXIST]
   read (head)
   write (head)
 complete (index)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: assume obj write is a first write

if fails and object already exists then retry. This improves first obj
write performance on the expense of overwrites.

Fixes: http://tracker.ceph.com/issues/18622
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: configurable write obj window size

Fixes: http://tracker.ceph.com/issues/18623
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

include/ceph_features: retire OSD_MIN_SIZE_RECOVERY, OSD_PROXY_FEATURES

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire OSD_REPOP, OSD_OBJECT_DIGEST, OSD_TRANSACTION_MAY_LAYOUT

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire OSD_SET_ALLOC_HINT

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire OSD_PACKED_RECOVERY

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire MON_SCRUB

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: remove ceph_sanitize_features workaround

Way back in 9ea02b84104045c2ffd7e7f4e7af512953855ecd we defined the
SNAPMAPPER feature such that it set bits 32-63 by sign-extending a negative
int32_t to int64_t.  In 4255b5c2fb54ae40c53284b3ab700fdfc7e61748 it was
fixed.  This affected v0.58-657-g9ea02b8 to v0.65-263-g4255b5c.

The workaround santized features whenever the top bit 63 was set to be the
intended set of features (bits 0-32).  Only buggy peers in that range
ever set bit 63.

Since these versions are truly ancient (pre-dumpling) and no clients
should be that old, we are dropping the workaround.  Note that the Linux
kernel client never had this bug: 2b3e0c905af43cfe402a2ef3f800be5dc1684005
went from 30 bits used to 38 bits defiend with the workaround in place.

We deprecate the bit now (i.e., stop paying attention to this bit).  After
another extended period of time we'll eventually be able to retire
and reuse it.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire OSD_SNAPMAPPER

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire {BACKFILL,RECOVERY}_RESERVATION

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_feaetures: retire MON_NULLROUTE

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire CHUNKY_SCRUB

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: use FEATUREMASK for testing for SERVER_KRAKEN on the server

The mon we are talking to may be an old ceph-mon with the prior meanging of
the SERVER_KRAKEN bit; use the mask to disambiguate instead of testing for
two featurse explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: return INDEP_PG_MAP

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire QUERY_T

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: retire MONCLOCKCHECK

Deprecated in Jewel.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: has_features()

Test for multiple features all being set together.  Note that this can be
used with the CEPH_FEATUREMASK_ defines to safely test for reused feature
bits.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: define HAVE_FEATURE helper

This lets you test a feature succintly by name, *AND* makes use of the
FEATUREMASK so that it can be used on bitmasks that may come from old
versions with prior usages of the bit.  This makes it usable for data
type encoding or with client-provided bits.

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: new macros, and document deprecation process

This was super confusing before.  Hopefully it is better now!

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: add SERVER_M feature placeholder

Signed-off-by: Sage Weil <sage@redhat.com>

include/ceph_features: CEPH_FEATURE_RESEND_ON_SPLIT

Signed-off-by: Sage Weil <sage@redhat.com>

tests: fix regression in qa/tasks/ceph_master.py

https://github.com/ceph/ceph/pull/13194 introduced a regression:

2017-02-06T16:14:23.162 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 722, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 839, in do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 305, in kill_osd
    output = proc.stderr.getvalue()
AttributeError: 'NoneType' object has no attribute 'getvalue'

This is because the original patch failed to pass "stderr=StringIO()" to run().

Fixes: http://tracker.ceph.com/issues/16263
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12979 from theanalyst/scripts/release-notes-tracker

script: ceph-release-notes check orig. issue only for backports

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13277 from jan--f/janf-teuth-cloud-doc

doc: minor change to a cloud testing paragraph

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13275 from idryomov/wip-require-osd-features-fix

osd/OSDMap: require OSD features only of OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: minor change to a cloud testing paragraph

Use terminate for the instances to match the terminology used by the
horizon interface.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #13256 from dongbula/correct-cached_crc-comment

osd: correct comment of perfcounter cached_crc in code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13220 from rzarzynski/wip-rgw-18364-2

rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12993 from jcsp/wip-cephfs-doc-authcaps

doc: clarify the path restriction mds cap example

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13242 from Jing-Scott/dev/fix-swift-anon-getobj-error-code

rgw: swift: fix anonymous user's error code of getting object

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

osd/OSDMap: require OSD features only of OSDs

Fix up commit 1a5cc32f0a3b ("osd/OSDMap: reflect REQUIRE_*_OSDS flag in
required features") -- require_*_osds shouldn't affect older clients.

Fixes: http://tracker.ceph.com/issues/18831
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #13271 from Liuchang0812/fix-file-description-leak

msg/async: fix file description leak in NetHandler

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: fix file description leak in NetHandler

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: swift: fix anonymous user's error code of getting object

The openstack swift will return 401 rather than 403 when
the anon user has no permission to get objects.

Fixes: http://tracker.ceph.com/issues/18806
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

ceph: do not throw TypeError on connection failure

otherwise we could concat None with a string on connection problem.
which will result in TypeError. and the caller will print misleading
error like

Error connecting to cluster: TypeError

see also #12934

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13237 from smithfarm/wip-18799

tests: Thrasher: eliminate a race between kill_osd and __init__

Reviewed-by: Sage Weil <sage@redhat.com>