crush: implement CrushWrapper::dump(choose_args)

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: disable modification API when choose_args is not empty

Adding, removing or move items / buckets via the CrushWrapper API when
choose_args is not empty is unlikely to produce the desired outcome. The
caller should instead add, remove or move items / buckets in a
decompiled crushmap, update the associated choose_arg and upload the new
crushmap.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: compile/decompile crush_choose_arg_map

A map of crush_choose_arg_map is added to the crushmap text syntax. The
key is an integer matching a pool number.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: add per pool choose_args when calling do_rule

If there is no crush_choose_arg_map for a given pool (the default) a
NULL pointer is given instead and crush_do_rule behavior remains
unchanged.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: implement weight and id overrides for straw2

bucket_straw2_choose needs to use weights that may be different from
weight_items. For instance to compensate for an uneven distribution
caused by a low number of values. Or to fix the probability biais
introduced by conditional probabilities (see
http://tracker.ceph.com/issues/15653 for more information).

We introduce a weight_set for each straw2 bucket to set the desired
weight for a given item at a given position. The weight of a given item
when picking the first replica (first position) may be different from
the weight the second replica (second position). For instance the weight
matrix for a given bucket containing items 3, 7 and 13 could be as
follows:

          position 0   position 1

item 3     0x10000      0x100000
item 7     0x40000       0x10000
item 13    0x40000       0x10000

When crush_do_rule picks the first of two replicas (position 0), item 7,
3 are four times more likely to be choosen by bucket_straw2_choose than
item 13. When choosing the second replica (position 1), item 3 is ten
times more likely to be choosen than item 7, 13.

By default the weight_set of each bucket exactly matches the content of
item_weights for each position to ensure backward compatibility.

bucket_straw2_choose compares items by using their id. The same ids are
also used to index buckets and they must be unique. For each item in a
bucket an array of ids can be provided for placement purposes and they
are used instead of the ids. If no replacement ids are provided, the
legacy behavior is preserved.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: cleanup test memory leaks

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: do not use TREE in tests

It is bugous and unsupported.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #14460 from dillaman/wip-qa-rbd-cli

qa/workunits: corrected issues with RBD cli test

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14467 from badone/wip-debug-mon-features-api-removal-cleanup

mon: Tidy up removal of debug mon features

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14361 from ShiqiCooperation/master

mailmap: shiqi affiliation

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14446 from badone/wip-cppcheck-errors

Wip cppcheck errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Suppress cppcheck error

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14419 from badone/wip-doc-pg-states-min_size

docs: Clarify the relationship of min_size to EC pool recovery

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Tidy up removal of debug mon features

545bc83 removed most of the plumbing for the debug mon features admin
socket commands but failed to remove the register/unregister command
pairs. This means the monitor asserts if an attempt is made to use any
of these commands.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14290 from wjwithagen/wip-wjw-freebsd-procprefix

core: Prefix /proc/ with FreeBSD emulation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14362 from majianpeng/fs-remove-redundant-code

os/fs/FS.cc: remove the redundant code.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14462 from pdinnen/patch-2

doc: correct arguments for ceph tell osd.N bench

Reviewed-by: Sage Weil <sage@redhat.com>

doc: correct arguments for ceph tell osd.N bench

Signed-off-by: Patrick Dinnen pdinnen@gmail.com

Merge pull request #14461 from dmick/master

stop.sh: boilerplate error (don't stop mon when stopping mgr)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

stop.sh: boilerplate error (don't stop mon when stopping mgr)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14376 from cbodley/wip-rgw-forward-canned-acl

rgw: fix for EINVAL errors on forwarded bucket put_acl requests
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13105 from rjfd/wip-18481

librbd: deferred image deletion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits: corrected issues with RBD cli test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14355 from cbodley/wip-19505

rgw: allow larger payload for period commit

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14319 from cbodley/wip-19488

rgw: multisite: fixes for zonegroup redirect

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14343 from Liuchang0812/wip-rgw-read-user-buckets

rgw: update is_truncated in function rgw_read_user_buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14438 from theanalyst/fix/build-doc

script: build-doc/serve-doc fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14455 from optimistyzy/411_1

os/bluestore/NVMEDevice: fix the compilation issue for collect_metadata

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13698 from ukernel/wip-multimds-misc

mds: miscellaneous multimds fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14453 from trociny/wip-test_mock_InstanceWatcher

test/rbd_mirror: race in TestMockInstanceWatcher on destroy

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind: librbd: added trash API to python RBD binding

Signed-off-by: Ricardo Dias <rdias@suse.com>

qa/workunits/rbd: cli testing of rbd trash commands

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: added image-id optional to Info command

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: Added image-id optional to Snap commands

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: added image_id param to Utils::init_and_open_image

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: added group image remove by id to CLI

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: added group image remove by id to API

Signed-off-by: Ricardo Dias <rdias@suse.com>

rbd: added CLI support for trash operations

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: added trash methods to API

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: open image gets name from directory or trash

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: added open an image by id to API

Signed-off-by: Ricardo Dias <rdias@suse.com>

cls_rbd: methods to manipulate the rbd_trash object

Signed-off-by: Ricardo Dias <rdias@suse.com>

bluestore/NVMEDevice: fix the compilation issue for collect_metadata

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

test/rbd_mirror: race in TestMockInstanceWatcher on destroy

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14295 from theanalyst/release/10.2.7/notes

doc: release notes for v10.2.7 Jewel

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14441 from theanalyst/doc/changelog/v10.2.7

doc: add changelog for v10.2.7

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #14426 from tchaikov/wip-13562-kefu

misc: SCA fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Silence cppcheck memory leak warnings in ceph_test_rados_api_io

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

denc: Silence cppcheck uninitialised variable errors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common: Restructure cppcheck suppresions

Some of the suppressions are currently redundant and some need to be
changed.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

crypto: Suppress cppcheck error

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

blkdev: Suppress cppcheck error

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13940 from Adirl/out

msg/async/rdma: Update fix broken compilation

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14409 from Adirl/conn_info

msg/async/rdma: RDMA-CM, Pass specific ConnMgr info in constructor

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14410 from Adirl/get_device

msg/async/rdma: RDMA-CM, get_device() by ibv_context

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13073 from chardan/jfw-wip-15475-admin_heap_command

osd: add "heap *" admin command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14235 from nvedant07/bug_fix_3

rados: more info added to pool deletion error

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14022 from batrick/admin-socket-close-error

common: do not print error when asok is closed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14292 from joscollin/wip-radosacl-warning

tools: warning, ‘%.16x’ directive output truncated writing 16 bytes into a region of size 9.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14301 from Andrey1994/master

build: Add Virtuozzo Linux support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14336 from Adirl/fix_sockaddr

src/msg/async: Update fix broken compilation for Posix

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: add changelog for v10.2.7

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #14039 from dachary/wip-19307-set-deviceclass

mon/OSDMonitor: osd crush set-device-class

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14208 from dachary/wip-crush-uniform-weights

crush: bucket: crush_add_uniform_bucket_item should check for uniformity

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14209 from dachary/wip-crush-create-default-tunables

crush: builder: creating crush map with optimal configurations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14401 from dillaman/wip-rbd-import-fixes

rbd: correct issues with image importing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14437 from cernceph/dvanders_mgrrotate

ceph-mgr: rotate logs on sighup

Reviewed-by: Sage Weil <sage@redhat.com>

script: serve-doc print the port we're serving

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

script: build-doc use python2 only for virtualenv

This is because the sphinx-ditaa package and ceph's rbd/rgw/rados pybind
require python2

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes for v10.2.7 Jewel

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #14413 from hansbogert/patch-3

doc: rgw: remove mention of megabytes for quotas

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

logrotate.conf: also sighup the ceph-mgr

Tell ceph-mgr to reopen its logs postrotate.

Fixes: http://tracker.ceph.com/issues/19568
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

ceph_fuse.cc: fix 64/32bit detection

Fix for:

[src/ceph_fuse.cc:111]: (style) Condition 'sizeof(long)==4' is always false

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

lua_bufferlist.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

BitAllocator_test.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_librbd.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rest_s3.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

PrimaryLogPG.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PG.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

BitAllocator.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #14430 from jcsp/wip-19566-monassert

mon: avoid segfault in wait_auth_rotating

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-mgr: reopen logs on SIGHUP

Wire up the sighup_handler to reopen logs, like the other daemons.

Fixes: http://tracker.ceph.com/issues/19568
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #14428 from tchaikov/wip-ceph_objectstore_tool-kill-all

tests: ceph_objectstore_tool.py: kill all daemons

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10636 from fullerdj/wip-djf-15069

cephfs: Permit recovering metadata into a new RADOS pool

Reviewed-by: John Spray <john.spray@redhat.com>

mon: avoid segfault in wait_auth_rotating

MonClient users should not be calling into MonClient
after calling shutdown().  However, MonClient should
assert out rather than proceeding to try and
follow a maybe-null pointer.

Related to http://tracker.ceph.com/issues/19566

Signed-off-by: John Spray <john.spray@redhat.com>

tests: ceph_objectstore_tool.py: kill all daemons

init-ceph kills all daemons that can be identified by get_name_list() by
default, so no need to list them explicitly. also, without this change,
mgr is not stopped when the test finishes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.cc: use static_cast instead of c-style

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rest_s3.cc: reorder checks in while loop

Fix for:

[src/rgw/rgw_rest_s3.cc:1224]: (style) Array index 'chunk_offset'
 is used before limits check.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

AsyncMessenger.cc: fix FORWARD_NULL

Use assert to ensure Message *m is not NULL.

Fix for:

CID 1399578 (#4 of 4): Dereference after null check (FORWARD_NULL)
 var_deref_model: Passing null pointer m to submit_message,
 which dereferences it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_file.h: fix potential data race condition

Fix for:

CID 1399581 (#1 of 1): Data race condition (MISSING_LOCK)
 missing_lock: Accessing this->state.atime without holding lock
 rgw::RGWFileHandle.mtx. Elsewhere, "_ZN3rgw13RGWFileHandle5StateE.atime"
 is accessed with rgw::RGWFileHandle.mtx held 2 out of 3 times
 (2 of these accesses strongly imply that it is necessary).

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

cls_rgw_client.h: Close resource leak

Fix for:

CID 1395687 (#1 of 1): Resource leak (RESOURCE_LEAK)
CID 1395679 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable c going out of scope leaks the storage it
 points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

mount.ceph.c: fix resource leak

Addressed issue:

CID 1219611 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable saw_name going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_ipaddr.cc: fix UNINIT issues around sockaddr_in struct

Fix for:

CID 1396088 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396090 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396091 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396092 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396096 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396097 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396099 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396101 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396103 (#1 of 1): Uninitialized scalar variable (UNINIT)
CID 1396104 (#1 of 1): Uninitialized scalar variable (UNINIT)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #14280 from tone-zhang/bug_19442

rgw: fix response header of Swift API

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #14378 from kestrels/wip-19454

arch: fix build on PowerPC with FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

docs: Clarify the relationship of min_size to EC pool recovery

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #14360 from joscollin/wip-Update-organizationmap

.organizationmap: Updated authors

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13803 from trociny/wip-18785

rbd-mirror: separate ImageReplayer handling from Replayer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14406 from wjwithagen/wip-wjw-freebsd.readme

README.FreeBSD: Update the status

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: rgw: remove mention of megabytes for quotas

Remove mention of confusing reference to megabytes in opening paragraph of "quota management".
To remove further confusion, mention that setting a size for '--max-size' is in bytes by default.

Signed-off-by: Hans van den Bogert <hansbogert@gmail.com>