rgw: instruct civetweb to drop permissions

Get uid set in g_ceph_context and pass to civetweb where setuid()/setgid()
will be invoked. This Allows for starting civetweb under privileged port
numbers.

Fixes: #13600
Signed-off-by: Karol Mroz <kmroz@suse.com>

rgw_main: break up frontend config parsing

Before invoking global_init(), determine which frontends are configured.
If Civetweb is to be used, ensure it will be responsible for dropping
permissions after radosgw startup.

Signed-off-by: Karol Mroz <kmroz@suse.com>

global_init: add run_pre_init argument to function signature

Allows running the global_pre_init() separately in order to parse config
options prior to invoking global_init().

Signed-off-by: Karol Mroz <kmroz@suse.com>

global: record target user and group strings

Useful so we don't have to do a reverse lookup when dropping permissions
in civetweb.

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7798 from ukernel/wip-mds-snap

A few more snapshot fixes, mostly around snapshotted inode/dentry tracking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'origin/greg-fs-testing-3-7-damage'

Merge pull request #7686 from david-z/wip-fix-fh-ref-leak

ceph-fuse: Fix potential filehandle ref leak at umount

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7685 from david-z/wip-fix-put-free-fd-back

[cephfs] fix free fds being exhausted eventually because freed fds are never put back

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7003 from jcsp/wip-cephfs-tmap-migrate

tools/cephfs: add tmap_upgrade

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8014 from ukernel/wip-15038

mds: fix inode_t::compare()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: fix inode_t::compare()

Fixes: #15038
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7978 from dillaman/wip-14908

librbd: refresh image if required before replaying journal ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8009 from dillaman/wip-14988

test: reproducer for writeback CoW deadlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8005 from dillaman/wip-rbd-lock-fence

pybind: flag an RBD image as closed regardless of result code

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8001 from dillaman/wip-rbd-cram

test: update rbd integration cram tests for new default features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: reproducer for writeback CoW deadlock

Refs: #14988

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7906 from dillaman/wip-14869

journal: re-use common threads between journalers

Conflicts:
src/journal/JournalPlayer.cc
src/librbd/Journal.cc
src/test/rbd_mirror/image_replay.cc
src/tools/rbd_mirror/ImageReplayer.h
src/tools/rbd_mirror/Mirror.cc

(merged interface changes to ImageReplayer, and reduced scope for
change to JournalPlayer due to pr #7884 (wip-14663)).

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7958 from trociny/wip-rbd-mirror-fixup

rbd-mirror: minor fix-ups for initial skeleton implementation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7884 from dillaman/wip-14663

librbd: allocate new journal tag after acquiring exclusive lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7772 from xinxinsh/wip-14789

librbd: truncate does not need to mark the object as existing in the object map

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8003 from liewegas/wip-ceph-disk-doc

man/8/ceph-disk: fix formatting issue

Merge pull request #7927 from oritwas/wip-rgw-fix-default

rgw: RGWZoneParams::create should not handle -EEXIST error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7768 from xinxinsh/wip-fix-zero

osd: filestore: FALLOC_FL_PUNCH_HOLE must be used with FALLOC_FL_KEEP_SIZE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7844 from xiexingguo/xxg-wip-14922

osd: replicatedpg: break out loop if we encounter fatal error during do_pg_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7864 from dillaman/wip-librados-aio-notify

librados: race condition on aio_notify completion handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7928 from branch-predictor/bp-radostool-reduce-mem

tools/rados: reduce "rados put" memory usage by op_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7967 from tchaikov/wip-si-cast-coverity

common/strtol.cc: fix the coverity warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7948 from avnerbh/fix_avoid_enqueue_dup_code

msg: remove duplicated code - local_delivery will now call 'enqueue'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7890 from liewegas/wip-reweight

mon: osd [test-]reweight-by-{pg,utilization} command updates

Reviewed-by: Samuel Just <sjust@redhat.com>

pybind: flag an RBD Image as closed regardless of result code

The image will always be closed -- the error code just alerts the
user to any possible error encountered when closing the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit watch flush to return error code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: remove range=100 from reweight-by-* commands

Specifying the range makes it skip instead of error when you give
it a bad value.

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>

test: update rbd integration cram tests for new default features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7847 from wido/mirroring-fixes

Updated scripts and documentation for mirrors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add script to test connectivity and availability of Ceph mirrors

Signed-off-by: Wido den Hollander <wido@42on.com>

mirror: Add examle rsync daemon configuration

This can be used by mirrors to offer a rsync service to end-users

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: Updated docs with additional mirror information

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #7552 from dachary/wip-14669-key-management

ceph-disk: key management support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7995 from majianpeng/bluestore-dump-onode

osd: bluestore: allow _dump_onode dynamic accept log level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: Allow _dump_onode dynamic accept log level.

Make _dump_onode dynamic accept different log level in different case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7996 from majianpeng/bluestore-fullextent-overwrite

osd: bluestore: for overwrite a extent, allocate new extent on min_alloc_size write

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: For overwrite a extent, allocate new extent rather than WAL.

This bug introduce by commit:b0b4b6de362f0edf. This commit forget this
case which overwrite(0, bluestore_min_alloc_size).
For this case it  need a new extent rather than WAL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7960 from SUSE/wip-docdev-mon

doc/dev: continue writing Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: Continue writing Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7986 from liewegas/wip-xxhash

xxhash: use clone of xxhash.git; add .gitignore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xxHash: add .gitignore for build artifacts

Signed-off-by: Sage Weil <sage@redhat.com>

.gitmodules: use github.com/ceph/xxHash clone

This lets us add our own commits, if necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: refresh image if required before replaying journal ops

Fixes: #14908
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/release-notes: v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7966 from dachary/wip-release-notes

release-notes: draft v10.0.4 release notes

Merge pull request #7880 from SUSE/wip_keyring_permisions_for_mon_deamon_14950

mon: fix keyring permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7121 from efirs/ef_eversion_t_no_sprintf

osd: Replace snprintf with faster implementation in eversion_t::get_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7141 from jecluis/wip-mon-clockskew

mon: make clock skew checks sane

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7899 from liewegas/wip-osd-markdown

test/osd: Relax the timing intervals in osd-markdown.sh

Merge pull request #7900 from liewegas/wip-bluefs-alignment

osd: bluefs: fix alignment for odd page sizes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7904 from liewegas/wip-log-delete

log: use delete[]

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

journal: possible race condition during fetch playback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: fix errors introduced by rebase to master

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: clean up playback notification handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated test cases for librbd journal tag allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: properly handle tag transition

Now that the tag concept has been re-used for delineating epochs for
librbd, we need playback to properly handle the cases where the active
playback tag abruptly ends and a newer tag is inserted in the first
splay offset object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: additional debug messages surrounding commit updates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improved debugging for journal client / tag processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate new journal tag when acquiring exclusive lock

The journal tag tracks a write epoch within the image, so before
allowing writes to proceed, ensure the journal entries will be
stored under a new epoch.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add tag handling to journal state machine

The journal will not retrieve the tag class for the image within
the journal in addition to the most recently allocated tag. Also
added helper methods to allocate new tags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to retrieve in-memory client data

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: local pool id no longer stored in client registration

The remote pool name should match the remote pool name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update journal client and tag data structures

The mirror peer now holds remote's image id as well as a collection
of bootstrap sync points.  The tag now contains uuids to map back
to the owning pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate new uuid when enabling mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: support for uuid to represent a mirrored pool

This uuid will be used as the client id for remote peer image
journals.  This will allow the uuid to be also utilized within
the librbd journal tag for tracking epoch ownership.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7742 from ErwanAliasr1/evelu-spec

Adding build requires

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

common/strtol.cc: fix the coverity warnings

* promote the compared types properly to address the signed/unsigned
  comparison warnings. this also fixes the potential problems of
  slicing a compared type down to a "smaller" type before the
  comparison.
* check for the width of resulting type and shift bits caused by SI
  prefix to avoid the -Wshift-count-overflow warnings. this again
  is a potential issue, as shifting n bits of an integer of m bits
  width, where n >= m, leads to undefined behaviour.
* add a test for the 2nd fixed issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: integrate single thread pool for all processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: own the lifecycle of the journaler's threads

This is the first step in merging the journal threads into the
librbd singleton thread pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: use provided work queue and timer

This avoids the need to open two threads per journaler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: pass metadata instead of finisher to FutureImpl

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: skip truncate request if object does not exist

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

scripts: ceph-release-note rewrite PR title case

Signed-off-by: Loic Dachary <loic@dachary.org>

release-notes: draft v10.0.4 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7959 from kmroz/wip-add-rgw-to-stopsh

script: add missing stop_rgw variable to stop.sh script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-damage-table' of git://github.com/jcsp/ceph into greg-fs-testing

Conflicts:
src/mds/MDCache.cc

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'wip-scrub-repair' of git://github.com/ukernel/ceph into greg-fs-testing

https://github.com/ceph/ceph/pull/6682

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge branch 'master' of github.com:ceph/ceph

Merge branch 'wip-conf-root' of https://github.com/liewegas/ceph

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/common/config.cc

Merge pull request #7914 from dillaman/wip-tests-template-specialization

test: correct librbd errors discovered with unoptimized cmake build

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

packaging: Moving Cython into distro specific area

To get the spec file into a consistent state, let's move Cython into the distro
specific are of the spec file.

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding redhat-rpm-config

Some part of the rados compilation process is expecting redhat-hardened-cc1.
That's surely because of the FORTIFY compilation flag.

redhat-hardened-cc1 is part of the redhat-rpm-config so let's put it as a build
dependency.

The typical output when this issue is occuring :

cd ./pybind/rados; CPPFLAGS="-iquote \/ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=\"/usr/local/lib\" -DCEPH_PKGLIBDIR=\"/usr/local/lib/ceph\" -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT    " CFLAGS="-iquote \/ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith  -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong  -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -g -O2" LDFLAGS="-L\/ceph/src/.libs -Wl,--as-needed  -Wl,-z,relro -Wl,-z,now  -latomic_ops " CYTHON_BUILD_DIR="/ceph/src/build" /bin/python ./setup.py build \
--build-base /ceph/src/build \
--verbose
Compiling rados.pyx because it changed.
Cythonizing rados.pyx
  CXXLD    libradosstriper.la
  CXXLD    librbd.la
  CXXLD    librgw.la
running build
running build_ext
building 'rados' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -iquote /ceph/src/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -fno-strict-aliasing -fsigned-char -rdynamic -O2 -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=2 -fexceptions --param=ssp-buffer-size=4 -fPIE -fstack-protector-strong -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -g -O2 -iquote /ceph/src/include -D__CEPH__ -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -D_GNU_SOURCE -DCEPH_LIBDIR=/usr/local/lib -DCEPH_PKGLIBDIR=/usr/local/lib/ceph -DGTEST_USE_OWN_TR1_TUPLE=0 -D_REENTRANT -fPIC -I/usr/include/python2.7 -c /ceph/src/build/rados.c -o /ceph/src/build/temp.linux-x86_64-2.7/ceph/src/build/rados.o
gcc: erreur: /usr/lib/rpm/redhat/redhat-hardened-cc1: Aucun fichier ou dossier de ce type
error: command 'gcc' failed with exit status 1

Signed-off-by: Erwan Velu <erwan@redhat.com>

packaging: Adding btrfs build require

When doing a 'make check', 'qa/workunits/ceph-helpers.sh' does expect the 'btrfs'
command to be available. If not, that make many tests failing.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #7628 from liewegas/wip-journal-block-size

os/filestore/FileJournal: set block size via config option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7705 from renhwsky/renhw-wip-pgmonitor-get_rule_avail

mon/pgmonitor: use appropriate forced conversions in get_rule_avail

Merge pull request #7775 from branch-predictor/bp-bl-eradication

os/ObjectStore: implement more efficient get_encoded_bytes()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7845 from aiicore/pthread_setname_fix

common: set thread name from correct thread

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7850 from jupiturliu/kstore-latency

osd: kstore: latency breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7861 from tchaikov/wip-14910

qa/workunits/cephtool/test.sh: wait longer in ceph_watch_start()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7863 from branch-predictor/bp-faster-name-gen

common/obj_bencher.cc: faster object name generation

Reviewed-by: Sage Weil <sage@redhat.com>

stop: Add missing stop_rgw variable

Signed-off-by: Karol Mroz <kmroz@suse.com>

Merge pull request #7954 from SUSE/wip-docdev

doc/dev: various refinements

Reviewed-by: Loic Dachary <ldachary@redhat.com>