mon/OSDMonitor: prevent FLAG_CREATING from getting set pre-nautilus

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: disallow pg_num changes while CREATING flag is set

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: set POOL_CREATING flag until initial pool pgs are created

Set the flag when the pool is created, and clear it when the initial set
of PGs have been created by the mon.  Move the update_creating_pgs()
block so that we can process the pgid removal from the creating list and
the pool flag removal in the same epoch; otherwise we might remove the
pgid but have no cluster activity to roll over another osdmap epoch to
allow the pool flag to be removed.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add pg_pool_t FLAG_POOL_CREATING

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: introduce last_force_resend_prenautilus

Previously, we renamed the old last_force_resend to
last_force_resend_preluminous and created a new last_force_resend for
luminous+.  This allowed us to force preluminous clients to resend ops
(because they didn't understand the new pg split => new interval rule)
without affecting luminous clients.

Do the same rename again, adding a last_force_resend_prenautilus (luminous
or mimic).

Adjust the OSD code accordingly so it matches the behavior we'll see from
a luminous client.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: merge_from helper

When merging two logs, we throw out all of the actual log entries.
However, we need to convert them to dup ops as appropriate, and merge
those together.  Reuse the trim code to do this.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: no cache agent or snap trimming during premerge

The PG is quiesced; not background activity.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: notify mon when pending PGs are ready to merge

When a PG is in the pending merge state it is >= pg_num_pending and <
pg_num.  When this happens quiesce IO, peer, wait for activate to commit,
and then notify the mon that we are idle and safe to merge.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add simple controller to adjust pg[p]_num_actual

This is a pretty trivial controller.  It adds some constraints that were
obviously not there before when the user could set these values to anything
they wanted, but does not implement all of the "nice" stepping that we'll
eventually want.  That can come later.

Splits:
- throttle pg_num increases, currently using the same config option
(mon_osd_max_creating_pgs) that we used to throttle pg creation
- do not increase pg_num until the initial pg creation has completed.

Merges:
- wait until the source and target pgs for merge are active and clean
before doing a merge.

Adjust pgp_num all at once for now.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: MOSDPGReadyToMerge to complete a pg_num change

This message allows pg_num to be decremented (once the final PGs are
ready).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: allow pg_num to adjusted up or down via pg[p]_num_target

The CLI now sets the *_target values, imposing only the subset of constraints that
the user needs to be concerned with.

new "pg_num_actual" and "pgp_num_actual" properties/commands are added that allow
the underlying raw values to be adjusted.  For the merge case, this sets
pg_num_pending instead of pg_num so that the OSDs can go through the
merge prep process.

A controller (in a future commit) will make pg[p]_num converge to pg[p]_num_target.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: make pg merge an interval boundary

Both the merge itself *and* the pending merge are interval transitions.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add pg_t::is_merge() method

This checks if we are a merge *source*, and if so, who the parent (target)
will be.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add pg_num_pending to pg_pool_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow multiple threads to block on wait_min_pg_epoch

Signed-off-by: Sage Weil <sage@redhat.com>

osd: restructure advance_pg() call mechanism

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: prune merged pgs

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: track pgs by state for each pool

We had this globally, but it's useful to have the per-pool breakdowns.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: allow split_bits to decrease (merge)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix osr_drain before merge

We need to make sure the deferred writes on the source collection finish
before the merge so that ops ordered via the final target sequencer will
occur after those writes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow reuse of osr from existing collection

We try to attach an old osr at prepare_new_collection time, but that
happens before a transaction is submitted, and we might have a
transaction that removes and then recreates a collection.

Move the logic to _osr_attach and extend it to include reusing an osr
in use by a collection already in coll_map.  Also adjust the
_osr_register_zombie method to behave if the osr is already there, which
can happen with a remove, create, remove+create transaction sequence.

Fixes: https://tracker.ceph.com/issues/25180
Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: (re)implement merge

Merging is a bit different then splitting, because the two collections
may already be hashed at different levels.  Since lookup etc rely on the
idea that the object is always at the deepest level of hashing, if you
merge collections with different levels that share some common bit prefix
then some objects will end up higher up the hierarchy even though deeper
hashed directories exist.

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: add _merge_collections post-check

Signed-off-by: Sage Weil <sage@redhat.com>

os: implement merge_collection

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add merge_collection operation to Transaction

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23634 from cbodley/wip-21154

rgw: RGWRadosGetOmapKeysCR takes result by shared_ptr

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRadosGetOmapKeysCR takes result by shared_ptr

Fixes: http://tracker.ceph.com/issues/21154
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23920 from cbodley/wip-rgw-cr-rados-fixes

rgw multisite: async rados requests don't access coroutine memory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #23959 from rubenk/doc-remove-unknown-option-from-manpage

doc: remove deprecated 'scrubq' from ceph(8)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23931 from cyx1231st/wip-msgr-test

tests: fix to check server_conn in MessengerTest.NameAddrTest

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: fix to check server_conn in MessengerTest.NameAddrTest

Signed-off-by: Yingxin <yingxin.cheng@intel.com>

Merge pull request #23944 from ceph/wip-s3a-update-mirror

qa/tasks: update mirror link for maven

Merge pull request #23963 from alfredodeza/wip-rm35535

ceph-volume:  batch tests for mixed-type of devices

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch use 'ceph' as the cluster name with filestore

Custom cluster names are currently broken on ceph-volume, should get
addressed with http://tracker.ceph.com/issues/27210 which is out of
scope for these changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update filestore xenial test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update bluestore xenial test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update filestore centos7 test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional update bluestore centos7 test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api use double -f flags when calling pvremove

Fairly destructive, just like everything else when zapping a device.
This is required in the case of double UUIDs detected, something that
surfaced when testing with a loop device to create an nvme (the loop
device ends up with the same UUID as the nvme).

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional remove duplicate test playbooks

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional setup playbook for nvme devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional single (test) playbook to zap and redeploy batch operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional create a noop playbook to allow symlinking always to setup.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add mixed-type scenarios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #23673 into master

* refs/pull/23673/head:
qa: automate distro/kernel matrix for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tested-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests/functional add CentOS7 bluestore mixed-type scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional add CentOS7 filestore mixed-type scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa: automate distro/kernel matrix for kclient

It's no longer necessary to pass `-k testing` to teuthology-suite. We're also
now regularly testing RHEL 7.5 kernel in upstream testing.

This work is prep for eventually integrating kclient into fs.

Fixes: http://tracker.ceph.com/issues/26995
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: remove deprecated 'scrubq' from ceph(8)

The option was removed in a30cbe9adea34782be8699c010d4281d245feb6b

Fixes: http://tracker.ceph.com/issues/35813
Signed-off-by: Ruben Kerkhof <ruben@rubenkerkhof.com>

Merge pull request #23846 from liewegas/wip-mgr-config-fixes

mon: fix mgr module config option handling

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23836 from sepia-liu/wip-cephfuse-doc

doc/cephfs: fixup add/remove mds docs

Merge pull request #23911 from votdev/cleanup

mgr/dashboard: Remove useless code

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23948 from libingyang-zte/master

doc: Fix Spelling Error of Radosgw

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23947 from tchaikov/wip-test-msgr-include

test/msgr: add missing #include

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23926 from Gangbiao/lgb-devel

doc: Document correction

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: Fix Spelling Error of Radosgw

Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>

Merge pull request #23850 from liewegas/wip-22544

osdc/Objecter: fix split vs reconnect race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #23557 from liewegas/wip-cmd-getval

common/cmdparse: cmd_getval_throws -> cmd_getval

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/msgr: add missing #include

it's a regression introduced by #23930

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: update mirror link for maven, the original mirror no longer exists

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #23851 into master

* refs/pull/23851/head:
mds: print is_laggy message once

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23890 into master

* refs/pull/23890/head:
msg: lower verbosity on normal event

Reviewed-by: Sage Weil <sage@redhat.com>

mds: print is_laggy message once

Beacon::is_laggy gets called frequently which causes the debug log to get
spammed with messages.

Steps to reproduce:
- setup vstart cluster
- start IO from client (e.g. `cp -av /usr /mnt`)
- find connection to mon by the mds
  $ ss -tnp | grep ceph-mds
    ...
    ESTAB      0      0      127.0.0.1:58988              127.0.0.1:40928               users:(("ceph-mds",pid=13095,fd=37))
- block the connection using iptables on the MDS node:
  $ sudo iptables -I INPUT -p tcp --sport 58988 --dport 40928 -j DROP
- verify MDS log using:
  $ tail -f mds.a.log | grep beacon
    ...
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.672 7f2908a9d700  1 mds.beacon.a is_laggy 19.8876 > 15 since last acked beacon
    2018-09-05 19:21:05.784 7f2905a97700  5 mds.beacon.a Sending beacon up:active seq 114
    2018-09-05 19:21:05.784 7f2905a97700  1 -- 127.0.0.1:6813/2277776624 --> 127.0.0.1:40929/0 -- mdsbeacon(14162/a up:active seq 114 v7) v7 -- 0x7d1e7e2a80 con 0
    2018-09-05 19:21:05.784 7f2905a97700 20 mds.beacon.a sender thread waiting interval 4s

Fixes: http://tracker.ceph.com/issues/35250
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23645 from jcsp/wip-mgr-unknown-error

mgr: replace "Unknown error" string on always_on

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23942 from trociny/wip-rados-rmpool

qa/workunits/rbd: replace usage of 'rados rmpool'

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: replace usage of 'rados rmpool'

This command was dropped.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #23907 into master

* refs/pull/23907/head:
doc: Fix Spelling Error of Cephfs Shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #23930 from tchaikov/wip-include-cleanup

common,mon: add implicit `#include` headers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23909 from theanalyst/doc/releases/12.2.8

doc: release notes for 12.2.8 luminous

doc: release notes for 12.2.8 luminous

- Add a special note for upgrades from earlier releases
- Add a notable section mentioning the scrub and rgw options

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #23852 from tchaikov/wip-cmake-cleanup

cmake: link compressor plugins against lib the modern way

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23925 from cfanz/wip-fix-data-sync-report

rgw-admin: fix data sync report for master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23626 from Liuchang0812/rgw-put-cors-compatible

rgw: return err_malformed_xml when MaxAgeSeconds is an invalid integer

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23835 from Songweibin/wip-import-34536

rbd: fix error import when the input is a pipe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23815 from shun-s/fix-acquire-lock-latency

librbd: fix possible unnecessary latency when requeue request

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23935 from LenzGr/bug-35686-missing-tooltip

mgr/dashboard: Added missing tooltip to settings icon

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23409 from ricardoasmarques/wip-role-management

mgr/dashboard: Role management from the UI

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23938 from dillaman/wip-qa-rbd-mkpool

qa/workunits/rbd: replace usage of 'rados mkpool'

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: replace usage of 'rados mkpool'

This command was dropped under commit 2c26fb0fe1, so use
'ceph osd pool create' instead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Role management from the UI

Fixes: https://tracker.ceph.com/issues/24447
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #23937 from tchaikov/wip-crimson-test-denc

test/crimson: do not use unit.cc as the driver of unittest_seastar_denc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/crimson: do not use unit.cc as the driver of unittest_seastar_denc

as unit.cc initializes the CephContext and all of Ceph's infratructure,
which is not necessary for the denc test. also, seastar's builtin allocator
only pre-allocates 32 << 20 bytes. it's enough for the denc test, but
not necessarily enough for create CephContext and its friends. an option is
to use seastar's app template to initialize the memory allocator properly.
another option is to avoid initializing CephContext in this test.

the latter is simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Added missing tooltip to settings icon

Fixes: https://tracker.ceph.com/issues/35686
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #23797 from votdev/fix_user_notifications

mgr/dashboard: Fix notifications in user list and form

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mon/MonClient.h: include necessary header

MonClient references LockPolicy, so it should include
common/lock_policy.h .

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/TrackedOp.h: include spinlock.h as this header uses it

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,crush,mds,messages,osd: add missing #include

they rely on dout.h to do this, but they should not.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23816 from cyx1231st/wip-seastar-msgr-fix

crimson/net: miscellaneous fixes to seastar-msgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Fix notifications in user list

The notification message title and text were inverted.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23912 from tchaikov/wip-install-deps-openldap

install-deps: s/openldap-client/openldap24-client/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

doc: Document correction
    "doc/ceph-volume/lvm/prepare.rst":  Remove duplicated words
    "doc/radosgw/cloud-sync-module.rst": Remove duplicated words

Signed-off-by: Gangbiao Liu <liugangbiao@cmss.chinamobile.com>

rgw-admin: fix data sync report for master zone

Signed-off-by: cfanz <songxinying@sensetime.com>

Merge pull request #23918 from dzafman/wip-fix-assert

osd: Change assert() to ceph_assert() missed in the transition

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23759 from cbodley/wip-rgw-curl-timeout

rgw: raise default rgw_curl_low_speed_time to 300 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23751 from joke-lee/wip-copy-response

rgw: etag in rgw copy result response body rather in header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23749 from joke-lee/wip-etag-copy-part

rgw: fix Etag error in multipart copy response

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23646 from ashitakasam/get_cors_err_code

rgw: the error code returned by rgw is different from amz s3 when getting cors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23590 from yaozongyou/fix-rgw-deadlock

rgw: fix deadlock on RGWIndexCompletionManager::stop

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWAsyncPutSystemObjAttrs does not access coroutine memory

also exposes optional objv_tracker to RGWSimpleRadosWriteAttrsCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWAsyncPutSystemObj does not access coroutine memory

Signed-off-by: Casey Bodley <cbodley@redhat.com>