options.cc: document rgw config options

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 79e58a346e743b95b4435fdc72f2fd7ac01e3f1e)

common/options: use user-defined literals for sizes

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6788ea0e9a51217cd56b282c40533a8971a36530)

common/options: use user-defined literals for default intervals

for better readablility

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3704fe283b606737533d62a082337381b95bdbd9)

common/options: use user-defined literals for default sizes

for better readablity.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d9b38a1d6172a43e7ebb755be5cf044f19d7035d)

Merge pull request #18908 from smithfarm/wip-luminous-fixup

qa/suites/fs/basic_functional/clusters: more osds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #18910 from ceph/backport-18882

luminous: ceph-volume: add functional tests for simple, rearrange lvm tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional update lvm/tox.ini paths after moving it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7d787849556788961155534039886aedfcdb2a88)

ceph-volume tests.functional move top level tox.ini to lvm/tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b14b96d3bf6c8326cfc91e657161ffab3b8c2b4)

ceph-volume tests.functional move xenial to lvm/xenial

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 00d576d266b63080129e58482bd6fba62534ee97)

ceph-volume tests.functional lvm move xenial/bluestore to xenial/bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d5ec7d1c58141f4593ed9d1e4501c9f3be6d9755)

ceph-volume tests.functional lvm move xenial/create to xenial/filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c504b8781b3a31475280a032cd75c9e1eaa6d8c9)

ceph-volume tests.functional move centos7 to lvm/centos7

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc1da3fd0aeafe59a82343fb16fe1fcea66ae3eb)

ceph-volume tests.functional lvm move bluestore to bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit aec1a80a3e1e61720f494980711808bfee783a03)

ceph-volume tests.functional lvm move create to filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d6905f3f620d4c733835e21a275d4127cdcb769b)

ceph-volume tests.functional simple xenial bluestore scenario

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f0fdc975951a2a8af31bbaa3abd2b5d807a0add)

ceph-volume tests.functional simple xenial filestore scenario

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7452e1b6f9b82c2ef62bae19c632e81039638ba5)

ceph-volume tests.functional simple tox.ini update for xenial scnearios

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 76ed6a8f2ecab7c58def0453ffed9d8a997e0d5b)

ceph-volume tests.functional simple tox.ini addition for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 02154e445b87a8fdcabbd699a2532833764140b5)

ceph-volume tests.functional add initial files for simple bluestore activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3320f28e799e1d77d1ea69be9ced603bb69cc5fa)

ceph-volume tests.functional add initial files for simple filestore activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9d18bae52ad77be0e64ef105c33a019446b6f1cf)

ceph-volume tests.functional create a specific test.yml for simple activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 37bde0d3b37a87c456438b0edc0cb9e2dcd39120)

ceph-volume tests.functional create a separate tox.ini environ for
`simple`

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89757ad12e6938fe2bf997e0f21a5ffac082e30b)

Merge pull request #18907 from ceph/luminous-wip-bz1498200

luminous: ceph-volume: support GPT and other deployed OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa/suites/fs/basic_functional/clusters: more osds

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 998122c324047c96c16d60d7fbb3a21f4dd63c56)

doc/ceph-volume include the top level systemd.rst in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e2f281c115c15e820588b478214c3600ff97470f)

ceph-volume simple.activate add --file in help docstring as an example

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ff60592897b63581ebce372b3f6ffa0f10d2fe5a)

doc/ceph-volume activate has a --file option for json files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 29913951b71d68b3f3f9b9fa076f309c0252da36)

ceph-volume simple.activate enable the osd when activating

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 366fb8c43af54e9c92f4e95d9a0582c1fc378dbc)

ceph-volume systemd.systemctl always force symlink when masking ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb470622fd4f13fafe9cb2f38fb25fd1fbd8d89c)

ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c954c913395557bcd89f25b0496c4428e6109494)

ceph-volume simple.activate only check for missing id and uuid if file is not passed in

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 826c2709dd5ab0fbf6eb419a735d15e718a7712c)

ceph-volume simple.activate allow to pass a file to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bd3498f3eabf07637cb936e6c036d1d8d75415b0)

ceph-volume tests.util add tests for OSDPath validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d7fc26ab4e2cebe485647510119ea300d47db184)

ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cda47c247b929401a7dce3f1ef70371ea80eeceb)

ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bad2f62b34243addc854abf691854b886aecdc19)

ceph-volume simple.scan if mounted device cannot be detected, raise an error

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6b2fc4c0ba63e4bd51774ea09cc6542738128621)

ceph-volume simple.activate remove str formatting from logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 230faf04a185c0b95c8ad497ddd5501dc70c1409)

ceph-volume util.arg_validators always convert the path to an absolute path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b5acf4d720b5395ae4ccbf007c94c99610e40089)

ceph-volume util.arg_validators require superuser privs to check paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 376dad74159532f901d0044f95330fc02f93bd25)

ceph-volume simple.scan skip directories to scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6bc1e9ee0b8d7bc475f218ae3dd8c6c7fe5a983a)

ceph-volume tests.devices.simple add activate CLI unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cde0d517563b537319d06c758b93e421cdabe59d)

ceph-volume tests.devices add help menu unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dcc7c72196a327f9501045f025761b228b5379df)

doc/ceph-volume systemd describe the interaction with 'simple' activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7071eeaeb5d77c6ab6e7a24f1b630cb0cf8a8e7c)

doc/ceph-volume index initial description of simple

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f9ee51f9feadb6d0f91b76a2b2ba8a24f1523a77)

doc/ceph-volume lvm add back a system.rst with just the lvm portion of it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f872cd29b93e02b77172f7b382c4746af9ba845d)

doc/ceph-volume make systemd.rst generic enough to be related to other sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cf4e32a0dc3ab4f55fe7857b51afbc63a638ec66)

doc/ceph-volume move lvm/systemd.rst to top level, making it generic

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0672dd608d4601a936d703ea5ba6edc14a161d8e)

doc/ceph-volume simple.activate initial take on documenting activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 80c796bd1c43125f5678710b494b03c486b3fce6)

doc/ceph-volume lvm update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 53420a4756a5e4fa6b0f2aa74d6ad1e2919f1954)

doc/ceph-volume lvm.activate update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7fc0e7ebcae06f28f6ed08c0db14075ada34e8a7)

doc/ceph-volume lvm.systemd update anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 01a8b7e6a45f6fd85bf53d3cab15acc81d0bae1b)

doc/ceph-volume document the scan functionality

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b30a3eaf17a20a1273724b2cfcaf0a7b36eb08f6)

doc/ceph-volume update the index to include the simple sub command and files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d7f85c64e4b409ea95e30ccb5a1388278843c478)

ceph-volume tests.simple add tests for trigger parsing from systemd

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0e2620234e82abf85274b3c08b39cdf76430ce48)

ceph-volume tests.simple create etc_path scan tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 90ebc4e1ffa7b3273787d708bb2c05180530b5d1)

ceph-volume tests.simple add checks for get_contents in scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 647bfe0883d2bb05b4ff7f99894075df3e73d2d5)

ceph-volume tests.util add binary verification tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 827aa57557561355dd4068e26801b5f1ed62f295)

ceph-volume tests create a fixture for temporary files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 23da3445718c01f71e2a5748fb4fc1c5fdc85ca5)

ceph-volume systemd.systemctl helpers to mask ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3da23fa4caea8c9dc91da3eee5b5acc1860bf242)

ceph-volume devices include simple at the module import level

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 027a09b3ea6bac416ff632a69a53393badd51708)

ceph-volume main add simple to the sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 94579c77ffe4d5ba2d457ad3981a2b9ed232a90f)

ceph-volume util add an OSDPath argument validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a826fdf4a2ed9ff704fb823d05d3cb706ba89e38)

ceph-volume simple.system add a helper to check if a file is a binary or not

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7bf8676045b4f966b60cdadf4d5af1c89fca69ae)

ceph-volume util.system add a context manager for temporary mounting/unmounting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6394cdd41943e3cda6867ecc59ab835cb2151707)

ceph-volume simple.scan initial take on directory/device scanning

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0e54b9be621cc0969a3bbc972d1870080ccdae77)

ceph-volume simple.trigger enable systemd triggering for simple setups

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cbc0c0f775e96a5b0ab5d5bc141acebe4cc48373)

ceph-volume simple.activate initial take on activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5a97374605408d18e9be29bcc051ae7b913d1675)

ceph-volume simple add __init__ for the module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 285e5bdd28a42d98418ca90ad0cdec6bd87849db)

ceph-volume simple create the initial, main module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f050502425a1993ae8edfdb5785b24129cffeeba)

Merge pull request #18801 from linuxbox2/luminous-rgw-readdir2

rgw_file: implement variant offset readdir processing -- pullup 2 of 2

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18734 from shinobu-x/wip-21648-luminous

luminous: mgr[zabbix] float division by zero

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #18851 from tchaikov/wip-18759-pr-luminous

mon/LogMonitor: "log last" should return up to n entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18741 from shinobu-x/wip-22019-luminous

luminous: tests: "ceph osd create" is not idempotent

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/LogMonitor: "log last" should return up to n entries

limit the # of returned entries to "num", and backoff the start iterator
by one if it reaches the rend().

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit de14103f5201da69b360d9b01e249896d59376b3)

rgw_file: implement variant offset readdir processing

Introduce new rgw_readdir2(...), which in which continues
from an arbitrary dirent name, which presumably has been
seen in a prior partial enumeration.

Add single-file unit test for READDIR cases, librgw_file_marker.cc.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit a2c34c597371809bd8fd40ee1fc6b8e6a5145609)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18800 from linuxbox2/luminous-rgw_file-fsid-mount

Luminous rgw file fsid mount -- pullup 1 of 2

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18767 from smithfarm/wip-22020-luminous

luminous: multisite: race between sync of bucket and bucket instance metadata

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #18794 from theanalyst/wip-18709

luminous: rgw: Stale bucket index entry remains after object deletion

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #18726 from shinobu-x/wip-luminous-22035

luminous: Spurious ceph-mgr failovers during mon elections

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18738 from shinobu-x/wip-21547-luminous

luminous: ceph-mgr gets process called "exe" after respawn

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18727 from shinobu-x/wip-luminous-22034

luminous: key mismatch for mgr after upgrade from jewel to luminous(dev)

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18723 from shinobu-x/wip-luminous-22023

luminous: osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

Merge pull request #18737 from shinobu-x/wip-21549-luminous

luminous: the dashboard uses absolute links for filesystems and clients

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18736 from shinobu-x/wip-21638-luminous

luminous: dashboard OSD list has servers and osds in arbitrary order

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18728 from shinobu-x/wip-luminous-22032

luminous: dashboard barfs on nulls where it expects numbers

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18699 from shinobu-x/wip-luminous-21538

luminous: upmap does not respect osd reweights

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18719 from tchaikov/wip-pr-18373-luminous

luminous: osd,mon: add max-pg-per-osd limit

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix the desc of "osd max pg per osd hard ratio"

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d940f6220346ad21dfc4f351c57389e8430c0257)

mon: change "mon_pg_warn_min_per_osd" to uint64_t

* this silences a warning from -Wsign-compare.
* also switch all its user to the new-style option

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c3cd0c13a1585388df0f17ab53c170e39c711615)

Merge pull request #18702 from liewegas/wip-pr-18670-luminous

luminous: qa/tasks/thrashosds: set min_in default to 4

Merge pull request #18596 from smithfarm/wip-21955-luminous

luminous: tests: add EC data pool to testing

Reviewed-By: Patrick Donelly <pdonelly@redhat.com>

rgw_file: introduce rgw_mount2 with a bucket name parameter

This explicitly allow mount a specified bucket in librgw.
Originally, mounting a bucket is implemented like a sub-directory
mount in nfs-ganesha FSAL_RGW with lookup_path.
With this change, we allow root_fh to points exactly to the root
of a mounted fs instance, which is a bucket or "/" rather than
always let root_fh points to "/".

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit 860716e17e840df11a7e1d8b6ca0c1ee694b038b)

rgw_file: new fsid for fs instance

We use an in-memory fs_inst_counter for fsid currently, but it
inevitably cause nfs automount problem across a nfs-ganesha crash.
So here we relate the fsid to the s3 uid with a hash.
With this, fsid remains consistent across all fs instances and
across crashes.

We should be able to upgrade from old fsid to new fsid with
update_fh().

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit 178b547e83a2c0eeb024bbe4ca2ff8d8e7d50572)

Merge pull request #18628 from batrick/i21953

luminous: mds: sanitize mdsmap of removed pools

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #18626 from dillaman/wip-21968

luminous: cls/journal: fixed possible infinite loop in expire_tags

Reviewed-By: Mykola Golub <trociny@gmail.com>

Merge pull request #18688 from liewegas/wip-balancer-luminous

luminous: mgr/balancer: backport crush-compat work from master

Reviewed-By: John Spray <jspray@redhat.com>

rgw: fix BZ 1500904, Stale bucket index entry remains after object deletion

We have a race condition:

 1. RGW client #1: requests an object be deleted.
 2. RGW client #1: sends a prepare op to bucket index OSD #1.
 3. OSD #1:        prepares the op, adding pending ops to the bucket dir entry
 4. RGW client #2: sends a list bucket to OSD #1
 5. RGW client #2: sees that there are pending operations on bucket
                   dir entry, and calls check_disk_state
 6. RGW client #2: check_disk_state sees that the object still exists, so it
                   sends CEPH_RGW_UPDATE to bucket index OSD (#1)
 7. RGW client #1: sends a delete object to object OSD (#2)
 8. OSD #2:        deletes the object
 9. RGW client #2: sends a complete op to bucket index OSD (#1)
10. OSD #1:        completes the op
11. OSD #1:        receives the CEPH_RGW_UPDATE and updates the bucket index
                   entry, thereby **RECREATING** it

Solution implemented:

At step #5 the object's dir entry exists. If we get to beginning of
step #11 and the object's dir entry no longer exists, we know that the
dir entry was just actively being modified, and ignore the
CEPH_RGW_UPDATE operation, thereby NOT recreating it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit b33f529e79b74314a2030231e1308ee225717743)

Merge pull request #18569 from kmroz/wip-21939-luminous

luminous: list bucket which enable versioning get wrong result when user marker

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #18591 from cbodley/wip-21938

luminous: rgw: fix error handling in ListBucketIndexesCR
(approved in irc & standups)

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18539 from linuxbox2/luminous-baixuyeyu-refcnt

luminous: baixuyeyu refcnt

Reviewed-By: Casey Bodley <cbodley@redhat.com>