From: Peter Dillinger <peterd@fb.com>
Date: Fri, 13 Mar 2020 20:36:45 +0000 (-0700)
Subject: Use an Amazon S3 bucket for downloading deps (#6526)
X-Git-Tag: v6.8.1~1
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=d7b6149563a43d6423aa570a896dab40256cf3cb;p=rocksdb.git

Use an Amazon S3 bucket for downloading deps (#6526)

Summary:
After we had a lot of failures with maven.org downloads, we
wanted an alternative location for downloading binary dependencies.
Hosting them through github would have been good in terms of
organizational and network dependencies, but that approach seems to be
awkward (fake releases, so would need a 'rocksdb-deps' repo) and
strangely complicated for Facebook policy on open source repositories.

This commit moves the downloads (that are not officially hosted by
others on github) from my personal rocksdb fork to an S3 bucket owned
by the Facebook RocksDB AWS account. Facebook employees can access
this through an internal tool, and we should be able to grant permission
to outside collaborators.

Assuming this works out, I will back-port to older branches to stabilize
their CI testing as well.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6526

Test Plan: CI

Differential Revision: D20430130

Pulled By: pdillinger

fbshipit-source-id: df52394a65e0a57942db3039bdaade8a4d520cb2
---

diff --git a/.travis.yml b/.travis.yml
index feca073a..300d729e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -81,7 +81,9 @@ install:
       sudo apt-get install -y mingw-w64 ;
     fi
   - if [[ "${JOB_NAME}" == cmake* ]] && [ "${TRAVIS_OS_NAME}" == linux ]; then
-      mkdir cmake-dist && curl --silent --fail --show-error --location https://github.com/Kitware/CMake/releases/download/v3.14.5/cmake-3.14.5-Linux-x86_64.tar.gz | tar --strip-components=1 -C cmake-dist -xz && export PATH=$PWD/cmake-dist/bin:$PATH;
+      CMAKE_DIST_URL="https://rocksdb-deps.s3-us-west-2.amazonaws.com/cmake/cmake-3.14.5-Linux-$(uname -m).tar.bz2";
+      TAR_OPT="--strip-components=1 -xj";
+      mkdir cmake-dist && curl --silent --fail --show-error --location "${CMAKE_DIST_URL}" | tar -C cmake-dist ${TAR_OPT} && export PATH=$PWD/cmake-dist/bin:$PATH;
     fi
   - if [[ "${JOB_NAME}" == java_test ]]; then
       java -version && echo "JAVA_HOME=${JAVA_HOME}";
diff --git a/java/CMakeLists.txt b/java/CMakeLists.txt
index b9ab8177..04d592bc 100644
--- a/java/CMakeLists.txt
+++ b/java/CMakeLists.txt
@@ -316,11 +316,9 @@ endif()
 if (DEFINED CUSTOM_DEPS_URL)
   set(DEPS_URL ${CUSTOM_DEPS_URL}/)
 else ()
-  # This is a URL for artifacts from a "fake" release on pdillinger's fork,
-  # so as not to put binaries in git (ew). We should move to hosting these
-  # under the facebook account on github, or something else more reliable
-  # than maven.org, which has been failing frequently from Travis.
-  set(DEPS_URL "https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps")
+  # Using a Facebook AWS account for S3 storage. (maven.org has a history
+  # of failing in Travis builds.)
+  set(DEPS_URL "https://rocksdb-deps.s3-us-west-2.amazonaws.com/jars")
 endif()
 
 if(NOT EXISTS ${JAVA_JUNIT_JAR})
diff --git a/java/Makefile b/java/Makefile
index c6acd416..155db694 100644
--- a/java/Makefile
+++ b/java/Makefile
@@ -217,11 +217,9 @@ ifneq ($(DEBUG_LEVEL),0)
 	JAVAC_ARGS = -Xlint:deprecation -Xlint:unchecked
 endif
 
-# This is a URL for artifacts from a "fake" release on pdillinger's fork,
-# so as not to put binaries in git (ew). We should move to hosting these
-# under the facebook account on github, or something else more reliable
-# than maven.org, which has been failing frequently from Travis.
-DEPS_URL?=https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps
+# Using a Facebook AWS account for S3 storage. (maven.org has a history
+# of failing in Travis builds.)
+DEPS_URL?=https://rocksdb-deps.s3-us-west-2.amazonaws.com/jars
 
 clean: clean-not-downloaded clean-downloaded