We Make Service Mesh Easy

Just provide a few details and you’ll be ready to use Aspen Mesh

F5 NETWORKS

BETA PRODUCT AGREEMENT

BY ACCEPTING THIS BETA PRODUCT AGREEMENT (THE “AGREEMENT”), REGISTERING FOR OR USING AN ACCOUNT (“ACCOUNT”) FOR THE BETA PRODUCT, OR BY ACCESSING THE BETA PRODUCT, YOU (1) agree to this BETA ServiceS Agreement (the “Agreement”) and the Aspen MESH portal terms of use on behalf of yourself and the Customer identified in the BETA PORTAL (the “Customer” or “YOU”), (2) represent and warrant to F5 (“f5,” “We, or “US”) that you are authorized to accept this Agreement on behalf of the Customer, and (3) agree that such Customer will be responsible for the acts and omissions of any individuals or other users who register for, access or use the BETA PRODUCT through your account. IF YOU AND/OR CUSTOMER DOES NOT AGREE TO ALL OF THE TERMS OF THIS AGREEMENT, DO NOT ACCEPT THIS AGREEMENT AND DO NOT ACCESS THE BETA PRODUCT.

All references to “F5” in this Agreement are references to the applicable F5 entity as follows: (i) if the Customer’s primary place of business is located in Europe, the Middle East or Africa (“EMEA”), the F5 entity is F5 Networks Ltd.; (ii) if the Customer’s primary place of business is located in the Asia-Pacific region (“APAC”), the F5 entity is F5 Networks Singapore Pte Ltd.; and (iii) if the Customer’s primary place of business is located in a region outside of EMEA or APAC, the F5 entity is F5 Networks, Inc.

You will provide accurate, current and complete information (including about Customer and Customer’s users) in all registration and other Account-related forms on the Beta Portal (“Customer Information”) and you will maintain the security of your username(s) and password(s). You will maintain and promptly update the Customer Information to keep it accurate, current and complete. YOU UNDERSTAND THAT ANY PERSON WITH YOUR USERNAME(S) AND PASSWORD(S) MAY BE ABLE TO ACCESS YOUR ACCOUNT (INCLUDING CUSTOMER INFORMATION AND OTHER CUSTOMER DATA (COLLECTIVELY, “CONTENT”)). YOU ACCEPT ALL RISKS OF UNAUTHORIZED ACCESS TO YOUR ACCOUNT BASED ON THE SHARING OR LOSS OF A USERNAME AND PASSWORD. You will promptly notify us if you discover or otherwise suspect any unauthorized access related to your Account or the Beta Product, including any unauthorized use or disclosure of a username or password.

  1. Definitions.

1.1.    “Beta Period” means the period in which you may use the Beta Product under the terms of this Agreement as set forth on the Beta Portal Site; and if no period is specified, until we specify otherwise. We reserve the right to terminate the Beta Period at any time upon notice provided via the Beta Portal.

1.2.    “Beta Portal” means the Aspen Mesh Beta Portal website located at https://my.aspenmesh.io.

1.3.    “Beta Product” means the F5 beta software product(s) offered through the Beta Portal for download.

 

2.      Use of Beta Product.

2.1.    License Grant. Subject to your compliance with this Agreement, we grant you a non-transferable, nonexclusive right, during the Beta Period, to use the Beta Product for your internal evaluation and testing purposes only in a non-production environment.

2.2.    Title. The Beta Product and all enhancements, modifications and improvements thereto and the associated copyrights and intellectual property rights are owned by F5 and/or its licensors. Except for the license grant provided in Section 2.1, no rights to the Beta Product are granted or conveyed to Customer by implication or otherwise. To the extent that the Beta Product incorporates third party software, your use is subject to licenses with the respective licensor(s). The protections given to F5 under this Agreement also apply to the licensor(s) of this third party software, who are intended third party beneficiaries of this Agreement.

2.3.    Restrictions.  You will only use the Beta Product in accordance with all documentation and instructions we provide. You will use the Beta Product solely for your internal testing, and you will not modify or copy the Beta Product or any portion thereof. You will not demonstrate, market, reuse, copy, modify, translate or create derivative works of the Beta Service or any portion thereof; rent, sell, lease, transfer or otherwise make available the Beta Product or any portion thereof; or use it or any portion thereof for the benefit of a third party. You may not dissemble, de-compile, reverse assemble, reverse compile or reverse engineer the Beta Product or any portion thereof, or otherwise attempt to discover any Beta Product source code or underlying proprietary information. You will not use any information in any way related to or acquired by use of the Beta Product other than in connection with your evaluation of the Beta Product for a potential commercial implementation.

2.4.    Additional Obligations.

2.4.1.   You will use and test the Beta Product during the Beta Period. We may monitor your usage of the Beta Product.

2.4.2.   You will report to F5 on the performance of the Beta Product and will provide suggestions, comments or other feedback to F5 in the format and on the timing we reasonably request with respect to improving the Beta Product (collectively, “Feedback”). We may freely use, disclose, reproduce, license, distribute, and otherwise commercially exploit the Feedback without obligation or restriction of any kind.

2.4.3.   You will promptly report to F5 any problems with the Beta Product.

3.          Term and Termination. The term of this Agreement will commence when you accept this Agreement and will continue until the completion of the Beta Period or until terminated in accordance with this Agreement. This Agreement may be terminated by any party for any reason or no reason upon written notice to another party, and in any case will expire at the end of the Beta Period. We may extend the Beta Period upon written notice to you. We may discontinue the beta program or your Account at any time, in which case this Agreement will automatically terminate at the time of such discontinuation. Upon termination, the license granted hereunder will terminate and we may terminate your access to and use of the Beta Product immediately and without notice. Upon termination for any reason, you will immediately return all documents, notes and other materials regarding the Beta Product including, without limitation, all proprietary information of F5 and all copies and extracts of the foregoing. Any terms by which their nature impose an obligation after termination will survive termination or expiration of this Agreement, including but not limited to confidentiality, disclaimer of warranties, limitation of liabilities, indemnification, and governing law.

4.          Confidentiality.  You will comply with the terms of the Nondisclosure Agreement entered into by the parties (the “NDA”), if applicable. In the event that the parties have not entered into an NDA covering the subject matter of this Agreement, you agree that any non-public information regarding the Beta Product provided or made accessible hereunder, the results of any beta testing, and the terms and content of this Agreement are part of F5’s confidential information. You will not disclose F5 confidential information to any third party other than your employees or agents who have a business need to know and who are bound by written confidentiality obligations to you not to disclose the confidential information that are substantially similar this Section 4. This obligation of confidentiality will not apply to information which you can show by contemporaneous documentation to be (i) rightfully available to the public; (ii) rightfully received by you from a third party without breach of a duty to F5; (iii) independently developed by your employees without access to the Beta Product; or (iv) rightfully known to Customer prior to first receipt from F5 while not under a contemporaneous duty of non-disclosure. You will not use any confidential information of F5 or proprietary information of F5 except for purposes of your evaluation of the Beta Product under the terms of this Agreement. You acknowledge that the Beta Product, its performance, evaluation results, features and mode of operation, as well as all internal documentation, specifications, product requirements, problem reports, analysis and performance information, benchmarks, software documents, and other technical, business, product, marketing and financial information, plans and data relating to the Beta Product are the proprietary information of F5. You will not publish or disclose to others any results of any benchmarking or other tests run on the Beta Service without our prior written consent. You will return or destroy all F5 confidential information upon our request. If you destroy the F5 confidential information, you will provide a written certification of destruction signed by an authorized officer of Customer.

5.          Representations and Warranties. You hereby warrant, represent and covenant as follows in the performance of your obligations and use of the Beta Product (by Customer and any Customer Representatives): (i), you will comply with all applicable laws, rules, and regulations of all applicable U.S. and foreign authorities; (ii) you will not infringe the proprietary or privacy rights of any third party; (iii) you will not use the Beta Product in a manner which constitutes Misuse; (iv) the information and other data (including Personal Data) that you transmit, process, and receive in connection with the use of the Beta Product provided hereunder complies and will at all times during the term of this Agreement comply with all applicable laws and do not and will not infringe the proprietary rights or privacy rights of any third parties; (v) when using the Beta Product (or allowing others to use the Beta Product including its end users) you will comply with all applicable acceptable use policies and will not cause or allow others to cause the disruption of other parties’ use or enjoyment of the Internet; and (vi) you do not currently provide services that compete with the Beta Product and will not at any time in the future use any of the Beta Product or any other confidential information of F5 for the provision of any services that compete with the Beta Product. In addition, you represent and warrant that you are not on the U.S. Treasury Department’s list of Specially Designated Nationals or the U.S. Department of Commerce’s Table of Denial Orders.

6.          Warranty Disclaimer; Warning. The parties acknowledge that the Beta Product is provided “AS IS”. Customer understands that the Beta Product has not completed F5’s full quality assurance program and may have errors and may produce unexpected results. F5 DISCLAIMS ALL WARRANTIES RELATING TO THE BETA PRODUCT, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES AS TO TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE (INCLUDING, WITHOUT LIMITATION, PREVENTION OF UNAUTHORIZED ACCESS),THAT THE BETA PRODUCT WILL BE FREE FROM ERROR OR INTERRUPTION OR FAILURE. F5 NEITHER ASSUMES, NOR AUTHORIZES ANY OTHER PERSON TO ASSUME FOR IT, ANY OTHER LIABILITY IN CONNECTION WITH THE BETA PRODUCT OR ANY INFORMATION PROVIDED IN CONNECTION THEREWITH, INCLUDING, WITHOUT LIMITATION, LIABILITY ARISING OUT OF THE OPERATION, SUPPORT, OR USE OF THE BETA PRODUCT. F5 DOES NOT WARRANT THE RESULTS OF THE BETA PRODUCT OR THAT ANY ERRORS IN THE BETA PRODUCT WILL BE CORRECTED, OR THAT THE BETA PRODUCT WILL MEET CUSTOMER’S REQUIREMENTS OR EXPECTATIONS. YOU SHOULD SAFEGUARD IMPORTANT DATA, USE CAUTION, AND NOT RELY IN ANY WAY ON THE CORRECT FUNCTIONING OR PERFORMANCE OF THE BETA PRODUCT AND/OR ACCOMPANYING MATERIALS.

7.          Limitation of Remedies and Damages. IN NO EVENT WILL F5 OR ITS LICENSORS BE LIABLE  FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE, DAMAGES INCLUDING, BUT NOT LIMITED TO LOSS OF PROFITS, REVENUE, DATA OR MACHINE USE, BUSINESS INTERRUPTION ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT, THE USE OR THE INABILITY OF THE USE OR PERFORMANCE OF THE BETA PRODUCT, THE PROVISION OF OR FAILURE TO PROVIDE BETA PRODUCT, OR ANY INFORMATION PROVIDED, REGARDLESS OF THE NATURE OF THE ACTION OR UNDERLYING LEGAL THEORY (INCLUDING UNDER ANY CONTRACT, NEGLIGENCE, STRICT LIABILITY OR OTHER THEORY), EVEN IF F5 AND/OR ITS LICENSORS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. F5 WILL NOT BE RESPONSIBLE FOR ANY MATTER BEYOND ITS REASONABLE CONTROL. IN NO EVENT WILL F5’S TOTAL AGGREGATE LIABILITY UNDER THIS AGREEMENT EXCEED ONE THOUSAND DOLLARS ($1,000). NO ACTION, REGARDLESS OF FORM, ARISING UNDER THIS AGREEMENT MAY BE BROUGHT BY CUSTOMER MORE THAN ONE YEAR AFTER THE END OF THE BETA PERIOD.

8.          Customer Acknowledgement. CUSTOMER ACKNOWLEDGES THE NATURE OF THE BETA PRODUCT, AND THAT IT MAY CONTAIN KNOWN OR UNKNOWN BUGS, DEFECTS AND ERRORS, MAY NOT FUNCTION AS INTENDED, MAY NOT FUNCTION AT THE LEVEL OF A FINAL, GENERALLY AVAILABLE PRODUCT, MAY CREATE UNFORESEEABLE EVENTS, AND MAY BE SUBSTANTIALLY MODIFIED PRIOR TO FIRST GENERAL COMMERCIAL AVAILABILITY, OR WITHDRAWN. CUSTOMER ACKNOWLEDGES THAT THE PRODUCT IS INTENDED TO BE USED ONLY IN A TESTING ENVIRONMENT, WITH TEST DATA, AND NOT FOR PRODUCTION PURPOSES. F5 HAS NO OBLIGATION TO CORRECT ANY BUGS, DEFECTS OR ERRORS IN THE BETA PRODUCT OR OTHERWISE SUPPORT OR MAINTAIN THE BETA PRODUCT. F5 reserves the right at any time not to generally commercially release the Beta Product or, even if released, to alter prices, features, specifications, capabilities, functions, release dates, general availability, or other characteristics of the Beta Product.

9.          Indemnification. Customer will defend, indemnify and hold F5, and its affiliates and their respective officers, directors, licensors, suppliers, service providers, employees, contractors and agents harmless against any claims, liabilities or expenses incurred (including reasonable attorneys’ fees), as well as amounts finally awarded in a settlement or by a court arising from any (i) breach by Customer or any Customer Representative of any representation, warranty, covenant or other obligation of this Agreement, (ii) Misuse, (iii) negligence or willful misconduct of Customer or Customer Representatives, or (iv) liability arising out of or relating to the use of the Beta Product. We will promptly notify you of any such claim or allegation, but any delay in providing such notice will not affect your obligations hereunder except to the extent that the defense of the claim is materially prejudiced by such delay. You will not agree to any settlement without our prior written approval. We will provide reasonable assistance upon request and at your sole cost.

10.        Customer Performance and Usage Data. We may use various third party software tools to measure and monitor Customer interaction with the Beta Product, and to generate reports relating to use of the Beta Product under this Agreement. All information and status reports derived from your use of the Beta Product will be our exclusive property. You will not disclose such information or reports. We may use aggregated data derived from the use of the Beta Product to support and improve F5 products and services, in the development of new features, products, tools, content, and for market research, provided that such data has been stripped of all information identifying you.

11.        General.

11.1.  Entire Agreement. This Agreement together with the Aspen Mesh Portal Terms of Use, and, if applicable, the NDA, constitutes the entire agreement between the parties pertaining to the subject matter hereof, and supersedes any and all prior proposals, negotiations, communications, and agreements, written or oral (except the NDA) previously existing between the parties pertaining to the subject matter hereof. Without limiting the generality of the foregoing, in the event of a conflict between the terms of this Agreement and any other agreement between the parties, this Agreement will govern with respect to any component of the Beta Product, including any portion thereof to the extent used as part of the Beta Product. We will not be bound by any use of any standardized form or correspondence (including any order form, purchase order, acknowledgement, shrink-wrap, boxtop, or click wrap license, or other form) containing additional or different terms. We may, at any time and in our discretion, modify this Agreement by posting a notice of changes to the beta portal. Your continued use of the Beta Product after we modify the Agreement constitutes your acceptance of the changes. If you do not agree to any changes, you must terminate the agreement and discontinue all use of the Beta Product.

11.2.  Governing Law, Attorneys’ Fees. This Agreement, and all matters arising out of or relating to this Agreement, will be governed by and construed in accordance with the laws of the jurisdiction set forth in the governing law column opposite the applicable undersigned F5 entity in the table below, without regard to that jurisdiction’s choice of law rules. Further, for any action arising out of or related to this Agreement, Customer consents to the exclusive jurisdiction and venue of the courts located in the venue column opposite the applicable undersigned F5 entity in the table below.

F5 Entity: Governing Law: Venue:
F5 Networks Singapore Pte Ltd The laws of Singapore Singapore
F5 Networks Ltd. The laws of England and Wales London, England
F5 Networks, Inc. The laws of the State of Washington Seattle, Washington

 

The Parties agree that the United Nations Convention on Contracts for the International Sale of Goods (CISG) and the Uniform Computer Information Transactions Act, in whatever form adopted, does not apply to this Agreement and the parties specifically opt out of the application of such laws. If either party engages attorneys to enforce any rights arising out of or relating to this Agreement, the prevailing party will be entitled to recover reasonable attorneys’ fees.

11.3.  Assignment. You may not assign this Agreement in whole or in part, without F5’s prior written consent. F5 may assign this Agreement or any of its rights and obligations under it at any time. Any attempted assignment or transfer in violation of this Section will be void and without effect. Subject to the foregoing, this Agreement will be binding upon and shall inure to the benefit of the parties and their respective permitted successors and assigns.

11.4.  U.S. Government Restricted Rights. If Customer uses the Beta Service by or for any unit or agency of the United States Government, this provision applies. The Beta Service and any associated documentation are “commercial items” incorporating “commercial computer software” and “commercial computer software documentation” as such terms are defined in the Federal Acquisition Regulations (48 C.F.R.) (the “FAR”) 2.101 and its supplements. The parties agree that the Beta Service was developed entirely at private expense, that no part of the Beta Service was first produced in the performance of a Government contract. Consistent with FAR 12.212 and DoD FAR Supp. 227.7202-1 through 227.7202-4, and notwithstanding any other FAR or other contractual clause to the contrary in any agreement into which this Agreement may be incorporated, Customer will acquire and may provide to a Government end user with only those rights set forth in this Agreement. Use and/or access of the Beta Service constitutes acknowledgment of F5’s and its licensors’ rights in the Beta Service.

11.5.  Notices. Any notice or other communication required or permitted under this Agreement must be in writing in English and will be deemed given three (3) business days after it is sent by registered or certified mail, return receipt requested and postage prepaid, one (1) business day after it is sent via reputable nationwide courier service, or upon personal delivery, or upon posting to or sending of a notification in the Beta Portal. Notices to Customer may be sent to the address set forth in Customer’s Account or other address provided to F5 in the registration process for the Beta Product, sent to Customer’s notification inbox in the Beta Portal, or posted in the Beta Portal. All notices to F5 will be sent to the address(es) of the applicable F5 entity in the table below. Either party may change its address by giving the other party written notice in accordance with this Section 16. Notice may also be sent by fax, with confirmation of receipt, or electronic mail provided that it is also provided in accordance with one other method described above within 3 business days.

F5 Entity: Notice Address: With a Copy to:
F5 Networks Singapore Pte Ltd F5 Networks Singapore Pte LtdAttn: Legal Department5 Temasek Boulevard#08-01/02/05 Suntec Tower 5Singapore 038985Singapore F5 Networks, Inc.Attention:  Legal Department401 Elliot Avenue WestSeattle, WA 98119USA
F5 Networks Ltd. F5 Networks Ltd.Attn: Legal DepartmentChertsey Gate West43-47 London Street ChertseySurrey KT16 8APUnited Kingdom F5 Networks, Inc.Attention:  Legal Department401 Elliot Avenue WestSeattle, WA 98119USA
F5 Networks, Inc. F5 Networks, Inc.Attn: Legal Department401 Elliott Avenue WestSeattle, WA 98119 USA

 

11.6.  Severability. Any provisions found to be invalid or unenforceable will not affect the validity or enforceability of the other provisions contained herein, but will instead be replaced with a provision as similar to the original as possible and the remainder of this Agreement shall remain valid and enforceable according to its terms.

11.7.   No Waiver. Failure of either party to insist upon strict performance of any of the terms and conditions of this Agreement will not preclude enforcement of such provisions or the exercise of any right. No waiver of a breach of this Agreement will be valid unless in writing. Waiver by either party in the exercise of any of its remedies shall not constitute a subsequent waiver of such terms and conditions or a waiver of any default or remedy.

11.8.  Relationship of the Parties. F5 and Customer are independent contractors and this Agreement will not establish any relationship of partnership, joint venture, employment, franchise, or agency between F5 and Customer. Customer shall not, and will have no power to, bind F5 or incur obligations on F5’s behalf.

11.9.  Interpretation. This Agreement will not be construed in favor of or against any party by reason of the extent to which any party participated in the preparation of this Agreement.

Submit Your Resume

Upload your resume. (5 MB max - .pdf, .doc, or .docx)

January 23, 2018

Building Istio with Minikube-in-a-Container and Jenkins

 

AspenMesh provides a supported distribution of Istio, which means that we need to be able to test and release bugfixes even if they are out-of-cadence with the upstream Istio project. To do this we’ve developed our own build and test infrastructure. Now that we’ve got many of these pieces up and running, we figured some parts might be useful if you are also interested in CI for Istio but not committed to Circle CI or GKE.

This post will show how we made an updated Minikube-in-a-Container and a Jenkins pipeline that uses it to build and test Istio. If you want, you can docker run the minikube container right now and get a functioning Kubernetes cluster inside the container that you can throw away when you’re done. The Jenkins bits will help you build Istio today and also give you a head-start if you want to build containers inside of containers.

Minikube-in-a-Container

This part describes how we made a Minikube-in-a-container that we use to run the Istio smoke tests during a build. This isn’t our idea – we started with localkube-dind. We couldn’t get it working out-of-the-box, we think due to a little bit of drift between localkube and minikube, so this is a record of what we changed to get it working for us. We also added some options and tooling so that we can use Istio in the resulting container. Nothing too fancy but we’re hoping it gives you a head start if you’re heading down a similar path.

Minikube may be familiar to you as a project to start up your own Kubernetes cluster in a VM that you can carry around on your laptop. This approach is very convenient but there are some situations where you can’t/don’t want to provision a VM, like cloud providers that don’t offer nested virtualization. Since docker can now run inside of docker, we decided to try making our own Kubernetes cluster inside of a docker container. An ephemeral Kubernetes container is easy to start, run a few tests, and throw away when you’re done and is a good fit for CI.

In our model, the Kubernetes cluster creates child docker containers (not sibling containers in the lingo of Jérôme Petazzoni’s consideration ). We did this intentionally – we preferred the isolation of child containers over sharing the docker build cache. But you should check out Jérôme’s article before committing to DinD for your application – maybe DooD (Docker-outside-of-Docker) is better for you. FYI – we’ve avoided the “it gets worse” part, and it looks like the “bad” and “ugly” parts are fixed/avoidable for us.

When you start a docker container, you’re asking docker to create and setup a few namespaces in the kernel, and then start your container inside these namespaces. A namespace is a sandbox – when you’re inside the namespace, you can generally only see other things that are also inside the namespace. A chroot, but for more than just filesystems – PIDs, network interfaces, etc. If you start a docker container with --privileged then the namespaces that are created get extra privileges, like the ability to create more child namespaces. That’s the trick at the core of docker-in-docker. For any more details, again, Jérôme’s the expert – check out his explanation (complete with Xzibit memes) here.

OK, so here’s the flow:

  1. Build a container that’s got docker, minikube, kubectl and dependencies installed.
  2. Add a “fake-systemctl” shim to trick Minikube into running without a real systemd installation.
  3. Start the container with --privileged
  4. Have the container start its own “inner” dockerd – this is the DinD part.
  5. Have the container start minikube --vm-driver=none so that minikube (in the container) talks to the dockerd running right alongside it.

All you have to do is docker run --privileged this container and you’re ready to go with kubectl. If you want, you can run the kubectl inside the container and get a truly throw-away environment.

You can try it now:

docker run --privileged --rm -it quay.io/aspenmesh/minikube-dind
docker exec -it <container> /bin/bash
# kubectl get nodes
<....>
# kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/shell-demo.yaml
# kubectl exec -it shell-demo -- /bin/bash

when you exit, the --rm flag means that docker will tear down and throw away everything for you.

For heavier usage, you’ll probably want to “docker cp” the kubeconfig file to your host and talk to kubernetes inside the container over the exposed kube API port 8443.

Here’s the Dockerfile that makes it go (you can clone this and support scripts here):

# Portions Copyright 2016 The Kubernetes Authors All rights reserved.
# Portions Copyright 2018 AspenMesh
#
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Based on:
# https://github.com/kubernetes/minikube/tree/master/deploy/docker/localkube-dind
FROM debian:jessie
# Install minikube dependencies
RUN DEBIAN_FRONTEND=noninteractive apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get -yy -q –no-install-recommends install \
iptables \
ebtables \
ethtool \
ca-certificates \
conntrack \
socat \
git \
nfs-common \
glusterfs-client \
cifs-utils \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common \
bridge-utils \
ipcalc \
aufs-tools \
sudo \
&& DEBIAN_FRONTEND=noninteractive apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Install docker
RUN \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add – && \
apt-key export “9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88” | gpg – && \
echo “deb [arch=amd64] https://download.docker.com/linux/debian jessie stable” >> \
/etc/apt/sources.list.d/docker.list && \
DEBIAN_FRONTEND=noninteractive apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get -yy -q –no-install-recommends install \
docker-ce \
&& DEBIAN_FRONTEND=noninteractive apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
VOLUME /var/lib/docker
EXPOSE 2375
# Install minikube
RUN curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.1/minikube-linux-amd64 && chmod +x minikube
ENV MINIKUBE_WANTUPDATENOTIFICATION=false
ENV MINIKUBE_WANTREPORTERRORPROMPT=false
ENV CHANGE_MINIKUBE_NONE_USER=true
# minikube –vm-driver=none checks systemctl before starting. Instead of
# setting up a real systemd environment, install this shim to tell minikube
# what it wants to know: localkube isn’t started yet.
COPY fake-systemctl.sh /usr/local/bin/systemctl
EXPOSE 8443
# Install kubectl
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.9.1/bin/linux/amd64/kubectl && \
chmod a+x kubectl && \
mv kubectl /usr/local/bin
# Copy local start.sh
COPY start.sh /start.sh
RUN chmod a+x /start.sh
# If nothing else specified, start up docker and kubernetes.
CMD /start.sh & sleep 4 && tail -F /var/log/docker.log /var/log/dind.log /var/log/minikube-start.log
view rawDockerfile.minikube hosted with ❤ by GitHub

Jenkins for Istio

Now that we’ve got Kubernetes-in-a-container we can use this for our Istio builds. Dockerized build systems are nice because developers can quickly create higher fidelity replicas of the CI build. Here’s an outline of our CI architecture for Istio builds:

  • Jenkins worker: This is a VM started by Jenkins for running builds. It may be shared by other builds at the same time. It’s important that any tooling we install on the worker is locally-scoped (so it doesn’t interfere with other builds) and ephemeral (we autoscale Jenkins workers to save costs).
  • Minikube container: The first thing we do is build and enter the Minikube container we talked about above. The rest of the build proceeds inside this container (or its children). The Jenkins workspace is mounted here. Jenkins’ docker plugin takes care of tearing this container down in success or failure, which is all we need to clean up all the running Kubernetes and Istio components.
  • Builder container: This is a container with build tools like the golang toolchain installed. It’s where we compile Istio and build containers for the Istio components. We test those components in the minikube container, and if they pass, declare the build a success and push the containers to our registry.

Most of the Jenkinsfile is about getting those pieces set up. After that, we run the same steps to build Istio that you would on your laptop: make dependmake buildmake test.

Check out the Jenkinsfile here:

node(docker) {
properties([disableConcurrentBuilds()])
wkdir = src/istio.io/istio
stage(Checkout) {
checkout scm
}
// withRegistry writes to /home/ubuntu/.dockercfg outside of the container
// (even if you run it inside the docker plugin) which won’t be visible
// inside the builder container, so copy them somewhere that will be
// visible. We will symlink to .dockercfg only when needed to reduce
// the chance of accidentally using the credentials outside of push
docker.withRegistry(https://quay.io, name-of-your-credentials-in-jenkins) {
stage(Load Push Credentials) {
sh cp ~/.dockercfg ${pwd()}/.dockercfg-quay-creds
}
}
k8sImage = docker.build(
k8s-${env.BUILD_TAG},
-f $wkdir/.jenkins/Dockerfile.minikube +
$wkdir/.jenkins/
)
k8sImage.withRun(–privileged) { k8s ->
stage(Get kubeconfig) {
sh docker exec ${k8s.id} /bin/bash -c \”while ! [ -e /kubeconfig ]; do echo waiting for kubeconfig; sleep 3; done\”
sh rm -f ${pwd()}/kubeconfig && docker cp ${k8s.id}:/kubeconfig ${pwd()}/kubeconfig
// Replace “127.0.0.1” with the path that peer containers can use to
// get to minikube.
// minikube will bake certs including the subject “kubernetes” so
// the kube-api server needs to be reachable from the client’s concept
// of “https://kubernetes:8443” or kubectl will refuse to connect.
sh sed -i” -e ‘s;server: https://127.0.0.1:8443;server: https://kubernetes:8443;’ kubeconfig
}
builder = docker.build(
istio-builder-${env.BUILD_TAG},
-f $wkdir/.jenkins/Dockerfile.jenkins-build +
–build-arg UID=`id -u` –build-arg GID=`id -g` +
$wkdir/.jenkins,
)
builder.inside(
-e GOPATH=${pwd()} +
-e HOME=${pwd()} +
-e PATH=${pwd()}/bin:\$PATH +
-e KUBECONFIG=${pwd()}/kubeconfig +
-e DOCKER_HOST=\”tcp://kubernetes:2375\” +
–link ${k8s.id}:kubernetes
) {
stage(Check) {
sh ls -al
// If there are old credentials from a previous build, destroy them –
// we will only load them when needed in the push stage
sh rm -f ~/.dockercfg
sh cd $wkdir && go get -u github.com/golang/lint/golint
sh cd $wkdir && make check
}
stage(Build) {
sh cd $wkdir && make depend
sh cd $wkdir && make build
}
stage(Test) {
sh cp kubeconfig $wkdir/pilot/platform/kube/config
sh “””PROXYVERSION=\$(grep envoy-debug $wkdir/pilot/docker/Dockerfile.proxy_debug |cut -d: -f2) &&
PROXY=debug-\$PROXYVERSION &&
curl -Lo – https://storage.googleapis.com/istio-build/proxy/envoy-\$PROXY.tar.gz | tar xz &&
mv usr/local/bin/envoy ${pwd()}/bin/envoy &&
rm -r usr/“””
sh cd $wkdir && make test
}
stage(Push) {
sh cd && ln -sf .dockercfg-quay-creds .dockercfg
sh cd $wkdir && +
make HUB=yourhub TAG=$BUILD_TAG push
gitTag = getTag(wkdir)
if (gitTag) {
sh cd $wkdir && +
make HUB=yourhub TAG=$gitTag push
}
sh cd && rm .dockercfg
}
}
}
}
String getTag(String wkdir) {
return sh(
script: cd $wkdir && +
git describe –exact-match –tags \$GIT_COMMIT || true,
returnStdout: true
).trim()
}
view rawJenkinsfile hosted with ❤ by GitHub

If you want to grab the files from this post and the supporting scripts, go here.

5 thoughts on “Building Istio with Minikube-in-a-Container and Jenkins

  1. Hi,

    I am testing your image, I am trying to launch a single pod with “kubectl run mynginx –image=nginx:alpine”, but it does not work at all.

    I see that in the logs:

    ==> /var/log/docker.log <==
    time="2018-06-13T09:00:03.369568213Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
    time="2018-06-13T09:00:03Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/afc6128431a88c858e7563354ce01a4cc943aac3c05a5012ca0dc5a5d76bbd9d/shim.sock" debug=false module="containerd/tasks" pid=363
    time="2018-06-13T09:00:03.490519933Z" level=warning msg="unknown container" container=afc6128431a88c858e7563354ce01a4cc943aac3c05a5012ca0dc5a5d76bbd9d module=libcontainerd namespace=plugins.moby
    time="2018-06-13T09:00:03.515864791Z" level=warning msg="unknown container" container=afc6128431a88c858e7563354ce01a4cc943aac3c05a5012ca0dc5a5d76bbd9d module=libcontainerd namespace=plugins.moby
    time="2018-06-13T09:00:28.924742602Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
    time="2018-06-13T09:00:28Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/703d9311dd05da5e5558556bbde8a7e1c02a52c3bcabb9449e8a1709ddd07276/shim.sock" debug=false module="containerd/tasks" pid=470
    time="2018-06-13T09:00:29.021090994Z" level=warning msg="unknown container" container=703d9311dd05da5e5558556bbde8a7e1c02a52c3bcabb9449e8a1709ddd07276 module=libcontainerd namespace=plugins.moby
    time="2018-06-13T09:00:29.049912568Z" level=warning msg="unknown container" container=703d9311dd05da5e5558556bbde8a7e1c02a52c3bcabb9449e8a1709ddd07276 module=libcontainerd namespace=plugins.moby
    time="2018-06-13T09:00:30.122556354Z" level=info msg="Container 703d9311dd05da5e5558556bbde8a7e1c02a52c3bcabb9449e8a1709ddd07276 failed to exit within 0 seconds of signal 15 – using the force"
    time="2018-06-13T09:00:30.165355369Z" level=warning msg="unknown container" container=703d9311dd05da5e5558556bbde8a7e1c02a52c3bcabb9449e8a1709ddd07276 module=libcontainerd namespace=plugins.moby

    Any idea why this container dies?

    1. I’m seeing the same thing myself when I try to run the container now. I’m using docker for mac and I know I’ve upgraded docker several times since January. Maybe something changed there?

      The dind is capable of running other docker containers – I could successfully “docker run hello-world” and “docker run -it ubuntu bash”. Just not full Kubernetes via localkube.

      Sorry zoobab, I’ll have to dig deeper.

  2. Have you put any thought into if its possible to run `minikube start..` in the docker build phase rather than the docker run phase? What you have described takes around 2mins to create a healthy 1 node kube cluster (with all the downloads done in the build phase). It would be great to reduce that by having the run phase of the container just start an already configured minikube instance. I’ve tried it, but hit weird issues where pods don’t have network access to the api server via 10.96.0.x. I assume its some left over configuration that has changed, Any ideas?

    1. That’s interesting.

      It should definitely be possible to do “minikube cache add …” in the docker build phase, which would mean you don’t have to download any of the kubernetes stuff during “docker run”, but you’d still have to run it.

      As far as “minikube start” I think you’re running into issues where Minikube wants to remember the IP address (it puts it into the .kubeconfig and probably other places) but this changes between when you “docker build” and “docker start” (and each instance of “docker start”).

      Even if you could work around that, I think there may be other side effects to “minikube start” during docker build. I think if you get it to work, one side effect would be that anyone with your docker container could connect to anyone else’s k8s cluster that was started from that same container, because the cert pair was baked in during “docker build”.

      That may not be a big deal since this is supposed to be run locally, but I’d hate to publicly host something with keys pre-baked. Might be just fine for your own local purposes.

      (You’re going to have to re-do at least part of “minikube start” during “docker run” because we at least have to start the associated kubernetes containers i.e. create the namespaces and start the processes)

      If I update this, I’ll definitely try the “minikube cache add …” pieces and I’m interested to hear if you have success with “minikube start” during docker build.

      1. Not sure about the minikube cache system, but at least for the big download that I noticed, this addition to your Dockerfile seems to suffice:

        # for a standalone Dockerfile: FROM quay.io/aspenmesh/minikube-dind
        ADD https://storage.googleapis.com/minikube/k8sReleases/v1.8.0/localkube-linux-amd64 /usr/local/bin/localkube
        RUN mkdir -p /root/.minikube/cache/localkube
        RUN cp /usr/local/bin/localkube /root/.minikube/cache/localkube/localkube-v1.8.0
        RUN echo 546bd1980d0ea7424a21fc7ff3d7a8afd7809cefd362546d40f19a40d805f553 > /root/.minikube/cache/localkube/localkube-v1.8.0.sha256

        Here https://github.com/kubernetes/minikube/blob/v0.24.1/pkg/minikube/constants/constants.go#L119-L120 and https://github.com/kubernetes/minikube/blob/v0.24.1/pkg/minikube/bootstrapper/localkube/localkube_caching.go#L74 are the relevant bits AFAICT.

Leave a Reply

Your email address will not be published. Required fields are marked *