We Make Service Mesh Easy

 

F5 NETWORKS

ASPEN MESH BETA AGREEMENT

Use of the BETA PRODUCT AND OTHER F5 PROPERTY is subject to the TERMS AND CONDITIONS of THIS ASPEN MESH BETA AGREEMENT (THIS “AGREEMENT”).

BY CLICKING ON THE “I AGREE” BUTTON IMMEDIATELY BELOW, REGISTERING FOR OR USING AN ACCOUNT (“ACCOUNT”) FOR THE BETA PRODUCT, OR BY ACCESSING THE BETA PRODUCT (THE FIRST INSTANCE OF ANY OF THE FOREGOING, AN “ACCEPTANCE”), YOU (1) agree to this “Agreement” and the BETA portal terms of use (THE “PORTAL TERMS OF USE”) on behalf of yourself and the Customer identified in the BETA PORTAL (the “Customer” or “YOU”), (2) represent and warrant that you are authorized to AGREE TO this Agreement AND THE PORTAL TERMS OF USE on behalf of the Customer, and (3) agree that such Customer will be responsible for the acts and omissions of any individuals or other users who register for, access or use f5 property through your account or under this agreement. FOR AVOIDANCE OF DOUBT, YOUR ACCEPTANCE IS ON BEHALF OF YOURSELF AND OF YOUR ORGANIZATION’S EMPLOYEES, AGENTS, CONSULTANTS OR OTHER REPRESENTATIVES (“CUSTOMER REPRESENTATIVES”).  IF YOU AND/OR CUSTOMER DOES NOT AGREE TO ALL OF THE TERMS OF THIS AGREEMENT OR THE PORTAL TERMS OF USE, DO NOT ACCEPT THIS AGREEMENT AND DO NOT ACCESS THE BETA PRODUCT.

All references to “F5” in this Agreement are references to the applicable F5 entity as follows: (i) if the Customer’s primary place of business is located in Europe, the Middle East or Africa (“EMEA”), the F5 entity is F5 Networks Ltd.; (ii) if the Customer’s primary place of business is located in the Asia-Pacific region (“APAC”), the F5 entity is F5 Networks Singapore Pte Ltd.; and (iii) if the Customer’s primary place of business is located in a region outside of EMEA or APAC, the F5 entity is F5 Networks, Inc.  As used herein, “Customer” or “you”, on the one hand, and “F5”, “we”, or “us”, on the other hand, shall each be “a Party” hereto, and may collectively be referred to as “the Parties” hereto.

You will provide accurate, current and complete information (including about Customer and Customer’s users) in all registration and other Account-related forms on the Beta Portal (“Customer Information”) and you will maintain the security of your username(s) and password(s). You will maintain and promptly update the Customer Information to keep it accurate, current and complete. YOU UNDERSTAND THAT ANY PERSON WITH YOUR USERNAME(S) AND PASSWORD(S) MAY BE ABLE TO ACCESS YOUR ACCOUNT. YOU ACCEPT ALL RISKS OF UNAUTHORIZED ACCESS TO YOUR ACCOUNT BASED ON THE SHARING OR LOSS OF A USERNAME AND PASSWORD. You will promptly notify F5 if you discover or otherwise suspect any unauthorized access related to your Account, the Beta Product, including any unauthorized use or disclosure of a username or password.

1.  Definitions.

1.1.    “Beta Period” means the period in which Customer may use the Beta Product, and if no period is specified, until F5 specifies otherwise.

1.2.    “Beta Portal” means the Aspen Mesh Beta Portal website located at https://my.aspenmesh.io/.

1.3.    “Beta Services” means the F5 beta service(s) and documentation offered through F5’s software-as-a-service platform, as such may be reasonably modified by F5 from time to time.

“Beta Software” means the F5 beta software product(s) and documentation offered through the Beta Portal for download.

1.5.    “Beta Product” means the Beta Software, the Beta Services, and the Beta Portal.

1.6.    “F5 Property” means (a) the Beta Product and all other software, hardware, technology, documentation, and information provided by F5 in connection with the Beta Product, and all enhancements, modifications and improvements thereto; (b) all internal documentation, specifications, product requirements, problem reports, analysis and performance information, benchmarks, software documents, and other technical, business, product, marketing and financial information, plans and data relating to the Beta Services; (c) all ideas, know-how, and techniques that may be developed, conceived, or invented by F5 during its performance under this Agreement; and (d) all worldwide patent, copyright, trade secret, trademark and other intellectual property rights in and to the property described in clauses (a), (b) and (c) above.

2. Use of Beta Product.

2.1.    License and Rights Grants. Subject to this Agreement, (a) F5 hereby grants Customer (i) a non-exclusive, non-transferable, non-sublicensable, revocable license, during the Beta Period to use the Beta Software, in object code form, solely for Customer’s internal evaluation and testing purposes (the “Purpose”), solely with Customer’s test data, and solely in a non-production environment, and (ii) a non-exclusive, non-transferable, revocable right, during the Beta Period, to access and use the Beta Services and Beta Portal solely for the Purpose; and (b) Customer hereby grants F5 a non-exclusive, non-transferable, worldwide right to use the electronic data that is transmitted to or through the Beta Services by Customer or on Customer’s behalf (collectively, “Customer Data”) for the purpose of providing the Beta Product.

2.2.    Ownership. Subject to this Agreement, F5 and its licensors and suppliers own and retain all right, title, and interest in and to the F5 Property.  Except for the rights granted in Section 2.1 above, no right, title or interest in and to the F5 Property is granted or conveyed to Customer by implication or otherwise.  Subject to this Agreement, Customer owns and retains all right, title, and interest in and to the Customer Data and all intellectual property rights therein.

2.3.    Restrictions.  Customer shall use the Beta Product solely as expressly permitted in Section 2 hereof and solely in accordance with any documentation or instructions supplied by F5. Customer shall not (a) demonstrate, market, reuse, copy, modify, translate or create derivative works of, or otherwise attempt to derive, analyze or use any source code of, or underlying ideas or algorithms pertaining to, the F5 Property or any portion thereof by any means whatsoever; (b) rent, sell, lease, transfer or otherwise make available the F5 Property or any portion thereof to any third party, or use it or any portion thereof for the benefit of a third party (such third parties including Customer affiliates); (c) dissemble, de-compile, reverse assemble, reverse compile or reverse engineer the F5 Property or any portion thereof, or otherwise attempt to discover any F5 Property source code or underlying proprietary information; (d) remove, obscure, or alter any intellectual property right or confidentiality notices or legends appearing in or on any aspect of any the F5 Property; or (e) use any information in any way related to or acquired by use of the Beta Services other than in connection with Customer’s evaluation of the Beta Product.  In addition to the foregoing, Customer will not introduce or permit to be submitted or received by the Beta Services: (x) any data other than Customer’s test data, (y) any Personal Data; or (z) any infringing, obscene, libelous, or otherwise unlawful data or material.  “Personal Data” shall mean information about any individual person that can be used to distinguish or trace the person’s identity, such as their name, social security number, or biometric records, whether alone or combined with other personal or identifying information that is linked or linkable to a specific individual such as date and place of birth or mother’s maiden name.  Personal Data further includes other information deemed “personal data”, “personal health information”, or “personally identifiably information” under applicable data privacy laws.  In addition to F5’s other rights and remedies hereunder, F5 may immediately (and without prior notice) suspend (including but not limited to any transmission of Customer Data to or through the Beta Services) or terminate this Agreement if Customer violates the restrictions in this subsection.

2.4.    Additional Obligations.

2.4.1.   Customer will not disclose the results of any benchmarking or other tests of the Beta Services or any portion thereof to any third party without F5’s prior written consent.

2.4.2.   Customer hereby agrees to report to F5 on the performance of the Beta Product and shall provide suggestions, comments or other feedback to F5 with respect to improving the Beta Product (collectively, “Feedback”).  F5 may freely use, disclose, reproduce, license, distribute, and otherwise commercially exploit the Feedback without obligation or restriction of any kind.

2.4.3.   Customer will promptly report to F5 any problems with the Beta Product.

3. Term and Termination.

3.1.    The term of this Agreement will commence upon Acceptance and will continue until the completion of the Beta Period or until earlier terminated in accordance with this Agreement. This Agreement may be terminated by any Party for any reason or no reason upon written notice to another Party, and in any case will expire at the end of the Beta Period. F5 may extend the Beta Period upon written notice to Customer. F5 may also, upon written notice, or earlier (with subsequent written notice) in the case of a question of the security or integrity of your Account or the Beta Product: (a) discontinue the beta program or your Account at any time, in which case this Agreement will automatically terminate at the time of such discontinuation, or (b) suspend your Account or access to any part of the Beta Product without otherwise terminating the Beta Period.

3.2.    Upon termination of this Agreement, the licenses and rights granted to Customer hereunder will terminate and F5 may terminate Customer’s access to and use of the F5 Property immediately.  Upon termination, Customer will immediately destroy and delete all F5 Property (including Beta Software and other Confidential Information) in Customer’s possession or control. Upon F5’s request, Customer agrees to certify in writing to Customer’s compliance herewith. Following termination of this Agreement, F5 may destroy Customer Data and other Customer Confidential Information that is in its possession or control. Any terms by which their nature impose an obligation after termination will survive termination or expiration of this Agreement, including but not limited to confidentiality, disclaimer of warranties, limitation of liabilities, indemnification, and governing law.

4. Confidentiality.

4.1.    Subject to the exceptions listed below, a disclosing Party’s “Confidential Information” shall be defined as information disclosed by the disclosing Party to the receiving Party under this Agreement that is either:  (a) clearly marked or otherwise clearly designated as confidential or proprietary; or (b) should be reasonably understood by the receiving Party to be the confidential or proprietary information of the disclosing Party.  For the avoidance of doubt, F5 Property and all pricing under this Agreement is the Confidential Information of F5, and Customer Data is the Confidential Information of Customer.  Confidential Information of the disclosing Party does not include information that (i) is previously rightfully known to the receiving Party without restriction on disclosure, (ii) is or becomes known to the general public, through no act or omission on the part of the receiving Party, (iii) is disclosed to the receiving Party by a third party without breach of any separate nondisclosure obligation, or (iv) is independently developed by the receiving Party without resort to the Confidential Information of the disclosing Party.

4.2.    The receiving Party will hold in confidence, using reasonable measures to protect, and not use any Confidential Information except in connection with its obligations and rights under this Agreement and shall not disclose the Confidential Information to any third party except to its employees, consultants, independent contractors and agents who have a need to know in connection with this Agreement and who have signed confidentiality agreements or are otherwise bound by confidentiality obligations at least as restrictive as those contained herein; provided, however, that F5 may use and disclose the Confidential Information of Customer as necessary for the limited purpose of performing the Beta Services hereunder.  If required by law or government order of competent jurisdiction, the receiving Party may disclose Confidential Information of the disclosing Party but will give adequate prior notice of such disclosure to the disclosing Party (unless prohibited by law or government order) to permit the disclosing Party to intervene and to request protective orders or other confidential treatment therefor.

4.3.    Money damages may not be an adequate remedy if this Section 4 is breached and, therefore, either Party shall, in addition to any other legal or equitable remedies, be entitled to seek an injunction or other equitable relief against such breach or threatened breach without the necessity of posting any bond or surety.

5. Representations and Warranties. Customer hereby warrants, represents and covenants as follows:  (a) Customer has the authority to grant the rights granted by Customer under this Agreement and to perform its obligations under this Agreement; (b) in the performance of its obligations and use of the Beta Product, Customer will not violate any applicable laws or infringe the proprietary or privacy rights of any third party; (c) the Customer Data that Customer transmits or permits to be transmitted to or through the Beta Services or Beta Portal complies and will at all times during the term of this Agreement comply with all applicable laws and does not and will not infringe the proprietary rights or privacy rights of any third parties; (d) when using the Beta Services Customer will comply with all applicable acceptable use policies for which Customer has received notice and will not cause or allow others to cause the disruption of other parties’ use or enjoyment of the Internet; (e) Customer does not currently provide services that compete with the Beta Product and will not at any time in the future use the Beta Services or any other Confidential Information for the provision of any services that compete with the Beta Services; and (f) Customer shall comply with all applicable export laws, restrictions and regulations of the U.S. Department of Commerce, and all other applicable U.S. and foreign authority.  In addition, Customer hereby represents and warrants that it is not (nor are any Customer Representatives) on the U.S. Treasury Department’s list of Specially Designated Nationals or the U.S. Department of Commerce’s Table of Denial Orders.

6. Warranty Disclaimer; Warning. Customer acknowledges that the Beta Product is provided “AS IS”. Customer understands that the Beta Product has not completed F5’s full quality assurance program and may have errors and may produce unexpected results. F5 DISCLAIMS ALL WARRANTIES RELATING TO THE BETA PRODUCT, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES AS TO TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE (INCLUDING, WITHOUT LIMITATION, PREVENTION OF UNAUTHORIZED ACCESS), THAT THE BETA PRODUCT WILL BE FREE FROM ERROR OR INTERRUPTION OR FAILURE. F5 NEITHER ASSUMES, NOR AUTHORIZES ANY OTHER PERSON TO ASSUME FOR IT, ANY OTHER LIABILITY IN CONNECTION WITH THE BETA PRODUCT OR ANY INFORMATION PROVIDED IN CONNECTION THEREWITH, INCLUDING, WITHOUT LIMITATION, LIABILITY ARISING OUT OF THE OPERATION, SUPPORT, OR USE OF THE BETA PRODUCT. F5 DOES NOT WARRANT THE RESULTS OF THE BETA PRODUCT OR THAT ANY ERRORS IN THE BETA PRODUCT WILL BE CORRECTED, OR THAT THE BETA PRODUCT WILL MEET CUSTOMER’S REQUIREMENTS OR EXPECTATIONS. CUSTOMER SHOULD SAFEGUARD IMPORTANT DATA, USE CAUTION, AND NOT RELY IN ANY WAY ON THE CORRECT FUNCTIONING OR PERFORMANCE OF THE BETA PRODUCT AND/OR ACCOMPANYING MATERIALS.

7. Limitation of Remedies and Damages. IN NO EVENT WILL F5 OR ITS LICENSORS OR SUPPLIERS BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE, DAMAGES INCLUDING, BUT NOT LIMITED TO LOSS OF PROFITS, REVENUE, DATA OR MACHINE USE, BUSINESS INTERRUPTION ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT, THE USE OR THE INABILITY OF THE USE OR PERFORMANCE OF THE BETA PRODUCT, THE PROVISION OF OR FAILURE TO PROVIDE THE BETA PRODUCT, OR ANY INFORMATION PROVIDED, REGARDLESS OF THE NATURE OF THE ACTION OR UNDERLYING LEGAL THEORY (INCLUDING UNDER ANY CONTRACT, NEGLIGENCE, STRICT LIABILITY OR OTHER THEORY), EVEN IF F5 AND/OR ITS LICENSORS OR SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. F5 WILL NOT BE RESPONSIBLE FOR ANY MATTER BEYOND ITS REASONABLE CONTROL. IN NO EVENT WILL F5’S TOTAL AGGREGATE LIABILITY UNDER THIS AGREEMENT EXCEED ONE THOUSAND DOLLARS ($1,000). NO ACTION, REGARDLESS OF FORM, ARISING UNDER THIS AGREEMENT MAY BE BROUGHT BY CUSTOMER MORE THAN ONE YEAR AFTER THE END OF THE BETA PERIOD.

8. Customer Acknowledgement. CUSTOMER ACKNOWLEDGES THE NATURE OF THE BETA PRODUCT, AND THAT IT MAY CONTAIN KNOWN OR UNKNOWN BUGS, DEFECTS AND ERRORS, MAY NOT FUNCTION AS INTENDED, MAY NOT FUNCTION AT THE LEVEL OF A FINAL, GENERALLY AVAILABLE PRODUCT, MAY CREATE UNFORESEEABLE EVENTS, AND MAY BE SUBSTANTIALLY MODIFIED PRIOR TO FIRST GENERAL COMMERCIAL AVAILABILITY, OR WITHDRAWN ENTIRELY. CUSTOMER ACKNOWLEDGES THAT THE PRODUCT IS INTENDED TO BE USED ONLY IN A NON-PRODUCTION ENVIRONMENT, WITH TEST DATA, AND NOT FOR PRODUCTION PURPOSES. F5 HAS NO OBLIGATION TO CORRECT ANY BUGS, DEFECTS OR ERRORS IN THE BETA PRODUCT OR OTHERWISE SUPPORT OR MAINTAIN THE BETA PRODUCT. F5 RESERVES THE RIGHT AT ANY TIME NOT TO GENERALLY COMMERCIALLY RELEASE THE BETA PRODUCT OR, EVEN IF RELEASED, TO ALTER PRICES, FEATURES, SPECIFICATIONS, CAPABILITIES, FUNCTIONS, RELEASE DATES, GENERAL AVAILABILITY, OR OTHER CHARACTERISTICS OF THE BETA PRODUCT.

9. Indemnification. Customer will defend, indemnify and hold F5, and its affiliates and their respective officers, directors, licensors, suppliers, service providers, employees, contractors and agents harmless against any claims, liabilities or expenses incurred (including reasonable attorneys’ fees), as well as amounts finally awarded in a settlement or by a court arising from any (a) breach by Customer or any Customer Representative of any representation, warranty, covenant or other obligation of this Agreement, (b) negligence or willful misconduct of Customer or Customer Representatives, or (c) liability arising out of or relating to the use of F5 Property. F5 will provide Customer (i) prompt written notice of any such claim or allegation, provided that any delay in providing such notice shall not impact Customer’s obligations hereunder except to the extent that Customer is materially prejudiced by such delay; (ii) control of the defense and settlement thereof, provided that any settlement shall be subject to F5’s prior written approval; and (iii) reasonable assistance in such defense or settlement, at Customer’s expense.  Notwithstanding the foregoing, if Customer does not timely assume control of the defense thereof, F5 may do so on its own, after prior notice to Customer, and in turn Customer shall provide reasonable assistance in such defense or settlement, at Customer’s expense.

10. Customer Performance and Usage Data. F5 may measure and monitor Customer interaction with the Beta Product, and to generate reports relating to use of the Beta Product under this Agreement. All information and status reports derived from such use of the Product will be considered F5 Property. Customer will not disclose such information or reports. F5 may use Aggregated Data derived from the use of the Beta Product to support and improve F5 products and services, in the development of new features, products, tools, content, and for market research. “Aggregated Data” means Customer Data and usage information that has been stripped of all Personal Data and any information that identifies Customer.

11. General.

11.1.  Entire Agreement. This Agreement constitutes the entire agreement between the Parties pertaining to the subject matter hereof, and supersedes any and all prior proposals, negotiations, communications, and agreements, written or oral previously existing between the Parties pertaining to the subject matter hereof.  Without limiting the generality of the foregoing, in the event of a conflict between the terms of this Agreement and any other agreement between the Parties, this Agreement will govern with respect to any component of the Beta Product or any portion thereof to the extent used as part of the Beta Product. F5 will not be bound by the use of any standardized form or correspondence (including any order form, purchase order, acknowledgement, shrink-wrap, boxtop, or click wrap license, or other form) containing additional or different terms. f5 may, at any time and in its discretion, modify this Agreement by providing written notice, including but not limited to the posting of a notice of changes to the beta portal.  customer’s continued use of the Beta product after f5 modifies this Agreement constitutes customer’s acceptance of the changes. If cUSTOMER doES not agree to any changes, CUSTOMER must terminate this agreement and discontinue all use of the beta product.

11.2.  Governing Law, Attorneys’ Fees. This Agreement, and all matters arising out of or relating to this Agreement, will be governed by and construed in accordance with the laws of the jurisdiction set forth in the governing law column opposite the applicable undersigned F5 entity in the table below, without regard to that jurisdiction’s choice of law rules. Further, for any action arising out of or related to this Agreement, Customer consents to the exclusive jurisdiction and venue of the courts located in the venue column opposite the applicable undersigned F5 entity in the table below.

F5 Entity: Governing Law: Venue:
F5 Networks Singapore Pte Ltd The laws of Singapore Singapore
F5 Networks Ltd. The laws of England and Wales London, England
F5 Networks, Inc. The laws of the State of Washington Seattle, Washington

 

The Parties agree that the United Nations Convention on Contracts for the International Sale of Goods (CISG) and the Uniform Computer Information Transactions Act, in whatever form adopted, does not apply to this Agreement and the Parties specifically opt out of the application of such laws. If either Party engages attorneys to enforce any rights arising out of or relating to this Agreement, the prevailing Party will be entitled to recover reasonable attorneys’ fees.

11.3.  Assignment. Customer may not assign this Agreement in whole or in part, without F5’s prior written consent. F5 may assign this Agreement or any of its rights and obligations under it at any time. Any attempted assignment or transfer in violation of this Section will be void and without effect. Subject to the foregoing, this Agreement will be binding upon and shall inure to the benefit of the Parties and their respective permitted successors and assigns. F5 may subcontract parts of the Beta Services to third parties or provision of services related to management and hosting of the Beta Product to third parties (such third parties including but not limited to F5’s affiliates). F5 is responsible for breaches of this Agreement caused by its subcontractors.

11.4.  Export Control; U.S. Government Restricted Rights.  F5 Property is subject to export control laws of various countries, including the laws of the United States. Customer will not submit F5 Property to any government agency for licensing consideration or other regulatory approval and will not export F5 Property to countries, persons or entities if prohibited by export laws.  Notwithstanding, in any case, if Customer uses the Beta Product by or for any unit or agency of the United States Government, this provision applies. The Beta Product and any associated documentation are “commercial items” incorporating “commercial computer software” and “commercial computer software documentation” as such terms are defined in the Federal Acquisition Regulations (48 C.F.R.) (the “FAR”) 2.101 and its supplements. The Parties agree that the Beta Product was developed entirely at private expense, that no part of the Beta Product was first produced in the performance of a Government contract. Use and/or access of the Beta Product constitutes acknowledgment of F5’s and its licensors’ and suppliers’ rights in the Beta Product.

11.5.  Notices. Any notice or other communication required or permitted under this Agreement must be in writing in English and will be deemed given three (3) business days after it is sent by registered or certified mail, return receipt requested and postage prepaid, one (1) business day after it is sent via reputable nationwide courier service, or upon personal delivery, or upon posting to or sending of a notification in the Beta Portal. F5 may also provide notice by email or other electronic delivery, which shall be deemed given immediately upon transmission to the address set forth on Customer’s Account. Notices to Customer may also be sent to the address set forth in Customer’s Account or other address provided to F5 in the registration process for the Beta Product. All notices to F5 will be sent to the address(es) of the applicable F5 entity in the table below. Either Party may change its address by giving the other Party written notice in accordance with this Section.

F5 Entity: Notice Address: With a Copy to:
F5 Networks Singapore Pte Ltd F5 Networks Singapore Pte Ltd

Attn: Legal Department

5 Temasek Boulevard

#08-01/02/05 Suntec Tower 5

Singapore 038985

Singapore

 

F5 Networks, Inc.

Attention:  Legal Department

401 Elliot Avenue West

Seattle, WA 98119

USA

F5 Networks Ltd. F5 Networks Ltd.

Attn: Legal Department

Chertsey Gate West

43-47 London Street Chertsey

Surrey KT16 8AP

United Kingdom

 

F5 Networks, Inc.

Attention:  Legal Department

401 Elliot Avenue West

Seattle, WA 98119

USA

F5 Networks, Inc. F5 Networks, Inc.

Attn: Legal Department

401 Elliott Avenue West

Seattle, WA 98119

USA

 

11.6.  Severability. Any provisions found to be invalid or unenforceable will not affect the validity or enforceability of the other provisions contained herein, but will instead be replaced with a provision as similar to the original as possible and the remainder of this Agreement shall remain valid and enforceable according to its terms.

11.7.  No Waiver. Failure of either Party to insist upon strict performance of any of the terms and conditions of this Agreement will not preclude enforcement of such provisions or the exercise of any right. No waiver of a breach of this Agreement will be valid unless in writing. Waiver by either Party in the exercise of any of its remedies shall not constitute a subsequent waiver of such terms and conditions or a waiver of any default or remedy.

11.8.  Relationship of the Parties. F5 and Customer are independent contractors and this Agreement will not establish any relationship of partnership, joint venture, employment, franchise, or agency between F5 and Customer. Customer shall not, and will have no power to, bind F5 or incur obligations on F5’s behalf.

11.9.  Interpretation. This Agreement will not be construed in favor of or against any Party by reason of the extent to which any Party participated in the preparation of this Agreement.

Submit Your Resume

Upload your resume. (5 MB max - .pdf, .doc, or .docx)

January 8, 2019

Alert Fatigue: Finding What’s Wrong when the Helpers are Hurting…

 

Here at Aspen Mesh, we’re committed to using our own application in production. We can’t expect you to use it if we’re not, right? Plus, we kind of needed a service mesh!

Besides using the platform, we needed to be sure that our platform (composed of a bunch of microservices) was operating properly and helping our users be successful.  The best idea to meet our basic needs (for now) was to make another microservice that impersonated a user and sent us alerts when parts of our platform were unavailable or failed in some defined way. When implemented, it rarely fired, and if it did, there was a good reason, and it helped us to quickly address the reason for the alert.

Fast-forward to KubeCon in December 2018. The interest in Aspen Mesh was almost overwhelming, and though we had no user complaints, and no real observations of trouble, our Slack alerts were going crazy.  We muted the threads, some people left the channel. Still, we figured there had to be something going on and we needed to look into it.

We’ll take you through the questions we had, and how we answered them, as we figured out what was going on. No spoilers, but the answer was unexpected, and the set of unexpected events showed us that there is a new alert we need to write!

Measuring Application Health: Be a User

How did we settle on impersonating users as a way to address application health? The idea of “application health” can be measured many different ways.  

At the most granular level, you can ask “is any application emitting a log message?  Are any requests not getting through?” But in the real world, we’ve found the answer is always “yes”, which makes this level of granularity not terribly helpful.  Even in the best case, users may disconnect their laptops in the middle of a request; no human can handle the deluge at this level. This level may be very noisy, but it’s also the most actionable.  

At the other end, you could just check and see if your users are calling support and screaming that the site is down.  Low noise, but also low actionability – when “the site is down” you don’t know when to start, and it’s way too late to start intervening on behalf of your users. Plus, any customer intervention at that point requires much more investment (time, personal attention, baked goods…) to restore that lost trust.  We needed a middle ground.

We know what our product is supposed to do.  So a solid middle ground is to “be a user”. We implemented a health check service that actively pretends to be a user of Aspen Mesh and does the things a user would normally do: log in to the GUI, check the service graph, tweak an Aspen Mesh Experiment, check for some Istio Vet notes, etc.

If this checker fails, we hope this is a high-fidelity signal that a user would experience the same failure.  It’s not as actionable as a log message in a particular microservice — if “the service graph fails”, the fault could be in any number of components. But it’s a good place to start.

Several months ago, we implemented this health checker that we could monitor via Prometheus and Grafana, and then send alerts to a Slack channel when a threshold is exceeded. It was exciting when we first launched it as we knew it would give us a deeper understanding of our microservices. Luckily, we have a few geniuses on staff and were quite pleased to find that everything was just peachy. We rarely got alerts, and when we did, we were able to trace the issues back to oddities like a dependency that restarted every Wednesday at 11 PM…

…But then we went to Kubecon and our alerts started firing every few minutes. Our sense of urgency quickly waned towards mild annoyance. Still, we wrote the code, and were pretty sure that the health checker wasn’t sending us bogus alerts, so we needed to figure out what it was trying to tell us.

Round I: Initial Questions

Grafana showed this:

Not great.

We created a goal:

As an Aspen Mesh engineer, I only want to get alerts in #prod-alerts for real issues, and those alerts should occur infrequently so that I don’t get alert fatigue.

One of our senior devs started a shared-doc with a list of his initial thoughts since I wasn’t sure where to start:

Andrew’s List of  Questions:

  • Is the problem that the platform is actually having these errors, or is it on the platform monitoring side?  For example, maybe the platform monitor can’t connect “out” to the platform sometimes, but our users are happy as can be.
  • This hypothesis can be falsified by running the platform-health locally and confirming that we see similar errors.
  • Anything really bad in the platform health logs?  Is the platform-health pod restarting?
  • If we curl the /metrics endpoint on the platform-health pod ourselves, do we see errors?  Or is it “synthetic” (like Prometheus can’t scrape the pod)?
  • Is the platform-health pod running with an Istio sidecar?
  • Is the problem correlated in time with any other issues?
  • Some of these are really simple – the “unauth login” page is just getting back a static webpage (after going through some of our platform components).  Why are these affected?

This doc became a sort of journal of things we tried, questions answered, screenshots. It was actually really helpful.

Initial Answers

The important results were that the Health pod was healthy, had been live since the last time we updated production. It (correctly) had no sidecar. Running the service locally produced no errors.  There weren’t any obvious issues with the pods.

Next we checked the logs for the pod. We have CloudWatch set up for this service, so I did some spot-checking to cover a few days worth of alerting. Remember that question of granularity? We implemented helpful error and logs messages in our code, and let Cloudwatch collect our logs so that we can dig in for info when needed (it does some other cool stuff I’ll get to in a moment).

By spot-checking logs for different times and dates, I found that the majority of the failed requests were timeouts, and occasionally our client was totally unavailable. But there didn’t seem to be a pattern of when the client was missing and when the check failed due to timeouts. So at least we now knew that the Alerts were really reporting alerts. But why was it happening so often, for such a short period of time, and why was it that our users weren’t reporting any issues?

Round II: Maybe Patterns?

Next, I decided to zoom in on a 2-hour section of the alert graph to see what patterns I could find, and made some notes for Andrew.

Lauren’s List of Possibly interesting things:

  • The Health service runs 1x/min.
  • There are times when there are only failures from the login page and dashboard that last about 1-2 minutes, then they succeed for anywhere from 2 to 10+ minutes.
  • There are times when everything fails for about 2-5 minutes at a time.
  • There are times when some things fail and then pass in a staggered way — this could indicate an order in which failures occur or resolve.
  • There are some points where there are Auth failures (a failure means ctx.HttpClient() == nil), but it’s not every time. There are times when the other pings fail but we have a valid working client. (See ~14:35pm, ~15:40, ~16:05)
  • There are 2 cases where we did not have the ability to reach the login page, all the other things failed, then the first thing to recover is the login page. (see ~15:15 and
    ~16:10ish)

Andrew’s thoughts on this were:

  • Around 14:25, 15:05,15:28, 15:56 it looks like specifically Dashboard and UnauthLogin fail but everything else succeeds.  That’s interesting because Dashboard and UnauthLogin are both pretty “boring” services, and they both go
    through an apiserver.
  • In a dashboard/unauth failure case, is the request getting all the way to the dashboard?
  • Can we change the HTTP UserAgent for the health monitor, which may make it easier to tell in logs if a request is from the health monitor?

Round III: Jimmy the Jamboard

We met up the next day to Jamboard the problem (I dubbed the Jamboard “Jimmy”… we’ll see if it sticks), and Andrew hypothesized that there might be something wrong with the node the Health pod was in — that it might be hitting a limit, but it could be a resource limit, or a traffic limit.

Finding the nodeName for a pod took a bit. When using AWS, the pod’s nodeName is the AWS Private DNS Name. But once I had that, I was back in the AWS platform. By selecting the Private DNS Name, I checked the instance’s Monitoring tab for CloudWatch data. In addition to graphs of CPU usage, read/write data, it provides graphs of Network In/Out bytes per minute.   This node was handling about 380MB/min In and 20MB/min Out, which seemed pretty high.

I checked some other nodes running our main application and the traffic was surprisingly lower.

One bummer with Cloudwatch for checking individual nodes was the inability to compare multiple nodes in one window, or even multiple browser windows at once (I could only have 1 selected at a time, regardless of window) — hence the screenshots. Last, I double-checked all the nodes in question for a matching 2 hour time frame to be sure I hadn’t been making assumptions based on an isolated incident.

We knew that the Health service itself was tiny, so what was going on with that node? And, if we were hitting a throughput limit, what was that limit? AWS doesn’t do a great job of making that info available, so Andrew found an article where the author was running his own tests on AWS Instance Types. Ours capped at 0.45GB in the author’s tests, so this node could certainly be hitting a network limit every few minutes.

Finally Getting Somewhere!

Next, it was back to the terminal to see what was going on in that node. I learned another new thing: printing out a list of pods by filtering for node name! (Thanks again, Andrew!)

kubectl get pods --all-namespaces --field-selector spec.nodeName=<nodenamehere> -o wide

It turns out that this node was running a ton of per-user microservices that monitor users’ clusters so they can get real-time data. While that seemed the likely culprit, we had noticed a few other deficiencies.  We focused on convincing ourselves we had the right root cause, and also took advantage of the downtime while running our experiment to fix some “nice-to-haves”.

Nice-to-have #1: There were two places in our Health Checker code where the closing the request was missing, so we checked to see if there was an overwhelming number of sockets that were open causing an ephemeral source port starvation. That didn’t seem to be the case, but since we found the bugs, we fixed them anyway.

Nice-to-have #2: We also changed the User-Agent string that is sent with requests.  This makes it easy to select only the health monitor requests in log messages or in our distributed traces.  I recommend setting the user-agent for your different services early on in a project so you’re not staring at a sea of “Go-http-client/1.1″s when debugging.

After all of this, it seemed the issue was that our little Health pod was living in a node with a ton of hungry services causing the node to reach its network bandwidth limit. The easiest way to test this was to either kill the pod and allow “Chance” to start it on a new node (hoping it wasn’t a busy node), or to go through setting taints or tolerations to exercise some control on where the pod restarted.  It seemed faster to just go the route of Chance, and then kill it again if the gamble was bad.  

With luck, it started on a mostly bare node and we went several hours without an alert.  Using Jaeger, we were able to isolate traces with the new user-agent string (via Tags) to see that we weren’t having the same kinds of issues. Just to check, we killed it again and let it restart (luck landed it on a moderately busy node this time) and the service continued to thrive. Jaeger still showed we were good.

Of course, relying on Chance to schedule favorably is not a solution, but it convinced us that we understood what was going on (intra-cluster network saturation), so now we can go off and get a handle on that…

I think we’ve all learned something today…

So the culprit ended up being a node with too much traffic that hit limits which blocked the little service from completing its checks, so the checker did its job and reported errors until we listened. We now know we need to address node traffic in production – maybe using taints and tolerations, maybe with a new service deployed to each node, or maybe something else. But our little Health checker alerted us to a bigger and unforeseen issue, and that’s pretty cool. Also, now when we get alerts in Slack, we care because we’ve fought the alert fatigue and won (at least for today).

Leave a Reply

Your email address will not be published. Required fields are marked *