What You Should Know About the Optimizely “Vulnerability”

Update 25th June 2014 7pm IST: With the new version of VWO, we have now given an option to mask experiment names and variation names.

Update 5th June 2014 11 am IST: It looks like Optimizely is planning to fix the issues that have been identified – particularly they will allow the exclusion of paused and draft tests from test settings. We’re glad they responded quickly to this. Gaining the trust of businesses by fixing such issues will be good for the A/B testing industry at large, and that was our original intention.

Download Free: A/B Testing Guide

In the comments below, people mention that knowing what tests are running is possible using VWO as well. As we write in the article, it’s by definition possible for all A/B testing tools. What is not possible from the VWO test settings code right now is to know what tests are planned or what tests were conducted in the past (and hence give away business strategy). We believe this was an important issue and that’s what we highlighted in the post. Like Optimizely, for integration with analytics products, VWO shows variation names and experiment names. We’re soon going to release an option to replace that with account id and experiment id.

** End of Update **With VWO being a prominent vendor in the A/B testing industry, it’s our duty to comment and elaborate on popular concerns and queries. After reading this post, if you find that we’re wrong about anything, have misinterpreted something, or if you know of better approaches, we invite corrections (and we will promptly update the article). A/B testers are a tight-knit group who are passionate (like us) about providing marketers and web developers the tools to beat the competition. This is why we feel so passionately about addressing these issues, because they may lead to the opposite effect.

Media outlets and the world at large recently found out what we’ve known for a long time: running A/B tests using Optimizely is not safe from a competitive point of view. Take a look at the following article:

Optimizely vulnerability lets you see what other sites are testing (Published on June 02, 2014)

To quote VentureBeat:

“The leak, which is embedded in the JavaScript code used to implement Optimizely on each site, doesn’t appear to endanger the sites or their fundamental security. But it does enable anyone, with very little effort, to see exactly what tests an Optimizely customer is conducting.”

The market at large is worried about this, and they should be. And we want to clarify why such vulnerabilities are unique to Optimizely’s approach and also expand on what makes VWO immune to them.

What exactly are the “Vulnerabilities” and why are they there?

Optimizely depends on external content delivery networks such as Akamai. There is a reason the delivery infrastructure has the word “content” in it. Akamai and other CDNs are primarily used to deliver static content like videos, images, and plain text. Although CDNs are becoming better and more flexible by the day, the fact is that they were never meant to deliver content that is dynamically retrieved (like your Facebook feed or Gmail inbox). There are nuances to this but the dynamic querying capabilities of a CDN are still very limited as compared to a database like MySQL.

CDNs have multiple servers across the world and they simply replicate these static content pieces across all servers. When someone visits a website using a CDN, it typically selects the geographically nearest server to the user and uses that to load the website content.

So what’s the problem?

All A/B testing platforms need to deliver campaign/test settings in a dynamic manner specific to the URL a visitor is on. The problem with Optimizely’s approach is that they use a CDN meant for static content to serve their JavaScript file that makes the changes in an A/B test. Since it’s a static file, Optimizely’s code contains ALL past, present, and work-in-progress campaigns set up across the account, running or not running. Once this file is downloaded, it checks what URL the visitor is on and then applies the relevant A/B test changes to it.

Since test settings are delivered as JavaScript code, anyone can simply “View Source” in the browser to see what’s in it. Test settings being visible to anyone in the browser is not unique to Optimizely, it’s the standard method for most A/B testing tools that integrate via JavaScript.

However, what’s unique with Optimizely is that you get to see not only the tests and campaigns for a specific page (say for optimizely.com) but for all other pages on the website as well (say optimizely.com/pricing and other pages that aren’t even linked from the main website). Moreover, you also get to see all un-archived tests in Optimizely – old, present and even the ones you’re currently working on.

How is VWO different?

We built our own home-grown CDN network that can serve dynamic content. The VWO approach is to dynamically generate page-specific settings for running campaigns only (not stopped or work-in-progress) with every page visit. For example, if visitors arrive on “URL:”vwo.com/early-access” they will only be served those tests that are running on the new VWO’s early access page, nothing else.

Example output of VWO test campaign settings

What’s contained in the publicly accessible Optimizely campaign settings file?

Everything! That’s concerning. As at the time of writing this post, their publicly viewable campaign setting file contained:

• All the A/B test and other optimization campaigns you are currently running across your entire website (even on the pages you wanted to keep secret from competitors)
• All the campaigns you have run in the past that you haven’t archived
• All the campaigns you are planning (which are in the drafting stage)

Someone actually made a dedicated website for you to take a peek at other website testing and optimization campaigns. Take a look at URL: whatyatesting.com.

Simply enter a URL of your competitor/target website and instantly know what campaigns they’re running and what ideas they’re planning to test in the future.

Know what CNN is testing using Optimizely –

URL: whatyatesting.com/sites/cnn.com

Why should you care?

In essence, if you are using Optimizely (or any other vendor that takes the similar route of relying on external CDNs to deliver test settings):

• All the URLs of any landing page that you are testing will be public even if you don’t link them from anywhere on the main website and even if you never wanted URLs to be seen by anyone else than who it was intended for.
• If your competitors know the last 10 campaigns you ran, they can predict the direction the business is heading. This is almost like they know what is being discussed in internal meetings.
• The new ideas you’re toying with will be public before you wanted them to be. They can know if you’re planning to test pricing or new offers and can react before you even had a chance to implement them.
• All old and discarded attempts will be there for your competition and technically savvy users to see. So, if you were testing $49 v/s $69 for a product and you retained $69, people will know that the same product was available for cheaper one week back. And in the worst case, even if you have stopped the campaign, people can still see and use the old versions that you no longer wanted to be seen.

All in all, Optimizely users are currently exposed because competitors can go through their tests and get a sense of the direction the business is taking. Marketers all over are evidently shocked and concerned about Optimizely’s approach.

Another consequence of Optimizely’s approach: large download sizes and potentially slow website load times

If you dump all the campaigns and settings of all the tests across the website, old, current, and future ones, the download size is bound to increase. People notice such stuff: settings of all the tests across the website, old, current and future ones, the download size is bound to increase. People notice such stuff:

And since Optimizely’s integration is “one line” and synchronous, the download of this hundred of kilobytes (sometimes which goes into MBs) of data takes time. And it’s quite obvious that it directly impacts sales and conversions on eCommerce and other websites. Sample a few recent tweets:

(The last time we had a complaining tweet about VWO being slow was in 2012 when we had synchronous code. Now we have an asynchronous code snippet for integration and our own blazing-fast CDN.)

Granted that the static file delivered through an external CDN is cached and will typically be only downloaded only once per session (when a visitor visits the website). However, first experiences matter, and if the website is slow even once, conversions, sales and revenue take a hit.

Like we said previously, VWO dynamically generates page-specific settings for running campaigns. This means that settings are not cached, but if you compare the response times of both approaches, you will see that they’re similar.

So, even without caching, our method of delivering test settings is not slow and obviously, it has the advantage of not leaking your entire testing strategy.

Download Free: A/B Testing Guide

How is VWO safe from these “vulnerabilities”?

As briefly mentioned above, we have an entirely different architecture. There are a couple of major differences in how tests are delivered and executed with VWO.

1. We have our own Content Delivery Network to allow for dynamic delivery

Instead of relying on an external CDN for delivering test and campaign settings, we have our own geographically distributed CDN which we keep expanding by adding new server locations. This enables us to deliver test settings specific to the URL a visitor is on.

2. With VWO, nobody can see what other pages you are running tests and campaigns on

(unless they know the URLs of the pages you are running tests on).

Plus, our custom CDN is lightning fast, delivering settings quickly to test participants from across the world. Check for yourself on an independent monitoring system such as WatchMouse or Site24x7.

3. Our code is not “one-line” but is smart and configurable asynchronous code

VWO Smartcode looks like this:

The reason it “looks” complex is because it is smart. Unlike other A/B testing vendors’ integration codes where they first download test settings and then load the rest of the page, this code downloads test settings parallel to the website.

The VWO Smartcode also has user-configurable timeouts. So you can configure VWO for specific situations, like if test settings are not downloaded in one second, just timeout and show the original website. On the contrary, synchronous code in the worst case hangs for 30 seconds in most browsers.

In an age where even a second of delay causes a 7% loss in conversions, your business will suffer massively if it happens to stall for 30 seconds!

As an added feature, VWO also offers self-hosted test settings where you can download and host test settings from your own server, removing all dependencies (except for logging) on any 3rd party servers. Many of our financial and banking customers use this.

VWO Asynchronous Smartcode still has all the advantages of one-time integration (you just have to copy-paste this code once on your website and you’re done). So apart from the aesthetic value of the “one-line” integration to developers, there’s no real reason why you should prefer that instead of having code that allows for greater flexibility.

Bottom line: A/B testing is important, but so is the issue of leaking sensitive information to competitors and having a really slow website

We know that A/B testing is just so crucial to businesses and the adoption of A/B testing has been exploding for the last many years. A/B testing and Conversion Rate Optimization (CRO) are here to stay. However, you should be prudent in making sure you don’t inadvertently expose information that you don’t want your competitors to know.

If you have any questions regarding the technology, architecture, or security of VWO or any other vendor, please reach out to us. Simply leave a comment here or email contact@vwo.com, and we are happy to answer. If you want to talk to our sales, reach out to us at sales@vwo.com and we can advise you on how you can easily port your campaigns, reports, and data from your existing vendor to VWO.

To make A/B testing earn the trust of businesses at larger, we sincerely hope the right architecture and approach are adopted by all vendors. In fact, our engineering team is ready and willing to help other vendors fix the issues. If this issue is fixed, everybody wins.

Again, if we’re wrong anywhere in the article or misinterpreted anything or if you know of better approaches, we invite corrections (and we will promptly update the article).

Update: As we were about to publish this post, Optimizely published an update on their blog on how they’re going to allow their customers to mask experiment names. But they don’t talk about the real issue of all planned, future and past campaigns (along with URLs) being discoverable by competitors and how the same approach leads to a large, bloated settings file that could slow down a website.