Introducing Harhar


Sometimes, you need to reproduce a typical HTTP user behavior under load. A typical example is the need to test a proxy with specific rules for specific pages – you need to ensure that applying a given rule won’t kill your performance, and the best way to check that is to test. Another reason would be to send a realistic traffic, representative of the typical traffic you see – so if you’re a web services host company, for instance, you test your infrastructures with the correct objects size and amount of requests.

Avalanche, as you might know if you read this blog, is totally able to do that and is even, arguably, the best tool for that. But you need to build the test by hand or by using the Fiddler plugin that I provide for free (source code is available on GitHub). There’s however a limitation with the Fiddler plugin: it will only generate a client Actions List. This means you can only test against real servers (which is fine) but you need to do some extra work if you want Avalanche to emulate the server side.

Enters Harhar

This is why I developed a tool called Harhar. This free (as in beer) tool parses .har (HTTP Archive) files generated by Chrome (Firebug-generated files support is under work) to create Actions Lists but also server objects, with the correct hierarchy. You simply need to upload the content on the Avalanche server-side and run. The tool comes with a detailed tutorial document so make sure to read it.

As a little background, HTTP Archives are files that store every requests from a HTTP client (URIs, cookies, etc) as well as the responses (including the content). The files are defined in a W3C Draft and are actually JSON content.

So feel free to experiment with this tool and let me know if you have any question, feedback or found any bug!

Posted in General | Tagged , , | Leave a comment

Avalanche 4.44 released


A couple of weeks ago, version 4.44 of Avalanche was released. It’s not a major (4.x) release, but some new features made it (on top of bug fixes). Here they are:

  • TCP Selective ACK (SACK) is now a fully supported feature (it was an alpha feature before 4.44). This will help making the TCP behavior of Avalanche closer to modern Operating Systems (Windows Vista+, Linux Kernel 2.16+).
  • DHCPv6 client – that one should be self-explanatory.
  • More of Eliptic Curve ciphers. I’m really excited about those because, even if in the past we supported a bunch of them, they required DSA keys. The new ones support RSA keys, which are much more common. Some more ciphers were implemented, including the popular AES128-GCM-SHA256. Here’s the full list:
    • AES128-GCM-SHA256
    • AES256-GCM-SHA384
    • DHE-RSA-AES128-GCM-SHA256
    • DHE-RSA-AES256-GCM-SHA384
    • ECDHE-RSA-AES128-GCM-SHA256
    • ECDHE-RSA-AES128-SHA256
    • ECDHE-RSA-AES256-GCM-SHA384
    • ECDHE-RSA-AES256-SHA384

There’s also an exciting new Alpha feature: TCP Vegas! A lot of people were asking for this mode modern Congestion Avoidance mechanism.

Posted in Releases | Tagged , , , | Leave a comment

Testing Server Name Indication (SNI) In The Lab


People have been saying that Internet will run out of IPv4 address space for a long time now, and somehow hosting companies always managed to cope. One of the “coping mechanisms” consists of regrouping several websites behind one IP address – either through NAT or just by hosting several websites on the same web server. In the past, this was not supported by browsers and servers if you wanted to encrypt the traffic: it wasn’t possible to have multiple certificates (and, therefore, domains) on the same IP address. This became a problem so RFC 3546 (“TLS Extensions”) introduced Server Name Indication (SNI).

This article explains how to test SNI in the lab. Avalanche supports this feature since release 4.39. I will not delve into methodologies specifically but instead how to setup an environment and configure Avalanche to use it. Customers will probably need to adjust this to mimic their own architecture, of course. Continue reading

Posted in Tutorial | Tagged , , , , , | Leave a comment

Avalanche Now Supports ETag and If-None-Match


With the release of version 4.37 this week, Avalanche now supports the ETag and If-None-Match concept. This will be highly useful for Content Delivery Network (CDN) testing as well as caching devices implementing this concept.

Let’s start with a little recap on the said concept.

What are Etag and If-None-Match?

Modern browsers help speeding up the loading of previously visited pages by caching as much static content as possible. If the header image, or the CSS template, or the JavaScript resources don’t change between two visits, it would be pretty dumb to download them each time you visit. It makes pages longer to load, generates more traffic on the network and more load on the servers.

Initially the browsers would “remember” the first time they downloaded a resource (like an image), and let the server know. They send an “If-Modified-Since: <some date>” header (which Avalanche also supports). The server would then compare with the local resource’s modified date (this is an attribute of the file on the file system). If the file was modified later than the client’s version (implying it’s a newer version of the same file), the server would return the resource, otherwise it would return a message saying the resource was not modified (and the browser would use the version it has in its cache).

If-None-Match (INM) does something similar, but in a smarter way. For INM to work, the server needs to “tag” the resources it sends. The first time you visit a website, the browser will retrieve the resources from the server. The server will return the content with an “ETag: <some value>” header. The browser stores the resource in its cache, along with the ETag value. The next time the resource is needed (for instance, the next time you visit the website), the browser will check against the server if the file is still the same: it will send the request to retrieve the resource along with an “If-None-Match” header with the same value as the server sent initially. When the server receives the request, it checks that value against its own. If it matches, it means the files are the same, so it will return a response “304 Not Modified” to let the browser know it can load the file from the cache. If the tags don’t match, it will return a “200 OK” and the new version of the file.

For instance, Bob requests the resource “spirent.png.” The server will return the image with an ETag header, for instance ETag: “123456789”. The next time Bob visits the page, the browser will send a request to the server to download “spirent.png”, with the header “If-None-Match: 123456789”. If the file hasn’t changed on the server, the server will return “304 Not Modified” and Bob’s browser will load the file from cache. Otherwise, the server will return the whole new image, with a different ETag, that the browser will save in its cache. Etc. ad nauseam.

Note: There’s no defined standard describing how to generate ETags. Each implementation can have its own method of doing so. Usually people will implement a collision-resistant hashing algorithm but it’s not required.

How does it work in Avalanche?

First, a caveat, to get that out of the way. Being a test tool for very high performance, Avalanche doesn’t “remember” ETags. Each Simulated User (SimUser) is like a fresh visit, there’s no simulated local browser cache. You can pre-define the ETags in your Action List, as I will show below, but it’s not automatic. It’s also more practical for CDN and caching device providers like this.

Let’s take a look at the server-side under “Transactions.” This is where we’ll define how we want to send the ETags. There are two ways (besides “None”): Incremental and Circulatory. The documentation is pretty clear on what they do, so allow me to quote it:

  • Incremental ETag: The server includes an ETag header in HTTP response messages, which starts with 1 and increments every number of milliseconds that you specify in the Interval field.
  • Circulatory ETag List: The server includes an ETag header in HTTP response messages, which gets assigned from the circulatory ETag List of strings every number of milliseconds that you specify in the Interval field.

In other words, when you choose “Incremental” Avalanche will generate the ETag value itself. Starting at 1 (i.e.: first header would be ETag: “1”) and incrementing by 1 every 1000 milliseconds (1 second). Note that this is since the start of the test, so if you have some delay at the beginning of your test (for ARPing, for instance), you won’t get a value of “1” in the first response (in my back-to-back tests it starts at 3).

When you choose “Circulatory” you can choose your own ETag values, and Avalanche will cycle through them. Using the default 1000 ms value, the returned header value will change every second.

Some Examples

Let’s create a few requests (with no If-None-Match yet) and see how the server behaves. I configured the clients to make 3 requests, at a 2 seconds interval.

Avalanche server-side Transaction configuration

Avalanche server-side Transaction configuration

This is the resulting PCAP:

Avalanche HTTP server returns the ETag values as configured.

Avalanche HTTP server returns the ETag values as configured.

It’s incrementing as expected! Now let’s see what happen when we do match. For this I need to add an “If-None-Match” header. To keep it simple I’ll make this static and the same for all requests, but it could be a variable or random, and different for each request of course. Server side stays the same, and this is the action list:

1 get http://192.168.1.1/firstEtag.html <ADDITIONAL_HEADER="If-None-Match: \"7\"">
THINK <2000>
1 get http://192.168.1.1/secondEtag.html <ADDITIONAL_HEADER="If-None-Match: \"7\"">
THINK <2000>
1 get http://192.168.1.1/thirdEtag.html <ADDITIONAL_HEADER="If-None-Match: \"7\"">

Note the escaping backslash characters. This is required because INM values are enclosed in double quotes, but the “ADDITIONAL_HEADER” as well, so we need to tell the backend parser we don’t want to close the quote at that point.

According to the previous PCAP, only the 3rd GET should match since the first response’s ETag is “3”, the second is “5” and the third one is “7.” Let’s see:

If-None-Match headers are sent, and the one we expected to match returns the correct 304 status code.

If-None-Match headers are sent, and the one we expected to match returns the correct 304 status code.

Looks fine! We can see (in blue) that the clients are now sending the “If-None-Match” value we configured in the Action List. The first two responses are a 200 OK since the ETag don’t match. However the third request sends a 304 Not Modified since the ETag match (and we can see there’s no content return, so we saved bandwidth and time).

There could be many more examples, so I’ll stop here. If you have specific questions, as usual feel free to contact me!

Image | Posted on by | Tagged , , , , | Leave a comment

Spirent Launches Avalanche NEXT


TL;DRGo here.

A Brief History of  Time Avalanche

When you look back at the history of Avalanche, you realize it was the first ever web testing stateful test tool – that could scale, anyway. It really picked up because it was the only way to test the first stateful firewalls of the time, mostly from major vendors such as Cisco and Check Point. The tools started in the R&D labs, you know, people who look at the bits and bytes and need a super-flexible tool because they are either developers or dedicated test engineers. And then quickly spread to be used in quality assurance, and by many other customers such as Service Providers.

As it grew Avalanche is probably the most widely used stateful test tool in the world. Some very large enterprises, with dedicated testing teams (such as banks or major car manufacturers, for instance) were also customers, but in much smaller numbers than R&D teams from Network Equipment Manufacturers. But these days, we see the need for Avalanche-like testing rise a lot in the Enterprise market.

We started to see that trend become more prevalent in the last couple of years. More and more Enterprise customers now want to test. The flurry of attacks we’ve seen in the past years might have something to do with that. Enterprises are starting to realize that you need to test as much as possible for many reasons – before buying, before changing the configuration, before upgrading, etc. This is a Good Thing ™. But it comes with challenges.

Challenges of a R&D GUI

When your daily job is to manage a corporate network, deal with security alerts, setup BGP Peers and all these tasks Network Administrators have to achieve, you are likely not expected to be an expert at testing. And when the GUI is made for Developers or expert Performance Testing Engineers, it can get tricky. Do we really need to put the IP Fairness Threshold or the HTTP Transactions per TCP connection values in your face? Probably not.

Another important area is “How do I test, anyway?” Again, is it the job of a Network Administrator to keep up-to-date with the latest Test Methodologies? Probably not. Companies like NSS Labs, ICSA, UNH or EANTC (kind of) share theirs, but there are a lot of them and they don’t tell you how to configure an Avalanche to match these testing advanced methodologies. When you buy from us you’ll always get somebody on the phone to help you, but you shouldn’t have to pick the phone in the first place, should you?

There are more points, but these are the two main ones. In short, the GUI need to be Methodology- or Test-Based, not a blank slate; and it should show only the settings that matter to non-test experts. The GUI should be simple, but not simplistic and still accommodate the needs of everyone. Easy, right?

Enter Avalanche NEXT

But we like a challenge at Spirent, and we took that one on. A few days ago, we announced Avalanche NEXT. It’s a web-based, test case-oriented GUI following the tenants of Responsive Web Design. You browse to the GUI, load a test case (we plan to provide a lot of those, and I wish I could share some of those already in the works!), pick where you want to send the traffic to and from, what kind of traffic and how to mix it, put some thresholds for pass/fail criteria, and then just run. At the end you get a scored based on how far off (or not) the test performed relatively to  your pass/fail criteria. You can also download the latest Applications and Cyber Security threats for the latest in testing twice a month!

My good friend Ankur Chadda has a ~30 minutes presentation of all this, available on Youtube. If you are interested in a shorter, less technical video, visit the Avalanche NEXT product page.

I will post more articles in the future about Avalanche NEXT, as this one is more an explanation as to why we initiated that project in the first place.

Posted in General, Releases | Tagged , , , | Leave a comment

Using the Action List To Create More Dynamic Tests


One of the key feature of Avalanche is the “Simulated User” concept, coupled with Action Lists. The Action Lists represent exactly that: a list of actions that each users assigned to it will execute. You can mix protocols, so it’s easy to simulate a user doing a DNS query, then do some HTTP, SMTP and Peer-to-Peer traffic.

The Action List comes with a macro scripting language. This gives users even more flexibility. I’ll give an example.

This Is What I Want, Do It

Last week I was in Germany working with a Next Generation Firewall vendor for an event we’re running this week. The idea was to get a given amount of source IP addresses (500) randomly going to a pool of 200 servers. The assignment was that the protocol should be HTTP, each user should retrieve 10 pages, and that half of the responses should be 64 bytes while the other half was 25KBytes. This would give us a nice mix of throughput and connection rate.

First, I didn’t really want to type 200 IP addresses. Only the last byte of the IP would change. The source:destination IP pairs should be random, so first we enabled the “Random” option in the Client/Subnets. Suddently, the source IPs were random, good.

Boom! Random source IP addresses. Just like that.

Boom! Random source IP addresses. Just like that.

I was still left with 200 IPs to assign. Fortunately, I didn’t have to, thanks to these few lines of script:

ASSIGN VARIABLE <myFirstOctets “10.1.0.”>
ASSIGN VARIABLE <myLastOctet SEQUENTIAL 5 204>
ASSIGN VARIABLE <myDstIp myFirstOctets myLastOctet>

The first variable (“myFirstOctets”) defines the first three bytes of the IP address. The second variable (“myLastOctet”) assigns a value starting from 5 all the way up to 204. This value is incremented for each SimUser that gets assigned to that Action List. Then the last variable (“myDstIp”) is a concatenation of the first two variables. We then applied it like so:

1 GET http://<APPLY myDstIp>?Transaction-Profile=64B

On the wire this goes out like this:

Wireshark view of the request

Wireshark view of the request

(If you’re wondering why the IP layer is in red, read this article)

We have a random source, but an incremental destination. The reason for this is that the firewall vendor needed to ensure each and every server IP was used. Using a random value would not ensure that, but we could have made it random quite easily:

ASSIGN VARIABLE <myFirstOctets “10.1.0.”>
ASSIGN VARIABLE <myLastOctet UNIFORM 5 204>
ASSIGN VARIABLE <myDstIp myFirstOctets myLastOctet>

See what I did there? “SEQUENTIAL” was changed to “UNIFORM”, and that’s it. Now the destination IP will be random between the two given bounds.

Posted in Tutorial | Tagged , , | Leave a comment

MATCH(NOT) Now 100% More Flexible


Avalanche is really good at testing firewalls, server-load balancers, UTMs and what-have-you. But it’s often overlooked how good it is to do “one-arm testing”, where only the clients are simulated. This is especially true for Web Application testing, because believe you me, if you can handle 1.2 million new connections per second, you can hurt websites quite a lot.

To keep improving this aspect of the product, Spirent introduced a new feature in the Avalanche 4.30 release: The ability to MATCH or MATCH_NOT against a variable.

TL;DR: Get the SPF and figure out yourself how it works. You’ll need the Application Testing feature for this to work, though.

But let’s first recap what MATCH is, because not many people use it.

Was ist Match?

MATCH and MATCHNOT are functions (sort of) that you can use in the Action List. They work only after level 1 HTTP requests (regardless of the method) and allow you to look for a specific text string in the response of that HTTP request. It’s useful to make sure the action you just did worked as expected. A typical use is something like this:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCH <“Welcome, “,1,BODY>

The above code is simple but let’s review it. I’m doing a level 1 HTTP Post against the /account/logon/ URL of the http://www.somewebsite.com host. I’m posting two variables (username and password), with hard-coded values. In a real scenario these values would be variable (come from a Forms Database for instance), but let’s keep it simple.

Now, the tester is supposed to know what happens when this log-on action succeeds: you get, somewhere in the page, a “Welcome, <username>” (John, in our example). The string that’ll be common to all users is “Welcome,”, and so this is what should be matched. If it doesn’t match, we’ll count this as “Match Failed” in the Avalanche statistics. You can’t match against a regular expression or anything like that. We do performance, and this comes with limitations. In that case, you can match only against a literal string.

The other function is the opposite: MATCHNOT. In that case, you don’t want to match the value, so a “Failed Match Not” means that you, in fact, matched what you didn’t want to. Let’s take the same example again:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCHNOT <“Wrong password or username”,1,BODY>

In the same scenario as before, we now that if we provide incorrect credentials we get the “Wrong password or username” error message. So we can make a check against that string, indicating we don’t want to see it.

Note that you can act on a failed MATCH/MATCHNOT. Your options are to stop the current user (if the credentials are wrong, no need to even try moving down the action list) or stop the whole test. Simply add this;

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCH <“Welcome, “,1,BODY>
STOP USER

Or this:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCHNOT <“Wrong password”,1,BODY>
STOP TEST

In the first example, we’ll “kill” the Simulated User in case we can’t match “Welcome” in the response following the POST. In the second example, we’ll stop the whole test in case we do match “Wrong password” in the response. STOP USER and STOP TEST are triggered only if the MATCH or MATCHNOT fail.

New in 4.30

One of the new features introduced by the 4.30 release is the ability use a variable for the value to MATCH against. Ok that phrase is a bit convoluted so let’s give an example:

LOOP myLoop START COUNT=10
    ASSIGN VARIABLE <myMatchString matchStrings.$1>
    1 get http://192.168.1.1/index.html
    MATCH<<APPLY myMatchString>,1,BODY>
    NEXT ROW <matchStrings>
LOOP myLoop STOP

In that example you can see that I’m defining a variable called “myMatchString”, coming from the “matchStrings” Forms Database (you can tell it’s a forms db because of the “.$”). I get whatever value is first in the forms db, make a HTTP GET, and make a match against that variable. Then increment the row of the form db, and go through the loop again.

This is great because what this means is that now, all from a Forms Database, you can specify a list of URLs with that their response should MATCH. For instance we could have a Forms Database like this:

URI,Match Value
account/login,Welcome
account/reset,Successfully
account/profile,Profile
(etc)

We could then make different MATCH for different pages, while still keeping the same action:

ASSIGN VARIABLE <myUri myFormDb.$1>
ASSIGN VARIABLE <myMatchValue myFormDb.$2>
1 GET http://www.somewebsite.com/<APPLY myURi>
MATCH<<APPLY myMatchValue>,1,BODY>

This is just an example but the possibilities are endless! I created a demo SPF showing some of these examples, feel free to play with it. Note that you’ll need the Application Testing license to use MATCH.

Posted in Uncategorized | Tagged , , , , | Leave a comment