Avalanche 4.44 released

A couple of weeks ago, version 4.44 of Avalanche was released. It’s not a major (4.x) release, but some new features made it (on top of bug fixes). Here they are:

  • TCP Selective ACK (SACK) is now a fully supported feature (it was an alpha feature before 4.44). This will help making the TCP behavior of Avalanche closer to modern Operating Systems (Windows Vista+, Linux Kernel 2.16+).
  • DHCPv6 client – that one should be self-explanatory.
  • More of Eliptic Curve ciphers. I’m really excited about those because, even if in the past we supported a bunch of them, they required DSA keys. The new ones support RSA keys, which are much more common. Some more ciphers were implemented, including the popular AES128-GCM-SHA256. Here’s the full list:
    • AES128-GCM-SHA256
    • AES256-GCM-SHA384
    • DHE-RSA-AES128-GCM-SHA256
    • DHE-RSA-AES256-GCM-SHA384
    • ECDHE-RSA-AES128-SHA256
    • ECDHE-RSA-AES256-SHA384

There’s also an exciting new Alpha feature: TCP Vegas! A lot of people were asking for this mode modern Congestion Avoidance mechanism.

Posted in Releases | Tagged , , , | Leave a comment

Testing Server Name Indication (SNI) In The Lab

People have been saying that Internet will run out of IPv4 address space for a long time now, and somehow hosting companies always managed to cope. One of the “coping mechanisms” consists of regrouping several websites behind one IP address – either through NAT or just by hosting several websites on the same web server. In the past, this was not supported by browsers and servers if you wanted to encrypt the traffic: it wasn’t possible to have multiple certificates (and, therefore, domains) on the same IP address. This became a problem so RFC 3546 (“TLS Extensions”) introduced Server Name Indication (SNI).

This article explains how to test SNI in the lab. Avalanche supports this feature since release 4.39. I will not delve into methodologies specifically but instead how to setup an environment and configure Avalanche to use it. Customers will probably need to adjust this to mimic their own architecture, of course. Continue reading

Posted in Tutorial | Tagged , , , , , | Leave a comment

Avalanche Now Supports ETag and If-None-Match

With the release of version 4.37 this week, Avalanche now supports the ETag and If-None-Match concept. This will be highly useful for Content Delivery Network (CDN) testing as well as caching devices implementing this concept.

Let’s start with a little recap on the said concept.

What are Etag and If-None-Match?

Modern browsers help speeding up the loading of previously visited pages by caching as much static content as possible. If the header image, or the CSS template, or the JavaScript resources don’t change between two visits, it would be pretty dumb to download them each time you visit. It makes pages longer to load, generates more traffic on the network and more load on the servers.

Initially the browsers would “remember” the first time they downloaded a resource (like an image), and let the server know. They send an “If-Modified-Since: <some date>” header (which Avalanche also supports). The server would then compare with the local resource’s modified date (this is an attribute of the file on the file system). If the file was modified later than the client’s version (implying it’s a newer version of the same file), the server would return the resource, otherwise it would return a message saying the resource was not modified (and the browser would use the version it has in its cache).

If-None-Match (INM) does something similar, but in a smarter way. For INM to work, the server needs to “tag” the resources it sends. The first time you visit a website, the browser will retrieve the resources from the server. The server will return the content with an “ETag: <some value>” header. The browser stores the resource in its cache, along with the ETag value. The next time the resource is needed (for instance, the next time you visit the website), the browser will check against the server if the file is still the same: it will send the request to retrieve the resource along with an “If-None-Match” header with the same value as the server sent initially. When the server receives the request, it checks that value against its own. If it matches, it means the files are the same, so it will return a response “304 Not Modified” to let the browser know it can load the file from the cache. If the tags don’t match, it will return a “200 OK” and the new version of the file.

For instance, Bob requests the resource “spirent.png.” The server will return the image with an ETag header, for instance ETag: “123456789”. The next time Bob visits the page, the browser will send a request to the server to download “spirent.png”, with the header “If-None-Match: 123456789”. If the file hasn’t changed on the server, the server will return “304 Not Modified” and Bob’s browser will load the file from cache. Otherwise, the server will return the whole new image, with a different ETag, that the browser will save in its cache. Etc. ad nauseam.

Note: There’s no defined standard describing how to generate ETags. Each implementation can have its own method of doing so. Usually people will implement a collision-resistant hashing algorithm but it’s not required.

How does it work in Avalanche?

First, a caveat, to get that out of the way. Being a test tool for very high performance, Avalanche doesn’t “remember” ETags. Each Simulated User (SimUser) is like a fresh visit, there’s no simulated local browser cache. You can pre-define the ETags in your Action List, as I will show below, but it’s not automatic. It’s also more practical for CDN and caching device providers like this.

Let’s take a look at the server-side under “Transactions.” This is where we’ll define how we want to send the ETags. There are two ways (besides “None”): Incremental and Circulatory. The documentation is pretty clear on what they do, so allow me to quote it:

  • Incremental ETag: The server includes an ETag header in HTTP response messages, which starts with 1 and increments every number of milliseconds that you specify in the Interval field.
  • Circulatory ETag List: The server includes an ETag header in HTTP response messages, which gets assigned from the circulatory ETag List of strings every number of milliseconds that you specify in the Interval field.

In other words, when you choose “Incremental” Avalanche will generate the ETag value itself. Starting at 1 (i.e.: first header would be ETag: “1”) and incrementing by 1 every 1000 milliseconds (1 second). Note that this is since the start of the test, so if you have some delay at the beginning of your test (for ARPing, for instance), you won’t get a value of “1” in the first response (in my back-to-back tests it starts at 3).

When you choose “Circulatory” you can choose your own ETag values, and Avalanche will cycle through them. Using the default 1000 ms value, the returned header value will change every second.

Some Examples

Let’s create a few requests (with no If-None-Match yet) and see how the server behaves. I configured the clients to make 3 requests, at a 2 seconds interval.

Avalanche server-side Transaction configuration

Avalanche server-side Transaction configuration

This is the resulting PCAP:

Avalanche HTTP server returns the ETag values as configured.

Avalanche HTTP server returns the ETag values as configured.

It’s incrementing as expected! Now let’s see what happen when we do match. For this I need to add an “If-None-Match” header. To keep it simple I’ll make this static and the same for all requests, but it could be a variable or random, and different for each request of course. Server side stays the same, and this is the action list:

1 get <ADDITIONAL_HEADER="If-None-Match: \"7\"">
THINK <2000>
1 get <ADDITIONAL_HEADER="If-None-Match: \"7\"">
THINK <2000>
1 get <ADDITIONAL_HEADER="If-None-Match: \"7\"">

Note the escaping backslash characters. This is required because INM values are enclosed in double quotes, but the “ADDITIONAL_HEADER” as well, so we need to tell the backend parser we don’t want to close the quote at that point.

According to the previous PCAP, only the 3rd GET should match since the first response’s ETag is “3”, the second is “5” and the third one is “7.” Let’s see:

If-None-Match headers are sent, and the one we expected to match returns the correct 304 status code.

If-None-Match headers are sent, and the one we expected to match returns the correct 304 status code.

Looks fine! We can see (in blue) that the clients are now sending the “If-None-Match” value we configured in the Action List. The first two responses are a 200 OK since the ETag don’t match. However the third request sends a 304 Not Modified since the ETag match (and we can see there’s no content return, so we saved bandwidth and time).

There could be many more examples, so I’ll stop here. If you have specific questions, as usual feel free to contact me!

Image | Posted on by | Tagged , , , , | Leave a comment

Spirent Launches Avalanche NEXT

TL;DRGo here.

A Brief History of  Time Avalanche

When you look back at the history of Avalanche, you realize it was the first ever web testing stateful test tool – that could scale, anyway. It really picked up because it was the only way to test the first stateful firewalls of the time, mostly from major vendors such as Cisco and Check Point. The tools started in the R&D labs, you know, people who look at the bits and bytes and need a super-flexible tool because they are either developers or dedicated test engineers. And then quickly spread to be used in quality assurance, and by many other customers such as Service Providers.

As it grew Avalanche is probably the most widely used stateful test tool in the world. Some very large enterprises, with dedicated testing teams (such as banks or major car manufacturers, for instance) were also customers, but in much smaller numbers than R&D teams from Network Equipment Manufacturers. But these days, we see the need for Avalanche-like testing rise a lot in the Enterprise market.

We started to see that trend become more prevalent in the last couple of years. More and more Enterprise customers now want to test. The flurry of attacks we’ve seen in the past years might have something to do with that. Enterprises are starting to realize that you need to test as much as possible for many reasons – before buying, before changing the configuration, before upgrading, etc. This is a Good Thing ™. But it comes with challenges.

Challenges of a R&D GUI

When your daily job is to manage a corporate network, deal with security alerts, setup BGP Peers and all these tasks Network Administrators have to achieve, you are likely not expected to be an expert at testing. And when the GUI is made for Developers or expert Performance Testing Engineers, it can get tricky. Do we really need to put the IP Fairness Threshold or the HTTP Transactions per TCP connection values in your face? Probably not.

Another important area is “How do I test, anyway?” Again, is it the job of a Network Administrator to keep up-to-date with the latest Test Methodologies? Probably not. Companies like NSS Labs, ICSA, UNH or EANTC (kind of) share theirs, but there are a lot of them and they don’t tell you how to configure an Avalanche to match these testing advanced methodologies. When you buy from us you’ll always get somebody on the phone to help you, but you shouldn’t have to pick the phone in the first place, should you?

There are more points, but these are the two main ones. In short, the GUI need to be Methodology- or Test-Based, not a blank slate; and it should show only the settings that matter to non-test experts. The GUI should be simple, but not simplistic and still accommodate the needs of everyone. Easy, right?

Enter Avalanche NEXT

But we like a challenge at Spirent, and we took that one on. A few days ago, we announced Avalanche NEXT. It’s a web-based, test case-oriented GUI following the tenants of Responsive Web Design. You browse to the GUI, load a test case (we plan to provide a lot of those, and I wish I could share some of those already in the works!), pick where you want to send the traffic to and from, what kind of traffic and how to mix it, put some thresholds for pass/fail criteria, and then just run. At the end you get a scored based on how far off (or not) the test performed relatively to  your pass/fail criteria. You can also download the latest Applications and Cyber Security threats for the latest in testing twice a month!

My good friend Ankur Chadda has a ~30 minutes presentation of all this, available on Youtube. If you are interested in a shorter, less technical video, visit the Avalanche NEXT product page.

I will post more articles in the future about Avalanche NEXT, as this one is more an explanation as to why we initiated that project in the first place.

Posted in General, Releases | Tagged , , , | Leave a comment

Using the Action List To Create More Dynamic Tests

One of the key feature of Avalanche is the “Simulated User” concept, coupled with Action Lists. The Action Lists represent exactly that: a list of actions that each users assigned to it will execute. You can mix protocols, so it’s easy to simulate a user doing a DNS query, then do some HTTP, SMTP and Peer-to-Peer traffic.

The Action List comes with a macro scripting language. This gives users even more flexibility. I’ll give an example.

This Is What I Want, Do It

Last week I was in Germany working with a Next Generation Firewall vendor for an event we’re running this week. The idea was to get a given amount of source IP addresses (500) randomly going to a pool of 200 servers. The assignment was that the protocol should be HTTP, each user should retrieve 10 pages, and that half of the responses should be 64 bytes while the other half was 25KBytes. This would give us a nice mix of throughput and connection rate.

First, I didn’t really want to type 200 IP addresses. Only the last byte of the IP would change. The source:destination IP pairs should be random, so first we enabled the “Random” option in the Client/Subnets. Suddently, the source IPs were random, good.

Boom! Random source IP addresses. Just like that.

Boom! Random source IP addresses. Just like that.

I was still left with 200 IPs to assign. Fortunately, I didn’t have to, thanks to these few lines of script:

ASSIGN VARIABLE <myFirstOctets “10.1.0.”>
ASSIGN VARIABLE <myDstIp myFirstOctets myLastOctet>

The first variable (“myFirstOctets”) defines the first three bytes of the IP address. The second variable (“myLastOctet”) assigns a value starting from 5 all the way up to 204. This value is incremented for each SimUser that gets assigned to that Action List. Then the last variable (“myDstIp”) is a concatenation of the first two variables. We then applied it like so:

1 GET http://<APPLY myDstIp>?Transaction-Profile=64B

On the wire this goes out like this:

Wireshark view of the request

Wireshark view of the request

(If you’re wondering why the IP layer is in red, read this article)

We have a random source, but an incremental destination. The reason for this is that the firewall vendor needed to ensure each and every server IP was used. Using a random value would not ensure that, but we could have made it random quite easily:

ASSIGN VARIABLE <myFirstOctets “10.1.0.”>
ASSIGN VARIABLE <myDstIp myFirstOctets myLastOctet>

See what I did there? “SEQUENTIAL” was changed to “UNIFORM”, and that’s it. Now the destination IP will be random between the two given bounds.

Posted in Tutorial | Tagged , , | Leave a comment

MATCH(NOT) Now 100% More Flexible

Avalanche is really good at testing firewalls, server-load balancers, UTMs and what-have-you. But it’s often overlooked how good it is to do “one-arm testing”, where only the clients are simulated. This is especially true for Web Application testing, because believe you me, if you can handle 1.2 million new connections per second, you can hurt websites quite a lot.

To keep improving this aspect of the product, Spirent introduced a new feature in the Avalanche 4.30 release: The ability to MATCH or MATCH_NOT against a variable.

TL;DR: Get the SPF and figure out yourself how it works. You’ll need the Application Testing feature for this to work, though.

But let’s first recap what MATCH is, because not many people use it.

Was ist Match?

MATCH and MATCHNOT are functions (sort of) that you can use in the Action List. They work only after level 1 HTTP requests (regardless of the method) and allow you to look for a specific text string in the response of that HTTP request. It’s useful to make sure the action you just did worked as expected. A typical use is something like this:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCH <”Welcome, “,1,BODY>

The above code is simple but let’s review it. I’m doing a level 1 HTTP Post against the /account/logon/ URL of the http://www.somewebsite.com host. I’m posting two variables (username and password), with hard-coded values. In a real scenario these values would be variable (come from a Forms Database for instance), but let’s keep it simple.

Now, the tester is supposed to know what happens when this log-on action succeeds: you get, somewhere in the page, a “Welcome, <username>” (John, in our example). The string that’ll be common to all users is “Welcome,”, and so this is what should be matched. If it doesn’t match, we’ll count this as “Match Failed” in the Avalanche statistics. You can’t match against a regular expression or anything like that. We do performance, and this comes with limitations. In that case, you can match only against a literal string.

The other function is the opposite: MATCHNOT. In that case, you don’t want to match the value, so a “Failed Match Not” means that you, in fact, matched what you didn’t want to. Let’s take the same example again:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCHNOT <”Wrong password or username”,1,BODY>

In the same scenario as before, we now that if we provide incorrect credentials we get the “Wrong password or username” error message. So we can make a check against that string, indicating we don’t want to see it.

Note that you can act on a failed MATCH/MATCHNOT. Your options are to stop the current user (if the credentials are wrong, no need to even try moving down the action list) or stop the whole test. Simply add this;

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCH <”Welcome, “,1,BODY>

Or this:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCHNOT <”Wrong password”,1,BODY>

In the first example, we’ll “kill” the Simulated User in case we can’t match “Welcome” in the response following the POST. In the second example, we’ll stop the whole test in case we do match “Wrong password” in the response. STOP USER and STOP TEST are triggered only if the MATCH or MATCHNOT fail.

New in 4.30

One of the new features introduced by the 4.30 release is the ability use a variable for the value to MATCH against. Ok that phrase is a bit convoluted so let’s give an example:

    ASSIGN VARIABLE <myMatchString matchStrings.$1>
    1 get
    MATCH<<APPLY myMatchString>,1,BODY>
    NEXT ROW <matchStrings>

In that example you can see that I’m defining a variable called “myMatchString”, coming from the “matchStrings” Forms Database (you can tell it’s a forms db because of the “.$”). I get whatever value is first in the forms db, make a HTTP GET, and make a match against that variable. Then increment the row of the form db, and go through the loop again.

This is great because what this means is that now, all from a Forms Database, you can specify a list of URLs with that their response should MATCH. For instance we could have a Forms Database like this:

URI,Match Value

We could then make different MATCH for different pages, while still keeping the same action:

ASSIGN VARIABLE <myUri myFormDb.$1>
ASSIGN VARIABLE <myMatchValue myFormDb.$2>
1 GET http://www.somewebsite.com/<APPLY myURi>
MATCH<<APPLY myMatchValue>,1,BODY>

This is just an example but the possibilities are endless! I created a demo SPF showing some of these examples, feel free to play with it. Note that you’ll need the Application Testing license to use MATCH.

Posted in Uncategorized | Tagged , , , , | Leave a comment

Installing Avalanche Anywhere on CentOS 64 bits

A couple of month ago, Spirent released Avalanche Anywhere (as well as STC Anywhere). The goal of these products is to give our customers the ability to put the Avalanche (or STC) backend on any Linux distribution. For support purposes we officially support Red Hat Enterprise Linux (RHEL) and Fedora.

This blog isn’t here for me to make sales pitch, but I will say that it’s great being able to put Avalanche… well… anywhere! Our marketing team is so good at naming things it’s unsettling!

For this article we’ll install Avalanche Anywhere (AvA) on CentOS 6.3. I’m using CentOS because it’s basically RHEL only free.

Preparing System

Installing CentOS itself is outside the scope of this article. This video right here should be more than enough. Linux got confusingly easy to install in the last few years (hey, don’t laugh Systems people, this is a Networks people blog!).

Install CentOS (6.3 minimal install is used in this document). The system needs at least two NIC, one for admin of Anywhere (this can be the same as the main IP of the system), one for load generation (this one should be dedicated to load generation). You need at least 2 GB of RAM (4 GB is recommended) and one core.

The very first thing to do is to disable SE Linux by editing its configuration:

[root@ava-1 /]# vi /etc/selinux/config

In that file, set its mode to « disabled » like here:

# This file controls the state of SELinux on the system.
 # SELINUX= can take one of these three values:
 #     enforcing - SELinux security policy is enforced.
 #     permissive - SELinux prints warnings instead of enforcing.
 #     disabled - No SELinux policy is loaded.
 # SELINUXTYPE= can take one of these two values:
 #     targeted - Targeted processes are protected,
 #     mls - Multi Level Security protection.

(Knowing how to use vi is outside the scope of this article)

We also have to disable iptables. We can do it temporarily like so (recommended):

[root@ava-1 ~]# /etc/init.d/iptables stop

Or permanently (not recommended, unless the system will be dedicated to load testing):

[root@ava-1 ~]# chkconfig iptables off

Once this is installed, several extra packages need to be added:

Note: If you didn’t install CentOS minimal, some of these packages are probably already present on the system.

  1. kernel-devel or kernel-PAE-devel if you use a Physical Address Extension-enabled Kernel (if you’re not sure, you probably don’t use that). If your OS is sitting in a Xen-like VM, you will need kernel-xen-devel instead.
  2. Gcc
  3. Make
  4. Openssh-clients
  5. Expect
  6. Xinetd
  7. Ntp
  8. Glibc
  9. Libstdc++
  10. perl

First we’ll install the packages. Type this:

[root@ava-1 ~]# yum install gcc make openssh-clients expect xinetd ntp glibc.i686 libstdc++.i686 perl

This will check for all these packages and their dependencies. The final summary should look like something akin to this:

Transaction Summary
================================================================= Install      27 Package(s)
 Upgrade       2 Package(s)

Total download size: 56 M
 Is this ok [y/N]:y

Once this is installed, we can move on to install the kernel-devel module.

First upgrade the current kernel:

[root@ava-1 ~]# yum upgrade kernel

Then reboot (“reboot”) and install the kernel-devel:

[root@ava-1 ~]# yum install kernel-devel

Activating Avalanche Anywhere

We now need to transfer the Avalanche Anywhere zip file to the server. There are many ways to do this but the easiest is to do this over SSH. FileZilla allows just that – FTP over SSH – and you don’t need to set anything else to work:


Transfer the file (the name should be something like “ava-4.10.2196-fs-1.1.tgz”), to the /root directory for instance if you are logged in as root as in the example.  Once the file is transferred we need to make it executable:

[root@ava-1 ~]# chmod +x ava-4.10.2196-fs-1.1.tgz.run

And then execute it.

[root@ava-1 ~]# ./ava-4.10.2196-fs-1.1.tgz.run
 Verifying archive integrity... All good.
 Uncompressing build/il/packages/fsimg/stca-4.10.2196-fs-1.1.tgz....
 Install STC Anywhere in your PC

 Stopping xinetd:                                         [  OK  ]
 Starting xinetd:                                         [  OK  ]


Please run admin.py before your first running.

[root@ava-1 ~]#

Avalanche Anywhere is now installed on the OS!

Configuring Avalanche Anywhere

We now need to configure it by running the admin.py script:

[root@ava-1 /]# /mnt/spirent/chassis/bin/admin.py

Do you want to reconfigure the admin.conf? y

Current config:


Please input license server address: []
Please input ntp server address: []
Please input port speed (100M/1G/10G): [100M]1G
Please input the port group size (1~1): [1]
Here is the list of ethernet interfaces:
1) eth0
2) eth1

Please choose the interface for admin (1-2): 1
1) eth0
2) eth1

Please choose the interface for test port No.1 (1-2): 2
Do you want to reconfigure the stca cpu affinity?n
Please restart stca to make the change take effect.

[root@ava-1 /]#

And now we just need to start stca to turn the daemon on!

[root@ava-2 ~]# stca start

Please make sure firewall (iptables) is disables.
Install sasm.ko ...OK
<quite a lot of stuff here, including error messages you can discard>

[root@ava-2 ~]#

That’s about it really! Now we simply need to add this newly added load generator to our GUI (“Administration/Spirent Test Center Chassis”):


Avalanche Anywhere in STC Administration tab.

Avalanche Anywhere in STC Administration tab.

Now to use this backend, it’s same as usual: reserve it and add it into a test. The nice thing is that since the test cases are 100% compatible between any Avalanche load generator (Virtual, STC-based, Appliance, etc.), using Anywhere can be a practical way to design tests on a local PC (or remote VMs). Then you only need to pass the test case onto the team or person that’ll do the actual Big Bad Load Test – and modify the load appropriately.

Posted in Tutorial | Tagged , , , , | Leave a comment