Using the Action List To Create More Dynamic Tests


One of the key feature of Avalanche is the “Simulated User” concept, coupled with Action Lists. The Action Lists represent exactly that: a list of actions that each users assigned to it will execute. You can mix protocols, so it’s easy to simulate a user doing a DNS query, then do some HTTP, SMTP and Peer-to-Peer traffic.

The Action List comes with a macro scripting language. This gives users even more flexibility. I’ll give an example.

This Is What I Want, Do It

Last week I was in Germany working with a Next Generation Firewall vendor for an event we’re running this week. The idea was to get a given amount of source IP addresses (500) randomly going to a pool of 200 servers. The assignment was that the protocol should be HTTP, each user should retrieve 10 pages, and that half of the responses should be 64 bytes while the other half was 25KBytes. This would give us a nice mix of throughput and connection rate.

First, I didn’t really want to type 200 IP addresses. Only the last byte of the IP would change. The source:destination IP pairs should be random, so first we enabled the “Random” option in the Client/Subnets. Suddently, the source IPs were random, good.

Boom! Random source IP addresses. Just like that.

Boom! Random source IP addresses. Just like that.

I was still left with 200 IPs to assign. Fortunately, I didn’t have to, thanks to these few lines of script:

ASSIGN VARIABLE <myFirstOctets “10.1.0.”>
ASSIGN VARIABLE <myLastOctet SEQUENTIAL 5 204>
ASSIGN VARIABLE <myDstIp myFirstOctets myLastOctet>

The first variable (“myFirstOctets”) defines the first three bytes of the IP address. The second variable (“myLastOctet”) assigns a value starting from 5 all the way up to 204. This value is incremented for each SimUser that gets assigned to that Action List. Then the last variable (“myDstIp”) is a concatenation of the first two variables. We then applied it like so:

1 GET http://<APPLY myDstIp>?Transaction-Profile=64B

On the wire this goes out like this:

Wireshark view of the request

Wireshark view of the request

(If you’re wondering why the IP layer is in red, read this article)

We have a random source, but an incremental destination. The reason for this is that the firewall vendor needed to ensure each and every server IP was used. Using a random value would not ensure that, but we could have made it random quite easily:

ASSIGN VARIABLE <myFirstOctets “10.1.0.”>
ASSIGN VARIABLE <myLastOctet UNIFORM 5 204>
ASSIGN VARIABLE <myDstIp myFirstOctets myLastOctet>

See what I did there? “SEQUENTIAL” was changed to “UNIFORM”, and that’s it. Now the destination IP will be random between the two given bounds.

Posted in Tutorial | Tagged , , | Leave a comment

MATCH(NOT) Now 100% More Flexible


Avalanche is really good at testing firewalls, server-load balancers, UTMs and what-have-you. But it’s often overlooked how good it is to do “one-arm testing”, where only the clients are simulated. This is especially true for Web Application testing, because believe you me, if you can handle 1.2 million new connections per second, you can hurt websites quite a lot.

To keep improving this aspect of the product, Spirent introduced a new feature in the Avalanche 4.30 release: The ability to MATCH or MATCH_NOT against a variable.

TL;DR: Get the SPF and figure out yourself how it works. You’ll need the Application Testing feature for this to work, though.

But let’s first recap what MATCH is, because not many people use it.

Was ist Match?

MATCH and MATCHNOT are functions (sort of) that you can use in the Action List. They work only after level 1 HTTP requests (regardless of the method) and allow you to look for a specific text string in the response of that HTTP request. It’s useful to make sure the action you just did worked as expected. A typical use is something like this:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCH <“Welcome, “,1,BODY>

The above code is simple but let’s review it. I’m doing a level 1 HTTP Post against the /account/logon/ URL of the http://www.somewebsite.com host. I’m posting two variables (username and password), with hard-coded values. In a real scenario these values would be variable (come from a Forms Database for instance), but let’s keep it simple.

Now, the tester is supposed to know what happens when this log-on action succeeds: you get, somewhere in the page, a “Welcome, <username>” (John, in our example). The string that’ll be common to all users is “Welcome,”, and so this is what should be matched. If it doesn’t match, we’ll count this as “Match Failed” in the Avalanche statistics. You can’t match against a regular expression or anything like that. We do performance, and this comes with limitations. In that case, you can match only against a literal string.

The other function is the opposite: MATCHNOT. In that case, you don’t want to match the value, so a “Failed Match Not” means that you, in fact, matched what you didn’t want to. Let’s take the same example again:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCHNOT <“Wrong password or username”,1,BODY>

In the same scenario as before, we now that if we provide incorrect credentials we get the “Wrong password or username” error message. So we can make a check against that string, indicating we don’t want to see it.

Note that you can act on a failed MATCH/MATCHNOT. Your options are to stop the current user (if the credentials are wrong, no need to even try moving down the action list) or stop the whole test. Simply add this;

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCH <“Welcome, “,1,BODY>
STOP USER

Or this:

1 POST http://www.somewebsite.com/account/logon username=john password=weaksauce
MATCHNOT <“Wrong password”,1,BODY>
STOP TEST

In the first example, we’ll “kill” the Simulated User in case we can’t match “Welcome” in the response following the POST. In the second example, we’ll stop the whole test in case we do match “Wrong password” in the response. STOP USER and STOP TEST are triggered only if the MATCH or MATCHNOT fail.

New in 4.30

One of the new features introduced by the 4.30 release is the ability use a variable for the value to MATCH against. Ok that phrase is a bit convoluted so let’s give an example:

LOOP myLoop START COUNT=10
    ASSIGN VARIABLE <myMatchString matchStrings.$1>
    1 get http://192.168.1.1/index.html
    MATCH<<APPLY myMatchString>,1,BODY>
    NEXT ROW <matchStrings>
LOOP myLoop STOP

In that example you can see that I’m defining a variable called “myMatchString”, coming from the “matchStrings” Forms Database (you can tell it’s a forms db because of the “.$”). I get whatever value is first in the forms db, make a HTTP GET, and make a match against that variable. Then increment the row of the form db, and go through the loop again.

This is great because what this means is that now, all from a Forms Database, you can specify a list of URLs with that their response should MATCH. For instance we could have a Forms Database like this:

URI,Match Value
account/login,Welcome
account/reset,Successfully
account/profile,Profile
(etc)

We could then make different MATCH for different pages, while still keeping the same action:

ASSIGN VARIABLE <myUri myFormDb.$1>
ASSIGN VARIABLE <myMatchValue myFormDb.$2>
1 GET http://www.somewebsite.com/<APPLY myURi>
MATCH<<APPLY myMatchValue>,1,BODY>

This is just an example but the possibilities are endless! I created a demo SPF showing some of these examples, feel free to play with it. Note that you’ll need the Application Testing license to use MATCH.

Posted in Uncategorized | Tagged , , , , | 2 Comments

Installing Avalanche Anywhere on CentOS 64 bits


A couple of month ago, Spirent released Avalanche Anywhere (as well as STC Anywhere). The goal of these products is to give our customers the ability to put the Avalanche (or STC) backend on any Linux distribution. For support purposes we officially support Red Hat Enterprise Linux (RHEL) and Fedora.

This blog isn’t here for me to make sales pitch, but I will say that it’s great being able to put Avalanche… well… anywhere! Our marketing team is so good at naming things it’s unsettling!

For this article we’ll install Avalanche Anywhere (AvA) on CentOS 6.3. I’m using CentOS because it’s basically RHEL only free.

Preparing System

Installing CentOS itself is outside the scope of this article. This video right here should be more than enough. Linux got confusingly easy to install in the last few years (hey, don’t laugh Systems people, this is a Networks people blog!).

Install CentOS (6.3 minimal install is used in this document). The system needs at least two NIC, one for admin of Anywhere (this can be the same as the main IP of the system), one for load generation (this one should be dedicated to load generation). You need at least 2 GB of RAM (4 GB is recommended) and one core.

The very first thing to do is to disable SE Linux by editing its configuration:

[root@ava-1 /]# vi /etc/selinux/config

In that file, set its mode to « disabled » like here:

# This file controls the state of SELinux on the system.
 # SELINUX= can take one of these three values:
 #     enforcing - SELinux security policy is enforced.
 #     permissive - SELinux prints warnings instead of enforcing.
 #     disabled - No SELinux policy is loaded.
 SELINUX=disabled
 # SELINUXTYPE= can take one of these two values:
 #     targeted - Targeted processes are protected,
 #     mls - Multi Level Security protection.
 SELINUXTYPE=targeted

(Knowing how to use vi is outside the scope of this article)

We also have to disable iptables. We can do it temporarily like so (recommended):

[root@ava-1 ~]# /etc/init.d/iptables stop

Or permanently (not recommended, unless the system will be dedicated to load testing):

[root@ava-1 ~]# chkconfig iptables off

Once this is installed, several extra packages need to be added:

Note: If you didn’t install CentOS minimal, some of these packages are probably already present on the system.

  1. kernel-devel or kernel-PAE-devel if you use a Physical Address Extension-enabled Kernel (if you’re not sure, you probably don’t use that). If your OS is sitting in a Xen-like VM, you will need kernel-xen-devel instead.
  2. Gcc
  3. Make
  4. Openssh-clients
  5. Expect
  6. Xinetd
  7. Ntp
  8. Glibc
  9. Libstdc++
  10. perl

First we’ll install the packages. Type this:

[root@ava-1 ~]# yum install gcc make openssh-clients expect xinetd ntp glibc.i686 libstdc++-devel.i686 perl

This will check for all these packages and their dependencies. The final summary should look like something akin to this:

=================================================================
Transaction Summary
================================================================= Install      27 Package(s)
 Upgrade       2 Package(s)

Total download size: 56 M
 Is this ok [y/N]:y

Once this is installed, we can move on to install the kernel-devel module.

First upgrade the current kernel:

[root@ava-1 ~]# yum upgrade kernel

Then reboot (“reboot”) and install the kernel-devel:

[root@ava-1 ~]# yum install kernel-devel

Activating Avalanche Anywhere

We now need to transfer the Avalanche Anywhere zip file to the server. There are many ways to do this but the easiest is to do this over SSH. FileZilla allows just that – FTP over SSH – and you don’t need to set anything else to work:

filezilla

Transfer the file (the name should be something like “ava-4.10.2196-fs-1.1.tgz”), to the /root directory for instance if you are logged in as root as in the example.  Once the file is transferred we need to make it executable:

[root@ava-1 ~]# chmod +x ava-4.10.2196-fs-1.1.tgz.run

And then execute it.

[root@ava-1 ~]# ./ava-4.10.2196-fs-1.1.tgz.run
 Verifying archive integrity... All good.
 Uncompressing build/il/packages/fsimg/stca-4.10.2196-fs-1.1.tgz....
 Install STC Anywhere in your PC

 Stopping xinetd:                                         [  OK  ]
 Starting xinetd:                                         [  OK  ]

Done

Please run admin.py before your first running.

[root@ava-1 ~]#

Avalanche Anywhere is now installed on the OS!

Configuring Avalanche Anywhere

We now need to configure it by running the admin.py script:

[root@ava-1 /]# /mnt/spirent/chassis/bin/admin.py

Do you want to reconfigure the admin.conf? y

Current config:

DEVICE=admin0
ADDR_MODE=dhcp
IPADDR=10.2.20.85
NETMASK=255.255.0.0
GATEWAY=10.2.0.1
HOSTNAME=
LSERVER=
NTPSERVER=10.100.15.140
NPORTGROUPS=1
NPORTS_PER_GROUP=1
DRIVERMODE=sockets
PORT_SPEED=100M
PROMISC=on

Please input license server address: []10.7.2.250
Please input ntp server address: [10.100.15.140]
Please input port speed (100M/1G/10G): [100M]1G
Please input the port group size (1~1): [1]
Here is the list of ethernet interfaces:
1) eth0
2) eth1

Please choose the interface for admin (1-2): 1
1) eth0
2) eth1

Please choose the interface for test port No.1 (1-2): 2
Do you want to reconfigure the stca cpu affinity?n
Please restart stca to make the change take effect.

[root@ava-1 /]#

And now we just need to start stca to turn the daemon on!

[root@ava-2 ~]# stca start

Please make sure firewall (iptables) is disables.
Install sasm.ko ...OK
<quite a lot of stuff here, including error messages you can discard>

[root@ava-2 ~]#

That’s about it really! Now we simply need to add this newly added load generator to our GUI (“Administration/Spirent Test Center Chassis”):

 

Avalanche Anywhere in STC Administration tab.

Avalanche Anywhere in STC Administration tab.

Now to use this backend, it’s same as usual: reserve it and add it into a test. The nice thing is that since the test cases are 100% compatible between any Avalanche load generator (Virtual, STC-based, Appliance, etc.), using Anywhere can be a practical way to design tests on a local PC (or remote VMs). Then you only need to pass the test case onto the team or person that’ll do the actual Big Bad Load Test – and modify the load appropriately.

Posted in Tutorial | Tagged , , , , | 2 Comments

HTML5 Is Now A Thing


HTML5_Badge_128After many years of debate and development (not sure if more development or debate happened), the W3C finally announced that the HTML5 specification is now feature complete. If you are interested (and that’d be a little sad), the full specification is available here. HTML5 is not a W3C Standard just yet as it still needs to pass all the interoperability and performance testing, but the hardest part is done. Work on HTML 5.1 has already started.

This news doesn’t mean a lot for the average user. HTML 5 was a draft for several years, and most browsers more or less implement it. Some of the not-so-implemented features are actually part of CSS 3.0. Like the border-radius. You can see moz-border-radius, webkit-border-radius and probably ie-border-radius. See, when an element (or style in that case) is not standardized yet, some people implement their own version. But since they can’t use the ‘real’ name, they prepend the name with their rendering engine – Firefox’s, Safari/Chrome’s or Internet Explorer in the previous examples. This is why some pages work better with some browsers than others. This is also why I will end up with an ulcer.

In any case, what does it mean for Avalanche ? Well, not much. HTML5 is application data  and Avalanche emulates protocols. Protocols are used to carry application data from one end to another. Avalanche couldn’t care less about the HTML version you’re using. So yes, you can test HTML5-enabled websites with Avalanche, no problem. Just like you can “test Ajax” – Ajax are just HTTP requests being send asynchronously from your browser. We can reproduce that. We don’t care if you use jQuery or Node.js or Prototype – in the end, what goes on the wire does it over HTTP, and we’ll do that.

Now HTTP 2.0 or SPDY 2/3.0 we’d really care about, but those are still drafts so we’ll have time to talk about that later.

Posted in General | Tagged , , | Leave a comment

No MOS in Avalanche Adaptive Bitrate – why?


Spirent has always been on the cutting-edge of video and audio testing. When the hot technology was Microsoft’s MMS, we added that. When RTSP rose, we implemented it with a myriad of player emulation (Microsoft, Apple, Real, BitBand …). When RTMP (Flash) video was the next best thing, we added that too, and so on and so forth.

When presenting the solutions to the customers, we always make it clear that in video/voice testing, you want to look not only at how many streams the system you test can handle, but also the quality of the streams. If your customers are plagued with bad video quality, they will not use your Video on Demand service any more. If they can’t understand what the person on the other side of the IP phone is saying, they will switch to a different VoIP provider.

This is why Avalanche (as well as Spirent Test Center and some of our other products) always implemented Quality of Experience (QoE) metrics. There are network layer metrics: the Media Delivery Index (MDI) -related stats; and the “human-level” metrics: the Mean Opinion Score (MOS). These are pretty much industry standard metrics that are totally relevant when testing RTSP and SIP.

But now we support Adaptive Bitrate (ABR) and…. We don’t provide MOS or MDI. And people are surprised, with reason. I was surprised too, at the beginning, until a discussion on our internal mailing list got me to think more about it. Let’s explore the reasons why we didn’t implement MOS and MDI for ABR, but let’s first recap what MOS means in the context of load testing.

What is MOS, again?

MOS is a score on a scale of 5 that reflects the quality of a video as a human would evaluate it. A score of 5 is “perfect” (and not achievable by design). A score below 1 is “please make it stop, my eyes are bleeding.” A typical good score is somewhere between 4.1 and 4.5.

As soon as a video is encoded, you can use tools to calculate its MOS. A good (usually one you have to pay for) encoder will give you a high score. A bad (that you sometimes also pay for) encoder will give you a bad score. I will not get into the details of how the score is achieved, but in two words it depends on your test methodology. Some people will compare the source and retrieved files (PEVQ). If you use R-Factor you’ll use codec and bitrate information and so on. There are other ways to calculate video quality, even when sticking to MOS.

When a video is streamed, the MOS on the receiving end cannot be higher than the source video. At best, in perfect network conditions, the MOS scores will be equal between the source and retrieved media. This is what you look for when looking at MOS during load tests: you’re looking at the evolution of the MOS (it shouldn’t get lower), not only its absolute value. If the source video MOS score is 2, it’s pretty bad, but if it’s still 2 when it reaches the client, your network is not degrading the quality: your network is good.

What makes MOS go down then? Typically, it’s packet loss. RTSP and Multicast video streaming typically use RTP/UDP for the data stream (RTSP, which is TCP-based, is only used for control). If you’re reading this blog, you know that UDP is an unreliable transport protocol – there’s no re-transmit feature, among other things. (People have been trying to work around that using RTCP, but it adds overhead, which is why RTP wasn’t based on TCP in the first place I think, so it’s not an ideal solution).

Why is it irrelevant then?

As we have just seen, in a live network, a MOS score will decrease due to bad network conditions because the underlying transport protocol (UDP) is unreliable. But Adaptive Bitrate is based on HTTP, which itself is based on TCP! And we know that TCP is a reliable protocol. There will be no packet loss – TCP’s retransmit mechanisms will kick-in to make sure you get that lost packet.

This means your clients video quality score will always be the same as the source, because ABR relies on TCP to make sure there’s no lost data. Therefore, measuring this is irrelevant.

But retransmits bring other problems. First, there is the overhead. Not much can be done about that. ABR is a technology that will favor quality over verbosity.

Then it takes time to re-transmit packets. You’ve got an extra round-trip to make when you re-transmit. On a fairly lossy network, the re-transmission will multiply and slow down the network. How will this manifest (pun intended) for the users? They will not have enough data (fragments) to keep playing the video without interruption. This is known as Buffering Wait Time. You don’t want that.

When this threatens to happen, the ABR client will tend to downshift to a lower bitrate. This is what makes this technology brilliant. As the name implies, it will adapt to the network conditions. This is what you want to look at. As we’ve seen, the video quality is a given. What is not a given, and a very good metric to look at, is the total number of downshifts. Or the total number of buffer underruns. Or the average Buffering Wait Time. And guess what, Avalanche measures all that!

One Metric To Rule Them All

People like to have one metric to simplify the results analysis, and they are right. While this metric cannot be as precise all looking in details at all the stats, it’s important to have it.

In Avalanche we call it the Adaptivity Score. We look at the total bandwidth used by the users, and compare it to the potential maximum bandwidth (that’s the maximum available bitrate multiplied by the number of users). We then rationalize it over 100.

Let’s have an example. If we have 10 users, connecting to an ABR server serving streams at bitrates of 1 Mbps and 500 Kbps. That’s a maximum potential bandwidth of 10 Mbps. If all 10 users are on the 1 Mbps stream, the score will be 100:

(current bitrate / max bitrate) * 100

((10×1 Mbps) / 10 Mbps) * 100 = 100

Now let’s pretend that half of the users go to the 500 Kbps stream.

(((5×0.5 Mbps) + (5x1Mbps)) / 10 Mbps) * 100 = 75

And since we do this calculation at every result sampling interval, you can analyze this after the test has been executed and get a nice graph.

In the example below I used Avalanche to emulate both the client and server of the Apple HLS implementation of ABR. I have a bunch of bitrates in the Manifest, and enabled a bitrate shift algorithm. The users are configured to start at the minimum possible bitrate and work their way up. The video lasts 5 minutes (to allow enough time to shift all the way up).

The first graph shows the Adaptivity Score. The second graph shows which bitrate “buckets” the users are streaming from. We can see that as the users go to higher bitrate channels the score goes higher.

Adaptivity Score over Time

This graphs shows the evolution of the Adaptivity Score over time.

Adaptive Bitrate "Buckets"

This graph shows that not all users start at the maximum bitrate – it takes some time for them to all shift up. This reflects on the Adaptivity Score.

And just for fun, here’s a screenshot of the throughput in that test. That’s almost 30 Gbps on a single device emulating both the clients and servers 🙂

Overall ABR Throughput (Back to Back)

This shows the overall ABR Throughput of a back to back test on a Spirent C-100 appliance.

Summary

If there is one thing to take from this article, it’s that in HTTP Adaptive Bitrate we know that thanks to TCP, all of the video data will reach the clients. There will be no lost data. We know that the quality of the video as viewed by the clients will be equal to the source. But the cost of this is that you might have an increased buffering time as packets are potentially re-transmitted.

The second part is that if a Service Provider wants to make sure its customers have the best possible experience, they need to make sure these clients can smoothly stream from the highest available bitrate. That’s your “quality of experience” measurement in ABR : how close to the maximum available bandwidth your clients can reach.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

FortiGate IPSEC VPN testing with Avalanche


Spirent recently launched the Spirent Test Center C1 small form-factor appliance, and I just received mine. I decided it would be a nice little machine to test the FortiGate 40C  UTM (also a small form-factor device) I just bought for demo purposes. With both the C1 and the FortiGate 40C, I have two tiny boxes that have the exact same features as the bigger ones we usually test.

This leads me to the topic of this entry: IPSEC testing. This technology is coming back strongly lately, mostly because of new LTE deployments. That seemed liked a good test case to have fun with. I will show how to configure both the FortiGate and the Avalanche to do an IPSEC Concurrent Tunnels Capacity test. Note that the blurred IP addresses are so because I try to obfuscate our corporate network details as much as possible 🙂

LAN Topology

The LAN subnet is 192.168.100.0/24. The gateway is .1, and I have a web server on .10. That’s pretty much all there is to it. Here’s the server-side association:

Server Associations

WAN Topology

The clients will come from the WAN, which is setup on the DUT as 192.168.128.0/24. Since I want to run an IPSEC concurrency test, I need a lot more addresses than what’s available on a /24 – this is because you can only have one IPSEC tunnel per IP address.

I will therefore use a Virtual Router, on the WAN side, and have my clients use a range in the 10.10.0.0/16 network. The VR IP address is 192.168.128.1, with a default route to 192.168.128.2 (the DUT IP on that network).

Client subnets – I use the highlighted one

Note that there is no Default Gateway set for the clients. This is because when using Virtual Routers, Avalanche will use the routes from the VR, not from the subnets.

Under Client/Ports, this is how the Virtual Router is configured:

The client-side Virtual Router

This is how the DUT is configured with regards to subnets:

FortiGate interfaces configuration

We also need to add a route to the clients’ subnet, so the DUT knows about that subnet. It’s set under Router/Static Routes:

FortiGate static routes – 0.0.0.0 goes to wan2 which is the admin port. We only need 10.10.0.0/16 for this test.

(You will notice I use ‘wan2’ as the management interface, so the default route goes there)

Now that we clearly see the network topology, onto IPSEC!

Configuring IPSEC

This is the Phase 1 configuration on the FortiGate. I will need to match it on the Avalanche.

The FortiGate IPSEC configuration

This is an easy configuration with many options disabled, but it serves its purpose. Let’s match it on Avalanche. For this we need to check the “IPSec” box under Client/Subnets. On the subnet, check “Enable IPSec”, and set the following options:

  • Check “Remote Access”
  • Vendor ID: Is ignored in before 4.10. This is used to customize the Vendor ID later on.
  • Remote Gateway: 192.168.128.2 in my case (the DUT’s WAN IP)
  • IKE Version: 1
  • DH-Group: I could choose 1 or 5 since both are enabled on the DUT.
  • Hash: SHA-1
  • Encryption: AES-CBC-128
  • Authentication: Preshared Key
  • XAuth: No XAuth
  • Preshared key: Value set on the DUT (“spirent” in my case)
  • ISAKMP ID is not used here

Then at the bottom half of the screen, we have more options. Under “IPSecIKEv1” I need to set the IKE Mode to “Aggressive” and leave the rest as default. Let’s now review the Phase 2 on the DUT:

To match this we need to go to the “IPSec” tab now. Leave all the options to default except:

  • Persistent Tunnel: If this box is checked, it tells the Avalanche not to tear-down the tunnel even if the SimUser that used it is dead. I’ll keep this checked since my test case is a concurrency test.
  • Hash: SHA-1
  • Encryption: ESP-AES-128
  • PFS (Perfect Forward Secrecy) : Disabled above, so this needs to be set to “None”

Finishing the Avalanche Configuration

I have absolutely no idea how many tunnels the FortiGate can handle. But I know the C1 can do 5,000 tunnels in that configuration, so this will be my goal. To match this, I just need to create a Load Profile with a “Sim Users” specification. I will use a Ramp Up of 300 seconds (that’s 17 tunnels per second!) and a steady phase of 120 seconds.

This means that the very first SimUser to be born will have to stay alive around 420 seconds. I need to make sure this will happen, and for this we usually use a Think Time (Client/Profile/HTTP User). I will set it to 50. What a think time does is force the SimUsers to wait (50 seconds in that case) between two actions (HTTP GETs in my case). I will create an Action List of 10 GETs, so it will take 500 seconds to go through it.

You don’t have to do this, since with the Persistent Tunnel option the tunnel will stay up even if the user has completed its action list. But I like to make sure all users do requests on a regular basis to make sure the DUT can keep forwarding traffic while handling the tunnels.

At the end, this is what the client association look like:

Final client assocations

Now, just need to run the test! Looks like the DUT likes what it’s seeing 🙂

And the Avalanche stats seem to agree!

Posted in General | Tagged , , , | 2 Comments

Adaptive Bitrate in Avalanche – Configuration & Stats


Starting with version 4.10, Spirent Avalanche officially supports Apple HLS (AHLS) as a server for Adaptive Bitrate (we supported the technology in a client-only fashion before then). That means you can emulate Apple HLS servers with Avalanche, generating random content. You can choose which bitrates to advertise in the Manifest, the Media length (duration of the video) as well as the Fragment length (how many seconds of video each fragment holds).

Server-side Setup (1 minute)

The server-side setup is very straightforward. Since ABR is based on HTTP, the tab to configure this is under Server/Transactions (where HTTP content is setup). You will notice a checkbox at the bottom (“HTTP ABR”) that you need to enable in order to start streaming.

Server Type: You can only choose Apple (HLS) here. In the future we will support Microsoft Smooth and Adobe Zeri.

Stream Type: VOD is for video on demand. The streams have got duration and you can use Trick mode on the client-side. Live emulates a live broadcast which by definition as no fixed duration and you cannot use trick mode.

Resource Name: The name of your playlist. If “Sample” is the name, our server will accept “Sample.m3u8” in the Manifest requests.

VOD Media Length: For VOD streams, the duration of the media. This way if you tell your clients to play until EOS (End of Stream) this setting can set the maximum duration of watching a video. The minimum value is 2 seconds. There is no maximum value.

Max Video Fragment Length: How many seconds of video each fragment holds. The “Max.” statement is here because the last fragment can potentially be shorter (ex: 15 seconds media. The first fragment will hold 10 seconds, the second one only 5 seconds.). The value cannot be longer than the Media Length. The minimum value is 2 (or the Media Length).

Then on the right side you can see a list of bitrates. These are the bitrates that we will announce in the Manifest. In the example above I have the maximum supported (1.2Mbps), then half (640 Kbps) and then minimum (64 Kbps). I used large differences because this way we can easily see the shifting.

To finish the setup, create a HTTP Server as usual (Server/Profiles) and choose the appropriate Transaction from the dropdown list. Then create an association that uses that Profile.

Client-side setup (3 minutes)

The client-side is also pretty easy to setup. Simply create an Action List, and then an Adaptive Streaming Profile. The window looks like this:

Server Type: Must match what you have on the other end. In our case it must be Apple because this is what we are emulating. But if you are in a client-only mode, we fully support Microsoft and Adobe’s solutions.

Stream Type: The actions and handling of the fragments are slightly different between VOD and Live streams for Avalanche. So please kindly specify which type you are connecting to. VOD in our case.

Reload Playlist Depth: This applies only to Apple HLS. When you use this technology, the client must download an updated playlist on a regular basis, because the original manifest doesn’t have the full list of fragments (unlike Microsoft Smooth). This parameter tells the client how many fragments before the last it needs to wait before requesting the manifest again. If you set this to 2, for instance, the client reloads the playlist before getting the next-to-last fragment in the current playlist. The maximum value is 5.

Manifest File Download Timeout: This is the maximum allowed time (in seconds) to retrieve the Manifest. The timer starts when the client issues the request. If the Manifest has not been downloaded in the specified amount of time, the session is aborted. The default value is 10, the maximum is 60. Using a value of 0 disables the timeout.

Starting Bitrate: This allows you to choose what the clients do when they are offered the Manifests. They can go for the Maximum available bitrate, the Median, the Minimum, or the one the closest to what you define (“User Defined”) .

Bitrate Shift Algorithm: This defines how the clients will shift. “Constant” means that no shifting will occur. “Normal” is based on the time it takes to download a fragment. For instance, if the next fragment has been downloaded after 80% of the current fragment has been played, a downshift will occur. If less than 50% have been played, an upshift will occur. If between these bounds, no shifting occurs. These values are customizable. “Smart” does pretty much the same, except it tries to calculate the available bandwidth and will go to the bitrate that is the closest. Refer to the on-line help for more details on this.

Once we are happy with the profile, we can just use it in the action list. The syntax is very simple:

as://192.168.1.1/Sample.m3u8 profile=”AdaptiveStreaming_0001″

Note that the IP address or the Playlist name can be a variable. The profile name needs to match what you created under Adaptive Streaming. We can now add the association and start the test!

Runtime Stats

There are a lot of very interesting stats for ABR. We will cover them in this section:

Channels per Interval: How many Channels are generated per second. A channel is not a stream. A channel is part of a stream, for instance it can be a video. If there’s also an audio track, that’s a different channel. A stream can have multiple channels. If your streams only have video, this will match.

Buffering Wait Time: Do you know that annoying time when you need to wait for your player to download enough video to start playing it? This is the buffering wait time. We measure the (min/max/average) time spent waiting for buffering aggregated for all the users, for the last 4 seconds.

Avg. Fragment Response and Download Time: This one is tricky. It measures the response time (time between the GET request and the first byte of data) as well as the download time (the time between the first and last byte of data), and add them up. Then it will check which bitrate the fragment was in. Then it compares the value to the other fragments in that “bucket.” Then it averages out the values of that bucket, and it shows it. It basically tells you the average time it takes to request and fully request fragments in the specified bitrate buckets.

Manifest per Interval: The amount of Manifests that are retrieved every second.

Fragments per Interval: The amount of Fragments that are retrieved every second.

Active Video Channels: Shows how many clients are connected to specific buckets of bitrate (<200kbps, 200-499kbps, 2000+ Kbps, etc.). Very cool to see them moving around as they shift J

Standalone Data: This one is a table, not a graph. It shows several stats:

  • Active Sessions: How many streams are currently active (should match the total amount of Active Video Channels)
  • Buffer Underruns: Shows how many times the users had to wait for the buffer to fill up (people had to “wait for their video”). You want this to be zero.
  • Total Active Channels: Tells you how many channels are active. As explained above, this number can be different than the amount of streams, since a stream can contain multiple channels (audio, video and subtitles, for instance)
  • Total Upshifts/Downshifts: How many times user had to Upshift or Downshift.
  • Total Rate Maintaining: Very interesting stat! This tells you how many fragments were retrieved, amongst all users, before any kind of shifting happened.

Summary and Real-time Stats

As we have seen, the Runtime stats are pretty good. As usual, the Summary and Real-Time ones are based on them. The Summary will show the totals. When you think summary, think “tables.” You will see how many upshift and downshift occurred, how many 200/OK status codes were received, and tons of other things.

Real-Time stat are something you should visualize as “Graphs.” They are very close to the Runtime stats, as they will show everything “per second.” Some of these stats are very interesting and not shown at runtime out of performance concerns.

One very interesting statistic is the “Adaptivity Score.” There are a lot of reasons why measuring Median Opinion Score (MOS) is not relevant in ABR, but is outside of the scope of this article. Since MOS is irrelevant, we came up with our own scoring mechanism. We take a look at the total current bandwidth for all the users, and compare it to the potential maximum available bandwidth (if they were all connected to the highest stream). We then rationalize this over 100 (to get a nice percentage). And that gives you a fairly good percentage of achieved bandwidth.

The exact formula is: (current bitrate / maximum potential bitrate) * 100%.

Summary

That’s pretty much it for this topic. ABR is a really cool technology, well thought-out and perfectly inline with the current needs of the people. Our support is pretty great (as I believe this post illustrates) and I’m really excited about the fact we can cover pretty much any test case where this technology is needed.

Trivia: I was talking with the BBC about ABR. They use this for streaming the Olympic Games. They expected a 3 Tbps of throughput at peaks on their CDNs. Friggin’ awesome  🙂

Posted in Tutorial | Tagged , , , , , , | Leave a comment