A comparision of NetStumbler, MiniStumbler, and AirMagnet.

A comparision of NetStumbler, MiniStumbler, and AirMagnet.

Postby Chris_Schear » Wed Sep 25, 2002 4:04 pm

I wanted to conduct some quick analysis regarding the usage of NetStumbler and MiniStumbler and compare the data against AirMagnet, to review the effectiveness of each product under normal operating conditions. In the past months, I have had ever growing issues and increased inconsistencies with using NetStumbler and MiniStumbler. I have had continual issues with certain combinations of software/hardware reporting particular access points with varying efficiency. I had never run all products, simultaneously, and compared the data. I submit, for your information, the findings of my analysis.

---------------
DEVICES USED
---------------

IBM Thinkpad : NetStumbler
- Cisco AIR-PCM350 PCMCIA NIC
- Version 5.02.005 ACU
- Version 8.2.3.0 NIC drivers from Cisco Systems
- Firmware V4.25.30

iPaq 3970 : MiniStumbler
- Compaq WL110 PCMCIA NIC
- Version 7.62 Build 622
- Client V2.10
- Firmware V4.4 (Primary)
- Firmware 8.42 (Station)

iPaq 3970 : AirMagnet
- Cisco AIR-PCM350 PCMCIA NIC
- Version 2.22.16 ACU
- Version 2.20.05 NIC drivers from AirMagnet
- Firmware V4.25.30

-----------------
ANALYSIS SCOPE
-----------------

Distance driven: 15 mi.
Total time: 42 min.

Starting from a downtown location in Des Moines, IA, I began my wireless survey at a local Park & Ride garage near the heart of the city. Most of the travel was conducted along I-253, with approximately 6 miles encompassed in residential areas and 2 miles in business zones. Three devices with three different sets of software running were employed during this testing. No add on antennas were used, all devices used reception abilities inherent to their own technology. Scan speeds for all platforms were configured as fast as possible. No GPS data was collected.


--------------------------
ACCESS POINT DISCOVERY
--------------------------

NetStumbler = 14 APs / 14 Unique SSIDs
MiniStumbler = 5 APs / 5 Unique SSIDs
AirMagnet = 16 APs / 25 Unique SSIDs (and 33 workstations + 2 Ad-Hoc clients)

NetStumbler
-----------------------
- IAWDSM01067
- IAWDSM01112
- IAWDSM01157
- IAWDSM01202
- IAWDSM01247
- IAWDSM01292
- IAWDSM01337
- l00ker4u
- linksys
- linksys
- links_sys
- pmon
- PI
- Wedgewood

MiniStumbler
------------------------
- IAWDSM01337
- looker4u
- linksys
- linksys
- link_sys

AirMagnet
------------------------
- IAWDSM01022
- IAWDSM01067
- IAWDSM01112
- IAWDSM01157
- IAWDSM01202
- IAWDSM01247
- IAWDSM01292
- IAWDSM01337
- l00ker4u (6 total)
- linksys
- linksys
- link_sys
- pmon
- PI
- Wedgewood
- zmtvs0999

***********************************************
* Access points discovered by AirMagnet
* that were NOT broadcasting SSID info
***********************************************
*
* G4 Network (2 Nodes, Apple) (Ad-Hoc)
* APs discovered w/out SSID broadcasting = 21
*
***********************************************
User avatar
Chris_Schear
 
Posts: 243
Joined: Mon Aug 05, 2002 12:54 pm
Location: WDM, Iowa

Postby Chris_Schear » Wed Sep 25, 2002 4:04 pm

-----------
FINDINGS
-----------

1. NetStumbler failed to report signals properly
Most of the access points and signals detected by NetStumbler w/Cisco NIC were reported incorrectly. Over half of the access points detected were of the vendor, "Symbol". Almost all of these access points reported a signal strength of "-10". Signal+ was always reported as "-9" or "-10" for ALL access points. Noise- was reported as "-100" for ALL access points. SNR was reported as "90" for almost all access points. It would seem NetStumbler reports signal strengths differently than MiniStumbler. NetStumbler also reports signal strengths differently than it has in previous versions. In version 0.3.30, the closer to "-100" you get, the stronger the signal is. A signal that reports as "-15" is further away. This is different compared to previous versions and the current version of MiniStumbler. In those versions, the closer you got to an access point, the lower the negative value would shrink. Standing mere feet away from an access point, NetStumbler reported a signal strength of "-97". When approximately 120 feet from an access point, line of sight, NetStumbler reported a signal strength of "-13". This would appear to be backwards.

2. NetStumbler maintained active 'ghost' access points
As has been reported by other users, several access point appear and remain in the active table, eternally reporting a signal strength of "-10" until the device is rebooted. Closing down NetStumbler and re-opening the application has no effect. Disabling and re-enabling the NIC has no effect. All access points reported as being made by vendor, "Symbol" suffered this issue. Access points with incorrect signal information also had incorrect data in every other field; SNR, SNR+, Signal+, Noise-, and Noise.

3. MiniStumbler w/Compaq NIC needed frequent attention
The Compaq WL110 NIC ceased to function on numerous occasions. 8 resets were necessary to restore active stumbling. Resets were facilitated by physically removing and re-inserting the NIC into the iPaq.

4. MiniStumbler reported the fewest access points
This may be the result of the Compaq WL110 having an inferior internal antenna compared to the Cisco NICs, no analysis has been performed. Regardless, the MiniStumbler was the least successful in finding access points.

4. Cisco NICs appear to have better antennas
The NetStumbler and AirMagnet devices were close to reporting the same access points. Only one broadcasted SSID was missing from the NetStumbler data. Although, almost 90% of the NetStumbled data is invalid due to misreporting of signals and associated information. AirMagnet logged more accurate and detailed information in all regards. The Cisco NICs appear to have more effective unidirectional capabilities. The Compaq NIC works at optimum reception when the back side of the NIC is facing the signal.

5. AirMagnet detected faster
For any discovered access point, the information was noted in AirMagnet before any other tool. AirMagnet was configured to scan each channel at 500ms intervals. Devices were detected by AirMagnet before NetStumbler or MiniStumbler.

6. AirMagnet detected more nodes
AirMagnet reported more nodes per SSID. For any SSID, AirMagnet reported several instances of multiple devices with the same SSID. These were usually access points that were not operating in infrastructure mode but rather, they were bridged together to extend local wireless coverage. NetStumbler and MiniStumbler reported NO access points in bridging mode. In cases were multiple access points were present with the same SSID, NetStumbler (and sometimes, MiniStumbler) would only report a single find. In one instance, 6 access points had the same SSID and were bridged together. AirMagnet reported 6 access points. NetStumbler and MiniStumbler reported only 1. AirMagnet also reported on 21 access points that were not broadcasting their SSID. This feature is outside the capabilities of NetStumbler and MiniStumbler. Only access points that broadcast SSIDs are detectable. AirMagnet is unique in this regard as it can sniff the available traffic and detect embedded SSID information in regular data exchanges. In short, AirMagnet can detect access points and any workstations associated to them even if SSID broadcasting is not present.

7. AirMagnet detected more than just access points
AirMagnet reported not only broadcast and non-broadcasting SSIDs of access points, but also any active devices on the discovered wireless networks. A total of 33 workstations were detected and reported on. SSID, device name, IP address, and other assorted information about each workstation was available. IP address data was also available for all access points discovered. AirMagnet successfully obtained any locally configured hostname of access points and/or workstations in its operation. A nice feature in AirMagnet as it will identify how many and which workstations are associated to which particular access point(s). The product also comes with a built-in promiscuous sniffing capabilities and partial packet decode features.

8. AirMagnet reported more information about access points
NetStumbler almost always failed to report signal/noise information accurately. AirMagnet was the only product that provided consistent and reliable data at all stages of the test. AirMagnet also reported on the following fields of information that NetStumbler and MiniStumbler do not: 802.1X/EAP Type, VPN/type, Auth. Algorithm, PCF/DCF, Channel agility, supported data rates, network mode, number of stations associated. AirMagnet also records detailed information about the traffic it sees from any access point or workstation, to include but not limited to: Management frame types and counts, association/disassociation and authentication/disauthentication requests, various RTS, PSP, CTS and ACK packet counts, how many 1, 2, 5, and 11mbit frames were seen, and provides graphical line charts that are updated in real time.

--------------
CONCLUSIONS
--------------
It is most obvious that you "get what you pay for" with regards to wireless products and services. With NetStumbler/MiniStumbler not being in the same playing field as AirMagnet, the quality received from a commercial/enterprise-grade product is most evident. The differences can most notably be seen in price over performance. NetStumbler and MiniStumbler are not without technical issues with various configuration setups, particular NICs, certain firmware and driver revisions. AirMagnet uses a custom flashed NIC to support the full gamut of form and function of its product. This gives the developers a bit more control. It also comes at a cost, in dollars.

NetStumbler and MiniStumbler cannot effectively be relied on to provide accurate analysis data. Whether you be concerned with signal strengths, access points that don't seem to "disappear" until you reboot your device, or network cards that repeatedly go to sleep, NetStumbler and MiniStumbler will often times fail to consistently perform. It is my opinion their usage will continue to be regarded as "novelty", while commercial/enterprise-grade products surpass their functionality and reliability. They will, undoubtedly in light of the fact they are free, be downloaded by the novice wireless user and employed in a more trivial fashion, centered around questionable activities. The demographic of NetStumbler and MiniStumbler will most likely continue along the path of the technical novice, charged with excitement at the prospect of being able to perform actual wireless surveying at little or no cost. There is cost, however, it comes in the form of reliability of the data and the scope/breadth of information obtained. NetStumbler and MiniStumbler offer very little with regards to meeting the needs of the network engineer who is charged with finding rogue or misconfigured access points. They will probably remain close favorites to the casual Internet-grown population who find value in the novelty.

-------------------
CLOSING COMMENTS
-------------------
While the obviously more expensive package of the three, and in some regards an unfair comparison, AirMagnet offers a more reliable means and more significant depth of data collection. It does what NetStumbler and MiniStumbler does and more. It also, does what they can do in a more effective way and with greater accuracy. As much as I have enjoyed the clean interface of NetStumbler and MiniStumbler, not to mention their price of usage, their inconsistent nature towards information reporting and the continued issues of support with particular hardware/software setups makes them a far less attractive solution. Each individual interested in wireless networks brings to the table specific motivations and goals. For the amateur who is excited by the prospect of driving around finding poorly configured access points, if for no other reason than to log them to a file and imports them into a large database, thus showing their "wardriving prowess" - NetStumbler and MiniStumbler hits right on the mark. For the network engineer who needs a high degree of accuracy and tools he can rely on, AirMagnet wins hands down. AirMagnet is a true "handheld, wireless network security analyzer". NetStumbler and MiniStumbler are little more than freeware applications akin to "NetBus probing tools". The frivolous activity of wardriving, has little to offer to the individuals interested in increasing their wireless security.

If more scrupulous individuals could afford AirMagnet, there would be a much greater wireless security threat, as a whole.
User avatar
Chris_Schear
 
Posts: 243
Joined: Mon Aug 05, 2002 12:54 pm
Location: WDM, Iowa

Postby blackwave » Wed Sep 25, 2002 4:16 pm

Originally posted by Chris_Schear
Scan speeds for all platforms were configured as fast as possible.


This may be a variable since it is possible that scan speeds are not constant to each software package.

Also you did not state if you were travelling at 35mph, which is the optimal speed for wardriving.

:)
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Postby Eyecannon » Wed Sep 25, 2002 4:34 pm

and kismet owns them all :p
No, officer, YOU are under arrest!

http://eyecannon.com/wardrive.html
User avatar
Eyecannon
 
Posts: 679
Joined: Mon Aug 05, 2002 3:28 pm
Location: Santa Monix

Postby TheSovereign » Wed Sep 25, 2002 5:32 pm

even if airmagnet was better airmagnet is 2000 DOLLARS!

netstumbler is donation please :)
i love netstumbler
SO SAYS TheSovereign
User avatar
TheSovereign
 
Posts: 658
Joined: Sun Jun 30, 2002 2:35 am
Location: chicago

Postby astcell » Wed Sep 25, 2002 6:25 pm

Did they all use the same antenna?
User avatar
astcell
 
Posts: 589
Joined: Tue Aug 06, 2002 10:12 am
Location: Southern CA

Postby blackwave » Wed Sep 25, 2002 6:26 pm

Originally posted by TheSovereign
i love netstumbler


I love NetStumbler/MiniStumbler!
Hail Marius, and Mini-Marius!
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Postby astcell » Wed Sep 25, 2002 6:31 pm

Maybe we need to rename Mini-Stumbler to Midget-Stumbler. :D
User avatar
astcell
 
Posts: 589
Joined: Tue Aug 06, 2002 10:12 am
Location: Southern CA

NetStumbler

Postby nashr » Thu Sep 26, 2002 4:15 am

Did you forget that NetStumbler is still in BETA???
Help! I've been Simpsonized!
User avatar
nashr
 
Posts: 1585
Joined: Fri Aug 09, 2002 6:12 am
Location: Virginia

Postby highfrequency » Thu Sep 26, 2002 4:24 am

I think this is a good comparison, however I have a possible explanation. One critical thing is to understand that there are great inconsistencies between the noise figures of different cards. This is because most of the WLAN PCMCIA cards are made from G 10 epoxy circuit board, which, from board to board, has inconsistent performance, regardless of how well the WLAn receiver chip sets are 'laid out'. The reason WLAN manufacturers use G10 is cost, although they would get much better performance from expensive 'microwave board' like FRR 10.
The sensitivity of a WLAN receiver is usually rated by noise figure (NF) which is specified in dB. The effective noise temperature of a typical WLAN receiver using the Harris PRISM chip set is in the order of 4 dB. When the chip set is implemented on the cards, there may be up to 2.5 dB (or higher) increase in 'system' noise figure. This means that there is a difference in the sensitivity between individual cards as those with higher noise figure are less sensitive.
A 2.4 GHz receiver with a 1 MHz bandwidth and a noise figure of 4dB has a sensitivity of approximately minus 142 dBW. A noise figure of 6.5 dB gives a sensitivity of about minus 138 db.
Clearly, noise figure affects sensitivity, as does the relative atmospheric noise figure in the region where the measurments are taking place.

It is possible that the Air Magnet cards are 'hand-picked'.

It would be interesting to see the sensitivity tests reproduced when you run a 3.5 dB noise figure pre-amplifier ahead of each system (that's how I stumble). With a pre-amp, you will get more cionsistent results.
highfrequency
Mini Stumbler
 
Posts: 22
Joined: Sat May 25, 2002 10:48 am
Location: Vancouver

Postby mvario » Thu Sep 26, 2002 6:03 am

If you have the card I'd like to see results with NetStumbler using an orinoco card. I've noticed flaky reporting of signal strength when it's using the NDIS API, and others have reported other anomalous behaviour in this configuration.
mvario
 
Posts: 137
Joined: Fri May 24, 2002 6:53 pm
Location: NYC

Airmagent AVI demo

Postby packetattack » Thu Sep 26, 2002 7:13 am

It's funny that you posted this. I had done the same thing but not nearly as detailed. However my own results back up yours. Airmagnet works extremely well relative to NS.. but, NS is free and does GPS location data which makes it invaluable over AM for certain tasks. I use them both for what they are each good at.

Sidebar: my own testing was done on an iPAQ 3835 with the AM Cisco 352 card and Orinoco Gold card. The AM does not support an external antenna without hacking the tweaked card. Although I am in talks with the engineers about offering that as an option.

For those who want to see AM.. go to this link. I made up a couple of AVIs with a voice over. The first link is the direct link and the second is the wireless page that has the 2nd link.

http://www.packetattack.com/airmag/airmag_opening_wmv.html

http://www.packetattack.com/wireless.html

We have links to several wireless sniffers. I will have the NAI wireless sniffer in my hands on friday and will be able to report on that in a few days.

Wy-
"Take advantage of the enemy's unreadiness, make your way by unexpected routes, and attack unguarded spots."
Sun Tzu
packetattack
Mini Stumbler
 
Posts: 74
Joined: Wed Aug 21, 2002 9:47 am

Postby TheSovereign » Thu Sep 26, 2002 9:19 am

hey do u think marius will ever add detection for adhoc users?
cuz the way i get around being detected is i just set all my stuff to adhoc and leave the access point disconnected
SO SAYS TheSovereign
User avatar
TheSovereign
 
Posts: 658
Joined: Sun Jun 30, 2002 2:35 am
Location: chicago

Postby Chris_Schear » Thu Sep 26, 2002 12:51 pm

"Also you did not state if you were travelling at 35mph, which is the optimal speed for wardriving."
- Two comments. First, I would argue that is at all an accurate statement. In fact, I will. The key issue is that the client must be within radio range in order to receive any of the beacon announcements transmitted by an access point. It is conceivable that a client node may travel through the reception range in between anouncements and, thus, not detect an access point. SSID beacons are transmitted every 100ms. Now, considering an access point even transmitting at 1mW has an effective line of sight range of approximately 40 feet. You would have to be travelling so fast that you pass through that 40 feet in less than 100ms.

I will give bonus points to anyone who takes the time to do the simple math and determine how many miles per hour you have to be travelling to go 40 feet in 100ms. Again, this assumes lowest broadcasting power - which you would not normally experience. At 100mW, your effective radio range is 100-350 feet with a 2db gain antenna. Any math, be damned, you will NOT be driving fast enough to travel 300 feet in less than 100ms.

Secondly, you mention I did not state I was driving 35mph in my testing. I, again, invite you to the realm of simple math. I will even provide you with the formula to answer that question yourself. However, I'm sure you don't need this help.

D = R * T

Given D = 15 miles
Given T = 42 minutes

Solve for R.

"even if airmagnet was better airmagnet is 2000 DOLLARS!"
- It's more, actually. However, they will negotiate to $2000.00 USD if you buy in bulk.

"It is possible that the Air Magnet cards are 'hand-picked'."
- It is not a "possibility", it is most honestly a "certainty". They offer two cards with their solution and each one has been flashed with proprietary images. You MUST use their NICs or their software will not function.

"If you have the card I'd like to see results with NetStumbler using an orinoco card."
- I'm sorry. At present, I have only an Entrasys, Cisco, and Compaq card. I do plan on purchasing an Orinoco Gold in the near future. Upon doing so, I would be happy to re-test and publish the findings.

"but, NS is free and does GPS location data which makes it invaluable..."
- The next version of AirMagnet will have GPS integration. I have coordinated with their development staff on a great many new features to be in the next release. You can be almost certain, any new "bells and whistles" you find in the updated version have come from my "Wouldn't it be nice if..." ideas. The new version is currently in beta and is being refered to as "v1.5.1". It is intended on going into pre-release status next week. It contains bug fixes and enhancements, to include GPS addon ability.

"my own testing was done on an iPAQ 3835 with the AM Cisco 352 card and Orinoco Gold card. The AM does not support an external antenna without hacking the tweaked card."
- You need to purchase the AIR-LMC352, which is a PCM Card with MMCX ports for external antennas. The core PCM model requires hacking. You can even purchase an adapter cable for MMCX to RP-TNC with part ID "AIR-420-1625-0500".

"For those who want to see AM.. go to this link..."
- I had previously reviewed your video. It was very nicely presented. Excellent work.

The largest benefit ( in my opinion ) present in AirMagnet is their integrated AirWise system of detecting access points, bridges, and stations that are not broadcasting SSIDs. Not only can they detect these nodes and identify them as "Unknown SSID" - but given normal client activity, they can sniff out the SSID and "stumble" them, as well. Even if broadcasting is disabled, you can still snag SSID information. There are obviously other benefits to the software, but this is the one I find most valuable. The graphical representation of signal/noise and bleeding signal data is also quite useful in troubleshooting wireless LANs.

At present, AirMagnet has two modes of operation, "Expert" and "Survey". I have worked with AirMagnet at great detail in the past month, assisting them in developing a third mode termed "Sercurity". It is uncertain whether or not this feature will be in the next official release. It is not currently found in the beta.

I will be doing more controlled testing in the future, with more tightly regulated environments. It should be said that I do not consider this preliminary testing to be scientifically sound. There are many aspects to the testing which would be discounted by experienced sources of analysis. It is by no means without error.

My hopes are to provide more detailed analysis and findings in subsequent experiments. My goals in this one were rather superficial and limited. I merely wanted to see if the three products detected the same quantity of access points under "normal" conditions of usage. Future experiments will have more defined goals and parameters within the scope.

Any comments/suggestions or requests are welcome.
User avatar
Chris_Schear
 
Posts: 243
Joined: Mon Aug 05, 2002 12:54 pm
Location: WDM, Iowa

Postby blackwave » Thu Sep 26, 2002 4:19 pm

Greetings Chris_Schear,

Thank you for the effort you have put forth. :)

How about adding ISS Wireless Scanner™ to your tests :)
http://www.iss.net/products_services/enterprise_protection/vulnerability_assessment/scanner_wireless.php

as well as the GULPIT(TM)Toolkit
http://www.crak.com/gulpit.htm

... the more the merrier :)
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Next

Return to Windows

Who is online

Users browsing this forum: No registered users and 5 guests

cron