by Chris_Schear » Wed Sep 25, 2002 4:04 pm
-----------
FINDINGS
-----------
1. NetStumbler failed to report signals properly
Most of the access points and signals detected by NetStumbler w/Cisco NIC were reported incorrectly. Over half of the access points detected were of the vendor, "Symbol". Almost all of these access points reported a signal strength of "-10". Signal+ was always reported as "-9" or "-10" for ALL access points. Noise- was reported as "-100" for ALL access points. SNR was reported as "90" for almost all access points. It would seem NetStumbler reports signal strengths differently than MiniStumbler. NetStumbler also reports signal strengths differently than it has in previous versions. In version 0.3.30, the closer to "-100" you get, the stronger the signal is. A signal that reports as "-15" is further away. This is different compared to previous versions and the current version of MiniStumbler. In those versions, the closer you got to an access point, the lower the negative value would shrink. Standing mere feet away from an access point, NetStumbler reported a signal strength of "-97". When approximately 120 feet from an access point, line of sight, NetStumbler reported a signal strength of "-13". This would appear to be backwards.
2. NetStumbler maintained active 'ghost' access points
As has been reported by other users, several access point appear and remain in the active table, eternally reporting a signal strength of "-10" until the device is rebooted. Closing down NetStumbler and re-opening the application has no effect. Disabling and re-enabling the NIC has no effect. All access points reported as being made by vendor, "Symbol" suffered this issue. Access points with incorrect signal information also had incorrect data in every other field; SNR, SNR+, Signal+, Noise-, and Noise.
3. MiniStumbler w/Compaq NIC needed frequent attention
The Compaq WL110 NIC ceased to function on numerous occasions. 8 resets were necessary to restore active stumbling. Resets were facilitated by physically removing and re-inserting the NIC into the iPaq.
4. MiniStumbler reported the fewest access points
This may be the result of the Compaq WL110 having an inferior internal antenna compared to the Cisco NICs, no analysis has been performed. Regardless, the MiniStumbler was the least successful in finding access points.
4. Cisco NICs appear to have better antennas
The NetStumbler and AirMagnet devices were close to reporting the same access points. Only one broadcasted SSID was missing from the NetStumbler data. Although, almost 90% of the NetStumbled data is invalid due to misreporting of signals and associated information. AirMagnet logged more accurate and detailed information in all regards. The Cisco NICs appear to have more effective unidirectional capabilities. The Compaq NIC works at optimum reception when the back side of the NIC is facing the signal.
5. AirMagnet detected faster
For any discovered access point, the information was noted in AirMagnet before any other tool. AirMagnet was configured to scan each channel at 500ms intervals. Devices were detected by AirMagnet before NetStumbler or MiniStumbler.
6. AirMagnet detected more nodes
AirMagnet reported more nodes per SSID. For any SSID, AirMagnet reported several instances of multiple devices with the same SSID. These were usually access points that were not operating in infrastructure mode but rather, they were bridged together to extend local wireless coverage. NetStumbler and MiniStumbler reported NO access points in bridging mode. In cases were multiple access points were present with the same SSID, NetStumbler (and sometimes, MiniStumbler) would only report a single find. In one instance, 6 access points had the same SSID and were bridged together. AirMagnet reported 6 access points. NetStumbler and MiniStumbler reported only 1. AirMagnet also reported on 21 access points that were not broadcasting their SSID. This feature is outside the capabilities of NetStumbler and MiniStumbler. Only access points that broadcast SSIDs are detectable. AirMagnet is unique in this regard as it can sniff the available traffic and detect embedded SSID information in regular data exchanges. In short, AirMagnet can detect access points and any workstations associated to them even if SSID broadcasting is not present.
7. AirMagnet detected more than just access points
AirMagnet reported not only broadcast and non-broadcasting SSIDs of access points, but also any active devices on the discovered wireless networks. A total of 33 workstations were detected and reported on. SSID, device name, IP address, and other assorted information about each workstation was available. IP address data was also available for all access points discovered. AirMagnet successfully obtained any locally configured hostname of access points and/or workstations in its operation. A nice feature in AirMagnet as it will identify how many and which workstations are associated to which particular access point(s). The product also comes with a built-in promiscuous sniffing capabilities and partial packet decode features.
8. AirMagnet reported more information about access points
NetStumbler almost always failed to report signal/noise information accurately. AirMagnet was the only product that provided consistent and reliable data at all stages of the test. AirMagnet also reported on the following fields of information that NetStumbler and MiniStumbler do not: 802.1X/EAP Type, VPN/type, Auth. Algorithm, PCF/DCF, Channel agility, supported data rates, network mode, number of stations associated. AirMagnet also records detailed information about the traffic it sees from any access point or workstation, to include but not limited to: Management frame types and counts, association/disassociation and authentication/disauthentication requests, various RTS, PSP, CTS and ACK packet counts, how many 1, 2, 5, and 11mbit frames were seen, and provides graphical line charts that are updated in real time.
--------------
CONCLUSIONS
--------------
It is most obvious that you "get what you pay for" with regards to wireless products and services. With NetStumbler/MiniStumbler not being in the same playing field as AirMagnet, the quality received from a commercial/enterprise-grade product is most evident. The differences can most notably be seen in price over performance. NetStumbler and MiniStumbler are not without technical issues with various configuration setups, particular NICs, certain firmware and driver revisions. AirMagnet uses a custom flashed NIC to support the full gamut of form and function of its product. This gives the developers a bit more control. It also comes at a cost, in dollars.
NetStumbler and MiniStumbler cannot effectively be relied on to provide accurate analysis data. Whether you be concerned with signal strengths, access points that don't seem to "disappear" until you reboot your device, or network cards that repeatedly go to sleep, NetStumbler and MiniStumbler will often times fail to consistently perform. It is my opinion their usage will continue to be regarded as "novelty", while commercial/enterprise-grade products surpass their functionality and reliability. They will, undoubtedly in light of the fact they are free, be downloaded by the novice wireless user and employed in a more trivial fashion, centered around questionable activities. The demographic of NetStumbler and MiniStumbler will most likely continue along the path of the technical novice, charged with excitement at the prospect of being able to perform actual wireless surveying at little or no cost. There is cost, however, it comes in the form of reliability of the data and the scope/breadth of information obtained. NetStumbler and MiniStumbler offer very little with regards to meeting the needs of the network engineer who is charged with finding rogue or misconfigured access points. They will probably remain close favorites to the casual Internet-grown population who find value in the novelty.
-------------------
CLOSING COMMENTS
-------------------
While the obviously more expensive package of the three, and in some regards an unfair comparison, AirMagnet offers a more reliable means and more significant depth of data collection. It does what NetStumbler and MiniStumbler does and more. It also, does what they can do in a more effective way and with greater accuracy. As much as I have enjoyed the clean interface of NetStumbler and MiniStumbler, not to mention their price of usage, their inconsistent nature towards information reporting and the continued issues of support with particular hardware/software setups makes them a far less attractive solution. Each individual interested in wireless networks brings to the table specific motivations and goals. For the amateur who is excited by the prospect of driving around finding poorly configured access points, if for no other reason than to log them to a file and imports them into a large database, thus showing their "wardriving prowess" - NetStumbler and MiniStumbler hits right on the mark. For the network engineer who needs a high degree of accuracy and tools he can rely on, AirMagnet wins hands down. AirMagnet is a true "handheld, wireless network security analyzer". NetStumbler and MiniStumbler are little more than freeware applications akin to "NetBus probing tools". The frivolous activity of wardriving, has little to offer to the individuals interested in increasing their wireless security.
If more scrupulous individuals could afford AirMagnet, there would be a much greater wireless security threat, as a whole.