Mapping accesspoints online and making gifs afterwards..

Dutch · 04-29-2004

Gents and ladies,

Thought this might be interesting, considering all the post recently regarding making maps via mappoint etc.

If one doesn't have Mappoint or similar, there is a way of doing it online, which is quite easy.. www.gpsvisualizer.com is a free service, where one can upload either a CSV file or use the simple waypoint formular, and it will make a map based on the coordinates. This map is a SVG (Scalable Vector Graphics) overlay on a GIF file, and it works down to street level. Best of all for us europeans, it can even make maps covering our local areas.

But you need to do some stuff first, in order to clean up the data from netstumbler, so it can be mapped correctly.

A) Export a summary from Netstumbler (File - Export - Summary)

B) Open the exported summary file in your favorite text-editor ( Edlin fans can rejoice, as it still works)
It will look something like this :

Code:

 
# $Creator: Network Stumbler Version 0.4.0
# $Format: wi-scan summary with extensions
# Latitude	Longitude	( SSID )	Type	( BSSID )	Time (GMT)	[ SNR Sig Noise ]	# ( Name )	Flags	Channelbits	BcnIntvl	DataRate	LastChannel
# $DateGMT: 2004-04-29
N 55.6345667	E 12.6406600	( rootsdkwlan )	BSS	( 00:03:2f:20:7b:37 )	10:13:13 (GMT)	[ 42 191 149 ]	# (  )	0011	00002000	100	540	13
N 55.6345650	E 12.6405667	( linksys )	BSS	( 00:0c:41:8c:25:f4 )	10:21:52 (GMT)	[ 20 69 49 ]	# (  )	0005	00000800	100	110	11
N 55.6330900	E 12.6377433	( Wireless )	BSS	( 00:a0:c5:5d:2f:46 )	10:25:42 (GMT)	[ 47 96 49 ]	# (  )	0001	00000040	100	110	6
N 55.6314233	E 12.6421383	( Michael & Mette )	BSS	( 00:0d:88:20:cc:ba )	10:31:36 (GMT)	[ 28 77 49 ]	# (  )	0011	00000040	100	110	6

We need to clean this up, so the gpsvisualizer understands it. What we need is the Lat/Lon coordinates, the station ID, and also the MAC address, and we need it comma-seperated.

C) First delete the four top lines, then search-and-replace the spaces with a comma, between the fields. Our file would then look like this :

Code:

N 55.6345667,E 12.6406600,( rootsdkwlan ),BSS,( 00:03:2f:20:7b:37 ),10:13:13 (GMT),[ 42 191 149 ],# (  ),0011,00002000,100,540,13
N 55.6345650,E 12.6405667,( linksys ),BSS,( 00:0c:41:8c:25:f4 ),10:21:52 (GMT),[ 20 69 49 ],# (  ),0005,00000800,100,110,11
N 55.6330900,E 12.6377433,( Wireless ),BSS,( 00:a0:c5:5d:2f:46 ),10:25:42 (GMT),[ 47 96 49 ],# (  ),0001,00000040,100,110,6
N 55.6314233,E 12.6421383,( Michael & Mette ),BSS,( 00:0d:88:20:cc:ba ),10:31:36 (GMT),[ 28 77 49 ],# (  ),0011,00000040,100,110,6

D) Now we need to remove the Type field, as we don't need it.. Search-and replace "BSS,(" with just a "(" and if there are peer type Wifi logged, do the same with the string "Ad-hoc,(". Now our file looks like this :

Code:

N 55.6345667,E 12.6406600,( rootsdkwlan ),( 00:03:2f:20:7b:37 ),10:13:13 (GMT),[ 42 191 149 ],# (  ),0011,00002000,100,540,13
N 55.6345650,E 12.6405667,( linksys ),( 00:0c:41:8c:25:f4 ),10:21:52 (GMT),[ 20 69 49 ],# (  ),0005,00000800,100,110,11
N 55.6330900,E 12.6377433,( Wireless ),( 00:a0:c5:5d:2f:46 ),10:25:42 (GMT),[ 47 96 49 ],# (  ),0001,00000040,100,110,6
N 55.6314233,E 12.6421383,( Michael & Mette ),( 00:0d:88:20:cc:ba ),10:31:36 (GMT),[ 28 77 49 ],# (  ),0011,00000040,100,110,6

E) Now we have to remove the N and E (or S & W) ID from the lat-lon coordinates. Gpsvisualizer expects N & E cooridnates to be positive, and S & W coordinates to be negative. So search-and-replace "N " and "E " with a "+" (or nothing, in effect deleting the "N " and the "E " prefix). And search-and-replace "S " and "W " with "-" if you are in that part of the world.

Now our file looks like this:

Code:

55.6345667,12.6406600,( rootsdkwlan ),( 00:03:2f:20:7b:37 ),10:13:13 (GMT),[ 42 191 149 ],# (  ),0011,00002000,100,540,13
55.6345650,12.6405667,( linksys ),( 00:0c:41:8c:25:f4 ),10:21:52 (GMT),[ 20 69 49 ],# (  ),0005,00000800,100,110,11
55.6330900,12.6377433,( Wireless ),( 00:a0:c5:5d:2f:46 ),10:25:42 (GMT),[ 47 96 49 ],# (  ),0001,00000040,100,110,6
55.6314233,12.6421383,( Michael & Mette ),( 00:0d:88:20:cc:ba ),10:31:36 (GMT),[ 28 77 49 ],# (  ),0011,00000040,100,110,6

F) We could clean up the rest of the file, as we don't need anything else, but if one is lazy, it's not necessary. Gpsvisualizer will just disregard the fields we don't need. Gps visualizer will make the map according to the top-left most coordinate and the bottomrightmost coordinate, so no need to specify anything else.. As an example, you can use the following data, containing 34 wifi-points that I found on a midday stroll near Copenhagen Airport, while U.S. Foggy Bottom Manager Colin Powell was visiting today ;-)

Code:

55.6345667,12.6406600,( rootsdkwlan ),( 00:03:2f:20:7b:37 ),10:13:13 (GMT),[ 42 191 149 ],# (  ),0011,00002000,100,540,13
55.6345650,12.6405667,( linksys ),( 00:0c:41:8c:25:f4 ),10:21:52 (GMT),[ 20 69 49 ],# (  ),0005,00000800,100,110,11
55.6330900,12.6377433,( Wireless ),( 00:a0:c5:5d:2f:46 ),10:25:42 (GMT),[ 47 96 49 ],# (  ),0001,00000040,100,110,6
55.6314233,12.6421383,( Michael & Mette ),( 00:0d:88:20:cc:ba ),10:31:36 (GMT),[ 28 77 49 ],# (  ),0011,00000040,100,110,6
55.6309600,12.6419483,( linksys ),( 00:0c:41:89:90:8a ),10:31:38 (GMT),[ 39 88 49 ],# (  ),0015,00000800,100,110,11
55.6329167,12.6424517,( default ),( 00:04:e2:60:72:b2 ),10:46:36 (GMT),[ 47 96 49 ],# (  ),0001,00000002,100,110,1
55.6326433,12.6434300,( default ),( 00:0d:88:36:0e:6b ),10:47:05 (GMT),[ 39 88 49 ],# (  ),0001,00000040,100,110,6
55.6320250,12.6447817,( Britter ),( 00:03:2f:1c:a7:59 ),10:48:29 (GMT),[ 55 104 49 ],# (  ),0015,00000040,100,110,6
55.6326750,12.6444117,( cph ),( 00:02:8a:97:c8:4f ),10:49:00 (GMT),[ 22 71 49 ],# (  ),0001,00000002,100,110,1
55.6330333,12.6441800,( cph ),( 00:02:8a:42:23:fa ),10:50:28 (GMT),[ 23 72 49 ],# (  ),0001,00000040,100,110,6
55.6346950,12.6446400,( NETGEAR ),( 00:09:5b:97:9f:ae ),10:55:26 (GMT),[ 31 80 49 ],# (  ),0421,00000800,100,540,11
55.6330167,12.6458250,( default ),( 00:0d:88:94:e7:6b ),11:02:17 (GMT),[ 36 85 49 ],# (  ),0461,00000040,100,540,6
55.6348183,12.6460217,( linksys ),( 00:06:25:98:50:a8 ),11:06:24 (GMT),[ 28 77 49 ],# (  ),0001,00000040,100,110,6
55.6348117,12.6457033,( Wireless ),( 00:a0:c5:5c:af:12 ),11:06:56 (GMT),[ 36 85 49 ],# (  ),0011,00000040,100,110,6
55.6346250,12.6442000,( RATHJ),( 00:03:2f:1c:8c:29 ),11:08:31 (GMT),[ 18 67 49 ],# (  ),0005,00000010,100,110,4
55.6347883,12.6433333,( linksys ),( 00:06:25:98:67:20 ),11:09:32 (GMT),[ 38 87 49 ],# (  ),0001,00000040,100,110,6
55.6364533,12.6426783,( steve ),( 00:a0:c5:46:7e:18 ),11:13:17 (GMT),[ 26 75 49 ],# (  ),0001,00000002,100,110,1
55.6366700,12.6434483,( Kastrupkrogen ),( 00:0c:41:18:65:0a ),11:13:27 (GMT),[ 31 80 49 ],# (  ),0011,00000040,100,110,6
55.6364717,12.6435667,( TDC ),( 00:a0:f8:51:63:a5 ),11:13:29 (GMT),[ 17 66 49 ],# (  ),0001,00000800,100,110,11
55.6366433,12.6435083,( Untitled ),( 00:e0:98:4c:3c:b2 ),11:13:45 (GMT),[ 34 83 49 ],# (  ),0431,00000400,100,540,10
55.6366433,12.6435083,( default ),( 00:03:2f:1f:48:f7 ),11:13:47 (GMT),[ 33 82 49 ],# (  ),0401,00000040,100,540,6
55.6366400,12.6422567,( birch ),( 00:80:c8:07:8d:f4 ),11:15:01 (GMT),[ 26 75 49 ],# (  ),0051,00000200,100,220,9
55.6369550,12.6424833,( TDC ),( 00:a0:f8:52:5e:ef ),11:17:14 (GMT),[ 20 69 49 ],# (  ),0001,00000800,100,110,11
55.6367883,12.6423517,( TDC ),( 00:a0:f8:51:63:dc ),11:18:34 (GMT),[ 20 69 49 ],# (  ),0001,00000040,100,110,6
55.6367883,12.6423517,( TDC ),( 00:a0:f8:51:64:fd ),11:18:37 (GMT),[ 20 69 49 ],# (  ),0001,00000040,100,110,6
55.6370300,12.6420750,( TDC ),( 00:a0:f8:51:64:e5 ),11:19:13 (GMT),[ 18 67 49 ],# (  ),0001,00000040,100,110,6
55.6370300,12.6420750,( TDC ),( 00:a0:f8:51:63:ae ),11:19:13 (GMT),[ 18 67 49 ],# (  ),0001,00000040,100,110,6
55.6373883,12.6416533,( TDC ),( 00:a0:f8:51:63:d5 ),11:19:48 (GMT),[ 20 69 49 ],# (  ),0001,00000800,100,110,11
55.6373883,12.6416533,( TDC ),( 00:a0:f8:51:63:f0 ),11:19:48 (GMT),[ 31 80 49 ],# (  ),0001,00000002,100,110,1
55.6384050,12.6403100,( COMPAQ ),( 00:50:8b:99:1d:c9 ),11:22:24 (GMT),[ 30 79 49 ],# (  ),0005,00000008,80,110,3
55.6377767,12.6351983,( default ),( 00:03:2f:20:01:53 ),11:29:19 (GMT),[ 25 74 49 ],# (  ),0005,00000040,100,110,6
55.6382783,12.6341733,( linksys ),( 00:06:25:bf:d2:d1 ),11:30:05 (GMT),[ 22 71 49 ],# (  ),0001,00000800,100,110,11
55.6376750,12.6355833,( Wireless ),( 00:a0:c5:5d:49:50 ),11:32:18 (GMT),[ 26 75 49 ],# (  ),0001,00000002,100,110,1
55.6367300,12.6363833,( Wireless ),( 00:a0:c5:61:09:92 ),11:34:19 (GMT),[ 36 85 49 ],# (  ),0401,00000040,100,540,6

G) Now for the fun part... Open a webbrowser and point it at the easy waypoinnts form on the gpsvisualizer site.
In the entry field for the waypoint data, change the first line to read:
latitude,longitude,name,description
then copy-paste the data from above underneath that line.

On the right side, change the max dimension to whatever size in pixels you like.. The higher the number, the more streetdetail you get.
Change the Background map selection to the Europe & US political map (street-level) entry.
If you want station ID and MAC address on the map, then select to show Waypoint names and Waypoint Descriptions.
Change the Waypoint Radius to 2 pixels.
Select "Open in a new window" under the waypoint field, and then click on the Draw the Map button. In a moment you will see the map in a new browser window. If you don't have the SVG plugin loaded, then it will install first.. (if it don't, then just use the link at the top of that page to install it.)
To save the map as a gif or jpg, maximise the browser window, press F11, and then just do a shift-printscreen (if you use Windows) to copy the current window to to the clipboard. Then paste it into your favourite graphicsprogram (or use Windows Paint), crop the picture so it only contains the map, and save in your pictureformat of choice, be it JPG, PNG, Gif or whatever.

Easy mapping, free as in gratis (but do support gpsvisualizer if you can.. Nice service, and deserves support).

Just my 0.02€'s worth...

Dutch

iceburn68 · 04-29-2004

Very Interesting, not bad for an online mapping client . . . good show!

G8tK33per · 04-29-2004

Quote:

Originally Posted by Dutch

F) We could clean up the rest of the file, as we don't need anything else, but if one is lazy, it's not necessary. Gpsvisualizer will just disregard the fields we don't need.

That being the case, why the manual steps in the beginning to remove unwanted fields?

Dutch · 04-29-2004

Quote:

Originally Posted by G8tK33per

That being the case, why the manual steps in the beginning to remove unwanted fields?

Because according to the information on the gpsvisualizer site, it needs the lat/lon fields first.. It disregards the fields there isn't any header information for in the first line. I would guess that one could just make a field header there, for the type field, which wouldn't get used, but didn't try it..
Found the site yesterday (google is your friend...

) and decided to test it out today after my stroll, so there might be other ways of doing it. Going to try to massage the data into shape for a csv upload on the main page, in order to test that later, and then write a VBscript macro to do it all in one go.
Should be fun to cut the steps down to
A) export summary from Netstumbler.
B) open summary in Word and running Macro
C) see map in browser.

Just my 0.02€'s worth...

Dutch

adamschneider · 04-29-2004

Quote:

Originally Posted by Dutch

Because according to the information on the gpsvisualizer site, it needs the lat/lon fields first.. It disregards the fields there isn't any header information for in the first line.

Just to clarify, The first line does need to have field names, but you can write your own header line, and the order of the fields doesn't matter, especially if you're plotting waypoints. (If you want to draw a track from plain-text, you would need to have a first column that indicated "T" for track, either as a field name or under a "type heading.")

If there's enough interest, I can certainly look into adding support for reading NetStumbler log files directly (using the file upload form, not the text-entry waypoints form). If you want to make that happen, send me a few sample files, both from the U.S. and from Europe; contact address is at the bottom of GPS Visualizer.

I also have a couple of questions:

1) Would you prefer to have your log files plotted as waypoints or as tracks? (Or both??) I would guess waypoints.

2) If they're waypoints, what should the "name" and "description" fields be? I would think the SSID for the name, but for the description, would it make more sense to use the MAC address, the SNR, or what? Keep in mind that the name and desc fields could also combine info from multiple NetStumbler fields, e.g., name = "linksys (42)", desc = "10:13:13 (GMT) / 00:03:2f:20:7b:37." But I'd need to make a decision and stick with it.

3) Is there a standard filename extension for NetStumbler log files, like ".nsl" or something? Usually, GPS Visualizer recognizes files by suffix.

adamschneider · 04-29-2004

By the way, if you manage to massage your data into a comma-separated file like Dutch did, try this: put the SNR in its own column with no brackets; label that field "N"; save it as a file whose name ends in ".csv"; and try this form: http://www.gpsvisualizer.com/map?form=data

The gimmick here is that the "N" field determines the SIZE of the waypoints; points with a stronger signal will appear larger than weak signals. It's not well documented yet, though; suffice to say the calculation is done differently if the minimum waypoint size is set to zero than if it's set to a number greater than zero.

G8tK33per · 04-29-2004

I suspect that a majority of Netstumbler users would prefer to have the plots as waypoints, I imagine there will be plenty of opinion as far as the 'name' and 'description' fields are concerned. The file extension is .ns1

Thorn · 04-29-2004

Quote:

Originally Posted by G8tK33per

I suspect that a majority of Netstumbler users would prefer to have the plots as waypoints, I imagine there will be plenty of opinion as far as the 'name' and 'description' fields are concerned. The file extension is .ns1

Just understand that .ns1 files are proprietary binary files.

adamschneider · 04-29-2004

Quote:

Originally Posted by Thorn

Just understand that .ns1 files are proprietary binary files.

Hmm, I didn't realize that. I guess if I had read Dutch's post more carefully, I'd have seen that what he posted was actually created by "File - Export - Summary." At any rate, those exported text files are what I'd be looking into supporting (and what I'd like to get a few samples of). I take it those don't have a special file suffix on them? Just "txt"?

Dutch · 04-29-2004

Quote:

Originally Posted by adamschneider

Hmm, I didn't realize that. I guess if I had read Dutch's post more carefully, I'd have seen that what he posted was actually created by "File - Export - Summary." At any rate, those exported text files are what I'd be looking into supporting (and what I'd like to get a few samples of). I take it those don't have a special file suffix on them? Just "txt"?

Netstumbler can export in a variety of formats. The summary is pure text, and only contain one entry Page Ranking accesspoint found. Which lat/lon coordinates or SNR data the entry uses from all the logged data Page Ranking accesspoint I don't know. Maybe Marius or some other in the knowhow can jump in here and tell if NS does some averaging of the data or just takes the first logged entry ?

If one exports in non summary text format, there will be a line for each logging of the accesspoint, where the lat/long can fluctuate due to movement (and GPS precision), as well as fluctuation in the SNR logged, due to distance to the accesspoint.

The same is true for the WiScan format export.

So these two formats are quite large, compared to the summary format. Their size depends on how much data is logged. As a comparision, the native .ns1 file from my "war-stroll" is 116 Kb, the summary is 4,34 Kb, the full text export is 177 Kb, and the WiScan export is 131 Kb.

I can send you samples of all three, as well as the binary native .ns1 format.
You need to contact Marius, and get the native .ns1 format definition, if you want to support that. My guess is that he wouldn't mind giving it to you, if you want to support NS natively.

As for the suffix to the file, it's up to one self to decide when exporting.. But yes, I use just .txt

Just my 0.02€'s worth...

Dutch

adamschneider · 04-29-2004

Please, send me all three files; I'll take a look at them and see what makes the most sense to support.

Adam

Dutch · 04-29-2004

Quote:

Originally Posted by adamschneider

Please, send me all three files; I'll take a look at them and see what makes the most sense to support.

Adam

Done!

Just my 0.02€'s worth...

Dutch

Thorn · 04-29-2004

Quote:

Originally Posted by Dutch

... You need to contact Marius, and get the native .ns1 format definition, if you want to support that. My guess is that he wouldn't mind giving it to you, if you want to support NS natively. ...

It is my understanding that Marius has supplied information on how to decode the .ns1 format to other people (notably the CoWF), but it will involve at least one NDA.

In the meantime, attached is a PERL file (GPS_vz.pl.txt) to convert a NS Summary file to the CSV for use with GPS Visualizer. (Basically, it does all of the conversion steps Dutch outlined above for you.) Make sure you rename it to GPS_vz.pl to run.)

To use:
1) Export a Summary file, naming it "FULLSUM.TXT"
2) Run the PERL script
3) The script will produce a GPS_vz.CSV file, containing the Lat, Long, SSID and MAC.
4) Upload the GPS_vz.csv file to GPS Visualizer.
5) Open favorite beverage and toast Thorn for removing drudgework.

uhtu · 04-29-2004

or just use wigle.net :-)

Dutch · 04-29-2004

Quote:

Originally Posted by uhtu

or just use wigle.net :-)

Apples and oranges....

I use wigle.net also, and have uploaded around 300+ newly discovered nets in copenhagen and surroundings the last couple of days. Wigle is great to get an overview of the nets in your area, but the webmap sucks when one wants to show people the problem of non secure accesspoints in their local area.
Coastal lines are off, no recognizable landmarks, etc etc.. People can relate to streetnames, they don't relate to areaoutlines....

With Adam's gpsvisualizer site, I can make a good hi-resolution map, with streetnames and show people where the problem exist in their area.. Chances are that they might know somebody on xxyy-street who have PC's and might use WLAN equipment. The chance that they might think "Hey.. it could be Bob on xxyy-street who runs unsecure" from the wigle.net webmap is almost non existant...

Remember, we are trying to get people to realize that setting a wlan up out-of-the-box is NOT a good idea.

By using Adam's free service, to make a map of the areas I check out, and provide it to the local community via the weekly newspapers, hanging a copy on the bulletinboard at the library, etc etc, I kind of feel I continue in the spirit of why Marius made Netstumbler free : To help in raising the awareness about (the lack of ) wlan security.

So thanks to Marius for making netstumbler, a great tool in its own right, and thanks to Adam for providing the gpsvisualizer service, another great tool in its own right. The two of them together provides a fantastic GIS tool for wardriver-persons who want to give a little back to their local community, and don't have the dough to invest in the commercial tools, such as Aeropeek, and Mappoint.

Just my 0.02€'s worth...

Dutch

P.S. Thorn, if you get to Copenhagen sometime, holler and I'll buy ya some good danish beer for the perl script.. You see, I would like to toast ya, but I hate to drink alone

04-29-2004	#2 (permalink)
iceburn68 Registered Member Join Date: Feb 2004 Posts: 206	Very Interesting, not bad for an online mapping client . . . good show! __________________ My Wardriving Website

04-29-2004	#6 (permalink)
adamschneider Registered Member Join Date: Apr 2004 Posts: 38	More mapping tricks By the way, if you manage to massage your data into a comma-separated file like Dutch did, try this: put the SNR in its own column with no brackets; label that field "N"; save it as a file whose name ends in ".csv"; and try this form: http://www.gpsvisualizer.com/map?form=data The gimmick here is that the "N" field determines the SIZE of the waypoints; points with a stronger signal will appear larger than weak signals. It's not well documented yet, though; suffice to say the calculation is done differently if the minimum waypoint size is set to zero than if it's set to a number greater than zero.

04-29-2004	#7 (permalink)
G8tK33per Asshole Emeritus Join Date: May 2003 Location: S.E. VA. Posts: 5,887	I suspect that a majority of Netstumbler users would prefer to have the plots as waypoints, I imagine there will be plenty of opinion as far as the 'name' and 'description' fields are concerned. The file extension is .ns1 __________________ "Butters, stop being such a pussy." Sons of Confederate Veterans

04-29-2004	#11 (permalink)
adamschneider Registered Member Join Date: Apr 2004 Posts: 38	Please, send me all three files; I'll take a look at them and see what makes the most sense to support. Adam __________________ adamschneider.net gpsvisualizer.com

04-29-2004	#14 (permalink)
uhtu Registered Member Join Date: Dec 2002 Posts: 17	just a little bit or just use wigle.net :-)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode