Page 1 of 7

Data source

PostPosted: Thu May 30, 2002 4:35 am
by Oralis
*errrggg*

Not having much luck with figuring out the NS data format. Maybe there could be an ODBC created for it. In short this would give us access to the data without giving away the format secrets.


---Oralis

PostPosted: Thu May 30, 2002 5:59 am
by 138
Here are a few (not a full list of) hints:
1) Look at the file in a Hex Editor.
2) The SSIDs and Names are obvious, but notice the value immediately preceding them.
3) The MAC address is obvious.
4) Some values are derived, they aren't obvious.

PostPosted: Thu May 30, 2002 6:09 am
by Oralis
I did look at in both binary and hex editors ..
..and like you saw the ssid's and some other usefull info, but couldnt make heads or tails of the other particulars ..the other issue is, is the file locked and if we try to append data to the file will it fail to open in NS.


---Oralis

PostPosted: Thu May 30, 2002 6:26 am
by 138
That's because you haven't done enough programming to realize that not every file is delimited, fixed-size, or written with ASCII characters.

You can append data to the file if you know where to update the record count in the file.

More hints:
1) The file keeps track of how many records are in it.
2) How well do you know your binary representations of negative numbers?
3) Do you know the difference between Big and Little Endians?
4) The information to construct the graphs are contained in each record.

PostPosted: Thu May 30, 2002 6:42 am
by blackwave
Originally posted by 138
More hints:
1) The file keeps track of how many records are in it.
2) How well do you know your binary representations of negative numbers?
3) Do you know the difference between Big and Little Endians?
4) The information to construct the graphs are contained in each record.


Thanks for all your help and hints 138, the few clock ticks we get to research this makes this project move at a snails pace.

PostPosted: Thu May 30, 2002 6:50 am
by 138
I have about 80% of the format figured out, but I need more data files to finish. Need files with GPS coordinates. Need file with more than 256 records. Need to get off my lazy ass and figure out which format timestamps are stored in.

PostPosted: Thu May 30, 2002 6:52 am
by Oralis
I do admit to not being a "Programming" so this is all new to me.

PostPosted: Thu May 30, 2002 6:56 am
by blackwave
Originally posted by 138
I have about 80% of the format figured out, but I need more data files to finish. Need files with GPS coordinates. Need file with more than 256 records. Need to get off my lazy ass and figure out which format timestamps are stored in.


138 FYI:
Trading NS1 Files
http://forums.netstumbler.com/showthread.php?s=&threadid=1783

the latest NS1 we have is ~30MB Nationwide (US).. in Monday's update it will be larger still.

PostPosted: Thu May 30, 2002 6:57 am
by blackwave
Originally posted by Oralis
I do admit to not being a "Programming" so this is all new to me.


Don't worry Oralis, this is a learning process for everyone :)

PostPosted: Thu May 30, 2002 7:01 am
by acidrayne
Originally posted by blackwave


138 FYI:
[B]Trading NS1 Files

http://forums.netstumbler.com/showthread.php?s=&threadid=1783

the latest NS1 we have is ~30MB Nationwide (US).. in Monday's update it will be larger still. [/B]


I believe this may have been stated but I cant remember for sure ... Were you also interested in files that do not have GPS??

PostPosted: Thu May 30, 2002 7:06 am
by blackwave
Originally posted by acidrayne


I believe this may have been stated but I cant remember for sure ... Were you also interested in files that do not have GPS??


In short yes.

Once we have found out how to take this file apart and repack we plan to separate the gps and non gps data and put it in an alternate database for further analysis that is non-gps based.

It is good to analyze all the other data as well. All information is good. In the current master list we currently have 4577 logged APs with GPS and around 500 without GPS info.

PostPosted: Thu May 30, 2002 7:10 am
by acidrayne
I have a master file that currently has 205 and will keep growing ... would you prefer I just keep sending you updates of the master as I get them?

PostPosted: Thu May 30, 2002 7:12 am
by 138
A record appears to start with the length of the string containing the SSID, followed by the SSID, and MAC address. Next appears to be where NS stores the information that is used to construct the graphs. (Guessing) Sig+, Noise-, stuff I haven't *quite* figured out yet. These sub-records are repeated for each spike on the graph. The last information stored in the record is the length of the string representing the Name of the AP, followed by the Name of the AP represented as a String (if applicable).

IE - If the AP doesn't have a name, the record ends with 16 zeroes.

PostPosted: Thu May 30, 2002 7:16 am
by blackwave
Originally posted by acidrayne
I have a master file that currently has 205 and will keep growing ... would you prefer I just keep sending you updates of the master as I get them?


That would be wonderful thanks! :)

PostPosted: Thu May 30, 2002 7:20 am
by acidrayne
Originally posted by blackwave


That would be wonderful thanks! :)


Will probably get that to you tonight after work ... Maybe on lunch if I have time.