Data source

Questions about the operation or expectations of the NetStumbler software

Postby 138 » Thu May 30, 2002 3:04 pm

Just think...one day, 44 replies, 229 view...just to decide to use the text format.

Oh well, on to creating a database and writing an ASP or J2EE web interface.
138
Mini Stumbler
 
Posts: 17
Joined: Thu May 30, 2002 5:53 am

Access Database

Postby pmn929 » Thu May 30, 2002 3:11 pm

I posted this a while back, and because of the damn old posts not showing up, I will post it again.

I wrote this so I could easly manage my files.

http://66.206.4.16/SummaryImport/SummaryImport.zip

In the DB, there is a module with the code necessary for importing the summary files.
pmn929
Mini Stumbler
 
Posts: 12
Joined: Sat Apr 13, 2002 11:49 am

Re: Access Database

Postby blackwave » Thu May 30, 2002 3:21 pm

Originally posted by pmn929
I posted this a while back, and because of the damn old posts not showing up, I will post it again.

I wrote this so I could easly manage my files.

http://66.206.4.16/SummaryImport/SummaryImport.zip

In the DB, there is a module with the code necessary for importing the summary files.


Thanks downloaded it, just a question how big were your NS1 files? Have you tested with ~50MB files, and perhaps 100 users accessing the data at the same time? ... Thanks again!
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Postby blackwave » Thu May 30, 2002 3:21 pm

Originally posted by 138
Just think...one day, 44 replies, 229 view...just to decide to use the text format.

Oh well, on to creating a database and writing an ASP or J2EE web interface.


Thanks for all your help, I will continue this whenver I get the ticks in. :)
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Access DB

Postby pmn929 » Thu May 30, 2002 3:31 pm

No,
It only imports the summary files, which are a lot smaller than the NS1 format, and once it puts them in the table, it is much smaller than that.

Currently, that only has an import function. It is meant for an individual user really. But I could modify it and actualy give it an ASP front end.

An export option could also be added to export a full summary file.

What is the end result you are trying to achive?
A personal data manager, or the ability to have a user go to one place and get a massive file to use.
pmn929
Mini Stumbler
 
Posts: 12
Joined: Sat Apr 13, 2002 11:49 am

Re: Access DB

Postby blackwave » Thu May 30, 2002 3:45 pm

Originally posted by pmn929

What is the end result you are trying to achive?
A personal data manager, or the ability to have a user go to one place and get a massive file to use.


I have a lot of ns1 files, I have them all merged into 1 large ns1.. some of them don't have gps, most of them do.

1. upload an ns1 (ASP)
2. unarchive file if archived
0.||3. check this file against other files for redundancy
4. open ns1 and import into sql database

5. sort data, filter out junk, filter out non-gps data
6. export into ns1 file as masterfile

7. query database, and database reacts...
for example let's say i wanted to see all data on linksys mac addresses.. i would query, and i would return all data relevant.. let's say i wanted to narrow it to linksys mac addresses in calfornia... linksys mac addresses in california irvine.. etc..

8. after query is complete pack back into ns1.
9. download newly packed file.

I want this to happen live, or email the ns1 to an account.. etc.
But to do this it requires full automation...

If ns1's were not being used and supported I would be fine with summary and text files.. but if someone gives you an ns1, they expect and ns1 back... no one ever explained why ns1 was used in the first place, or why we even use it since it is not compressed and rather large compared to a summary file... :)

The only issue I am having with this mechanism is the ns1 file :) everything else is cake.
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Postby lincomatic » Thu May 30, 2002 4:37 pm

the ns1 files and the text files contain the same stuff, as far as i know...which is EVERYTHING including all the samples, so you can plot out the graphs of signal strength vs time.

but there is still this 60 byte optional chunk in .NS1 files that i don't understand.

the wi-scan files are a little stripped down...some fields missing but otherwise similar to text out.

and the summary files are just what you see in the UI at the top level...just 1 summary sample in time for each AP.

curse u BW! i'm supposed to be working but u got me playing detective today. i just have no self control these days! :rolleyes: i actually have a little C prog working now which can churn thru a whole .NS1 file now and print out tons of stuff like ESSID, BSSID, AP name, signal strengths, Lat/Lon, the whole kit & kaboodle. the only pieces left to the puzzle are cracking the date format and that mysterious 60byte optional chunk. but i guess in the current form there is enuf known to slice & dice and re-merge NS1 files.
~lincomatic
User avatar
lincomatic
Mini Stumbler
 
Posts: 1682
Joined: Tue Apr 16, 2002 12:53 am
Location: Tinsel Town

Postby blackwave » Thu May 30, 2002 5:32 pm

Originally posted by lincomatic
curse u BW!


Very cool... When I started stumbling I wondered what the differences were in the NS1 vs Summary, Text, etc.. I understand perhaps the formatting, choice of delimiter, etc... but now that we have pried this a little I am sincerely wondering why does ns1 exist?

Seriously comparing a large file:

30MB of binary gunk vs a 588KB flat file with EXACTLY the same information??? :confused: What?

Why isn't everyone trading Summary files instead of NS1 files, obviously it is a lot more efficient. I would like to know why.

why... why... why...
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Postby sparafina » Thu May 30, 2002 5:46 pm

Originally posted by blackwave
30[B]MB of binary gunk vs a 588KB flat file with EXACTLY the same information??? :confused: What?

why... why... why... [/B]


Not to bitch an whine, but the choice of a tab and square bracket delimited text file is also puzzling, as well as grouping signal, noise, and snr in the same field. I'll never attain even 1NF at this rate ;)
User avatar
sparafina
Mini Stumbler
 
Posts: 1428
Joined: Tue May 07, 2002 2:12 pm
Location: TX,MD,NY

Postby lincomatic » Thu May 30, 2002 5:49 pm

no..the summary files are way stripped down. maybe i didn't explain well enough above.

summary files only have 1 datapoint per AP...just like what u see in the main NS UI.

NS1 & Export/Text are basically equivalent to each other (except for the mysterious optional 60byte chunks in the NS1 files). they contain sample histories of signal strengths over time for each AP.

the Export/wi-scan format is similar to Text, but doesn't have certain fields which are NS-specific such as beacon, AP name, etc.

so it's all up to what your purpose is. if you don't need temporal data for each AP then summary files are all you need. personally, i am a pack rat, so i'd rather have everything just in case.

now why NS1 binary instead of Text? because the text format is HUGE. it repeats stuff over and over so it's much bigger than NS1 format even if you take out the advantage of NS1 being binary. for instance, in Text format, every time sample has SNR and Lat/Lon, but in the NS1 file, there is 1 SNR and one Lat/Lon for each AP (in the summary record). only signal & noise are kept for each time sample.
~lincomatic
User avatar
lincomatic
Mini Stumbler
 
Posts: 1682
Joined: Tue Apr 16, 2002 12:53 am
Location: Tinsel Town

Postby blackwave » Thu May 30, 2002 6:07 pm

Originally posted by lincomatic
so it's all up to what your purpose is. if you don't need temporal data for each AP then summary files are all you need. personally, i am a pack rat, so i'd rather have everything just in case.


Yes I certainly need to keep the integrity of the temporal information as accurate as possible. :)
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Postby lincomatic » Thu May 30, 2002 6:32 pm

btw, i successfully cranked your whole giant NS1 file thru my little program and it actually parsed all the AP's correctly! now if we can just crack those few unknown fields...
~lincomatic
User avatar
lincomatic
Mini Stumbler
 
Posts: 1682
Joined: Tue Apr 16, 2002 12:53 am
Location: Tinsel Town

Postby blackwave » Thu May 30, 2002 6:43 pm

Originally posted by lincomatic
btw, i successfully cranked your whole giant NS1 file thru my little program and it actually parsed all the AP's correctly! now if we can just crack those few unknown fields...


Sweetness lincomatic! You are the man! :) I am trying to finish up a few things here before jumping back into helping figure out stuff... :)
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

Check it...

Postby blackwave » Thu May 30, 2002 11:52 pm

I was checking the 2001 search area.. and found this:


Originally posted by Marius
Since the shouting about NMEA's name has died down, the answer to the original question is that the elevation is stored in the binary (NS1) files but it isn't output on the display or in the text files. The reasons for this are (1) it's not as useful as lat/long; (2) if I'm going up and down in a skyscraper then I probably have a really poor GPS fix; (3) I couldn't be bothered to put it in.



here:
Elevation and .ns file format?
http://forums.netstumbler.com/showthread.php?s=&threadid=114&highlight=format

Sooo now we know we also store elevation in the NS1 format, perhaps that is also what is in the mystery chunk? :)
-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

More old data...

Postby blackwave » Fri May 31, 2002 12:03 am

-=BW=-
User avatar
blackwave
 
Posts: 4507
Joined: Mon Apr 15, 2002 3:00 am
Location: SoCal, OC

PreviousNext

Return to Help

Who is online

Users browsing this forum: No registered users and 8 guests

cron