NetStumbler.org Forums - View Single Post

Scruge · 02-06-2006

@ Dutch

Actually I don't have any use for the csv converter, but because several others here had a need and it was easy to add to my NS1 and Wigle cache converter I proceeded with the addition. The stupid mistake I made was never testing the executables on machines that hadn't had dbase installed on them first.

I used to develop in xbase many years ago and about 6 months ago I dusted off one of my old compilers (db2k) to write a few programs for my own needs. Although I first I tried developing a few things with the Visual languages I had, I kept going back to xbase because it was 20-50x faster.

db2k is based on the 32 bit Borland Database Engine (BDE 5.1). It screams with speed being its all hard coded in assembler. As an example it will physically sort a 5 million record 1GB file in less than 5 minutes, typical is less than 3 mins depending on the sort key. It can build an index in less than 90 seconds on the same file. All test were on machines running 1.4ghz cpus or less. I've got another ap that has a very complex relational structure between several tables/dbf files. It processes network densities and some other things plus produces 4 different output files that runs almost linear with your cpu speed at ~ 1000 records per second per 1ghz. It takes my 1.1ghz machine ~1.4hrs to process 5 kk records.

However, from what I can tell it looks like my "Installshield Express" is out of date and therefore isn't including all necessary files when it compiles. The "Installshield" determines whether the executable is deployable. So I'll have to see what's necessary to fix that before I post anymore non functioning apps.

However if anyone wants to test the object code I can provide. You'll need to find a copy of "db2k" or "dbase 7.5" and install. There are several trial and student versions floating around. FYI don't bother with versions earlier like 5, 5.5 or 7 or later like vdb2.1, vdb2.5 etc. Borland sold the product off to DBI in the late 90s which resulted in a name version change.

edit...
Dutch I almost forgot.. yes it removes all duplicates from the kml output files and processed Frag's sample 3000+ nets .csv files in 2.5seconds.

02-06-2006	#11 (permalink)
Scruge Nyuk nyuk! Join Date: Jan 2005 Location: TX Posts: 1,380	@ Dutch Actually I don't have any use for the csv converter, but because several others here had a need and it was easy to add to my NS1 and Wigle cache converter I proceeded with the addition. The stupid mistake I made was never testing the executables on machines that hadn't had dbase installed on them first. I used to develop in xbase many years ago and about 6 months ago I dusted off one of my old compilers (db2k) to write a few programs for my own needs. Although I first I tried developing a few things with the Visual languages I had, I kept going back to xbase because it was 20-50x faster. db2k is based on the 32 bit Borland Database Engine (BDE 5.1). It screams with speed being its all hard coded in assembler. As an example it will physically sort a 5 million record 1GB file in less than 5 minutes, typical is less than 3 mins depending on the sort key. It can build an index in less than 90 seconds on the same file. All test were on machines running 1.4ghz cpus or less. I've got another ap that has a very complex relational structure between several tables/dbf files. It processes network densities and some other things plus produces 4 different output files that runs almost linear with your cpu speed at ~ 1000 records per second per 1ghz. It takes my 1.1ghz machine ~1.4hrs to process 5 kk records. However, from what I can tell it looks like my "Installshield Express" is out of date and therefore isn't including all necessary files when it compiles. The "Installshield" determines whether the executable is deployable. So I'll have to see what's necessary to fix that before I post anymore non functioning apps. However if anyone wants to test the object code I can provide. You'll need to find a copy of "db2k" or "dbase 7.5" and install. There are several trial and student versions floating around. FYI don't bother with versions earlier like 5, 5.5 or 7 or later like vdb2.1, vdb2.5 etc. Borland sold the product off to DBI in the late 90s which resulted in a name version change. edit... Dutch I almost forgot.. yes it removes all duplicates from the kml output files and processed Frag's sample 3000+ nets .csv files in 2.5seconds. Last edited by Scruge : 02-06-2006 at 09:29 AM.