How Bytes Add Up

I've been working on a process which handles hand summarizes trade data from multiple markets. During development I didn't take much time on memory calculations because the structures used were light weight and the bulk of the data was persisted to a flat files.
With a code review approaching fast, I guessed it was time to do some mathematics. The result...

Worst case scenario

Market Count = 20
Instrument Count = 1000 (per partition)
Time = 8hrs. = 3600 * 8 s = 28800 s

Total Memory = 28800 * 20 * 1000 * 16 = 9,216,000,000B = 9.2 Gb

Yep..Even though this is the worst case, the figure was unacceptable. So I had to find a way to prove that in reality, the memory consumption of my process would be less.

The secret lay in the number of trades which occur on a given day. For the 10 or so markets that we connect to, this would be around 10 million. I took this down to 5 million to be on the safe side and it changed the result drastically.

Trade Count = 5,000,000

Memory = (16 * 5000000)B = 80,000,000 b = 80 MB

For the worst case, the trade count would be;

Trade count in order for this memory level to be reached = 28800 * 20 * 1000 =

576,000,000 = 576 million

So I can safely say that my process would operate at between 100MB and 1GB as long as the trade count remains below 50 million... Since it's unlikely that we will be receiving this many messages any time soon, the current architecture could be used for a long time to come.

If this fails we'll have to go for plan B, which is to move our indexes into files, and cope with having to do 2 file reads for each data segment retrieval...

Importing a resource from one MFC project to another

God only knows how I hate MFC. When I first came across it I was already a sort of a veteran at C# and .Net and had quite a bit of experience developing front ends using these technologies. My first impression on MFC was "Maaaaaan!!! this feels AAAAAnnnnnnncient". Yep... But since it was the platform being used at our company I had to learn to live with it...

So, it happens that I'm developing a front-ends for a product(I'm mainly a back end developer :)) and it has to be customized for different projects. Recently, I made an enhancement in one project and now I have to do almost the exact same thing for another one. Importing the controls from the original project seemed like the obvious solution.


It was a surprise to see how easy it was to do this in VS6. The steps are;

* In VC6, pick "Open" from the File menu.
* At the very bottom of the open file dialog is a combo box labeled
"open as", which says "Auto" by default.
* Change that so it says "Resources" instead.
* Locate the resource file, select it, and hit open
* Drag and drop the required control from the main window to your resource view.

These steps were adequate for me to get the control integrated with all the control IDs properly assigned. But sometimes the IDs might get reset to some meaningless numeric values. In that case, those would have to be renamed manually by going to the "Properties" of each control.

(Thanks to Katy Mulvey who provided this excellent guide down at the devX forum. Link)

Online Stores Shipping to Anywhere in the World

There are millions of online shops to buy almost anything that you need but unfortunately most of them do not ship to exotic locations like Sri Lanka. I've experienced this first hand a lot of times. You go to a site, there's nothing what so ever about shipping limitations. You add stuff to the shopping cart and go to checkout and BLAM! Google checkout says the store doesn't ship to your address. The least that these sites could do is to mention up front that they only ship to the US or Canada or the UK and so on.
I've been looking around for stores that ship worldwide for the past couple of weeks and have listed the ones I've found, below.

Computer Components

Numinix Technology

Numinix Technology is the place to upgrade or build your personal or workspace computer. We offer high quality system building components at extremely low prices. Regardless of your budget, you will always get a great deal and a great system.










Excaliber PC

EXcaliberPC.com Corp., headquartered in Fremont, California, is a leading computer component e-tailer dedicated to providing our valued customers with the best online shopping experience. We strive to make EXcaliberPC.com your one stop shop by offering you the widest selection of products at competitive prices. Unlike other websites that are selling toys, home appliances etc. we focus strictly on the needs of both tech savvy shoppers as well as first time hardware upgrade shoppers. Come and see what made EXcaliberPC.com one of the favorite places to shop online!





Sharp Notebooks


Founded in September 2003, Closo.com (http://www.closo.com) is where your local and international market is even closer™. Closo.com vies to differentiate itself in the e-commerce market by helping buyers find local good, thus increasing trust and reducing fraud because of the face-to-face nature of the trade. Also, Closo is an international site where international items allover the world are displayed in the search results, to increase competition on the site and offer better prices to the buyers. Closo is also very seller-friendly, helping sellers achieve the results they want by helping them with marketing, customer service, fulfillment by providing sellers many advanced tools. Closo is the future of e-commerce both locally and internationally.


Zip Zip Thumb Drives

Zip Zip Memory Bricks are 1 Gigabyte USB 2.0 Memory encased within Lego ® bricks.
Memory Bricks interconnect and stack ontop of each other like normal building blocks.

Memory Bricks retail for $AU59, approximately $US48.
For secure Online Credit Card payment powered by PayPal simply click on a Zip Zip Memory Brick. Alternatively, please email orders@zipzip.net.au if you would like to pay via direct debit, online transfer or cheque.




Books

A1 Overstock (Wholesale only)


A1Overstock is a wholesale-only company. If you are a retail customer, please go to your local bookstore. If you are in the business of selling books, this is the best place to find the best books at the lowest prices!









Electronic Components

Farnell

Farnell, established in 1939, is a world leading high service, low volume distributor of electronic, electrical, industrial and maintenance, repair & operations (MRO) products – with fast, easy access to over 430,000 stocked products, 24 hours a day, 365 days a year.We offer buyers, design and maintenance engineers flexible ordering through a choice of channels, delivery options to suit individual requirements.

24 hour ordering – online
FAST, easy online ordering
FAST same day despatch
FREE technical support

PERLing Your Way Out of Chaos

Managing and analyzing logs can sometimes be a nightmare, specially if the process has been running for hours. Things get more interesting when a bug is discovered and no one has a clue as to where things went wrong. Since the beginning of time man has been using Perl to crawl his way out of the logging wilderness. This, and the reason that the new process I'm writing for out stock trading app is going to produce logs amounting to the size of a Tolkein fantasy, prompted me to try my hand at clearing things up using the trusty old scripting language.

My prior experience using Perl is limited to writing a few scripts for the Internet Programming module during my under grad days. So I had to start from scratch. With a lot of help from various tutorials and articles on the web, I finally put together a script which was pretty handy.


Original Log

Some useful links:
PERL string searching with user input
Multi-line string (pattern) matching
Trouble shooting file handling related issues
Guide to metacharacters used in perl regular expressions









After Initial Extraction

After Final Extraction

Memories: Our Trip to Udawalawe National Park

Me, Lanil, Nilanka, Shakila, Yohan, Shiraz and Prasanna