Monday, December 24, 2007

Utilities Updates

IrfanView has hit version 4.10. Yeah, old news, you might say, but I just noticed it myself! Download the entire plugins setup package.


Programmer's Notepad 2 has hit v2.0.7.706-devel.



The best parts:
  • It has PyPN which allows users to write their own extensions in Python.
  • It finally supports Bash syntax highlighting!
Python 2.5.1 has been released. Python 3000 alpha 2 is also ready for download and evaluation.

OpenOffice has reached 2.3.1.


This is JUST-IN! XAMPP has reached version 1.6.5 on 24 December 2007.

Thursday, December 20, 2007

Attachment Extractor

I was looking for a way to store all the attachments from an email onto disk in Thunderbird, without any success. Normal programs like Outlook (Express) shows you a list of attached files that you can select and drag-drop to the desktop, but this was plain missing in Thunderbird. Hmm, interesting.

Anyway, I decided to do a quick search and found the following right away.

Extracts all attachments from selected messages and then can delete, detach or mark-read.

Just select the messages containing the attachments you want to extract and select 'Extract Attachments' and this extension does the rest. No more having to go through each message to extract the attachments!
Check out the settings dialog for the different options.

by Andrew Williamson (eviljeff)

Friday, December 14, 2007

Interesting Snippets

Ruckus Wireless Technology Review

Ruckus Wireless Technology Review at Gadget Center talks about new wireless access point and wireless client products from Singaporean company Ruckus Wireless which promises to solve the problem of low quality IPTV over wireless transmission with a combination of technologies called BeamFlex and SmartCast.

Joel Spolsky's Talk at Yale

Joel has published three articles - transcripts of a speech he gave at at the Yale Computer Science department - describing his experiences as a Computer Science student, his various job stints, right up to starting his own business.

Part 1 | Part 2 | Part 3

Thursday, December 06, 2007

Software Gems

I feel tempted to update this Comsul blog today because I made some discoveries this week while at work.

... darn ... now I have forgotten what they are... alright, let me try to refresh my memory ...

Safari 3 Public Beta (now available for Windows) I like the default font, though it doesn't looked like the Times New Roman that the settings page seems to say. Guess this is why it is still a beta?
The fastest web browser on any platform, Safari loads pages up to 2 times faster than Internet Explorer 7 and up to 1.7 times faster than Firefox 2.
Well of course this is what is found on Apple's website. Look at what ArsTechnica has to say about it.

A friend recommended me PHP on Trax when I asked about a PHP framework with object persistence. My first experience with object persistence was with SQLObject in Python.

Well, I just came back from a search which lead me to Wikipedia page that contains a list of object-relational mappers for PHP and various other languages. There is a whole lot of choices!

Microsoft Internet Explorer Inline Search from IEforge
is an extremely useful free add-on for Internet Explorer that mimics Firefox's search behavior. It turns searching in a web page into a non-modal research experience coupled with a find as you type facility.
I've just installed WINE on the development servers for generating software release packages that require some processing in Windows. Since all our servers are Linux based, using WINE allows us to add functionality for Windows-based software without having to bear the extra cost of installing extra Windows servers. Even though there are still fatal bugs, especially with running Cygwin applications (you might be thinking who would want to run Cygwin on WINE/Linux!!!) and it doesn't run well with older libraries, when it works, it is a gem to have. For those who have sneered with disdain at WINE in the past, now is probably the time to start looking at it.

Update: I have been running Wine with varies success on various systems. Sometimes, even running "wine cmd" would cause Wine to crash and throw exceptions, specifically with libpthread. Anyone has any idea why this is?

Thursday, November 01, 2007

I am Famous!

Well, that is to the people who subscribe to the Embedded Muse newsletter by Jack Ganssle.

The following is an excerpt of his 151st edition of the newsletter. The "engineer" was me! I mentioned this to him during his Better Firmware Faster seminar in Singapore on the 18 and 19 October 2007.

Heh, Jack Ganssle actually learnt something from ME! :-)

On an unrelated note, an engineer recently referred me to another of
Joel's posts that I had somehow missed:
http://www.joelonsoftware.com/articles/fog0000000043.html . It's
Joel's test of a healthy software organization. Do read the article,
but he believes we should all be able to answer "yes" to these
questions:

- Do you use source control?
- Can you make a build in one step?
- Do you make daily builds?
- Do you have a bug database?
- Do you fix bugs before writing new code?
- Do you have an up-to-date schedule?
- Do you have a spec?
- Do programmers have quiet working conditions?
- Do you use the best tools money can buy?
- Do you have testers?
- Do new candidates write code during their interview?
- Do you do hallway usability testing?

I'm a bit unconvinced about the last question for most embedded apps,
but these others are spot-on.

Thursday, October 25, 2007

Five easy ways to fail

Joel, I love you! I identify with EVERYTHING you say. Please, please come to Singapore and teach our people here - especially the marketing types - a thing or two.

Wednesday, October 24, 2007

Netrino - The Embedded Systems Reference

Just a quick post to document a fabulous resource I've found related to embedded systems programming. Netrino offers a Glossary and Articles of extremely high quality.

Saturday, October 20, 2007

Better Firmware Faster

I attended the 2-day Better Firmware Faster seminar conducted by renown embedded systems guru Jack Ganssle, organized by OmniScient International over the past 2 days.

To help lay-persons get an idea of how this feels, imagine being in the same room with Jacky Cheung of Cantopop, Pete Sampras of tennis, Paul McCartney of English pop, etc. You get what I mean.

My greatest achievement was staying awake (and alert!) for the entire duration of the course despite the lunch buffet, sumptuous morning and afternoon tea breaks!

The seminar was not cheap, but learning the cumulative experience of 30 years of product development experience in 2 days was worth every cent! I forked out my own money (4 four figure sum) to attend the seminar when the budget wasn't approved. (When will they learn that free seminars are the ones NOT worth attending??!!) It is such a rare opportunity to attend this seminar since Jack does not travel to Singapore often.

I'm quite surprised (more disappointed rather) that the turnout at this seminar was quite low - only 11 people attended - I heard the turnout in Penang, the next stop, is twice the number. I would expect that many more people would recognize the value in attending such a seminar. A few reasons I can offer:
  1. Bad publicity - insufficient promotion was done for the event. I came to know of it only because I subscribe to Jack's The Embedded Muse
  2. Lack of firmware practitioners in Singapore - we are a dying trade in Singapore, even though (I believe) demand for firmware is increasing.
  3. Singapore engineers are too full of themselves - they believe that no one can teach them how to develop firmware better
  4. Employers, managers and engineers alike are too myopic - they don't think the seminar is worth the meager amount that would have gone into the directors' road taxes; they believe in motivating their engineers by threatening their increments and bonuses, and increasing productivity by management fiat; they believe the only way to ship quality firmware faster is to keep screwing in the schedules, working overtime, feverishly pounding keyboards, pressuring vendors and individual heroics (tools, systems, reviews are just a waste of time).
Notice my longest rant above is about the employer mindsets. :-)

I feel sad for firmware/embedded software in Singapore. It is the most expensive thing in the universe to produce, but it is only fetching the price of the hardware. Since when did you PAY for firmware upgrades? Since when did you pay MORE for a device because it can do more? Fresh graduates coming out of 'world class' universities think Java is the only programming language in the world. How many programmers can read schematics? Why are huge companies who are eager to expand their firmware/software team not willing to pay the premium for scarce brilliant, experienced practitioners?

Anyway, the wisdom to be gained from this seminar does not just apply to firmware. A lot of the best practices can be applied to all areas of software development in general. Simply attending the seminar does increase your efficiency and effectiveness overnight - you have to understand and FOLLOW the advice presented. There is no one-size-fits-all solution. Every company, team, domain has it's own peculiarities and it is up to YOU to find out and do what fits YOU.

Seeing that this is a chance not to be missed - he says he is retiring (please Jack, stay and teach us more!). Immediately after the seminar ended, I took out my camera and asked for a photo with Jack.

This one was taken without flash. (notice Jack's grin).
This one was taken with flash.

I also follow Joel Spolsky's blog on software development. This guy is funny and makes a lot of sense. A lot of things Jack mentioned during his seminar ring a bell with what I hear from Joel, even though they put it across in completely different ways. The underlying wisdom is the same! Two great minds cannot be wrong!

At the end of the two days, I get rewarded with this:


A word to young firmware engineers and managers out there: There is a lot of wisdom and experience you can gain by listening to others. Why learn all these things the hard way when you can learn all of these NOW, and start being more productive and efficient? People like Jack and Joel make a lot of sense. Listen to them! Kudos to you guys!

P.S. Joel, if you do have time, please swing by Singapore to hold a seminar. I'll attend for sure!

Tuesday, August 28, 2007

Installed Ubuntu

I've tried a Linux distribution apart from Fedora. Ubuntu 7.04! True to it's name, it's really easy to install and use. No more excuses NOT to run Linux.

This is posted with Firefox on Ubuntu.

Sunday, July 29, 2007

My Linux System

I've upgraded my Linux system from Fedora Core 5 to Fedora 7 overnight with the help of the Internet and apt-get. Just one night with apt-get has made me abandon YUM. Now my system is spanking new, and the install/upgrade process also tidied up my system files in the process! E.g. /etc/fstab, /etc/modprobe.conf, /etc/grub.conf, etc.

1. Asus Pundit-R
2. Intel P4 3.0GHz Hyper-Threading
3. 512MB DDR SDRAM
4. 60GB Western Digital (IDE, 8MB cache, 3.5")
5. Samsung 40x CD-RW (IDE)
6. Fedora 7 (2.6.21-1.3228.fc7)

Here are some of my prevous computer related posts that I put in another blog. I'll just move them here for consistency sake.

Motive Certification



TR-069: DSL1000EU is Motive Baseline Certified Just received confirmation that our DSL1000EU has been Motive certified. Next step is to get DSL605EW the same certification as well. SSL verification need to be done properly.Hoping to get Siemens verification behind us - hope to do so next week.

DSL605EW is Motive certified. 'Nuff said.


A PUT script in Python+Apache



I finally managed to write a PUT script properly.

Initially I thought it was simply reading the file data from stdin and writing the contents out to a filename that is retrieved from the environment. Well, it used to be that simple on Unix/Linux. On Windows it is a lot more complicated.

  1. In Windows, file streams are in TEXT mode, even stdin!
  2. To fix this, "import msvcrt", msvcrt(sys.stdin.fileno(), os.O_BINARY)
  3. How do you know you are running Windows?
  4. "if sys.platform=='win32':"


Here is the code in all its glory:


#!"C:\Python24\Python.exe"
import os, sys, time
if sys.platform=='win32': import msvcrt

apacheHelp = '''This directive adds an action, which will activate cgi-script when a file is requested using the method of method. The cgi-script is the URL-path to a resource that has been designated as a CGI script using ScriptAlias or AddHandler. The URL and file path of the requested document is sent using the standard CGI PATH_INFO and PATH_TRANSLATED environment variables.'''
def reportError(errMsg):
print 'Content-Type: text/plain\r\n\r\n'
print errMsg

def debugEnv():
print 'Content-Type: text/plain\r\n\r\n'
for x in os.environ.keys():
print x,'=',os.environ[x]

def getenv(name):
try:
return os.environ[name]
except KeyError:
return ''

def writeFile(path, srcFD):
dstFD = open(path, 'wb')
numWrite = 0
while True:
buf = srcFD.read(1024)
if len(buf)==0: break
dstFD.write(buf)
dstFD.flush()
numWrite += len(buf)
dstFD.close()
return numWrite

# debugEnv()
def main():
pathInfo = getenv('PATH_INFO')
if (not pathInfo.startswith('/uploads/')):
reportError('Disallowed upload path ' + pathInfo)
sys.exit(1)
if pathInfo.endswith('/'):
reportError('Cannot upload file as a directory!')
sys.exit(2)

if sys.platform=='win32': msvcrt.setmode(sys.stdin.fileno(), (os.O_BINARY))
size = writeFile(getenv('PATH_TRANSLATED'), sys.stdin)

reportError('')
print '%d bytes written to %s\n' % (size, pathInfo)

if '__main__'==__name__:
main()



Rare error on Google?




Thursday, July 26, 2007

Linux Server Performance Tuning

For the past 2 years, I've had 2 Dell PowerEdge 800 machines running at unacceptably sub-standard performance. They were primarily used as cross compile machines for our embedded Linux and RTOS systems.

For the record , they were bought in mid 2005, and were running
  • 3.2GHz with Hyper-Threading
  • 1GB DDR SDRAM
  • 160GB SATA-1 hard disk
  • Broadcom Gigabit Ethernet
Although by today's standards, 1GB of RAM was not a big deal, but that was quite impressive back in those days.

Anyway, to cut a long story short, I had intended to install RedHat 9 on these machines since this was what the cross compiler suite was developed on, and I knew better than to try than to rock the boat.

Problem #1: RedHat 9 install CD did not support SATA disks and GigaBit cards (there were no such things when RH9 was released). Simple, you'd say - install onto an IDE drive, a compatible Fast Ethernet card, recompile the kernel, remove the compatible hardware, and get back to business! This leads us to ...

Problem #2: Our MIS did not want to crack open the case; they controlled the inventory and didn't want us pesky R&D personnel messing around with the stuff, lest I make off with the RAM or hard disk or something.

So I had to install using the latest Fedora install CD at that time, which I think was Fedora Core 3. I realized that I did not need to install Fedora Core 3 fully. I booted the CD in rescue mode such that the SATA and Gigabit Ethernet were up and running, created and formatted the filesystem on the hard disk, and then copied using NFS, the entire root filesystem of an existing RedHat 9 installation which I had done earlier on an old IBM machine with an IDE drive and a 'normal' Ethernet card. In the process, I had to update e2fsprogs to handle the newer filesystem formatted with Fedora Core 3.

The system seemed to run ok, the kernel moving from 2.4.26 to 2.4.33. In our compilation work, we had to deal with large source trees, including a full Linux kernel source tree, tens of applications, some big, some small, and uClibc. To handle this behemoth (67806 files) of a source
tree, we used SubVersion as the Version Control System instead of CVS.

We hosted the SubVersion repositories on a central server, an older P4 2.4GHz which used to be the only compilation system until we got the PowerEdge 800's.

Running svn update took an average of 5 minutes. Looking back now, I could not imagine how we could have accepted and lived through that kind of performance. But for tens of thousands of files, we thought it was an acceptable performance. Little did I realize how much injustice I was doing Linux!

Among the many things I did to try to improve performance:
  • Installed a Gigabit Ethernet Card on the SVN server.
  • Connected all three machines with a Gigabit switch.
  • Updated the kernel to 2.4.33 (hoping for better disk and network drivers)
  • Installed the latest versions of SubVersion,
  • Installed latest versions of SAMBA and SSH (disable DNS reverse lookups)
The one nagging problem that still remains is the very slow performance when running svn update on the source tree. This is not all, because the whole system slows to a crawl when somebody is doing a compilation. Further more, accessing shared drives via SAMBA would always hang Windows Explorer or the application that is accessing the SAMBA files (like text editors and source analyzers) for half a minute. This is hardly like a state of the art Linux server, which is still considered pretty fast in recent times (3.2GHz).

Last year-end, we bought a new server to take over the role of a source code server in anticipation of the increasing space and computational demands of our firmware development efforts. I had Fedora Core 6 installed on the machine and used LVM to manage the filesystems on the hard disks. (Yes , two hard disks!) I did all the needful, copied RH9 into one of the partitions, did a chroot and used it to compile firmware. On this machine, the performance was surprisingly fast, even though it had a slower CPU (2.8GHz Pentium D). In particular, 'svn update' (which became my definitive benchmarking test) mostly took around 1 second to complete on the same source tree (67k files).

Doing 'strace svn update' shows that the system was hanging on random file accesses. svn was creating, locking and removing thousands of lock files while performing the update command. I copied the FC6 filesystem to one of the PowerEdge 800 machines and booted off that thinking that perhaps the older kernel was really lacking some good SATA driver (2.4.33 had libata 1.2, which 2.6.18 had libata 2.0). To my dismay, nothing changed - same OS, same kernel - different performance. svn update was averaging 40 seconds to complete.

Analysis ruled out the effects of differences in CPU speed and RAM size. I discovered that the new machine had SATA-2 while the 'older' machines only had SATA-1. However, these differences could not explain the huge discrepancy between the two systems. I suspected two things: (a) The on-board SATA controller was under-performing, or (b) the hard disk was under-performing. Running badblocks on the hard disk showed nothing.

I got another hard disk installed in the older machine (same controller) today, partitioned, formatted, mounted, and checked-out the same source tree, and the results:
  1. First svn update: 1m (this is fine, since the partition was freshly mounted)
  2. 2nd svn update: 1s !!!
  3. 3rd svn update: 0.7s !!!

So, (a) controller is alright, so (b) had to be somewhat true. I remember that I had partitioned the older machines way back in 2005, and the new one in 2006/2007. Surely there had to be some changes either in the ext2/ext3. Using dumpe2fs indeed showed an obvious difference - one of those head-slapping moments ... the journal size on the old machine was 32MB (on a 150GB partition) while the newer machine had it at 128MB.

I reclaimed the original swap partition on the original hard disk, reformatted with the latest mke2fs (-j option) and re-tested. Speedy gonzales! Mounting the partition as ext2 also had no speed issues. I re-created the journal on the compile partition (the 150GB one) with 'tune2fs -O ^has_journal' followed by 'tune2fs -j -J size=128' and voila! excellent performance. While at 1.5Gbps, is not as fast as SATA-2's 3Gbps, it is good enough for me.

During lunch the same day, I expanded the journal of the other old machine still running 2.4.33 and had the same performance boost already. My colleagues came to feedback that the system felt faster and more responsive.

And I agree!


I later learnt, that setting up and using NFS was still not as spiffy as using rsync with ssh. Using Rsync, you just needed to install and run an SSH server. No complex configuration files to mess around with, because SSH configuration files work well right out of the box.

Broadband 'speed race'? No real need for one


Who is this guy kidding? "There is no need for anything above 20Mbps". This is like the misquoted Bill Gates adage "640kB of RAM should be enough for everybody".

Using current infrastructure, SingTel cannot offer anything more than 20Mbps. (ADSL2+ has a top speed of 25Mbps at short distances, which is rarely achievable in reality). There is a gap that needs to be filled, and SingTel is fast losing ground.

They are not just losing ground, they are also losing profits - with the sky high broadband prices enabled by the monopoly that SingTel holds over the phone lines in the country. Only the emergence and deployment of competing technologies like DOCSIS 2.0 and 3.0 provide the incentive for SingTel to upgrade their services. Otherwise, they would tell you 512kbps is enough for everybody. What's on the horizon? Bonded ADSL, VDSL2 and fibre.

Why is Starhub offering 100Mbps? Why are countries like Korea offering 8Mbps as early back as 2004? YahooBB in Japan is offering 8 and 12Mbps back in 2003!

His argument is misleading because a computer can receive data from multiple websites at once. Connecting to 5 different websites, each with capacities to deliver 20Mbps, is enough to max out a 100Mbps Internet connection.

Popular foreign content, like Yahoo, CNN, maybe even YouTube, are cached in local servers by content accelerators like Akamai and LimeLight. Most of the foreign content that you surf will already be coming from local servers.

Speculative downloading in the background is another feature supported by web browsers and applications like Adobe Acrobat Reader to download pages that are linked from the current page in anticipation of the user visiting those links in the next. Often there will be 10 or more links per document. Having higher bandwidth makes for a snappier user experience.

more to come...

This is StarHub's rebuttal:

Higher broadband speeds benefit consumers

I REFER to the letter by SingTel's Mr Peter Heng, 'Broadband 'speed race'? No real need for one' (ST, July 24).

In spite of Mr Heng's assertion that 'any speed beyond 20Mbps today cannot be feasibly used by individual customers', our research indicates that many homes in Singapore now use multiple computers to access the Internet simultaneously over a single connection.

It is also very common for some users to have multiple, concurrent applications and downloads.

Therefore, there will always be a clear, present need for broadband speeds (including beyond 20Mbps) that offer better performance to each user in a home network.

We believe this is also why the global broadband industry is evolving the home networking technology to the newer IEEE802.11n standard, such that consumers will be able to enjoy these higher speeds wirelessly.

According to the Infocomm Development Authority of Singapore's online publication, Consumer Information On Residential Broadband Services In Singapore (http://www.ida.gov.sg/Publications/20061213184450.aspx), with higher broadband speeds consumers enjoy better value and benefit more from multiple Internet connections over a single broadband service, as well as a single Internet connection that supports several concurrent online applications.

We agree with Mr Heng that Internet content for consumers in Singapore is largely hosted overseas but we wish to highlight that the nature of the public Internet is such that it will always be a shared infrastructure, and the access is only as good as the weakest and slowest link. StarHub ensures that the weakest link is never within the local-access network.

StarHub ensures there is sufficient bandwidth capacity within the network to deliver the speeds that our customers are paying for. Our cable broadband service, with the new state-of-the-art channel-bonding technology, deployed over fibre-optic and coaxial cables, delivers higher broadband speeds of up to 100Mbps nationwide. No other country in the world offers these speeds to households.

In addition, our cable broadband network uses dedicated frequencies/channels for the individual delivery of video, data and voice services without compromising quality.

Jeannie Ong (Ms)
Head
Corporate Communications & Investor Relations
StarHub
This was the original article by Alfred Siew that started it all (way to go Alfred!):

SINGAPORE: SingTel takes a breather in speed race

SingTel is considering a network upgrade to keep pace with competitors

The Straits Times
Friday, July 20, 2007

By Alfred Siew

Singtel has scaled back its bid to offer the fastest broadband service here for the time being, as it mulls over a network upgrade to keep pace with rivals.

Top-end services -- costing more than $100 a month -- attract heavy users like gamers who crave the extra bandwidth for smoother action on screen.

But after a year of intense competition, SingTel is rethinking this niche market.

The telecom operator stopped selling its 25 megabits per second (mbps) service as a standalone offering a few months ago. It is available now only as part of a $150 bundle of services, including cellphone calls.

Mr Allen Lew, SingTel's chief executive officer for Singapore, cited "low demand and take-up from customers" as reasons for this.

However, the company will continue providing the 25mbps service to existing subscribers, who pay $125 a month.

Mr Lew argued that most home users do not need speeds of more than 20mbps because websites overseas -- where 85 per cent of traffic heads to -- have a cap on how fast users can download.

"We're not saying we are not competing on speed, all we're saying is that any speed above 20mbps is irrelevant in the context of what consumers are using broadband Internet for," he said.

The next fastest service from SingTel now offers 10mbps.

This leaves the high-end market open to rival StarHub, which launched a 100mbps service last December.

Mr Thomas Ee, StarHub's senior vice-president for cable, fixed and Internet protocol services, told The Straits Times it is ready to boost speeds up to 160mbps "when the demand picks up."

The company had said previously that faster speeds would benefit users with multiple computers at home.

SingTel and StarHub are neck and neck in their share of the home broadband market, which also includes smaller players like MobileOne.

Despite not pushing for more speed now, SingTel has been testing other technologies, like fibre optic cables, that support faster speeds, over the past year. But these require running new cables into homes.

Ultra-fast broadband has seen some early success in places like Hong Kong, where videos and other online content are available locally.

Date Posted: 7/20/2007

Monday, May 21, 2007

Considerations when choosing an NAS

Size

Are you just running a small home business or are you running a whole enterprise?

Security
  1. Are you dealing with sensitive data?
  2. Do you (or your associates/customers) access your data remotely?
  3. If you answer yes to BOTH, then you need:
  4. Secure protocols like HTTPS/SFTP/SSH (or any form of encrypted access method).
  5. Do NOT use plain FTP, or HTTP with basic authentication (these allow your passwords to be intercepted).
Speed

You can never get fast enough.
  1. USB 2.0 high speed (480Mbps) is a MUST.
  2. Gigabit Ethernet is preferred. (Fast ethernet is too slow)
  3. Serial-ATA (SATA) support is preferred.
Space
  1. If you have lots of data, but they are not very important, then you need a lot of space.
    1. Example: large photos and videos that you want to share for a short period of time, and you have the originals somewhere else.
Reliability
  1. If you have critical data that you cannot afford to lose then you need reliability. You need to dedicate some portion of the space for redundant storage (i.e. backups).