My Linux System

I've upgraded my Linux system from Fedora Core 5 to Fedora 7 overnight with the help of the Internet and apt-get. Just one night with apt-get has made me abandon YUM. Now my system is spanking new, and the install/upgrade process also tidied up my system files in the process! E.g. /etc/fstab, /etc/modprobe.conf, /etc/grub.conf, etc.

1. Asus Pundit-R
2. Intel P4 3.0GHz Hyper-Threading
3. 512MB DDR SDRAM
4. 60GB Western Digital (IDE, 8MB cache, 3.5")
5. Samsung 40x CD-RW (IDE)
6. Fedora 7 (2.6.21-1.3228.fc7)

Here are some of my prevous computer related posts that I put in another blog. I'll just move them here for consistency sake.

Motive Certification

TR-069: DSL1000EU is Motive Baseline Certified Just received confirmation that our DSL1000EU has been Motive certified. Next step is to get DSL605EW the same certification as well. SSL verification need to be done properly.Hoping to get Siemens verification behind us - hope to do so next week.

DSL605EW is Motive certified. 'Nuff said.

A PUT script in Python+Apache

I finally managed to write a PUT script properly.

Initially I thought it was simply reading the file data from stdin and writing the contents out to a filename that is retrieved from the environment. Well, it used to be that simple on Unix/Linux. On Windows it is a lot more complicated.

1. In Windows, file streams are in TEXT mode, even stdin!
2. To fix this, "import msvcrt", msvcrt(sys.stdin.fileno(), os.O_BINARY)
3. How do you know you are running Windows?
4. "if sys.platform=='win32':"
   

Here is the code in all its glory:

#!"C:\Python24\Python.exe"
import os, sys, time
if sys.platform=='win32': import msvcrt

apacheHelp = '''This directive adds an action, which will activate cgi-script when a file is requested using the method of method. The cgi-script is the URL-path to a resource that has been designated as a CGI script using ScriptAlias or AddHandler. The URL and file path of the requested document is sent using the standard CGI PATH_INFO and PATH_TRANSLATED environment variables.'''
def reportError(errMsg):
 print 'Content-Type: text/plain\r\n\r\n'
 print errMsg

def debugEnv():
 print 'Content-Type: text/plain\r\n\r\n'
 for x in os.environ.keys():
  print x,'=',os.environ[x]

def getenv(name):
 try:
  return os.environ[name]
 except KeyError:
  return ''

def writeFile(path, srcFD):
 dstFD = open(path, 'wb')
 numWrite = 0
 while True:
  buf = srcFD.read(1024)
  if len(buf)==0: break
  dstFD.write(buf)
  dstFD.flush()
  numWrite += len(buf)
 dstFD.close()
 return numWrite

# debugEnv()
def main():
 pathInfo = getenv('PATH_INFO')
 if (not pathInfo.startswith('/uploads/')):
  reportError('Disallowed upload path ' + pathInfo)
  sys.exit(1)
 if pathInfo.endswith('/'):
  reportError('Cannot upload file as a directory!')
  sys.exit(2)

 if sys.platform=='win32': msvcrt.setmode(sys.stdin.fileno(), (os.O_BINARY))
 size = writeFile(getenv('PATH_TRANSLATED'), sys.stdin)

 reportError('')
 print '%d bytes written to %s\n' % (size, pathInfo)

if '__main__'==__name__:
 main()

Rare error on Google?

Linux Server Performance Tuning

For the past 2 years, I've had 2 Dell PowerEdge 800 machines running at unacceptably sub-standard performance. They were primarily used as cross compile machines for our embedded Linux and RTOS systems.

For the record , they were bought in mid 2005, and were running

• 3.2GHz with Hyper-Threading
• 1GB DDR SDRAM
• 160GB SATA-1 hard disk
• Broadcom Gigabit Ethernet
  

Although by today's standards, 1GB of RAM was not a big deal, but that was quite impressive back in those days.

Anyway, to cut a long story short, I had intended to install RedHat 9 on these machines since this was what the cross compiler suite was developed on, and I knew better than to try than to rock the boat.

Problem #1: RedHat 9 install CD did not support SATA disks and GigaBit cards (there were no such things when RH9 was released). Simple, you'd say - install onto an IDE drive, a compatible Fast Ethernet card, recompile the kernel, remove the compatible hardware, and get back to business! This leads us to ...

Problem #2: Our MIS did not want to crack open the case; they controlled the inventory and didn't want us pesky R&D personnel messing around with the stuff, lest I make off with the RAM or hard disk or something.

So I had to install using the latest Fedora install CD at that time, which I think was Fedora Core 3. I realized that I did not need to install Fedora Core 3 fully. I booted the CD in rescue mode such that the SATA and Gigabit Ethernet were up and running, created and formatted the filesystem on the hard disk, and then copied using NFS, the entire root filesystem of an existing RedHat 9 installation which I had done earlier on an old IBM machine with an IDE drive and a 'normal' Ethernet card. In the process, I had to update e2fsprogs to handle the newer filesystem formatted with Fedora Core 3.

The system seemed to run ok, the kernel moving from 2.4.26 to 2.4.33. In our compilation work, we had to deal with large source trees, including a full Linux kernel source tree, tens of applications, some big, some small, and uClibc. To handle this behemoth (67806 files) of a source
tree, we used SubVersion as the Version Control System instead of CVS.

We hosted the SubVersion repositories on a central server, an older P4 2.4GHz which used to be the only compilation system until we got the PowerEdge 800's.

Running svn update took an average of 5 minutes. Looking back now, I could not imagine how we could have accepted and lived through that kind of performance. But for tens of thousands of files, we thought it was an acceptable performance. Little did I realize how much injustice I was doing Linux!

Among the many things I did to try to improve performance:

• Installed a Gigabit Ethernet Card on the SVN server.
• Connected all three machines with a Gigabit switch.
• Updated the kernel to 2.4.33 (hoping for better disk and network drivers)
• Installed the latest versions of SubVersion,
  
• Installed latest versions of SAMBA and SSH (disable DNS reverse lookups)

The one nagging problem that still remains is the very slow performance when running svn update on the source tree. This is not all, because the whole system slows to a crawl when somebody is doing a compilation. Further more, accessing shared drives via SAMBA would always hang Windows Explorer or the application that is accessing the SAMBA files (like text editors and source analyzers) for half a minute. This is hardly like a state of the art Linux server, which is still considered pretty fast in recent times (3.2GHz).

Last year-end, we bought a new server to take over the role of a source code server in anticipation of the increasing space and computational demands of our firmware development efforts. I had Fedora Core 6 installed on the machine and used LVM to manage the filesystems on the hard disks. (Yes , two hard disks!) I did all the needful, copied RH9 into one of the partitions, did a chroot and used it to compile firmware. On this machine, the performance was surprisingly fast, even though it had a slower CPU (2.8GHz Pentium D). In particular, 'svn update' (which became my definitive benchmarking test) mostly took around 1 second to complete on the same source tree (67k files).

Doing 'strace svn update' shows that the system was hanging on random file accesses. svn was creating, locking and removing thousands of lock files while performing the update command. I copied the FC6 filesystem to one of the PowerEdge 800 machines and booted off that thinking that perhaps the older kernel was really lacking some good SATA driver (2.4.33 had libata 1.2, which 2.6.18 had libata 2.0). To my dismay, nothing changed - same OS, same kernel - different performance. svn update was averaging 40 seconds to complete.

Analysis ruled out the effects of differences in CPU speed and RAM size. I discovered that the new machine had SATA-2 while the 'older' machines only had SATA-1. However, these differences could not explain the huge discrepancy between the two systems. I suspected two things: (a) The on-board SATA controller was under-performing, or (b) the hard disk was under-performing. Running badblocks on the hard disk showed nothing.

I got another hard disk installed in the older machine (same controller) today, partitioned, formatted, mounted, and checked-out the same source tree, and the results:

1. First svn update: 1m (this is fine, since the partition was freshly mounted)
2. 2nd svn update: 1s !!!
   
3. 3rd svn update: 0.7s !!!

So, (a) controller is alright, so (b) had to be somewhat true. I remember that I had partitioned the older machines way back in 2005, and the new one in 2006/2007. Surely there had to be some changes either in the ext2/ext3. Using dumpe2fs indeed showed an obvious difference - one of those head-slapping moments ... the journal size on the old machine was 32MB (on a 150GB partition) while the newer machine had it at 128MB.

I reclaimed the original swap partition on the original hard disk, reformatted with the latest mke2fs (-j option) and re-tested. Speedy gonzales! Mounting the partition as ext2 also had no speed issues. I re-created the journal on the compile partition (the 150GB one) with 'tune2fs -O ^has_journal' followed by 'tune2fs -j -J size=128' and voila! excellent performance. While at 1.5Gbps, is not as fast as SATA-2's 3Gbps, it is good enough for me.

During lunch the same day, I expanded the journal of the other old machine still running 2.4.33 and had the same performance boost already. My colleagues came to feedback that the system felt faster and more responsive.

And I agree!


I later learnt, that setting up and using NFS was still not as spiffy as using rsync with ssh. Using Rsync, you just needed to install and run an SSH server. No complex configuration files to mess around with, because SSH configuration files work well right out of the box.

Broadband 'speed race'? No real need for one

Who is this guy kidding? "There is no need for anything above 20Mbps". This is like the misquoted Bill Gates adage "640kB of RAM should be enough for everybody".

Using current infrastructure, SingTel cannot offer anything more than 20Mbps. (ADSL2+ has a top speed of 25Mbps at short distances, which is rarely achievable in reality). There is a gap that needs to be filled, and SingTel is fast losing ground.

They are not just losing ground, they are also losing profits - with the sky high broadband prices enabled by the monopoly that SingTel holds over the phone lines in the country. Only the emergence and deployment of competing technologies like DOCSIS 2.0 and 3.0 provide the incentive for SingTel to upgrade their services. Otherwise, they would tell you 512kbps is enough for everybody. What's on the horizon? Bonded ADSL, VDSL2 and fibre.

Why is Starhub offering 100Mbps? Why are countries like Korea offering 8Mbps as early back as 2004? YahooBB in Japan is offering 8 and 12Mbps back in 2003!

His argument is misleading because a computer can receive data from multiple websites at once. Connecting to 5 different websites, each with capacities to deliver 20Mbps, is enough to max out a 100Mbps Internet connection.

Popular foreign content, like Yahoo, CNN, maybe even YouTube, are cached in local servers by content accelerators like Akamai and LimeLight. Most of the foreign content that you surf will already be coming from local servers.

Speculative downloading in the background is another feature supported by web browsers and applications like Adobe Acrobat Reader to download pages that are linked from the current page in anticipation of the user visiting those links in the next. Often there will be 10 or more links per document. Having higher bandwidth makes for a snappier user experience.

more to come...

This is StarHub's rebuttal:

Higher broadband speeds benefit consumers

I REFER to the letter by SingTel's Mr Peter Heng, 'Broadband 'speed race'? No real need for one' (ST, July 24).

In spite of Mr Heng's assertion that 'any speed beyond 20Mbps today cannot be feasibly used by individual customers', our research indicates that many homes in Singapore now use multiple computers to access the Internet simultaneously over a single connection.

It is also very common for some users to have multiple, concurrent applications and downloads.

Therefore, there will always be a clear, present need for broadband speeds (including beyond 20Mbps) that offer better performance to each user in a home network.

We believe this is also why the global broadband industry is evolving the home networking technology to the newer IEEE802.11n standard, such that consumers will be able to enjoy these higher speeds wirelessly.

According to the Infocomm Development Authority of Singapore's online publication, Consumer Information On Residential Broadband Services In Singapore (http://www.ida.gov.sg/Publications/20061213184450.aspx), with higher broadband speeds consumers enjoy better value and benefit more from multiple Internet connections over a single broadband service, as well as a single Internet connection that supports several concurrent online applications.

We agree with Mr Heng that Internet content for consumers in Singapore is largely hosted overseas but we wish to highlight that the nature of the public Internet is such that it will always be a shared infrastructure, and the access is only as good as the weakest and slowest link. StarHub ensures that the weakest link is never within the local-access network.

StarHub ensures there is sufficient bandwidth capacity within the network to deliver the speeds that our customers are paying for. Our cable broadband service, with the new state-of-the-art channel-bonding technology, deployed over fibre-optic and coaxial cables, delivers higher broadband speeds of up to 100Mbps nationwide. No other country in the world offers these speeds to households.

In addition, our cable broadband network uses dedicated frequencies/channels for the individual delivery of video, data and voice services without compromising quality.

Jeannie Ong (Ms)
Head
Corporate Communications & Investor Relations
StarHub

This was the original article by Alfred Siew that started it all (way to go Alfred!):

SINGAPORE: SingTel takes a breather in speed race

SingTel is considering a network upgrade to keep pace with competitors

The Straits Times
Friday, July 20, 2007

By Alfred Siew

Singtel has scaled back its bid to offer the fastest broadband service here for the time being, as it mulls over a network upgrade to keep pace with rivals.

Top-end services -- costing more than $100 a month -- attract heavy users like gamers who crave the extra bandwidth for smoother action on screen.

But after a year of intense competition, SingTel is rethinking this niche market.

The telecom operator stopped selling its 25 megabits per second (mbps) service as a standalone offering a few months ago. It is available now only as part of a $150 bundle of services, including cellphone calls.

Mr Allen Lew, SingTel's chief executive officer for Singapore, cited "low demand and take-up from customers" as reasons for this.

However, the company will continue providing the 25mbps service to existing subscribers, who pay $125 a month.

Mr Lew argued that most home users do not need speeds of more than 20mbps because websites overseas -- where 85 per cent of traffic heads to -- have a cap on how fast users can download.

"We're not saying we are not competing on speed, all we're saying is that any speed above 20mbps is irrelevant in the context of what consumers are using broadband Internet for," he said.

The next fastest service from SingTel now offers 10mbps.

This leaves the high-end market open to rival StarHub, which launched a 100mbps service last December.

Mr Thomas Ee, StarHub's senior vice-president for cable, fixed and Internet protocol services, told The Straits Times it is ready to boost speeds up to 160mbps "when the demand picks up."

The company had said previously that faster speeds would benefit users with multiple computers at home.

SingTel and StarHub are neck and neck in their share of the home broadband market, which also includes smaller players like MobileOne.

Despite not pushing for more speed now, SingTel has been testing other technologies, like fibre optic cables, that support faster speeds, over the past year. But these require running new cables into homes.

Ultra-fast broadband has seen some early success in places like Hong Kong, where videos and other online content are available locally.

Date Posted: 7/20/2007