In order to try to find the fault or problem, I had to look in many different places and discovered a lot about Linux and MySQL. I enjoy these fact finding missions, even though there are always a lot of dead ends.

I thought I would use this blog as a place to write down all the areas that are worth checking, in case anyone else finds themselves in this situation.

Things to check:

• The Process List
• MySQL Slow Query Log
• Server free disk space
• Server memory usage
• Server load
• Your queries
• Your indexes

Our server was running on Linux Centos 3, and many of the commands below are Linux only. For executing the MySQL commands and queries I suggest getting something like WebYog’s excellent SQLYog, or using MySQL’s own MySQL Workbench.

Listing active MySQL queries

MySQL contains the ability to show a list of currently running queries. From within MySQL – execute the following query:

SHOW PROCESSLIST;

This will show a list that details which users, hosts, databases are being used to execute queries – how long they have been going for, and what state they are in. Look for queries that have been running for a long time, or have the state of ‘locked’. For more information see here.

Breaking down your queries

It is crucial that you enable slow query logging and it is quite simple to set up. Edit your my.cnf file, and make sure the following is present:

log-slow-queries = /var/log/mysql/mysql-slow.log
long_query_time = 3

(Adjust the log_query_time setting to suit the length of the query you are trying to detect)

Once this is in place, restart the MySQL server and start checking the new log file.

The MySQL command EXPLAIN allows you to see a breakdown of the steps taken by MySQL to get your query results, including information about joins and the sequence of queries called. You can use the information it outputs to help optimise your own queries.

The utility is easy to use. Simply prefix your SELECT statement with the word EXPLAIN. eg.

EXPLAIN
SELECT userName, firstName, lastName
FROM users
WHERE firstName LIKE 'da%';

top

This command shows you a list of applications and services running on the machine, along with some basic information about the resources they are using.

From here you can see the system load, free memory and the % of CPU for each process.

To quit and return to the shell, use CTRL+C.

df -h

This will show you how much free space is on your mounted drives, in human readable form.

[sam@webserver1 /]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              9.8G  3.9G  5.9G  40% /
none                 1014M     0 1014M   0% /dev/shm
/dev/md2               57G   31G   27G  54% /disk1

tail

This is a quick way of looking at the last few lines of a log file. The following will show you the last 10 lines of the mysql log.

tail /var/log/mysql.log

You can also specify that you want to view a specific number of lines by using the -n parameter. eg.

tail -n 50 /var/log/mysql.log

Eventually, after going through lots of checking and tests, we discovered that the server was simply at capacity. It hadn’t been upgraded for 8 years and the number of users had quadrupled during that time – with users from 2 other continents now logging on regularly. The database was on the same hardware as the web-server software and the system was just becoming overloaded.

We managed to speed things up considerably before we were forced to upgrade, mostly by examining the slow-query-log and adding indexes where appropriate. Also, many of our queries were refined and tidied up – so we were definitely not wasting our time running the examinations. It has made me realise that it is probably worth going back over old code every now and then to check for potential improvements – something we only tend to do if there is a client requirement, or an OS upgrade required.

We have now performed a complete upgrade to the latest hardware available, with 8GB of memory, more CPU cores than I suspect are really needed, the latest Centos 5 OS and all the most recent MySQL/Apache and PHP versions – which has made the system run incredibly quickly.

Some of those functions need to use the database, and to do so you have to explicitly tell the function to load the Code Igniter framework and database class.

Its pretty simple when you know how, but it took quite a while to work it out.

function get_user_full_name($userId) {
 
    //the database functions can not be called from within the helper
    //so we have to explicitly load the functions we need in to an object
    //that I will call ci. then we use that to access the regular stuff.
    $ci=& get_instance();
    $ci->load->database();
 
    //select the required fields from the database
    $ci->db->select('firstName, lastName');
 
    //tell the db class the criteria
    $ci->db->where('userId', $userId);
 
    //supply the table name and get the data
    $query = $ci->db->get('user');
 
    //ensure that there is something in the variable - in case of no match
    $fullName = "";
 
    foreach($query->result() as $row):
 
        //get the full name by concatinating the first and last names
        $fullName = $row->firstName . " " . $row->lastName;
 
    endforeach;
 
    // return the full name;
    return $fullName;
 
}

Firstly, install Eclipse from the repositories. Then open it up, select your default workspace (if you havent already) and then:

1. From the Help menu, select “Install new software”.
2. Click the “Add” button and add the following:
   • Name: Eclipse Updates (Galileo)
   • Location: http://download.eclipse.org/releases/galileo/
3. Apply your changes and wait for the contents to download.
4. When all is ready, select your new source from the drop down and look for the “Programming languages” section.
5. Open it up and add the “PHP Development Tools (PDT) SDK Feature”.
6. Step through using Next and Finish etc, and after a few moments downloading and installing you now have a PHP ready IDE.

It is only now that I realise quite how powerful this is!

The great thing about it is that you do not have to edit your data-load function if you add a new field to the database. Because the jQuery function loops through all the fields in the JSON data, it will automatically insert the values in to the correct fields, providing those fields are named the same as the html form controls.

Here is the code again for those who dont want to check back.

$.getJSON("/yourJson.js, function(myJson){
	$.each(myJson.rows, function(i,item) {
		 for (prop in item) {
		 	 $('#' + prop).val(item[prop]);
		 }
	});
});

This is actually quite simple to do, as long as you know what to look for. The ODBC manager that you are able to access from within Windows is not going to work for you, but it turns out that Microsoft provide a 32 bit manager that will.

To use this manager, run the following command:

c:\windows\syswow64\odbcad32.exe

Please note that it is not good enough to just type odbcad32.exe, as there is another file in the path with the same file name (great work MS!). However, once you have opened this application up, it works in the exact same way as if you were running XP.

I suggest creating a virtual machine with something like VirtualBox for this. It is much more flexible that way, and you can move it to a more powerful machine if you decide you need to.

Phase 1. Install required applications:

1. Open ‘Synaptic package manager’ from the System/Administration menu.
2. Click the ‘Search’ button.
3. Enter ‘Apache’ and press return.
4. You will see an item called ‘Apache2′ in the list of results. Click the box to the left to mark it as something you want to install.
5. Repeat this process for:
   • Php5
   • MySql
   • Php5-mysql
6. Click the ‘Apply’ button on the toolbar.
7. This will start the installation process, during which you will be prompted to provide a password for the MySQL root account. Supply a strong password and make sure you do not forget it.
8. When everything has finished installing, I suggest rebooting. There are ways of starting the servers without rebooting, but a reboot works just as well and is much easier.
9. After you have logged back in to Ubuntu, open up Firefox and visit http://localhost.
10. You should see a page that says “It works!” which means that your webserver is now working.

Phase 2: Configure Virtual Hosts (So you can have more than one site on your server)

I have chosen to use the command line to do much of this phase rather than the GUI. For me, it is much easier to access protected system files from the shell rather than use a GUI. However, should you want to use a GUI, you will need to be running with root privileges. You can do this by entering: sudo nautilus in to a command line (and enter your password). However, the instructions below are for using the shell.

Any time you see the sudo command, that means that you will run the command as root (administrator). Linux is much more secure than Windows thanks to it being locked down by default with a standard set of permissions. It might seem like hassle to keep having to switch to be administrator but it makes things much safer. Deal with it!

1. Open a new Terminal from Applications / Accessories in the menus.
2. In the terminal, enter the following commands and review the results so you know what you are doing.
3. cd /etc/apache2/sites-enabled (this moves you to the folder where you can define different websites)
4. ls (lower case LS: this will show you a list of files)
5. sudo cp 000-default 001-mysite (this will copy the default config file so you can use it as the basis for your new site)
6. Enter your password when prompted (this should only happen once, but if you are prompted again – just re-enter it)
7. sudo gedit 001-mysite (this will use the gedit text editor to open and allow you to edit the new site config file)
8. Change the ServerAdmin row to use your email address. ie. ServerAdmin sam@mysite.net
9. Add a row below with the tag: ServerName followed by the name of your site. I will continue to use mysite.local throughout. ie. ServerName mysite.local
10. Edit the row DocumentRoot so that the new path is a sub-folder of www called 001-mysite. ie. /var/www/001-mysite
11. Edit the row <Directory…> so that the new path is also used. ie. <Directory /var/www/001-mysite>
12. Save the file and closed GEdit.
13. We now need to let your computer know that it IS mysite.local. The easiest way to do this is to edit your hosts file. There is plenty of information online about what this file does.
14. From the terminal, type: sudo gedit /etc/hosts
15. Add a new row below 127.0.0.1    localhost that reads: 127.0.0.1     mysite.local (spaces are tabs).
    
16. Save the file and close GEdit.
17. Now we need to create the file structure for where the sites files will live.
18. Enter: sudo mkdir /var/www/001-mysite (this will create a new folder that will store your php/html files etc)
    
19. cd /var/www/001-mysite
    
20. sudo gedit index.php (this will open the text editor again and will allow you to create the index page for your site).
21. In the text editor, enter the following bold text (including all the punctuation): <?php phpinfo(); ?>
22. Save the file and close.
23. sudo /etc/init.d/apache2 restart (this restarts the Apache service so that the new virtual host is available)
    
24. Now open firefox and visit your new site: http://mysite.local
25. You should see a whole load of information about your server, and more specifically about your PHP installation.
26. Thats it! Repeat this phase to add as many virtual hosts as you like.

You will now almost certainly want to secure your server. If you have got this far then you can at least get your site(s) up and running on your local machine. Ill cover securing your server in another post.

HAR panel from TorrentFreak on Vimeo.

For information about HAR (Hacking at random) please see their website: https://wiki.har2009.org/

Google have launched a new service called “Google Voice” which is basically an application for your PC or your mobile which works via the internet connection, that then behaves like a telephone.

They supply you with a new phone number, for free, and then you add all of your own phone numbers to it. If someone calls your Google phone number – all of your phones will ring. This means that you will only ever need to give out that one number ever again! This is very useful, especially if you work internationally, as a freelancer or change mobiles often.

The service is only available at the moment go Google GrandCentral users, but should be launching soon to the general public.

The only reason I even heard about this service is that Apple have now blocked the sale of the Google Voice application from their iPhone’s App store, claiming the reason: “it duplicates functionality available with the iPhone” – which surely translates as: “Our partners such as O2 and AT&T will be forced to do something about their prices and they really dont like it“…

Google Voice is a service that gives you one number for all your phones, voicemail that is easy as email, and many enhanced calling features like call blocking and screening, voicemail transcripts, call conferencing, international calls, and more.

Google Voice is currently available for GrandCentral users only, but will be open to new users soon. In the meantime, please leave us your email address and we’ll notify you as soon as Google Voice becomes available. To learn more about Google Voice, check out our feature videos.

Im sure there are probably a few problems with the software at the moment, but knowing Google they will work at it until they have an unrivaled service, just like with their Search tool and Apps. Not sure how they will pay for this, but I heard a roumor that they may eventually play adds during calls?

I’m looking forward to trying it out.

Following the tutorials on the developer.joomla.org website was actually very straight forward and I have been able to create a simple component of my own. I will be working on this over the next few days / weeks and will release it to the open source community when its done.

For those who haven’t heard of Joomla!, it is a template based web content management system. Once you get the hang of things you can use it to create entire websites in no time at all. It comes with loads of built in functionality and best of all, you dont need to know how to do any programming. Its almost as easy to create a web page as it is to type an email.

Aptana Studio IDE

UPDATE: With the latest version of Aptana 2 and Ubuntu 9.10 it seems you dont need to do this hack anymore. Try it without first of all, then if you get problems – apply the fixes as described.

I have been using the Aptana IDE for PHP and Ajax development for a while now on my Windows system, and have decided that it is much better than the standard Eclipse, at least for me. I wanted to start using it on my Linux machine but encountered problems which prevented it from running.

It took me a while to find out how to get it to work, but thanks to Google and some others who have had this problem before me, I now have a working Aptana Studio on my Ubuntu system.

Here is how to get it working:

1. Download and extract Aptana Studio for Linux (All in one) from here.
2. Use Synaptic to install xulrunner (version 1.8).
3. Use Synaptic to install a JRE.
4. Create a text file (in your home folder etc) which we will use to set up the Aptana environment and execute the application. Call it startAptana.
5. Paste the text from below in to the new text file, and be sure to change the bottom line so that it matches the path to your Aptana installation.
6. Make the startAptana file executable.
7. Double click the startAptana file to start using Aptana, or create menu icon that links to it.

#!/bin/sh
MOZILLA_FIVE_HOME=/usr/lib/xulrunner
if [ $LD_LIBRARY_PATH ]; then
   LD_LIBRARY_PATH=$MOZILLA_FIVE_HOME:$LD_LIBRARY_PATH
else
   LD_LIBRARY_PATH=$MOZILLA_FIVE_HOME
fi
export MOZILLA_FIVE_HOME LD_LIBRARY_PATH
/home/sam/Apps/aptana/AptanaStudio