RSS
 

Sample Page for sending JSON data as string to web-service

05 Dec

Just realised it has been a year since I posted anything on here! Time really flies when you’re busy!

I remember when I was first starting off with jQuery, JSON and all that jazz that I struggled to find some simple examples of how to do things like send data to a web-service.

Here is a quick example of how to send some JSON data (as a single post field) to a web service, using jQuery’s post method.

Keep in mind that if you are posting to another domain you may need to use something called jsonP – but if you are just learning, this is a good starting point!

<!DOCTYPE html>
<html>
    <head>
        <title>JSON Web Service Demo</title>
 
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 
        <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
 
        <script type="text/javascript">
            $(document).ready(function() {
 
                //this code will be added to the onclick event
                $('#save_user_button').click(function() {
 
                    //add the values we need for the API to an object
                    var data = new Object;
 
                    //build the object
                    data.guid                   = $('#guid').val();
                    data.defaultallocation      = $('#projecttype').val();
                    data.email                  = $('#email').val();
                    data.username               = $('#username').val();
                    data.fullname               = $('#fullname').val();
 
                    //turn the object in to a JSON string
                    var myJson = JSON.stringify(data);
 
                    var url = "http://testurl/service/user";
 
                    //call the post
                    $.post(url, {jsonUser : myJson}, function(myResult) {
                        alert(myResult)
                    });
 
                });
 
            });
        </script>
 
    </head>
    <body>
        <div>
            <table>
 
                <tr>
                    <td>Project Type</td>
                    <td><input id="projecttype"/></td>
                </tr>
 
                <tr>
                    <td>GUID</td>
                    <td><input id="guid"/></td>
                </tr>
 
                <tr>
                    <td>User Name</td>
                    <td><input id="username"/></td>
                </tr>
 
                <tr>
                    <td>Full Name</td>
                    <td><input id="fullname"/></td>
                </tr>
 
                <tr>
                    <td>Email</td>
                    <td><input id="email"/></td>
                </tr>
 
            </table>
 
            <input type="button" id="save_user_button" value="Save"/>
 
        </div>
    </body>
</html>
 
 

Slow and Locked MySQL Queries

17 Dec

One of our clients’ servers recently saw a huge drop in speed. None of the hardware had been changed, and while the total number of users has been steadily increasing for the last few years, it has not seen any kind of spike that I thought might cause this type of problem.

In order to try to find the fault or problem, I had to look in many different places and discovered a lot about Linux and MySQL. I enjoy these fact finding missions, even though there are always a lot of dead ends.

I thought I would use this blog as a place to write down all the areas that are worth checking, in case anyone else finds themselves in this situation.

Things to check:

  • The Process List
  • MySQL Slow Query Log
  • Server free disk space
  • Server memory usage
  • Server load
  • Your queries
  • Your indexes

Our server was running on Linux Centos 3, and many of the commands below are Linux only. For executing the MySQL commands and queries I suggest getting something like WebYog’s excellent SQLYog, or using MySQL’s own MySQL Workbench.

Listing active MySQL queries

MySQL contains the ability to show a list of currently running queries. From within MySQL – execute the following query:

SHOW PROCESSLIST;

This will show a list that details which users, hosts, databases are being used to execute queries – how long they have been going for, and what state they are in. Look for queries that have been running for a long time, or have the state of ‘locked’. For more information see here.

Breaking down your queries

It is crucial that you enable slow query logging and it is quite simple to set up. Edit your my.cnf file, and make sure the following is present:

log-slow-queries = /var/log/mysql/mysql-slow.log
long_query_time = 3

(Adjust the log_query_time setting to suit the length of the query you are trying to detect)

Once this is in place, restart the MySQL server and start checking the new log file.

The MySQL command EXPLAIN allows you to see a breakdown of the steps taken by MySQL to get your query results, including information about joins and the sequence of queries called. You can use the information it outputs to help optimise your own queries.

The utility is easy to use. Simply prefix your SELECT statement with the word EXPLAIN. eg.

EXPLAIN
SELECT userName, firstName, lastName
FROM users
WHERE firstName LIKE 'da%';

top

This command shows you a list of applications and services running on the machine, along with some basic information about the resources they are using.

From here you can see the system load, free memory and the % of CPU for each process.

To quit and return to the shell, use CTRL+C.

df -h

This will show you how much free space is on your mounted drives, in human readable form.

[sam@webserver1 /]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              9.8G  3.9G  5.9G  40% /
none                 1014M     0 1014M   0% /dev/shm
/dev/md2               57G   31G   27G  54% /disk1

tail

This is a quick way of looking at the last few lines of a log file. The following will show you the last 10 lines of the mysql log.

tail /var/log/mysql.log

You can also specify that you want to view a specific number of lines by using the -n parameter. eg.

tail -n 50 /var/log/mysql.log

Eventually, after going through lots of checking and tests, we discovered that the server was simply at capacity. It hadn’t been upgraded for 8 years and the number of users had quadrupled during that time – with users from 2 other continents now logging on regularly. The database was on the same hardware as the web-server software and the system was just becoming overloaded.

We managed to speed things up considerably before we were forced to upgrade, mostly by examining the slow-query-log and adding indexes where appropriate. Also, many of our queries were refined and tidied up – so we were definitely not wasting our time running the examinations. It has made me realise that it is probably worth going back over old code every now and then to check for potential improvements – something we only tend to do if there is a client requirement, or an OS upgrade required.

We have now performed a complete upgrade to the latest hardware available, with 8GB of memory, more CPU cores than I suspect are really needed, the latest Centos 5 OS and all the most recent MySQL/Apache and PHP versions – which has made the system run incredibly quickly.

 

Using the database from inside a helper (CodeIgniter)

17 Jun

I have some custom functions that I store in a helper in CodeIgniter.

Some of those functions need to use the database, and to do so you have to explicitly tell the function to load the Code Igniter framework and database class.

Its pretty simple when you know how, but it took quite a while to work it out.

function get_user_full_name($userId) {
 
    //the database functions can not be called from within the helper
    //so we have to explicitly load the functions we need in to an object
    //that I will call ci. then we use that to access the regular stuff.
    $ci=& get_instance();
    $ci->load->database();
 
    //select the required fields from the database
    $ci->db->select('firstName, lastName');
 
    //tell the db class the criteria
    $ci->db->where('userId', $userId);
 
    //supply the table name and get the data
    $query = $ci->db->get('user');
 
    //ensure that there is something in the variable - in case of no match
    $fullName = "";
 
    foreach($query->result() as $row):
 
        //get the full name by concatinating the first and last names
        $fullName = $row->firstName . " " . $row->lastName;
 
    endforeach;
 
    // return the full name;
    return $fullName;
 
}
 

Using Eclipse as a PHP IDE in Ubuntu 9.10

25 Nov

This is actually much easier than any of the guides I found online make out.

Firstly, install Eclipse from the repositories. Then open it up, select your default workspace (if you havent already) and then:

  1. From the Help menu, select “Install new software”.
  2. Click the “Add” button and add the following:
      • Name: Eclipse Updates (Galileo)
      • Location: http://download.eclipse.org/releases/galileo/
      1. Apply your changes and wait for the contents to download.
      2. When all is ready, select your new source from the drop down and look for the “Programming languages” section.
      3. Open it up and add the “PHP Development Tools (PDT) SDK Feature”.
      4. Step through using Next and Finish etc, and after a few moments downloading and installing you now have a PHP ready IDE.
       

      Loading JSON data – the EASY way!

      18 Nov

      json160A few weeks ago I wrote a blog post about using jQuery and JSON to automatically load data from a JSON file in to name-matched controls on an HTML page.

      It is only now that I realise quite how powerful this is!

      The great thing about it is that you do not have to edit your data-load function if you add a new field to the database. Because the jQuery function loops through all the fields in the JSON data, it will automatically insert the values in to the correct fields, providing those fields are named the same as the html form controls.

      Here is the code again for those who dont want to check back.

      $.getJSON("/yourJson.js, function(myJson){
      	$.each(myJson.rows, function(i,item) {
      		 for (prop in item) {
      		 	 $('#' + prop).val(item[prop]);
      		 }
      	});
      });
       

      Installing MyODBC 3.51 on Windows 7 x64

      16 Nov

      mysql2In order to get some of our 32 bit applications working on Windows 7 64 bit edition, we need to provide support for the MySQL 3.51 ODBC connector.

      This is actually quite simple to do, as long as you know what to look for. The ODBC manager that you are able to access from within Windows is not going to work for you, but it turns out that Microsoft provide a 32 bit manager that will.

      To use this manager, run the following command:

      c:\windows\syswow64\odbcad32.exe

      Please note that it is not good enough to just type odbcad32.exe, as there is another file in the path with the same file name (great work MS!). However, once you have opened this application up, it works in the exact same way as if you were running XP.

       

      Setting up a basic web server using Ubuntu 9.10 (Desktop)

      12 Nov

      Ubuntu LogoI have been using Ubuntu 9.10 for a while and wanted to see how it performs as a web server. The GUI interface is very slick and it is an ideal OS for beginners and advanced users alike. I suspect that many people will want to use it as the basis for their webservers.

      I suggest creating a virtual machine with something like VirtualBox for this. It is much more flexible that way, and you can move it to a more powerful machine if you decide you need to.

      Phase 1. Install required applications:

      1. Open ‘Synaptic package manager’ from the System/Administration menu.
      2. Click the ‘Search’ button.
      3. Enter ‘Apache’ and press return.
      4. You will see an item called ‘Apache2′ in the list of results. Click the box to the left to mark it as something you want to install.
      5. Repeat this process for:
        • Php5
        • MySql
        • Php5-mysql
      6. Click the ‘Apply’ button on the toolbar.
      7. This will start the installation process, during which you will be prompted to provide a password for the MySQL root account. Supply a strong password and make sure you do not forget it.
      8. It worked!When everything has finished installing, I suggest rebooting. There are ways of starting the servers without rebooting, but a reboot works just as well and is much easier.
      9. After you have logged back in to Ubuntu, open up Firefox and visit http://localhost.
      10. You should see a page that says “It works!” which means that your webserver is now working.

      Phase 2: Configure Virtual Hosts (So you can have more than one site on your server)

      I have chosen to use the command line to do much of this phase rather than the GUI. For me, it is much easier to access protected system files from the shell rather than use a GUI. However, should you want to use a GUI, you will need to be running with root privileges. You can do this by entering: sudo nautilus in to a command line (and enter your password). However, the instructions below are for using the shell.

      Any time you see the sudo command, that means that you will run the command as root (administrator). Linux is much more secure than Windows thanks to it being locked down by default with a standard set of permissions. It might seem like hassle to keep having to switch to be administrator but it makes things much safer. Deal with it!

      1. Open a new Terminal from Applications / Accessories in the menus.
      2. In the terminal, enter the following commands and review the results so you know what you are doing.
      3. cd /etc/apache2/sites-enabled (this moves you to the folder where you can define different websites)
      4. ls (lower case LS: this will show you a list of files)
      5. sudo cp 000-default 001-mysite (this will copy the default config file so you can use it as the basis for your new site)
      6. Enter your password when prompted (this should only happen once, but if you are prompted again – just re-enter it)
      7. sudo gedit 001-mysite (this will use the gedit text editor to open and allow you to edit the new site config file)
      8. Change the ServerAdmin row to use your email address. ie. ServerAdmin sam@mysite.net
      9. Add a row below with the tag: ServerName followed by the name of your site. I will continue to use mysite.local throughout. ie. ServerName mysite.local
      10. Edit the row DocumentRoot so that the new path is a sub-folder of www called 001-mysite. ie. /var/www/001-mysite
      11. Edit the row <Directory…> so that the new path is also used. ie. <Directory /var/www/001-mysite>
      12. Save the file and closed GEdit.
      13. A working PHP virtual hostWe now need to let your computer know that it IS mysite.local. The easiest way to do this is to edit your hosts file. There is plenty of information online about what this file does.
      14. From the terminal, type: sudo gedit /etc/hosts
      15. Add a new row below 127.0.0.1    localhost that reads: 127.0.0.1     mysite.local (spaces are tabs).
      16. Save the file and close GEdit.
      17. Now we need to create the file structure for where the sites files will live.
      18. Enter: sudo mkdir /var/www/001-mysite (this will create a new folder that will store your php/html files etc)
      19. cd /var/www/001-mysite
      20. sudo gedit index.php (this will open the text editor again and will allow you to create the index page for your site).
      21. In the text editor, enter the following bold text (including all the punctuation): <?php phpinfo(); ?>
      22. Save the file and close.
      23. sudo /etc/init.d/apache2 restart (this restarts the Apache service so that the new virtual host is available)
      24. Now open firefox and visit your new site: http://mysite.local
      25. You should see a whole load of information about your server, and more specifically about your PHP installation.
      26. Thats it! Repeat this phase to add as many virtual hosts as you like.

      You will now almost certainly want to secure your server. If you have got this far then you can at least get your site(s) up and running on your local machine. Ill cover securing your server in another post.

       

      Inserting JSON data in to name-matched controls

      01 Sep

      Thought this might be handy… This loop will insert the values from a JSON object in to text boxes.

      It loops around each key-pair and inserts the value if there is a text box with an ID that matches the key name.

      This is the JSON data: (yourJson.js)

      {"page":1,"total":1,"records":1,"paging":20,"rows":[
      {"clipId":"14792","barcode":"02528","title":"My Project Sample","format":"DigiBeta","source":"1242",
      "runningTime":"14 Mins","location":"Library","timeCode":"10:00:00","cUser":"Sam",
      "cDate":"1 Jan 2007","mUser":"Sam","mDate":"1 Jan 2008"}]}

      This is the form HTML:

      <form>
      	<table>
      		<tr>
      			<th>Clip Id</th>
      			<td><input id='clipId'></td>
      		</tr>
      		<tr>
      			<th>Barcode</th>
      			<td><input id='barcode'></td>
      		</tr>
      		<tr>
      			<th>Title</th>
      			<td><input id='title'></td>
      		</tr>
      		<tr>
      			<th>Format</th>
      			<td><input id='format'></td>
      		</tr>
      		<tr>
      			<th>Source</th>
      			<td><input id='source'></td>
      		</tr>
      		<tr>
      			<th>Running Time</th>
      			<td><input id='runningTime'></td>
      		</tr>
      		<tr>
      			<th>Location</th>
      			<td><input id='location'></td>
      		</tr>
      		<tr>
      			<th>Time Code</th>
      			<td><input id='timeCode'></td>
      		</tr>
      	</table>
      </form>

      This is the jQuery:

      	$.getJSON("/yourJson.js, function(myJson){
      		$.each(myJson.rows, function(i,item) {
      		    for (prop in item) {
      				$('#' + prop).val(item[prop]);
      		    }
      		}); 
      	});
       
       

      HAR panel discussing copyright and file sharing

      26 Aug

      An interesting discussion about file sharing for those interested in DRM, copyright and patents. Most people agree that the business model based around copyright no longer works in the digital age. The question is, what other options do we have, and how many people are going to get sued along the way?

      HAR panel from TorrentFreak on Vimeo.

      For information about HAR (Hacking at random) please see their website: https://wiki.har2009.org/

       

      Creating JSON data from MySQL in PHP

      24 Aug

      Here is a quick post that demonstrates how to create some neat JSON data from MySQL using PHP. The script is really simple, and has been working well for me over the last few weeks.

      If anything is confusing, it might be worth checking out the previous post about creating XML data from PHP. It explains things in more detail and will help if you are not familiar…

      //set up the php headers so that the page doesnt cache etc
      header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");
      header("Last-Modified: " . gmdate( "D, d M Y H:i:s") . " GMT");
      header("Cache-Control: no-cache, must-revalidate");
      header("Pragma: no-cache");
      header("Content-type: text/json");
       
      $jsonArray->page = $page; 
      $jsonArray->total = $totalPages;
      $jsonArray->records = $recordCount; 
       
      $i=0; 
      while ($row = $db->fetch_array($countRows)) {
      	$jsonArray->rows[$i]['id']=$row['userAccountId']; 
      	$jsonArray->rows[$i]['userName']=$row['userName'];
      	$jsonArray->rows[$i]['fullName']=$row['fullName'];
      	$jsonArray->rows[$i]['accessCode']=$row['accessCode'];
      	$jsonArray->rows[$i]['email']=$row['email'];
      	$jsonArray->rows[$i]['addedBy']=$row['addedBy'];
      	$jsonArray->rows[$i]['addedDate']=$row['addedDate'];
      	$jsonArray->rows[$i]['modifiedBy']=$row['modifiedBy'];
      	$jsonArray->rows[$i]['modifiedDate']=$row['modifiedDate'];
      	$i++; 
      } 
       
      echo json_encode($jsonArray);

      The above code will create the following style JSON output.

      {"page":1,"total":1,"records":"3","rows":[
      {"id":"1","userName":"demo","fullName":"Demo User","accessCode":"","email":"demo@myexample.com","addedBy":"1","addedDate":"2009-05-02 00:00:00","modifiedBy":"10","modifiedDate":"2009-06-04 00:00:00"},
      {"id":"2","userName":"sam","fullName":"Sam Lasagne","accessCode":"","email":"sam@myexample.com","addedBy":"2","addedDate":"2009-05-02 00:00:00","modifiedBy":"2","modifiedDate":"2009-08-02 00:00:00"},
      {"id":"10","userName":"adrian","fullName":"Adrian Spaghetti","accessCode":"","email":"adrian@myexample.com","addedBy":"1","addedDate":"2009-05-02 00:00:00","modifiedBy":"2","modifiedDate":"2009-09-01 00:00:00"}
      ]}