After looking into migrating to PostgreSQL, which seems to be a popular pastime among database people (migrating, not looking into), I decided to do my own benchmarks. Here are the results of the simple ones (I have yet to code the complex ones). I wrote all the code in perl, and ran it on quentin. I use InnoDB for mysql, with defaults, and everything default on postgres.
Read the rest of this entry »Posts Tagged Coding
For any given weird, seemingly pointless action, you will be able to find at least 3 good, interesting reasons for wanting to take that action (provided you look around hard enough). In my case, the action was calling a cgi script from php, and the reason was to implement a good logging system. I run a bugzilla installation here, and I didn’t want to go editing every page to get a logger. I also didn’t want to use Apache’s own loggers, since a MySQL database is much easier to handle than a super-sized file.
My solution was to create an index.php file and modify the htaccess so that every request to the bugzilla installation went through index.php. Then, I could have index.php log the event, and call the appropriate cgi script.
Most sites will want to avoid “bad bots” – bots that harm a site more than they help it. Frequent traits of these bots are:
- Not pausing in between requests, causing server overload
- Ignoring robots.txt
- Using the ‘disallow’ lines of robots.txt to get more URLs to look at
- Harvesting email addresses for spam lists
Luckily for the webmaster, each of these traits forces a bot to reveal itself. I’ll focus here on #3, since it is the easiest to exploit, but I’ll also explain how to modify the scripts to take care of bots that don’t happen to use the robots.txt file to get more URLs.
Read the rest of this entry »For a new version of the robotics team history page at /history, I was creating a pure-javascript (for reasons I will not enumerate here), cross-browser (for reasons I need not enumerate here) application to display information about the team in a certain year when that year was clicked. Most of the code was relatively simple (the cross-browser part remains to be done), since I just had to get the information from the document structure and then re-output it, with some of it hidden and modified in an appropriate fashion.
The most difficult part was blessing my application with fades. When I clicked on a year, I wanted to see a short (~1 second) transition in which the current year faded out, and when it was gone, the next year faded in. I chose not to use scriptaculous, which is built on prototype, because the combination is well over 200KB and would dwarf my 15-20KB application. Scriptaculous was overkill (an in my opinion, is overkill for almost anything). As usual, the complete code is displayed at the bottom.
Read the rest of this entry »For the minification scripts mentioned here, I had to write a program that would go through every file of a certain type in a directory and process (in this case, minify) that file. At the heart of this problem was the need to get a directory listing.
The obvious way to get a directory listing in C is to use the system call, as in system("ls");. But this is not only lame, it does not work on all systems, and the result cannot be used very easily. The better way to get a directory listing is to use system calls to the filesystem. A complete example is shown at the bottom of this post, as usual.
The functions used in getting a directory listing are opendir and readdir (note: these functions also exist in php, and the code to get a directory listing is in fact nearly the same in php and c). opendir() can be seen as kind of like a fopen() for directories. Following this analogy, readdir() is like a fgetc() for directories. To get a directory listing, we first open the directory with opendir("/dir/name/here"), and then read the name of each file in the directory with a loop of readdir(directory_handle)s. Read the rest of this entry »
I recently noted that several students in a beginning computer science class at my school had been learning event-driven programming by themselves, without bothering to use google or any textbooks. While I approve of this experimental learning approach in general, in this case, the result was less than good.
The programmers in question wanted to design a system in which, so long as the number of button presses was less than the number of lines in a certain file, pressing the button would retrieve the next line, process it, and print out the result. They did this as follows (I’ve made some modifications for readability):
- button.addActionListener(new MyActionListener());
- Scanner s=new Scanner(new File("values.txt"));
- while(s.hasNextLine()) {
- while(!button_pressed);
- button_pressed=false;
- processLine(s.nextLine());
- }
- private class MyActionListener implements ActionListener {
- public void actionPerformed(ActionEvent e) {
- button_pressed=true;
- }
- }
From the programmer’s standpoint, I suppose this actually is the intuitive way to do it. If the user wasn’t involved, we would just loop until there were no more lines, but since the user is involved, we have to wait for the user to press the button, hence the while loop. Nevertheless, this is a Very Bad way to do it. While in the loop on line 5, we are using 100% CPU, even though all we are doing is waiting. If we want to do another operation at the same time, we are doomed.
Read the rest of this entry »For a program I was writing (GCenter – a productivity tool – saves you time by minimizing the time it takes to load your games), I found it necessary to create a plugin architecture with Java and ClassLoader. (I expect Scout449 to eventually use a similar system, although probably with cleaner code.) My code has several cheap hacks, but remains an effective demonstration of a decent plugin system.
Java lends itself particularly well to plugin systems, since classes are loaded dynamically anyway. The idea of a plugin system is to modify how Java loads things. Normally, Java will load your program from the class files in its directory or jar file. When creating a plugin architecture, you want it to also load classes from different jar files (generally the jar files in a certain directory). The solution is to create your own version of the ClassLoader.
Most of the code below is just to navigate the jar files. In the execution of the application, a PluginReader is created for each jar file in the plugin directory. The read() method is then called (line 39), which first fetches a manifest file named “MANIFEST.GMF” which contains the name the actual plugin class (the class that has information like author’s name, as well as functions for starting the plugin running). This class is then loaded, and the class itself is registered with PluginUtils, so that it shows up as an option on the application screen. Then, all the other classes are loaded via the same method (but they are not registered), so that they can be used by the plugin. Finally, copyFiles is called, which copies the entire contents of the jar file to a temporary directory, so that the plugin can fetch graphics and stuff if it wants (using a path conversion method that takes a jar file path and turns it into a temporary directory path).
The real meat is at the very end: line 206 and on. Here, we define a ClassLoader that takes as an argument to its constructor a JarFile, and can be used to load classes from that JarFile. But the real beauty of the ClassLoader is not simply that we can load classes from a jar file, its that when those classes then want to load more classes or find more resources, it is still done within the context of the jar file. Which means that we don’t have to be 100% thorough, we just have to get the plugin going – Java will be happy to handle the rest for us.
Creating a class is simple. We just have to read in a .class file and call the ClassLoader’s defineClass method – in this instance, using our own createClass method as a proxy (just for convenience).
Minifying CSS and JS
May 27
Many web developers want to make use of large, flashy Javascript libraries that allow fancy effects. (Most such libraries also come with large CSS files). Team 449’s website uses Prototype (an AJAX library) and Scriptaculous (a JS effects library built on top of prototype). While many or most viewers of the website may enjoy the experience, others, who have slower connections or are farther away or have the bad luck to view the website during peak viewing time, get frustrated with the long load time.
The solution to this is twofold. First, install gzipping on the server – I’ll discuss this in a later entry. Second, use a minification program, such as the one Yahoo provides. These programs take the JS or CSS files and remove unnecessary whitespace and comments to decrease the file size. The end result – the load time is often cut in half or better.
Yahoo’s compressor must be called from the command line and told to actually compress the file. Naturally, most people will want to automate this process, so that they don’t have to remember to call the compressor everytime they install a new version of scriptaculous or prototype, or make a change.
One possibility is to set up a serving script that first calls the compressor and then serves the page. This can be done in the htaccess like so:
Options +FollowSymlinks RewriteEngine on RewriteRule ^js/(.*)$ jserve.php/$1 [L] RewriteRule ^style/(.*)$ cserve.php/$1 [L]
This is grossly inefficient. For a medium or high-traffic website, it will completely kill your server. The better solution (and the one we used at the Blair Robot Project) is to have the compressor be called in your crontab, and then have the serving script serve the minified file. The crontab entry would then look like:
*/30 * * * * /var/www/robot/scripts/manage-roboweb.sh
The above entry runs manage-roboweb.sh every 30 minutes. In manage-roboweb.sh, we have a call to a c minification program that automatically finds every file in a specified directory (in this case /var/www/robot/js) that has a specified extension, and processes that file with a call to YUI (the Yahoo compression application).
This can be improved by having the serving script first check to see if the minified file is up-to-date, by checking the last modified times of each. You can also have the managing script do the same, so that the files are only re-minified if they were modified. I’ve placed the code we used for both serving scripts below.
cserve.php
- function minify_version($fn) {
- $min_fn = str_replace(".css", ".min.css", $fn);
- $min_stat = stat($min_fn);
- $norm_stat = stat($fn);
- return $min_stat && $min_stat['mtime'] >= $norm_stat['mtime']
- ? $min_fn
- : $fn
- ;
- }
- if (substr_count($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip')) {
- ob_start("ob_gzhandler");
- } else {
- ob_start();
- }
- $offset=1000*3600*48;//48 hour cache
- header("Expires: ".gmdate("D, d M Y H:i:s",time()+$offset)." GMT");
- header("Cache-Control: max-age=$offset, must-revalidate");
- $gmdate_mod = gmdate('D, d M Y H:i:s', time()) . ' GMT';
- header("Last-Modified: $gmdate_mod");
- header("Pragma: public");
- header("Content-Type: text/css");
- $url=$_SERVER["REQUEST_URI"];
- $fn=minify_version("/var/www/robot/style/" .
- str_replace("/style/","",$url));
- $file=file_get_contents($fn);
- echo $file;
jserve.php
- if (substr_count($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip')) {
- ob_start("ob_gzhandler");
- } else {
- ob_start();
- }
- function minify_version($fn) {
- $min_fn = str_replace(".js", ".min.js", $fn);
- $min_stat = stat($min_fn);
- $norm_stat = stat($fn);
- return $min_stat && $min_stat['mtime'] >= $norm_stat['mtime']
- ? $min_fn
- : $fn;
- }
- $offset=1000*3600*48;//48 hour cache
- header("Expires: ".gmdate("D, d M Y H:i:s",time()+$offset)." GMT");
- header("Cache-Control: max-age=$offset, must-revalidate");
- $gmdate_mod = gmdate('D, d M Y H:i:s', time()) . ' GMT';
- header("Last-Modified: $gmdate_mod");
- header("Pragma: public");
- header("Content-Type: text/javascript");
- $url=$_SERVER["REQUEST_URI"];
- $fn = minify_version("/var/www/robot/js/" .
- str_replace("/js/", "", $url));
- $contents=file_get_contents($fn);
- echo $contents;
Week Five: 1/31/09 – 2/6/09
Feb 12
We took advantage of the first full week of school in nearly a month to get a lot done on the robot. The three-wheeled backerman drive was completed and prepared for transfer onto the robot itself. The dumper and harvester were altered to better manipulate moon rocks within the robot and the catapult was completed and tested in conjunction with the camera. Thanks to this, we now have a much better idea of the catapult’s range and ability. The final parts for the gyro were acquired and construction began on it. Our programmers continued to work diligently on the LabVIEW code and made changes to the website. Meanwhile, the public relations subteam worked on the team’s entries for various awards and provided the programmers with additional content for the website.
Today we could be found at Katherine’s house working in the basement as all of our stuff had been located Monday night (good thing too). We made significant progress on fixing the code, making different possible manipulators to grab the tubes (we’re going to try them out once we get back to school, where we have a model rack), and fabricating the ramp. It was quite a day!
