Posts Tagged Website Development

Migrating MySQL to PostgreSQL By Hand

Sep 14

Posted by Scott in Website Development

I recently completed a manual migration of my data for CTDA (an analysis system for team data collected from thebluealliance.net and other sources) to PostgreSQL from MySQL (see my comparison for why). By ‘manual’, I do not mean that I typed the data in by hand – I mean that for each table, I wrote a separate script to transfer the data, instead of writing (or finding online) a single script to copy a whole database. My advice to you, if you are planning to migrate data to PostgreSQL from MySQL, is this: don’t do it by hand.

Let’s assume I would have been unable to find a workable script online to transfer the database for me. This is unlikely at best, but if I couldn’t, I could probably have written one that would cover all the cases needed for my database in about 3 or 4 hours (I’m rather new to postgres). So I sit down one day, write the script, test it a bit, and set it to run overnight, and I’m done.

But that’s not what I did. I decided that it would be ‘easier’ to write a script for each table. Perhaps it saved me some thinking – but the process ended up taking me two weeks, and much more than 4 hours. I would estimate that, because of this delay, CTDA will be released 3 weeks later than it would have been if I had written a good script to do this for me.

Oh, and in case you can’t guess. My advice to those who are planning to migrate from PostgreSQL to MySQL: don’t.

Tags: , , , ,

No Comments

New Additions to Website

Aug 4

Posted by Scott in Website Development

We’ve been trying to get quality content up on the website, emphasizing resources for robotics and programming. I put together version 1 of an article system, which can split a long page into multiple sections based on header tags, and automatically generate a table of contents. We’ve written a few new articles, but we’ve mostly just been trying to get old content online. The current article list can be viewed here. Most recently, I put up Eric’s Subversion Crash Course.

We’ve also been working on a new index page, which would look somewhat better, have more content, and be more inviting overall.

Finally, we should have breadcrumb navigation soon.

Tags: , , ,

No Comments

Creating Basic Breadcrumb Navigation with PHP

Jul 10

Posted by Scott in Website Development

“Breadcrumb navigation” is the feature on many websites (including ours, now), where there is a line of links showing the position of the current page in the overall hierarchy. This often corresponds directly to the URL. For example, if the URL was http://robot.mbhs.edu/resources/web/html, the breadcrumb bar might display something like Team 449 >> Resources >> Website Development >> HTML (except with links). This is not only useful to users, since they can have more navigational tools available, but search engines also like it, because it results in a lot of links to your site within your site. As usual, I’ve put the source code used in the blair robot project website below.

Read the rest of this entry »

Tags: , ,

2 Comments

New UI Uploaded on Website

Jul 5

Posted by Scott in Website Development

After several months of waiting, previewing, fixing, and starting over, the new look of the Team 449 website is almost finished. The beta version has been uploaded, although work continues on it. The main changes that will be made soon are:

  • typesetting
  • the left navbar
  • add statusbar

Improvements are still being made and bugs are still being ironed out, but for now, the site validates as XHTML 1.0 transitional.

It is our intent to eventually create an HTML 5 version that can be viewed by compatible browsers. (CSS3 will be included no matter what browser is being used, since it causes no adverse effects.)

Tags: , , , ,

1 Comment

Writing a Search Engine – Part 2

Jul 2

Posted by Scott in Programming, Website Development

Note: this article is a continuation of a previous article on search engines, and has been continued with part 3.

After a bit over a week coding (and learning various Perl libraries), I have completed stages 1 and 2 of the search, although stage 2, the indexer, could do with a little improvement. Both are written in perl, and as usual, the complete code listings are below. I decided to write the entire spider and indexer in perl and optimize as necessary later on, so that I could get done with the thing and not get bogged down in C code. If the perl turns out not to be fast enough as the site grows, then I plan to port to C. Likewise, the actual search part (stage 3) will be written in PHP to save time. If the PHP is not fast enough, I’ll rewrite it in C – but I expect there to be no problems.

Read the rest of this entry »

Tags: , , , , , , ,

No Comments

Calling a CGI Script from PHP

Jun 28

Posted by Scott in Uncategorized

For any given weird, seemingly pointless action, you will be able to find at least 3 good, interesting reasons for wanting to take that action (provided you look around hard enough). In my case, the action was calling a cgi script from php, and the reason was to implement a good logging system. I run a bugzilla installation here, and I didn’t want to go editing every page to get a logger. I also didn’t want to use Apache’s own loggers, since a MySQL database is much easier to handle than a super-sized file.

My solution was to create an index.php file and modify the htaccess so that every request to the bugzilla installation went through index.php. Then, I could have index.php log the event, and call the appropriate cgi script.

Read the rest of this entry »

Tags: , , , , , , ,

1 Comment

Using Access Control Lists (ACL)

Jun 25

Posted by Scott in Uncategorized

If you’ve been using unix or linux long enough, you’ve probably found yourself wishing for more powerful permissions management. For example, if your web development group has been cooperating with another group on a somewhat sensitive project, you’ve probably wished you could easily set it so that those two groups – and only those two groups – could read and write to certain files. Or, as in my case, wished that you could allow the web server user to read and write to files without having to mess with any group permissions. ACLs are the tool that lets you do that.

Read the rest of this entry »

Tags: , , , ,

No Comments

Writing a Search Engine – Part 1

Jun 24

Posted by Scott in Programming, Website Development

I decided that the website had grown to the point where it needed a search engine. I didn’t want to use a google search or an embedded yahoo search – they look disgusting. I also didn’t want to use any of the third party searching scripts, since most of them were costly and all of them had commercial licenses. I like free software. So I set out to write my own.

General Considerations

Let me start off by saying that what I have below is not a magic, easy solution to writing a search engine. If you are planning to write the world’s “next google”, I have a recommendation: go to http://bing.com – Microsoft’s “next google”. Notice how “copycatted” it looks. Then search around (on google, please) to find out exactly how popular it is. Hint: not very. Microsoft tried and failed. Don’t waste your time. My problem is to build an internal search engine, which only needs to deal with a small number of pages, and is low traffic so it doesn’t have to be super fast. When I told my co-working friends about the project, the responses I got varied from “maniac” to “shoot yourself now rather than afterwards – save some time”. And that’s with a highly simplified version of the problem.

Read the rest of this entry »

Tags: , , , , , , ,

1 Comment

Important Differences in HTML 5 from HTML 4

Jun 21

Posted by Scott in Programming

Realizing that the age of HTML 5 was coming up on us faster than I had thought (I thought it was going to be at least a year before and major browser had full support, now I’m thinking it’ll be about a month), I set out to learn what made it different. Looking at a publication by the W3C that outlined the differences, I realized just how old HTML 4 is. I’ve put together a a complete list of the differences between HTML 5 and HTML 4, and I’ve outlined the important – and best – ones below.

Read the rest of this entry »

Tags: , , , ,

4 Comments

Maintaining a Separate Draft Copy of a Website

Jun 21

Posted by Scott in Programming, Website Development

On websites with traffic outside of a group of coworkers, you may find it desirable to modify a copy of the website, and then periodically upload the new version. Both the main robotics website and TMS (”Team Management System”) are developed apart from the main website, and then the drafts are periodically “pushed” onto the main site.

While this technique may not seem particularly impressive to some, some problems come up when you actually try to implement it. The most major problem is that when you push, all the links are now broken. A link to /draft/page.html needs to become /page.html when the page is pushed, and this is hard to automate. (A simple regexp is not enough: what about favicons and stylesheets?) The more minor problem that comes up is design-based, and depends on how you plan to store your pages. If you use a simple filesystem-oriented storage method, there will be no problem.

Read the rest of this entry »

Tags: , , , , , ,

1 Comment