Posts Tagged mysql

Migrating MySQL to PostgreSQL By Hand

I recently completed a manual migration of my data for CTDA (an analysis system for team data collected from thebluealliance.net and other sources) to PostgreSQL from MySQL (see my comparison for why). By ‘manual’, I do not mean that I typed the data in by hand – I mean that for each table, I wrote a separate script to transfer the data, instead of writing (or finding online) a single script to copy a whole database. My advice to you, if you are planning to migrate data to PostgreSQL from MySQL, is this: don’t do it by hand.

Let’s assume I would have been unable to find a workable script online to transfer the database for me. This is unlikely at best, but if I couldn’t, I could probably have written one that would cover all the cases needed for my database in about 3 or 4 hours (I’m rather new to postgres). So I sit down one day, write the script, test it a bit, and set it to run overnight, and I’m done.

But that’s not what I did. I decided that it would be ‘easier’ to write a script for each table. Perhaps it saved me some thinking – but the process ended up taking me two weeks, and much more than 4 hours. I would estimate that, because of this delay, CTDA will be released 3 weeks later than it would have been if I had written a good script to do this for me.

Oh, and in case you can’t guess. My advice to those who are planning to migrate from PostgreSQL to MySQL: don’t.

Tags: , , , ,

No Comments

MySQL vs PostgreSQL: Benchmarking Data

After looking into migrating to PostgreSQL, which seems to be a popular pastime among database people (migrating, not looking into), I decided to do my own benchmarks. Here are the results of the simple ones (I have yet to code the complex ones). I wrote all the code in perl, and ran it on quentin. I use InnoDB for mysql, with defaults, and everything default on postgres.

Read the rest of this entry »

Tags: , , , ,

2 Comments

Risky Plans

It’s been a strange week – it seems that I decided, once I was almost done with the search engine, that it was time to start some more crazy projects. The disease is catching – Eric has his own project, not yet approved, that even I consider to be extreme. We won’t go into it here.

My three projects are, in order of dubiousness of completion: an in-house listserv to be hosted on ogodei, an IRC server, and an analysis system for FIRST team data (not yet online). The first two will both be linked to the Eric’s “Team Management System” – which is what makes them especially challenging. The third is designed to find patterns (defined by me as a lack of randomness) in team data, including sponsors, team size, geographic location, and performance at competition. Obviously there are some trivial patterns with which I can test the system: “hey! most teams in the D.C. region go to the D.C. regional!” My hope is to find something slightly more useful than that: “hey! everytime ARL decides to sponsor a team, Microsoft joins them the next year!”

Read the rest of this entry »

Tags: , , , , , , ,

No Comments

Writing a Search Engine – Part 2

Note: this article is a continuation of a previous article on search engines, and has been continued with part 3.

After a bit over a week coding (and learning various Perl libraries), I have completed stages 1 and 2 of the search, although stage 2, the indexer, could do with a little improvement. Both are written in perl, and as usual, the complete code listings are below. I decided to write the entire spider and indexer in perl and optimize as necessary later on, so that I could get done with the thing and not get bogged down in C code. If the perl turns out not to be fast enough as the site grows, then I plan to port to C. Likewise, the actual search part (stage 3) will be written in PHP to save time. If the PHP is not fast enough, I’ll rewrite it in C – but I expect there to be no problems.

Read the rest of this entry »

Tags: , , , , , , ,

No Comments

Calling a CGI Script from PHP

For any given weird, seemingly pointless action, you will be able to find at least 3 good, interesting reasons for wanting to take that action (provided you look around hard enough). In my case, the action was calling a cgi script from php, and the reason was to implement a good logging system. I run a bugzilla installation here, and I didn’t want to go editing every page to get a logger. I also didn’t want to use Apache’s own loggers, since a MySQL database is much easier to handle than a super-sized file.

My solution was to create an index.php file and modify the htaccess so that every request to the bugzilla installation went through index.php. Then, I could have index.php log the event, and call the appropriate cgi script.

Read the rest of this entry »

Tags: , , , , , , ,

1 Comment