Jason Fox

Icon

programming, products, and pontifications…

Installing MySQL with the InnoDB Plugin

I found after installing MySQL 5.1 that the “fast index creation” feature is not utilized by the default InnoDB storage engine that comes with MySQL 5.1. This is the main reason why I went through the trouble of installing MySQL 5.1 in the first place. :-/ However, have no fear, the InnoDB plugin is here! (ok, that was lame, anyways…)

The InnoDB plugin is a replacement InnoDB storage engine developed with the help of Sun, Google and Percona. It supposedly provides better overall performance compared with the default InnoDB storage engine that comes with MySQL and it adds a few new features that I want such as “fast index creation.”

If you’d like to try out the InnoDB plugin you’ll have to recompile MySQL as a pre-compiled binary drop-in is not provided yet for OSX. So, here are the steps for compiling and installing MySQL 5.1 with the InnoDB plugin while maintaining a working MySQL 5.0 installation on your computer.

  1. Download the MySQL 5.1 source distribution (v.5.1.43 for me; Change Platform to: Source Code; Scroll down to the last distribution: Generic Linux (Architecture Independent))
  2. Download the InnoDB plugin source distribution (v.1.0.6 for me)
  3. Create a new install directory for MySQL 5.1: sudo mkdir /usr/local/mysql-5.1.43
  4. Change ownership of the install directory to the mysql user: sudo chown -R mysql /usr/local/mysql-5.1.43
  5. Extract the MySQL source: tar -zxf mysql-5.1.43.tar.gz
  6. Extract the InnoDB plugin source: tar -zxf innodb-1.0.6.tar.gz
  7. Change into the MySQL source directory for storage engines: cd mysql-5.1.37/storage
  8. Remove the version of the InnoDB plugin that MySQL comes with: rm -fr innobase
  9. Replace the InnoDB plugin with the one you downloaded: mv innodb-1.0.6 innobase
  10. Change into the MySQL source root: cd ..
  11. Create the make file: ./configure --prefix=/usr/local/mysql-5.1.43 --with-extra-charsets=complex --enable-thread-safe-client --enable-local-infile --enable-shared --with-plugins=innobase --without-plugin-innodb_plugin --with-federated-storage-engine; note, I’m also including the Federated Storage Engine here in case you find a use for it in the future; NOTE: You will receive the following warning, however, IT DID WORK, so just ignore it:
    configure: WARNING: unrecognized options: --without-plugin-innodb_plugin, --with-federated-storage-engine
  12. Compile MySQL: make; takes about 10 minutes or so
  13. Install MySQL: sudo make install
  14. Change into your install directory: cd /usr/local/mysql-5.1.43
  15. Create the MySQL database: sudo ./bin/mysql_install_db --user=mysql
  16. Change ownership of the var directory to the mysql user: sudo chown -R mysql ./var

For more information see…

If you get stuck you can check out the following resources that I used to complete this process:

Installing Apache2 with SSL Support on Mac OS X

Here are the steps for installing Apache2 with SSL support on your local development machine. Several of our applications require HTTPS for certain actions and I’ve been bitten in the past by not testing this in development. So, here’s how to get Apache up and running with SSL.

Compiling and Installing Apache

  1. Download the latest source distribution (v2.2.15 for me) from: http://httpd.apache.org
  2. Extract the source: tar -xvf httpd-2.2.15.tar.gz
  3. Create an install directory (I put it in /usr/local): sudo makedir /usr/local/apache2
  4. Configure the makefile with the following options: ./configure --prefix=/usr/local/apache2 --enable-ssl --enable-setenvif --enable-proxy --enable-headers
  5. Compile the source code: make
  6. Install apache: sudo make install
  7. Create your self-signed SSL keys by following this tutorial: http://developer.apple.com/internet/serverside/modssl.html
    • Scroll down to the Configuring SSL section and start from there
    • You will need to download mod_ssl to get the sign.sh script mentioned in the tutorial but that’s all you need it for. You’ll notice the mod_ssl site says it’s only for Apache 1.3.X. That is because Apache2 has built in SSL support which we enabled in the makefile configuration above so mod_ssl is no longer needed.
  8. Edit your httpd.conf file: sudo vi /usr/local/apache2/conf/httpd.conf
  9. Add the following configuration to the bottom of the file to proxy all HTTP and HTTPS requests to port 3000, i.e., Rails:
    • This assumes Rails is running on port 3000 on your machine
    • This assumes you put the SSL keys where the tutorial told you to
    • The last tag might already be in your httpd.conf file, if so, no need to repeat it here
      # Apache needs to know you want to accept connections over HTTPS
      Listen 443
      
      SSLCertificateFile /etc/httpd/ssl.key/server.crt
      SSLCertificateKeyFile /etc/httpd/ssl.key/server.key
      
      # Below is optional, but was helpful to me in debugging this setup
      CustomLog logs/ssl_request_log  "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" 
      
      <VirtualHost *:80>
          ServerName localhost
          ServerAlias 127.0.0.1
      
          ProxyPass / <a href="http://localhost:3000/">http://localhost:3000/</a>
          ProxyPassReverse / <a href="http://localhost:3000">http://localhost:3000</a>
          ProxyPreserveHost on
      
      <VirtualHost *:443>
          SSLEngine On
          ServerName localhost
          ServerAlias 127.0.0.1
      
          ProxyPass / <a href="http://localhost:3000/">http://localhost:3000/</a>
          ProxyPassReverse / <a href="http://localhost:3000">http://localhost:3000</a>
          ProxyPreserveHost on
          RequestHeader set X_FORWARDED_PROTO 'https'
      
      <ifmodule ssl_module>
          SSLRandomSeed startup builtin
          SSLRandomSeed connect builtin
      </ifmodule>
      

Auto-starting Apache

If you’d like to have apache start automatically whenever your computer starts do the following:

  1. Create a new plist file for apache with a unique name: sudo vi /Library/LaunchDaemons/org.apache.httpd
  2. Put the following in the file (assuming you installed apache in /usr/local/apache2):
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
        <dict>
            <key>KeepAlive</key>
            <true />
            <key>Label</key>
            <string>org.apache.httpd</string>
            <key>ProgramArguments</key>
            <array>
                <string>/usr/local/apache2/bin/apachectl</string>
                <string>start</string>
            </array>
            <key>RunAtLoad</key>
            <true />
            <key>UserName</key>
            <string>root</string>
            <key>WorkingDirectory</key>
            <string>/usr/local/apache2</string>
        </dict>
    </plist>
    
  3. Test it out with launchd: sudo launchctl load -w /Library/LaunchDaemons/org.apache.httpd; You should see several instances of httpd running if you do a: ps aux | grep httpd

Testing it out

  1. Fire up Rails on whatever port you are forwarding your HTTP/HTTPS requests to
  2. Request an action with HTTP
  3. Request an action with HTTPS

Possible Problems

I encountered the following cryptic error in FireFox when I was testing out an Apache install on a second dev box:

SEC_ERROR_REUSED_ISSUER_AND_SERIAL

I found out by reading through some forums that each SSL certificate needs to have a unique serial number. Since I had followed the above procedure to install Apache and SSL on both my laptop and the second dev box I already had a certificate with the same serial number stored in my browser (for my local machine) as the one on the second dev box. If you follow the above process the serial number of your certificate will be 01. So to fix this do the following on your new certificate (for me it was the one on the dev box):

sudo openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out server.crt

This simply assigns 02 to the second certificate and keeps FireFox happy. You probably will not encounter this problem unless you are testing SSL on multiple machines like I was but I figured I’d mention it anyways. Note, you can see the serial number of a certificate in FireFox by going to:

Firefox > Preferences > Advanced > View Certificates (button)

Then double click on a certificate of interest. Also note that Chrome did not complain about this but FireFox won’t even let you visit a site with a duplicate serial number.

If you get stuck…

Try one of the following resources that I used to get this up and running:

Installiing and Running MySQL 5.1 and MySQL 5.0 simultaneously on Mac OS X

I am writing a few new applications and decided to upgrade to MySQL 5.1 from MySQL 5.0.  However, I have several legacy applications that still require MySQL 5.0.  So, I set out to get them both up and running side-by-side so I could forge a new path forward to 5.1 while not abandoning my 5.0 apps.

Installing MySQL 5.1

  1. Download the source distribution of the latest 5.1 (for me it was 5.1.42)
  2. Decompress it and extract it into a directory
  3. Create a new home for the 5.1 files, I put it in /usr/local/mysql-5.1.42
  4. Next cd into the directory where you extracted the files
  5. Run the following command substituting your install directory created in step 3 above: ./configure --prefix=/usr/local/mysql-5.1.42 --with-extra-charsets=complex --enable-thread-safe-client --enable-local-infile --enable-shared --with-plugins=innobase
  6. Compile the code by running: make; this will take a few minutes (10ish?)
  7. Install into your chosen directory with: sudo make install
  8. Next cd into your install directory: cd /usr/local/mysql-5.1.42
  9. Create the mysql database with: sudo ./bin/mysql_install_db --user=mysql
  10. Change ownership of the var directory (this is the data directory): sudo chown -R mysql ./var
  11. Change ownership of the install directory: sudo chown -R mysql /usr/local/mysql-5.1.42
  12. Create a socket file in your install directory: sudo -u mysql touch mysql.sock; mysql may create this itself on start-up, but, I created it ahead of time
  13. Start the server: sudo -u mysql ./libexec/mysqld --basedir=/usr/local/mysql-5.1.42 --port=6666 --socket=/usr/local/mysql-5.1.42/mysql.sock --user=mysql

Running MySQL 5.1 and 5.0

  1. Edit your my.cnf file and explicitly add: socket=/tmp/mysql.socket; if you do not do this when you start the 5.1 server the socket will be deleted
  2. Create a new my.cnf file in your 5.1 installation root; for me: */usr/local/mysql-5.1.42/; your my.cnf file should look like this:
        [mysqld]
        basedir=/usr/local/mysql-5.1.42/
        port=6666
        socket=/tmp/mysql.5.1.socket
        user=mysql
    
  3. Change ownership of the my.cnf file to the mysql user: sudo chown -R mysql my.cnf
  4. Create a plist file for launchd called something like: com.mysql.mysqld.5.1.plist in /Library/LaunchDaemons; your plist file should look like this:
        <?xml version="1.0" encoding="UTF-8"?>
        <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
        <plist version="1.0">
            <dict>
                <key>KeepAlive</key>
                <true />
                <key>Label</key>
                <string>com.mysql.mysqld.5.1</string>
                <key>ProgramArguments</key>
                <array>
                    <string>/usr/local/mysql-5.1.42/bin/mysqld_safe</string>
                    <string>--defaults-file=/usr/local/mysql-5.1.42/my.cnf</string>
                </array>
                <key>RunAtLoad</key>
                <true />
                <key>UserName</key>
                <string>mysql</string>
                <key>WorkingDirectory</key>
                <string>/usr/local/mysql-5.1.42</string>
            </dict>
        </plist>
    
  5. Change ownership of the plist file to root like so: sudo chown root /Library/LaunchDaemons/com.mysql.mysqld.5.1.plist
  6. Test it out by launching mysql: sudo launchctl load -w /Library/LaunchDaemons/com.mysql.mysqld.5.1.plist; The next time you start up both MySQL 5.0 and 5.1 should be started up. Check the /tmp directory after start-up to ensure both sockets were created successfully.

A few notes about this process

  • When you install from source the layout of the installation is different than when you install from the binary distribution, i.e., the data direectory is var not data and the server binary is in libexec not bin, etc. For more information check out this article.
  • When starting a second instance of mysqld you must specify different values for the following options: port, socket, pid-file, tmpdir, datadir. However, if you specify the basedir you only need to explicitly set values for socket and port as we did in the last step above. For more information check out this article.
  • When you start the mysql client you must specify the socket and port like so: /usr/local/mysql-5.1.42/bin/mysql -P 6666 -S /usr/local/mysql-5.1.42/mysql.sock -u root; Alternatively you can create a hidden .my.cnf file in your home directory and specify which server should be connected to by default when you run the mysql client. Your .my.cnf file should look like this:
        [client]
        port=6666
        socket=/tmp/mysql.5.1.socket
    

Working with Rails

Now that you have two MySQL servers running you need to tell Rails which one you want to use in a more explicit fashion. Here’s an example from my database.yml file:
    # Main database connection for the media_service;
    # Uses MySQL 5.1
    development:
      adapter: mysql
      encoding: utf8
      database: my_development
      username: foo
      password: bar
      socket: /tmp/mysql.5.1.socket
      port: 6666

killall Dock – the most awesome command in the world

Ever have Expose freeze up on you? Command-TAB stops working, Spaces stops working, the Dock stops working, show desktop stops working, show windows stops working; you’re trapped!  Well, if you happen to have a terminal window open or can launch one, fear not!  Simply issue the following command (make sure to use a capital ‘D’) and you’re golden.  No restart required!

$ killall Dock

Share and enjoy!

Backgroundjob (Bj) Won’t Start

Recently I had a problem with Bj where it would not start up.  Nothing was written to the backgroundjob log or Rails log and no exception was being thrown.  To make the problem even stranger, Bj would start-up just fine in development but not in production but worked just fine in production from script/console.  After digging into the Bj code and adding some debug statements I found the problem.

# database.yml
development:
  adapter: mysql
  database: my_development
  username: me
  password: password
  host: localhost
  port: 3306

test:
  adapter: mysql
  database: my_test
  username: me
  password: password
  host: localhost
  port: 3306

production:
  development

Bj was getting an ActiveRecord::ConnectionNotEstablished exception but was swallowing it.  The solution was to explicitly define the production database connection in database.yml.

acts_as_universally_unique

I recently found the need to provide UUIDs for ActiveRecord models in a service that I’m developing.  I wasn’t able to find a suitable soution, so, I rolled my own.  Enter acts_as_universally_unique.  The plugin simply adds a (customizabe) UUID field to all ActiveRecord models that act_as_universally_unique.  I will be adding additional methods (and test cases) to it shortly.

XML-RPC, SOAP and Polymorphism

According to the XML-RPC specification a XML_RPC request may only contain scalar <value>s or non-scalar <struct>s. The specification unfortunately does not provide any standard for encoding the type of data encoded in the <struct>s. This has the side effect of not being able to support polymorphism in service method parameters as it leaves the sever no choice but to rely on the method signature in the API declaration when trying to determine what to instantiate for a given <struct> in the XML-RPC request.

Let’s say you have the following declarations:

class SubscriptionsApi < ActionWebService::API::Base
  api_method(
    :create_subscription,
    :expects => [
      { :customer => Logical::Customer },
      { :payment_method => Logical::PaymentMethod }
    ]
  )
end
module Logical
  class PaymentMethod < ActionWebService::Struct
  end
  class CreditCard < PaymentMethod
    member :card_number, :string
    # ...
  end
  class PayPal < PaymentMethod
    member :login, :string
    # ...
  end
end

Now you want to make a call to the service method and pass either a CreditCard or a PayPal. XML-RPC will encode the request like so:

<methodCall>
  <methodName>create_subscription</methodName>
  <param>
    <struct>
      <member>
        <name>card_number</name>
        <value>4111-1111-1111-1111</value>
      </member>
    </struct>
  </param>
</methodCall>

This provides no type information to the server so the server will attempt to instantiate a Logical::PaymentMethod which will of course not have a card_number member as it’s specific to the CreditCard subclass. SOAP, on the other hand, does encode the parameter types allowing you to utilize this type of polymorphism in your service parameters. Here’s the same request encoded in SOAP.

<?xml version="1.0" encoding="utf-8" ?>
<env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<env:Body>
  <n1:CreateSubscription xmlns:n1="urn:ActionWebService" env:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
    <payment_method xmlns:n2="http://www.ruby-lang.org/xmlns/ruby/type/custom" xsi:type="n2:Logical..CreditCard">
      <card_number xsi:type="xsd:string">1</card_number>
    </payment_method>
  </n1:CreateSubscription>
 </env:Body>
</env:Envelope>

The current implementation of ActionWebService resurrected by datanoise did not support this type of polymorphism in SOAP requests. However, I submitted a patch recently which provides for this functionality. Hopefully it’s accepted. :)

MySQL Allows NULLs Where They Are Not Welcome

I recently came across an annoying bug in MySQL v5.1 (also in 6.0 apparently) that bit me hard, so, I thought I’d post on it in case you are being bitten by the same bug.

If you attempt to update a column that does not allow NULL to NULL, MySQL will set the column’s value to the default value for that column’s data type.  This is true only when you are not running MySQL strict mode.  Here’s an example to illustrate.

mysql> create table null_test (id int not null unique(id), name varchar(25)
null default null);
Query OK, 0 rows affected (0.01 sec)

mysql> show create table null_test;
+-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table     | Create Table                                                                                                                                           |
+-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| null_test | CREATE TABLE `null_test` (
  `id` int(11) NOT NULL,
  `name` varchar(25) default NULL,
  UNIQUE KEY `id` (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |
+-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> insert into null_test (id, name) values (1, 'Jane');
Query OK, 1 row affected (0.00 sec)

mysql> update null_test set id = null;
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0  Changed: 0  Warnings: 0

mysql> select * from null_test where name = 'Jane';
+----+------+
| id | name |
+----+------+
|  0 | Jane |
+----+------+
1 row in set (0.00 sec)

More information about this bug can be found in the bug report submitted Janurary 4, 2008.

Check Constraints and MySQL

Unfortunately, MySQL does not support check constraints out of the box.  This makes the task of enforcing business logic in the database layer difficult, but not impossible.  I recently found this approach to implementing check constraints in MySQL.  It’s not as pretty and clean as I’d like, but, it’s the best approach that I’ve found so far.

Now. why would you want to encode business logic in the database?  Can’t you make due with your ActiveRecord::Validations?

Well, have you ever updated the database directly?  Have you ever called update_attribute on an object?  How about save_with_validation(false)?   Yeah, I thought so.  Read more about why you should treat your database as a fortress in Dan Chak’s recently released book, Enterprise Rails (review coming soon).

Rewriting Sub-selects as Joins

Your RDBMS will usually rewrite your sub-selects behind the scenes as joins.  However, there are times where you’ll want to do this yourself.  For example, past versions of MySQL did not play well with sub-selects.  Here are a couple examples of how to rewrite sub-selects as joins.

select *
from   table_a
where  id not in (select a_id from table_b);
-- can be rewritten as...
select *
from   table_a
left   outer join table_b on table_a.id = table_b.a_id
where  table_b.a_id is null;

select *
from   table_a
where  id in (select a_id from table_b);
-- can be rewritten as...
select *
from   table_a
left outer join table_b on table_a.id = table_b.a_id
where  table_b.a_id is not null;

About

Jason Fox is the Co-Founder of Initiate Commerce, Inc. and the Head of Technology and Development at readMedia, Inc. Jason has over 10 years experience designing and building scalable, internet-based, applications for start-up companies both large and small.

Twitter

    github.com/jfoxny

    • No feed items.

    Recent Wines

    View Jason Fox's profile on LinkedIn
    Jason Fox's Facebook profile